Programmatic SEO: Managing Millions of URLs

Programmatic SEO involves creating large numbers of pages targeting specific keyword patterns. Sites using this strategy often have hundreds of thousands or millions of URLs---presenting unique sitemap challenges.

The Scale Challenge

Consider a real estate site with pages for every city, neighborhood, and property type combination. The math adds up quickly:

• 50,000 cities x 3 property types = 150,000 URLs
• Add neighborhoods and you're at 500,000+ URLs
• Add filters and variations: millions of URLs

Strategy 1: Incremental Updates

Instead of regenerating your entire sitemap on every change, update incrementally:

// Only sync changed URLs
const changedUrls = await getUrlsModifiedSince(lastSyncTime);

await fetch('/api/sitemap/generate', {
  body: JSON.stringify({
    urls: changedUrls,
    replace: false, // Append, don't replace
  }),
});

Strategy 2: Database-Driven Generation

For very large sites, generate sitemaps directly from your database in batches:

const BATCH_SIZE = 50000;
let offset = 0;

while (true) {
  const urls = await db.query(
    'SELECT url, updated_at FROM pages LIMIT ? OFFSET ?',
    [BATCH_SIZE, offset]
  );

  if (urls.length === 0) break;

  await syncToSitemapHost(urls, { replace: offset === 0 });
  offset += BATCH_SIZE;
}

Strategy 3: Priority-Based Segmentation

Not all pages are equally important. Segment your sitemap by priority:

• High priority: Main category pages, updated daily
• Medium priority: Individual listing pages, updated weekly
• Low priority: Archive pages, updated monthly

Monitoring at Scale

With millions of URLs, monitoring becomes critical:

• Track index coverage in Search Console
• Monitor crawl stats for anomalies
• Set up alerts for sitemap errors
• Regularly audit for broken/removed URLs

Starting a new programmatic SEO project? Use our free sitemap generator to prototype your sitemap structure, or audit an existing sitemap for errors before scaling up.