Technical SEO
robots.txt and XML sitemaps for SaaS websites
Configure robots.txt and XML sitemaps without leaking private routes or accidentally blocking the public pages that should rank.
robots.txt and an XML sitemap solve different problems. One gives crawlers instructions. The other gives search engines a clean list of canonical public URLs. Neither file protects a private page.
For SaaS sites, the common failure is mixing the marketing site with the logged-in product. Public guides belong in the sitemap. Account, billing, internal API, and dashboard routes usually do not.
Use robots.txt for crawl guidance, not access control
A disallowed path is still public. Anyone can read robots.txt, and a URL may still be indexed from links elsewhere. Protect sensitive routes with authentication and authorization. Use noindex on a public page when the goal is to keep that page out of search results.
- Keep production robots.txt available at the root hostname.
- Do not block CSS or JavaScript needed to render public pages.
- Reference the canonical sitemap URL.
- Review broad rules such as Disallow: / before every launch.
Build a sitemap from pages worth indexing
Include canonical, indexable URLs that return a successful response. Exclude dashboard screens, search results, temporary campaign variants, redirected URLs, and routes that carry private identifiers.
The sitemap should agree with the page. A URL marked noindex or canonicalized to another page should not be listed as if it were a primary result.
Separate product and marketing surfaces
If the app and marketing site share one host, make ownership explicit. The marketing build can generate the sitemap while product routes default to noindex where appropriate. If they use separate hosts, give each host its own crawl rules and only publish a sitemap for content that should be found.
- Homepage, pricing, feature pages, comparisons, and useful guides are typical sitemap entries.
- Login may be crawlable but rarely needs high sitemap priority.
- Authenticated dashboards should not rely on robots.txt for privacy.
- Staging hosts should require access control and stay outside public sitemaps.
Test the files after deployment
Fetch both files from the public domain, follow a sample of sitemap URLs, and check their status, canonical, and robots directives. Then submit the sitemap through your search engine tooling and watch for excluded or redirected entries.
Check your production site
See which of these issues are visible now.
ScanMySaaS checks the public surface and keeps the affected URL with every finding. Verify results before changing production.
Run a free scan