Technical SEO
robots.txt and XML sitemaps for SaaS websites
Configure robots.txt and XML sitemaps without leaking private routes or accidentally blocking the public pages that should rank.
robots.txt and an XML sitemap solve different problems. One gives crawlers instructions. The other gives search engines a clean list of canonical public URLs. Neither file protects a private page.
For SaaS sites, the common failure is mixing the marketing site with the logged-in product. Public guides belong in the sitemap. Account, billing, internal API, and dashboard routes usually do not.
Use robots.txt for crawl guidance, not access control
A disallowed path is still public. Anyone can read robots.txt, and a URL may still be indexed from links elsewhere. Protect sensitive routes with authentication and authorization. Use noindex on a public page when the goal is to keep that page out of search results.
- Keep production robots.txt available at the root hostname.
- Do not block CSS or JavaScript needed to render public pages.
- Reference the canonical sitemap URL.
- Review broad rules such as Disallow: / before every launch.
Build a sitemap from pages worth indexing
Include canonical, indexable URLs that return a successful response. Exclude dashboard screens, search results, temporary campaign variants, redirected URLs, and routes that carry private identifiers.
The sitemap should agree with the page. A URL marked noindex or canonicalized to another page should not be listed as if it were a primary result.
Discovery, crawling and indexing are different decisions
Googlebot discovers URLs through links and sitemaps, checks robots.txt before fetching pages, then can read canonical and noindex signals on allowed pages. A sitemap does not override robots.txt; blocking crawling can prevent a noindex directive from being seen.
Separate product and marketing surfaces
If the app and marketing site share one host, make ownership explicit. The marketing build can generate the sitemap while product routes default to noindex where appropriate. If they use separate hosts, give each host its own crawl rules and only publish a sitemap for content that should be found.
- Homepage, pricing, feature pages, comparisons, and useful guides are typical sitemap entries.
- Login may be crawlable but rarely needs high sitemap priority.
- Authenticated dashboards should not rely on robots.txt for privacy.
- Staging hosts should require access control and stay outside public sitemaps.
Fix "Sitemap contains URLs which are blocked by robots.txt"
Search Console reports this when a URL listed in your sitemap matches a Disallow rule in robots.txt. The two files disagree: the sitemap asks for the page to be indexed while robots.txt tells crawlers not to fetch it. Decide which one is right for each URL.
If the page should rank, remove or narrow the Disallow rule that matches it (a broad rule such as Disallow: /app or a wildcard often catches public pages by accident). If the page should stay out of search, remove it from the sitemap instead; to keep it out of the index entirely, allow crawling and serve a noindex directive, because a blocked page cannot show its noindex.
A related warning is a sitemap that robots.txt does not reference. Add a Sitemap: line with the absolute URL of your canonical sitemap, for example Sitemap: https://www.example.com/sitemap.xml, and make sure that URL returns 200 on the same host and protocol as your canonical pages.
- Test each reported URL against your robots.txt rules (most specific matching rule wins).
- Keep only canonical, indexable 200 URLs in the sitemap; drop redirects and noindex pages.
- Reference the sitemap in robots.txt with an absolute URL on the canonical host.
- Resubmit the sitemap in Search Console and validate the fix.
Test the files after deployment
Fetch both files from the public domain, follow a sample of sitemap URLs, and check their status, canonical, and robots directives. Then submit the sitemap through your search engine tooling and watch for excluded or redirected entries.
Check your production site
See which of these issues are visible now.
ScanMySaaS checks the public surface and keeps the affected URL with every finding. Verify results before changing production.
Run a free scan