How to find and fix sitemap problems

Download the public sitemap and inspect the evidence

Use this macOS/Linux GET request after replacing example.com. It creates a new local directory, so existing project files are not overwritten. The header file records redirect responses; the body file contains the final response. HTTP errors are retained for inspection, so read the reported status rather than treating a completed download as success.

Check the final URL and Content-Type against the body. A sitemap route may return an HTML login page, application fallback or error page with status 200. An unexpected Location can reveal the wrong host or route. Correct that response before investigating XML tags.

sitemap_check_dir=$(mktemp -d ./sitemap-check.XXXXXX) &&
curl -q -sS -L --max-redirs 5 --max-time 30 --compressed \
  -D "$sitemap_check_dir/headers.txt" \
  -o "$sitemap_check_dir/sitemap.xml" \
  --write-out 'HTTP %{http_code}\nFinal URL: %{url_effective}\nSaved: %{filename_effective}\n' \
  'https://example.com/sitemap.xml'

1. Read the saved headers and confirm the intended final URL returns a successful sitemap response.
2. Inspect the saved body as text: confirm XML content rather than an HTML challenge, login or error page.
3. Parse the XML, then open representative listed URLs and check their content, canonical and indexing instructions.

Reference: curl: GET responses, redirects and output files

Produce well-formed XML with the intended URLs

Use UTF-8, the sitemap namespace and absolute URLs. This example includes a content-changing query with its ampersand escaped as &. Keep such a URL only if it is the intended canonical page. Replace the illustrative lastmod date with the page’s actual significant modification date, or omit it when that date is unknown.

Do not stamp every entry with the current time whenever the sitemap is requested. Include the final preferred URLs, avoiding staging hosts, redirecting aliases and pages deliberately excluded from search. A URL can be syntactically valid while still being the wrong page to submit.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/guides/tls</loc>
    <lastmod>2026-10-20</lastmod>
  </url>
  <url>
    <loc>https://example.com/reference?language=en&amp;version=2</loc>
  </url>
</urlset>

Reference: Sitemaps protocol: XML structure, escaping and lastmod

Make the right files discoverable within their scope

Publish an absolute Sitemap line in robots.txt, such as Sitemap: https://example.com/sitemap.xml, or submit the file through Search Console. A custom filename is valid. Hosting the sitemap at the site root simplifies URL scope; cross-site hosting and broader directory scope require the applicable ownership or submission setup.

Limit each sitemap to 50,000 URLs and 50 MB uncompressed. Gzip does not raise that uncompressed limit. Split larger collections into child sitemaps and list those files in a sitemap index. An index needs its own valid entries, and each child must also be reachable, parseable and within its limits.

The curl example handles HTTP content compression. A downloaded .gz sitemap file may still need gzip decompression before XML inspection; distinguish the file format from Content-Encoding. Do not interpret compressed bytes or a renamed HTML document as broken XML.

Reference: Google Search Central: sitemap discovery, location and size limits

Understand exactly what SiteAudit checks

SiteAudit requests two fixed paths on the supplied origin: /robots.txt and /sitemap.xml. It lists up to ten Sitemap references found in robots.txt but does not visit those references. It does not fetch sitemap-index children or check whether the audited page appears in a sitemap.

The fixed-file requests allow up to three redirects on the same hostname. A cross-host redirect, timeout or download limit can produce unknown even when another crawler can use the file. The sitemap download cap is 1 MiB, well below the protocol’s maximum; a larger valid sitemap can exceed the scan’s scope.

Its XML check covers parsing, an accepted root and namespace, and rejection of DTDs. It is not full schema validation and does not inspect every loc, lastmod, URL response or canonical. A pass therefore does not prove completeness or indexing; a missing /sitemap.xml does not prove there is no sitemap elsewhere.

Finish with a parser check and a small URL sample

If xmllint is installed, run the command below against the XML downloaded earlier. It checks well-formedness without requesting network resources; it is not a sitemap-schema or URL-content check. Fix broken markup at the generator and regenerate the file instead of manually patching an output that will be overwritten.

Submit the intended sitemap or index in Search Console and inspect its fetch and parsing report. Successful submission or processing does not guarantee that every listed page will be crawled or indexed. Use URL Inspection for specific pages whose indexing remains in question.

xmllint --nonet --noout "$sitemap_check_dir/sitemap.xml"

1. Check the deployed XML and any index children, including a child that previously failed.
2. Sample a recent page, an older page, a language version and any meaningful query URL; verify status, content, canonical and noindex.
3. Confirm robots.txt or Search Console points at the intended file, then retain the failing URL and error if further investigation is needed.

Reference: Google Search Console: sitemap processing and errors