Two focused validation modes
Inspect robots groups, rules and declared sitemaps, or parse one XML urlset or sitemap index with entry and metadata checks.
Free · no signup
Robots.txt & Sitemap Validator
Check one public robots.txt or XML sitemap through a guarded server fetch. The report separates protocol errors, contextual warnings, crawler-specific behavior and WebTaskKit's own inspection limits—without an opaque score.
Your categorized report will appear here
Errors, warnings, crawler-specific behavior, information, tool limits and unverifiable facts remain distinct. No ranking or health score is invented.
The submitted public URL and optional robots path are used in memory for one check. They and the fetched file are not stored or written to WebTaskKit access or event logs; only this canonical tool path and an allowlisted action name may be counted.
Simple by design
How to use Robots & sitemap
- 1
Choose Robots.txt or XML sitemap. Submit only a public, canonical HTTP or HTTPS address with no credentials, query string, fragment or custom port.
- 2
For robots.txt, enter the website origin. You may test one path against a crawler product token; that path is evaluated against the fetched rules and is never requested.
- 3
For a sitemap, enter the exact .xml or .xml.gz file you want to inspect. Review the categorized findings, then verify ownership and indexation in the relevant search-engine console.
Crawler behavior stays explicit
RFC-level findings are kept separate from Google-specific interpretation and from facts the file alone cannot prove.
One bounded public fetch
The server validates every destination and redirect, pins public DNS results, and applies fixed time, redirect, header, body and parser limits.
Put it to work
Practical ways to use Robots & sitemap
Check a deployment change
Confirm that a production origin still serves a readable /robots.txt after an ingress, framework or CDN change, then test one important path without fetching it.
Review one sitemap before submission
Inspect the root element, locations, duplicates and optional last-modified values before submitting the exact sitemap URL in a search console.
Investigate an unexpected crawl block
Test the affected path with the crawler product token you care about and inspect the winning rule, while remembering that a live crawler may cache an older robots response.
Choose deliberately
Make the right output
Fix definite protocol errors first
A malformed document or unusable location is different from an optional field or a crawler-specific interpretation. Start with findings marked as errors.
Read warnings in site context
A broad Disallow rule, cross-origin declaration or old lastmod value may be intentional. Confirm the publishing and ownership model before changing it.
Take tool limits literally
A tool-limit finding identifies an intentionally shortened returned preview; the accepted document's counts and checks still cover what was parsed. A request rejected at a hard size or complexity cap is not fully validated.
Continue the workflow
Useful next steps
Primary guidance
What the checks are based on
Protocol and crawler guidance reviewed August 19, 2026. Crawler policies can change; the report labels crawler-specific statements and never treats a sitemap as proof of indexation.
Good to know
Frequently asked questions
Does the validator crawl my website?
No. It fetches exactly one robots.txt or XML sitemap per run. It does not request the optional robots test path, sitemap entries, sitemap-index children or sitemap URLs declared in robots.txt.
What URL should I enter for robots.txt?
Enter the final public origin, such as https://example.com. WebTaskKit requests that origin's standard /robots.txt location. The origin cannot contain a query; the separate optional test path may include a query because it participates in robots matching, but that path is never fetched.
What URL should I enter for a sitemap?
Enter the exact public .xml or .xml.gz URL you want to validate. The tool does not guess, discover or recursively fetch a different sitemap.
Does Allowed mean Google will index the page?
No. A robots result concerns crawling rules for the tested path and product token. Indexation also depends on discovery, response content, canonical signals, quality, policies and search-engine systems.
Why can the report differ from Google's robots tester?
RFC 9309 defines a shared core, but crawlers add documented behavior for file size, redirects, caching and HTTP errors. WebTaskKit also uses stricter safety limits and cannot see a crawler's cached copy.
Is a missing changefreq or priority a sitemap error?
No. Both are optional in the protocol. The report surfaces values when present without treating their absence as a failure.
Can robots.txt protect a private file?
No. Robots rules are voluntary crawl instructions, not authentication. Protect private content at the application or server layer and avoid naming secrets in a public robots file.
Does a valid sitemap guarantee indexing?
No. A sitemap helps discovery and provides hints. Confirm processing, coverage and page-level reasons in the verified property for each search engine.
Are my submitted URL and fetched file stored?
No. They are used in memory for the response and are not written to WebTaskKit access or event logs. The destination may still log the incoming public request.