Robots.txt Tester
Check whether the entered public path is permitted by the site robots policy.
How to use the Robots.txt Tester
Check whether the entered public path is permitted by the site robots policy.
Interpret the results
Robots rules can prevent crawling but cannot reliably keep a URL out of search indexes if it is referenced elsewhere. For nonpublic content use authentication and proper access controls.
Example
Test /guides/ on a site whose robots.txt blocks /admin/.
Limits and methodology
Wildcard and edge-case robots syntax needs verification with platform-specific testers.
Use this tool together with related checks for a fuller picture. Recommendations are based on the available observed evidence, not on secret search engine ranking data.
Tool workflow and verification
Test /guides/ on a site whose robots.txt blocks /admin/.
Robots rules can prevent crawling but cannot reliably keep a URL out of search indexes if it is referenced elsewhere. For nonpublic content use authentication and proper access controls.
Wildcard and edge-case robots syntax needs verification with platform-specific testers.
This tool page is an educational guide as well as a functional utility. Demand trends and real search impressions must come from verified search data, not keyword guesses.
Frequently asked questions
What does this tool measure?
Check whether the entered public path is permitted by the site robots policy.
How should I use its findings?
Robots rules can prevent crawling but cannot reliably keep a URL out of search indexes if it is referenced elsewhere. For nonpublic content use authentication and proper access controls.
What are the limitations?
Wildcard and edge-case robots syntax needs verification with platform-specific testers.