Loading IndexHaven…
Free website tools
Fetch robots.txt for a URL’s origin and inspect whether its path is allowed by the parsed rules. Robots.txt controls crawling; it is not a reliable way to remove a URL from search results.
Retrieve a site’s robots.txt and check path rules for IndexHavenBot, Googlebot and Bingbot. View the response and file evidence.
Rule interpretation uses robots-parser for the selected user agents; search engines may differ. The robots file is fetched without crawling the submitted page. Missing files and unavailable files are distinguished; displayed file evidence is capped at 64 KB.
User-agent: * Disallow: /private/ Allow: /
This example asks crawlers not to crawl /private/ while allowing other paths. It is not an access-control mechanism.
A one-time result describes one response. Read our guide to ongoing monitoring to understand scheduled checks, domain ownership and confirmed changes. Explore monitoring features and plan limits.