Free tool · no account needed

Robots.txt Checker

Test your robots.txt against 42 crawlers: Googlebot, Bingbot, the AI bots, the ad, SEO-tool and social preview bots. Every rule checked line by line, every problem explained, and a corrected file you can copy.

Free robots.txt check

Check your robots.txt file for free

Robots.txt checker and validator for every rule in the file

The ProStar SEO robots.txt checker reads the robots.txt file of any domain and tests every rule the way crawlers apply it. Enter a domain and the checker fetches robots.txt, the http and www versions of the file, the homepage with its links, CSS, JavaScript and images, each declared sitemap and llms.txt. Then it runs 42 crawlers through your rules: Googlebot and the other Google crawlers, Bingbot, Applebot, the AI search and training crawlers, the SEO tool bots and the social preview bots.

Every finding carries a severity, the line it comes from, the reason it matters and the fix. Critical findings stop crawling or indexing. Warnings cost crawl coverage or AI citations. Notices are worth a look. The checker gives no score out of 100, because a single score hides which line causes the problem.

What the robots.txt validator checks

Server checks come first. A robots.txt file has to return HTTP 200 as plain text. A 404 tells crawlers to crawl everything. A 5xx or 429 error makes Google treat the whole site as disallowed until the file comes back. Google follows up to five redirects for robots.txt, reads only the first 500 KiB and skips a byte order mark.

Syntax checks follow. The validator flags misspelled directives such as “Dissallow”, missing colons, rules written before any User-agent line, full URLs where a path belongs, relative sitemap URLs and the directives Google ignores: Noindex, Crawl-delay and Host.

Logic checks catch the expensive mistakes. Disallow: / under User-agent: * shuts out every crawler you don’t name. A named group for Googlebot silently drops your * rules. Blocked CSS and JavaScript files leave Google rendering a broken page. A blocked /wp-content/ folder hides a WordPress theme. A sitemap that lists URLs robots.txt blocks sends Google two opposite instructions.

Robots.txt generator: a corrected file built from yours

The Recommended file tab rewrites your robots.txt with every safe fix applied, and lists each change with its line number. Typos are corrected, unsupported directives removed, full URLs turned into paths, blocked CSS and JavaScript reopened with targeted Allow rules and a Sitemap line added. You can edit any version and re-run every check before it goes live on your server.

The 42 crawlers the robots.txt checker tests

The checker runs every rule in your file against 42 crawlers, each under the user-agent name it reads in robots.txt. Half of them are AI bots. The other half are the search engine crawlers, the ad crawler, the SEO tools and the social preview bots that decide how your pages get crawled, ranked and shared. Any other user-agent can be tested in the URL tester.

Search engines10

Crawl and index pages for search results. Applebot follows your Googlebot rules when it has no group of its own.

  • GooglebotGoogle
  • Googlebot-ImageGoogle
  • Googlebot-VideoGoogle
  • Googlebot-NewsGoogle
  • Storebot-GoogleGoogle Shopping
  • BingbotMicrosoft
  • ApplebotApple
  • DuckDuckBotDuckDuckGo
  • YandexBotYandex
  • BaiduspiderBaidu

AI search5

Build the indexes that AI answers cite. A site that blocks OAI-SearchBot is left out of ChatGPT search answers.

  • OAI-SearchBotOpenAI
  • Claude-SearchBotAnthropic
  • PerplexityBotPerplexity
  • DuckAssistBotDuckDuckGo
  • YouBotYou.com

AI assistants5

Fetch a page when a user asks for it. OpenAI and Perplexity say these visits may not follow robots.txt.

  • ChatGPT-UserOpenAI
  • Claude-UserAnthropic
  • Perplexity-UserPerplexity
  • MistralAI-UserMistral
  • meta-externalfetcherMeta

AI training13

Collect content for model training. Google-Extended and Applebot-Extended are control tokens, and anthropic-ai and Claude-Web are outdated names the checker flags.

  • GPTBotOpenAI
  • ClaudeBotAnthropic
  • Google-ExtendedGoogle
  • Applebot-ExtendedApple
  • CCBotCommon Crawl
  • meta-externalagentMeta
  • BytespiderByteDance
  • AmazonbotAmazon
  • cohere-training-data-crawlerCohere
  • AI2BotAllen Institute
  • DiffbotDiffbot
  • anthropic-aiAnthropic, outdated
  • Claude-WebAnthropic, outdated

Ads1

AdsBot-Google checks Google Ads landing pages and ignores the * group. Only a group that names it applies.

  • AdsBot-GoogleGoogle

SEO tools5

Backlink and audit crawlers. Blocking them hides your site from those tools, your own audits included.

  • AhrefsBotAhrefs
  • SemrushBotSemrush
  • MJ12botMajestic
  • DotBotMoz
  • Screaming Frog SEO SpiderScreaming Frog

Build the title, image and description card when someone shares your link on social media.

  • facebookexternalhitMeta
  • TwitterbotX
  • LinkedInBotLinkedIn

How Google reads a robots.txt file

Google parses robots.txt under RFC 9309, the Robots Exclusion Protocol standard published in 2022. A crawler obeys one group only: the group whose User-agent line names it most specifically. Googlebot-Image falls back to the Googlebot group, and the * group applies only when no group names the crawler. Groups that name the same crawler are merged into one.

Inside that group the longest matching path wins. When an Allow rule and a Disallow rule match with the same length, Google applies Allow. The * wildcard matches any sequence of characters and $ marks the end of the URL. Paths are case-sensitive, so Disallow: /Admin/ leaves /admin/ open.

User-agent: *
Disallow: /private/
Allow: /private/press/

With these three lines, Googlebot may crawl /private/press/kit.pdf, because the Allow rule is longer than the Disallow rule, and may not crawl /private/plans.pdf.

Robots.txt tester for any URL and any crawler

The URL tester tab answers the everyday question: is this URL blocked? Paste a path or a full URL, pick a crawler, and the tester returns Allowed or Blocked with the exact line that decided it. A second table runs the same URL through every crawler at once, which is the fastest way to check robots.txt for a new section before launch.

AI crawlers in robots.txt: search, assistants and training

AI crawlers do three different jobs, and a robots.txt file should treat each one separately. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes that AI answers cite. Assistant fetchers such as ChatGPT-User and Claude-User load a page when a user asks for it. Training crawlers such as GPTBot, ClaudeBot and CCBot collect content for model training, while Google-Extended and Applebot-Extended are control tokens for the same purpose.

Blocking GPTBot keeps your content out of OpenAI’s training data and leaves your pages eligible for ChatGPT search, which OAI-SearchBot controls. Blocking Google-Extended has no effect on Google Search, AI Overviews or AI Mode, which all rely on Googlebot. The checker asks what you want from AI tools first, then grades your file against that goal. To see whether AI answers cite your brand today, run the free AI visibility checker.

A robots.txt rule is a request that each crawler decides to honor. OpenAI and Perplexity say user-triggered fetches may not follow it, and Bytespider is widely reported to ignore it. A firewall rule is the only hard block.

Robots.txt blocks crawling. Noindex removes pages.

A disallowed URL can still appear in Google when other pages link to it. Google lists it without a description, because robots.txt never let Googlebot read the page. To take a page out of search results, let Google crawl it and add a noindex meta tag or an X-Robots-Tag header. Blocking and noindexing the same URL keeps the noindex invisible, so the page stays in the index.

The checker tests the internal links on your homepage and a sample of 50 URLs from your sitemaps against your rules. Any URL that your sitemap lists and robots.txt blocks shows up as a conflict, with the line responsible. If rankings fell after a robots.txt change, the penalty checker shows when page-one keywords dropped.

Robots.txt checker FAQ

How do I check my robots.txt file?

Enter your domain in the robots.txt checker above. To read the raw file, open https://yourdomain.com/robots.txt in a browser. Google Search Console’s robots.txt report, under Settings, shows the versions Google fetched for each host in your property and when it fetched them.

Where does the robots.txt file go?

The file sits at the root of each host, at exactly /robots.txt. It applies to one protocol and one host, so https://www.example.com/robots.txt doesn’t cover https://shop.example.com or http://example.com. Every subdomain needs its own file.

Does Google support Crawl-delay or Noindex in robots.txt?

No. Google ignores Crawl-delay, and it stopped supporting Noindex in robots.txt in September 2019. Bing and Yandex still honor Crawl-delay, so a high value slows their crawling of your whole site.

Should I block GPTBot in robots.txt?

That depends on what you want from AI tools. Blocking GPTBot keeps content out of OpenAI model training and doesn’t affect ChatGPT search. Pick “Get cited, but no AI training” in the checker and the recommended file blocks the training crawlers while the AI search crawlers stay open.

Is the robots.txt checker free?

Yes, with no account. The checker reads public files only and changes nothing on your site. Each network can run 30 checks an hour.

Get crawl and indexing problems fixed

Robots.txt is one file in the crawl path. ProStar SEO’s technical SEO work covers the rest: redirects, canonical tags, sitemaps, rendering and the indexing reports in Search Console. Book 30 minutes with Eric St-Cyr and bring your checker results.