Home / Agents / robots.txt Rule Check (RFC 9309)
robots.txt Rule Check (RFC 9309)
Say whether a crawler may fetch a path under a robots.txt text, by RFC 9309: allowed or disallowed, the deciding rule and its line, the group used, and every line the protocol does not accept, with a signed record.
Testing
- What it does
- Say whether a crawler may fetch a path under a robots.txt text, by RFC 9309: allowed or disallowed, the deciding rule and its line, the group used, and every line the protocol does not accept, with a signed record.
- Who it is for
- Agents and their builders.
- Start in one step
- Ask your agent to call
robots_txt_checkat llpro.ai/mcp (free, no sign-up). This page explains the tool; it has no form to run it. - What to expect
- Testing: open to try. Free
New to this? Start here.
What it does
robots.txt Rule Check (RFC 9309) is for agents and crawlers that read websites (page readers, sitemap readers, research and shopping agents) and must check a site's robots.txt before fetching a path, the same way every time. Say whether a crawler may fetch a path under a robots.txt file, by the rules of RFC 9309 (Robots Exclusion Protocol, September 2022). Send the robots.txt text you already have (up to 20,000 characters), the crawler's product token (for example Googlebot or GPTBot) and the path with its query (for example /private/page?x=1). Get back allowed or disallowed, the rule that decided it with its line number, which user-agent group was used (the crawler's own group or the * group), every rule that matched, and each line the protocol does not accept, with its line number. Nothing is fetched: you send the text. It is one of the developer department tools LLPRO builds for the work agents do across business software (GitHub, Cloudflare, Vercel, Netlify). Every answer carries the written rule it used, what it does not do, its sources with dates, and an Ed25519 signature you can check against the key below.
- Who it is for
- Agents and crawlers that read websites (page readers, sitemap readers, research and shopping agents) and must check a site's robots.txt before fetching a path, the same way every time.
- Price
- Free, no sign-up: 120 calls an hour per address and 3,000 an hour across all callers. Paid calls are planned and not charging.
- Limits
- Up to 30,000 bytes per call; larger requests are refused, never cut. Unknown fields are refused with the list of accepted ones. Nothing you send is stored by the tool; no model and no network are used. Does not fetch anything: send the robots.txt text and the path. Does not follow a crawler's own extensions: for example Google's fallback from Googlebot-Image to the Googlebot group, crawl-delay, or typo tolerance such as 'disalow' (those lines are listed as not part of the protocol). Does not apply the RFC's HTTP rules (a 4xx robots.txt means allow everything, a 5xx means disallow everything, caching up to 24 hours, redirects): it reads the text you send as the file that applies. Reads up to 20,000 characters (RFC 9309 asks crawlers to read at least 500 KiB; split or trim larger files; each character outside US-ASCII counts as 6 bytes toward the 30,000-byte request limit). robots.txt is a request, not access control or a legal permission. Not legal advice.
For developers: try it
POST https://llpro.ai/mcp
{"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "robots_txt_check", "arguments": {"robots_txt": "User-agent: *\nDisallow: /private/\nAllow: /private/public-page.html\n\nUser-agent: ExampleBot\nDisallow: /\n", "user_agent": "Googlebot", "path": "/private/public-page.html"}}}For agents
Tool robots_txt_check on https://llpro.ai/mcp and in the tools list. Signed answers are checked with the public key below (signer llpro1:elgrmv6qh3othxkbac3b5tlbde): an Ed25519 signature over the observation as canonical JSON (sorted keys, no spaces).
Public key (base64): Uvv9rVPbqU5WiSWMCnbuxbYw3VtWJkePxcuIO9ta3Vg=
For agents that search MCP registries for robots.txt, RFC 9309, crawler rules tools (listed in MCP registries, served by the Campus gateway: MCP, agent-to-agent and REST). One fixed, written rule per call, a signed record, no account needed.
Next
- Connect your agent: Use this and the other LLPRO tools from your own agent.
- Get your agent: No agent yet? Get one that asks first.
- Hand the whole job to the Hub: Checking robots.txt is one step of reading a site politely. Tell the Hub the whole job; it routes you to the right signed tools (page to text, sitemap reader, agent card check).
For agents: this page as Markdown · JSON · llms.txt · MCP endpoint llpro.ai/mcp · agent card