llms.txt and AI Crawler Tools, Ranked

What llms.txt actually is, and who wins this list

llms.txt is a proposed standard, maintained at llmstxt.org, for a plain Markdown file at the root of a website (/llms.txt) that gives large language models a concise, structured map of a site’s content, written for machines instead of browsers rendering HTML, CSS, and JavaScript. It is not a technical requirement any AI company has committed to honoring, and it does not replace robots.txt for controlling crawler access. It is closer to a curated index: a short list of links and descriptions that helps an LLM find the pages worth reading if it decides to fetch your site at all. Separately, a growing set of tools now let you both generate that file and control which AI crawlers touch your site in the first place. If you only look at one tool, make it Cloudflare’s AI Crawl Control, because it pairs crawler visibility with real enforcement at the network level, and it is available on every Cloudflare plan.

Below is a ranked breakdown of the real tools doing this work today, generators and access-control platforms both, since in practice most teams need one of each.

1. Cloudflare AI Crawl Control

Cloudflare’s AI Crawl Control, formerly AI Audit, sits at the network edge and gives you visibility into which named AI services (GPTBot, ClaudeBot, PerplexityBot, and dozens of others) are hitting your site, plus one-click controls to allow, block, or rate-limit them. It also tracks the health of your robots.txt and shows which crawlers ignore your directives. It does not generate llms.txt, so pair it with one of the generators below. Because it runs at the DNS/CDN layer, it works regardless of your CMS or hosting stack, which is the main reason it edges out pure generator tools. Cloudflare lists it as available on all plans; more granular controls and analytics sit behind Cloudflare’s paid tiers.

Best for: any site already on Cloudflare that wants crawler visibility and blocking in one place, not just a static file.

One real advantage: it is the only tool on this list that can actually stop a crawler, not just ask it politely via a text file. The tradeoff: it only helps sites on Cloudflare’s network, so it is a non-starter if your DNS lives elsewhere.

2. Dark Visitors (Known Agents)

Dark Visitors, now Known Agents (the old darkvisitors.com domain redirects there), maintains a continuously updated directory of AI crawlers, scrapers, and assistant bots, and uses that directory to generate a robots.txt that blocks or allows specific categories, AI Assistant, AI Data Scraper, AI Search Crawler, and so on, without you having to track every new bot by name. It also logs which agents actually visited your site, so you get analytics instead of a static guess. It ships as a hosted service and a WordPress plugin.

Best for: teams that want an auto-updating robots.txt against AI bots without manually researching every new crawler user-agent string.

Pro: the crawler list updates itself as new agents show up, which matters given how fast the bot landscape changes. Con: the free tier covers basic blocking and identification, but the deeper analytics and historical logs sit behind paid plans.

3. TollBit

TollBit is a licensing and monetization layer, not a generator. It sits between your site and AI crawlers, giving you visibility into bot traffic and the ability to charge AI companies per scrape or per query instead of simply blocking them. Publishers can use its bot analytics for free and set rates for AI access, which AI companies pay through TollBit’s bot paywall. In March 2026, the publishing platform Arc XP announced a TollBit integration for its publishers.

Best for: publishers and content-heavy sites that want to get paid for AI crawler access rather than just gate it.

Pro: it turns a cost center (unwanted scraping) into a revenue line, which no other tool here does. Con: it only works if the AI companies on the other end are willing to pay, and adoption is still limited to a subset of major AI buyers, so smaller sites may see little volume.

4. Vercel Bot Management / BotID

Vercel’s bot tooling is built for a different problem: distinguishing legitimate crawlers, including AI crawlers, from bots imitating real users on high-value routes like checkouts, logins, and LLM-powered API endpoints. BotID runs an invisible client-side challenge to catch scripted traffic without a visible CAPTCHA, while Vercel’s broader Bot Management lets you classify and allow known-good crawlers (search engines, payment webhooks, legitimate AI agents) versus blocking the rest. It is not an llms.txt generator, but it is the tool of record if your concern is AI agents hammering expensive API routes rather than indexing your marketing pages.

Best for: Vercel-hosted apps that need to protect expensive routes from bot abuse while still letting legitimate crawlers through.

Pro: invisible to real users, no friction added to checkout or signup flows. Con: it is scoped to Vercel-hosted projects and to route-level protection, so it does nothing for llms.txt generation or robots.txt management.

5. Mintlify llms.txt auto-generation

If your documentation runs on Mintlify, you already have this. Mintlify automatically generates /llms.txt and /llms-full.txt for every hosted docs site with zero configuration, pulling page descriptions from frontmatter and serving Markdown versions of each page so an LLM can fetch content directly instead of parsing rendered HTML. Mintlify says it co-developed llms.txt and llms-full.txt support with Anthropic, one of its documentation customers, in 2024. It is bundled into Mintlify’s documentation hosting, not sold separately.

Best for: product and engineering teams whose docs already live on Mintlify and want llms.txt handled without a separate tool.

Pro: zero setup, the file updates automatically as docs change. Con: it only applies to Mintlify-hosted docs, useless for your marketing site, blog, or anything outside that one subdomain.

6. WordLift llms.txt Generator

WordLift’s generator crawls your site starting from the homepage, extracts titles, descriptions, and structural metadata, and assembles a spec-compliant llms.txt file, positioned as part of WordLift’s broader AI SEO and knowledge graph toolset. It is the most accessible option for a marketing or content team that wants a one-time or periodically refreshed llms.txt without writing one by hand.

Best for: marketing teams without developer resources who need a working llms.txt fast and don’t need ongoing crawler blocking.

Pro: no code required, works on any CMS since it crawls the live site rather than needing a platform integration. Con: it is a point-in-time generator, not a live service, so the file goes stale as soon as you publish new pages unless you regenerate it manually.

7. Firecrawl llms.txt Generator

Firecrawl offers an API endpoint and open-source project that generate both llms.txt and llms-full.txt for any URL, priced on Firecrawl’s standard credit system (roughly one credit per URL processed, controllable via a max-URL parameter). It is the developer-friendly option: scriptable, API-first, and easy to slot into a build pipeline so the file regenerates automatically on deploy. Worth flagging that Firecrawl’s docs mark this endpoint as an alpha feature being deprecated in favor of its main endpoints, with maintenance ending June 30, 2025, so check current docs before building anything on it.

Best for: engineering teams who want llms.txt generation as a scripted, repeatable step in CI/CD rather than a one-off manual export.

Pro: cheap, API-driven, and open-source, so you can self-host the generation logic if needed. Con: it is a deprecated alpha feature no longer under active maintenance, which is a real risk if you wire it into a production pipeline.

Generator vs. access control: pick both, not one

The tools above split into two jobs that get conflated constantly. Generators (Mintlify, WordLift, Firecrawl) produce the llms.txt file itself, a courtesy index that well-behaved AI crawlers might read. Access-control platforms (Cloudflare, Dark Visitors, TollBit, Vercel) decide whether a crawler gets in at all and what happens when it does. An llms.txt file with no enforcement behind it is a suggestion, nothing more. If AI visibility and citation are part of your strategy, and for most brands doing digital GEO/SEO work today they should be, you need a generated file that accurately represents your content and a policy layer that decides which bots are worth letting through. Our GEO glossary entry covers where llms.txt fits into the broader generative engine optimization picture if you want the underlying mechanics.

How we ranked these

We ranked based on four factors: whether the tool does what it claims (generation, blocking, or monetization) without requiring a platform migration, how current the underlying AI crawler data is (since new bots appear weekly and stale lists are worse than no list), pricing accessibility for small and mid-size teams versus enterprise-only, and evidence of real adoption rather than vaporware. Every tool listed here was verified against its own documentation or product pages as of this writing; none were included on the basis of marketing copy alone. If you’re trying to figure out which combination of generation and enforcement makes sense for your site, talk to us and we’ll walk through what actually matters for your traffic and content.

Frequently asked questions

Does llms.txt help you get cited by ChatGPT?

No, llms.txt does not guarantee ChatGPT or any other AI tool will cite you. It is a proposed standard that no AI company has committed to honoring, so it works as a helpful index at best. Citations depend far more on clear, quotable content and third-party mentions of your brand. Adding the file is low effort and harmless, but treat it as a small supporting step, not a citation strategy.

What is the difference between llms.txt and robots.txt?

robots.txt tells crawlers which parts of your site they may access, while llms.txt suggests which pages are worth reading. robots.txt is a long-established convention that most major crawlers respect, and it controls access. llms.txt is a newer Markdown index of your key pages written for language models, and it controls nothing. You need robots.txt either way. llms.txt is optional and only useful once access is allowed.

Should you block AI crawlers from your website?

Block AI crawlers only if the cost of being crawled outweighs the value of appearing in AI answers. Blocking AI search and assistant crawlers can remove your pages from the answers buyers see in ChatGPT or Perplexity, which hurts visibility. Blocking training-data scrapers is a separate decision, often made by publishers protecting content. Most brands chasing AI visibility allow search crawlers, then decide category by category using a tool like Dark Visitors or Cloudflare.

How often should you update your llms.txt file?

Update your llms.txt file whenever you publish, retire, or significantly change a key page. A stale file points models toward outdated or missing content, which defeats its purpose. The simplest fix is automation: platforms like Mintlify regenerate the file as pages change, and developer teams can rebuild it on each deploy. If you use a one-time generator like WordLift, schedule a regular refresh so the index keeps pace with your site.