Is llms.txt Worth It? What 600+ Websites Show

TL;DR

• An llms.txt is fine to add and changes nothing we can measure. Across 600+ websites we've studied, sites with the file got the same AI assistant visits and the same search traffic as sites without it.
• Google says Search ignores the file, and no AI company documents reading yours. The hype came from Semrush audit alerts and Shopify's default file.
• Spend the time on what AI already reads, starting with which AI crawlers your robots.txt lets in.

The llms.txt questions started landing in our inbox in late 2025, and they're still coming in.

Clients forward audit alerts, partner agencies ask what we're doing, our own team asks what to say on the next client call.

Source: StudioHawk Slack and HubSpot email records, October 2025 to November 2026. Names and client details removed.

This week a client's in-house SEO asked us for one, along with case studies showing it works.

We went looking and couldn't find a single one.

We gave the answer we've given since 2025: it's fine to put on a website, it doesn't hurt, and it doesn't cause any significant uplift.

Add it if it takes minutes, then focus on the bigger fish to fry.

Start with what the file is.

What is llms.txt?

llms.txt is a Markdown file at the root of a website that lists the pages an AI tool should read first.

Jeremy Howard proposed it on September 3, 2024. He wanted a way "to provide information to help agents use a website."

The file holds the site name as a heading, a one-line summary, then lists of links with a note after each.

It's still a proposal, and no search engine or AI company has adopted it as a standard.

The proposal's own site says the files "are used most heavily for software documentation, where coding agents follow them."

Our Hawk Academy glossary entry on llms.txt walks through the format line by line.

The more useful question is how a file built for coding tools ended up in marketing audits.

Why are clients asking for llms.txt?

Clients ask for an llms.txt for one of four reasons. We've seen all four in our client conversations since mid 2025:

• An audit tool flagged it. Semrush and similar tools list a missing llms.txt as an issue, so it shows up in a report as a gap to close.
• A competitor or a big brand has one. A client checks a rival's site, finds the file, and wants the same.
• An executive asked for it. The request comes down from the top after a conference talk or a LinkedIn post, and someone has to answer it.
• Shopify switched it on. Ecommerce clients found files on their stores that nobody had asked for.

All four start from how a site looks to a tool or a rival.

None of them starts from a result someone measured.

Attention ran two years ahead of the evidence.

How did llms.txt go from a proposal to a must-have?

Tools, platforms and brands adopted llms.txt one after another until it looked like a standard.

No search engine or AI company ever said it reads the file.

Source: llmstxt.org, Search Engine Journal, Chrome for Developers, Shopify developer changelog and Google Search Central (guide updated July 10, 2026). Dates checked November 3, 2026.

• September 2024. Jeremy Howard proposes the file to help AI tools read software documentation.
• 2024 and 2025. AI companies publish llms.txt files for their own developer docs. OpenAI, Anthropic and Perplexity all have one. To a marketer, that reads like an endorsement.
• June 2025. Yoast adds a one-click llms.txt to its WordPress plugin, which puts the file on sites as a side effect of a plugin setting.
• July 2025. The first audit alert reaches us from a client: a Semrush report listing the missing file.
• May 2026. Chrome's Lighthouse adds an llms.txt check to its experimental agentic browsing audit, and Shopify starts serving a default file on every store.
• 2026. Big brands publish their own, which gives every in-house SEO an example to point at.

Google said no the whole way through. In April 2025, John Mueller compared llms.txt to the keywords meta tag, a tag Google has ignored for ranking since at least 2009.

Even the Lighthouse check marks a missing file as not applicable, because providing it is "optional at the moment".

The evidence on what the file does for search and AI answers points the same way.

Does llms.txt change anything in search or AI answers?

No source we can find shows llms.txt changing rankings, citations or AI traffic. Here is the evidence, strongest first.

• Google: Its AI optimization guide says an llms.txt "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." In January 2026, Mueller was asked whether the llms.txt on Google's Gemini API docs meant Google endorsed the format. He replied: "to be direct, no."
• The AI companies: OpenAI, Anthropic and Perplexity publish an llms.txt for their own docs. None of their crawler documentation says they read yours (checked November 2, 2026). All three point site owners to robots.txt.
• Server logs: Ahrefs checked logs across 137,210 domains and found 97% of llms.txt files got zero requests in May 2026. Our founder Harry Sanders has reviewed log files for hundreds of businesses, he has said publicly that he's seen the file crawled "less than the fingers on one hand."
• Citation data: SE Ranking modeled AI citations across about 300,000 domains, its model predicted citations better without the llms.txt variable than with it.
• Our own data, from 600+ websites we've studied: In our analytics data, over 12 months, sites with the file had a median of 125.8 AI assistant sessions per 1,000 non-brand Google clicks.
  • Sites without it had 124.8. A gap that small sits well inside what chance alone produces.
  • We also checked Australian business sites ranking for 45 buying searches in DataForSEO's search traffic estimates.
  • Once Shopify stores were set aside, because they get the file by default, sites with and without an llms.txt changed by the same amount between October 2025 and October 2026. Shopify stores led other retailers by the same margin before Shopify switched the file on in May 2026 as after.
  • Our Hawk Academy guide to whether llms.txt is worth it has the method and its limits.

Source: StudioHawk analytics data, 394 websites, October 2025 to September 2026; llms.txt checked November 2, 2026. Full method in the Hawk Academy guide.

Shopify and the proposal's authors keep changing the file anyway, because its job is changing.

What is llms.txt turning into?

llms.txt is turning into instructions for AI agents, the tools that browse and buy on a person's behalf.

Two changes in 2026 pushed it there.

First, version 2 of the proposal, published in August 2026, lets one site carry several files. In its words, "a file covers the pages under its path, and the most specific file applies."

That suits large documentation sites, where a coding agent needs the right section and nothing else.

Second, Shopify's default file stopped being a link list.

A Shopify store's /llms.txt now serves the same content as its /agents.md, a file of instructions for shopping agents.

Shopify's changelog of May 28, 2026, says the paths "/llms.txt and /llms-full.txt also point to this content by default."

You can see it on our client Sylvane's store.

Its llms.txt opens with the heading "Agent Instructions" and reads like a manual for a shopping agent: how to search products, build a cart and check out with Shop Pay.

Most clients assume the file tells AI who the brand is, this one mostly tells an agent how to buy.

Shopify store owners should open yourstore.com/llms.txt before writing anything.

You almost certainly have one: 121 of the 125 Shopify stores in our sample served a file.

Keep any edit small, and leave the agent instructions working.

What does implementing llms.txt actually involve?

Writing an llms.txt takes minutes. Getting it served is where the time goes, we've planned or shipped the file on more than 20 client sites.

Each time, the hard part was access:

• The file sits at the server root, so the client's developer uploads it.
• A CMS module breaks when it's fed raw Markdown.
• The file arrives in a format the CMS can't upload.
• The platform serves the file as a web page, with an HTML content type.

That last one is ours, our own llms.txt is built as a HubSpot page, so HubSpot serves it as text/html.

The proposal names no content type, no AI company documents reading the file at all, so none says what format it needs.

The file still loads, and nothing we can measure depends on it.

Keep the file accurate if you run one.

We've had to strip old company names and retired URLs out of client files, and out of our own.

That upkeep matters less than the fixes below.

What should you fix before llms.txt?

Fix the things AI systems demonstrably read before you touch llms.txt.

Our short list:

1. AI crawler access. Check that robots.txt lets in the AI bots you want. Our free AI bot access checker shows you in a minute.
2. Content in the raw HTML. Most AI crawlers don't run JavaScript, so they may never see content that loads late.
3. An HTML sitemap and internal links, so every important page is a click or two from a page crawlers already visit.
4. An entity map, structured data that ties your brand, people and pages to one consistent identity.
5. An AI info page, if AI tools describe your business wrongly.
6. Content worth citing, such as original data and answers nobody else has published.

None of this is new. It's the same groundwork that gets you into Google's AI Overviews. Of the six, the AI info page needs the most explaining.

Is an llm-info page worth it instead?

An llm-info page is worth building if AI tools describe your business wrongly. It's an ordinary HTML page of plain facts: what you do, who runs it, where you operate. AI crawlers read it under the same robots.txt rules as any other page.

That's why it gets read.

We tested one in September 2025 by placing a unique marker on a client's AI info page, then asking ChatGPT about the brand. ChatGPT reached the page through a live web search, because the page ranked for the brand name. It worked because it was a normal page that ranked. No special file was involved.

Ours sits at studiohawk.com.au/llm-info.

Put the same facts on your About page if AI tools already describe you correctly.

People read that page too.

The Hawk Academy glossary entry on llm-info pages covers what to include.

FAQ

Can an llms.txt hurt my rankings?

No. An llms.txt can't hurt your rankings in Google, because Google Search ignores the file. Google's guide says it "will neither harm nor help" your visibility in Search.

Does Shopify add an llms.txt automatically?

Yes. Shopify adds an llms.txt to stores by default, and since May 2026 it mirrors the store's agents.md, a file of instructions for shopping agents. Check yours at yourstore.com/llms.txt before building a new one.

Can I block AI bots with llms.txt?

No. You can't block AI bots with llms.txt, because the file has no rules for blocking anything. Use robots.txt, which OpenAI, Anthropic and Perplexity all document for controlling their crawlers.

Does it matter if my llms.txt is served as HTML?

No documented rule says it matters. The proposal names no content type, and the Lighthouse check only flags server errors. Ours serves as text/html through HubSpot, and the text inside is still plain Markdown.

Check your robots.txt before your llms.txt

Add the file if it takes minutes.

Then spend the real time on crawler access, starting with the free AI bot access checker, which shows which AI crawlers your robots.txt lets in.

For the wider picture, download our AI search whitepaper, or talk to us about AI SEO.

Sources

• StudioHawk study S11: 394 websites from our own analytics data, GA4 and Search Console, October 1, 2025, to September 30, 2026; llms.txt presence checked November 2, 2026. Pooled results only.
• StudioHawk check with DataForSEO: 247 Australian business sites from the top 10 results for 45 buying searches across 9 industries (DataForSEO SERPs, November 3, 2026); Google Australia organic traffic estimates, October 2025 against October 2026; llms.txt checked live November 3, 2026. Together with S11, 641 websites.
• Jeremy Howard, The /llms.txt file, llmstxt.org, September 3, 2024; version 2, August 2026.
• Google Search Central, Optimizing your website for generative AI features on Google Search, last updated July 10, 2026.
• Search Engine Journal, Google says llms.txt comparable to keywords meta tag, April 17, 2025.
• Search Engine Journal, Google's llms.txt guidance depends on which product you ask, May 20, 2026.
• John Mueller, reply on Bluesky, January 20, 2026.
• Chrome for Developers, Lighthouse agentic browsing audit "llms.txt", last updated May 5, 2026.
• Shopify developer changelog, "Customize llms.txt, llms-full.txt and agents.md", May 28, 2026.
• Search Engine Journal, Yoast SEO WordPress plugin adds support for llms.txt, June 18, 2025. Yoast announced the feature on June 2, 2025, in version 25.3.
• Search Engine Journal, AI crawlers account for 28% of Googlebot's traffic, study finds, December 19, 2024, reporting Vercel's study: of the major AI crawlers, only Googlebot (which Gemini uses) and Applebot run JavaScript.
• Ahrefs, llms.txt study, June 15, 2026.
• SE Ranking, llms.txt and AI citations study, about 300,000 domains, October 15, 2026 version; first reported by Search Engine Journal, November 20, 2025.
• OpenAI, Anthropic and Perplexity crawler documentation, read November 2, 2026.