Explore. Build. Grow
Explore. Build. Grow

Our flexible three-stage framework to craft digital experiences.

Learn more
Expertise
Outcomes
View our portfolio
View our portfolio

Explore our range of digital solutions.

Learn more
Industries
Our Approach to AI
Our Approach to AI

Keeping ourselves at the forefront of emerging technology is part of our DNA. But we recognise that ‘technology for technology’s sake’ is not the end goal – there is a balance to be struck.

Learn more
DXP
Headless
Cloud Platforms
Marketing Optimisation
Contact Us
Contact Us
Explore. Build. Grow
Explore. Build. Grow

Our flexible three-stage framework to craft digital experiences.

Learn more
Expertise
Outcomes
View our portfolio
View our portfolio

Explore our range of digital solutions.

Learn more
Industries
Our Approach to AI
Our Approach to AI

Keeping ourselves at the forefront of emerging technology is part of our DNA. But we recognise that ‘technology for technology’s sake’ is not the end goal – there is a balance to be struck.

Learn more
DXP
Headless
Cloud Platforms
Marketing Optimisation
Contact Us
Google search
Home / Insights / ＿

Testing the llms.txt file: Does it help AI discover content?

Tested: Can llms.txt help AI discover content? Results show it’s not a discovery tool. SEO fundamentals still drive visibility in AI search.

Shayna Burns
Shayna Burns

15 July 2026

4 minute read

As AI search evolves, new tactics are emerging quickly, often ahead of clear evidence of how they actually perform.

One of those is the llms.txt file, a proposed standard designed to highlight priority pages for large language models. It provides a simplified, markdown-based version of key URLs, reducing reliance on complex HTML pages that include heavy CSS and JavaScript.

The new standard and website, https://llmstxt.org/, were introduced in September 2024 by Australian AI researcher Jeremy Howard, co-founder of Answer.AI. Since then, many in the industry have questioned its use case and benefits, such as those explored in this March 2025 Search Engine Land article. One debated benefit is its potential use for AI discoverability.

In July 2025, Luminary partnered with a long-term client to launch their own llms.txt file as part of their broader push toward AI readiness. But the core question remained: does this file actually help AI systems find content?

To answer that, we ran a controlled test to isolate its impact on discoverability.

How we tested the llms.txt’s impact on content discoverability

To test the discoverability theory, we created a new page on the client's website that could only be discovered through the llms.txt file. 

  • Test URL: We created a new page on the client's website.
  • No discovery pathways: Excluded the test page from XML sitemap, no internal links, meta robots 'noindex' setting to prevent search engine indexation.
  • One entry point: The only way LLMs could find the page was through the llms.txt file.

We validated that the page had no external links or alternate discovery paths before running the test.

Screenshot showing Google had not indexed the test page.

We validated that the URL was not indexed in Google before commencing the test.

If the industry theories were true – that llms.txt could be a primary discovery tool used by AI – then LLMs should be able to find and use the new page despite the lack of traditional SEO signals.

What we learned

Initial findings and test iteration

After the page had been available in the llms.txt file for three weeks, none of the LLMs we tested (ChatGPT and Google AI Mode) were able to answer a prompt about the page content using the test page URL. This implied they had not discovered the page.

Our Managing Director, Adam Griffith, queried whether perhaps this was because they were obeying the SEO 'noindex' directive.

To test this, we iterated by keeping all test variables the same but removing the meta robots 'noindex' tag. 

Final results

After another two weeks, the test results concluded: ChatGPT remained unable to answer prompts relating to the test page’s content, but Google’s AI Mode was able to cite the page:

Screenshot showing AI Mode’s response and citation source

Success! AI Mode referenced the correct test page to generate its answer to our test prompt.

Further analysis showed Google had indexed the page after discovering it via the llms.txt file:

Screenshot showing Google had indexed the client's llms.txt file.

1. Discovery is possible, but limited

Google Search Console confirmed that Google did discover the URL specifically through the llms.txt file:

Google Search Console URL Inspection Tool showing the page was discovered via llms.txt as the referring page.

Google Search Console confirms the test page was discovered through the llms.txt file, not a referring page or an XML sitemap.

2. Indexation still governs visibility

Our test proved that llms.txt cannot bypass traditional search rules. For the period the page was set to 'noindex', it remained invisible to AI outputs. The file acted as a 'crawl hint' for a bot, but not a "source of truth" for the model.

Once the 'noindex' tag was removed, Google could index the URL it had already found in the llms.txt file it indexed:

Screenshot of Google search results showing the test page indexed in Google.

Google indexed the test page after the noindex tag was removed.

3. Impact varies by market

We ran this test across Australia, the US, and Korea. Interestingly, only the Korean variant was indexed in Google and eventually used by Gemini.

This suggests that, in markets where the index is less saturated, a crawl hint like llms.txt might have a faster impact, but in competitive markets like Australia and the US, it wasn't enough to overcome the lack of internal linking and authority.

Key takeaway: Discovery still depends on SEO fundamentals

The data is clear: llms.txt alone does not enable LLMs to discover web pages.

AI systems still rely on the same foundations that underpin search: indexable content, smart internal linking and site authority. When those signals are absent, a reference in llms.txt is not enough to surface or cite a page.

Where the file may have a role is not discovery, but interpretation. By presenting a clean, structured version of key URLs, llms.txt could support how models process and synthesise content once it has already been found – particularly on complex pages where heavy CSS and JavaScript can obscure the core information.

This is consistent with Luminary’s approach to SEO and generative engine optimisation, which prioritises indexable content, internal linking and content structured for both search engines and AI systems.

Thank you to our client for partnering with us on this experiment.

Shayna Burns
Shayna Burns

15 July 2026

4 minute read

Enjoyed this article? Share it...

Want to tap into the expertise of an agency that’s been in operation since 1999?

Get in touch

Keep Reading

Want more? Here are some other blog posts you might be interested in.

Follow
Contact Us
1300 554 891
[email protected]

Our locations

Stay informed
Our best articles, observations and news delivered to your inbox
Subscribe
Flags

We acknowledge the Traditional Custodians of the land on which we work, live and engage. We pay respect to Elders past and present.

Luminary is a registered trademark of Luminary Digital Pty Ltd

Terms of Services
Privacy Policy