Dries Buytaert

In January, I made every page on my site available as Markdown. Immediately AI crawlers quickly found the Markdown versions. I was excited, but excitement isn't data. Now that the dust has settled, I pulled a month of Cloudflare logs and analyzed them.

I compared how much AI bots crawl my site to how often AI answer engines link back. For every citation they sent, their crawlers had fetched 1,241 pages. That is a lot of reading for very little traffic in return. It is the deal AI is offering creators right now, and it is not a good one.

People also asked whether serving Markdown reduced my bot traffic since the files are smaller. It does not. Bots fetch both versions, and my crawler traffic increased by about 7%. Offering a lighter format does not replace the heavier one. It simply gives bots more to crawl.

As the table below shows, several AI companies crawl my site. Some fetch thousands of pages each month, but very few request the Markdown versions.

Interestingly, OpenAI runs three bots with different roles. OAI-SearchBot indexes content for search, GPTBot crawls for training data, and ChatGPT-User fetches pages in real time during live ChatGPT sessions.

When I added Markdown support to my site, I exposed it in two ways. The first is through dedicated Markdown URLs: append .md to any page and you get the Markdown version. The second is through content negotiation, where the original URL returns Markdown instead of HTML when the request includes an Accept: text/markdown header.

No AI crawler uses content negotiation. Not one. They only discover the Markdown pages through the dedicated URLs, and only via the auto-discovery link. To be fair, the auto-discovery link points to the .md version. As a next step, I will experiment with pointing it to the content-negotiated version instead.

And then my favorite: llms.txt, a proposed standard where sites describe their content for AI systems. My site received 52 requests for it last month. Every one came from SEO audit tools. Not a single request came from an AI answer engine or crawler. (I don't use or pay for SEO tools but apparently that doesn't stop them from auditing my site.)

For fun, we also looked across Acquia's entire hosting fleet and found about 5,000 llms.txt requests out of 400 million total (0.001%), nearly all from SEO tools. llms.txt is a solution looking for a problem. The bots it was designed for don't look for it.

So should you add Markdown support to your site? Probably not. There is no clear benefit today. It does not reduce crawl traffic, and I can't demonstrate that it improves how AI systems use your content.

We do know that AI systems love Markdown, and they fetch it when it is available. At best, it may become more useful over time.

If it is easy to add and you enjoy experimenting, go ahead. If it takes real effort, spend that effort on your content instead. What still works is what has always worked: clear writing, authoritative content, and timely publishing.