Can verified crawl logs prove how content was subsequently used?
Crawl logs alone cannot prove subsequent use. Even when a bot has been verified as genuine, a successful crawl proves only that it retrieved the content—not that the page was cited, paraphrased, displayed in an answer, or later mentioned by a model. Consider three questions separately: who made the request, whether content was successfully retrieved, and how that content was subsequently used. Establishing the first two does not automatically prove the third.
Verifying identity does not establish where the content went
In a June 2026 article, Duane Forrester noted that confirming a crawl and confirming subsequent use are different matters. A genuine bot may retrieve a page, but the log itself cannot tell the site owner whether the content appeared in an answer a user saw, whether it was attributed, or whether it was used at all. Even when discussing model training, crawl logs cannot establish that the model will later mention the website by name.
- Crawler identity: Confirming that a visitor is a genuine bot does not tell you how it later used the content.
- Crawling activity: Sending a request and successfully retrieving content are different events and should not be counted as equivalent.
- Content use: Crawl counts cannot tell you whether content was cited, paraphrased, or included in an answer.
Check the status code first: a request does not mean a file exists
When assessing llms.txt traffic on your own site, Ahrefs’ Louise Linehan recommends checking the logs first. In a June 2026 article, she noted that Ahrefs had not published the file at the time but still received bot requests for it. The 404 response was the key detail: the traffic was real, but the requested file did not exist.
Seeing /llms.txt in a report therefore does not justify concluding that “AI has read the instructions provided by the website.” First confirm the requested URL, bot type, and response status. Filtering only for URLs containing “llms.txt” may also capture articles about the file rather than the file itself. Even successful retrieval does not establish that its contents were used.
Neither crawling nor indexing proves trust
In an August 2026 article, Mark Williams-Cook used a cats.txt file describing his cats’ job roles as an example: several crawlers requested the file, and Google indexed it. This example illustrates that crawling does not prove content was given extra weight, trusted, or used. Inclusion in an index is likewise not an endorsement of its truthfulness, usefulness, or importance.
To establish how content was used, you need evidence directly related to that use. If you want to know whether content appeared in an answer or received a citation, examine the answers and citations, not download records. But if you want to know whether content influenced answer generation, a site owner’s server logs alone cannot independently establish that.
In September 2026, Suganthan Mohanadasan reported that Cloudflare was running a pilot to shift its default billing approach from pay-per-crawl to pay-per-use, because retrieving a page does not mean its content was used. He also cautioned that pay-per-use depends on platforms reporting usage and value. Site owners cannot independently determine how content influenced an answer solely from their own logs. This is a trade-off in the reported pilot, not a rule currently applied by all platforms.
Verified crawl logs: perspectives from across the web
If an answer includes information found only in llms.txt, does that prove the system places special trust in the file?
That conclusion cannot be drawn directly. Williams-Cook noted that such a result could also be consistent with ordinary retrieval-augmented generation. An answer containing information from the file does not mean the file was treated as a specially trusted source. Determining whether it received extra weight or trust requires additional evidence supporting that specific use.
Does pay-per-use allow site owners to independently verify whether their content influenced an answer?
Not necessarily. Mohanadasan noted that once a page leaves the server, site owners cannot independently determine whether it influenced an answer solely from their own records. Pay-per-use therefore relies on platforms reporting usage and value. By contrast, pay-per-crawl allows site owners to check requests and payments independently, but still does not establish subsequent use. This is a trade-off in the reported pilot, not a rule currently applied by all platforms.
References
- 81.8% Of My ‘AI Assistant’ Traffic Was Fake. The Googlebot Number Was Worsewww.searchenginejournal.com · 2026-06-25T13:30:24+00:00 — Source used in the article above.
- We Analyzed 137K Sites: 97% of llms.txt Files Never Get Readahrefs.com · 2026-06-15T10:27:40+00:00 — Source used in the article above.
- How Cats.txt Showed LLMs.txt Evidence Is GEO Astrologywww.searchenginejournal.com · 2026-08-07T12:00:42+00:00 — Discusses: If an answer includes information found only in llms.txt, does that prove the system places special trust in the file?
- I Made My Website Charge AI Agents A Penny Per Page, Then I Watched Claude Pay Itwww.searchenginejournal.com · 2026-09-17T19:00:29+00:00 — Discusses: Does pay-per-use allow site owners to independently verify whether their content influenced an answer?


Join the discussion 0 comments
What are you working through? Share a question or experience and let’s find the next step.
No comments yet. Start the conversation.