How can you log AI crawler visits yourself?
You can monitor crawler visits through your website logs. If your site uses Cloudflare, another option is to run a Worker at the edge to capture request information and send it to a collection endpoint you specify. However, matching User-Agent (UA) strings only identifies requests that claim to come from a particular crawler; it does not verify the crawler’s actual identity. The implementation details below are based on an iThome example. That article describes Cloudflare behavior as of September 7, 2026; current rules have not been reverified here.
First, decide where to log requests and which fields to capture
Semrush recommends using logs to track how often AI crawlers visit and which pages they frequently crawl. This shows crawling activity, but it should not be taken as direct evidence that a platform values a particular piece of content, much less that it has used that content.
The iThome example uses a Cloudflare Worker to monitor requests. It creates records for requests matching a UA list and sends them to a collection endpoint via HTTP POST, rather than storing the data in the Worker. The recorded fields include:
- The timestamp, identification label, and original UA, to preserve the basis for classification.
- The request method, hostname, path, and query string, to identify what was crawled.
- The IP address, country, and, where available, ASN, organization, data center, and verified bot category.
The edge-specific fields come from request.cf. The example uses an empty object as a fallback and records missing values as null to avoid read errors when the runtime does not provide this information.
UA matching is a starting point, not identity verification
The example uses a list you maintain yourself, converts the UA to lowercase, and checks whether it contains specified strings. For example, a UA containing GPTBot is labeled GPTBot without verifying the source IP or ASN, so requests spoofing the same string will also be logged. Requests that do not match the list are excluded from this log.
Reports should describe these as “UA-matched requests,” not confirmed visits from official crawlers. Even if verifiedBotCategory is saved, the article does not explain how to interpret that field, so it should not be treated as proof of a specific crawler’s identity.
Keep log delivery from delaying the website’s response
The example uses ctx.waitUntil() to send records to the collection endpoint, allowing the website to respond to visitors without waiting for the POST to finish. The code also catches exceptions during delivery. According to the article’s description at the time, all waitUntil() tasks share a maximum execution window of 30 seconds after an HTTP Worker sends its response.
This does not guarantee delivery: the example simply ignores exceptions from failed sends, so a successful website response does not mean that the record was successfully saved.
Test the workflow, then check the deployment architecture
Following the example, you can test locally with wrangler dev, send a request with a GPTBot UA, and check whether the collection endpoint receives the expected fields. This only verifies the matching and delivery workflow; it does not prove that a real crawler visited. Local request.cf data should not be treated as production traffic information either.
To avoid logging a request again if it loops back, the example first checks for a header that marks it as already processed. Before forwarding a request, it copies the Request and adds that header to the copy. The article also warns that if the origin itself is a Worker bound to a Route, adding this monitor directly on top may block requests. Check your architecture before deployment rather than treating the example as a universal configuration.
Logging AI crawler visits yourself: perspectives from across the web
What else is needed to verify a crawler’s identity?
In addition to UA matching, you can add identity verification based on the source IP or ASN. iThome describes an approach that involves obtaining each provider’s official IP list, updating it regularly, and checking against it at the edge. This requires more engineering work than simply matching strings. The material does not provide complete verification steps or the providers’ lists, so the example should not be treated as already implementing identity verification.
What compatibility date should you put in the configuration file for local testing?
The iThome example uses Wrangler to run a Worker locally. The compatibility_date in the configuration file should be a date actually supported by the installed version of Wrangler. The article’s author was unable to start the Worker after entering a date beyond what the local runtime supported, so if you encounter a compatibility-date error, start by checking the version and the error message.
Why can’t you directly modify the incoming request headers?
According to the explanation in the iThome example, the incoming request.headers is read-only, and setting headers directly causes an error. To add a marker that prevents duplicate processing to the forwarded request, the example first creates a copy with new Request(request), then modifies the copy’s headers. This is the approach used in that Worker implementation.
References
- How Do Technical SEO Factors Impact AI Search? [Study]www.semrush.com · 2026-01-05T15:12:00Z — Source used in the article above.
- 靠人不如靠己:不到 90 行 Cloudflare Worker,親手榨出 AI 爬蟲的真實足跡ithelp.ithome.com.tw — Discusses: What else is needed to verify a crawler’s identity?; What compatibility date should you put in the configuration file for local testing?


Join the discussion 0 comments
What are you working through? Share a question or experience and let’s find the next step.
No comments yet. Start the conversation.