--- title: "We Logged 30 Days of AI Bot Traffic. About 2,500 Fetches Reached a Live Answer" description: "In 30 days of groas.com edge logs, only about 30% of AI-bot hits to two key paths passed IP verification. Here’s why answer-linked fetches beat raw crawler counts as a visibility metric." url: "https://groas.com/post/we-logged-30-days-of-ai-bot-traffic-most" image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/6f84b502-ecbb-4031-b6ea-a8901bc9c613.png" published: "2026-10-17T05:25:40.831Z" modified: "2026-10-17T05:25:40.901Z" --- October 17, 2026 · 10 min read # We Logged 30 Days of AI Bot Traffic. About 2,500 Fetches Reached a Live Answer [Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/) ![A bouncer checks IDs at a night door while a long queue of people all wear 'BOT' name stickers, implying most claimed bot visitors are impostors.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/6f84b502-ecbb-4031-b6ea-a8901bc9c613.png) In this article 1. [The headline: about 30% passed verification on two key paths](#the-headline-about-30-passed-verification-on-two-key-paths) 2. [Finding 1: the bot name did most of the pretending](#finding-1-the-bot-name-did-most-of-the-pretending) 3. [Finding 2: 250 verified fetches can still mean zero answers](#finding-2-250-verified-fetches-can-still-mean-zero-answers) 4. [Finding 3: answer-linked activity clustered on a few paths](#finding-3-answer-linked-activity-clustered-on-a-few-paths) 5. [Put prompt results beside logs, not in place of them](#put-prompt-results-beside-logs-not-in-place-of-them) 6. [What I’d change on Monday](#what-id-change-on-monday) 7. [The limit, and the decision](#the-limit-and-the-decision) About **2,500 verified fetches were linked to a live answer** across 30 days of groas.com logs. That is the number I would report. The raw logs looked far more impressive: `/robots.txt` alone drew 13,494 hits claiming an AI crawler user agent, and the homepage drew 6,506. On both paths, roughly 70% failed verification against the bot vendors’ published IP ranges. I have spent enough late nights mining search terms to distrust any dashboard that gets bigger when you stop asking questions. So we split the traffic by path and bot, then asked three different things of it: Did a request claim to be an AI bot? Did its IP match that bot’s published range? Could we link the verified fetch to a live answer served to a user? Those are not interchangeable questions. As [Canonry explains in its server-log walkthrough](https://canonry.ai/blog/ai-traffic-server-logs), bots do not run JavaScript, so Google Analytics misses their requests. The evidence starts in access logs, but a user agent string is only a claim until the IP checks out. ## The headline: about 30% passed verification on two key paths The first cut is from our edge logs for a single 30-day window. _Total hits_ means requests claiming an AI crawler user agent. _Verified_ means the source IP matched the published range for the bot it claimed to be. This table does **not** say whether any of those requests fed an answer. | Path | Total claimed hits | IP-verified hits | Verified share | | --------------- | -----------------: | ---------------: | ---------------: | | `/robots.txt` | 13,494 | 3,912 | **29.0%** | | Homepage `/` | 6,506 | 1,930 | **29.7%** | | Decorative SVGs | Thousands | Not counted here | **Roughly 1–2%** | _Source: groas.com edge logs, 30-day window. The SVG figure is a verified share, not a verified-hit count._ A raw crawler chart makes `/robots.txt` look like a busy front door. The IP check says most of that apparent traffic cannot be attributed to the bots named in the requests. Decorative SVGs were starker: only about one or two requests in a hundred passed verification. **A claimed bot hit is not verified bot traffic**, much less evidence that someone saw your brand in an AI answer. Across all logged pages, about **2,500 verified fetches were linked to live answers**. That count comes from joining verified requests to answers served to users, rather than from a mention score or the raw crawler total. It is the useful third cut, with an important limit: the join shows that a fetch was associated with an answer, not that the answer cited the fetched page or copied a particular sentence from it. ![Funnel from claimed AI-bot hits to IP-verified fetches to fetches linked to live answers](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/90850f43-ae51-4027-a653-9d49dbc34fd7.png) A user agent is self-reported. Anyone can put `ChatGPT-User` in a request header. Verification means checking the source IP against the vendor JSON for that exact bot, not merely checking its network operator. This is not a fussy distinction invented for a cleaner spreadsheet. [HUMAN found that 5.7% of traffic claiming an AI crawler user agent was spoofed](https://www.humansecurity.com/learn/blog/ai-crawler-spoofing-chatgpt-mistral-perplexity/), with `ChatGPT-User` abused at roughly one in five. [DataDome reported a 45% increase in AI agent spoofing from February to July 2026](https://ppc.land/full-bot-protection-drops-to-2-4-of-popular-websites-datadome-finds/). Our unverified share is much larger than HUMAN’s broader figure. That is a reason to examine this site’s traffic, not to promote our percentage into an internet average. ## Finding 1: the bot name did most of the pretending Start with `/robots.txt`. It drew 13,494 requests claiming an AI bot identity, but only 3,912 passed the IP check: a roughly 71% haircut. The homepage was nearly identical, with 1,930 verified out of 6,506 claimed, or a 70.3% haircut. I used to tell clients bot logs were directionally true. I was wrong. On this small site with open robots, the unfiltered direction was mostly costume. Decorative SVGs gave the game away. Thousands of requests, a verified share near one or two percent. No answer engine needs a divider graphic to write a sentence about Google Ads pricing. An unverified request that claims a bot identity and walks an asset tree is not evidence of search visibility. It belongs outside the visibility count, however authoritative its user agent looks in a CDN panel. The IP check needs to be specific. HUMAN reported that much of the spoofed `ChatGPT-User` traffic it observed came from [the same ASN as real traffic, using adjacent IPs and rotation across more than 10,000 IPs](https://www.humansecurity.com/learn/blog/ai-crawler-spoofing-chatgpt-mistral-perplexity/). Checking the ASN alone would let those requests through. Checking the exact published CIDR for the claimed bot is the difference between verification and recognising a familiar-looking neighbourhood. **If the report verifies only the ASN, do not label the result verified AI visibility.** ## Finding 2: 250 verified fetches can still mean zero answers Our category pages show why IP verification is necessary but insufficient. Several drew roughly **250 verified fetches each** during the window. These were real bot requests by our check, not merely familiar names in a header. Yet those pages fed **zero live answers** in the answer-linked count. A report built from verified crawls would make them look productive. A report built from what reached a live answer would not. That gap is easier to understand when you separate bulk crawling from a user-triggered fetch. [Canonry distinguishes bulk crawlers such as `GPTBot` and `ClaudeBot` from user-facing fetchers such as `ChatGPT-User` and `Claude-User`](https://canonry.ai/blog/ai-traffic-server-logs). A bulk crawl can establish that an engine has read a page. It cannot establish that the page was read while answering someone’s question. Our category pages received the former kind of attention without showing up in the latter count. The content distinction matters too. These category pages were broad and thin on a direct answer; they were easy to revisit without supplying much an engine could use in a response. That is an interpretation of the pattern, not something an edge log can prove about a model’s reasoning. The measured result is narrower and still useful: verified fetches accumulated, while answer-linked fetches did not. Citations are a further step beyond that. Across 22 categories, [Semrush found that fewer than half of the top-100 mentioned brands were also top-100 cited domains](https://martech.org/your-ai-visibility-score-may-be-hiding-a-problem/). A brand can be present in an answer without its site being cited. A page can be fetched without appearing in the answer at all. **Crawled, used in a live answer, mentioned and cited are four different labels.** Put them in separate columns before deciding which page has won. ## Finding 3: answer-linked activity clustered on a few paths The roughly 2,500 answer-linked verified fetches were not spread evenly across the site. Our edge-log and answer join showed this shape: | Path | Verified fetches linked to a live answer | | -------------------------------- | -----------------------------------------------: | | Homepage `/` | **801** | | Google Ads updates for 2026 post | **402** | | Several category pages | **0**, despite roughly 250 verified fetches each | _Source: groas.com verified edge logs joined to live answers during the same 30-day window. These are answer-linked fetches, not citation counts._ `/robots.txt` was also rechecked within many live sessions. It matters to the activity trail, but it does not supply answer text, so I would not present it as a content win alongside the homepage and the updates post. The window is short enough that the 2026 post’s burst of attention can tilt the distribution. Still, the operating lesson is plain: **a sitewide bot total hides which URLs were present when answers were served.** That is the part a content owner can inspect and change. The bot names hid another split. In our join, `ChatGPT-User` accounted for most live-answering fetches, including requests to the homepage and the updates post around the time a question was asked. `Claude-User` showed steadier reading and far fewer fetches tied to an answer served seconds later. I would not collapse those patterns into one AI visibility score. The platforms also produce different answer shapes: [Semrush found differences among Gemini, ChatGPT and AI Overviews in brand mentions, source entries and overlap](https://martech.org/your-ai-visibility-score-may-be-hiding-a-problem/). One bot’s strong count tells you little about another platform’s answers. ## Put prompt results beside logs, not in place of them Most businesses I talk to start with one of two reports. One is a prompt dashboard: run a fixed set of buyer questions across AI search products and record mentions, citations and sentiment. [Search Engine Land describes a Brand Visibility Score based on answers mentioning the brand divided by total answers](https://searchengineland.com/measure-brand-visibility-ai-search-464524); [HubSpot frames measurement around visibility, engagement and revenue](https://blog.hubspot.com/marketing/ai-search-visibility-roi). That tells you what appeared in sampled answers. By itself, it does not tell you which page was fetched near the answer or whether your server was asked at all. The other report is a firewall or CDN bot total. It tells you something knocked. Without IP verification, it cannot even confirm who knocked. With verification, it still cannot tell you whether a fetch was associated with a live answer. Our earlier [postmortem on 3,800 fetches that produced zero citations](https://groas.com/post/postmortem-3-800-ai-bot-fetches-on-our-c) dealt with the page-level gap. This 30-day log cut deals with the measurement gap. Neither a prompt chart nor an edge log closes both on its own. Here is the practical join I would keep: 1. **Hold the questions steady.** Use a fixed set of 25 to 50 buyer prompts naming your category. Each week, record whether each answer mentioned you, cited you with a link and how it described you. Keep ChatGPT, Gemini and AI Overviews separate. 2. **Split requests by path.** Record claimed AI-bot hits, IP-verified hits and verified fetches linked to a live answer. Do not silently promote a request from one column to the next. 3. **Compare the paths with the answers.** Join by date, prompt, bot name and path where the records allow it. Inspect which URLs appear around cited answers, then read those pages as a buyer would. The mismatches are the useful part. If claimed hits rise while verified hits do not, fix the filter before telling anyone visibility grew. If verified fetches rise but answer-linked fetches do not, more crawling is not the result you wanted. If answer-linked fetches rise while your fixed prompts still show no citations, inspect what the page says and what the answer actually used; do not call the fetch a citation. The join narrows the question. It does not give you a window into the model. ## What I’d change on Monday If I owned this report for a business starting Monday, I would keep three measures: - **Verified live-answer fetches by path.** This is the crawl measure closest to an answer someone saw, rather than to a bot’s claim about itself. - **Citation rate on a fixed prompt set.** Track cited prompts divided by total prompts each week, separately for ChatGPT, Gemini and AI Overviews. - **The URLs associated with cited answers.** Join prompt results to verified requests by date and bot where possible. Use the pairing to decide what to inspect or rewrite, without pretending timing proves causation. I would stop using raw AI-bot hits as a visibility headline. On our two busiest paths, they were about 3.4 times the verified count; on SVGs, the verified share was only roughly one to two percent. I would also stop treating mention-only scores as citation scores. The [mention-versus-citation gap Semrush observed](https://martech.org/your-ai-visibility-score-may-be-hiding-a-problem/) is precisely why those labels need their own lines in the report. This will not work for everyone. If your site gets fewer than a few hundred verified AI fetches a month, the path-level log split may be too thin to read. I would keep the fixed prompt tracking and wait for more volume rather than manufacture a trend from a handful of requests. ![Clay stop-motion scene of a marketer discarding fake bot logs and keeping a verified answer card](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c5dc0b80-5e95-4127-ac91-25dc1fc98f84.png) ## The limit, and the decision This is one small B2B site, 30 days of logs, open robots and a narrow topic cluster around paid search. Our roughly 70% unverified share is not an internet average. A larger site with different bot rules, or a publisher blocking AI crawlers, will see a different mix. The window is also short enough for one heavily read post, such as our 2026 updates post, to skew where answer-linked fetches appear. There is a harder limit. We can match a verified fetch to a live answer using timing, path and bot name. **We cannot see inside the model and prove that the fetch caused a citation.** Training-data memory may also contribute to what an answer says. That is why I want the prompt record beside the logs, and why I will not relabel the 2,500 answer-linked fetches as 2,500 citations. The decision the numbers support is boring, which is why I trust it. Stop reporting the big bot total. Report the **about 2,500 verified fetches linked to live answers in 30 days**, show which paths carried them, and compare those paths with a fixed prompt set every week. When a page gets read but does not appear in your sampled answers, inspect the page. When a buyer question gets answered without you, work on the answer your page fails to provide. Everything else in the AI visibility deck is scenery. Count what reached an answer, not what knocked. ## Frequently Asked Questions ### Why do raw AI bot hit counts overstate real crawler traffic? A user agent string is self-reported, so anyone can put ChatGPT-User in a request header. On groas.com, roughly 70% of claimed AI-bot hits on /robots.txt and the homepage failed IP verification against the bot vendors' published ranges. A claimed bot hit is therefore not verified bot traffic. ### How do I verify that a request really comes from an AI crawler like ChatGPT-User? Check the source IP against the vendor's published JSON for that exact bot, not merely its network operator. HUMAN found spoofed ChatGPT-User traffic often came from the same ASN as real traffic using IP rotation, so checking the ASN alone would let spoofed requests through. ### Can a page get lots of verified AI crawler visits but still never appear in AI answers? Yes. Several groas.com category pages drew roughly 250 IP-verified fetches each in 30 days yet fed zero live answers. Bulk crawlers such as GPTBot and ClaudeBot can read a page without that read being tied to answering a user's question. ### Is being mentioned in an AI answer the same as being cited? No, they are different labels. Semrush found that across 22 categories, fewer than half of the top-100 mentioned brands were also top-100 cited domains. A page can be fetched without appearing in an answer, and a brand can be mentioned without its site being cited. ### Which pages on the site actually fed AI answers? The homepage carried 801 verified fetches linked to live answers and the Google Ads updates for 2026 post carried 402, while several category pages had zero despite roughly 250 verified fetches each. A sitewide bot total hides which URLs were present when answers were served. ### How should I measure AI search visibility alongside my server logs? Run a fixed set of 25 to 50 buyer prompts each week, recording mentions and citations separately for ChatGPT, Gemini and AI Overviews. Split requests by path into claimed hits, IP-verified hits and verified fetches linked to a live answer, then join the two records by date, prompt, bot name and path. ### Does a verified fetch matched to a live answer prove the fetch caused the citation? No. The join uses timing, path and bot name to show a fetch was associated with an answer, but it cannot see inside the model or prove causation. Training-data memory may also contribute to what an answer says, so the author declines to relabel 2,500 answer-linked fetches as 2,500 citations. ### Does this log-based measurement method work for small sites with little AI bot traffic? Not always. If a site gets fewer than a few hundred verified AI fetches a month, the path-level log split may be too thin to read. In that case, keep fixed prompt tracking and wait for more volume rather than manufacture a trend from a handful of requests. ## Pay For Results, Not For Hours Businesses buy the outcome, agencies resell it, and groas answers for it either way. [See If You Qualify](https://groas.typeform.com/to/xC1bQNUT) ## Structured data ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/we-logged-30-days-of-ai-bot-traffic-most#webpage","url":"https://groas.com/post/we-logged-30-days-of-ai-bot-traffic-most","name":"We Logged 30 Days of AI Bot Traffic. About 2,500 Fetches Reached a Live Answer","description":"In 30 days of groas.com edge logs, only about 30% of AI-bot hits to two key paths passed IP verification. Here’s why answer-linked fetches beat raw crawler counts as a visibility metric.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/6f84b502-ecbb-4031-b6ea-a8901bc9c613.png"},"breadcrumb":{"@id":"https://groas.com/post/we-logged-30-days-of-ai-bot-traffic-most#breadcrumb"},"dateModified":"2026-10-17T05:25:40.901Z","mainEntity":{"@id":"https://groas.com/post/we-logged-30-days-of-ai-bot-traffic-most#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/we-logged-30-days-of-ai-bot-traffic-most#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"We Logged 30 Days of AI Bot Traffic. About 2,500 Fetches Reached a Live Answer","item":"https://groas.com/post/we-logged-30-days-of-ai-bot-traffic-most"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/we-logged-30-days-of-ai-bot-traffic-most#article","url":"https://groas.com/post/we-logged-30-days-of-ai-bot-traffic-most","mainEntityOfPage":{"@id":"https://groas.com/post/we-logged-30-days-of-ai-bot-traffic-most#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"We Logged 30 Days of AI Bot Traffic. About 2,500 Fetches Reached a Live Answer","description":"In 30 days of groas.com edge logs, only about 30% of AI-bot hits to two key paths passed IP verification. Here’s why answer-linked fetches beat raw crawler counts as a visibility metric.","datePublished":"2026-10-17T05:25:40.831Z","dateModified":"2026-10-17T05:25:40.901Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/6f84b502-ecbb-4031-b6ea-a8901bc9c613.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"answer, fetches, verified, bot, about, live answer, live, traffic","wordCount":2135,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/we-logged-30-days-of-ai-bot-traffic-most#faq","mainEntity":[{"@type":"Question","name":"Why do raw AI bot hit counts overstate real crawler traffic?","acceptedAnswer":{"@type":"Answer","text":"A user agent string is self-reported, so anyone can put ChatGPT-User in a request header. On groas.com, roughly 70% of claimed AI-bot hits on /robots.txt and the homepage failed IP verification against the bot vendors' published ranges. A claimed bot hit is therefore not verified bot traffic."}},{"@type":"Question","name":"How do I verify that a request really comes from an AI crawler like ChatGPT-User?","acceptedAnswer":{"@type":"Answer","text":"Check the source IP against the vendor's published JSON for that exact bot, not merely its network operator. HUMAN found spoofed ChatGPT-User traffic often came from the same ASN as real traffic using IP rotation, so checking the ASN alone would let spoofed requests through."}},{"@type":"Question","name":"Can a page get lots of verified AI crawler visits but still never appear in AI answers?","acceptedAnswer":{"@type":"Answer","text":"Yes. Several groas.com category pages drew roughly 250 IP-verified fetches each in 30 days yet fed zero live answers. Bulk crawlers such as GPTBot and ClaudeBot can read a page without that read being tied to answering a user's question."}},{"@type":"Question","name":"Is being mentioned in an AI answer the same as being cited?","acceptedAnswer":{"@type":"Answer","text":"No, they are different labels. Semrush found that across 22 categories, fewer than half of the top-100 mentioned brands were also top-100 cited domains. A page can be fetched without appearing in an answer, and a brand can be mentioned without its site being cited."}},{"@type":"Question","name":"Which pages on the site actually fed AI answers?","acceptedAnswer":{"@type":"Answer","text":"The homepage carried 801 verified fetches linked to live answers and the Google Ads updates for 2026 post carried 402, while several category pages had zero despite roughly 250 verified fetches each. A sitewide bot total hides which URLs were present when answers were served."}},{"@type":"Question","name":"How should I measure AI search visibility alongside my server logs?","acceptedAnswer":{"@type":"Answer","text":"Run a fixed set of 25 to 50 buyer prompts each week, recording mentions and citations separately for ChatGPT, Gemini and AI Overviews. Split requests by path into claimed hits, IP-verified hits and verified fetches linked to a live answer, then join the two records by date, prompt, bot name and path."}},{"@type":"Question","name":"Does a verified fetch matched to a live answer prove the fetch caused the citation?","acceptedAnswer":{"@type":"Answer","text":"No. The join uses timing, path and bot name to show a fetch was associated with an answer, but it cannot see inside the model or prove causation. Training-data memory may also contribute to what an answer says, so the author declines to relabel 2,500 answer-linked fetches as 2,500 citations."}},{"@type":"Question","name":"Does this log-based measurement method work for small sites with little AI bot traffic?","acceptedAnswer":{"@type":"Answer","text":"Not always. If a site gets fewer than a few hundred verified AI fetches a month, the path-level log split may be too thin to read. In that case, keep fixed prompt tracking and wait for more volume rather than manufacture a trend from a handful of requests."}}]}]} ```