A question I hear a lot from business owners goes like this: “We spent three months working on AI search. Did it actually do anything?” Usually they open GA4, find a few hundred visits from ChatGPT, and feel either disappointed or confused. Many believe that if AI search mattered, it would show up as a big line in their referral report.
It often doesn’t, and that doesn’t mean nothing happened. A lot of AI influence never turns into a click you can track. Someone reads your name in ChatGPT, then types your brand into Google two days later, and your analytics calls it organic search or direct traffic.
So instead of asking “how many clicks did AI send?”, I suggest a better question: “What changed that would not have changed without our AI search work?” That is an experiment question, and in this guide I will show you simple ways to answer it: pre/post tests, page-group holdouts and branded demand checks. If you need formulas for return on investment, see my separate guide to AI search ROI. For paid channel tests, my guide to incrementality testing covers the basics. This post focuses on AI visibility only.
Why click-based attribution undercounts AI influence
Ahrefs said it plainly in its October 2026 guide on measuring AI search visibility:
“Most AI influence happens out of sight.”
— Ahrefs, How to Measure AI Search Visibility When Attribution Falls Short
Let me explain the main reasons, in plain words:
- Most AI answers don’t produce a click. The answer itself is the product. Ahrefs’ February 2026 study on AI Overviews found that top-ranking pages get about 58% fewer clicks than expected when an AI Overview appears.
- Some AI visits arrive without a referrer. Apps and some browsers strip the referral information, so the visit lands in “direct.”
- Google mixes AI and classic search. Google’s documentation says traffic from AI Overviews and AI Mode is reported in Search Console’s Performance report within the “Web” search type. You can’t simply filter it out of normal organic traffic there.
- The buyer journey is delayed. A person may see your brand in an answer today and buy next month from a different device.
- There is no query report for ChatGPT. As Ahrefs put it, “There's no Search Console for ChatGPT.”
GA4 does help a little. It now has an AI Assistant default channel. According to Google’s channel group documentation, it covers visits from AI chatbots such as ChatGPT, Gemini and Copilot, and it excludes Google’s AI Overviews and AI Mode. That gives you clean click data, but only for the clicks. If you haven’t set this up, my guide to tracking AI traffic in GA4 walks you through it.
What “incremental” means for AI visibility
Incremental simply means “extra.” It is the result you got because of your work, minus what would have happened anyway.
Here is a simple example. Say your branded searches grew 20% this quarter. Was that your AI work? Maybe. But if your whole category grew 15% because of a seasonal peak, your incremental lift is closer to 5 points, not 20.
To measure the extra part, you need a comparison. There are three kinds that work for AI search:
| Method | What you compare | Best when | Main weakness |
|---|---|---|---|
| Pre/post test | The same metric before and after a change | You make one clear push, like a PR campaign or a big content update | Seasonality and other changes can fool you |
| Page-group holdout | Pages you optimized vs similar pages you left alone | You have many similar pages (products, locations, guides) | Needs enough pages and enough tracked prompts |
| Branded demand correlation | AI visibility trend vs branded search and direct traffic | Referral data is thin or missing | Shows a relationship, not proof of cause |
None of these is perfect. Used together, they give you a much more honest answer than a referral report alone.
Running a pre/post test around a content or PR push
A pre/post test is the easiest place to start. You pick a moment when you change something, and you compare a window before with a window after.
- Write down the change and the date. For example: “Published 12 comparison pages and a pricing page update on March 3.”
- Pick a small set of AI prompts the change should affect. Ten to thirty is enough. My guide to prompt research shows how to choose them.
- Record a baseline for at least two to four weeks before. Track mention rate (how often your brand appears), citation rate and AI referral sessions.
- Make the change, and change nothing else if you can. If a big ad campaign starts the same week, your test is muddied.
- Measure the same metrics for the same length of time after. Use equal-length windows, like 28 days vs 28 days.
- Compare against a control. This can be a set of prompts the change should not affect, or a competitor’s visibility on the same prompts.
- Write down the result, including “no change.” A null result is still useful.
Why repeated runs matter in pre/post tests
AI answers are not stable. A SparkToro and Gumshoe study, reported by Search Engine Land in January 2026, had 600 volunteers run 12 prompts nearly 3,000 times. The study found that the chance of getting the same list of brands twice was under 1 in 100. Its authors’ lesson is one I agree with:
“Repeat presence means something. Exact rank does not.”
— Search Engine Land, AI recommendation lists repeat less than 1% of the time
So don’t compare a single answer before with a single answer after. Run each prompt many times, or use a tool that does, and compare the share of answers that mention you. Ahrefs makes the same point in its article on why you can’t track AI like traditional search: individual mentions jump around, but aggregated data is far more stable.
Page-group holdouts: optimize some pages, not others
A holdout is the closest thing to a real experiment you can run on your own site. You split similar pages into two groups. You improve one group (the test group) and leave the other alone (the holdout). Then you compare.
How to set up a holdout
- Find a set of similar pages. For example, 40 service-area pages, 60 product category pages or 30 “best X for Y” guides.
- Match them into pairs by traffic, topic and age. Two pages with similar clicks and similar prompts go in a pair.
- Randomly assign one page from each pair to test and one to holdout. A coin flip is fine. Randomness stops you from picking your favorite pages for the test group.
- Apply the same change to every test page. For example: add a clear answer paragraph, a facts table, an updated date and original data.
- Track both groups for 6–12 weeks. Watch AI citations to those URLs, AI referral sessions to those URLs, and Search Console clicks and impressions.
- Compare the change in each group, not the raw numbers.
Example: a hypothetical holdout on 40 location pages
Let me show the math with an example. These numbers are made up for illustration.
| Metric (28 days) | Test group: before | Test group: after | Holdout: before | Holdout: after |
|---|---|---|---|---|
| AI citations to these URLs | 50 | 90 | 48 | 60 |
| AI Assistant sessions (GA4) | 120 | 190 | 115 | 135 |
| Search Console clicks | 2,000 | 2,150 | 2,050 | 2,100 |
Citations in the test group rose 80% (50 to 90). In the holdout, they rose 25% (48 to 60). So the incremental lift is about 55 percentage points. Something lifted all pages, perhaps a model update or a seasonal rise, but the test pages rose much more.
For sessions, the test group rose about 58% and the holdout about 17%, so roughly 41 points of lift. Search Console clicks barely moved in either group, which tells me this change helped AI visibility more than classic rankings. That is a useful finding on its own.
One honest note: with only 20 pages per group, a couple of pages can swing the result. I would want to see the pattern hold in a second round before rolling the change out everywhere.
Using branded search and direct traffic as lagging signals
When people see your brand in AI answers, some of them will search for you later. That makes branded search one of the best indirect signals you have. Ahrefs suggests comparing AI mentions with branded search volume over time, and I agree. Its example showed AI mentions up 100% and branded search volume up 15% over the same period.
Here are the lagging signals I watch, and where to find them:
- Branded clicks and impressions in Search Console. My guide to using branded search as an AI visibility proxy explains how to filter them.
- Direct traffic to deep pages. People rarely type a long URL. A rise in direct visits to product or pricing pages can point to copied links from AI answers.
- “How did you hear about us?” answers on forms. Ahrefs shared that ChatGPT and Claude together accounted for 21.7% of its own self-reported signups. My post on self-reported attribution for AI covers how to set this up.
- Branded mentions on the web. Ahrefs’ study of about 75,000 brands found branded web mentions correlated with AI mentions at around 0.66–0.71, much higher than links.
Ahrefs also gives a useful rule of thumb for direct traffic: if direct traffic converts at 2% and you see 1,000 extra unexplained visits, roughly 20 conversions might be AI-driven. I treat that as an estimate to discuss, not a number to put in a board report as fact.
Pulling Search Console data for this
If you need more than the 1,000 rows the interface shows, use the Search Console API. Google’s Search Analytics API reference says each request can return up to 25,000 rows, and you can page through more with startRow. Group by date and query, then tag brand queries with a simple list of your brand names and common misspellings.
Matching visibility tracking to pipeline by date
The final step is to connect visibility to money. You won’t get perfect attribution, but you can line things up by date and look for patterns.
- Build one weekly table with these columns: week, AI mention rate, AI citations, AI Assistant sessions, branded clicks, direct sessions, leads, pipeline value.
- Add a notes column for anything that happened: launches, PR wins, ad changes, model updates, holidays.
- Chart AI mention rate against branded clicks and look for a lag. Does branded search rise two or four weeks after mentions rise?
- Check the leads that came from AI Assistant sessions and from “heard about you in ChatGPT” answers. Look at their deal size and close rate.
- Review quarterly, not weekly. Weekly numbers are too noisy to judge.
If you want this in a dashboard, my guide to an AI search dashboard in Data Studio has a template you can adapt.
Reading noisy AI search results honestly
This part matters most. It is very easy to fool yourself with AI data. Even Ahrefs admits it:
“Nobody can definitively prove the value of AI search—us included.”
— Ahrefs, How to Measure AI Search Visibility When Attribution Falls Short
Here is how I read results without fooling myself:
| What you see | Possible honest reading | What to do |
|---|---|---|
| Test and holdout both rose a lot | Something outside your test moved everything, such as a model update | Report the difference only. Note the outside event. |
| Test rose, holdout flat, tiny sample | Promising, but could be chance | Repeat on a new set of pages before scaling. |
| Mentions up, branded search flat | Visibility hasn’t reached buyers yet, or prompts aren’t commercial | Check whether your tracked prompts match real buying questions. |
| Citations up, sessions flat | Answers cite you but people don’t need to click | Look at mention quality and branded demand instead of clicks. |
| Big spike in one week | Likely noise or a one-off answer change | Wait for three or more weeks of data. |
A few rules of thumb I use:
- Decide what “success” means before the test starts. For example, “at least 15 points more mention-rate lift in the test group than in the holdout.”
- Don’t chase a single metric. Ahrefs’ AI search trends report mentions Wil Reynolds seeing a 1,900% jump in ChatGPT citations to one page with little business impact.
- Keep a written log. Memory is a poor judge of what caused what.
A simple AI search test log template
Here is the template I suggest. Copy it into a spreadsheet and fill one row per test.
| Field | What to write | Example |
|---|---|---|
| Test name | Short, specific name | “Facts tables on location pages” |
| Hypothesis | What you expect and why | “Clear facts will raise ChatGPT citations to these URLs” |
| Method | Pre/post, holdout or correlation | Holdout, 20 vs 20 pages |
| Prompts tracked | Number and where the list lives | 25 prompts, “prompts-locations” tab |
| Primary metric | The one number that decides success | Mention rate across 10 runs per prompt |
| Success threshold | Set before you start | +15 points vs holdout |
| Start / end dates | Equal windows before and after | Mar 3 – Mar 30 vs Mar 31 – Apr 27 |
| Outside events | Anything that could muddy results | Model update Apr 10, Easter promo |
| Result | Numbers for test and control | +80% vs +25% |
| Decision | Scale, repeat or drop | Repeat on 40 more pages |
Questions marketers ask about proving AI search impact
Can I prove AI search ROI with GA4 alone?
Not fully. GA4’s AI Assistant channel shows clicks from chatbots, but most AI influence doesn’t produce a click. Combine it with branded search, direct traffic and self-reported attribution.
How long should an AI search test run?
I suggest at least four weeks before and four weeks after for a pre/post test, and 6–12 weeks for a holdout. Shorter tests are too easily swayed by answer variability.
How many times should I run each prompt?
Several times per measurement period. Because brand lists rarely repeat exactly, you want the share of answers that mention you, not a single snapshot.
What if my site is too small for a holdout?
Use a pre/post test with a control set of prompts, and watch branded search over a longer period. Small sites can still learn a lot from a careful log.
Is a correlation between AI mentions and branded search enough?
It is a useful signal, not proof. Correlation can come from a third cause, like a seasonal peak. Use it alongside a test where you can.
Where to start with your first AI search experiment
Pick one change you were going to make anyway, like updating a set of comparison pages. Split the pages into test and holdout, write your success threshold down, and track 20–30 prompts for a month on each side. That one test will teach you more than months of staring at a referral report.
For more context, read my framework for deciding which AI visibility data to act on, see how ChatGPT turns one prompt into many searches, and check the AI search trends shaping visibility. If you want to automate the data joins, my guide to Claude Code for SEO shows how.
If you would like help designing and running these tests, our SEO team does this for clients. Contact us for a quote and we will suggest a first experiment that fits your site.
Further reading on AI search incrementality
- ChatGPT vs Google: Search Volume vs the Traffic They Actually Send
- How Much Traffic Do AI Overviews Take? Estimating Click Loss by Keyword
- Too Much AI Visibility Data? A Framework for Deciding What to Act On
Sources for this AI search incrementality guide
- Ahrefs – How to Measure AI Search Visibility When Attribution Falls Short
- Ahrefs – You Can’t Track AI Like Traditional Search
- Ahrefs – AI Overviews reduce clicks (2026 update)
- Ahrefs – How ChatGPT Does Query Fan-Out
- Ahrefs – 5 AI Search Trends I’m Seeing in 2026
- Search Engine Land – AI recommendation lists repeat less than 1% of the time
- Google Search Central – AI features and your website
- Google Analytics Help – Default channel group
- Google for Developers – Search Analytics: query (Search Console API)
