Challenges to Pew Research Validity on Google AI Search Results

Google pushes back hard on Pew Research’s claims about AI summaries hurting traffic. The issue started with a Pew study alleging decline in web traffic caused by Google’s AI-powered search summaries. Soon after, Google fired back, calling the methodology flawed and the sample size way too small.

Google’s spokesperson said users are increasingly using AI features and asking more questions. They denied any web traffic drop, stressing billions of clicks still flow to websites daily.

Duane Forrester, ex-Bing, flagged the Pew sample size as tiny—66,000 queries against 500 billion monthly Google searches is barely a rounding error.

Advertisement

Google shared:

“People are gravitating to AI-powered experiences, and AI features in Search enable people to ask even more questions, creating new opportunities for people to connect with websites.

This study uses a flawed methodology and skewed queryset that is not representative of Search traffic. We consistently direct billions of clicks to websites daily and have not observed significant drops in aggregate web traffic as is being suggested.”

Pew’s reliability scores for groups aged 18–65+ showed errors up to ±13.7 percentage points. That’s a huge margin making the stats unreliable, at best rough guesses.

Pew also compared searches from March to scraped queries done a month later in April. That’s a big problem because Google’s AI summaries evolve constantly. The same exact query can pull completely different AI overviews—varied by browser even.

Screenshots show how the query “What is the RLHF training in OpenAI?” pulls different top links on Vivaldi versus Chrome Canary. AI summaries and embedded links shifted dramatically, underlining the dynamic results’ nature.

The takeaway: AI search results are fluid, changing for users and over time. Pew’s month-apart snapshots don’t truly capture this.

Publishers and SEOs see this shift too. AI-driven search results shuffle the top spots constantly, with Google showing broader site variety. That might explain why traffic feels inconsistent.

The controversy boils down to Pew’s questionable sampling and ignoring Google AI’s dynamic updates—casting serious doubt on the reported impact to websites.

Add a Comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Advertisement