# AI Overviews and fault codes: when the right answer has no source

Two studies checked whether Google AI Overviews are backed by the pages they cite. Most answers were right, but many weren't supported, usually because the source never said it. Six checks before you act on one for a fault code.

**Short answer:** AI Overviews usually get simple facts right, but the cited page often doesn't back the answer. A New York Times test with Oumi found 56% of correct answers "ungrounded". An IMC '26 study found 11.0% of claims unsupported by the cited pages, mostly facts no source mentions. For a fault code, open the cited page and check it's your manual.

## What does this look like on site?

A compressor stops. The drive is a Yaskawa A1000 and the keypad shows oC. The technician pulls out a phone and types "what does oC mean on a Yaskawa drive".

That's a question, and question-form searches are the ones most likely to get an AI Overview. In a 40-day study of 55,393 Google searches, 64.7% of question-form queries showed one, against 9.5% of other queries (Xu et al., IMC '26, p. 7).

If the summary says overcurrent, that part is right: the A1000 manual lists oC as overcurrent, with a long list of causes (see our [Yaskawa fault codes post](https://hangersearch.com/blog/yaskawa-vfd-fault-codes)). The harder question is where that answer came from. Drive makers code the same fault differently. On an INVT GD20, for example, overcurrent is split into OC1, OC2 and OC3 by when it tripped (see our [INVT GD20 post](https://hangersearch.com/blog/invt-vfd-fault-codes)). A summary built from three websites can be right about the code and still list causes or checks that aren't in your manual.

Two studies this year measured how often that happens. Neither looked at fault codes, but both measured the thing that matters here: whether the cited page actually says what the summary says.

![Open electrical cabinet with a variable frequency drive, contactors and terminal blocks, and a ring-bound manual open on the door shelf with one passage highlighted under a magnifier](/blog/media/ai-overviews-fault-code-accuracy-mw07r4i9.png)

## What did the New York Times and Oumi test find?

At the Times' request, Oumi, an AI company, ran Google searches built from SimpleQA, a benchmark of a bit over 4,000 short factual questions with known answers that OpenAI released in 2024. Search Engine Land reports the Times' figures for 4,326 searches:

- AI Overviews were correct 85% of the time in October (Gemini 2) and 91% in February (Gemini 3).
- Grounding got worse at the same time. In October, 37% of correct answers were "ungrounded", meaning the linked sources didn't fully support the answer. In February, it was 56%.

Oumi's own write-up puts it another way: only 39% of the Gemini 3 overviews were both correct and fully supported by their cited sources, and only 67% of individual claims were supported.

One of the Times' examples shows the pattern. Asked when Bob Marley's former home became a museum, AI Overviews cited three pages. Two didn't mention the date. The third, Wikipedia, gave two conflicting years, and the overview picked the wrong one (Ars Technica).

Treat these numbers with care. Oumi used AI models as judges (GPT-5 for correctness and its own HallOumi++ model for unsupported claims), checked by hand on small samples. It also sells tools for building models like HallOumi. Google said the study "has serious holes" and "doesn't reflect what people are actually searching on Google".

## What did the IMC '26 study find?

Haofei Xu, Umar Iqbal and Jacob M. Montgomery at Washington University in St. Louis ran a larger study, presented at the ACM Internet Measurement Conference in October 2026. Their crawler sent 55,393 trending Google searches across 19 topic categories from March 13 to April 21, 2026, from a single US location (pp. 1, 4).

They split every AI Overview into single factual claims, 98,020 in total, and checked each against the full text of the pages it cited (pp. 5, 10):

| Label | What it means | Share of claims |
|---|---|---|
| Clear | A cited page says it directly | 84.6% |
| Vague | A cited page supports it in broader terms | 4.4% |
| Omitted | No cited page mentions it | 7.0% |
| Incorrect | A cited page contradicts it | 2.7% |
| Ambiguous | The cited pages disagree with each other | 1.4% |

That's 89.0% supported and 11.0% unsupported (Table 2, p. 6). The authors call 11.0% a conservative figure. Their pipeline couldn't read social and video pages, and if every claim in an overview that cited one of those pages was in fact supported, the rate would fall to about 5.3% (p. 10). Weather and other live values also counted as unsupported when the page had changed by the time it was read (p. 11).

Three findings matter for anyone looking up a fault code:

- **The usual failure is a fact no source mentions.** An unsupported claim was about 2.6 times more likely to be omitted than contradicted (p. 10).
- **A contradiction can't be seen from the summary.** On the Incorrect claims, the authors write that it is "a failure mode a reader who sees only the AIO summary has no way to detect" (p. 10).
- **Cited sources aren't just the top results.** Google describes AI Overviews as "built to surface information that is backed up by top web results" (quoted on p. 1). Yet 29.8% of the domains they cited didn't appear on the first page of results at all (p. 2).

The closest category to machine faults was Autos & Vehicles, and it was small: 138 queries, 19 with an AI Overview (Table 3, p. 7). There, 27.4% of cited pages were social or video platforms, one of the highest shares of any category (p. 9 and Table 8, p. 10), and 86.4% of 272 claims were supported (Figure 5, p. 11). That's too few to say anything firm about fault codes, but it isn't a reason to relax either.

## Why do the two studies give different numbers?

They measured different things. Oumi used trivia questions with known answers, counted only citations that pointed to an exact text passage, and checked claims against that passage. Xu et al. used real trending searches and checked claims against the full cited page. Oumi's 67% of claims supported and the IMC study's 89.0% aren't measuring the same thing, so don't average them.

| | NYT / Oumi | Xu et al., IMC '26 |
|---|---|---|
| Queries | 4,326 SimpleQA trivia questions | 55,393 trending searches, 19 categories |
| When | October 2025 and February 2026 | March 13 to April 21, 2026 |
| Checked against | The cited text passage | The full cited page |
| Main result | 91% correct; 56% of correct answers ungrounded | 11.0% of claims unsupported; 7.0% omitted |
| Judge | AI models, hand-checked on samples | AI model, hand-checked on samples |
| Fault codes tested | No | No |

Both point the same way: a right answer and a supported answer aren't the same thing, and the summary alone won't tell you which one you have.

## What should you check before acting on an AI Overview for a fault code?

1. **Search the code exactly as displayed, with the brand and series.** **Expect:** results for your drive. **If not:** Yaskawa's oC starts with a lowercase letter o, not a zero, and other makers code the same fault differently. See our post on [exact fault-code lookup](https://hangersearch.com/blog/fault-code-lookup-keyword-search-vs-chat).
2. **Open the cited link behind the claim you're about to act on.** **Expect:** the page jumps to a highlighted passage that says the same thing. Oumi found that most citation links point to an exact text passage on the page. **If not:** if nothing on the page says it, treat the claim as Omitted.
3. **Check what the source is.** **Expect:** the maker's manual for your series, with an edition and a page. **If not:** a forum post or a video is one person's report. It may be a real fix, but it isn't the manual.
4. **Check the causes and the order of checks against the manual, not just the code name.** **Expect:** each cause in the summary is in the manual's list for that code. **If not:** a cause that isn't in the manual may have come from a different drive.
5. **If the cited pages disagree, go to the manual.** That's the Ambiguous case, and the Bob Marley example shows the overview can pick the wrong side.
6. **Then check your own service history for the same code on the same machine.** The manual lists every possible cause. Your old reports say which one it was last time. Our posts on [finding a past service fix](https://hangersearch.com/blog/find-past-service-fix-repeat-machine-fault) and the [Siemens MM440 F0001 that won't reset](https://hangersearch.com/blog/siemens-mm440-f0001-fault-wont-reset) show why that matters.

Google says the same thing at the bottom of every overview: "AI can make mistakes, so double-check responses" (quoted by Ars Technica).

## What these studies don't tell you

Neither study tested fault codes, service manuals or industrial equipment. The IMC study used trending searches from one US location, and the authors say it "may not reflect the full distribution of Google Search traffic, particularly less time-sensitive or long-tail queries" (p. 12). A fault-code search is a long-tail query. Results may differ in India or elsewhere. Both studies used AI models to judge claims, checked by people on samples. Google disputes the Oumi study, and AI Overviews change as Google changes the model behind them.

What they do show is enough to change one habit: before you act on a summary, open the page it cites.

## Where does HangerSearch fit?

We built HangerSearch so every result is a source you can open. It's private, retrieval-only search over your team's own finalized service reports, searched where they already are. You search by fault code, model or symptom. It shows the prior reports that match and why each matched, and opens the cited page. Nothing is generated, and if there's no similar report, it says so. See [how we handle your reports](https://hangersearch.com/trust).

Book a 20-minute demo: hangersearch.com

## FAQ

### How accurate are Google AI Overviews?
In a New York Times test with Oumi on 4,326 trivia questions, AI Overviews were correct 91% of the time in February 2026, up from 85% in October 2025. Accuracy isn't the whole story: 56% of the correct answers were "ungrounded", meaning the linked sources didn't fully support them.

### What does "ungrounded" mean for an AI Overview?
The answer may be right, but the pages it links to don't fully support it. In the IMC '26 study, the most common unsupported claim was one no cited page mentioned at all (7.0% of claims), ahead of claims a cited page contradicted (2.7%).

### Can I trust an AI Overview for a fault code?
Use it as a pointer, not an answer. Neither study tested fault codes. Open the cited page, check that it's the maker's manual for your series, and check each cause and step against the manual's entry for that code before you act on it.

### Why does an AI Overview show up for some searches and not others?
In the IMC '26 study, searches phrased as questions got an AI Overview 64.7% of the time, against 9.5% for other searches. "How" questions triggered one 84.3% of the time. Google doesn't publish its rules for when an overview appears.

## Sources

1. Haofei Xu, Umar Iqbal, Jacob M. Montgomery, "Measuring Google AI Overviews: Activation, Source Quality, Claim Fidelity, and Publisher Impact", IMC '26, Karlsruhe, October 2026: https://doi.org/10.1145/3777912.3839818 (preprint arXiv:2605.14021v2: https://arxiv.org/abs/2605.14021). Read 3 Nov 2026. Supports: study setup and dates (p. 1), Google's "top web results" description (p. 1), 29.8% of cited domains not on the first page (p. 2), single US vantage point (p. 4), full-page claim checking and text-fragment links (p. 5), label definitions and Table 2 shares (p. 6), question-form activation and Table 3 (p. 7), Autos & Vehicles UGC share (p. 9; Table 8, p. 10), omission vs contradiction, the 5.3% bound and the "no way to detect" quote (p. 10), Figure 5 and live-data caveat (p. 11), long-tail limitation (p. 12). PDF page numbers.
2. Danny Goodwin, "Google AI Overviews: 90% accurate, yet millions of errors remain: Analysis", Search Engine Land, 7 Apr 2026: https://searchengineland.com/google-ai-overviews-accuracy-wrong-answers-analysis-473837. Read 3 Nov 2026. Supports: the Times/Oumi figures (4,326 searches, 85% and 91%, 37% and 56% ungrounded) and Google's response.
3. Ryan Whitwam, "Testing suggests Google's AI Overviews tell millions of lies per hour", Ars Technica, 7 Apr 2026: https://arstechnica.com/google/2026/04/analysis-finds-google-ai-overviews-is-wrong-10-percent-of-the-time/. Read 3 Nov 2026. Supports: SimpleQA description, the Bob Marley example, Google's quotes, the "double-check responses" notice.
4. Stefan Webb, "Oumi's Study Finds 50% of AI Overviews Untrustworthy", Oumi blog, 14 Apr 2026: https://oumi.ai/blog/oumis-study-finds-50-of-ai-overviews. Vendor write-up of its own study. Read 3 Nov 2026. Supports: 39% correct and fully supported, 67% of claims supported, judge models, text-fragment citations, hand checks on samples.
5. Yaskawa A1000 and INVT GD20 fault-code meanings: as cited in our [Yaskawa](https://hangersearch.com/blog/yaskawa-vfd-fault-codes) and [INVT](https://hangersearch.com/blog/invt-vfd-fault-codes) posts (Yaskawa A1000 Technical Manual SIEP C710616 27C, p. 311; INVT GD20 manual V2.1, p. 112).