Exact fault-code lookup: why keyword search still beats chat for service manuals

Short answer: If you have the exact fault code, search for the exact code. In a 2025 study, plain retrieval found the right code 86% of the time, against 70% for an LLM alone and 57–64% for retrieval-augmented chat. Chat helped mainly when the symptom used different words from the manual. Either way, check the manual page.

Applies to: service and maintenance engineers looking up fault codes in OEM manuals and old service reports. Built on two published studies (PHM Society 2025, ICSE-SEIP 2026); neither tested HangerSearch. Last verified: 2 Nov 2026. Author: Muhammed Fahiz

Why is finding the fault code in the manual so slow?

Nitin Aji put the problem in four lines in a public LinkedIn post in September 2026:

"A 400-page service manual. A blinking fault code. Zero internet in the middle of the sea. For a marine service engineer, finding the right page can take longer than the repair itself." (Nitin Aji, LinkedIn post, 22 Sep 2026)

Anyone who has stood at a tripped drive with a PDF on a phone knows the shape of it. The code is on the display. The manual has a fault table somewhere, a troubleshooting chapter somewhere else, and a parameter list that the troubleshooting chapter refers to. The code might be written "E043" on the display and "E-43" in the table.

The obvious modern answer is to ask a chatbot. The question is whether that's faster and safer than searching for the code. Two recent studies tested close versions of that question.

What did the PHM 2025 study test?

Researchers at LMI (Sarah Lukens, Matthew Bishof, Nadir Siddiqui and Destiny West) presented a paper at the 2025 Annual Conference of the PHM Society on recommending fault codes from a technical manual. The setup, as the paper describes it (pp. 5–6):

• The manual: a public 1998 US Army operator's manual for light medium tactical vehicles, TM 9-2320-365-10. It lists 224 malfunctions in 22 systems, each with an operator troubleshooting procedure.
• The test data: 300 operator-style observations, such as "truck feels sluggish and struggles to accelerate", each generated by an LLM from a known fault code. The data is synthetic, which the authors list as a limitation.
• The systems compared: an LLM with no manual ("LLM-only"), three RAG variants that gave the LLM chunks of the manual (raw text, cleaned text, and a structured table indexed by fault code), and, in the second task, a retrieval-only baseline that returns fault codes from the manual with no LLM at all.

The question for each observation was simple: does the system's shortlist include the true fault code?

Did keyword retrieval or chat find the right fault code more often?

Retrieval did. In the task where answers had to be one of the manual's own fault codes, the results were (Table 5, p. 10):

System	Right code in the shortlist	Time per 100 observations
Retrieval only, no LLM	86.3%	117 s
LLM only, no manual	69.6%	301 s
RAG 1: raw manual text in chunks	64.3%	323 s
RAG 2: cleaned text in chunks	62.3%	334 s
RAG 3: structured, indexed by fault code	57.3%	334 s

Figures from Lukens et al. (2025), Table 5. Our layout.

The authors' summary is that "across all metrics, the Retrieval-only baseline outperformed every LLM-based approach" (p. 9). Retrieval was also the only system that never suggested a code that isn't in the manual (0.00% against 0.14–1.41% for the others), and it ran in well under half the time.

Two details are worth knowing before you lean on those numbers.

Adding the manual made the LLM worse in this task. Every RAG variant scored below the LLM on its own. The authors group cases by which systems got them right, and in 16.7% of observations retrieval and the LLM alone were both right while two or more RAG variants failed. Their label for that group is "RAG adds noise" (p. 9; Table 8, p. 11).

At the level of "which system is it?", everyone did better. When a suggestion only had to land in the right one of the 22 systems (electrical, engine and so on), hit rates ranged from 87.0% to 95.7%, with retrieval still highest (Table 6, p. 10). Narrowing a fault to the right subsystem is easier than naming the exact code.

When does chat beat keyword search?

When the words don't match. The paper's own examples make the split clear (p. 9):

• Retrieval won when the observation used the manual's phrasing. "Cab does not raise" found "Cab Will Not Raise" easily.
• The LLM won when the observation needed paraphrasing. "Temperature gauge rises into the red zone during operation" is the manual's "Engine Overheats". The LLM systems got it; retrieval returned codes about temperature indicators instead.

Across the whole set, the authors count about 10% of cases where the LLM was potentially preferable, against 24.7% where retrieval was preferable and 16.7% where RAG added noise (Table 8, p. 11).

In the first, chat-style task, where systems answered in free text, there was a second problem. The LLM on its own tended to answer with causes, such as "faulty or worn out air tank valves", rather than the manual's fault code, "large quantity of moisture expelled from air reservoirs". The structured RAG variant came closer to the manual's wording (Table 3, p. 7; p. 8). A cause list sounds helpful, but it doesn't take you to the troubleshooting page.

The authors' practical lesson: "in high-stakes applications that require efficient interaction with large volumes of technical content, retrieval-based approaches may be more suitable than LLM-based ones" (p. 10).

What happens when the question is vague?

A second study looked at the chat side in more depth. Maria Teresa Rossi and colleagues at the University of Milano-Bicocca tested RAG on troubleshooting procedures from Fincantieri, which, the paper says, involves over 100 manuals with hundreds of procedures in each (ICSE-SEIP 2026, p. 1).

When the people inspecting the best configuration's answers looked at questions phrased with the manual's terms and with context about the system, only 8% of answers were wrong. That rose to 36% when either the question was imprecise or the context was missing, and to more than half for imprecise questions with no context (p. 8). Their example of an imprecise question is "What should I do if the CIP pump leaks?", where "leaks" doesn't say whether it's the seals, the fittings or something else.

Two of their conclusions match what we see in the field:

• The sources used to build an answer "must always be returned together with the answer, so that operators can cross-validate recommendations" (p. 2).
• Training people on the documentation's own terms improves results (p. 9). In other words, the closer your words are to the manual's, the better any search works.

What should you do with a fault code on the display?

Search for the code first, and use the manual's words. A working order:

1. Write the code exactly as displayed, including letters, dashes and leading zeros. Some codes use letters that look like digits; Yaskawa's oC, for example, starts with a lowercase letter o, not a zero (see our Yaskawa fault codes post).
2. Search the manual for the code in each likely format. Aji's post lists the problem directly: "E043, E-043 and E-43 can all find the same fault-code entry" in his tool. A plain PDF search won't do that for you, so try each form.
3. Go to the fault table, then the troubleshooting section it points to. Note the manual's edition and page. Editions of the same manual can be paginated differently, so the page number in someone else's note may not match yours.
4. If you only have a symptom, translate it into the manual's words before searching. This is where a chat assistant can help: ask it what the manual would call the symptom, then search for that phrase. The Milano-Bicocca results suggest a precise question with context matters more than the tool.
5. Never act on an answer you can't trace to a page. Both studies say the same thing in different words: check the source.
6. Then search your own service reports for the same code on the same model. The manual lists every possible cause. Your old reports say which one it actually was last time. Our posts on finding a past service fix for a repeat fault and the alarm someone already solved three years ago cover where those reports usually live.

What these studies don't tell you

Both studies have limits, and the authors say so. The PHM study used synthetic observations, a single public manual that, the authors note, was likely in the LLM's training data, and gpt-3.5-turbo as the recommender (pp. 5, 9). Newer models might score differently. The Milano-Bicocca study used one company's procedures. Neither study looked at searching a team's own service reports, and neither tested HangerSearch.

What they do show is narrower and still useful: when the input is an exact code from a fixed list, exact retrieval was more accurate, faster and never made up a code. Generation earned its place on paraphrase, and only with the source shown.

Where does HangerSearch fit?

We built HangerSearch on the same split. It's private, retrieval-only search over your team's own finalized service reports, searched where they already are. You search by fault code, model or symptom. It shows the prior reports that match and why each matched, and opens the cited page. Nothing is generated, and if there's no similar report, it says so. See how we handle your reports.

Book a 20-minute demo: hangersearch.com

FAQ

Is keyword search better than ChatGPT for looking up fault codes?

For an exact code, in the one published comparison we found, yes. In Lukens et al. (PHM 2025), retrieval with no LLM put the right code in its shortlist 86.3% of the time, against 69.6% for an LLM alone and 57.3–64.3% for RAG setups. That study used synthetic observations and one manual, so treat it as evidence, not a law.

When is a chat assistant useful for troubleshooting?

When your description doesn't use the manual's words. In the same study, an LLM matched "temperature gauge rises into the red zone" to "Engine Overheats" where keyword retrieval missed. Use it to find the manual's term, then go to the page.

Why does RAG sometimes do worse than an LLM alone?

In the PHM 2025 study, the retrieved manual chunks sometimes pulled the answer toward the wrong fault. The authors call this "RAG adds noise" and saw it in 16.7% of cases. How the manual is split and cleaned also changed results.

How should I search a service manual by fault code?

Search the code exactly as displayed, then in its other likely formats (with and without dashes or leading zeros). Go from the fault table to the troubleshooting section it references, and note the manual's edition and page. Then search your own past service reports for the same code on the same model.

Sources

1. Nitin Aji, public LinkedIn post, "Offline Service Manual Assistant for Marine Engineers" (22 Sep 2026): https://www.linkedin.com/posts/nitin-aji-42922416b_rag-llm-genai-activity-7508008441571897344-lno6. Read 2 Nov 2026. Supports: both quotes from Aji.
2. Sarah Lukens, Matthew Bishof, Nadir Siddiqui, Destiny West, "An Evaluation Framework for Fault Diagnosis Using Technical Manuals in Retrieval-Augmented LLMs", Annual Conference of the PHM Society 2025: https://doi.org/10.36001/phmconf.2025.v17i1.4549. Read 2 Nov 2026. Supports: study setup (pp. 5–6), chat-task findings (Table 3, pp. 7–8), hit rates, hallucination rates and latency (Table 5, p. 10), system-level hit rates (Table 6, p. 10), examples and case groups (p. 9, Table 8 p. 11), limitations and lessons (pp. 9–10).
3. Maria Teresa Rossi, Leonardo Mariani, Oliviero Riganelli, Giuseppe Filomeno, Danilo Giannone, Paolo Gavazzo, "'Where is My Troubleshooting Procedure?': Studying the Potential of RAG in Assisting Failure Resolution of Large Cyber-Physical System", ICSE-SEIP 2026, arXiv:2601.08706. Read 2 Nov 2026. Supports: Fincantieri manual count (p. 1), returning sources (p. 2), wrong-answer rates and the "leaks" example (p. 8), terminology lesson (p. 9).