Hallucinations in a nutshell

Hallucinations are instances where large language models (LLMs) generate outputs that are fluent and plausible but factually incorrect, unsupported, or fabricated. They arise from models’ tendency to generalize patterns in training data without grounding answers in verified facts, especially when asked about niche, time-sensitive, or ambiguous topics. Hallucinations undermine trust and safety in real-world applications (e.g., search, helpdesks, legal summaries). Mitigation strategies include prompt engineering, model calibration, retrieval-augmented generation (RAG), grounding on structured data, answer verification, uncertainty estimation, and human-in-the-loop review. RAG—with an explicit retrieval step that supplies the model with relevant source passages—stands out as a practical, scalable approach to reduce hallucinations by anchoring generation in external evidence.

Using Retrieval Augmented Generation (RAG)

RAG integrates a retrieval system with a generative LLM. Instead of relying only on learned parameters, the model conditions its output on retrieved documents or passages relevant to the user query. This reduces hallucination by:

• Providing explicit evidence the model can cite.
• Narrowing the model’s generation space to verifiable facts in the retrieved texts.
• Enabling post-generation checks (e.g., citation matching, fact verification) and fallback behaviors (e.g., “I don’t know”) when evidence is absent.

How RAG does work

A typical RAG pipeline (concise):

1. Query encoding: Convert user query to a dense vector (or keyword query).
2. Retrieval: Search a document store (vector index or traditional search) for top-k relevant passages.
3. Context assembly: Concatenate or format retrieved passages into a context block or provide them via specialized input channels (e.g., retrieval tokens, memory).
4. Generation: Feed the model the query plus retrieved context; the model generates an answer conditioned on those passages.
5. Attribution & verification: Extract which passages support each claim, surface citations, and optionally run a verifier (rule-based, secondary model, or human) to check consistency.
6. Response policies: If retrieved evidence is weak/conflicting, return a cautious reply (e.g., ask for clarification or state uncertainty).

Detailed RAG example (customer-support knowledge base) Scenario: A user asks, “How do I reset ProductX to factory settings?” The company maintains a knowledge base (KB) of product manuals and support articles.

Step-by-step flow

1. Preprocessing and index creation
   • Split KB documents into smaller passages (100–500 tokens) with metadata (doc id, section, date).
   • Create dense embeddings for each passage using an embedding model; store them in a vector index (e.g., FAISS, Pinecone).
   • Optionally also maintain an inverted-index or BM25 index for hybrid retrieval.
2. Query handling
   • User query: “How do I reset ProductX to factory settings?”
   • Encode query into an embedding.
3. Retrieval
   • Retrieve top-k passages by vector similarity (k = 5–10). Also retrieve top matches by BM25 if using hybrid.
   • Example retrieved passages: P1: „ProductX — Factory Reset: Press and hold the reset button for 10 seconds. LED will flash red once. Model A only.“ P2: „ProductX Model B: Factory reset via settings > system > reset > confirm; requires admin login.“ P3: „Reset precautions: Back up settings. Firmware v2.0+ adds cloud-linked reset that also deregisters device.“
4. Context assembly & prompt template
   • Construct a prompt that clearly instructs the LLM to use only the retrieved passages as evidence and to cite them: „You are an assistant that answers using only the evidence in the supplied passages. If the passages conflict or do not contain an exact procedure, say you are unsure and provide next steps. Passages: [P1], [P2], [P3]. User question: [How do I reset ProductX to factory settings?]“
5. Generation with grounding
   • Model generates:
     • Determine device model from user or ask clarifying question if unknown.
     • If user specified Model A: reply with procedure from P1, cite P1.
     • If model unspecified: present both procedures and note the difference, cite P1 and P2, and warn about firmware caveat from P3.
6. Attribution and confidence
   • Attach citations inline: “For ProductX Model A, press and hold the reset button for 10 seconds (Source: KB doc id X, passage P1).”
   • Provide a confidence score or label: e.g., “High confidence for Model A (direct match in KB); moderate for cross-model recommendation.”
7. Verification & fallback
   • Run a fact-checker: ensure the generated steps exactly match retrieved text; flag mismatches.
   • If passages conflict or none are authoritative, respond with: “I don’t have a definitive, sourced procedure—please confirm your model number or consult official manual [link].”

Key implementation details to reduce hallucination

• Retrieval quality: Use high-recall indexes, tune embedding model, and include metadata filtering (product model, firmware).
• Prompt constraints: Explicitly instruct the generator to cite sources and not invent procedures. Use answer templates that require quoting text spans when asserting facts.
• Source ranking: Re-rank retrieved passages by domain-specific heuristics (freshness, document authority).
• Verifier: Post-process outputs with a separate model that checks each claim against retrieved texts (claim verification).
• Uncertainty handling: If no supporting evidence, prefer “don’t know” or ask clarifying questions rather than guessing.
• Human review: For high-risk domains, require human sign-off for answers without strong, consistent evidence.

Short pseudocode (conceptual)

1. embed_q = Embed(query)
2. candidates = VectorSearch(embed_q, k=10)
3. ranked = ReRank(candidates, query)
4. context = AssembleTopK(ranked, limit_tokens)
5. prompt = BuildPrompt(context, query, instructions_to_cite)
6. answer = LLM.Generate(prompt)
7. claims = ExtractClaims(answer)
8. for claim in claims: verify(claim, context)
9. if any claim fails verification: flag/modify answer or ask user for clarification
10. return answer + citations + confidence

Benefits and limitations

• Benefits: Substantially reduces unsupported assertions, enables traceable citations, allows updates without retraining.
• Limitations: RAG is as good as the retrieval corpus; missing or outdated sources still cause errors. Poor retrieval can mislead the model. Additional engineering (indexing, reranking, verification) is required for robust production behavior.

Conclusion

RAG reduces hallucination by grounding generation in retrieved evidence and combining retrieval, targeted prompting, and verification to produce traceable, more reliable outputs.