Updated October 12, 2026. LangChain APIs and provider integrations change quickly. This guide explains a version-aware way to build and evaluate a retrieval-augmented generation (RAG) pipeline; it does not claim that one package command, chunk size or model is universally correct.
RAG connects a language model to documents it can retrieve at question time. A useful system has two paths: an indexing path that prepares and stores source material, and a question path that retrieves evidence and generates an answer from it. The hard part is not making these components run once. It is showing that the right evidence is retrieved, the answer stays within that evidence, and changes do not silently break the system.
First, separate the two workflows
| Indexing: prepare knowledge | Question answering: use knowledge |
|---|---|
| Collect documents and retain source, date, access rights and version metadata. | Normalize the question and apply the user’s access scope. |
| Extract text; preserve headings, tables and page references where they matter. | Retrieve candidate passages using the chosen search method. |
| Split material into retrievable units; attach stable document and chunk identifiers. | Optionally rerank, filter or expand candidates; record why a passage was selected. |
| Create embeddings and store chunks, vectors and metadata in a persistent index. | Give the model the question and selected evidence; require source-grounded claims. |
| Refresh changed or deleted content and remove stale versions from search. | Return answer with source links—or say that evidence is insufficient. |
LangChain’s current documentation still describes this core as loading and splitting documents, embedding and storing chunks, then retrieving relevant passages for generation. The exact tutorial now also demonstrates a more elaborate Deep Agents pattern: index the documentation, search it with a tool, write retrieved chunks to an agent file system, and delegate focused analysis. Treat that as one current example—not a requirement for every RAG application. See the current official RAG tutorial.
Build from the official docs, but make the project reproducible
Older tutorials can mix APIs from different LangChain generations. A package may move to a separate integration package, a high-level chain may be replaced by a lower-level composition, or a sample may use a model name that is not available to your account. A copied example that imports successfully is not proof that the complete pipeline works.
- Choose one current official tutorial for your language and use case. Open its linked API references and changelog before relying on a code sample copied from a third-party blog.
- Pick the smallest architecture that meets the need. A simple retrieve-then-answer chain may be enough. Add a tool-using agent, subagents or a graph only when a measured requirement justifies the extra routing and state.
- Record the environment. Pin Python/Node, framework packages, integration packages, model identifier, embedding model, vector-store version and index schema in a lockfile or equivalent. Record the official doc URL and the date you checked it.
- Run clean-room checks. Create a fresh environment from the lockfile; run imports, indexing, one search, one answer and one update/delete test. Save console output and exact versions. Do not describe import-only checks as an end-to-end test.
- Keep provider credentials out of code. Use the provider’s documented secret handling, separate development and production credentials, and avoid indexing material users are not authorized to retrieve.
For example, the live LangChain tutorial lists separate provider integrations and vector-store packages, and its own quick-start storage option is in-memory. The documentation explicitly says to use persistent storage and refresh it when the documentation changes for a production index. Those distinctions matter more than a universal “pip install everything” command. The official tutorial and package examples are the source of truth for today’s names; still lock and test the exact versions used by your application.
Do not copy chunk size or top-k as magic numbers
Chunk size and overlap affect what the retriever can find and how much irrelevant context reaches the model. A short chunk may separate a condition from its exception; a large chunk may mix several topics and dilute search. Likewise, retrieving too few passages can miss needed evidence, while retrieving too many can bury it in noise. Values such as “1,000 characters,” “200 overlap,” or “top-k 4” are starting settings in a particular example, not a standard that transfers to every corpus.
- Preserve natural structure first: headings, paragraphs, list items, table rows, code blocks and document boundaries.
- Keep source identifiers, page/section, publication or effective date, and permission metadata attached to every chunk.
- Test several reasonable chunking and retrieval settings on the same representative questions. Change one factor at a time.
- Include questions requiring a definition, a detail buried in one source, evidence from multiple sources, a date-sensitive fact, and an answer that is absent.
- Measure the trade-off between retrieving the needed evidence and adding distracting passages. Inspect failures, not just a mean score.
Evaluate retrieval and generation separately
A plausible answer can still be wrong because retrieval found the wrong passages; a strong retriever can still be followed by an answer that invents details. Measure both stages and the end result.
| Layer | Questions to ask | Useful evidence |
|---|---|---|
| Retrieval | Did the system find the source and passage that contain the answer? Were permissions and dates applied? | Recall@k or hit rate on questions with known source passages; rank of the expected passage; sampled retrieval traces. |
| Context quality | Are returned passages relevant, focused, current and mutually consistent? | Human review on a small representative set; relevance and coverage labels; duplication and stale-source checks. |
| Answer grounding | Can each important claim be supported by a cited passage? Does the answer contradict the source? | Claim-to-source review; groundedness/faithfulness measures; adversarial and “not in documents” cases. |
| Reader usefulness | Does the response answer the actual question clearly, with the right caveats and next step? | Task-specific human rubric and user feedback; latency and cost recorded separately from quality. |
Research explains why one automated score is not enough. The RAGAS paper separates the quality of retrieved context from how faithfully the model uses it and from answer quality; its scores are evaluation signals, not ground-truth certification. RAGTruth supplies human-annotated examples of unsupported or contradictory claims, demonstrating that RAG can still hallucinate even when documents are provided. A 2025 EMNLP Industry paper also notes that models may add unsupported details despite relevant context and studies human-annotated examples to evaluate faithfulness. Use automated metrics to find likely regressions, then inspect errors with people who understand the domain.
Sources: Es et al., “RAGAS: Automated Evaluation of Retrieval Augmented Generation,” EACL 2024; Niu et al., “RAGTruth,” ACL 2024; and Tamber et al., “Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards,” EMNLP 2025.
A small evaluation plan you can maintain
- Create a reviewed set of real questions and expected supporting sources. Include answerable, ambiguous, out-of-scope and unanswerable questions. Remove personal or confidential information that is not needed.
- For each case, save the query, retrieved document/chunk IDs and order, answer, citations, model and package versions, and latency.
- Run the same set when changing chunking, embeddings, retrieval settings, rerankers, prompts, model versions or source documents.
- Review the misses: retrieval failure, stale or missing source, permission filter, generation unsupported by context, bad citation, or unclear abstention. Assign the fix to the layer that failed.
- Keep a few previously failed cases as regression checks. Do not treat one successful demo or an LLM judge score as proof of production readiness.
Common failure patterns
- Imports work, but the tutorial is called “tested”: run indexing, search, answer, refresh and deletion from a clean lockfile before making an end-to-end claim.
- Good-looking answer, weak evidence: inspect retrieved passages and claim citations; have the system abstain when the corpus does not support a conclusion.
- Old or unauthorized material appears: version source records, refresh/delete stale chunks, and enforce document permissions during retrieval—not only in the UI.
- Latency worsens after adding an agent: record time spent loading/indexing, retrieving, reranking, model generation and tool calls independently. Keep the simplest workflow that meets the measured quality bar.
- Scores rise but people still find errors: check whether the test set reflects actual user questions, whether references are correct, and whether the metric rewards style rather than evidence.
Practical takeaway
Use current official documentation to assemble the pipeline, but treat the whole application as a versioned system: source corpus, splitting, embeddings, index, retrieval, prompt, model and evaluation set. Start with a simple retrieve-then-answer flow, test it on representative questions, and add agent behavior only when the evidence shows a need. A RAG system is ready to rely on only to the extent that its sources are current, its retrieval is appropriate, and its answers have been checked against real tasks.
Sources and limits
- LangChain, current RAG with Deep Agents tutorial and linked integrations (accessed October 12, 2026).
- Es et al. (2024), “RAGAS: Automated Evaluation of Retrieval Augmented Generation,” EACL System Demonstrations. DOI: 10.18653/v1/2024.eacl-demo.16.
- Niu et al. (2024), “RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models,” ACL. DOI: 10.18653/v1/2024.acl-long.585.
- Tamber et al. (2025), “Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards,” EMNLP Industry Track. DOI: 10.18653/v1/2025.emnlp-industry.54.
This revision does not reproduce the previous four-site benchmark: the original selection method and execution environment could not be independently reconstructed from retained evidence. It removes its unsupported “we tested” and exact benchmark claims. No LangChain pipeline was executed as part of this article revision.