Skip to content

RAG vs agentic RAG: when retrieval needs an agent

  • How traditional (single-pass) retrieval-augmented generation works, and the specific ways it fails.
  • What changes when retrieval becomes an action a model can plan, check and repeat.
  • The main agentic RAG patterns: router, multi-agent, hierarchical, corrective, self-reflective, adaptive and graph-based.
  • What published research and production reports actually show, and what they do not.
  • How to decide, per query type, how much agentic machinery to use.

Retrieval-augmented generation (RAG) gives a language model relevant passages from your own data at question time, so answers can draw on information that is not in the model’s weights (Lewis et al., 2020). Agentic RAG is not a competing product. It is the same pipeline placed inside an agent loop, where the model decides whether to retrieve, what to search for, whether the results are good enough, and what to do next if they are not.

Traditional RAG runs one fixed sequence for every question: embed, search, stuff the top results into a prompt, generate. It is cheap, fast and predictable, and it is the right first design for almost every project. It fails on questions that need several lookups, on vague questions, and whenever retrieval returns the wrong passages, because nothing checks.

The evidence points to an ordering of fixes. Improve retrieval quality first (hybrid keyword and vector search, better chunking, reranking), then add agentic behavior selectively, for the query types where one pass demonstrably fails.

  • Single-hop questions dominate many workloads. “What is the return window?” needs one passage. Paying for planning and multiple retrieval rounds on every such question wastes money and time.
  • Some questions are multi-hop. “How long do returns take for an order shipped from Denver?” requires finding the warehouse’s carrier first, then the carrier’s processing time. A single search on the whole question often finds neither reliably.
  • Retrieval can fail silently. If the top results are irrelevant, a single-pass pipeline still hands them to the model, which may produce a fluent, wrong answer.
  • Data is heterogeneous. Policies in documents, orders in a database, live tracking behind an API. One vector index cannot serve them all.
  • Latency and cost budgets are real. Every extra model call adds both, and agent loops make them variable.
question ─► embed ─► vector search (top k) ─► prompt with passages ─► model ─► answer

One embedding call, one search, one generation call, the same steps for every question. Its known failure modes:

  • Multi-hop questions: the embedding of a combined question does not closely match either half’s answer.
  • Ambiguous questions: similarity search returns “close enough” passages that are subtly wrong.
  • No relevance check: irrelevant passages still reach the model.
  • Fixed k: a one-sentence answer and a ten-document synthesis get the same number of passages.

Much of this can be improved without an agent. Anthropic’s contextual retrieval experiments, for example, reduced the top-20 retrieval failure rate by 49% by combining contextual embeddings with contextual BM25 keyword search, and by 67% when a reranking step was added, on its test datasets.

question ─► plan: retrieve? from where? which sub-questions?
│
▼
retrieve ─► check: relevant and sufficient? ── no ─► reformulate, other source, or stop
│ yes
▼
generate ─► check: supported by the passages? ── no ─► retrieve more or correct
│ yes
▼
answer with citations

Retrieval becomes a tool the model can call zero or more times. The loop is bounded by a step and cost budget, and ends in an answer with sources or an explicit “cannot answer”. This is the general agentic loop applied to retrieval.

The survey by Singh et al. (2025) describes agentic RAG as applying reflection, planning, tool use and multi-agent collaboration to retrieval. The common patterns:

  1. Router RAG. One agent chooses the source per query: vector store, SQL database, web search or none. The simplest agentic pattern, and often the most valuable.
  2. Multi-agent RAG. Specialist agents per domain (orders, policies, product data) under a coordinator. Useful when sources cannot share one index.
  3. Hierarchical RAG. A top-level agent decomposes the question into sub-questions, delegates each, and synthesizes the results. This is the direct answer to multi-hop questions.
  4. Corrective RAG (CRAG). A lightweight evaluator rates retrieved documents and triggers corrective actions, such as web search or query rewriting, when retrieval looks wrong (Yan et al., 2024). The authors report significant gains over standard RAG on four datasets covering short- and long-form generation, and show it can be layered onto other RAG methods.
  5. Self-reflective RAG (Self-RAG). The model is trained to emit reflection tokens that decide when to retrieve and critique whether passages are relevant and whether its own output is supported (Asai et al., 2023). It reported large gains over same-size baselines on open-domain QA, fact verification and reasoning benchmarks, but it requires a specially trained model.
  6. Adaptive RAG. Classify query complexity first, then send each query to the cheapest pipeline that can answer it: no retrieval, single pass, or the full multi-step path. This is the pattern most directly aimed at cost.
  7. Graph RAG. Retrieve over a knowledge graph of entities and relationships, often with an agent that traverses it. It helps most when the hops in a question match real relationships in the data (warehouse, carrier, processing time).

The sample is plain Python 3.9 or later (tested with Python 3.11; no packages needed). Keyword-overlap retrieval and a scripted plan stand in for embeddings and a model, so the control flow runs deterministically without an API key. Save it as rag_compare.py and run python3 rag_compare.py.

Worked example: a multi-hop help-center question

Section titled “Worked example: a multi-hop help-center question”

An invented shop, OrdersPortal, answers customer questions from a small help-center corpus. A customer asks how long return processing takes for an order shipped from the Denver warehouse. The answer needs two facts from two documents.

# rag_compare.py - single-pass RAG versus a small agentic loop on a multi-hop question.
# Keyword-overlap retrieval and a scripted "planner" stand in for embeddings and a model,
# so the control flow runs deterministically without any API.
import re
DOCS = {
"wh-denver": "Orders shipped from the Denver warehouse use NorthLine for returns.",
"wh-austin": "Orders shipped from the Austin warehouse use SwiftPost for returns.",
"ret-policy": "Returns are accepted within 30 days of delivery for unused items.",
"northline": "NorthLine return parcels are processed within 5 business days of pickup.",
"swiftpost": "SwiftPost return parcels are processed within 2 business days of pickup.",
"refund-time": "Refunds are issued to the original payment method after the return is processed.",
}
STOP = {"the", "a", "for", "of", "from", "is", "are", "how", "long", "does", "do", "it", "take",
"which", "what", "and", "use", "to", "within", "after", "my", "an", "i", "was"}
def words(text: str) -> set[str]:
return {w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}
def retrieve(query: str, k: int = 2) -> list[tuple[str, int]]:
scored = sorted(((len(words(query) & words(t)), d) for d, t in DOCS.items()), reverse=True)
return [(d, s) for s, d in scored[:k]]
QUESTION = "My order was shipped from the Denver warehouse. How long does return processing take?"
# --- Single pass: retrieve once with the raw question, answer from whatever came back
hits = retrieve(QUESTION)
print("single pass retrieved:", [d for d, _ in hits])
print(" has the processing time?", any(d == "northline" for d, _ in hits))
# --- Agentic loop: plan sub-questions, check relevance, feed each answer into the next query
def agentic(max_steps: int = 4, min_overlap: int = 2):
plan = [("carrier", "Which carrier handles returns for the Denver warehouse?", r"\buse (\w+)"),
("days", "How long does {carrier} take to process return parcels?", r"within (\d+)")]
facts, sources = {}, [] # a model would write the plan and read the passages
for step, (key, template, pattern) in enumerate(plan, start=1):
if step > max_steps:
return None, sources # budget exhausted: escalate to a person
query = template.format(**facts)
(doc, score), = retrieve(query, k=1)
print(f" step {step}: '{query}' -> {doc} (overlap {score})")
if score < min_overlap: # relevance check failed: do not guess
return None, sources
facts[key] = re.search(pattern, DOCS[doc]).group(1)
sources.append(doc)
return facts, sources
print("agentic:")
facts, sources = agentic()
if facts:
print(f"answer: {facts['carrier']} processes returns within {facts['days']} business days "
f"(sources: {', '.join(sources)})")
else:
print("no confident answer: escalate")

Output:

single pass retrieved: ['wh-denver', 'wh-austin']
has the processing time? False
agentic:
step 1: 'Which carrier handles returns for the Denver warehouse?' -> wh-denver (overlap 3)
step 2: 'How long does NorthLine take to process return parcels?' -> northline (overlap 3)
answer: NorthLine processes returns within 5 business days (sources: wh-denver, northline)

What the example shows:

  • Single pass found the first hop only. The question mentions the warehouse, so the warehouse documents ranked highest; the carrier’s processing time never reached the model. A model given these passages either says it does not know or guesses.
  • The second query depends on the first answer. “NorthLine” appears nowhere in the customer’s question. Decomposition plus feeding the first result into the next query is what makes multi-hop work.
  • Relevance checks and budgets make it safe to stop. If either hop retrieves nothing relevant, the loop returns “escalate” instead of answering from weak evidence.
  • The cost is visible. Two retrievals here, plus in production a planning call and possibly a checking call. Multiply by query volume before choosing this path for every question.
Dimension Traditional RAG Agentic RAG
Model calls per question One generation call Several: planning, checking, generation, sometimes self-critique
Latency Lowest; one round trip to the model Higher and variable; grows with each step
Cost per question Lowest and predictable Multiplies with steps; needs hard caps
Single-hop accuracy Good when retrieval is good Similar; routing can skip needless retrieval
Multi-hop and ambiguous questions Degrades The case the extra complexity pays for
Failure behavior Silent: bad retrieval becomes a bad answer Can detect, retry, switch source or decline
Sources Usually one index Vector, SQL, APIs, web, knowledge graph
Debugging One deterministic path Variable-length traces need tracing tools

The cost question is about volume and error tolerance, not which design is better. As an illustration, a pipeline that is more accurate but costs twenty times more per question can be the wrong choice for a high-volume FAQ and the right one for low-volume legal or financial research. Published papers mostly report accuracy without latency or cost, so measure both on your own stack.

  • LinkedIn combined RAG with a knowledge graph built from historical support tickets. After about six months in its customer service team, it reported a 28.6% reduction in median per-issue resolution time (Xu et al., 2024). The gain came from relationships between tickets that text similarity missed.
  • Large search and enterprise products describe hybrid retrieval (keyword plus vector), multi-stage ranking and knowledge graphs as their core, with multi-step reasoning applied to the queries that need it. Few, if any, run a full agent loop on every query.

The common thread: invest in retrieval quality first, then apply agentic behavior selectively.

  • Loops that do not stop. An agent keeps reformulating without improving. Cap steps, time and tokens per question, and return “cannot answer” at the cap.
  • Checks that rubber-stamp. A model judging relevance agrees with whatever came back. Calibrate checks on a labeled set and track how often they reject.
  • Error compounding across hops. A wrong first hop makes every later step confidently wrong. Require citations for each intermediate fact and show them with the answer.
  • Cost variance. Average cost looks fine while a tail of complex questions explodes the bill. Monitor percentiles and set per-question budgets.
  • Prompt injection through documents. Retrieved content can contain instructions, and an agent that can call tools may follow them. Treat passages as data and restrict the tools the loop can call.
  • Stale or unauthorized data. Retrieval must respect each user’s permissions, and indexes must be refreshed when sources change.
  • Evaluation gaps. Measure retrieval (context recall and precision) separately from generation (faithfulness, answer correctness), on single-hop and multi-hop sets. See RAG evaluation.
  • Most questions are single-hop and the corpus is clean. Improve chunking, hybrid search and reranking instead.
  • Latency budgets are tight, such as autocomplete or interactive chat at high volume.
  • You have not measured the baseline. Without a test set that shows where single-pass RAG fails, an agent adds cost without evidence of benefit.
  • The data is structured. A question about order totals needs a SQL query, possibly chosen by a router, not iterative document search.

1. What is the difference between RAG and agentic RAG? RAG retrieves once with the raw question and generates from whatever came back. Agentic RAG makes retrieval an action inside an agent loop: the model decides whether and where to retrieve, decomposes questions, checks relevance, and retries or stops within a budget.

2. Why does single-pass RAG struggle with multi-hop questions? The answer depends on facts found in different documents, and the second search often depends on the first answer, which is not in the question. One similarity search over the combined question rarely ranks both passages highly.

3. What would you try before making a RAG system agentic? Fix retrieval quality: chunking that keeps context, hybrid keyword and vector search, a reranker, metadata filters and permissions. Build an evaluation set separating single-hop and multi-hop questions to see where failures remain.

4. Senior scenario: OrdersPortal’s help assistant answers 2 million questions a month with single-pass RAG. Accuracy is good on simple questions but poor on questions that combine orders, warehouses and policies, which are about 8% of traffic and drive most escalations. Design the next version. Keep single pass as the default and add an adaptive router in front. A cheap classifier (a fast-tier model, or rules on entities detected) sends simple questions down the existing path and multi-hop ones to an agentic path. The agentic path decomposes the question, calls typed tools (an order lookup API with the customer’s permissions, a warehouse-to-carrier table, the policy index) rather than searching documents for structured facts, checks each hop’s evidence, and returns an answer with citations or escalates. Cap it at a fixed number of steps and a token budget per question, and alert on the 95th-percentile cost. Before launch, build evaluation sets for both paths from real escalated conversations, and measure answer correctness, citation accuracy, latency and cost per path; ship behind a flag to a share of multi-hop traffic and compare escalation rates. Since only 8% of questions take the expensive path, total cost rises modestly while the escalation driver is addressed.

  • Agentic RAG is RAG inside an agent loop: plan, retrieve, check, retry or stop.
  • Single-pass RAG is the right default; fix retrieval quality before adding agents.
  • Agentic patterns earn their cost on multi-hop, ambiguous and multi-source questions.
  • Route per query: most traffic should take the cheap path.
  • Bound every loop, cite every hop, and evaluate retrieval and generation separately.