RAG architectures compared
Retrieval-augmented generation, RAG for short, is the idea of letting a language model look something up before it answers, instead of only relying on what it memorized during training. Almost everything written about RAG shows exactly one way of doing this, usually the simplest one, with nothing to compare it against. I wanted to know how the more elaborate versions actually stack up, and specifically where each one wins or loses, not just which one wins on average.
Try the live dashboard or read the full write-up and code.
The setup
I used MultiHop-RAG, a set of 609 real news articles paired with questions that are each labeled as one of four kinds:
- Inference questions need a fact that isn’t stated outright. You have to connect two things that are stated to work it out.
- Comparison questions ask which of two things is bigger, earlier, or more, so they need a separate lookup for each side.
- Temporal questions depend on getting dates and the order of events right.
- Null questions are trick questions. They sound like they’re about a real article, but the specific fact being asked for was never actually reported anywhere in the corpus. The right move is to say so, not guess.
That last category matters a lot for this experiment. A system that always answers, even when it shouldn’t, is hallucinating, and I wanted a clean way to catch that.
Every architecture ran on the same local, free model (llama3.1:8b through Ollama), against the same 609 articles, on the same 80 sampled questions, split evenly across the four types.
flowchart LR
A["609 news articles"] --> B["Cut into searchable chunks"]
C["80 sampled questions"] --> D{"6 architectures"}
B --> D
D --> E["llama3.1:8b, same model for all"]
E --> F["An answer, plus how it got there"]
Six ways to search and answer
- Naive RAG: embed the question, grab the 5 most similar chunks, answer from them. The plain baseline.
- Hybrid RAG: search twice, once by meaning (embeddings) and once by exact keyword match, then merge the two rankings.
- Reranked RAG: cast a wide net (20 chunks) by meaning, then have a second, slower model read the question against each one and narrow it down to the best 5.
- HyDE: before searching, ask the model to write a made-up passage that would answer the question, and search using that instead of the question itself.
- Query Decomposition RAG: split the question into 2 or 3 smaller questions, search for each one separately, then answer from everything found.
- Corrective RAG: after searching, have the model grade whether what it found is actually relevant before trusting it. If nothing relevant turns up, search again a different way, or admit defeat instead of guessing.
How I measured whether any of this actually worked
Three separate questions need three separate kinds of measurement, and mixing them up hides what’s actually going on.
Did it find the right evidence? Three things capture this. Recall asks, of everything it pulled up, did the one piece of evidence that actually proves the answer make it into the pile at all. Precision asks the opposite question: of everything it pulled up, how much of it was actually useful, versus just noise riding along. And because a useful chunk buried at the bottom of a long pile is worth less than one sitting right at the top, I also tracked how early the useful evidence tended to show up, not just whether it showed up.
Did it give a good answer? I checked whether the predicted answer matched the correct one, both strictly (an exact match) and loosely (partial credit for getting most of it right). I also checked faithfulness: whether the words in the answer actually trace back to something it found, rather than sounding right while citing evidence that doesn’t really back it up. And for the null (trick) questions specifically, I tracked two different abstention rates: how often it correctly admitted it didn’t know, and, just as important, how often it wrongly said “I don’t know” on a question it actually could have answered.
What did it cost? How many separate trips to the model one question took, how many separate searches, how much text went in and came out of each trip, how long a trip took end to end, how long before the first word of the answer even appeared, and how often a search came back with nothing useful at all or a call to the model just failed outright.
What actually happened
| Architecture | Got the answer right | Found the right evidence | Admitted “I don’t know” correctly | Wrongly gave up on an answerable question |
|---|---|---|---|---|
| Naive RAG | 57.5% | 53.3% | 100% | 28.3% |
| Hybrid RAG | 61.9% | 65.0% | 100% | 23.3% |
| Reranked RAG | 52.2% | 58.3% | 100% | 31.7% |
| HyDE | 52.8% | 45.0% | 90% | 31.7% |
| Query Decomposition RAG | 67.5% | 51.7% | 95% | 20.0% |
| Corrective RAG | 49.7% | 45.0% | 100% | 40.0% |
Three things surprised me enough to actually dig into why.
The architecture that gave the best answers wasn’t the one that found the best evidence. Hybrid RAG found the right evidence more reliably than anything else, by a clear margin. But Query Decomposition RAG, whose evidence-finding was only middling, ended up with the best final answers overall. The reason turned out to be the news corpus itself: it’s stuffed with duplicate coverage of the same events, so a name central to a big story (a company collapsing, a court case) tends to show up in a dozen related articles, not just the one that officially “counts” as the answer. Query Decomposition searches per sub-question and ends up pulling in more chunks overall, so even when it misses the one designated correct piece of evidence, it often stumbles onto the right name somewhere else in that wider pile anyway. Good retrieval and a good final answer turned out to be two different questions, and it took measuring both separately to see that they’d split apart like this.
Corrective RAG, the one design built specifically to avoid guessing, guessed wrong more than anything else. Its whole idea is to grade the evidence before trusting it, so it doesn’t confidently make something up. On the trick questions, that bought it nothing: three simpler architectures hit the exact same 100% correct “I don’t know” rate with no grading step at all, just a plain instruction not to guess. What the grading step did add was the worst rate of wrongly giving up on real, answerable questions (40%, double the plainest baseline), plus roughly double the cost per question. The grading step is itself just another model call, and an imperfect one, so a good piece of evidence sometimes gets rejected before the answer step ever sees it. That’s a known weak spot of this kind of design, not something new I stumbled onto, but it was still striking to watch it show up this clearly in the numbers.
HyDE’s exact failure mode showed up on one question in a way worth describing. Asked which country was responsible for a conflict the corpus never actually covers in that specific way, HyDE’s made-up passage confidently named a real country, drawing on the model’s general knowledge rather than anything in the articles. That fake passage’s meaning then pulled in real articles that were genuinely about the right general topic, and the final answer step, seeing evidence that looked plausible, went along with it instead of admitting there was no real support for that specific claim. That’s the exact risk this technique accepts on purpose: a made-up passage doesn’t need to be true to be useful for search, but when the model’s guess is specific and confident enough, it can drag in evidence that ends up looking like confirmation.
I also ran a smaller side experiment on chunking, meaning how a long article gets cut into smaller pieces before any of this searching happens. Bigger pieces found the right evidence more often, simply because there’s more text per piece to contain the fact somewhere, but that didn’t translate into the best final answers either. Pieces that were cut along whole sentences instead of a rigid word count did about as well as the biggest pieces, using far less text per piece, which is the more useful takeaway for anyone actually building one of these systems day to day.
The full write-up has every architecture’s exact prompts, a working demo you can click through, and the honest caveats (a sample of 80 questions is enough to see clear directional differences, not enough to trust an exact percentage point gap; this only tells us as much about a design as this specific news corpus lets it show).
Read the details, or explore the dashboard yourself.