Evaluation criteria across LLM, RAG, and agent systems
After three of these experiments, I had a real problem: too many metrics and no clear sense of which ones actually mattered where. Each write-up tracked its own list, quality metrics here, retrieval metrics there, agentic metrics somewhere else, and I noticed I was reaching for whichever one felt right rather than being able to say why. So instead of running a fourth experiment straight away, I went back through the first three and built a framework: across a plain LLM call, a RAG pipeline, a single agent, and a multi-agent system, which evaluation criteria actually decide something, and which ones just describe? Then I tested that framework directly, holding one task completely fixed and varying only the shape, to see whether its central claim actually held up against real numbers instead of an inference drawn across experiments.
Read the framework and its empirical test, or try both live dashboards linked at the end.
Four shapes, one spectrum
A plain call to a model has exactly one decision point: what to output. Add retrieval and there are two: what to retrieve, and what to generate from it. Add a loop that decides whether to search again, and there are as many decision points as steps taken. Add a second role checking the first role’s work, and there are that many again, times however many roles hand off to each other. That’s the whole spectrum the framework is built around, plain LLM, RAG, single agent, multi-agent, and the core claim is simple: the more decision points a system has, the less a single quality number tells you about what actually happened inside it.
That’s not just an abstract worry. In my agentic architectures experiment, four of five designs tied at 47.5% exact match. Read only that number and the honest conclusion is “architecture doesn’t matter here.” It’s wrong, and the only reason I know it’s wrong is that a second kind of metric, one that measures how the system stopped, not just whether it was right, was tracked at all.
The criteria catalog: what to track, and where it even applies
Not every metric belongs to every shape, and treating them as if they did is exactly how a dashboard ends up rewarding a system for succeeding at something it was never even trying to do.
Task and quality metrics (accuracy, Exact Match, F1) apply everywhere. It’s the one metric every shape needs, and for a plain LLM call it’s often the whole story, since a single call has no other layer to check.
Retrieval metrics (Recall@k, Precision@k, MRR) only mean something once a system retrieves anything at all. They answer whether the system had a chance to be right, independent of whether it used that chance well, a genuinely different question from whether the final answer was correct.
Operational metrics (tokens, latency, time to first token, context payload, error rate) apply everywhere too, since every shape makes at least one model call. They almost always explain a cost rather than decide a winner, the one exception being when a cost buys literally nothing.
Agentic metrics are where shape actually matters most. Step and tool-call count only mean something once there’s a tool to call. Handoff count is 0, correctly, not badly, for any shape with only one role. Early termination rate only exists for a shape with a loop to hit a cap on; a system that always completes in one pass doesn’t get a score here, it gets “not applicable,” a different thing entirely from scoring 0.
Six questions, not one scorecard
Rather than one universal metrics dashboard, the framework is six questions, asked roughly in the order you’d actually ask them, each stating up front which shapes it even applies to.
1. Is the output good? Applies everywhere. In my intent classification experiment, one technique beat every other by 23 points of accuracy (90.5% vs. the next best 67.1%), a gap wide enough that no other metric was needed to call that winner.
2. If quality is tied, why? Applies to RAG, single agent, and multi-agent, not plain LLM, which has nowhere else to look. In my RAG experiment, Hybrid RAG had the best retrieval scores of any architecture there and still lost on final answer quality to Query Decomposition RAG, which had worse retrieval scores than almost everything else in the comparison. Good retrieval and a good final answer turned out to be two different claims.
3. Is the system’s autonomy trustworthy, or is an external cap doing the work? Only applies to a system with a loop. In my agentic architectures experiment, one design’s coordinator never once decided on its own that it had gathered enough information, in all 40 runs. A hard cap of 3 rounds did all the stopping. Its accuracy looked unremarkable next to the others, tied at 47.5%, but that number was never really testing its judgment, because its judgment never got the chance to fire.
4. Where does the cost come from, and is it buying anything? Applies everywhere, but only ever decides something when a cost buys literally nothing. In intent classification, sampling three chain-of-thought answers and voting cost three times the tokens of a single pass and scored lower, a clean case of cost with no return.
5. Is the system reliable, or just accurate on average? Error rates matter as a floor to rule things out, not usually as a decider. Across every experiment I’ve run so far, error rate, tool error rate, and empty retrieval rate sit at or near their best possible value, every time.
6. Is coordination between roles adding value, or adding failure surface? Only applies once there’s more than one role. In the agentic experiment, the worst-performing design and the best-performing one averaged the identical number of handoffs between roles. The number alone said nothing; what the coordination was actually for separated a productive handoff from an expensive one.
Testing it for real
Every experiment behind that framework had varied technique within one shape, which RAG design, which agent control flow, never the shape itself. So I built one more experiment that holds the task and the dataset completely fixed and varies only the shape:
- Plain LLM: one prompt, one answer, from training knowledge alone.
- RAG: one search, one generation call, no second try.
- Single-Agent ReAct: loops, deciding turn by turn whether to search again or finish.
- Multi-Agent: a Supervisor routing between a Retriever, a Reasoner, and a Verifier, with a rejected draft sending it back to refine the query.
flowchart LR
A["HotpotQA, 40 sampled questions"] --> B{"4 shapes"}
B --> C["llama3.1:8b, same model for all"]
C --> D["Same search() tool where a shape uses one at all"]
D --> E["An answer, plus how it got there"]
Same task as my agentic architectures experiment, HotpotQA, 40 questions, each needing facts from two different paragraphs out of 10 candidates, and one new metric added specifically for this run: redundant step rate, how often a shape re-retrieves a paragraph it already had in the same run. It’s a direct, repo-native test of the single most common failure mode documented in MAST, a real study of why multi-agent systems fail: agents redoing work they already did.
| Shape | Exact match | Mean tool calls | Mean wall-clock |
|---|---|---|---|
| Plain LLM | 35.0% | 0.0 | 0.4s |
| RAG (single search) | 60.0% | 1.0 | 1.1s |
| Single-Agent ReAct | 47.5% | 3.0 | 11.5s |
| Multi-Agent | 60.0% | 2.0 | 9.5s |
RAG tied the multi-agent system on accuracy while making one tool call, zero handoffs, and finishing in about a ninth of the time. Three things explain why.
A plain retrieval of the question beat the agent’s own rephrasing of it. RAG’s single search, using the raw question verbatim, found the right paragraph 92.5% of the time. Single-Agent ReAct’s own first search, phrased by the model from its own reasoning, found it only 82.5% of the time, a worse start despite getting to try again afterward. On this corpus, the raw question was simply a better search query than the model’s rewording of it, which undercuts the reasonable-sounding assumption that a system that can loop and search more should retrieve at least as well as one that can’t. This is question 2 from the framework, playing out again in a fourth experiment.
Early termination rate lied by omission, in a way that took real digging to catch. Single-Agent ReAct’s rate of hitting its turn cap without an answer (35%) is numerically better than the multi-agent system’s (47.5%), which reads as “more reliable,” exactly question 3’s warning. But every single one of the single agent’s capped-out runs submitted nothing at all, an automatic miss. Every one of the multi-agent system’s capped-out runs still submitted its last draft anyway, right 52.6% of the time, well above chance. The system with the worse-looking rate was the one that failed more gracefully, and that’s most of why it won overall.
The new redundant-step metric split this experiment’s multi-agent runs almost perfectly by outcome. Every run that exhausted its full round budget, 19 of 40, had at least one redundant retrieval, its query refinement had genuinely run out of new things to try. Every run that settled within budget had zero. That’s about as clean a mechanistic confirmation of a failure mode as I’ve gotten out of any of these experiments: when the loop is actually finding new evidence, it finishes; when it isn’t, it starts repeating itself, every time.
One more thing worth naming honestly: RAG matched the looping systems even on the two-hop “bridge” questions I expected it to fail outright. That’s not a new discovery about my architectures, it’s a confirmation of research already published on this exact dataset showing a real share of HotpotQA’s “multi-hop” questions don’t actually require both hops to answer. A fairer test of that specific claim would need a dataset where that shortcut doesn’t exist, which is next on my list.
What surprised me putting this together
The multi-agent failure taxonomy I leaned on has its own real limits. MAST, the paper behind question 6, is a post-hoc taxonomy: failures get labeled by a human or an LLM judge reading a finished trace, not by anything live. Its own automated classifier agrees with human labels on whether a run failed at all about 94% of the time, but only about 72% on which of its 14 specific failure modes applies. Worth knowing before treating it as ground truth rather than a starting point, and part of why I built the redundant-step-rate metric above, a signal computable directly from a run’s own trace instead of an after-the-fact judge call.
Not every metric was even trackable, and that gap was informative on its own. Abstention rate, whether a system correctly says “I don’t know,” only ever got tracked in my RAG experiment, because that was the only one where I’d actually given the model the option to decline. That’s a gap in my prompts, not a property of the other systems, and it only became visible by trying to apply one framework across all of them at once.
Reliability metrics still haven’t moved, four experiments in. Error rate, tool error rate, empty retrieval rate: every single one sits at or near its best possible value, every time, this newest experiment included. That’s genuinely useful (it means every other finding above is a real behavioral difference, not one system just breaking more often), but it also means I still haven’t tested whether these metrics would actually catch anything if something did go wrong. A noisier, harder task that can actually break something is next on my list, right after the HotpotQA shortcut fix.
Try it yourself
The framework’s dashboard has the complete criteria catalog with a definition for every metric, all six questions worked through against real numbers, and the open gaps it still doesn’t resolve. The experiment’s dashboard has every metric broken out per shape, scoped to only the other shapes it’s actually comparable against, so a shape’s “not applicable” never gets mistaken for a bad score.