AI Experiments and Learnings
A running set of practical experiments across AI concepts and techniques, one dataset and approach at a time.
Overview
An ongoing side project to build real intuition for AI concepts and techniques, not just read about them. Each experiment picks one question (does technique X actually help, and by how much, at what cost?), runs it against real data, and measures the result instead of assuming it.
Experiments
1. Comparing prompting techniques for intent classification
Compared seven prompting techniques on a 77-class banking intent classification task, measuring both accuracy and operational cost. Semantic few-shot retrieval won by a wide margin (90.5% vs. 59 to 67% for the rest), splitting the decision into two steps made things worse instead of better, and self-consistency tripled the cost without improving accuracy. Read the detailed write-up.
2. PEFT, LoRA, and QLoRA: a theoretical comparison
Compared full fine-tuning against five parameter-efficient fine-tuning
techniques by computing real trainable-parameter counts and memory
footprints against llama3.1:8b’s architecture, no training run required.
QLoRA’s entire memory advantage over LoRA turned out to be weight
compression, not training fewer parameters, and BitFit trains zero
parameters on Llama because its architecture has no bias terms to select.
Read the detailed write-up.
3. Attention Is All You Need: a deep dive
A from-scratch walkthrough of the original transformer architecture, a worked attention example computed by hand on a tiny sentence, and how BERT, GPT, T5, and the efficiency and Mixture-of-Experts designs since 2017 (RoPE, grouped-query attention, FlashAttention, DeepSeek’s multi-head latent attention and DeepSeekMoE) changed the original design, and why. Not a technique comparison like the other experiments; a conceptual deep dive aimed at someone seeing the architecture for the first time. Read the detailed write-up.
4. RAG architectures compared
Built six ways to do retrieval-augmented generation, from a plain baseline to hybrid search, reranking, HyDE, query decomposition, and a self-correcting design, and compared them on the same 609-article news corpus and the same 80 questions. The architecture with the best final answers wasn’t the one that found the best evidence, and the design built specifically to avoid guessing guessed wrong more than anything else. Read the detailed write-up.
5. Agentic architectures compared
Built five ways to structure a multi-step AI agent, from a single agent deciding everything itself to a supervisor with a dedicated fact-checking role, and compared them on the same multi-hop question-answering task. The design that re-plans after every step, the one that sounds smartest on paper, never once decided on its own that it was done, in 40 separate tries; a supervisor with a skeptical verifier role won instead, by catching unsupported answers rather than out-reasoning the hardest questions. Read the detailed write-up.
6. Evaluation criteria across LLM, RAG, and agent systems
A framework built from the first three experiments, which evaluation criteria actually matter for which kind of system (plain LLM, RAG, single agent, multi-agent), and how to use that to decide something, then tested directly: the same question-answering task run through all four shapes with the task itself held fixed. Single-shot RAG tied the winning multi-agent system on accuracy while doing a fraction of the work, and a new redundant-step-rate metric split the multi-agent system’s runs almost perfectly by outcome: every run that exhausted its budget had re-retrieved evidence it already had. Read the detailed write-up.
7. Context window management compared
Built four ways to decide what stays in a model’s context window when there’s more content than comfortably fits, from keeping everything to keeping only what’s recent, only what’s relevant, or a compressed summary of what came before, and tested them on a controlled needle-in-a-haystack grid built from SQuAD. Retrieval Selection matched Full Context’s accuracy at a sixth of the tokens and beat it outright on the longest documents. Sliding Window failed as a hard, all-or-nothing cutoff rather than a gradual decline, and Hierarchical Summarization failed completely, 16 documents out of 16, whenever the needed fact landed in the middle of a chunk it summarized away. Read the detailed write-up.
More experiments get added here over time.