Beyond LLM Routing: A New Way to Optimize Agent Pipelines

⬅️ Back to Videos

🎥 Beyond LLM Routing: A New Way to Optimize Agent Pipelines

Melissa (UC Berkeley). Duration: 30 min

Timestamps

  • 0:00 The agent configuration problem
  • 2:52 How people configure systems today (vibe coding)
  • 6:04 Pipeline changes beat model changes
  • 8:21 Per-query variance
  • 11:24 The Matei vs. Melissa example
  • 13:19 Research question formulation
  • 15:32 BRAIN system overview
  • 19:08 Query characteristics
  • 20:25 Per-configuration predictors
  • 22:26 Results and cost reduction
  • 23:52 Classical ML beats end-to-end deep learning
  • 26:05 Future work and coding agents

Most teams building agent systems configure them by gut feel: pick a strong model within budget, run A/B tests, ship if it passes. Melissa ran the numbers on that approach and found costs vary by three orders of magnitude for the same accuracy. She presents BRAIN, a system that picks the right pipeline for each query automatically.

  1. Choosing an agent architecture matters more than choosing a model. Fix the model to GPT-5 Nano and vary the pipeline (LM-only, simple RAG, agentic RAG, full agent loop), and you span three orders of magnitude in cost and a wide accuracy range. Vary the model while holding the pipeline fixed, and the range is much smaller. Pipeline configuration, not model selection, is where the real leverage is.

  2. Current industry practice is vibe configuration. Melissa ran a study of production agent systems. Teams pick a model from the Arena leaderboard, run some test cases, and ship. Nobody systematically considers cost-performance tradeoffs. And anecdotally, the bill is starting to bite: Microsoft and Uber recently flagged that their AI spend no longer justifies the automation gains.

  3. Queries vary massively in what they need, even within the same workload. “Who is Matei Zaharia?” GPT-5 already knows, answer with LM only. “Who is Melissa?” Same semantic structure, but the model hallucinates. It needs web search. A static pipeline that handles both cases is either over-engineered for the easy ones or under-powered for the hard ones.

  4. BRAIN’s core idea: transform the query into structured characteristics, then predict the best configuration. Instead of feeding raw natural language to a router, BRAIN uses an LM to generate workload-specific binary features for each query: “involves a famous person?”, “requires multi-hop reasoning?”, “references cooking?”. This vector is much closer to the system configuration space than the raw text is.

  5. The characteristics are generated per-workload, not universal. Given 25 sample queries from a benchmark, an LM proposes the dimensions that separate them. The FinanceBench characteristics look different from the BrowseComp ones. This keeps the feature space small and meaningful.

  6. BRAIN trains one classical ML predictor per candidate configuration. Each predictor (XGBoost or logistic regression) learns: given a query with these characteristics, will this configuration answer it correctly? At runtime, a new query hits all predictors, and BRAIN picks the cheapest configuration whose predictor says it will succeed.

  7. The results: 89% cost reduction at 100% static accuracy on BrowseComp Plus. BRAIN is the only method tested that meets the 100% accuracy target across all three benchmarks (MUSIC, BrowseComp Plus, FinanceBench) while cutting cost. And the savings come from doing less work per query, not from guessing wrong on hard ones.

  8. Classical ML beats end-to-end deep learning here. Melissa tried fine-tuning BERT and transformers on the profiling data to predict configurations directly. They performed worse than XGBoost on the query characteristics. The reason: profiling data is expensive. Just one benchmark cost $11,000 to profile. Deep learning needs more data than that to beat a good feature engineering approach.

  9. Per-query routing is the natural next step beyond per-workload tuning. Most optimization today happens at the workload level (tune one pipeline for all queries). BRAIN shows that the intra-workload variance is large enough that per-query routing is worth the complexity. The system that treats every query the same is burning budget on easy ones and under-serving hard ones.

  10. The next frontier is coding agents. Melissa is extending BRAIN beyond retrieval-heavy agent systems to coding agents, where the configuration space includes tools, context windows, and test strategies. That is a harder problem. It is also where most production money is going.

Related TMFNK Content

Crepi il lupo! 🐺