Understanding LLM Self-Routing in Inference

You don't need a 2 trillion parameter model to tell you the capital of France is Paris. Be smart and route between a panel of models according to query difficulty and model specialty! New paper proposes a framework to train a router that routes queries to the appropriate LLM to optimize the trade-off b/w cost vs. performance. Overview: Model inference cost varies significantly: Per one million output tokens: Llama-3-70b ($1) vs. GPT-4-0613 ($60), Haiku ($1.25) vs. Opus ($75) The RouteLLM paper propose a router training framework based on human preference data and augmentation techniques, demonstrating over 2x cost saving on widely used benchmarks. They define the problem as having to choose between two classes of models: (1) strong models - produce high quality responses but at a high cost (GPT-4o, Claude3.5) (2) weak models - relatively lower quality and lower cost (Mixtral8x7B, Llama3-8b) A good router requires a deep understanding of the question’s complexity as well as the strengths and weaknesses of the available LLMs. Explore different routing approaches: - Similarity-weighted (SW) ranking - Matrix factorization - BERT query classifier - Causal LLM query classifier Neat Ideas to Build From: - Users can collect a small amount of in-domain data to improve performance for their specific use cases via dataset augmentation. - Can expand this problem from routing between a strong and weak LLM to a multiclass model routing approach where we have specialist models(language vision model, function calling model etc.) - Larger framework controlled by a router - imagine a system of 15-20 tuned small models and the router as the n+1'th model responsible for picking the LLM that will handle a particular query at inference time. - MoA architectures: Routing to different architectures of a Mixture of Agents would be a cool idea as well. Depending on the query you decide how many proposers there should be, how many layers in the mixture, what the aggregate models should be etc. - Route based caching: If you get redundant queries that are slightly different then route the query+previous answer to a small model to light rewriting instead of regenerating the answer

𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹-𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗲𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 (𝗥𝗔𝗚) and 𝗟𝗼𝗻𝗴-𝗰𝗼𝗻𝘁𝗲𝘅𝘁 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗟𝗖) are two actively discussed approaches for GenAI architectures. RAG combines capabilities of Language Models (LLMs) with a search/information retrieval system to retrieve relevant information from external sources to augment the models responses. LCs can process long sequences of text to understand and integrate large amounts of information to generate coherent and contextually accurate responses over extended interactions. The paper 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗲𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝗼𝗿 𝗟𝗼𝗻𝗴-𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗟𝗟𝗠𝘀? 𝗔 𝗖𝗼𝗺𝗽𝗿𝗲𝗵𝗲𝗻𝘀𝗶𝘃𝗲 𝗦𝘁𝘂𝗱𝘆 𝗮𝗻𝗱 𝗛𝘆𝗯𝗿𝗶𝗱 𝗔𝗽𝗽𝗿𝗼𝗮𝗰𝗵 provides practical guidelines for applying RAG and LC and highlights the tradeoffs in cost and performance. While LCs provide better performance and accuracy, RAG has significantly lower computation cost due to decreased input lengths, especially considering that LLM API pricing is based on number of input tokens. The paper compares Gemini 1.5 Pro (with a million token context window), GPT-4o, and GPT-3.5-Turbo and the Contriver and Dragon retrievers (for RAG extensions). The experiments show that LC and RAG predictions are identical for up to 70% of the queries. This leads to a design decision to leverage RAG for majority of the queries and reserve the computationally expensive LC for a small subset of queries where it truly excels. The authors propose a hybrid approach called 𝗦𝗲𝗹𝗳-𝗿𝗼𝘂𝘁𝗲 𝘄𝗵𝗶𝗰𝗵 𝗵𝗮𝘀 𝗮 𝗥𝗮𝗴-𝗮𝗻𝗱-𝗿𝗼𝘂𝘁𝗲 𝘀𝘁𝗲𝗽: 𝘀𝗲𝗻𝗱 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗲𝗱 𝗥𝗔𝗚 𝗰𝗵𝘂𝗻𝗸𝘀 𝗮𝗻𝗱 𝘁𝗵𝗲 𝗾𝘂𝗲𝗿𝘆 𝘁𝗼 𝗮𝗻 𝗟𝗟𝗠 𝘄𝗵𝗶𝗰𝗵 𝘄𝗶𝗹𝗹 𝗱𝗲𝗰𝗶𝗱𝗲 𝘄𝗵𝗲𝘁𝗵𝗲𝗿 𝘁𝗵𝗲 𝗾𝘂𝗲𝗿𝘆 𝗶𝘀 𝗮𝗻𝘀𝘄𝗲𝗿𝗮𝗯𝗹𝗲 𝗼𝗿 𝗻𝗼𝘁. If it is, then the RAG answer is taken as the final answer. If not, the query and the full context is provided to the LC for the final answer. Examples: 𝗤𝘂𝗲𝗿𝘆: "𝘾𝙤𝙢𝙥𝙖𝙧𝙚 𝙩𝙝𝙚 𝙛𝙚𝙖𝙩𝙪𝙧𝙚𝙨 𝙖𝙣𝙙 𝙥𝙧𝙞𝙘𝙚𝙨 𝙤𝙛 𝙩𝙝𝙚 𝙩𝙤𝙥 𝙩𝙝𝙧𝙚𝙚 𝙣𝙤𝙞𝙨𝙚-𝙘𝙖𝙣𝙘𝙚𝙡𝙞𝙣𝙜 𝙝𝙚𝙖𝙙𝙥𝙝𝙤𝙣𝙚𝙨." 𝗦𝗲𝗹𝗳-𝗥𝗼𝘂𝘁𝗲 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻: 𝗥𝗔𝗚:The query is routed to RAG to fetch specific product details and current pricing from e-commerce websites. 𝗤𝘂𝗲𝗿𝘆: "𝘿𝙚𝙫𝙚𝙡𝙤𝙥 𝙖 𝙘𝙤𝙢𝙥𝙧𝙚𝙝𝙚𝙣𝙨𝙞𝙫𝙚 𝙩𝙧𝙚𝙖𝙩𝙢𝙚𝙣𝙩 𝙥𝙡𝙖𝙣 𝙛𝙤𝙧 𝙖 𝙥𝙖𝙩𝙞𝙚𝙣𝙩 𝙬𝙞𝙩𝙝 𝙢𝙪𝙡𝙩𝙞𝙥𝙡𝙚 𝙘𝙝𝙧𝙤𝙣𝙞𝙘 𝙘𝙤𝙣𝙙𝙞𝙩𝙞𝙤𝙣𝙨, 𝙘𝙤𝙣𝙨𝙞𝙙𝙚𝙧𝙞𝙣𝙜 𝙩𝙝𝙚𝙞𝙧 𝙢𝙚𝙙𝙞𝙘𝙖𝙡 𝙝𝙞𝙨𝙩𝙤𝙧𝙮, 𝙘𝙪𝙧𝙧𝙚𝙣𝙩 𝙢𝙚𝙙𝙞𝙘𝙖𝙩𝙞𝙤𝙣𝙨, 𝙖𝙣𝙙 𝙡𝙞𝙛𝙚𝙨𝙩𝙮𝙡𝙚 𝙛𝙖𝙘𝙩𝙤𝙧𝙨." 𝗦𝗲𝗹𝗳-𝗥𝗼𝘂𝘁𝗲 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻: 𝗟𝗖: The query is routed to the LC model because it requires integrating extensive patient history and detailed context. https://lnkd.in/gsAkFQCu

Reasoning Agentic RAG: The Evolution from Static Pipelines to Intelligent Decision-Making Systems The AI research community has just released a comprehensive survey that could reshape how we think about Retrieval-Augmented Generation. Moving beyond traditional static RAG pipelines, researchers from leading institutions including Beijing University of Posts and Telecommunications, University of Georgia, and SenseTime Research have mapped out the emerging landscape of Reasoning Agentic RAG. The Core Innovation: System 1 vs System 2 Thinking Drawing from cognitive science, the survey categorizes reasoning workflows into two distinct paradigms: Predefined Reasoning (System 1): Fast, structured, and efficient approaches that follow fixed modular pipelines. These include route-based methods like RAGate that selectively trigger retrieval based on model confidence scores, loop-based systems like Self-RAG that enable iterative refinement through retrieval-feedback cycles, and tree-based architectures like RAPTOR that organize information hierarchically using recursive structures. Agentic Reasoning (System 2): Slow, deliberative, and adaptive systems where the LLM autonomously orchestrates tool interaction during inference. The model actively monitors its reasoning process, identifies knowledge gaps, and determines when and how to retrieve external information. Under the Hood: Technical Mechanisms The most fascinating aspect is how these systems work internally. In prompt-based agentic approaches, frameworks like ReAct interleave reasoning steps with tool use through Thought-Action-Observation sequences, while function calling mechanisms provide structured interfaces for LLMs to invoke search APIs based on natural language instructions. Training-based methods push even further. Systems like Search-R1 use reinforcement learning where the search engine becomes part of the RL environment, with the LLM learning policies to generate sequences including both internal reasoning steps and explicit search triggers. DeepResearcher takes this to the extreme by training agents directly in real-world web environments, fostering emergent behaviors like cross-validation of information sources and strategic plan adjustment. The Technical Architecture What sets these systems apart is their dynamic control logic. Unlike traditional RAG's static retrieve-then-generate pattern, agentic systems can rewrite failed queries, choose different retrieval methods, and integrate multiple tools-vector databases, SQL systems, and custom APIs-before finalizing responses. The distinguishing quality is the system's ability to own its reasoning process rather than executing predetermined scripts. The research indicates we're moving toward truly autonomous information-seeking systems that can adapt their strategies based on the quality of retrieved information, marking a significant step toward human-like research and problem-solving capabilities.

LLM inferencing at scale involves finding the right combination of hardware, software, drivers, kernels, and routing. vLLM handles the engine layer. But when you scale to multiple replicas, a standard load balancer round-robins blind. It has no idea which pod already has your prompt prefix cached or which one has a full queue. That's the gap llm-d fills with its EPP (Endpoint Policy Processor).                                 I put together a 6-part hands-on series that walks through the full stack, running on a MacBook with no GPU needed:                                                               ➡️ Part 1: Deploy vLLM on a local Kubernetes cluster ➡️ Part 2: Add the llm-d gateway and EPP routing layer ➡️ Part 3: Scale to 3 replicas and observe load distribution ➡️ Part 4: Model aliasing with InferenceModelRewrite ➡️ Part 5: Fault tolerance: delete a pod mid-traffic and watch recovery ➡️ Part 6: Scrape EPP Prometheus metrics and watch pool size change live                                 Full series with architecture diagram here: https://lnkd.in/gecdtpSw                                 #llmd #Kubernetes #LLMInference #vLLM #OpenSource #GenAI 

Recursive Introspection: LLM finetuning approach to teach models how to self-improve Usually LLM does not exhibit the ability of continually improving their responses sequentially, even in scenarios where they are explicitly told that they are making a mistake. In this paper, authors present RISE: Recursive IntroSpEction, an iterative fine-tuning procedure, which attempts to teach the model how to alter its response after having executed previously unsuccessful attempts to solve a hard test-time problem, with optionally additional environment feedback. 𝗥𝗜𝗦𝗘 𝗔𝗽𝗽𝗿𝗼𝗮𝗰𝗵 𝗢𝘃𝗲𝗿𝘃𝗶𝗲𝘄 i) Problem formulation - convert single-turn problems into multi-turn Markov decision processes(MDPs) - The state is given by the prompt, history of prior attempts, and optional feedback from the environment. - An action is a response generated from the LLM given the state of multi-turn interaction so far ii) Data collection - collect data by unrolling the current model 𝑘 − 1 times followed by an improved version of the response, which is obtained by either (1) self-distillation: sample multiple responses from the current model, and use the best response, or (2) distillation: obtain oracle responses by querying a more capable model. In either case, RISE then trains on the generated data. 𝗥𝗜𝗦𝗘 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗮𝘁 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁 𝗧𝗶𝗺𝗲 i) with oracle - each time the model improves its response, it is allowed to check its answer against an environment and terminate early as soon as a correct answer is found ii) without oracle - ask the model to sequentially revise its own responses j times, and perform majority voting on all candidate outputs from different turns to obtain the final response - If turn number 𝑗 is larger than the iteration number 𝑘, the agent only keeps the most recent history with 𝑘 interactions to avoid test-time distribution shift. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 i) Metrics used: - with oracle, “p1@t5”: this run terminates the rollout as soon as response is correct. - without oracle, “m1@t5”: this run does not terminate rollout before five turns, and we compute maj@1 performance on candidates produced in each turn ii) Results: - RISE attains the biggest performance improvement between 1-turn (m5@t1) and 5-turn (m1@t5) performance w/o an oracle on both GSM8K and MATH - prompting-only self-refine largely degrades performance across the board - Using RISE on top of Mistral-7B exceeds even state-of-the-art math models such as Eurus-7B-SFT 𝗟𝗶𝗺𝗶𝘁𝗮𝘁𝗶𝗼𝗻𝘀 - Improving with self-generated supervision will likely require more computation and more iterations, since it will be slower than when using an off-the-shelf expert model - RISE requires running manual iterations and hence, a more “online” variant of RISE is likely solution in the long run 𝗕𝗹𝗼𝗴: https://lnkd.in/eAcCi99S 𝗣𝗮𝗽𝗲𝗿: https://lnkd.in/eP8VwHrz

Understanding LLM Routing What is LLM Routing? LLM routing is a technique used to dynamically direct user queries to the most appropriate Large Language Model (LLM) based on the complexity and specificity of the query. The primary goal is to balance response quality and computational cost by leveraging both high-quality closed LLMs (e.g., GPT-4) and cost-effective open-source LLMs (e.g., Mixtral-8x7B). Key Points on Building an LLM Router by Anyscale(h/t: Amjad Almahairi) 1. Data Collection and Labeling: - Anyscale collected a diverse set of queries from the Nectar dataset, which includes responses from various models, including GPT-4. - Queries were labeled using a 1-5 scoring system based on the quality of responses from Mixtral-8x7B, with higher scores indicating better quality. 2. Model Selection: - GPT-4 was chosen as the closed LLM for its superior response quality. - Mixtral-8x7B was selected as the open-source LLM for its cost-effectiveness. 3. Causal LLM Classifier: - A Llama3-8B model was finetuned as a causal LLM classifier to route queries based on their complexity. - The classifier was trained to predict the quality score of Mixtral-8x7B's response to a given query. 4. Training Process: - The training involved full-parameter finetuning of the Llama3-8B model using Anyscale's API. - The dataset was balanced to ensure the model was not biased towards any specific label. 5. Evaluation: - Offline evaluations were conducted using benchmarks such as MT Bench and GSM8K. - The performance of the LLM router was compared against random routing and other public LLM routing systems. 6. Routing Decision: - The router directs "simple" queries to Mixtral-8x7B if the predicted score is high (4-5), maintaining high response quality while reducing costs. - More complex queries are routed to GPT-4 to ensure high-quality responses. 7. Results: - The LLM router achieved significant cost reductions while maintaining response quality. - Evaluations showed that the router could achieve up to a 70% cost reduction on MT Bench and a 40% cost reduction on GSM8K compared to using GPT-4 alone. Advantages of LLM Routing - Cost Efficiency: By routing simpler queries to cost-effective models, LLM routing significantly reduces computational costs. - High-Quality Responses: Complex queries are directed to high-quality models like GPT-4, ensuring that response quality is not compromised. - Scalability: The system can handle a high volume of queries by efficiently distributing the load between different models. - Flexibility: The routing framework can be adapted to include new models and updated based on evolving performance metrics. - Optimized Resource Utilization: Balances the use of computational resources, ensuring that high-cost models are only used when necessary.

I learned these 4 concepts later than I should have. I’ve summarized everything you need to know in this post. For the longest part of my career in AI, I worked primarily with Deep Learning systems. I didn’t spend much time on tabular data, XGBoost, time-series forecasting, or recommender systems. Instead, I focused on AI for Video, Audio, and everything in between — from image generation to OCR to large-scale vision systems and robotics. But here's something I realized: Deployment strategies for deep learning models are very different from traditional ML. In fact, LLM systems feel a lot like deploying real-time vision systems. Both require high-performance inference, concurrency handling, and GPU optimization. The same principles apply: Optimized models. Optimized infrastructure. Optimized inference and GPUs. Distributed systems. But here’s where LLMs are different: inference is way more nuanced. In previous projects, I had to choose between real-time, batch, single-stage/ensemble, edge, or streaming inference. With LLMs, there are 4 distinct inference patterns you need to understand. Why? Because LLM inference happens in two phases: Prefill and Decode. ⇢ In the prefill phase, the model processes the entire input prompt and computes the key-value (KV) cache. ⇢ In the decode phase, it autoregressively generates tokens one at a time, updating the KV cache at each step. This matters because: 1/ Prefill is compute-bound (parallelizable and fast). 2/ Decode is memory-bound (sequential and slow, due to GPU memory traffic from KV cache updates). Understanding this gives you these four distinct LLM inference patterns: SISO – Short Input, Short Output → Low latency with fast generation. If using this pattern, aim to optimize for high concurrency. LILO – Long Input, Long Output → The most compute-intensive requires distributed inference, ideally with shared KV cache across nodes or tensor parallelism. LISO – Long Input, Short Output → This one has heavy prefill and quick decode. GPUs with enough VRAM would parallelize the prefill and generate fast. SILO – Short Input, Long Output → Light prefill, which runs fast but is heavier on decode. Decode latency dominates here. Takeaway: The longer the output, the more time is spent in the decode loop. Thus, before fine-tuning your own LLM and deploying it, keep in mind the inference pattern your system will target. Summary: - In LLM inference, there are 4 patterns. - For LISO, the generation and TFTT are fast. - On SISO, everything is fast. - For LILO, a lot of computing. - For SILO, prefill is fast, generation is slow. --- ♻️ Reshare and let others learn this too. Follow me to learn more about AI Engineering and production-ready AI Systems!

GPT-5’s secret ingredient for efficiency: the real-time router. The new router decides which LLM, path, or reasoning depth to use at inference time, balancing efficiency with accuracy under the one-recipe policy. This validates the vision our lab has pursued for the past a couple of years. Starting with router profiling and reasoning routes, we’ve since pushed the work further: 1. Evaluating exterior routers (https://lnkd.in/g7vQ6s4P): revealing why prior benchmarks fall short and proposing a holistic framework. 2. LLMs as self-routers during inference (https://lnkd.in/gUMWQ8i2; https://lnkd.in/gQyRvsuN): using uncertainty to enable on-device LLM routing. 3. Adaptive reasoning routes (https://lnkd.in/gkDBjhMz): dynamically adjusting reasoning depth to cut waste while preserving accuracy. We believe routing will strengthen the future of efficient inference. The open-source routing ecosystem is only just beginning, and we’d love to build it together.

We are at the inflection point, as multi-agent systems scale they are starting to break at the infrastructure and inference layer in ways most teams did not expect. Here is what is actually happening: When you deploy a multi-agent system, you are no longer running a single LLM request. One orchestrator spins up multiple sub-agents. Those agents make parallel tool calls, reason independently, and continuously pass context between each other. By the time a workflow reaches step-15, the system is often processing tens of thousands of tokens across multiple agents at the same time: all competing for the same GPU memory, KV cache, and inference scheduler. The inference stack was never built for this kind of workload, so what happens? KV cache fills across the workflow. Cache evictions cascade between agents. One agent’s prefill blocks another’s decode cycle. GPU utilization looks high, but real throughput drops hard. Latency spikes. Cost grows 2–3x beyond the expected model budget. This is where the architecture has to evolve, a few shifts become critical: 1. Shared KV memory across the workflow instead of isolated agent silos 2. Passing KV state between agents instead of recomputing context every time 3. Smarter model routing where smaller models handle simple tasks and larger models activate only when needed 4. Tracking cost at the workflow level, not just per token 5. Runtime controls for context limits, retries, concurrency, and tool execution This is distributed systems thinking applied to agent infrastructure. Agents behave like services, KV cache becomes shared infrastructure, Tool calls impact latency, Model tiers impact cost and throughput. That mental model is the shift (bringing distributed systems discipline into AI inference and agent execution.)

LLM inference is not prompt → model → response. It is a runtime system. While working on the next part of my series Architecting LLM Inference, I drew this diagram to simplify what actually happens inside a modern LLM runtime inference engine. The detailed article is still in progress, but I thought this visual might be useful to share meanwhile. Most people think LLM inference is: prompt → model → response But production inference is much more than a simple model.forward() call. A runtime like vLLM, TensorRT-LLM, SGLang, or TGI is really a serving engine that coordinates: request processing and tokenization queueing and runtime scheduling prefill/decode batching KV cache memory management GPU worker execution attention, MLP, and logits computation sampling and stop-condition checks detokenization and streaming And when tools or agents are involved, the runtime story expands further. The model may emit a tool-call instruction, but the application or agent layer usually parses it, validates it, executes the tool, and may call the model again with the tool result. The key mental model: Scheduler decides what runs. KV Cache Manager decides where attention memory lives. GPU Workers run the transformer math. Sampler chooses the next token. Tool execution usually sits outside the runtime. This is why LLM inference is not just an ML problem anymore. It is a systems problem involving distributed serving, GPU memory management, batching, runtime scheduling, kernel optimization, streaming, and agent orchestration. Detailed article coming soon on my Substack (link in comments) as part of the Architecting LLM Inference series. Subscribe if you’re interested in deep dives on LLM runtimes, KV cache, batching, and inference optimization. Would love to hear from folks building inference platforms: