TokenRouter: Efficient Serving System for Token-Level LLM Routing

0:00Hello and welcome to Daily Papercast. Today, we're diving into a paper from the Hugging Face Daily Paper List of October 9, 2026, which has received 84 upvotes. The paper is titled, Token Router, Efficient Serving System for Token Level LLM Routing. The first two authors are Tien-Yu Fu and Teng-Suan Lu, and the corresponding author is Yu-Wang, all from Tsinghua University. Great! Let's dive into the introduction. Large language models, or LLMs, have shown strong capabilities across various applications. That's right. To meet diverse demands, modern LLMs vary significantly in size and domain expertise, leading to different latencies and capabilities. And we're seeing a common serving paradigm emerge called model routing to exploit this diversity and improve the cost-quality trade-off. Exactly. Course-grained routing at the session or query level is already widely adopted in production systems.

1:06For instance, commercial systems like chat GPT and cursor route each user query to different back-end models based on factors like predicted difficulty or topic. Ah, so once a request is routed in these systems, the response is generated entirely by a single model until the next round, right? Correct. But recent algorithmic work indicates that routing at the finer token level unlocks benefits that query-level routing can't achieve. Can you break down those benefits for us? Sure. On the efficiency side, token-level routing leverages the sharp variation in generation difficulty within a single query. This means you can use a lighter model for easier tokens and only switch to a heavier model when needed, which saves computational resources. And it seems like that can also enhance the quality of responses, right? Exactly. Token-level routing allows models with complementary expertise to collaborate within a single response, often surpassing the generation quality of any single LLM.

2:10However, existing LLM serving systems aren't quite set up for this finer level of granularity, are they? No, they're not. Current systems like SGLANG and VLLM are designed under the assumption that all active requests stay synchronous at each decoding step, because they usually serve a single LLM. But how does token-level routing break this assumption? And what challenges does that introduce? Good question. Token-level routing allows the target LLM to change at every decoding step, which introduces three main challenges, step desynchronization, batch admission delays, and increased implementation complexity. Could you elaborate on each of those challenges? Sure. First, step desynchronization. Different LLMs have different per step latencies, so keeping requests in a single batch synchronized forces every step to wait for the slowest model,

3:12leaving faster models idle. And that causes inefficiencies right from the get-go. Exactly. Second, batch admission delay. Since token-level routing causes frequent model switches, a routed request might arrive while its target model is already processing a previous batch. This means it has to wait to be admitted into the next batch, creating bubbles and fragmenting batches. Got it. And the third challenge, implementation complexity? Implementing token-level routing in current systems requires extensive modifications, because they don't provide a programming interface for per step routing decisions. Developers have to coordinate routing logic with features like continuous batching and prefix caching, which aren't designed for this level of granularity. So the paper proposes a solution to tackle these challenges? Yes. They propose token router, which is designed to be both efficient and developer-friendly for token-level routed LLM inference. It follows the principle of, request-centric programming, model-centric execution.

4:15What does that mean exactly? It means that developers describe the routing logic from the perspective of a single request, while the runtime handles the launch of subsurvers for each LLM, and dispatches requests asynchronously. This design keeps developers away from the system complexity, and lets the runtime optimize cross-model execution and batch scheduling. Sounds like a smart separation of concerns. How does token router perform? token router uses a delayed batching scheduler, whose optimal hyperparameters are derived from a mathematical throughput model called the system. According to the experiments in the paper, token router achieves between 2.01 to 64.15 times higher decoding throughput compared to existing systems across various routing algorithms, workloads, and model pairs. Wow! That's a significant improvement! Indeed, and that wraps up the introduction section. Let's delve into the core methods proposed in the token router paper.

5:18They start by highlighting the main principle behind token router. Request-centric programming, model-centric execution. This means developers describe how a single request should flow across multiple LLMs, and the system handles the execution details. That sounds like it simplifies the development process considerably. How do they implement this? Token router achieves this through three main components. Rout, send, and receive functions. Developers use these to define when a request should switch models, what state should be transferred, and how returned tokens should be incorporated. Could you break down each of these components? Sure, the route function is called after each decoding step. It determines a destination index based on the four-repass results, like hidden states, next-token logits, or sample tokens. A destination of zero means the request continues on the current model. Any other number points to a peer model. And the send function?

6:19The send function is invoked when route delegates a request. It creates a peer-req message, which contains the required details like request ID, token suffix, status, and any additional algorithm-specific fields needed by the receiver. So, it essentially packages the request for the next model? Exactly. Then there's the receive function, which processes the incoming peer-req. It converts this message into the local format, allowing the receiver model to continue decoding. That covers the programming interface. What about the system design for execution? TokenRatter's execution model is based on decoupled tri-loop asynchronous execution. The three loops are client server, decoding, and intermodel. This design resolves step desynchronization by allowing each sub-server to proceed at its own pace. Can you explain how these loops work together? Sure. The client server loop handles request admission and response streaming.

7:21The decoding loop schedules and executes model tasks. The intermodel loop manages peer requests by sending and receiving routed requests between sub-servers. Essentially, requests move asynchronously across these loops. How does the system handle handoffs between models? TokenRatter uses a handoff and resume mechanism. When a request is routed to another model, it enters a pending state. Upon resumption, the request status is toggled back to running or finished, based on the peer model's output. This minimizes redundant operations like prefix matching and KV cache allocation. And what about scheduling the batches efficiently? That's where the delayed batching scheduler comes in. Instead of processing a request immediately upon arrival, the scheduler waits for a short, controlled interval to gather more requests targeting the same model. This reduces batch admission delays and leads to larger, more efficient batches. How do they determine the optimal waiting time for these batches?

8:24TokenRatter uses a mathematical model to derive the throughput, optimal delayed batching threshold. This model balances the tradeoff between batch size and waiting time, ensuring efficient request processing without unnecessary delays. What kind of performance improvements do they report with TokenRouter? The paper shows that TokenRouter achieves significant efficiency gains across different routing algorithms, workloads, and model pairs. They report throughput improvements ranging from 2.01 to 64.15 times compared to existing systems. For example, in concurrency scenarios, TokenRouter's throughput increased by 8.61 times while maintaining more than half of the single user speed. Those are impressive figures. Did they test this across different models and scenarios? Yes, they evaluated TokenRouter with various token-level routing algorithms like CIDER, R2R, RStitch, CoLLM, and ME using different pairs of LLMMs.

9:26For example, RStitch delegates high entropy tokens to a large model while the large model sends low entropy tokens back. And how did TokenRouter fare against traditional methods? In addition to throughput, TokenRouter also reduced end-to-end latency substantially. Compared to the standard implementation, they saw latency improvements ranging between 2.03 to 63.64 times. Do they provide any detailed comparisons using specific benchmarks? They used benchmarks like AIM, E2, 024 for reasoning workloads, and SWE Smith for agentic tasks. In these tests, TokenRouter consistently outperformed other implementations, demonstrating it can handle both short input output tasks and more complex multi-turn interactions. So it really does show robust performance across different types of workloads. Yes, indeed. Even when evaluated under the original settings of the token-level routing algorithms,

10:32TokenRouter showed substantial improvements. For example, in the original settings for the citer algorithm, TokenRouter's throughput was nearly 2.73 times higher than the official implementation with a corresponding latency reduction. It sounds like TokenRouter not only handles the fine-grained token-level routing, but does so very efficiently. Exactly. And that wraps up the method section. All right, Ashley. Let's move on to the experiment section. How did the authors of TokenRouter validate their system? They conducted comprehensive experiments to measure the performance of TokenRouter across a wide range of setups. These experiments focused on evaluating five different token-level routing algorithms, citer, R2R, R-stitch, co-LLM, and ME. Which models did they use for these experiments? TokenRouter was tested with QN3W0.6 billion and QN3,

11:323.2 billion for most algorithms, and QN3 8 billion was also included for the ME algorithm. They also evaluated each algorithm with model pairs and data sets from their respective original papers. Got it. How about the hardware setup? All experiments were run on an 8 times A1 00080G GPU server. For two model algorithms, the small model was deployed on one GPU and the large model on two GPUs using tensor parallelism. For the ME algorithm, each model was deployed on a separate GPU. Interesting choice of hardware. What kinds of workloads did they test? They tested three types of workloads. First, low effort reasoning with short input and output links. Second, high effort reasoning where both input and output are longer. And third, agentic tasks which involve multi-turn interactions. Can you give us examples of data sets used for these workloads?

12:34For low effort reasoning, they used AIME2024 prompts. High effort reasoning also used AIME2024 but with longer output links. For agentic tasks, they used SWE Smith trajectories. What kind of performance improvements did token router demonstrate? Token router showed significant improvements in throughput and latency compared to standard serving systems. Across 15 algorithm workload combinations, token router achieved throughput gains ranging from 2.01 to 64.15 times higher than existing systems. That's an impressive range. Could you highlight some specific results? Certainly. For instance, token router improved throughput by 2.73 to 21.97 times under the original settings of the token level routing algorithms. This includes latency reductions of up to 21.61 times from the official implementations of these algorithms.

13:37How does token router perform in terms of efficiency under different concurrent request levels? Great question. According to Figure 1B in the paper, token router's throughput increases by 8.61 times when concurrency rises from 1 to 16 while still maintaining 51.7% of its single user speed. In contrast, a traditional system like R2R's official implementation only increases by 5.14 times and retains just 31.3% of its single user speed. That's quite a leap. Did they also test token router with different model pairs? Yes, they did. Table 3 in the paper shows that token router outperformed the official R2R implementation by 1.99 to 3.21 times across different SLM and LLM scales. This indicates that their optimizations are robust across various models sizes. Were there any interesting insights from these tests?

14:41Indeed, the paper includes an ablation study breaking down the contributions of various optimizations. For example, extended coutographs improved throughput by approximately 1.88 to 2.12 times. asynchronous execution and delayed batching added further gains. How about the deployment configurations? Did they explore different setups? For example, they evaluated non-overlapping and cross-node deployment configurations. Cross-node deployment showed comparable throughput to single-node setups, translating to 1.93 to 32.81 times improvements over official implementations. You mentioned something about a throughput accuracy trade-off earlier. How does token router hold up in that regard? According to Figure 7, token router shifts token-level routing to a new Pareto frontier. It shows that token routers fine-grained routing achieves higher throughput without sacrificing accuracy, making it competitive against query-level routing on state-of-the-art frameworks like SGLang.

15:47Token router really does seem to bring a lot to the table in terms of efficiency and performance. Definitely, and that wraps up the experiment section. Let's move on to the related work section. Ashley, can you give us an overview? Of course. The related work section of this paper focuses on three main areas, model routing, token-level routing, and LLM serving systems. Let's start with model routing. What is the paper highlight there? Model routing is about distributing requests across different LLMs to improve efficiency or quality. Most methods work at the session or query-level, deciding on models for entire requests or conversation turns. I see. And these methods typically send easier requests to smaller models, while reserving larger models for more complex queries? Exactly. Efficiency-oriented routers often choose cheaper models for easier requests under a quality constraint, aiming to balance cost

16:51and performance. On the other hand, quality-oriented routers select the best expert for the request, and sometimes combine outputs from multiple models to exceed the capabilities of any individual LLM. And this is why production systems can reuse standard single LLM batching and scheduling. Correct? Yes, that's correct. But token router takes a different approach, targeting token-level routing. This requires coordinating model switches within a single response, which is quite different from session or query-level routing. How does token-level routing differ then? Token-level routing selects a model for each token or short token span within a response. Efficiency-oriented methods use larger models for challenging tokens or segments, relying on indicators like confidence and entropy. Quality-oriented methods can combine model expertise by either accepting expert tokens or fusing output distributions. And these methods share a common life cycle, right?

17:52Correct. The life cycle typically involves selecting the next model, transferring the request state, receiving tokens or distributions, and resuming generation. This life cycle motivates token routers roots and receive programming interface. Interesting. Now, what about LLM serving systems? What are their key characteristics? Modern LLM serving systems accelerate inference using techniques like continuous batching, paged key value cache management, and efficient structured generation runtimes. These techniques are optimized for single model decoding, where each request stays with one model advancing step-by-step at that model's pace. Does this system also support speculative decoding? Yes, although speculative decoding uses multiple models, it couples draft generation and target verification in a predictable pattern. Token-level routing is different because it enables dynamic model switches based on data dependency at each token, leading to irregular arrivals and fragmented batches. How does token router

18:57address these irregularities and fragmented batches? Token router adopts a model-centric run time. It hosts each model as an autonomous subserver, preserving continuous batching and local prefix caching. This means each subserver operates independently, reducing synchronization overhead, and addressing the unique challenges posed by token-level routing. And this model-centric approach is quite distinct from traditional single LLM servers, isn't it? Exactly. Each subserver and token router implements three user-defined routing functions, route, send, and receive, and progresses requests asynchronously through three scheduler loops. This structure allows token router to scale efficiently by simply adding more subservers as needed. So, we've looked at model routing, token-level routing, and LLM serving systems. How does token router integrate these into its core operations? Token router synthesizes the principles of model routing with the detailed

20:02control of token-level routing and advanced serving optimizations to offer robust, high-efficiency LLM operations. It's truly a step forward in bridging the gap between theoretical models and practical serving systems. And with that, we conclude the related work section. Let's wrap up today's discussion by summarizing the key contributions and takeaways from the token router paper. Sure, the paper introduces token router, an efficient token-level routing system for large language models. Right. And this system addresses some major challenges in token-level LLM routing, like step-decycronization, batch admission delays, and implementation complexity. Exactly. Token router adopts a request-centric programming, model-centric execution approach. This allows developers to describe routing logic simply while the system takes care of efficient model execution. This approach simplifies the development process and hides the complexity

21:04from developers. Token router's architecture features decoupled, try-loop, a synchronous execution, a hand-off and resume mechanism for model transitions, and a delayed batching scheduler to optimize batch admission. The delayed batching scheduler, with its throughput optimal threshold, is particularly interesting as it balances batch size and waiting time efficiently. Indeed, token routers experimental results are impressive, showing significant improvements in throughput and latency across various routing algorithms, workloads, and model pairs. With improvements ranging from 2.01 to 64.15 times in throughput, token router really sets a new standard for efficient LLM routing. Token router not only demonstrates the feasibility and benefits of token-level routing, but also provides a practical, scalable framework for real-world applications. And that's a wrap on today's episode of Daily Papercast. We hope you found our discussion on Token router insightful.

22:07Remember to check out the paper for more detailed information. Thank you for tuning in. We look forward to having you join us again tomorrow for another dive into cutting-edge research. Until next time, stay curious and keep exploring.