RankGuide: Tensor-Rank-Guided Routing and Steering for Efficient Reasoning

RankGuide is a training-free serving framework that makes small-reasoning-model / large-reasoning-model (SRM–LRM) collaboration both faster and more accurate. Unlike prior routers, RankGuide:

• Uses tensor-rank signals from consecutive hidden states — not entropy alone — to detect both uncertain and overconfident SRM failures.
• Guides routing (when to fall back to the LRM) and steering (keeping the SRM on a productive trajectory) both from the tensor-rank signal.

• 🚀 [2026/08/08] Code Released!
• 🚀 [2026/04/17] We are excited to introduce RankGuide, a tensor-rank-guided routing-and-steering framework for efficient SRM–LRM collaborative reasoning.

Clone with submodules (a patched vLLM and LiveCodeBench are submodules):

git clone --recurse-submodules https://github.com/TTTTTTris/RankGuide.git
cd RankGuide
# or, if already cloned:
git submodule update --init --recursive

Install dependencies:

conda create -n rankguide python=3.12 -y
conda activate rankguide
pip install -e vllm          # patched vLLM (adds the static activation-steering hook)
pip install datasets tensorly openai transformers tqdm torch safetensors
pip install -r TensorRouter/evaluation/requirements.txt   # answer graders only

The routing code uses tensorly for TT-SVD. Steering needs no changes to your inference code: the hook lives in the vLLM submodule and is switched on entirely through STATIC_STEER_* environment variables when the SRM server is launched.

Everything runs from the TensorRouter/ directory. One script launches both vLLM servers and sweeps the full threshold grid — no need to start servers separately:

cd TensorRouter
bash scripts/run.sh          # RankGuide + GlimpRouter + SpecReason on AIME24
bash scripts/eval_grid.sh    # pass@1 for the steered configs, once the sweep finishes
STEERING=0 bash scripts/eval_grid.sh   # ... and for the unsteered configs + baselines

run.sh runs two phases, because the baselines must face an unsteered SRM:

Every setting is an overridable default in scripts/_config.sh — models and ports, GPUs, steering vector, WINDOW_SIZE/TT_RESHAPE, the grid lists — so you change behaviour by exporting, not by editing:

CUDA_VISIBLE_DEVICES=0,1 TP_SIZE=2 HF_HOME=/my/hf/cache bash scripts/run.sh
TASKS="aime25 gpqa" TOKEN_BUDGET=8192 bash scripts/run.sh
RANK1_LIST="7" RANK2_LIST="60" ENTROPY_LIST="1.1" bash scripts/run.sh   # a single config
PHASE_STEERED=0 bash scripts/run.sh                                    # baselines only

Each sweep writes one pickle per (problem, repeat) under TensorRouter/rankguide/, with the sweep config encoded in the directory name:

rankguide/results/<config>/<dataset>_<budget>/<lrm>_<srm>/<problem_id>/<repeat_id>.pickle
rankguide/logs/<config>/...              # per-problem stdout logs
rankguide/logs/timings.csv               # shell wall-clock per problem / config / sweep

timings.csv grows as the sweep runs (timestamp,scope,label,dataset,seconds,hms,detail, one row per problem / config / sweep), so a long grid can be watched live. That is driver wall-clock; the per-step model latency is in the pickles:

Each pickle is a list of per-step metadata dicts whose timing fields are independent wall-clock spans around sequential, non-overlapping blocking calls:

Other fields: score (rank_1 / rank_2 / first_token_entropy in tensor mode), step_str, num_output_tokens_base / _small, forced_direct_answer, and stop_reason (finished / budget) on the last reasoning step.

A small reasoning model drafts most steps cheaply, but fails in three characteristic ways:

• Overconfidence — confident but wrong. Incorrect trajectories collapse to low rank (median min TT-rank 5 vs. 9 on the 1st core, 59 vs. 79 on the 2nd; panels 1-2), yet those low-rank steps carry low entropy (0.388 vs. 0.983; panel 3) — so entropy-only routers miss exactly the steps that need the LRM.
• Uncertainty — the complementary failure: the model makes no progress and its output entropy is high. Entropy catches this one, which is why RankGuide routes on rank and entropy.
• Heavy revalidation — the SRM spends ~2× the LRM's share of steps re-checking earlier work, and more still when it is wrong (panel 4: 0.29 vs. 0.15 overall, 0.36 on incorrect responses).

Key insight: the tensor-train (TT) ranks of a sliding window of hidden states measure the intrinsic dimensionality of the reasoning dynamics — a signal entropy alone does not provide.

Both signals are computed at reasoning-step boundaries ("\n\n") over a sliding window of W steps; see the diagram at the top of this README.

Rank-guided routing (online) — from the window's hidden states, take the reasoning-step rank r1 and hidden-feature rank r2 (error-bounded TT-SVD, score.py) plus the SRM's next-token entropy H:

Rank-guided steering (offline → online) — offline (calibration.py), split calibration steps into execution vs. validation, keep only high-rank samples, and set v = mean(high-rank execution) − mean(high-rank validation). At inference the vLLM hook adds it at step boundaries, H ← H + α·v. The paper's vectors ship precomputed in vector_500_500/, so reproducing the main results needs no calibration run; to build a new one:

python calibration.py --correct_hidden <hidden.pt> --incorrect_hidden <hidden.pt> \
  --layer_idx 28 --rank_threshold 40 --output vector_500_500/<model>/<name>.pt

DeepSeek runs on 4× A100 40GB; Qwen3 on 2× H100. DeepSeek uses greedy decoding; Qwen3 uses temperature 0.6 / top-p 0.95.

• SRM-only / LRM-only — lower-latency vs. upper-bound-accuracy references.
• SpecReason — speculative reasoning with LLM-based verification and rollback (--score_method greedy).
• GlimpRouter — entropy-based routing without rollback (--score_method glimprouter).

T_e ∈ {0.7, 0.9, 1.0, 1.1}, T_{r1} ∈ {5, 6, 7, 8}, T_{r2} ∈ {50, 60, 70}, window W = 10, steering scale α = 1.0.

Accuracy vs. end-to-end latency for both model pairs on all three benchmarks. RankGuide (orange stars) sits on the accuracy–latency frontier in every panel: at matched accuracy it is the fastest collaborative method, and it dominates SpecReason and GlimpRouter — up to 1.75× lower latency than LRM-only and 1.36× lower than the best prior collaborative method. On AIME24 (DeepSeek) it matches LRM accuracy at 1.41× lower latency, with 19.3% fewer reasoning steps.

TensorRouter/                  # RankGuide research code (built on SpecReason)
  rankguide.py                 # main entry: SRM/LRM collaboration + rank-guided routing
  score.py                     # error-bounded TT-SVD ranks of a hidden-state window
  calibration.py               # offline rank-guided steering-vector extraction
  scripts/                     # entry points are sweep_grid* and eval_grid*
    _config.sh                 #   all defaults + shared helpers; override via the environment
    sweep_grid.sh              #   launch servers + sweep the grid, no steering (+ baselines)
    sweep_grid_steer.sh        #   same, with rank-guided steering
    sweep_grid_steer_lcb.sh    #   same, for the Qwen3 + LiveCodeBench setup
    sweep_one.sh               #     <- one config over one dataset (servers must be up)
    sweep_one_steer.sh         #     <- same, with steering
    eval_grid.sh               #   score a swept grid, pass@1 (AIME / MATH / GPQA)
    eval_grid_lcb.sh           #   score a swept grid, LiveCodeBench
  evaluation/                  # answer parsing + graders (math/GPQA)
  vector_500_500/              # precomputed steering vectors per model/layer
  LiveCodeBench/               # submodule: LiveCodeBench harness
vllm/                          # submodule: vLLM patched with the STATIC_STEER hook
assets/                        # figures used in this README

Sweeps additionally create TensorRouter/rankguide/{results,logs}/ and TensorRouter/data/ at runtime; these are gitignored.

@article{tian2026rankguide,
  title={RankGuide: Tensor-Rank-Guided Routing and Steering for Efficient Reasoning},
  author={Tian, Jiayi and Su, Yupeng and Solgi, Ryan and Kundu, Souvik and Zhang, Zheng},
  journal={Conference on Language Modeling  (COLM)},
  year={2026}
}

Our code is developed based on SpecReason and vLLM.