A GPU stress test using frozen 4-bit Qwen2.5-7B-Instruct on HotpotQA bridge questions demonstrates that task-aware semantic routing significantly improves efficiency and accuracy under high-distractor conditions. The study compared full capped context against a query-aware semantic selector retaining 60% of the budget, revealing substantial gains in speed and precision.

  • Mean Qwen input tokens dropped by 41.32%, from 13,124.5 to 7,701.8.
  • Mean prefill time decreased by 46.85%, from 870.8 ms to 462.8 ms.
  • End-to-end latency including selection fell by 45.02%, from 943.9 ms to 518.9 ms.
  • Peak VRAM usage was reduced by 2.55 GiB, from 11.71 GiB to 9.16 GiB.
  • Token F1 score increased by 0.0468, rising from 0.5342 to 0.5810.

The results indicate that retaining only 60% of context via semantic routing reduces computational costs while producing higher answer quality, though the test specifically measures robustness to long, irrelevant context rather than natural-distribution performance.