media Together AI Blog · 1h ago · inference

ThunderAgent achieves 2x faster agentic inference via program-level scheduling

from English

ThunderAgent is a system that introduces a program abstraction for agentic LLM request scheduling to address KV cache thrashing in high-concurrency synthetic data generation. By treating agent workflows as schedulable programs rather than independent requests, it significantly improves throughput and reduces latency.

Achieves up to 2.5× higher single-node throughput and 10× lower P50 latency at high concurrency compared to default SGLang schedulers.
Delivers a 2.4× speedup on an 8-node cluster with near-linear throughput scaling from 16 to 64 GPUs.
Mitigates KV cache thrashing through program-level admission control and a global waiting queue for load balancing.
Compatible with existing offloading strategies like HiCache and speculative decoding via OpenAI-compatible endpoints.

The system allows users to adopt agentic inference optimizations without changing their underlying inference backend, offering practical scalability for large-scale agent training pipelines.

Importance 1/3 New harness with differentiators Trust 1/3 Together AI Blog NVIDIA AI agents Inference efficiency

Read original

lab NVIDIA Technical Blog · 19d ago

NVIDIA BioNeMo Agent Toolkit accelerates end-to-end co-folding performance

NVIDIA has introduced the BioNeMo Agent Toolkit to accelerate biomolecular structure prediction and co-folding workloads driven by AI agents. This toolkit addresses the need for speed and scalability across key pipeline steps, including Multiple Sequence Alignment (MSA) generation, co-folding inference, serving, and multi-GPU scale-out.

media r/LocalLLaMA · 20d ago

Untuned Qwen3.6-27B outperforms tuned Nemotron Puzzle-75B in agentic tasks

An evaluation of agentic capabilities shows that the untuned Qwen3.6-27B model successfully completed all tested tasks using 6-9 tool calls, whereas the tuned Nemotron Puzzle-75B required hand-tuned prompts and significantly more turns to pass.

media r/LocalLLaMA · 31d ago

Running Llama 3.1 405B on a Single 8xA100 Node with Hot-Loaded LoRA Adapters

A user demonstrates successfully running the Llama 3.1 405B model quantized to AWQ-INT4 on a single node equipped with eight A100 80GB GPUs, enabling up to 30 fine-tuned specialists to be loaded and switched in under 200ms.

arxiv arXiv cs.AI · 40d ago · 2 views

UltraQuant: 4-bit KV Caching for Context-Heavy Agents

UltraQuant enables 4-bit KV caching for context-heavy agents, reducing P50 time-to-first-token by 3.47x in late rounds and boosting output throughput by 1.63x over FP8 KV baseline. It achieves this using FP8 queries, FP4 KV tensors, UE8M0 group scales, and native scaled-MFMA on AMD CDNA4 GPUs, with optimizations for decode-attention kernels and robust design choices like asymmetric K/V treatment and Walsh-Hadamard rotation.

arxiv arXiv cs.LG · 41d ago · 1 view

UltraQuant: 4-bit KV Caching for Context-Heavy Agents

UltraQuant introduces a 4-bit KV caching method tailored for context-heavy agent workloads. It achieves 3.47x reduction in P50 time-to-first-token in late rounds and 1.63x higher output throughput compared to FP8 KV caching, using FP8 queries, FP4 KV tensors, and native AMD CDNA4 scaled-MFMA support.