The vLLM project has upgraded its transformers modeling backend to dynamically apply inference-specific layer fusions at runtime, allowing compatible models to achieve native vLLM inference speeds without custom code. This update leverages torch.fx for static analysis and AST manipulation to identify and optimize patterns such as fused operations for Expert Parallelization in Mixture-of-Experts models.
- Benchmarks on Qwen3 models (4B, 32B, and 235B-parameter FP8 MoE) show the updated backend matches the performance of vLLM's hand-written native implementations across single GPU, tensor parallelism, and data/expert parallelism setups.
- The integration supports standard parallelism options like --tensor-parallel-size and --data-parallel-size without changing serving setup configurations.
- Optimized operations include mappings to vLLM kernels for Expert Parallelization and fused MergedColumnParallelLinear/QKVParallelLinear blocks that enable automatic inference of tensor and pipeline parallel plans.
- Models processed through this backend remain fully torch compilable via torch.compile and CUDA Graphs, while retaining the ability to be used for training, evaluations, and RL rollouts.
This improvement allows model authors to run Hugging Face models inside vLLM with optimized performance without porting code, bridging the gap between ease of use and maximum hardware utilization.