Turboderp has released ExLlamaV3 v1.0.0, the first production version of the library after over a year of development. This update introduces significant architectural changes and kernel optimizations aimed at improving inference speed and reducing dependencies.
- Removed flash-attention-2 and xformers dependencies.
- Extended tensor-parallel support to most models, including Gemma4.
- Introduced a new attention kernel with online cache quantization that eliminates KV quantization slowdowns.
- Added graph paths for all attn/GDN modules and a new conv1d kernel.
- Greatly improved GEMM/GEMV performance on Ampere GPUs and added a new INT8 GEMV kernel.
- Implemented a new MoE kernel ticket scheduler and added support for GptOssForCausalLM and NemotronHForCausalLM.
The release aims to provide faster inference capabilities and broader model compatibility through these extensive optimizations and bug fixes.