zhongkaifu has released TensorSharp, an open-source local LLM inference engine built entirely in C# that supports Windows, macOS, and Linux. The engine handles GGUF models from Unsloth, including Gemma 4 and Qwen 3.6 with multimodal capabilities, while providing APIs compatible with OpenAI and Ollama.

  • TensorSharp implements the inference engine from scratch rather than wrapping llama.cpp, offering CPU, CUDA, MLX, and GGML backends.
  • It incorporates optimizations such as paged KV cache, continuous batching, and SSD-based cache for MoE models.
  • Benchmarks show TensorSharp achieving performance ratios ranging from 0.84x to 1.28x compared to llama.cpp across various decode, prefill, and TTFT metrics on CUDA and Vulkan backends.

The project provides a console application, a web-based chatbot interface, and HTTP APIs for programmatic access, aiming to leverage full GPU capability across Nvidia, Apple, AMD, and Intel hardware.