The ggml project has introduced an initial Ethos-T (ET) backend in pull request #24179, enabling the use of Arm Cortex-M and Ethos-U NPUs within the llama.cpp framework.
- The implementation adds support for numerous kernels including MUL_MAT, ROPE, RMS_NORM, GLU, SOFT_MAX, GET_ROWS, and SET_ROWS.
- It includes performance logging, quantization helpers, and vectorized operations such as FP32 MUL_MAT using TensorFMA and parallelized softmax.
- The backend supports kernel fusion (e.g., RMS_NORM + MUL), FlashAttention, and various data types like Q4_0 and Q8_0.
- Build improvements allow importing the runtime via find_package() and support building on Python versions prior to 3.8.
This addition allows users to offload inference operations to Arm Ethos hardware, potentially improving performance and efficiency for compatible devices.