The ggml project has introduced an initial Ethos-T (ET) backend in pull request #24179, enabling the use of Arm Cortex-M and Ethos-U NPUs within the llama.cpp framework.

  • The implementation adds support for numerous kernels including MUL_MAT, ROPE, RMS_NORM, GLU, SOFT_MAX, GET_ROWS, and SET_ROWS.
  • It includes performance logging, quantization helpers, and vectorized operations such as FP32 MUL_MAT using TensorFMA and parallelized softmax.
  • The backend supports kernel fusion (e.g., RMS_NORM + MUL), FlashAttention, and various data types like Q4_0 and Q8_0.
  • Build improvements allow importing the runtime via find_package() and support building on Python versions prior to 3.8.

This addition allows users to offload inference operations to Arm Ethos hardware, potentially improving performance and efficiency for compatible devices.