PrismML Bonsai 27B runs on Jetson Orin Nano 8GB with 4.31 t/s and 6.2GB RAM

The PrismML Bonsai 27B model is reported to be usable on the Jetson Orin Nano 8GB, consuming approximately 25W of power.

• Prompt processing speed reaches 27 tokens per second.
• Generation speed is 4.31 tokens per second with a context size of 48k.
• RAM usage is 6.2GB on cold start without memory mapping.

The setup is described as capable of handling complex tasks effectively despite the modest performance metrics.