SonicBoom: LLM Inference in C++ and CUDA without Python

HardenedVault (HardenedLinux) has released SonicBoom on GitHub — a runtime for local LLM inference in C++/CUDA without Python. In v0.0.3 on an RTX 3050, it decodes Gemma 4 E4B (53.6 tok/s) and Qwen2.5-0.5B (345.9 tok/s), running 13–20% faster than llama.cpp.

Compiled by Sergey KostenchukPublished 2026-10-20Updated 2026-10-20

2026-10-20 Coding OpenAI

💻 SonicBoom: LLM Inference in C++/CUDA without Python

HardenedVault (HardenedLinux) has released SonicBoom on GitHub — a local LLM inference runtime: a single libsonicboom.so file with a C ABI, Scheme (GNU Guile) as the primary language, GGUF support, CPU, CUDA, and a server with an OpenAI-compatible API. In v0.0.3 on an RTX 3050 8GB, Gemma 4 E4B achieves 53.6 tok/s compared to 44.8 for llama.cpp, while Qwen2.5-0.5B reaches 345.9 compared to 306.9.

🌍 The performance gain comes from a level above the kernels: the entire decoding step is executed as a single CUDA Graph, and an MLIR-based compiler generates kernels for specific hardware. This is an auditable stack (GPL-3.0, C ABI) for air-gapped deployments.

👤 If you have an NVIDIA RTX 30-series GPU (sm_86 verified), you can build it, run Gemma 4 E4B or Qwen2.5-0.5B locally, and measure speed using tools/bench.sh. However, the project is young (v0.0.3, 18 stars), supports only greedy sampling and two model families — it is an experiment, not a replacement for llama.cpp.

Source 1: https://github.com/hardenedlinux/SonicBoom

Sources