HardenedVault (HardenedLinux) has published the SonicBoom project on GitHub — a runtime for local LLM inference written entirely in C++ and CUDA and not using Python: the entire library compiles into a single libsonicboom.so file with a C ABI. According to the authors' measurements in version v0.0.3, decoding on an RTX 3050 8GB card is 13–20% faster than llama.cpp with identical quantized model files. Since this has only been verified on two small models so far, SonicBoom should be viewed as an early experimental platform for local inference, not as a ready-made replacement for familiar tools.
What happened
The hardenedlinux/SonicBoom project is published under the GPL-3.0 license, with the current version at the time of release being v0.0.3. The runtime supports CPU and CUDA, with Scheme chosen as the primary interaction language on top of the GNU Guile engine, and models are loaded in GGUF format. The included local server is compatible with the OpenAI API, so existing client wrappers can connect to it without modification. Exact prefix reuse is claimed: the sequence continuation when caching matches the full recomputation character-for-character. Benchmarks compare two models with identical GGUF files: Gemma 4 E4B-it in Q3_K_M quantization and Qwen2.5-0.5B-Instruct in Q4_K_M. On an RTX 3050 8GB, medians of three runs yield 53.6 tok/s vs 44.8 tok/s for llama.cpp for Gemma 4 E4B and 345.9 tok/s vs 306.9 tok/s for Qwen2.5-0.5B.
Context
Usually, local inference speed is achieved through hand-written computational kernels: they are written and manually adapted for each graphics architecture, which sets the pace of development for llama.cpp and similar runtimes. SonicBoom moves this work to a higher level: a graph scheduler merges adjacent operations and writes the entire token generation step into a single CUDA Graph, while an MLIR-based compiler generates part of the CUDA kernels, including attention and bf16 GEMM, for a specific card, eliminating manual adaptation for each new NVIDIA architecture. Compiler-generated kernels in open local inference runtimes have been rare until now, so the project attracts attention even with a modest percentage gain. This also includes the guarantee of determinism with prefix cache: such strict reproducibility of results is usually not claimed by runtimes, and it is precisely this that should be checked first when getting acquainted with the project.
Why this matters for the industry
For the industry, the signal is broader than specific numbers: local inference gains can be achieved at a level above computational kernels — the graph scheduler and compilation — and not only through new hand-written kernels. If independent checks confirm the win on a wider class of cards and models, runtimes with hand-written kernels will have to reconsider their optimization stack, and compiler-generated kernels for specific hardware will likely move into mainstream projects. The second message is packaging: a fully open and auditable stack without Python dependencies is suitable for air-gapped and enterprise deployments where control over every execution level is critical. Finally, a fresh independent engine strengthens competition among tools for local model launching, where decoding speed has long been a key argument for choice.
Why this matters for users
It makes sense to try SonicBoom if you have an NVIDIA card from the RTX 30 series: only this architecture has been verified so far. The project builds and runs locally without Python, and both tested models are small — in tests, they were run on a budget RTX 3050 with 8GB of video memory. After building, you can start a local server, connect an existing prototype to it via an OpenAI-compatible endpoint, and measure real speed on your hardware using the tools/bench.sh script. A separate niche is tasks where a reproducible result is needed: thanks to the determinism guarantee with prefix cache, it is suitable for structured extraction tasks and regression checks of output. For everyday work, the runtime is not yet suitable: only greedy sampling is supported, and it is not possible to configure generation parameters like temperature or top-k.
What is still unknown / limitations
Benchmarks are currently weak by reproducibility standards: two model families, three runs with median, one RTX 3050 8GB card, and the same GGUF files. Comparison only with llama.cpp with identical quantized files is a correct control of the numerical base, but there are no measurements against vLLM and TensorRT-LLM, nor data on prefill, memory consumption, and batching, so the project's overall position among runtimes cannot be assessed. The claimed win was obtained only on the sm_86 architecture: it is unknown whether it will be preserved on sm_89 and newer, on larger models, and under parallel load. The conclusion that hand-written quantized kernels are no longer a defensible advantage is not justified by current data: for now, this is a testable hypothesis that must undergo third-party measurements. The project itself is very young: it is still version v0.0.3 with eighteen stars on GitHub.
Sources
Author
Look at AI, editorial team