Base Compute, the developer of the local inference engine BaseRT, released the September edition of its monthly State of Local AI Report — a practical map of which AI models actually work on local hardware, on exactly what hardware, and at what speed. At the center of this issue is the new local all-rounder Qwen3.8-27B, the agentic Meta Muse Glimmer 30B, open GLM-5.3 weights, and the Mac Studio M5 with Thunderbolt 5 RDMA clustering. The report reduces model selection to three physical constraints and sets speed thresholds for typical tasks.

image

What happened

The September issue of the State of Local AI Report gathered the month's main events in local inference. The new all-rounder for local deployment is named Qwen3.8-27B: a dense model under the Apache 2.0 license with built-in vision-language, 262K context, and weights of about 17 GB in 4-bit quantization. Meta Muse Glimmer 30B is recommended for agentic tool-calling loops. GLM-5.3 weights appeared on Hugging Face on August 28 under a bespoke license with a revenue threshold, while GLM-5.3-Flash, an open MoE with 320B total and 18B active parameters under the MIT license, was published on August 26. The main hardware event is the Mac Studio M5 Max/Ultra, announced on August 25 with shipments starting September 22: M5 Ultra memory bandwidth increased from 819 GB/s to 1.2 TB/s, and Apple claims up to 3x inference gains on a cluster of four machines connected via Thunderbolt 5 RDMA. The report also sets speed thresholds by task: 10–30 tok/s for chat, 30+ for coding, and 50+ for agentic scenarios.

Context

The methodological value of the report lies in its model selection framework. Local deployment reduces to three physical constraints: memory capacity determines whether the model fits, memory bandwidth sets token generation speed, and compute power sets prompt processing speed. On this framework rests the main shift of this issue: the class of 27–31B dense models in 4-bit quantization is displacing 70B dense as the working standard on a single 24 GB memory card, explained by improved quantization quality and the VRAM ceiling. The second shift is open MoEs like GLM-5.3-Flash and gpt-oss-120b, making frontier quality achievable on 128–192 GB unified memory: the math works out, GLM-5.3-Flash in 3-bit takes about 120 GB and fits in a 128 GB Mac Studio. However, the source is a local inference vendor: Base Compute sells the BaseRT engine, so the report is better read as a practical map rather than a scientific paper.

Why this matters for the industry

For the industry, the report marks the transformation of local inference from a hobby experiment into a planned product platform: the combination of model, hardware, and expected tok/s becomes a calculated task rather than a matter of taste. The 27–31B dense class on a single 24 GB card makes pilots cheaper, and the emergence of open MoEs that only need 128–192 GB unified memory makes self-hosting a cost-effective alternative to cloud APIs — the decision increasingly comes down to token cost, data requirements, and operational readiness. For startups, this is simultaneously a squeeze on pure API-wrapper margins and an opening of niches in private, edge, and on-prem inference. If monthly updates accumulate a reproducible methodology with fixed configurations, the report could become the de facto benchmark for local inference and create pressure on model publishers to publish tok/s on standard configurations; the likely market response is a growth in dense releases, products with local-first defaults, and tools for automatic model selection for the user's machine.

Why this matters for users

For readers, the report is a working reference for building local AI. Using the tables, you can pick a model for a task and hardware for a model: from Gemma 4 E4B, which runs in 4 GB RAM, to GLM-5.3-Flash, which needs a Mac Studio with 128 GB unified memory, with realistic tok/s and prices across six levels — from a laptop to an on-prem server costing $150,000 and above. The stated speed thresholds help match a configuration to a task: chat needs the lower bound, coding requires a noticeably higher rate, and agentic loops need even more. The practical step today is to build a prototype on Qwen3.8-27B: the model weights in 4-bit quantization fit on a single 24 GB memory card, and vision-language and 262K context cover most typical scenarios. The report is updated monthly, so it can be bookmarked to track dynamics from issue to issue.

What is still unknown / limitations

The report's key figures are vendor-provided, and before production decisions they should be verified against official model cards and your own benchmark on your hardware. The claim that Qwen3.8-27B improves Qwen3.6-27B's result by 77.2% on SWE-bench Verified was published without methodology: the agentic wrapper, number of runs, and decoding settings are unknown, and SWE-bench results depend heavily on scaffolding. The report's measurement methodology is not disclosed: batch size, context lengths, and quantization kernels are not specified, so tok/s figures cannot be directly reproduced. Apple's phrasing "up to 3x inference on four machines" does not clarify whether this refers to aggregated throughput or single-request latency, and what the scaling efficiency is as the number of nodes grows; distributed inference over a consumer interconnect has non-trivial costs. The question of MoE quality under aggressive 3-bit quantization also remains open. The first independent Mac Studio M5 measurements will only appear after shipments begin.

Sources

Author

Look at AI, editorial team