Local model landscape

Living document. Rewritten as new models ship. Last updated: 2026-08-11.

Two-clock status (2026-08-11) — both moved, opposite directions. Open: Meta shipped Muse Glimmer (2026-08-10) — the first open mover in ~a month that both runs on the reference fleet and beats the incumbents in its size class. 30B (2B vision encoder + 28B text decoder), Apache 2.0, multimodal (text/image/video), agentic (multimodal tool calling, self-quantization/deployment). Vendor benchmarks SWE-Bench Pro 51.2 / MCP Atlas 75.5 / Charxiv 78.8 — reportedly beats Gemma4-31B and Qwen3.6-27B (both in active use) on SWE-Bench Pro + MCP Atlas. At Q4_K_M (~18GB) it fits the M3 Max/M2 Max inference budget and offloads on the 3060 box; day-0 GGUF (unsloth quant already trending), llama.cpp/Transformers/vLLM. This is a recommendation-changer for the 27–31B local band — it earns a real eval, not a note. Verify-don’t-trust: benchmarks vendor-reported, unreproduced; the Gemma4/Qwen3.6 beat needs independent SWE-Bench Pro confirmation before displacing the incumbent pick. Two smaller open ticks alongside: NVIDIA Magpie TTS (364M open multilingual voice, sub-second latency — voice-agent infra going open) and LiquidAI LFM2.5-2.6B (tiny, M3-Max-class). Closed: OpenAI shipped GPT-5.6-Cyber (2026-08-10) into Daybreak Red — a gated, purpose-built cyber model behind identity verification / attestation / partner-only access; GPT-5.6 Sol went to Daybreak Blue (approved defenders). Hosted-only, local relevance nil; tracked as the closed clock’s gate-and-distribute posture (fail-closed matured from the 08-07 Astra brake into a distribution channel). See reports/2026-08-11-the-gate-and-the-open-door.md.

Two-clock status (2026-07-30) — both settled. Closed: no new weights. Anthropic index unchanged (Opus 5 / Sonnet 5 tracked); OpenAI’s two 07-29 posts (“GPT-5.6 fuses frontier intelligence with efficiency,” “two settings tripled ARC-AGI-3”) are explainer content about the July-9 GPT-5.6 launch, not a new model — the ARC-AGI-3 note (Sol 13.3%→38.3% via retained-reasoning + compaction, ~6× fewer tokens) is a harness-config finding, not a weights move. Open: no new base. HF trending is derivative churn — unsloth/Kimi-K3-GGUF quants (K3 weights DID land 07-27, correcting the “escrowed” framing in the Kimi block below), Qwen3.6-35B-A3B variants (family tracked since April — not a bump), GLM-5.2, Inkling. Unfamiliar-but-non-local names appeared (poolside/Laguna-S-2.1 118B, upstage/Solar-Open2-250B, Kwaipilot/KAT-Coder-V2.5-Dev 35B) — none change a reference-hardware recommendation. Frame-check saves: two would-be phantom capability events (Qwen3.6 “new,” GPT-5.6 “launched today”) caught by date/version discipline. See reports/2026-07-30-the-version-manager-grows-a-build-graph.md.

Reference hardware

Three representative consumer profiles, used as a baseline for the fit recommendations below. They span the common local-inference envelope: high-bandwidth Apple Silicon for fast small-model fleets, and a consumer NVIDIA card for GPU-offload of larger models.

Preferences

• Abliterated/uncensored variants preferred — no alignment tax
• Key producers: huihui-ai (Ollama + HF), mlabonne (HF), bartowski (GGUF quants), DavidAU (HERETIC method)
• Inference: Ollama 0.19+ (MLX backend — 57% faster prefill, 93% faster decode vs 0.18)

Cloud model context (benchmarks for reference)

Claude Opus 5 (July 24/25) — new Anthropic Opus generation, shipped as the default Opus in Claude Code v2.1.219 with its own newsroom post (/news/claude-opus-5, alongside /news/claude-sonnet-5). 1M context; fast mode at $10/$50 per Mtok (Opus 4.7 retired from fast mode; /fast now applies to Opus 5 + Opus 4.8). The claude-api skill’s default moved to Opus 5 with an Opus-4.8 migration path. Why it matters for the two-clocks read: the closed weights clock moved after ~5 quiet days — a genuine generation step, not a repackage — but it shipped already productized for latency ($10/$50 fast tier on day one), fusing the “labs compete on throughput” read with an actual intelligence step rather than falsifying it. The June export-control recall also appears to be resolving: /news/redeploying-fable-5 is live in the same index (a reappearance — the mirror of a recall). Local relevance nil (hosted-only); the significance is the collapsing gap between “new frontier model” and “cheap fast-tier access to it,” which raises the bar open weights must clear while leaving the cost/control value proposition intact. See reports/2026-07-25-the-plateau-steps.md.

OpenAI GPT-5.6 — Sol / Terra / Luna (June 26, limited preview, NOT GA) — OpenAI’s next-generation family, gov-gated to ~20 approved companies (GA “in coming weeks”). Sol = flagship (“strongest model yet”); Terra = “competitive with GPT-5.5 while 2× cheaper”; Luna = fast/cheap. New “max reasoning effort” tier + “ultra mode” (subagents to accelerate complex work); improved agentic coding/biology/cybersecurity; “most robust safety stack to date.” Local relevance nil (hosted-only, and not even GA). Why it’s tracked as cloud context: it is the closed clock’s announce-don’t-ship move — a vapor-tier preview (alongside the never-GA’d Gemini 3.5 Pro) whose Terra tier is a direct price-answer to the open free-weight flood (GLM-5.2 made coding intelligence free; OpenAI responds by halving its own commodity-tier price and gating the frontier behind a government list). Same governability leash as Fable/Mythos. The local-inference consequence is indirect: the closed labs are competing on price + gated access + presence, not on shippable frontier capability — which keeps the open-weight value proposition (cost/control/run-it-local) intact and arguably strengthens it. See reports/2026-06-29-the-free-weight-and-the-trapped-context.md and radar/signals/2026-06-26-previewing-gpt-5-6-sol-a-next-generation-model.md.

Claude Fable 5 / Mythos 5 (June 9) — new Anthropic frontier ceiling, replacing Opus 4.8 as the GA top. One set of weights, two names: Fable 5 (general, safeguarded) and Mythos 5 (ungated, Glasswing/bio-research partners only). SOTA “on nearly all tested benchmarks”; higher than Opus 4.8 on FrontierCode even at medium effort; new vision SOTA; “millions of tokens” long-context focus. Pricing $10/$50 (2× regular Opus 4.8). Safety is a routing layer, not a weights property: cyber/bio-chem/distillation queries classifier-fall-back to Opus 4.8 — the n-1 frontier is now the next one’s safety floor. Local-inference consequence: the bar TurboQuant-class compression has to clear on consumer hardware just moved up again, and the frontier↔local capability gap widened — but the governance gap (per-query capability gating) is something open weights structurally cannot copy, which keeps the open-weight value proposition on cost/control rather than frontier parity. Local relevance otherwise low (hosted-only). Full analysis: reports/2026-06-10-the-fable-and-the-fallback.md.

North Mini Code (Cohere, June 9) — Cohere’s first developer model and a new open-weight coding contributor: 30B MoE / 3B active, Apache 2.0 (bf16+fp8), 128K context, trained across multiple agent scaffolds for harness-robustness. 80.2% pass@10 SWE-Bench Verified, 55.1% pass@10 Terminal-Bench v2; positioned above similarly-sized Qwen3.5/Gemma 4/Devstral Small 2 and larger Nemotron 3 Super/Mistral Small 4/Devstral 2. 3B-active = sub-agent/worker economics (cf. Mellum 2); 30B at fp8 (~30GB) exceeds Mac budgets, Q4 (~15–16GB) fits. Candidate local coding worker. See radar/signals/2026-06-09-introducing-north-mini-code-cohere-s-first-model-for-develop.md.

Gemma 4 12B (June 3) — new encoder-free, native-audio mid-tier in a tracked family; full detail + hardware fit in models/families/gemma/README.md. Tail (06-21): huihui-ai/Huihui-gemma-4-12B-coder-fable5-composer2.5-v1-abliterated — a tracked abliterator (huihui-ai, RG’s preferred register) wrapping the trending gemma-4-12B-coder-fable5 coding tune; 12B fits the M3 Max (~21–24GB budget). A derivative of the open clock’s settling tail (no new open base since GLM-5.2, 06-16), not a frontier move — the abliterated/run-it-local surface maturing around the existing Gemma-4-coder / GLM-5.2 bases. Tail (06-23, settling day 4): empero-ai/Qwythos-9B-Claude-Mythos-5-1M (+GGUF) — a 9B, 1M-context open model named after the recalled Mythos 5; 9B fits the M3 Max comfortably, a small-model tail object (not a head). And HauhauCS/Qwen3.6-35B-A3B-Uncensored (“Aggressive”) — MoE 3B-active in the tracked HauhauCS abliteration register, fits M3 Max. Both confirm the open layer keeps naming targets after recalled closed models while shipping no new base. Tail (06-24, settling day 8): yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF and …-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF — GGUF quants of the already-tracked gemma-4-fable5 coder/agentic tunes; the local-run race reaching the Gemma-4-coder tail (mirrors GLM-5.2’s GGUF/FP8 spread). Quant-of-a-tune, not a new base — open clock still settled, eighth straight day.

Qwen-AgentWorld-35B-A3B (June 25) — a new open base on a new axis: the agent’s world, not the agent. Official Qwen, Apache 2.0, finetune of Qwen3.5-35B-A3B-Base, MoE 3B-active (~20GB Q4, fits M3 Max), 262K context (≥128K recommended for multi-turn). Self-described as “the first language world model to cover seven agent interaction domains within a single model” — tool calling, search, terminal, software engineering, Android, web, OS. It does not act as an agent; it predicts environment state transitions in response to agent actions (extended chain-of-thought, trained via continual pre-training + SFT + RL). 56.39 on AgentWorldBench open-ended eval, “competitive with leading proprietary models” (vendor-reported; arxiv:2606.24597, dataset:Qwen/AgentWorldBench). Why it matters: this is the first genuinely-new open base since GLM-5.2 (06-16) — the coding clock stayed settled eight days, but the open layer opened a new front. It’s infrastructure for making and evaluating agents — a downloadable simulator you can host locally to generate/score agent trajectories instead of paying an API. The missing half of a local agent-eval loop. Verify before any adoption claim — vendor benchmark only, and world-model quality is hard to read off a leaderboard. See reports/2026-06-25-the-form-and-the-world.md. Update (06-26, day 2 of traction): AgentWorld is now trending with real downloads (~13k) — community pickup validates it as more than a one-off, but it’s still the only world-model; no second lab has joined the front. Watch whether world-modeling becomes a lane. Update (06-29, day ~4): an unsloth/Qwen-AgentWorld-35B-A3B-GGUF quant is now trending alongside the base — the local-run race has reached the world-model (the GGUF/FP8 pattern that followed GLM-5.2), confirming sustained pickup. Still the only world-model; no second lab.

Tail (06-26, coding settled day 9) — two new open non-coding bases (verify-don’t-trust, both kept off the coding-frontier slot): deepreinforce-ai/Ornith-1.0 (35B + 9B GGUF, MIT, 06-25) — generic text-generation/conversational tags, no code/agent specialization, no benchmark claim; a real new open general base that fits the M3 Max at Q4 (~20GB) but does not displace the GLM-5.2 coding head. LiquidAI/LFM2.5-230M (06-24) — a tiny multilingual edge model (lfm2.5/edge tags, en/ar/zh/fr/de/ja/ko), on-device tier, fits anything including phones; new base from the LFM2.5 line, niche by design. Both surfaced by list-don’t-query and killed as frontier candidates by the verify step — logged as tail, not head. The coding clock stays settled (GLM-5.2, day 9).

Architecture-experiment lane (06-29) — nvidia/Nemotron-TwoTower-30B-A3B-Base: surfaced trending and unfamiliar; tags two-tower + diffusion + mamba — a hybrid non-transformer base, MoE 3B-active (fits M3 Max at Q4), NVIDIA open-model license, multilingual, trained on the Nemotron-Pretraining-Code/CC datasets, arxiv:2606.26493. Created 04-11, resurfaced via a 06-26 checkpoint update (hence the trending slot). Verify-don’t-trust: it is a Base model, no instruct/coding specialization and no coding benchmark — it does NOT displace the GLM-5.2 coding head. What it is: the second entry in the architecture-experiment lane after DiffusionGemma-26B-A4B (06-18) — open labs trying diffusion/state-space (Mamba) decoding for parallel-draft speed rather than chasing the coding frontier. Logged as a hardware-relevant architectural curiosity (a runnable non-transformer base on the M3 Max), sized to evidence. Watch whether this lane ever produces a coding-capable diffusion/mamba head, or stays generic-base experiments. Same window also surfaced DeepSeek-V4-Flash-DSpark (fp8/8-bit quant of DeepSeek-V4-Flash) and a Chunjiang-Intelligence/DeepSeek-v4-Fable tune — derivatives of the DeepSeek-V4 base, not new bases.

Second open coding base — tencent/Hy3 (surfaced 07-06): the first candidate the createdAt check couldn’t demote in ~20 days of a “settled at GLM-5.2” read. 295B total / 21B active MoE, Apache-2.0, 256K ctx, fused fast/slow-thinking reasoning+agent model — Tencent’s first major release after rebuilding its pre-training/RL infra in early 2026. Vendor coding/agent benchmarks: SWE-bench Verified 74.4, Terminal-Bench 2.0 54.4, BrowseComp 67.1, WideSearch 70.2 — “competitive with GLM-5, closing on Opus 4.6 / GPT-5.4.” Verify-don’t-trust: vendor-reported only, no independent SWE-bench confirmation. Not brand-new — newly-seen: Hy3-preview open-sourced April 23 (pre-GLM-5.2); the canonical tencent/Hy3 repo is dated 07-02. The distinction from GLM-5.2 is the axis: not raw scale (753B) but serving efficiency (21B active ≈ GLM-5.2-class coding at ~1/36th the active params → expert-parallel vLLM economics). Local fit (ref HW): ❌ — 295B total is >150GB even at Q4, exceeds all three profiles; cloud/heavy-quant only, same verdict as GLM-5.2. The open coding tier now has two heads on two axes. See reports/2026-07-06-the-settled-clock-had-a-second-hand.md and radar/signals/2026-07-06-tencent-hy3-open-coding-moe.md.

Not two heads — a wave (07-07): the createdAt check fired positive a second consecutive day, which kills the “settled” read entirely. meituan-longcat/LongCat-2.0 (createdAt 07-05, 2 days old, unlogged until now): 1.6T total / ~48B active MoE, 1M context, MIT. Vendor benchmarks SWE-bench Pro 59.5, Terminal-Bench 2.1 70.8, strong agentic (FORTE 73.2, RWSearch 78.8). Its card states it is “deeply integrated with mainstream harnesses such as Claude Code, OpenClaw, and Hermes” — the open models now ship harness integration as a launch feature. Positioned general-purpose, coding+agentic headline. Verify-don’t-trust: vendor-only benchmarks; 385 downloads in 2 days = community-unvalidated. Local fit (ref HW): ❌ — 1.6T total is cloud/datacenter-only at any quant. Alongside it, a miss the settled frame walked past four times: deepseek-ai/DeepSeek-V4-Pro-DSpark (createdAt 06-27, 889B, MIT, 15.5k downloads) — a scale-axis base older than Hy3, never logged. The scale axis alone now reads GLM-5.2 (753B) → DeepSeek-V4-Pro (889B) → LongCat-2.0 (1.6T) inside three weeks, all MIT. The open coding tier is not a head; it is a wave. See reports/2026-07-07-the-harness-hardens-the-models-flood.md.

A marquee lab debuts open — thinkingmachines/Inkling (07-15, HF createdAt 07-14): the first genuinely-new open coding/agentic base since GLM-5.2 (06-16) / Hy3 (07-02), and the strategic shape matters more than the number. Thinking Machines Lab — Mira Murati’s startup (ex-OpenAI CTO + research leadership, among the largest seed rounds ever) shipped its first public model fully OPEN-WEIGHT under Apache-2.0. 975B total / 41B active MoE, 66-layer decoder-only, hybrid local/global attention, 1M context, 45T-token multimodal (text/image/audio/video). Vendor self-reported: SWEBench Verified 77.6%, HLE 46.0 (w/ tools), MMMU 73.5, VoiceBench 91.4; day-one Inkling-NVFP4 + unsloth/inkling-GGUF; card explicitly names “coding assistants and agentic tool-use applications.” Artificial Analysis: “the leading US open-weights model.” Verify-don’t-trust: every benchmark is vendor-only and unreproduced — treat SWEBench 77.6 as a claim. Local fit (ref HW): ❌ — 975B total is >150GB even at Q4, exceeds all three profiles; cloud/heavy-quant tier, same verdict as GLM-5.2 (41B active is cheap to serve, but the full weight file is a datacenter download). Why it matters beyond the leaderboard: a would-be-closed frontier lab chose open-weight debut over hosted product — the open tier absorbed a lab that could have been closed. The open coding tier now has three bases on three axes: GLM-5.2 (scale/China/MIT), Hy3 (21B-active efficiency/China/Apache), Inkling (multimodal/US/Apache, ex-frontier-lab leadership) — and a US lab is on the open coding frontier for the first time. The “no-borders open weight” posture (GLM-5.2’s MIT framing vs the Fable export-control leash) is now trans-Pacific, not a Chinese-lab tactic. See reports/2026-07-18-open-at-the-frontier-closed-at-the-door.md. Watch: independent SWEBench reproduction; whether a second marquee lab open-debuts (Mistral’s early-access MoE the live candidate); whether open-debut is Thinking Machines’ standing posture or a launch tactic. [Update 2026-07-19 — the haircut] Independent context (Sebastian Raschka’s architecture notes, BenchLM, VentureBeat) lands: the 77.6% SWEBench Verified is harness-advantaged — Thinking Machines ran Inkling on a bash-only harness while the comparison figures are self-reported by competing models’ developers (not apples-to-apples). On coding Inkling sits behind DeepSeek V4 Pro and GLM-5.2, ahead of Nemotron 3 Ultra. Corrected read: Inkling is the leading US open-weight, not the open coding frontier — that seat is still GLM-5.2 (unmoved ~33 days). Its genuine differentiators are multimodal breadth and an explicit uncensored / “censorship-resistant” posture, not a coding-capability lead.

The largest open base ever, escrowed — moonshotai/Kimi K3 (announced 07-16, open weights promised 07-27): Moonshot AI launched Kimi K3 via app + API on 07-16: ~2.8T total parameters, MoE with 16-of-896 experts active per token (~1.8%, ≈50B active) under a “Stable LatentMoE” framework, 1M context, multimodal (text/image/video), ~2.5× K2’s scaling efficiency. VentureBeat: “the largest open-source model ever”; Simon Willison covered it day-one; Moonshot paused new K3 subscriptions 07-20 under demand surge. The catch is the whole point: it is live but not yet open. Weights are promised by 07-27 — right now K3 is API-and-benchmarks, the artifact escrowed for a week. A self-imposed dated escrow (demand management, not a government recall like Fable), but structurally the same announce-then-ship shape as the GPT-5.6 preview→GA gate: “available” means announced, not downloadable. Local fit (ref HW): ❌❌ — 2.8T total at 4-bit ≈ 1.4TB; even an aggressive 1.5-bit community quant lands ~500GB, nowhere near an M3 Max’s 36GB. The thread this crystallizes — open ≠ local, increasingly: the open coding/agentic frontier is scaling GLM-5.2 (753B) → Inkling (975B) → Kimi K3 (2.8T); active params stay efficient (~40–50B across all three) but total params — the disk-and-RAM constraint — climb into cloud-only territory. The open-weight and locally-runnable axes are diverging: the frontier stays open (you may download it) while ceasing to be local (you can’t run it). RG’s machines get the derivatives and the sub-30B bases (Hy3’s 21B-active tier, Agents-A1’s 35B, the abliterated small heads), not the open frontier itself. “Open” is decoupling from “yours to run.” Verify-don’t-trust: benchmarks are lab-reported; the 07-27 weight date is a promise (could slip like Gemini 3.5 Pro). See reports/2026-07-20-everything-ships-next-week.md. Watch: does 07-27 land (weights downloadable) or join the escrow tail; community MXFP4/GGUF quant sizes; whether K3 contests GLM-5.2 as the open coding head once weights exist (cloud-tier for RG regardless).

GPT-5.5 “Spud” (April 23) — first fully retrained base since GPT-4.5. 1M context (API). Natively omnimodal.

Pricing: Standard $5/$30, Pro $30/$180 per 1M tokens. The frontier is now a surface, not a point — no single model wins every benchmark. Local models compete on cost at the expense of benchmark position.

DeepSeek V4 (April 23-24) — MIT license, largest open-weight model, radical efficiency gains:

V4-Pro #1 open on Vibe Code Bench. Flash $0.14/$0.28, Pro $1.74/$3.48 per 1M tokens — 36-107x cheaper than GPT-5.5. Not viable for local inference (too large), but the CSA/HCA attention compression architecture will propagate to smaller models. When it reaches 7-27B scale, local agents get dramatically longer context without hardware upgrades.

The open-weight flood (sustained — 2026-06-18 → 06-19)

While the closed Western frontier sits frozen (Fable 5 recalled/export-controlled, Gemini 3.5 Pro never GA’d — only n‑1 gemini-3.1-pro GA-listed; no new closed weights since June 9), the open layer is in a release storm. The HuggingFace trending index — listed, not queried, per the corrected model-check discipline — surfaced a dense open field on 2026-06-18, and it sustained into 06-19 (NOT a one-day snapshot artifact — yesterday’s named falsifier is retired). Three tracked families (Gemma, Qwen, Kimi) had new entries trending simultaneously; two in the abliterated/uncensored register. Day-2 deltas: GLM-5.2 now shipped as GGUF + FP8 quants (the local-run race began within a day — community is racing to run the weights, not admire the benchmark); DeepSeek-V4 now trending as both Pro and Flash; diffusionGemma-26B-A4B gained an NVIDIA NVFP4 quant; and a new uncensored cyber-pentester agent fine-tune (RavenX-CyberAgent-Qwen3.6-35B-A3B-…-OpenMythos-Pentester-BugHunter) — the exact dual-use class (analyze-and-fix-code-flaws ↔ offensive cyber) the state export-controlled in Fable/Mythos, now a download. Entries are trending-surface observations — verify before adopting; only GLM-5.2 is independently benchmarked so far.

Also trending (not RG-hardware-relevant, landscape signal): DeepSeek-V4-Pro, MiniMax-M3, Cohere North-Mini-Code (a coding variant of the North line above), and prefeitura-rio/Rio-3.5-Open-397B — a 397B open model released by the city government of Rio de Janeiro (sovereignty signal: governments shipping open weights they can’t be made to recall).

The capability read: GLM-5.2’s within-a-point-of-Opus-4.8 coding scores (independently confirmed) cross a threshold for the local-sovereign path — “open weights aren’t good enough for real coding yet” is harder to defend this week. The asymmetry that makes this matter: the closed frontier can be frozen (it has an endpoint a government or vendor can hold); the open frontier cannot (you can’t recall a weight file), so the closed freeze accelerates the open flood. Governability is now a flood axis, not just a moat/leash. The 06-19 sustain adds a sharper read: the open layer is naming its targets after the recalled closed model (coding tunes named …-fable5-…; a cyber agent carrying OpenMythos) — you can export-control an endpoint, not a name or an imitation. The GGUF/FP8 quants arriving in a day mean the runnable open wins for RG hardware are the Gemma-4-12B coder tunes + Qwen3.6-A3B MoEs (GLM-5.2 at 753B remains > 36GB even at Q4). See reports/2026-06-18-the-clock-with-no-off-switch.md and reports/2026-06-19-fences-for-a-flood.md.

What just shipped

Mellum 2 (June 1, 2026) — JetBrains’ worker-tier coding MoE

JetBrains’ model card positions it not as a flagship coder but for routing/orchestration in multi-model systems and sub-agent tasks (planning, validation, transformation), RAG context compression, and private-code deployment — i.e. the worker slot in an agent fleet. ~2× faster inference than comparable dense models from the low active-param count. Context length not stated in the launch post (arXiv 2605.31268 + model card to confirm).

Hardware fit (the low active-param count is the story — fully GPU-resident on all three):

• M3 Max 36GB: Q4 ~7GB — comfortable; can host several parallel instances, high tok/s
• M2 Max 32GB: Q4 ~7GB — ideal dispatch worker for batch sub-agent jobs
• RTX 3060 12GB: Q4/Q5 ~7–8GB — fully in VRAM, no CPU offload; Q8 (~13GB) spills, stay at Q4/Q5

Significance / recommendation change: for a local sub-agent / code-completion worker role, Mellum 2 (Q4/Q5) is now the model to reach for on the 3060 box — the rare 12B that stays GPU-resident on a 12GB card because only 2.5B params activate, fast enough to run fanned-out rather than as a single assistant. On the Macs it’s a strong dispatch-worker default. Lands in the slot the orchestration layer is creating: as fleet-of-cheap-workers-under-one-planner becomes the dominant pattern (Opus 4.8 Dynamic Workflows; 60%+ of Codex users run parallel tasks), an open, fast, code-specialized local worker is the piece an open-source agent stack was missing.

Huihui4-8B-A4B-v2 (April 27, 2026) — Expert-pruned Gemma 4 variant

Expert pruning (128 → 32 experts) + SFT. Uses GLM-5.1 thinking mode format. INT4/INT8: 6-9GB VRAM. Cross-architecture lineage: Google model base, Chinese training data + reasoning format.

Hardware fit:

• M3 Max 36GB: INT4 ~6GB — comfortable, room for multi-model fleet
• M2 Max 32GB: INT4 ~6GB — comfortable
• RTX 3060: INT4 ~6GB — fits entirely in VRAM

Significance: huihui-ai’s first technique beyond abliteration. Expert pruning restructures the model rather than removing safety guardrails. The v2 suffix indicates iterative refinement. At 6-9GB, this is the smallest Gemma 4-based coding-capable model — evaluate against Gemma 4 E4B (4.5B active, 9.6GB Ollama) to see if pruning preserves coding quality.

Qwen3.6-27B Dense (April 22, 2026) — Apache 2.0, most important local model release since Gemma 4

Dense model (not MoE). “Thinking Preservation” mechanism. Outperforms the 397B MoE Qwen3.6 on agentic coding benchmarks — 14x smaller, better at the specific task. Unsloth MLX quants (4/6/8-bit) available same day.

Hardware fit:

• M3 Max 36GB: Q4_K_M (~15 GB) — fits with ~7GB headroom. Primary coding model candidate.
• M2 Max 32GB: Q4_K_M (~15 GB) — fits with ~7GB headroom. Dispatch upgrade.
• RTX 3060: Does not fit in 12GB VRAM. CPU offload with 64GB RAM possible but slow.

Priority evaluation. If the agentic coding benchmarks hold on practical tasks, this replaces the MoE models as the recommended local coding model for Apple Silicon. Dense architecture = more predictable inference, no routing overhead.

Kimi K2.6 (April 20, 2026) — Modified MIT, agent swarm architecture

First model designed for massive multi-agent orchestration: 300 sub-agents, 4,000 coordinated steps. SWE-Bench Pro 58.6 beats GPT-5.4 (57.7) and Opus 4.6 (57.3). Too large for local at full scale. Watch for distilled variants targeting the 32B active param slice.

Qwen3.6-35B-A3B (April 15-16, 2026) — Apache 2.0, MoE variant

Terminal-Bench 2.0: 51.5. Best open-weight agentic coding MoE at this parameter count. huihui-ai shipped abliterated + Claude-named variants. Unsloth Dynamic 2.0 + bartowski imatrix available.

Hardware fit:

• M3 Max 36GB: Q4_K_M (~18-19 GB) — fits but tight
• M2 Max 32GB: Q3_K_M (~15 GB) safer
• RTX 3060: does not fit. CPU+GPU split viable for batch.

Gemma 4 (April 2, 2026) — Apache 2.0 license (major change from Gemma 3’s custom license)

Gemma 4 E2B beats Gemma 3 27B on most benchmarks with only 2.3B active params. Most efficient model per byte I’ve tracked. ~75-85 tok/s on M3 Max.

Abliterated variants expanding — see abliteration section below.

Nemotron 3 Nano — Mamba-Transformer hybrid, benchmarks now available

Independent benchmarks (via NeMo Evaluator):

• AIME 2025: 89.1% (beats Qwen3-30B-A3B at 85.0%; 99.2% with Python tools)
• LiveCodeBench v6: 68.3% (beats Qwen3 66.0% and gpt-oss 61.0%)
• Arena-Hard-v2: 67.7% (vs Qwen3-30B 57.8%, gpt-oss-20b 48.5%)
• RULER: 87.5% at 64K, 82.9% at 128K, 70.6% at 512K (supports 1M context)
• 3.3x throughput vs Qwen3-30B-A3B on single H200

Verdict: At 3.2B active params, this runs on RTX 3060 12GB comfortably. Strong coding/reasoning at tiny active parameter count. Priority recommendation for the 3060 profile. GGUF quants available from Unsloth.

Hardware x Model fit matrix

M3 Max 36GB — Tiny fleet for background tasks

Multi-model strategy: Set OLLAMA_MAX_LOADED_MODELS=4. Example fleet: Qwen3.5-0.8B (1GB) + Gemma 4 E2B (4GB) + SmolLM3-3B (3.5GB) + Qwen3.5-2B (2.7GB) = ~11GB total, plenty of headroom.

M2 Max 32GB — Dispatch workhorse

Avoid: Gemma 4 26B MoE — community reports 11 tok/s vs 60+ for similarly-sized dense models. MoE has higher bandwidth demands per active param.

WSL + 3060 12GB — Heavy compute

Coding-specific models

Abliterated variant sources

Quick Ollama access:

ollama pull huihui_ai/qwen3.5-abliterated       # Qwen 3.5 uncensored
ollama pull huihui_ai/gemma3-abliterated         # Gemma 3 uncensored

gpt-oss-20b abliterated landscape (complete)

All fit comfortably on all three machines. MXFP4 at ~14GB or IQ4_NL at ~11.5GB. HERETIC variant claims complete refusal removal.

Independent benchmarks (via BenchLM, DataRobot, Artificial Analysis):

• Arena-Hard-v2: 48.5% (behind Nemotron 3 Nano at 67.7%)
• LiveCodeBench v6: 61.0% (behind Nemotron 3 Nano at 68.3%)
• Matches or exceeds o3-mini on most benchmarks
• Outperforms gpt-oss-120B on HumanEval and MMLU despite being much smaller
• “Low thinking effort” mode outperforms more expensive competitors
• Fits 16GB devices — runs on RTX 3060 and M3 Max easily

Verdict: Solid general-purpose model but Nemotron 3 Nano beats it on coding benchmarks at similar active params. Best use: general reasoning/chat where abliterated variant is preferred.

Quantization reference

Rule of thumb for Apple Silicon: model should be <=60-70% of total unified memory.

Key insight: TurboQuant — 6x KV cache compression (NEW — April 12)

Google Research’s TurboQuant (March 25, ICLR 2026) compresses KV cache to 3 bits with zero accuracy loss. No retraining required. 6x reduction in KV memory.

Impact on the reference hardware:

• M3 Max 36GB: Gemma 4 31B at full 262K context becomes possible. KV cache drops from ~22GB to ~3.7GB. 31B Q4 (~20GB) + 3.7GB KV = 23.7GB total — fits.
• M2 Max 32GB: Nemotron 30B-A3B and Qwen3.5-27B can serve dramatically longer contexts within existing memory.
• RTX 3060 12GB: Context length multiplied within same VRAM budget. 14B models can run at very long context.

Implementation status:

• Google official: Q2 2026
• llama.cpp: turboquant_plus project, experimental, Metal support on Apple Silicon
• Validated from 1.5B to 104B parameter models

The synthesis: TurboQuant + Ollama 0.19 MLX backend = two multiplicative improvements. MLX accelerates compute, TurboQuant expands context. Together they make Apple Silicon the most improved local inference platform.

Key insight: Unsloth MLX-native Gemma 4 lineup (NEW — April 14)

Unsloth uploaded MLX-native quantizations for the full Gemma 4 family:

Why this matters: MLX-native quants skip GGUF→MLX conversion overhead. Combined with Ollama 0.19’s MLX backend, these are the optimal format for Apple Silicon. The Gemma 4 26B MoE at MLX 4-bit (~17GB) fits M3 Max and M2 Max comfortably. The 31B Dense at MLX 3-bit may also fit within budget.

Updated recommendation: For general-purpose inference on Apple Silicon, prefer Unsloth MLX quants over GGUF when available.

huihui-ai abliteration wave (April 14-16)

• Huihui4-48B-A4B-abliterated (April 16) — experimental expanded architecture: takes Gemma 4 26B-A4B and replaces MLP layers with 256-expert MoE, expanding to 48B total. Not fine-tuned yet. Experimental.
• Huihui3.5-67B-A3B (April 16) — Qwen3.5-35B-A3B base expanded to 512 experts, 67B/3B active MoE
• Gemma 4 E2B, 31B, 26B MoE abliterated v2 — refreshed versions
• Full Qwen3.5, GLM-4.7, Kimi, Mistral-Small-4 lineups now abliterated

DavidAU LFM2 HERETIC series (NEW — April 14-16)

Liquid AI’s LFM2 foundation models (SSM-Transformer hybrids) getting the HERETIC treatment:

• LFM2-12B-A1B High-Intelligence Series-B (April 16)
• LFM2-12B-A1B Deckard-II HERETIC Uncensored (April 16)
• LFM2-8B-A1B Deckard-II HERETIC Uncensored (April 15)
• LFM2-8B-A1B GLM-4.7-Flash Thinking (April 15)
• gemma-4-19B-A4B-it INSTRUCT Heretic-Uncensored (April 15)
• gemma-4-E4B-it Deckard-V2 Strong HERETIC (April 14)

At 1B active params, the LFM2-8B models are extremely efficient — run on all three machines. SSM-Transformer hybrid architecture is distinct from standard transformer; worth evaluating for latency characteristics.

Key insight: Ollama 0.19 MLX backend

Released March 2026. On Apple Silicon: 57% faster prefill, 93% faster decode vs v0.18 (llama.cpp). The M3 Max has higher memory bandwidth than M4 Pro, so it outperforms newer chips for memory-bound inference. Make sure Ollama is updated.

Models NOT practical on the reference hardware

Other models to assess

• DavidAU LFM2-8B-A1B variants: SSM-Transformer hybrid from Liquid AI, HERETIC uncensored. 1B active params. Extremely efficient — fits all three machines. New architecture worth evaluating for latency and agentic task performance.
• DavidAU gemma-4-19B-A4B HERETIC: Compact uncensored Gemma 4 variant. Fits M3 Max and M2 Max at Q4.
• Nemotron 3 Nano 4B: Mamba-Transformer hybrid, claims 5x throughput. Tiny enough for fleet member on M3 Max. Priority evaluation.
• Nemotron 3 Nano 30B-A3B: MoE with only 3B active, Mamba hybrid. Local benchmarks now available: ~77 tok/s M2 Max (MLX), 40-60 tok/s RTX 3060 (Q4, ~5GB VRAM). Fits 3060 comfortably — top priority.
• MiniMax M2.7: “Self-evolving” training. 56.22% SWE-Pro — approaching Claude Opus 4.6. Too large for local but watch for quants/distills.
• Cogito v1 (3B/8B/14B/32B/70B): Dense, hybrid reasoning toggle. Llama/Qwen-base variants. On Ollama.
• Phi-4-mini-reasoning (3.8B): 128K context, reasoning-capable. Worth testing as alternative to SmolLM3.
• Gemma 4 31B HERETIC+Thinking (DavidAU): Chain-of-thought reasoning + uncensored 31B.
• Qwen3-Coder abliterated (huihui-ai): Abliterated variant for the coding model line.
• gpt-oss-20b HERETIC (DavidAU): Claims complete refusal removal. Priority evaluation.
• Gemma 4 31B abliterated GGUF (amarck): Q4_K_M ~19GB, fits M3 Max at short context.

Known issues

• Qwen 3.5 GGUF + Ollama incompatibility: GGUF versions do not work in Ollama due to separate mmproj vision files. Use llama.cpp directly for now.
• Gemma 4 GGUF chat template bug: Community GGUF uploads ship with incorrect chat templates (wrong delimiters), causing ”---” output loops. pmarreck/gemma4-heretical fixes this via Ollama RENDERER/PARSER support.
• Gemma 4 31B flash-attention bug in Ollama: Hangs on prompts over ~500 tokens. Workaround: OLLAMA_FLASH_ATTENTION=0 but tanks speed to ~15 tok/s on Apple Silicon. The 26B MoE is the better pick at ~20-30 tok/s.
• Gemma 4 31B context limits on 36GB Macs: 31B Q4 needs ~20GB weights + ~22GB KV at full 262K context. Only works at short context (<16K) on M3 Max 36GB.

Open threads

• Meta Muse Spark — open-weight contraction: Meta went proprietary. Llama future unclear. The open-weight producers (Google Gemma, Alibaba Qwen, Zhipu GLM, community) become more important. Google’s Apache 2.0 shift for Gemma 4 looks prescient.
• Heretic ARA quality: trohrbaugh’s gemma-4-31b-it-heretic-ara achieves KL divergence 0.012 (virtually no quality loss) while reducing refusals 98→5/100. Current best-quality abliteration for Gemma 4 31B. Needs evaluation.
• Nemotron 3 Nano evaluation: Benchmarks available. AIME 89.1%, LCBv6 68.3%, Arena-Hard 67.7%. At 3.2B active params, top priority for the 3060 profile. Beats Qwen3-30B-A3B and gpt-oss-20b on coding.
• gpt-oss-20b evaluation: Benchmarks available. Arena-Hard 48.5%, LCBv6 61.0%. Solid but Nemotron 3 Nano beats it. Best for general reasoning with abliterated variant.
• Ollama v0.20.5 (April 9): New release. Gemma 4 all sizes available. Check for stability/perf fixes.
• TrevorJS abliteration technique: Biprojection + EGA, cross-validated against 686 prompts. New method worth tracking.
• Qwen 3.6-Plus: API-only (Alibaba Bailian, OpenRouter). 1M context, agentic coding. Watch for local release.
• huihui-ai Huihui4-8B-A4B: New model family uploaded April 25. 8B total/4B active (MoE), image-text-to-text. GGUF variant also available. If original work (not abliteration), marks huihui-ai’s transition to model producer. Fits all three machines easily. Evaluate.
• huihui-ai Qwen3.6-27B abliterated: Uploaded April 23, 539 downloads. Abliterated dense Qwen3.6-27B — the model that outperforms 397B MoE. Combined with Unsloth MLX quants for Apple Silicon deployment.
• DeepSeek V4: Shipped April 23-24. V4-Pro 1.6T/49B active, V4-Flash 284B/13B active. MIT license. CSA/HCA attention compression (27% FLOPs, 10% KV cache vs V3.2). Too large for local. The architecture matters more than the weights — watch for compression techniques propagating to smaller models.
• vllm-mlx: Server framework claiming 400+ tok/s on tiny models, continuous batching, Claude Code compatible.
• OpenClaw community adopting Kimi K2.5: Signal that model preference is shifting away from Claude for agentic work in open-source community.
• GLM-5.1: 744B MoE (40B active), #1 SWE-Bench Pro (58.4), MIT license. Open-weight since April 7 (corrected from “cloud-only”). huihui-ai abliterated GGUF available. MLX community version exists. Too large for local at full scale (~206GB) but distills/aggressive quants may change this. Watch Z.ai for smaller variants.
• Copilot CLI BYOK: Now supports Ollama, vLLM, any OpenAI-compatible endpoint. Local models become usable inside a major agent’s workflow for the first time.