Major Local LLMs in 2026

Overview

Local LLMs are no longer limited to running a chat application on a PC. They are also becoming a foundation for Tool Use—the ability to call external functions such as search and programs—and for Agents that carry out multiple tasks toward a goal. Meta, Liquid AI, NVIDIA, Ornith, Qwen, Google, OpenAI, and Mistral now offer models designed with local inference or downloadable model weights in mind, while Perplexity has released a product that places both the model and its Agent runtime on the local system.

This article introduces major models as of September 1, 2026, and explains how to read specifications such as quantization, which reduces model size and memory requirements, and Context Length, the amount of text a model can reference at once. It also covers an Agent runtime that combines models, tools, context management, and sandboxing, providing an overview for initial selection rather than instructions for a specific product.

K2 Horizon and NVIDIA PAIR were added after checking their primary sources on September 6, 2026. Earlier entries retain their stated dates and have not all been revalidated for this update.

On September 18, 2026, primary sources were checked for Spark-X2.5, Bonsai 2, Edge0, and Iris-mini. These additions distinguish compact Agent models, low-bit compression, SSD expert streaming, and harness-dependent evaluation; they do not revalidate every older entry.

Conclusions

• The October 13 check corrects Flash-Next's existing license summary and adds the Ollama RC MLX memory-retention fix. Separate commercial-use conditions, logical cache usage, and actual allocated memory; earlier entries retain their information dates.
• The October 12 check confirms paused Ollama background conversion, upstream Bonsai Hadamard support for the dev Q2_0 artifact, and conditional Gemma 4 OpenCL decode support. These change evaluation paths, not measured quality or memory requirements.
• The October 11 check adds LLM-jp-4.1 as a Japanese-Agent research candidate and confirms K2 Horizon tensor parallelism re-enablement and MLX decision-model support. MoE caching, loading buffers, disk conversion, and ML Drift affect different resources and execution paths. Smaller files or added support do not guarantee Japanese quality, lower total memory, or concurrent-game performance.
• The October 7 check confirms a non-prerelease Ollama v0.40.0 for evaluating the MLX path, plus upstream llama.cpp support and official quantized GGUF options for K2 Horizon. Pin the runtime, model, and context, and separate availability from measured performance.
• The September 29 check adds MiMo-V2.6-Distill-Qwen-9B as an agentic candidate. Evaluate Transformers packed GGUF inference and Ollama's automatic MLX selection within their supported environments and versions. This does not revalidate every earlier entry.
• Spark-X2.5 expands the compact Agent shortlist. Bonsai 2 compresses weights, while Edge0 streams experts from SSD; their sizes and memory figures describe different things. Iris-mini illustrates why an Agent score belongs to its model and harness together.
• K2 Horizon 3.7B / 7B add candidates for compact local Agents; developer benchmark results do not establish NPC quality or latency on consumer PCs.
• NVIDIA PAIR distributes independent requests across local PCs, expanding concurrent capacity without pooling VRAM.
• Local LLM competition now covers reasoning—the ability to work through intermediate steps before answering—Tool Use, image understanding, and long-running tasks, not just conversation.
• Meta's Muse Glimmer is notable for explicitly targeting multi-step Agent work on consumer PCs.
• Liquid AI's LFM2.5-2.6B combines an official 1.59 GB 4-bit checkpoint with training for Tool Use, making it a compact on-device Agent candidate. QAD and DSpark address quantization loss and generation speed as separate concerns.
• NVIDIA's Nemotron 3.5 Lightning makes a clear distinction between large models that handle complex planning and execution models that repeatedly perform tasks such as Tool Use.
• Ornith 1.5 9B emphasizes Tool Calling and Agentic Coding. Its official Q4_K_M file is 5.63 GB, making it a candidate for evaluation on an 8 GB-class GPU, but that number covers weights only.
• Qwen3.8-Flash-Next previews the efficiency direction planned for Qwen4. Activating about 6B parameters per token, however, does not make it a roughly 6B-scale PC model.
• Mistral's Ministral 3 3B is a compact edge candidate with a 256K context window, Structured Outputs, and Function Calling.
• Runtime support also changes practical viability. LM Studio 0.4.22 added support for DFlash, DSpark, and MTP assistant drafters.
• Perplexity Portable Computer demonstrates that co-designing the model, context management, tools, scheduler, and sandbox can change Agent outcomes even when the underlying model is the same.
• A model name and parameter count alone do not determine memory requirements, speed, or quality.
• The best model depends on the workload and hardware, so shortlisted candidates should be compared on the same tasks and system.

How local LLMs are changing

Earlier local LLM discussions focused on whether a quantized small model could run on a personal computer and generate natural text at all.

The current question is how much of the following can be handled locally:

• Reasoning
• Tool Use
• Inspecting a failed tool result and trying again
• Understanding images and video as well as text
• Referring to long documents and earlier interactions
• Adjusting the balance between reasoning effort and response speed
• Generating output fast enough to sustain multiple tasks
• Assigning complex planning and repetitive Tool Use to different models

Models should therefore be evaluated as part of a complete runtime that includes data compression, inference software, techniques for faster generation, and the harness that manages context and tools.

Evaluate the model and harness together

Portable Computer co-designs the model and harness

Perplexity's Portable Computer is a local-first Agent runtime released on August 25, 2026. At launch, it runs Qwen 3.8 27B or PPLX 27B, a version post-trained for Agent workloads, on NVIDIA DGX Spark. The Linux release is available to Pro and Max subscribers. RTX PC and Windows support was still planned in that initial announcement.

Checked on October 4, 2026. NVIDIA announced Windows availability on September 14 for compatible GeForce RTX / RTX PRO GPUs with at least 24 GB VRAM. AMD announced support on September 24 for supported Ryzen AI Max systems, including Ryzen AI Halo. The current product page lists Pro / Max subscriptions. These expand the hardware evaluation options; our language-quality, latency, and concurrent-game tests remain unperformed. Product availability does not establish checkpoint redistribution rights.

The distinguishing feature is that it does not only run the model locally. Its orchestrator, planner, tool router, scheduler, durable task queue, and local search index also run on the device. Ordinary work and files remain local. Web search, connected services, and stronger cloud models are used only when required and approved by the user. Work completed by the local model does not consume cloud-inference credits.

Perplexity's technical report says that a deterministic orchestrator manages context, the execution loop, and policy while the LLM proposes the next action. Tools execute in an OS-level sandbox. If the sandbox is unavailable, the harness disables itself instead of falling back to unsandboxed execution. A cloud advisor receives only approved context and has no direct access to local files or tools.

On Perplexity's own 53-task Local Knowledge Work Bench, all configurations using Qwen 3.8 27B on NVIDIA DGX Spark scored 82.6% with the Portable Computer harness, 77.6% with Pi, and 74.0% with Hermes. Portable Computer with PPLX 27B reached 85.4%. Each task was evaluated three times. This developer evaluation shows that harness design can affect results with the same model, but the benchmark is an unpublished internal set and does not establish the same advantage for other workloads or hardware.

Perplexity's PPLX checkpoint repository describes a DGX Spark configuration using NVFP4, FP8 KV cache, and DFlash2. However, the repository labels the license as “other” and does not establish redistribution or commercial-product integration terms. Availability inside Portable Computer and permission to redistribute the checkpoint in another product must be evaluated separately.

Iris-mini makes context management part of the evaluation

Checked on September 18, 2026. Iris-mini is a search Agent post-trained from Qwen3.6-35B-A3B, with 35B total and 3B active parameters and a 256K context. The developer reports BrowseComp scores of 64.7 without context management and 82.2 with discard-all, which resets the conversation to the opening question after a prompt threshold. These are developer measurements, not an independent reproduction. The cross-model table uses different harness conditions for its baselines and is not a controlled ranking.

The useful lesson is to evaluate the model, search tools, and context management together. Active parameters do not determine full weight memory, and serving the model locally does not make external web searches offline. This is a research-Agent candidate, not evidence for NPC quality.

NVIDIA PAIR routes inference requests across PCs

Checked on September 6, 2026. NVIDIA PAIR is a beta local inference router that sends independent requests to compatible computers on the same LAN. It exposes Ollama-compatible and OpenAI-compatible endpoints and chooses eligible nodes according to engine availability, model availability, and current workload.

Each request completes on one node. PAIR does not pool GPU memory, shard a model across computers, or split an in-flight request. It is a candidate for concurrent multi-agent workloads; it does not establish that one request will become faster. Evaluate total completion time and queueing separately from single-request latency.

Major models

This is not a comprehensive ranking. The models below are representative examples of different directions in local LLM development in 2026.

The release date is the official release date of the model or major derivative covered here, not the last-modified timestamp of its distribution repository. Month-level precision is used when the official source does not provide a specific day.

MiMo-V2.6-Distill-Qwen-9B

Checked on September 29, 2026. Xiaomi's MiMo-V2.6-Distill-Qwen-9B is an agentic model produced by supervised fine-tuning of Qwen3.5-9B on MiMo-generated data. It adds a candidate for evaluating coding, tool use, and multi-step work.

These are developer evaluations, not independent reproductions or evidence of Japanese NPC quality. Gains vary by task. The official quickstart recommends a recent SGLang build with Qwen3.5 support and the supplied MiMo v2.6 chat template. For another runtime, check template and tool-call handling as well as weight format. Quantized file size alone does not establish practical operation on an 8, 12, or 16 GB GPU; measure context, runtime memory, latency, and concurrent game load.

This model generates dialogue or tool calls, whereas other approaches only select among candidates. The agent harness article compares Kev, SemIf, Laya, and Tev1.

LLM-jp-4.1 8B is a research candidate for Japanese Agents

Checked on October 11, 2026. NII's September 28 announcement releases LLM-jp-4.1 with Tool Calling and official quantized files. The 8B card specifies 8.59B total parameters, 65,536-token context, and Apache 2.0. The official Q4_K_M GGUF is 5.5 GB of weights, not total RAM / VRAM; its repository modification date is not the model release date.

The October 1 prerelease llama.cpp b11320 includes PR #29681, a dedicated Harmony parser for LLM-jp-4.1's spacing and parallel-tool-call delimiters. Use the matching template and tokenizer; ordinary gpt-oss parsing or LLM-jp-4.0 support is not equivalent. Developer parser / schema tests are not our measurements. The card describes research-and-development models not intended for direct production use. Short Japanese tool tasks, reasoning separation, schema validity, completion time, and total memory are evaluation candidates; Japanese success rates, NPC quality, and concurrent-game performance remain unmeasured. Verify exact artifact, runtime, and redistribution terms before adoption.

Spark-X2.5 4B / 1.7B

Checked on September 18, 2026. Spark-X2.5 adds 4B and 1.7B options for compact Agents. The developer's release log dates the model launch to September 1, llama.cpp support to September 6, LM Studio support to September 8, and Ollama support to September 15. These are developer-announced support dates; verify the installed engine, chat template, and Tool Call parser before evaluation.

The developer reports BFCL-v4 65.1 and MCP-Atlas 54.6 for 4B, but all its evaluations use thinking mode. These scores do not establish non-thinking quality, Japanese NPC dialogue, or low response latency. Native context up to 1M tokens is a capacity limit, not a memory budget. Begin with the context and output budget required by the task, and compare intent accuracy and total response time.

K2 Horizon 3.7B / 7B

Checked on September 6, 2026. IFM's K2 Horizon family, released on September 3, adds compact Agent candidates. The 3.7B and 7B model cards describe dense, core-size models with a native 524,288-token context. Core-size labels alone should not be used to estimate full checkpoint or runtime memory.

The developer reports SWE-bench Verified scores of 68.6 and 70.6 respectively. However, its evaluation guidance uses high reasoning effort and allows at least 32,768 output tokens. The 3.7B model scores 50.9 on BFCL v4 versus 55.7 for Qwen3.5-4B, so it is not uniformly ahead in Tool Use. These results support shortlisting, not claims about low-latency NPC dialogue, Japanese quality, or quantized PC performance.

The September 6 check identified BF16 files in the official 3.7B GGUF repository and the IFM fork as an evaluation path. The October 7, 2026 check confirms that the prerelease llama.cpp b11454, published October 6 at 20:20 UTC (October 7 at 05:20 JST), adds K2 Horizon dense / MoVA support, conversion and loading, tokenization, reasoning, and tool calls. This implementation is newer than the model card's still-pending upstream-PR guidance. However, the final code in that build disables tensor parallelism (-sm tensor) for K2 Horizon.

As of October 7, official files include a 3.16 GB 3.7B Q4_K_M and a 5.59 GB 7B Q4_K_M. The 3.7B file is named K2-Horizon-4B-Q4_K_M.gguf. The quantized files' first publication dates are unconfirmed and remain separate from the original models' September 3 release. These are weight-file sizes, excluding KV cache, runtime allocations, and game rendering. Evaluate the context actually needed instead of assuming the full 512K fits a consumer GPU.

The official GGUF card's quantization evaluation covers non-agentic tasks; do not transfer the original models' Agent scores to Q4. Pin the runtime, exact GGUF, and chat template, then compare short Japanese tool tasks, reasoning separation, schemas, timeouts and fallback, total memory, and completion latency. We have not tested each artifact, integration into Ollama / LM Studio, or concurrent-game performance. Distinguish the model cards' Apache-2.0 labels from llama.cpp's MIT license, and check the original model LICENSE and distribution conditions before product bundling. We could not retrieve the original model LICENSE in this check.

IFM describes open training resources, but its announcement includes future releases and distinguishes training data from construction recipes. Avoid treating all resources as already released. It also discloses a 7B run that downloaded SWE-bench answers and inflated its score to 82. That run is not evidence of software-engineering performance; the model card reports 70.6. Transparency and task performance should be assessed separately.

The October 11, 2026 check confirms that prerelease llama.cpp b11528, published October 9 at 14:53 UTC (23:53 JST), re-enables K2 Horizon's -sm tensor. PR #30217 fixes host tensor-view handling and an assertion with partial offload plus tensor parallelism. The versioned code confirms the change; retain the b11454 history above. The server documentation still labels tensor parallelism EXPERIMENTAL. It requires supported multiple GPUs, a compatible GGUF, and RAM / VRAM for weights, KV cache, and runtime allocations. Compare startup, completion, prefill / decode, and total memory against layer splitting or a single GPU with identical GGUF, context, and inputs. Availability proves neither faster inference nor lower total memory. We have not tested each artifact, Japanese quality, or adoption by Ollama / LM Studio; check model and runtime redistribution terms separately.

Meta Muse Glimmer 30B

Meta's Muse Glimmer 30B combines a Dense language model of about 29.6 billion parameters with an image encoder. Its stated use cases include Agents, Tool Use, reasoning, and retrying after failures.

The model card describes quantizing the language model weights to below 20 GB and targets 24 GB or 32 GB-class configurations that also account for the KV cache used to retain recent input, the image encoder, and a smaller helper model for predicting upcoming tokens.

That helper enables speculative decoding, a technique that generates multiple candidate tokens ahead of time to improve output speed without intentionally changing the answer.

Meta's involvement is interesting for more than the release of another set of model weights. The company presents Agents that repeatedly use tools on a personal computer as an explicit product use case.

However, 24 GB or 32 GB does not guarantee the same speed on every system. Requirements and performance change with the quantization format, Context Length, inference software, and number of concurrent requests.

Liquid AI LFM2.5-2.6B

Liquid AI's LFM2.5-2.6B is a text model for on-device Agents released on August 4, 2026. It has approximately 2.69 billion parameters, a Context Length of 131,072 tokens, and covers 16 languages including Japanese. It is post-trained for Tool Use and multi-step workloads, while the official model card does not recommend it for demanding agentic coding or knowledge-heavy tasks.

The official GGUF repository lists both the standard Q4_0 file and the QAD Q4_0 file added on August 19, 2026, at approximately 1.59 GB, with Q4_K_M at approximately 1.67 GB. Quantization-Aware Distillation, or QAD, transfers knowledge from a high-precision teacher to a 4-bit student to reduce quantization loss. Across an evaluation of four model sizes including 2.6B, Liquid AI reports that QAD recovered 97% of the average accuracy lost when moving from BF16 to ordinary Q4_0. This does not mean that the 2.6B model alone achieves 97% of BF16 performance on every workload.

DSpark, released on August 20, 2026, uses a 328M draft model to propose upcoming tokens for verification by the main model. In Liquid AI's M4 Max evaluation with FP16 GGUF weights, llama.cpp, and temperature 0, average 2.6B generation speed increased from 61 to 139 tokens per second. The company also reports a 57% average reduction in Function Calling latency across multi-tool scenarios. The design preserves output under the published temperature-0 conditions, but those figures were not measured with the QAD Q4_0 checkpoint.

DSpark shipped with integrations for llama.cpp and SGLang. LM Studio 0.4.22, released on August 28, 2026, added official support for DFlash, DSpark, and MTP assistant drafters and requires llama.cpp engine 2.29.1 or newer. This makes compatible draft models easier to evaluate through a GUI or local server. The official LFM2.5-2.6B DSpark GGUF repository lists the Q4_K_M draft at about 200 MB.

This does not mean that LM Studio will always make LFM2.5-2.6B 2.27 times faster. LM Studio notes that speculative decoding can become slower when the draft model is not fast enough or its suggestions are accepted too infrequently. The approximately 1.59 GB target weight file and approximately 200 MB draft file also exclude KV cache, runtime overhead, and other applications. LM Studio is useful as a prototyping environment, but it is not automatically the runtime that should be shipped with a product.

The model uses the LFM Open License v1.0. Liquid AI's official summary states that commercial use is free for companies below US$10 million in annual revenue and requires a paid license above that threshold. Its small weights and Tool Use focus make it a candidate for evaluating on-device intent selection and short tool calls, but they are not measurements of game coexistence, Japanese output quality, or failure rate.

NVIDIA Nemotron 3.5 Lightning 30B-A3B

NVIDIA's Nemotron 3.5 Lightning 30B-A3B is a Mixture-of-Experts (MoE) model released on August 11, 2026. It has about 30 billion parameters in total and uses about 3 billion for each token. It supports reasoning and Tool Use, with a maximum stated Context Length of 1,000,000 tokens.

Its defining characteristic is an explicit focus on the execution layer rather than acting as a universal model for every task. This layer repeatedly performs Tool Use, validates results, and delegates work to subagents. NVIDIA describes an architecture in which a larger reasoning model handles complex planning while Nemotron 3.5 Lightning carries out repetitive execution.

NVIDIA reports about 85% accuracy on PinchBench, a benchmark for Agent execution, and says the model completes 10,000 tasks about 30% faster than Qwen3.6 35B-A3B at what it describes as similar accuracy. In the same official evaluation table, however, it scores below Qwen3.6 35B-A3B on SWE-bench Verified and Terminal-Bench 2.1. It should therefore be evaluated for a defined execution role, balancing speed and task success, rather than treated as a universal replacement for demanding reasoning or coding models.

Local runtimes including llama.cpp, Ollama, and LM Studio are documented. About 3 billion Active Parameters describes the computation used for a token; it does not mean the model has the storage footprint of a 3B model. As of August 21, 2026, the GGUF repository linked by the official model card lists an approximately 22.5 GB NVFP4 file and an approximately 25.4 GB Q4_K_M file. KV cache usage also grows with Context Length, so deployments that share a GPU with games or other workloads must measure total available memory rather than model file size alone.

The maximum 1,000,000-token Context Length is also not a default that every personal computer can use. In the official BF16 deployment examples, one H100 has a validated Context Length of 256,000 tokens, while the 1,000,000-token configurations require larger systems. Tests should limit the context to what the workload actually needs.

The model card describes the model as ready for commercial use under the OpenMDW 1.1 license. Before adoption, review the terms for the model, the selected quantized weights, and the inference software together.

Ornith 1.5 9B / 35B-A3B

Ornith 1.5 9B is a Dense model that uses all of its approximately 9 billion parameters. Its maximum Context Length is 262,144 tokens, and its stated focus includes reasoning, Tool Calling, and Agentic Coding. It is released under the MIT license and positioned by the developer for single-GPU use and quantized edge deployments.

In the official 9B GGUF distribution, Q4_K_M is 5.63 GB and Q5_K_M is 6.47 GB. The Q4_K_M file size makes an 8 GB-class GPU a reasonable evaluation target, but it excludes the inference runtime, KV cache, inputs, CPU-offloaded state, and rendering memory used by a concurrently running game. Do not conclude that the model can remain on the same 8 GB GPU as a game merely because its weights are 5.63 GB; measure free VRAM while the game is running at the required Context Length.

The developer reports five-run averages of 70.6 on SWE-bench Verified and, on Terminal-Bench 2.1, 47.0 with the Claude Code Harness and 46.2 with the Terminus-2 Harness. The Terminal-Bench 2.1 evaluation conditions include a 128,000-token context, 32 CPU cores, 48 GB of RAM, and a four-hour timeout. These are developer evaluations of coding and terminal operation. They do not measure speed on a typical PC, NPC dialogue quality, persona consistency, or real-time response latency in a game.

The 35B-A3B version is an MoE model with approximately 35 billion total parameters and 3 billion active per token. Its developer-reported five-run averages are 79.0 on SWE-bench Verified and 68.5 on Terminal-Bench 2.1 with the Claude Code Harness, but Q4_K_M in the official GGUF distribution is 21.7 GB. Low Active Parameters do not make its weights suitable for sharing an 8 GB-class GPU with a game.

As of August 22, 2026, the ornith:9b and ornith:35b entries in Ollama's official library are still described as Ornith 1.0. When evaluating 1.5, do not identify the version from those generic tags alone. Specify the 1.5 distribution and quantization explicitly, such as the official hf.co/ornith-ai/Ornith-1.5-9B-GGUF repository documented on the GGUF page.

The 9B model is a reasonable validation candidate for designs that assign high-level decisions and Tool Calling to a small model, but this is an inference from its features and size. An NPC evaluation still needs to test whether it selects a schema-constrained action from game state, whether a validator can reject invalid output, whether it preserves a persona, and whether it meets the latency target.

Qwen3.8-27B / Qwen3.8-Flash-Next

Qwen3.8-27B is a Dense model with about 27 billion parameters. It handles images and video as well as text, supports adjustable reasoning effort, and emphasizes coding, research, and long-running work.

Its standard Context Length is described as 262,144 tokens, with extension up to 1,000,000 tokens. A token is a unit into which text is divided for model processing.

The maximum is not always necessary. Longer Context Length increases KV cache use and the time required to read the initial input, known as Prefill, and may introduce irrelevant older information. Limit it to what the workload needs.

The official model card also publishes benchmark results, but those values depend on the developer's chosen harness and conditions. They should not be compared directly with figures published under a different developer's setup.

Qwen3.8-Flash-Next, released on August 26, 2026, is a multimodal MoE model that previews the architecture planned for Qwen4. Its main model has about 125 billion parameters and activates about 6 billion per token. It also includes about 51 billion parameters of N-gram Embedding and about 4 billion for Multi-Token Prediction, or MTP. Native Context Length is 262,144 tokens and can be extended to 1,000,000 with YaRN.

Qwen Sparse Attention, or QSA, reduces long-context computation by selecting candidate context at micro-block granularity. N-gram Embedding uses recent token patterns to look up entries in a large table, adding capacity with little extra computation. The table is designed to reside in host memory and be prefetched while GPU computation continues. The locally relevant development is therefore not that Flash-Next is already lightweight, but that it separates the trade-offs among computation, GPU memory, and model capacity.

In Qwen's own evaluation table, Flash-Next moves only slightly above Qwen3.8-27B on SWE-bench Pro, from 61.7 to 62.5, while DeepSWE 1.1 rises from 42.2 to 58.7, Toolathlon Verified from 67.1 to 73.5, and JobBench from 33.4 to 55.7. These results support the stated emphasis on Agents and Tool Use, but they are developer-published evaluations rather than third-party reproductions or performance measurements on ordinary PCs.

The approximately 6 billion Active Parameters should not be read as a 6B-class hardware requirement. The third-party Unsloth GGUF distribution lists even its aggressively compressed UD-IQ1_S files at approximately 72.5 GB at publication time. The official repository provides paths for Transformers, llama.cpp, SGLang, vLLM, and TokenSpeed, but this is a newly released architecture. Runtime version, image support, Tool Calling, and the selected quantization need separate verification. For now, it is better treated as an evaluation target for systems with large memory capacity than as a resident model for a typical gaming PC.

Checked on October 4, 2026. The October 3 prerelease llama.cpp b11372 reduces indexer intermediate allocations in the qwen4exp path. In PR #29825, the author reports compute buffers of 6056 → 3748 MiB on CUDA at 128K context, with the N-gram table in RAM, and 6224 → 3924 MiB on Metal at 64K. These are configuration-specific implementation measurements, not Qwen hardware specifications or a halving of total RAM / VRAM including weights and KV cache. For an existing large-memory evaluation, pin the runtime, quantization, context, batch / ubatch, and backend; measure temporary allocations, total memory, prefill, and decode separately. We have not reproduced the results or verified downstream Ollama / LM Studio integration. The consumer-PC conclusion above remains unchanged.

Checked on October 13, 2026 JST. This corrects the existing summary against Flash-Next's Qwen Community License 1.0; it is not an announcement of changed license terms. Clause 1 requires prominent model-name display for commercial products or services exceeding 100 million monthly active users or US$20 million monthly revenue. Under clause 2, if the licensee or an affiliate conducts a Model as a Service / AI Work Assistant business, the licensee must obtain a separate license before any commercial use of the model or derivatives. The condition is not limited to use in those products. An exception covers qualifying internal use that does not make the model, outputs, or underlying capabilities available to third parties. Check the original business definitions and exclusions; we do not determine legal applicability to an individual business or game feature. These terms differ from Qwen3.8-27B's Apache-2.0 license.

Qwen3.8-27B and Flash-Next are therefore not direct replacements in the current shortlist. The former is a comparatively smaller Dense model, while the latter is a large-scale preview for evaluating the efficiency direction of Qwen4. If this architecture later reaches smaller models, its quality and resource use should be measured again rather than inferred from Flash-Next alone.

GPU expert caching keeps host-weight requirements

Checked on October 11, 2026. Prerelease llama.cpp b11480, published October 8 at 01:36 UTC (10:36 JST), includes the host-expert GPU cache from PR #29887. b11507, published that day at 15:27 UTC (October 9 at 00:27 JST), adds multiple-GPU support from PR #30112. This reuses experts in GPU cache; it neither compresses the checkpoint nor replaces host RAM with Edge0-style SSD streaming.

The b11507 argument definition disables --moe-cache-mib by default. The final implementation uses the cache for small batches of up to 32 tokens, rejects tensor parallelism, and splits the total budget according to --tensor-split or free memory. Do not reuse a PR example as a budget per GPU. The author's single-GPU results use Qwen3.8-Flash-Next Q4_0 (93.7 GiB), EPYC 7742 / 16 cores, PCIe 4.0 x16, 32K context, and no MTP. They do not establish consumer-PC or concurrent-game performance. For large-memory offload evaluations, retain enough host RAM for weights and runtime allocations and GPU headroom for cache, then compare disabled / enabled cold and warm prefill / decode, transfers, total RAM / VRAM, and rendering load. Japanese quality, p95 / p99, and adoption by Ollama / LM Studio remain untested by us; runtime MIT and model terms remain separate.

Ternary Bonsai 2 27B: low-bit weights require a matching runtime

Checked on September 18, 2026. Ternary Bonsai 2 27B is derived from Qwen3.8-27B. Its language weights occupy 5.95 GB in PTQ1_0 or 7.21 GB in PQ2_0; image input uses a separate optional projector of about 0.63 GB. These are file sizes, not total runtime VRAM. For these PTQ1_0 / PQ2_0 artifacts, the model card requires PrismML's llama.cpp fork with matching ternary and Hadamard-transform support; ordinary GGUF support is insufficient.

PrismML reports 84.78 versus FP16's 86.32 averaged over 14 thinking-mode benchmarks, or about 98.2% of that baseline average. This is not a guarantee of retaining 98.2% of capability on every task. Its batch-1, depth-0, text-only llama-bench TG128 results use different packings: RTX 4090 reaches 91.1 tok/s with PTQ1_0, while RTX 5090 reaches 129.9 and M5 Pro 28.1 with PQ2_0. Do not combine the smallest file and every platform's fastest result into one configuration. NPC quality and performance while rendering a game remain unverified.

Checked on October 12, 2026 (JST). Prerelease llama.cpp b11556, published October 11, includes upstream Hadamard support from PR #29600. The evaluation artifact is the official dev Q2_0 GGUF, 7.63 GB. This does not establish upstream support for the 5.95 GB PTQ1_0 or 7.21 GB PQ2_0 files above; PTQ1_0 PR #29672 remains open at this check. The versioned adapter code rejects LoRA targeting Hadamard-folded weights. Pin the exact artifact and runtime, and budget separately for weights, KV cache, runtime allocations, and the game. File size is not required RAM. Japanese quality, quantization quality, and game performance remain untested by us; runtime support does not grant model redistribution rights.

Gemma 4

Google's Gemma 4 is a family with several configurations for different hardware targets rather than a single model.

• 2B and 4B-class models for smaller devices
• A unified model of about 12B parameters
• An MoE model with about 26B total parameters that activates only part of its experts for each token
• A Dense model with about 31B parameters

The family covers Tool Use, reasoning, image input, and multiple languages across devices from small hardware to PCs. The useful question is therefore not simply whether “Gemma 4 runs,” but which checkpoint, precision, and runtime are being evaluated.

Smaller models offer advantages in memory use and response time but do not guarantee the same quality as larger models. Verify instruction following, natural output in the required language, and correct Tool Use.

Checked on October 11, 2026. Google's October 8 ML Drift announcement opens LiteRT's GPU foundation as a standalone C++ library, adding an in-app GPU-inference path rather than a new Gemma model or a complete game SDK. The pinned README lists OpenCL / Metal / WebGPU / OpenGL ES. There are no standalone distribution releases at the check date; LiteRT C++ SDK binaries are separate. Desktop LLM results are a preview; “up to 12% lower memory overhead” does not mean 12% less total memory. Pin supported model formats, SDK, and backend, then measure cold initialization, prefill / decode, total memory, and rendering contention. Ordinary GGUF replacement, Japanese quality, tool use, and game performance remain unverified. Check Apache 2.0, model, and dependency terms separately.

Checked on October 12, 2026 (JST). Prerelease llama.cpp b11559, published October 11 at 19:14 UTC (October 12 at 04:14 JST), includes PR #30266 for Gemma 4 E2B / E4B OpenCL Flash Attention decode with head dimension DK512. This updates the runtime, not the model. The versioned eligibility checks require Adreno, F32 queries / F16 KV, a single query position, GQA 4 or 8, and successful kernel initialization; GGML_OPENCL_FA_DK512_DECODE=0 disables this path. A7x mixed-type kernels are excluded because of compiler crashes. The OpenCL guide also warns that Flash Attention does not always improve performance. Pin the device, driver, GGUF, and context, and measure prefill / decode, thermal behavior, total memory, rendering contention, and fallback. We have not tested these, and this is not a speedup guarantee for every GPU.

OpenAI gpt-oss

OpenAI's gpt-oss-20b and gpt-oss-120b are reasoning-focused models with downloadable weights under the Apache 2.0 license. They run on computers, servers, or compatible hosting services managed by the user.

They differ from cloud models delivered through the OpenAI API in execution location, update process, hardware requirements, and operational responsibility. Do not group them together solely because they are OpenAI models.

The resource requirements of the 20b and 120b versions differ substantially. Record the exact model and compression format rather than evaluating only at the family level.

Ministral 3 3B

Mistral's Ministral 3 3B, released on December 2, 2025, is a compact model designed for edge deployment. Mistral lists it as a 3B model with text and image input, a 256,000-token context window, Structured Outputs, Function Calling, downloadable weights, and the Apache 2.0 license.

Its size makes it easier to shortlist for on-device evaluation. It does not guarantee natural output in a required language, consistent character behavior, correct Tool selection, or acceptable latency. Runtime memory including the vision encoder and KV cache also cannot be inferred from a quantized weight file alone.

Mistral Small 4

Mistral Small 4 combines reasoning, coding, Agents, and image input in a single MoE model.

Its official documentation describes about 119 billion total parameters and about 6 billion Active Parameters per token. Active Parameters are useful for understanding computation, but the memory and download size required to hold the full model are not equivalent to those of a 6B model.

Availability, deprecation, and distribution status can also change. Check the current model, lifecycle status, and license at the time of adoption rather than continuing to use an older candidate without review.

Edge0 streams experts from SSD

Checked on September 18, 2026. Edge0 combines SSD expert streaming, Recover-LoRA, and a prerouter that predicts which experts to load. For its Qwen3.6-based 35B-A3B preview, the developer reports about 23 GB of stored weights and 2.9 GiB of peak active memory. Unlike Bonsai's compressed resident weights, this approach changes which experts are loaded at a given time.

The reported 14.9–17.7 tok/s uses a Mac mini M4 Pro with 24 GB memory, roughly 3.3K input tokens, 10 warmup steps, and 200 timed output tokens over two runs. Active memory is not total system RAM including the OS and page cache; the warm prefill measurement explicitly uses resident page-cache data. It does not show that a 3 GB computer can run the model. Storage, cache state, and long-context KV growth must be measured separately.

The current backend is Apple Silicon with MLX; CUDA is on the roadmap. Ordinary Ollama or LM Studio GGUF loading does not reproduce this streaming pipeline. The OpenAI-compatible server provides an integration boundary, but establishes neither Windows support nor suitability for game NPCs. Evaluate cold/warm behavior and storage contention alongside quality and latency.

Pin versions when evaluating Apple Silicon runtimes

Checked on September 29, 2026. Hugging Face's GGUF execution update for Transformers adds a packed inference path that retains quantized weights instead of expanding them. This differs from the earlier dequantizing import path. Initial support is MPS-only and focuses on Qwen3.5 dense and MoE architectures, including compatible Qwen3.8 checkpoints. Padding and batching have limitations; this does not imply support for every GGUF or GPU, or arbitrary fine-tuning with weights kept quantized. Treat it as an option for Python-based experiments and check the runtime version and supported model.

Ollama v0.40.0-rc0 automatically routes supported architectures to MLX on Apple Silicon. It was a prerelease at the September 29 check. The change applies to supported models, not every model or Windows / NVIDIA execution. Do not present earlier MLX benchmarks as the speedup of this RC; compare with fixed context, cache conditions, and concurrency.

The October 7, 2026 check confirms that Ollama v0.40.0 is available as a non-prerelease. Its official metadata sets prerelease to false, so evaluating the MLX path no longer requires choosing an RC. The release entry retains a September 25 published_at, while the current macOS assets were created October 6 at 18:27 UTC (October 7 at 03:27 JST); neither establishes the exact transition time. October 7 is our availability-check date.

The pinned macOS requirements specify macOS 14 or later. The MLX path requires supported Apple Silicon hardware and models, with sufficient unified memory; check model terms separately from Ollama's MIT license. Compare RC and non-prerelease cold / warm responses using the same model, context, cache conditions, and concurrency. Availability alone does not establish improvements in Japanese quality, total memory, p95 / p99 latency, or concurrent-game performance; we have not measured these.

Checked on October 13, 2026 JST. Prerelease Ollama v0.40.4-rc0, published October 12 at 19:00:12 UTC (October 13 at 04:00:12 JST), includes the MLX memory-retention fix in PR #18955. A recurrent state restored after speculative rollback could retain a view into a larger buffer. Request-close compaction detaches the needed state before retaining it in the prefix cache. This does not compress weights or establish a benefit for every model. Compare runtime versions with the same model, digest, context, speculative settings, and request sequence; distinguish logical cache usage, actual allocated memory, swap, and latency. We have not reproduced the fix or measured memory savings, Japanese quality, or concurrent-game performance.

Checked on October 11, 2026. Ollama v0.40.0 also incorporates MLX decision-model support for Nimble, Tev1, Clef, and Clef Flash. RC selection is no longer required to evaluate this path. The v0.40.2 decision documentation describes /v1/systemone choices, probabilities, and ordered ratings, separately from text generation. Clef images use raw base64 in images; URLs and data URLs are rejected, unlike the llama.cpp path. Pin the supported Apple Silicon model, engine, and input representation. More runtime options prove neither accuracy nor probability calibration, memory savings, or lower latency; game-side evaluation is covered in the NPC article.

Checked on October 4, 2026. Splash adds a dedicated Apple Silicon engine; LM Studio 0.4.25 added support on September 19. Its stated requirements are M3 or newer, macOS 26.4 or later, and at least 36 GB unified memory, with 48 GB or more recommended by the developer. The Qwen3.8-27B Splash package is 17.4 GB including the 4-bit model, DFlash 2 draft, and vision encoder; it is a Splash-specific package rather than a normal MLX / Transformers checkpoint. Download size is not total memory. Check engine and model terms separately, and measure cold prefill, cache reuse, latency tails, and concurrent-game load before adoption; we have not run those tests.

Reading the specifications

Several numbers used for local LLMs look similar but describe different things.

Dense models generally use the full model for processing. MoE models reduce computation by selecting some experts, but the full set of experts still has to be stored.

Quantization reduces memory requirements but can affect accuracy, runtime compatibility, and speed. Different methods can produce different results even when all are described as “4-bit,” so record the specific quantization format.

Checked on October 11, 2026. Prerelease llama.cpp b11324, published October 1 at 19:24 UTC (October 2 at 04:24 JST), includes PR #29749. On Linux with -lm dio, it replaces a temporary full-tensor copy with a reusable buffer capped at 64 MB. Measure loading peak RSS, free RAM, and load time separately from inference RAM / VRAM and game load. The author's DGX Spark / CUDA observations are configuration-specific, not a reduction shared by every OS or loading mode; we have not reproduced them.

Checked on October 11, 2026. The non-prerelease Ollama v0.40.2, published October 8 at 16:53 UTC (October 9 at 01:53 JST), currently explains that older models are converted on first use and an original backup is kept for downgrades. PR #18874 hides duplicate list entries; it does not remove disk backups. Allow disk headroom and measure first-use conversion separately from downloads and runtime RAM / VRAM. If backups are removed, reverting to a version earlier than 0.40 requires re-pulling those models. The conversion feature's first introduction date is unconfirmed; do not claim v0.40.2 introduced it. We have not converted or deleted models, measured the extra disk usage or duration, or tested recovery.

Checked on October 12, 2026 (JST). Ollama v0.40.3, published October 11 UTC, pauses background local-model compatibility migration. PR #18908 cites per-request overhead and GC pressure, especially for high-volume embedding use; the versioned request path disables the migration call. Retain the v0.40.2 history above, but check the installed version before assuming background conversion occurs. This change does not establish that converted weights are reverted or backups deleted. Compare request latency and GC under the same model and concurrency, separately from disk usage and runtime RAM / VRAM. We have not measured an improvement or VRAM reduction, or tested conversion and recovery.

Selection criteria

Define the workload and constraints before choosing a model name.

• Can the task be limited to classification or summarization, or does it require open-ended conversation and Agent behavior?
• How much system RAM and GPU VRAM are available after accounting for other applications?
• Is text sufficient, or are images and video required?
• What language quality, Context Length, and Tool Use accuracy are required?
• What are the licenses and lifecycle status of the model, quantized data, and inference software?

After narrowing the list to two or three candidates, compare them with the same prompts, Context Length, quantization format, and hardware. Measure time to first output, completion time, system and GPU memory, task success rate, and failure cases. Developer benchmarks are useful for creating a shortlist, but the final decision should be based on the real workload.

A planned local-SLM integration inside a game

At gamescom 2026, NVIDIA announced that the creature-collection RPG Aniimo plans to add NVIDIA ACE support in early 2027. A fine-tuned Nemotron SLM is expected to run on the player's GPU, interpret intent from natural language or voice, and trigger the game's behavior APIs.

This is a planned example of using a local model to convert natural language into structured game operations rather than only providing open-ended chat. Production VRAM requirements, GPU contention with rendering, latency, and invalid-action rates are not yet established in public material. Model selection therefore needs to measure total VRAM under concurrent game load, latency, and validation of the output sent to behavior APIs, not just weight size.

The architecture and the other game-AI cases presented at gamescom are covered in What gamescom 2026 showed about the state of game AI.

The NPC integration article also covers AI Follower Jessica, a public Fallout 4 mod combining local dialogue, memory, and actions.

Current perspective

The September 18 additions distinguish compact Agent models, compressed resident weights, and streamed experts. Iris-mini also reinforces the role of context management. These broaden evaluation choices without establishing NPC adoption or game co-residency.

K2 Horizon expands the compact Agent shortlist, while PAIR adds a way to distribute concurrent requests across PCs. Both require workload-specific validation.

The important change in local LLMs as of September 2026 is not that one model has become universally best. Instead, more options can now provide reasoning, Tool Use, image input, recovery after failure, and faster generation in local environments. Runtime updates such as LM Studio's assistant-drafter support can also make new evaluation configurations available for an existing model.

Meta's Muse Glimmer illustrates that shift clearly. Liquid AI's LFM2.5-2.6B shows a direction that combines a small Tool Use model with QAD for quantized quality and DSpark for faster generation. NVIDIA's Nemotron 3.5 Lightning also demonstrates a move away from building an Agent around one model by separating planning from repetitive execution. Perplexity Portable Computer makes another direction concrete by co-designing the model, harness, sandbox, and optional cloud escalation. Ornith 1.5 9B shows that smaller Tool Calling models are also becoming candidates for Agent experiments. Qwen, Gemma, gpt-oss, and Mistral broaden the options with different scales and architectures.

Because models change quickly, treat this list as a dated snapshot rather than a permanent recommendation. Re-evaluate when:

• A new model or quantization format is released
• Inference software support changes
• The target hardware or other concurrent applications change
• Language, Tool Use, Context Length, or latency requirements change
• The license or lifecycle status of a model or dependency changes

Caveats

• Spark-X2.5, Bonsai 2, Edge0, and Iris-mini were checked on September 18, 2026. Their performance figures are developer-reported, and we have not reproduced them.
• Earlier entries draw on public information compiled by September 2, 2026, retaining their individual information dates. K2 Horizon and NVIDIA PAIR were added after checking primary sources on September 6. This update did not revalidate every earlier entry.
• The models listed are representative examples, not a complete catalog of local LLMs.
• Performance and memory requirements vary with quantization, inference software, Context Length, concurrency, and hardware.
• Developer-published benchmarks reflect each developer's evaluation conditions.
• The October 4 check covers Portable Computer Windows availability, Splash, and Qwen indexer memory only; it does not revalidate all earlier models or establish measured game performance.
• Review the model license, terms of use, inference software, and dependency licenses before adoption.

Related resources

• Muse Glimmer 30B Model Card
• LFM2.5-2.6B announcement
• LFM2.5-2.6B Model Card
• LFM2.5-2.6B official GGUF repository
• LFM2.5 QAD Q4_0
• LFM2.5-DSpark
• LFM2.5-2.6B-DSpark official GGUF repository
• LFM Open License summary
• LM Studio 0.4.22 release notes
• LM Studio Speculative Decoding
• Introducing Portable Computer
• A Local-First Agent for Private and Cost-Effective Knowledge Work
• PPLX Computer Qwen 3.8 27B DFlash2 checkpoint
• Nemotron 3.5 Lightning Technical Blog
• Nemotron 3.5 Lightning BF16 Model Card
• Nemotron 3.5 Lightning GGUF
• Ornith 1.5 9B Model Card
• Ornith 1.5 9B GGUF
• Ornith 1.5 35B-A3B Model Card
• Ornith 1.5 35B-A3B GGUF
• Ollama ornith library
• Qwen3.8-27B Model Card
• Qwen3.8-Flash-Next announcement
• Qwen3.8-Flash-Next Model Card
• Qwen Community License 1.0
• Unsloth Qwen3.8-Flash-Next GGUF
• Gemma 4 model overview
• OpenAI API model catalog
• Ministral 3 3B model overview
• Introducing Mistral Small 4
• NVIDIA gamescom 2026 announcements
• What gamescom 2026 showed about the state of game AI
• K2 Horizon
• K2 Horizon 3.7B
• K2 Horizon 7B
• K2 Horizon 3.7B GGUF
• NVIDIA PAIR