Qwen3.8-Flash-Next only activates 6B parameters per token. Then you open Unsloth's GGUF repo and see that the smallest file, a 1-bit quant, is 66.4 GB, and the UD-Q4_K_XL that everyone defaults to is 111 GB.
"Flash" describes how fast it decodes. It says nothing about how small it is. If you want to run Qwen3.8-Flash-Next locally, the number that matters is how much memory you have in total, and the 6B on the card won't tell you that.
The 6B does pay off once the weights fit somewhere. In a community llama.cpp benchmark with the experts in system RAM, a 24GB VRAM budget (simulated on a 96GB card, with 96GB of DDR5 behind it) still decoded UD-IQ4_XS at 39.01 tok/s, so for this model RAM matters more than the size of your card.
I haven't run it myself. The numbers below come from the model cards or from community benchmarks, and each one is linked.
What Qwen3.8-Flash-Next actually is
The base Qwen repo calls it an "experimental preview" and the first open-weight release of the architecture that will underpin Qwen4. Neither card gives a release date. The citation says August 2026, the Qwen collection was updated on Aug 26, and llama.cpp support (PR #27742) was merged on August 27, 2026.
It differs from Qwen3.8-27B and the other Qwen3.8 releases in a few ways:
- 125B total parameters and 6B active, plus 51B of n-gram embeddings and a 4B MTP (multi-token prediction) head
- 48 layers in a 3:1 pattern of Gated DeltaNet to Qwen Sparse Attention (QSA), where earlier models used Gated Attention
- MoE with 512 experts, of which 10 routed and 1 shared are active per token
- a Gated Residual design, and a training recipe that mixes Muon and AdamW
- 262,144 tokens of native context, which YaRN can stretch to 1,000,000
- a vision encoder
- the
qwen-community-1.0license, not Apache 2.0, so read it before you ship anything commercial
Qwen also sells Qwen3.8-Flash as the "official version" built on Flash-Next, with a 1M default context and built-in tools. Flash-Next is the open-weights preview.
Why a 6B-active model ships as a 111 GB file
MoE only saves compute. All 512 experts still have to sit somewhere you can read them fast, because the router picks a different 10 on every token.
The other reason is the n-gram embedding table, which is easy to overlook. It's a lookup table of bigrams and trigrams with about 20 million rows, and it feeds layer 2. The r/LocalLLaMA benchmark thread, which calls it the per-layer token embedding (PLE) table, measured it at 27.2 GiB inside an 87.2 GiB UD-IQ4_XS file. Unsloth's docs list it at 102.4 GB in BF16, 54.4 GB in Q8_0, 38.4 GB in Q5_1 and 28.8 GB in IQ4_NL. It sits in its own shard (-00004-of-00004 for UD-Q4_K_XL), and Unsloth ships a set_split.py script for swapping in a different precision.
Each token only looks up a few rows of that table, so you can keep it in system RAM, or even page it in from SSD through mmap. The one place you shouldn't put it is your GPU (details below).
The quant table and an estimated memory budget
File sizes come from the file listing in Unsloth's GGUF repo. The "estimated total" column is my own back-of-envelope figure, not an official one: file size plus KV cache plus a few GB for runtime overhead.
The KV cache is small for a 262K model. Only 12 of the 48 layers are attention layers, and each has 2 KV heads at head dim 256, which comes to roughly 24 KB per token in F16. That works out to about 6.4 GB at the full 262K, or under 1 GB at 32K. The QSA indexer keys add a little on top, and the DeltaNet layers keep a fixed-size state.
| Quant | File size | Estimated total at 32K ctx |
|---|---|---|
| UD-IQ1_S | 66.4 GB | ~70 GB |
| UD-IQ1_M | 68 GB | ~72 GB |
| UD-Q2_K_XL | 79.6 GB | ~84 GB |
| UD-Q3_K_XL | 89.5 GB | ~94 GB |
| UD-IQ4_XS | 93.7 GB | ~98 GB |
| UD-Q4_K_XL | 111 GB | ~116 GB |
| UD-Q5_K_XL | 158 GB | ~163 GB |
| Q8_0 | 188 GB | ~193 GB |
| BF16 | 354 GB | ~359 GB |
"Total" means RAM plus VRAM combined. None of this has to fit on a GPU.
Watch out for Unsloth's own docs page, which lists "4-bit: 64 GB" in its requirements table. That doesn't line up with a 111.3 GB UD-Q4_K_XL file, and the same page elsewhere says you need at least 75 GB. Leaving the n-gram table on disk doesn't get you there either. Take away shard 4 (the 28.8 GB table) and the rest of the UD-Q4_K_XL is still about 82.5 GB. Plan around the file size.
How to run Qwen3.8-Flash-Next locally with llama.cpp
Use a llama.cpp build from after August 27, 2026, because older builds won't recognize the qwen4exp architecture (qwen4_exp is the Hugging Face model_type). The "Use this model" snippets on the Hugging Face page give the quick path:
# install (macOS/Linux)
curl -LsSf https://llama.app/install.sh | sh
# or
brew install llama.cpp
# OpenAI-compatible server + web UI
llama serve -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
# terminal
llama cli -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XLTo pull only the quant you want, then serve it with the MTP draft head for speculative decoding:
hf download unsloth/Qwen3.8-Flash-Next-GGUF --local-dir unsloth/Qwen3.8-Flash-Next-GGUF --include "*UD-Q4_K_XL*"
hf download unsloth/Qwen3.8-Flash-Next-GGUF --local-dir unsloth/Qwen3.8-Flash-Next-GGUF --include "*mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf*"
llama.cpp/llama-server -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL \
-md unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
--spec-type draft-mtp --spec-draft-n-max 5Unsloth claims MTP gives a 1.3–1.7x speedup, with smaller gains on low-bandwidth hardware. Unsloth's docs still build from a custom branch (qwen4exp/mtp), and the MTP README in the repo says a stock llama.cpp build can't use its draft heads. Mainline has since added MTP for this model (PR #29761, merged October 1, 2026). If a stock build ignores the head, fall back to the branch.
Use the card's sampling settings. For thinking mode (the default) that's --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0. For non-thinking mode it's temperature 0.7, top_p 0.80 and presence_penalty 1.5.
Running it with Ollama
Hugging Face's "Use this model" menu offers this:
ollama run hf.co/unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XLUnsloth's docs page doesn't mention Ollama at all, and Ollama has to support the new architecture before this works. If you get an unknown-architecture error, go straight to llama.cpp. The Ollama command also pulls the whole 111 GB, so check your disk space first.
Real tok/s on RTX PRO 6000, 24GB, CPU-only, M1 Max and DGX Spark
These are third-party numbers on very different setups. Treat them as rough guides.
| Setup | Quant | Decode | Source |
|---|---|---|---|
| RTX PRO 6000 96GB + 96GB DDR5 | UD-IQ4_XS | 109.07 tok/s (2K prompt), 21.61 at 245K | r/LocalLLaMA |
| Same box, 24GB VRAM simulated | UD-IQ4_XS | 39.01 tok/s | same |
| Same box, CPU only | UD-IQ4_XS | 8.34 tok/s | same |
| M1 Max 64GB | 84.9 GB AtomicChat IQ1_M-heavy mix | ~18 tok/s | lilting.ch |
| DGX Spark | UD-Q4_K_XL + draft | 24.2 tok/s | contextstudios.ai |
That 39 tok/s on 24GB is what makes 6B active worth having. Even the 8GB tier, with all 48 expert layers in RAM, decoded at 35.69 tok/s. Once the experts are in RAM, a single consumer GPU stays usable. The 8–48GB tiers were simulated by reserving VRAM on the 96GB card, so they show what offloading costs, not how a real 4090 performs.
I wouldn't read much into the M1 Max number. The quant pushed KLD up to 0.2277, the author got a made-up game title and an insertBefore bug that kept a generated page from rendering, and the machine was swapping. It ran, but I wouldn't trust what came out.
Keep the n-gram table off your GPU (it's 55x slower)
With the per-layer token embedding table in system RAM, decode ran at 108.5 tok/s. Forcing it onto the GPU with per_layer_token_embd=CUDA0 dropped decode to 1.95 tok/s, about 55x slower. The author didn't find the cause and only tested build b10666. If your speed looks terrible and you've been tweaking -ot overrides, check this first.
The same post found that --load-mode none (fully RAM-resident, no mmap) gave 746.7 tok/s prefill against 400.4 with mmap, provided you have enough free RAM.
Don't turn on 1M context (YaRN) by default
Static YaRN keeps the scaling factor constant "regardless of input length, potentially impacting performance on shorter texts." Leave it off unless you really need more than 262K tokens, and if you only need 524,288, set factor to 2.0 rather than 4.0. The card only documents the YaRN override for vLLM, SGLang and TokenSpeed.
Qwen3.8-Flash-Next benchmarks vs Qwen3.8-27B, with salt
These are all Qwen's own numbers. SWE-bench Pro is 62.5, against 61.7 for Qwen3.8-27B and 53.4 for Claude-Opus-4.6 Max. That Opus figure is its officially published score, while the Qwen models were run under the Claude Code harness, so it isn't a like-for-like comparison. LiveCodeBench v6 is 91.9, GPQA Diamond 91.7, and SWE-bench Multilingual 81.0. Some benchmarks are in-house (CoWorkBench, RecreationBench), and HLE was judged by GPT-4o.
The number I keep coming back to is the 27B dense model sitting 0.8 points behind on SWE-bench Pro. If 125B of weights only buy you 0.8 points over a model that fits in 24GB at Q4, Flash-Next makes sense when you have plenty of RAM and want speed, not when you want the best quality per GB. If you're on 64GB, I'd sooner run the 27B at a sane quant than an IQ1-heavy Flash-Next that hallucinates game titles.
What's the biggest GGUF your machine can actually run at a usable speed, and would you run Flash-Next at IQ1 over a dense 27B at Q6?