How to Run GLM-5.3-Flash Locally: NVIDIA's 'Local' FP4 Needs Two Blackwell GPUs

The model is called "Flash" and it has 320 billion parameters. NVIDIA's new "Local" build of it needs about 166 GiB of VRAM. Both of those words are doing a lot of marketing work.

If you want to run GLM-5.3-Flash locally, NVIDIA's GLM-5.3-Flash-FP4-Local checkpoint is validated on exactly two setups, two RTX PRO 6000 Blackwell cards or two DGX Spark boxes, and it runs on vLLM only, from community containers. It also isn't smaller than a 4-bit GGUF. Everyone without Blackwell is choosing between GGUF and MLX anyway.

I haven't run it myself. What follows comes from the model cards, the llama.cpp threads and the quant repos, plus the numbers you need to decide whether your hardware is in the game.

What GLM-5.3-Flash is (and why "Flash" stopped meaning small)

The last Flash model I paid attention to was GLM-4.7-Flash: a 30B-A3B text MoE under MIT. You could run that on one consumer GPU.

GLM-5.3-Flash is a different animal:

320B total parameters, 18B active per token (the Hub widget says 321B)
Z.ai calls it "the first natively multimodal model in the GLM-5 series", tagged image-text-to-text
Hybrid sparse + linear attention, plus Manifold-Constrained Hyper-Connections (mHC)
1M-token context window per Z.ai's announcement
MIT license
reasoning_effort takes low, high or max, and defaults to max

Z.ai claims it beats GLM-5.2 "at one-tenth the price" and approaches Claude Opus 4.8 on coding and agent benchmarks. Those are vendor numbers. The card lists Terminal-Bench 2.1 at 84.3 and DeepSWE at 63.4.

In this release "Flash" means cheap per token on an API, and that does nothing to get it onto your gaming PC.

What FP4 means in this build, and yes, it's Blackwell-only

There are two NVIDIA quants, and they are easy to confuse.

nvidia/GLM-5.3-Flash-NVFP4 is a plain post-training quantization (ModelOpt 0.47.0), tested on GB200, with examples using 4-way tensor parallelism. That one is datacenter-oriented. nvidia/GLM-5.3-Flash-FP4-Local is the interesting one. It uses quantization-aware distillation (QAD): the BF16 model acts as teacher, the student's routed-expert weights get quantized to NVFP4 on every forward pass, and gradients update the underlying BF16 weights. The quantized model gets trained to recover what quantization broke, which a one-shot rounding pass has no way of doing.

The precision mix, per the card:

| Component | Precision |
|---|---|
| Decoder routed experts | NVFP4 (E2M1 values, E4M3 block scales) |
| Decoder attention | MXFP8 |
| Vision attention | MXFP8 |
| Vision MLP and merger linears | NVFP4 |
| Other vision tensors, norms | BF16 |

NVFP4 is the Blackwell tensor-core format, and the card lists Blackwell as the only supported microarchitecture, Linux as the preferred OS. It says nothing about Hopper or Ampere. The card also says the scale factors use "a new lossless compression format", which is why upstream vLLM can't load it yet and you need a community container. If you own an H100 or a 4090, this checkpoint is not for you, whatever your VRAM.

The VRAM math: FP4-Local vs BF16 vs GGUF vs MLX

These are file sizes and published memory figures. GiB and GB are mixed because the sources mix them: 166 GiB is about 178 GB.

| Build | Size / memory | Runs on |
|---|---|---|
| BF16 (Unsloth GGUF repo) | 642 GB | Multi-GPU server |
| Q8_0 GGUF | 341 GB | Server / big RAM box |
| NVIDIA FP4-Local | ~166 GiB VRAM | 2x RTX PRO 6000 Blackwell, 2x DGX Spark |
| UD-Q4_K_XL GGUF | 200 GB | llama.cpp, anything with enough RAM+VRAM |
| UD-IQ4_XS GGUF | 157 GB | llama.cpp |
| UD-Q2_K_XL GGUF | 109 GB | llama.cpp |
| UD-IQ1_S GGUF | 93.1 GB | llama.cpp |
| MLX 4-bit (orcarouter) | ~204 GB disk, 224 GB memory | Apple Silicon |

Two RTX PRO 6000s give you 192 GB total. After the ~166 GiB of weights and runtime, NVIDIA's validated config fits 1.8M tokens of FP8 KV cache. The two-Spark setup (128 GB each) gets 2M tokens. The linear attention layers are why a million-token context isn't a fantasy on this hardware.

The FP4-Local build is not smaller than a 4-bit GGUF; IQ4_XS is 157 GB. What FP4 buys you is native tensor-core math on Blackwell and a published accuracy number. That number is the part I care about.

Published accuracy loss: almost none, on the two benchmarks shown

From the FP4-Local card:

| Precision | GPQA Diamond | ToolEvalBench |
|---|---|---|
| BF16 | 92.1% | 91.9% |
| FP4-Local | 91.3% | 92.4% |

0.8 points down on GPQA, 0.5 up on tool calling. The PTQ NVFP4 build reports a similar story across more benchmarks: GPQA Diamond 0.9211 vs 0.9217, MMMU Pro 0.763 vs 0.7688, Terminal Bench 2.1 0.8315 vs 0.8258.

Two benchmarks is a thin evaluation for a vision-language model, though. There's no MMMU or long-context number on the Local card, and those are exactly where I'd expect aggressive quantization to show. Meanwhile the Unsloth GGUFs publish file sizes and no accuracy numbers at all, so on paper you're comparing a measured 0.8-point drop against a shrug.

How to serve it: the exact vLLM path on RTX PRO 6000

The Local card has no TensorRT-LLM or SGLang instructions, so vLLM in a container is the only path.

For 2x RTX PRO 6000 Blackwell, the image is ghcr.io/local-inference-lab/vllm. The card's simplified run uses tag release-10152026 with PRESET=glm53-tp2, while another line references release-10142026. Check which tag exists before you pull.

The verbose path runs vllm serve with roughly 40 flags. These are the ones that matter, excerpted from the card (copy the full command from the card, not from here):

bash

vllm serve nvidia/GLM-5.3-Flash-FP4-Local \
  --tensor-parallel-size 2 \
  --decode-context-parallel-size 2 \
  --quantization modelopt_mixed \
  --kv-cache-dtype fp8 \
  --max-model-len 1048576 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.985 \
  --attention-backend B12X \
  --moe-backend b12x \
  --linear-backend b12x \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"reasoning_effort":"max","clear_thinking":false}' \
  --speculative-config '{"method":"mtp","draft_sample_method":"probabilistic","rejection_sample_method":"standard","moe_backend":"b12x","attention_backend":"B12X","num_speculative_tokens":3}'

--max-num-seqs 8 tells you who this is for: one developer or a small team.

For two DGX Sparks, the card points to eugr/spark-vllm-b12x:latest, tensor parallel 2 across both nodes, --max-model-len of 500,000, --max-num-seqs 4, and a run-recipe.sh --setup launcher from the spark-vllm-docker repo. The card says to start the worker before the head node.

One thing I'd change for chat use: the base card recommends clear_thinking: true for chat, while this config ships false. And reasoning_effort: max is the default, so drop it to high or low if latency matters more than the last benchmark point.

No Blackwell? GGUF in llama.cpp, or MLX on a Mac

llama.cpp is messier than it should be. The Unsloth repo still tells you to use Unsloth's PR #27754 or Unsloth Desktop. That PR was closed without merging on October 1, 2026, and a maintainer wrote: "Support is already in, but does not work with Unsloth GGUFs until they are updated." So before you download 200 GB, check whether the Unsloth files have been re-uploaded for upstream llama.cpp, or build the PR branch yourself.

Either way, read the caveats in that PR before you report garbage output: set NVIDIA_TF32_OVERRIDE=0 and use -fa off. The author says the TF32 flag moved top-1 agreement on a test fixture from 0.896 to 0.9995, and flash attention casts an F32 latent to F16, which MLA can't tolerate. The PR also adds the vision tower, so image input is covered.

bash

NVIDIA_TF32_OVERRIDE=0 llama-server -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL -fa off

With an 18B-active MoE, offloading experts to system RAM and keeping attention on a 24-32 GB card is the realistic budget path. It'll be slow, but it'll run.

On a Mac you have two options. MLX: it's a vision model, so use mlx-vlm 0.6.17 or newer, not mlx-lm. The 4-bit MLX build is ~204 GB on disk and wants 224 GB of RAM, so that's a 256 GB Mac Studio. Orcarouter's guide sends 128 GB machines to a ~102 GB "2bit-lite" build, after raising the wired-memory limit with sudo sysctl iogpu.wired_limit_mb=114688, and the model card itself warns that long code generation is unreliable at that precision.

llama.cpp on Metal is also an option now. Issue #29867 reported decode stalls of 1 to 9 seconds per token on an M5, blamed on the fused Lightning Indexer falling back to CPU. ggerganov couldn't reproduce it and pointed to a fix already on master, and the reporter then measured about 31 t/s at low context and 25 t/s at high context with no stalls. The issue is still open, but if you're on an old build, update before you blame the model.

Is the FP4 build worth buying Blackwell for?

The QAD work is the most honest thing in this release: NVIDIA trained the quant back toward BF16 behaviour and published the delta, which most GGUF releases can't say. But the "Local" label assumes you have two workstation Blackwell cards or two Sparks, and the checkpoint needs a community vLLM container, with two different image tags on the same model card. I'd call that a preview for the very small group that already owns the hardware.

If you have a 4090 and 192 GB of DDR5, UD-Q4_K_XL (200 GB against your 216 GB total) is the build you'll actually run this month, with very little room left for context. IQ4_XS at 157 GB is the saner pick on that box.

So, for local models: is FP4 with published accuracy numbers worth buying Blackwell for, or is a Q4 GGUF good enough for what you do? If you've run either build, tell me what tokens/sec you got and on what hardware.