NVIDIA's new 4B model scores 61.8 on SWE-bench Verified by NVIDIA's own count. That's higher than the 61 it lists for Qwen3.5-9B, a model more than twice its size. It also gets 10 on a knowledge benchmark where Qwen3.5-4B gets 15. And as of today, October 18, you can't run it on stock llama.cpp, vLLM or Transformers, because support for all three is still unmerged or missing.
That's NVIDIA Nemotron Labs Dory 1.0 4B. It showed up on Hugging Face on October 16 with a model card, a GGUF repo, and no NVIDIA blog post. The card doesn't link a paper either. This post covers what's new in it, how much RAM and VRAM it needs, and how to run Dory 4B locally right now with a llama.cpp PR build. At the end I go through where Ollama, vLLM and Transformers stand.
Dory is a looped model, so "4B" undersells the compute
Most small models run each token through a fixed stack of layers once. Dory has three parts:
- an input block of 8 layers
- a shared recurrent block of 20 layers that can run 1, 2, 3 or 4 times per token
- an output block of 8 layers
The input gets fed back in on every pass through the loop. You choose the loop count when you start the model, and you don't retrain anything. One loop is the cheapest, two is NVIDIA's recommended default ("Most of the accuracy gain from looping comes from the second loop"), and four is for agentic coding work like SWE-bench and Terminal-Bench.
With 4 loops a token goes through 8 + 80 + 8 = 96 layers, against 36 with 1 loop. "4B" is the size on disk. At 4 loops you pay per-token compute like a much deeper model's while still storing a small one.
The other specs from the card and config.json:
- normalized-GPT architecture, mixing sliding-window attention (1,024-token window) with full-attention layers, one full layer for every three sliding ones
- 256K context (the card lists 256K as validated; the vLLM command sets
--max-model-len 262144) - 131,072-token vocab, hidden size 2,560, 16 query heads, 4 KV heads, head dim 256
- about 6T pretraining tokens. The first ~4.5T were trained natively in NVFP4 and the rest in BF16, followed by 300B tokens of context extension, then SFT and GRPO-based RL across math, code, science and other environments (the RL ran on NeMo RL and NeMo Gym)
- English only, with 43 programming languages in training
- OpenMDW License v1.1. The card says it's "ready for commercial or non-commercial use"
The license deserves credit. OpenMDW grants rights to the "Model Materials", architecture and parameters included, "without restriction", and for a company choosing a 4B model that matters more than any single benchmark score.
The benchmarks: strong at agents, weak at facts
These are all NVIDIA's numbers from its own harness (NeMo Gym / NeMo Evaluator). The card's Artificial Analysis index chart is an internal estimate that, per NVIDIA, may deviate from official AA scores. I haven't found an independent reproduction of any of these yet.
| Benchmark | Dory 1 loop | Dory 2 loops | Dory 4 loops | Qwen3.5-4B | Qwen3.5-9B |
|---|---|---|---|---|---|
| SWE-bench Verified | 50.3 | 59.5 | 61.8 | 54.7 | 61 |
| Terminal-Bench 2.1 | 12.5 | 18.1 | 22.2 | 15 | 24 |
| IFBench | 74.29 | 77.94 | 79 | 53.83 | 59 |
| AA-LCR | 63 | 71 | 73 | 64.7 | 68.4 |
| GPQA Diamond | 69.19 | 72.41 | 72.16 | 77.71 | 80.93 |
| HLE (text, no tools) | 8.25 | 10.89 | 10.43 | 11.31 | 15.43 |
| AA-Omniscience | 10.03 | 10.08 | 10.03 | 15.02 | 18.05 |
AA-Omniscience, the last row, barely moves from 1 loop to 4 (10.03, 10.08, 10.03). Looping gives the model more steps to work through a problem, but it can't add facts that never fit into 4B parameters. GPQA and HLE show the same thing: Dory trails Qwen3.5-4B there at every loop count.
The tasks where it wins are instruction following (IFBench, 20+ points over Qwen3.5-4B), plus long-context reasoning (AA-LCR) and SWE-bench once you run 2+ loops. That's the profile you want in a local coding agent or a tool-calling worker, and a bad one for a pocket encyclopedia.
A few caveats on the comparison. The card also lists Gemma 4 26B A4B and Qwen3.6-35B-A3B, which are bigger models, and the Qwen3.6 SWE-bench score (73.4) is a reported figure that NVIDIA didn't run. The card has no Phi row and no Gemma model of the same size. If you were hoping for Dory against Phi-4-mini at equal compute, NVIDIA didn't publish that.
How much RAM and VRAM Dory 4B needs
The card doesn't give minimum memory, so here's what the files and config tell you.
| Format | Weights on disk | Where it runs |
|---|---|---|
| BF16 | 7.95 GB (model.safetensors) | Any GPU with ~10 GB+ free, plus KV cache |
| Q5_K_M GGUF | 3.67 GB (3,673,288,864 bytes) | Laptops, Apple Silicon, 6-8 GB GPUs |
| MXFP8 | 4.75 GB | Hopper/Blackwell. 1.69x to 1.84x faster than BF16 on DGX Spark at batch size 1 |
| NVFP4 | 3.20 GB | Hopper/Blackwell. 2.17x to 2.44x faster than BF16 on DGX Spark at batch size 1 |
The KV cache is where looping costs you. In the default per_loop mode, every pass through the recurrent block keeps its own cache. I worked this out from layer_types in config.json. There are 9 full-attention layers: 2 in the input block, 5 in the recurrent block, 2 in the output block. Each one stores 4 KV heads × 256 dims × K and V × 2 bytes = 4 KiB per token. With per_loop, the recurrent five are counted once per loop, so 2 loops means 14 full-attention caches: about 1.75 GiB at 32K context and about 14 GiB at the full 256K. At 4 loops it's 24 caches, around 24 GiB at 256K. The sliding-window layers only ever hold 1,024 tokens, so they barely count.
The fix is last_loop mode. It "keeps a single cache per recurrent attention layer" no matter how many loops you run, so you're back to 9 caches (about 9 GiB at 256K, about 1.1 GiB at 32K). By NVIDIA's account accuracy "stays close to the per-loop mode." If you're short on memory, set this first.
Run Dory 4B locally with the llama.cpp PR build
Support lives in llama.cpp PR #30306. It's a draft that needs two approvals and is still waiting on reviews from CISC and ggerganov. Someone asked on October 17 whether any blockers remain, with no answer so far. The PR's own description says Metal is untested and long-context qualification isn't finished. Oddly, the GGUF card's build instructions are for Metal on Apple Silicon, using a pinned commit from the author's fork:
git init llama.cpp-dory
cd llama.cpp-dory
git remote add origin https://github.com/brianchmielNV/llama.cpp.git
git fetch --depth 1 origin 98a74c97ea09ac72f6ceda1559808fc0a2f447ce
git checkout --detach FETCH_HEADThe card's full macOS cmake invocation (Metal, Apple BLAS, server only) is on the GGUF page. On Linux with an NVIDIA card I'd build the same commit the usual way (cmake -B build -DGGML_CUDA=ON). The PR reports 96/96 numerical checks passing on CPU and CUDA, but NVIDIA doesn't publish a Linux recipe.
The file in the repo is named NVIDIA-Nemotron-Labs-Dory1.0-4B-Q5_K_M.gguf, while the card's commands expect dory-q5_k_m.gguf, so rename it after downloading. Check the file, then start the server:
echo "96080be49ffd67b916018b7bdef8c484762576fb52780e10ab98ac4978a660e0 dory-q5_k_m.gguf" | shasum -a 256 -c -
./build/bin/llama-server \
-m ./dory-q5_k_m.gguf \
--alias dory \
--host 127.0.0.1 --port 8080 \
-ngl 999 \
--ctx-size 4096 --parallel 1 \
--batch-size 512 --ubatch-size 512 \
--cache-type-k f16 --cache-type-v f16 \
--flash-attn off \
--no-context-shift \
--jinja --reasoning on \
--temp 1.0 --top-p 0.95 --top-k 0 --min-p 0 --repeat-penalty 1 \
--override-kv "dory.num_loops=int:2,dory.recurrent_kv_cache_mode=str:per_loop"To trade accuracy for speed, change dory.num_loops (1 to 4). Change per_loop to last_loop to cap the cache. --no-context-shift is required because the PR doesn't support context shifting. When you hit 4,096 tokens the request stops, so raise --ctx-size if you have the memory.
For coding agents, the card says to pass "chat_template_kwargs": {"force_nonempty_content": True} in extra_body.
What about Ollama, vLLM and Transformers?
Hugging Face's Local Apps menu on the GGUF page offers ollama run hf.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF:Q5_K_M, but the card itself only documents the llama.cpp PR build and never mentions Ollama or loop overrides through it. I don't expect the one-liner to work yet. Ollama can only run architectures its engine supports, and the dory architecture isn't in upstream llama.cpp. The PR doesn't mention Ollama at all. Wait for the merge plus an Ollama release.
vLLM support is in PR #60149, which is open with no approvals yet. One maintainer asked for the model to go into Transformers first so vLLM doesn't have to churn its config. The author said there's no plan for a Transformers PR, and the maintainer settled for a best effort at Transformers config conventions. Separately, the author argued for keeping the PR native rather than going through vLLM's Transformers backend, and no vLLM maintainer has replied to that yet. If you build that branch, the card's serve command looks like this:
vllm serve --model nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 \
--hf-overrides '{"n_recurrent_loops": 2, "recurrent_kv_cache_mode": "per_loop"}' \
--max-model-len 262144 \
--max-num-seqs 128 \
--enable-prefix-caching \
--async-scheduling \
--reasoning-parser nemotron_v3 \
--tool-call-parser qwen3_coder \
--enable-auto-tool-choiceReasoning is always on in vLLM. If your GPU doesn't have room for 256K of per_loop cache, lower --max-model-len.
Transformers is the strangest part. The repo is tagged library_name: transformers, but the card has no Transformers snippet. config.json declares "model_type": "dory", the repo has no modeling_*.py, and as of today Transformers main has no models/dory directory (latest PyPI release: 5.20.1). In the vLLM thread the author said NVIDIA doesn't currently plan a Transformers PR, so I'd expect an unknown-architecture error on a current release. If AutoModelForCausalLM loads it for you, please tell me which version you're on.
Is it worth trying this week?
On a Mac with 8 GB of free memory, yes. Build the pinned commit, run 2 loops with last_loop, and point a small coding agent at it. The looping design is the most interesting thing to come out of a 4B release this year, and being able to change depth at launch time is something I want in every small model.
On anything else, I'd give it a couple of weeks. The benchmarks are self-reported, both runtime PRs could still change config keys, and the knowledge scores mean you'll still want Qwen3.5-4B for anything that relies on facts the model has to remember.
Which 4B model do you actually run locally right now, and does a looped model like Dory have a shot at replacing it? If you get the PR build running on CUDA, post your tokens/sec at 1 vs 4 loops. I haven't seen anyone publish those numbers yet.