How to Run Mage-VL-Cowork Locally: Microsoft's 4B Agent Beats Sonnet 4.5

Microsoft's Mage-VL-Cowork is a 4B computer-use model that scores 66.0% on OSWorld-Verified. That's ahead of Claude Sonnet 4.5 (62.9%, and Sonnet got twice the step budget) and of an open 72B model that manages 44.9%. The BF16 weights are 9.48 GB and the license is Apache-2.0, so a 16 GB GPU is enough to run it locally. A 4-bit GGUF fits in 8 GB. Before you point it at anything, though, read the part of the model card that tells you to keep it in a disposable VM.

What Mage-VL-Cowork is (and how it differs from Phi vision)

Microsoft's earlier small VLMs were general models. Phi-4-multimodal is about 5.6B params, takes text, images and audio, has a 128K context and ships under MIT. You ask it about an image and it answers.

Mage-VL-Cowork does a different job. Every turn has the same output format: a block, a one-line Action: summary, then a . There are two tool channels:

• computer_use: 11 GUI primitives with coordinates normalized to [0, 999]
• bash: arbitrary shell commands

The project page's tagline is "One 4B policy that knows when to click—and when to code." That claim holds up best in the training mix: 68.19% of the 318,837 hybrid trajectories are GUI tool calls and 31.81% are terminal calls. The model learned to stop dragging the mouse around LibreOffice when a Python one-liner will do the job.

It's also not a Phi descendant. The language backbone is Qwen3-4B-Instruct-2507 and the vision encoder is Microsoft's own Mage-ViT. The closer Microsoft relative is Fara-7B (Qwen2.5-VL base, MIT, released November 2025), which is built for web tasks.

Mage-VL-Cowork specs: 4B or 4.7B parameters?

| | Mage-VL-Cowork | Phi-4-multimodal | Fara-7B |
|---|---|---|---|
| Params | 4B (card), 4.74B (Hub metadata) | ~5.6B | 7B |
| Backbone | Qwen3-4B-Instruct-2507 + Mage-ViT | Phi-4-mini | Qwen2.5-VL-7B |
| Context | 32,768 active (262,144 max in config) | 128K | 128K |
| License | Apache-2.0 | MIT | MIT |
| Job | Desktop + terminal agent | General multimodal | Web agent |

About the mismatch: the card says 4B, but the Hub's safetensors metadata counts 4.74B parameters. That matches 9.48 GB divided by 2 bytes per BF16 weight. The config has 36 text layers, hidden size 2560, and 8 KV heads. The vision side is 24 layers at 1024 hidden.

The context number trips people up. The architecture supports 262,144 tokens, but Microsoft trained and evaluated at 32,768 and runs a sliding window that keeps the 3 latest screenshots and 4 full assistant turns. Older turns get folded into their Action: lines. Don't push it to 262K and expect it to behave.

OSWorld-Verified: where 66.0% actually sits

The headline number is OSWorld-Verified: 361 tasks with Google Drive tasks excluded and a 50-step budget. The 66.0% is the mean of two runs, 65.33% and 66.75%.

| Model | Params | OSWorld-Verified |
|---|---|---|
| GPT-5.4-mini | — | 69.6 |
| ScaleCUA-Qwen3.5-9B | 9B | 68.7 |
| Mage-VL-Cowork | 4B | 66.0 |
| Muse Glimmer-30B | 30B | 65.9 |
| Qwen3.6-27B | 27B | 63.9 |
| Claude Sonnet 4.5 (100 steps) | — | 62.9 |
| OpenCUA-72B | 72B | 44.9 |
| Fara1.5-4B | 4B | 13.9 |

The frontier models are still well ahead. GPT-5.5 is at 78.7 and Claude Opus 4.7 at 78.0, and the card's human reference is 72.4. The 66.0 mostly shows how much a small model gets out of a task-specific curriculum. The RL stage alone took it from 55.84% to 66.04% while cutting output tokens per task from 7,374 to 5,499.

Two things I'd flag before anyone quotes these numbers in a pitch deck:

How much VRAM Mage-VL-Cowork needs

The card does not state a minimum VRAM. Here's what the published files and config let you work out.

Start with the weights. BF16 is 9.48 GB across 2 safetensors shards. The community Q4_K_M GGUF is 2.72 GB plus a 0.35 GB Q8_0 mmproj. The MLX 4-bit build is about 3.14 GB.

Then the KV cache. I worked this out from config.json and haven't measured it: 36 layers × 8 KV heads × 128 head dim × 2 (K and V) × 2 bytes comes to about 144 KB per token. At the full 32,768 window that's roughly 4.8 GB in FP16.

| Setup | Weights | + KV at 32K | Card to plan for |
|---|---|---|---|
| BF16 (Transformers/SGLang) | 9.48 GB | ~4.8 GB | 16 GB |
| Q8_0 GGUF + mmproj | 5.05 GB | ~4.8 GB | 12 GB |
| Q4_K_M GGUF + mmproj | 3.07 GB | ~4.8 GB | 8 GB |
| MLX 4-bit (Apple Silicon) | 3.14 GB | — | 16 GB unified memory Mac |

Screenshots are the other big cost. Microsoft recommends SGLANG_MAX_PIXELS=4000000 so a 1920×1080 capture keeps its full detail, which means a lot of vision tokens per turn. If you're tight on memory, lower the screenshot resolution before you drop to Q2_K. Grounding is what I'd expect quantization to break first. In the MLX port's tiny 3-task demo, the 4-bit build missed a Dark Mode toggle by 42px while the BF16 build hit it.

Which runtimes load it today: SGLang, Transformers, vLLM, llama.cpp, Ollama, MLX

Transformers

Works, needs trust_remote_code=True:

pip install "transformers>=4.57.3" accelerate pillow torch torchvision qwen-vl-utils

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

processor = AutoProcessor.from_pretrained("microsoft/Mage-VL-Cowork", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "microsoft/Mage-VL-Cowork", trust_remote_code=True,
    torch_dtype=torch.bfloat16, device_map="auto",
).eval()

Use the card's sampling settings (temperature 0.7, top_p 0.8, max_new_tokens 4096) and its full MAGE_HYBRID_SYSTEM_PROMPT. That prompt is where the tool schemas and the reply format are defined, so don't skip it.

SGLang

This is the path the card's Quick Start recommends, and the one I'd pick:

python -m sglang.launch_server --model-path microsoft/Mage-VL-Cowork \
  --trust-remote-code --dtype bfloat16 --context-length 32768

There's also a prebuilt image, xiaoyizhang/mage-vl-cowork:0.1.0 (SGLang 0.5.2, PyTorch 2.8.0+cu128). Screenshots go through the native POST /generate endpoint with image_data.

vLLM

The Hub's "Use this model" panel suggests vllm serve "microsoft/Mage-VL-Cowork", but the card doesn't test that path. With a custom llava_onevision2 architecture, I wouldn't count on it until someone posts a working run.

llama.cpp

Only through a fork. The GGUF repo points to ch686/llama.cpp-locateanything-magevl at commit e07bdc8, and the upstream PR (#26796) was closed without being merged. Loading the text side is easy enough; the fork exists because of the custom Mage-ViT projector.

Ollama and LM Studio

The Hub lists ollama run hf.co/Hansda/Mage-VL-Cowork-GGUF:Q4_K_M, but the GGUF author says outright: "Stock llama.cpp, Ollama and LM Studio have not been validated for this projector." Maybe it works fine. For an agent that has to click within a few pixels, though, I'd treat it as broken until someone posts grounding results.

MLX on Apple Silicon

Works with mlx-vlm 0.7.11, which has a built-in mage_vl model:

pip install -U mlx-vlm
python -m mlx_vlm.generate --model pratikgx/Mage-VL-Cowork-MLX-4bit \
  --image screenshot.png --prompt "Turn on Dark Mode." --max-tokens 512

None of the community quants has an OSWorld score. The only published numbers are for Microsoft's BF16 checkpoint.

The part nobody reads: Microsoft's sandbox rules

The card isn't vague about deployment. Run the agent inside an isolated container or VM with no access to host filesystems, local networks or personal environment variables. Require human confirmation before irreversible actions like external submissions, account logins, payments and privilege changes. Don't mount SSH keys, cloud tokens, browser cookies or password stores. Restrict outbound network access when the model handles untrusted web pages.

So the "cowork" fantasy, a local model watching your real desktop and helping out, is exactly what Microsoft tells you not to do. The model has a bash channel, and a web page with injected instructions is just more pixels to it. A local model keeps your data off someone else's server, but it doesn't stop the model from running rm on your files.

What I'd do with it today: spin up a snapshot-reverted Ubuntu VM with no credentials in it, serve the model with SGLang on a 16 GB card, and use the osworld_eval/ harness in the repo as a starting point for your own agent loop. That setup gives you a 4B agent that beats several 27–72B open models on OSWorld-Verified, and the worst it can do is wreck a VM you'll throw away anyway.

Would you let a local vision model watch your screen and act on it, as long as nothing leaves your machine? Or does a bash tool make "local" irrelevant? Tell me where you draw the line.

Sources

• microsoft/Mage-VL-Cowork model card and config.json
• Mage-VL-Cowork project page
• Hansda/Mage-VL-Cowork-GGUF
• pratikgx/Mage-VL-Cowork-MLX-4bit
• microsoft/Phi-4-multimodal-instruct
• microsoft/Fara-7B