Smaug-Nemotron-3.5-NVFP4: an agentic fine-tune that lives in 4 bits

Each adapted layer every step: the frozen base weight plus the low-rank update is rounded to the layer format and used by the forward pass; a zero-valued straight-through term carries the gradient back to A and B.

By Sandeep Singh Sandha and Arvind Sundararajan

We are happy to welcome a new member to the Smaug family – Smaug-Nemotron-3.5-NVFP4

We started the Smaug line with the original Smaug-72B that topped the Open LLM Leaderboard in early 2024, and more recently we have released a few agentic fine-tunes of various leading models. Smaug-Nemotron-3.5-NVFP4 continues that line – with a twist – it is our first model to be fine-tuned in NVFP4.

A week ago, NVIDIA released Nemotron 3.5 Super VL, a 120B-parameter hybrid Mamba-2 / MoE / Attention VL model with 12B active parameters, a 1M-token context, and text, image, and video input. Along with the BF16 weights, NVIDIA published an NVFP4 checkpoint, produced with Model Optimizer and quantization-aware distillation from the BF16 model, that runs on one DGX Spark, one H200, or one GB200. A 120B model with a million tokens of context on a desk-side box is a big deal, so we set out to apply our very successful agentic fine-tuning to it.

A big model in 4 bits beats a small model in 16 or 8

That a large quantized model beats a small full or half precision one of the same size in bytes is now well understood:

Dettmers and Zettlemoyer ran more than 35,000 experiments across model families from 19M to 176B parameters and found that, for a fixed budget of total model bits, 4-bit precision is almost universally optimal for zero-shot accuracy: a 60B model at 4 bits beats a 30B model at 8 bits (The case for 4-bit precision: k-bit inference scaling laws, ICML 2023).
NVFP4 from NVIDIA, bakes quantization into the hardware so that weights stored in FP4 can share a FP8 scale per block of 16 values and an FP32 scale per tensor. This lets Blackwell GPUs run 4-bit matrix multiplications natively while keeping the accuracy loss of plain FP4 to a minimum. (Introducing NVFP4 for efficient and accurate low-precision inference).
The format holds up at scale. NVIDIA pretrained a 12B model on 10 trillion tokens in NVFP4 and matched an FP8 run to within about 1.5% in loss (Pretraining Large Language Models with NVFP4). For Nemotron 3.5 Super VL itself, NVIDIA reports an APEX shortlist score of 89.36 for the NVFP4 checkpoint against 88.36 for BF16.

In practice, for a 120B model, the 240 GB required for a BF16 mode shrinks to 75GB in NVFP4 – letting it fit on a single H200 or the 128 GB of a DGX Spark.

Two ways to fine-tune a 4-bit model

We considered two routes.

Fine-tune in BF16, quantize afterwards. Take our BF16 fine-tune (Smaug-Nemotron-3.5) and round it to NVFP4 at the end.
Fine-tune the NVFP4 checkpoint itself. Start from NVIDIA’s 4-bit weights and never leave them.

The first route is the usual one, and it fails here for a specific reason. LoRA fine-tuning keeps changes tiny and surgical, so that reasoning, instruction following, long context, and vision are untouched. A change that small disappears in a 4-bit rounding step. When the learned update is added to the base weights and rounded into the shipped format, most of it simply vanishes. Kumar et al. make the general version of this point: post-training quantization degrades a model more the better trained it is (Scaling Laws for Precision, ICLR 2025), and a fine-tune is exactly a small, well-trained delta.

Fine-tuning in BF16 and rounding afterwards erases a small update; merge-aware training keeps it because every step is computed on the rounded weights that ship.

So we took the second route, with a custom loss designed for it. First, the results:

Evaluation results

Smaug-Nemotron-3.5-NVFP4 improves on the NVFP4 base across the agentic coding benchmarks it was trained for, and holds reasoning, coding, instruction-following, long-context and vision scores at parity.

Score change against the NVFP4 base on all eleven benchmarks: every agentic coding benchmark improves, general benchmarks sit near the base.

	Smaug-Nemotron-3.5-NVFP4	Nemotron 3.5 Super VL NVFP4 (base)	Nemotron 3.5 Super VL BF16
Agentic coding
SWE-bench Verified	71.8	71.1	71.1
SWE-bench Multilingual	69.0	67.0	67.7
Terminal-Bench 2.1	64.0	62.8	60.7
DeepSWE	21.2	18.2	24.7
NL2RepoBench	30.2	28.3	30.2
Reasoning
GPQA Diamond	90.1	89.9	89.5
Coding
LiveCodeBench v6	88.2	87.4	88.2
SciCode (subtask)	48.4	46.6	47.1
Instruction following
IFBench	70.7	71.5	70.9
Long context
AA-LCR v1.1	69.0	67.7	68.6
Vision
V*	86.5	86.0	85.6

Higher is better; bold is the best score in the row. Details in the model card.

On four of the five agentic benchmarks the 4-bit fine-tune even matches or beats NVIDIA’s 16-bit base.

On five agentic coding benchmarks Smaug-Nemotron-3.5-NVFP4 moves past its NVFP4 base; the tick marks the BF16 base.

Custom loss for 4-bit weights

Our training loop is configured so that the model learns only from its tool calls (actions), while the loss holds thinking token generation close to the base. The new idea here is that the loss always sees the model exactly as it will be served. We call this merge-aware training.

Train on the served weights

Each layer that we are adapting, has a base weight W, and a small low-rank update sBA, with trainable factors A and B, and a fixed scale s. The forward passes use the weight that will be released, rounded by Q_fmt to the layer’s own format (BF16, or FP8 with the per-tensor scale kept frozen):

W′ = Q_fmt(W + sBA)

Rounding has no gradient, so for a layer input x, each layer adds a correction built from its low-rank path ℓ(x). It is zero in value, but carries the low-rank gradient, with sg the stop-gradient:

h = x·W′ᵀ + (ℓ(x) − sg[ℓ(x)]), ℓ(x) = s · sg[x] · Aᵀ Bᵀ

The layer output h equals x·W′ᵀ, exactly what the release serves, while A and B train in full precision. Small progress updates that change a stored value build up until they cross a rounding boundary, and the loss sees it then. Layers in coarser formats get larger learning rates.

One weight during training: progress accumulates in full precision and the stored, served value changes only when it crosses a rounding boundary.

The trained model π_θ is the whole network run this way, with the serving engine’s numerics (FP8 KV cache, FP8 and NVFP4 activations, 4-bit experts). The reference model π_ref is the same network with B = 0, the base model.

Learn from tool calls while holding the thinking in place

ℒ(θ) = −(1/|𝒜|) Σ_t∈𝒜 log π_θ(y_t | y_<t) + β (1/|𝒯|) Σ_r |𝒯_r| k₃(R_r)

The loss has two terms.

Actions. The usual next-token loss on each token y_t, applied only to the action tokens 𝒜. Thinking tokens get no likelihood term of their own.
Thinking. A penalty on drift in the reasoning. For each trajectory r, R_r is the average gap log π_ref − log π_θ over its thinking tokens 𝒯_r; it is zero when the model on average rates that reasoning as the base does. The penalty k₃(R) = eᴿ − R − 1 is never negative, zero only at R = 0, close to R²/2 for small drift and exponential once the model starts rating its own reasoning below the base. Trajectories count in proportion to their share of all thinking tokens, and β sets the penalty’s strength.

A training trajectory: action tokens carry the next-token loss, thinking tokens carry only the drift penalty; below, the penalty k₃ against the drift R.

Release what we train

Training starts from the base model itself: with B = 0, W′ = W and the thinking penalty is exactly zero. And what ships is what was trained: the released checkpoint stores exactly the W′ the loss was computed on. Only the language model was adapted. The C-RADIO vision encoder, the vision-language connector and the MTP head are identical to NVIDIA’s release, and so is the NVFP4 quantization layout.

Block map of the model: vision encoder, connector and MTP head are unchanged; only the hybrid language model is adapted.

Download it today

The weights are published on Hugging Face (under the OpenMDW 1.1 license inherited from the base): abacusai/Smaug-Nemotron-3.5-NVFP4 along with our BF16 version at abacusai/Smaug-Nemotron-3.5 – the models are fully compatible with NVIDIA compatible tooling and you can use it as a drop-in replacement for the corresponding base models in vLLM, SGLang, or TensorRT-LLM.

The model card has the full vLLM command. Use temperature 1.0 and top-p 0.95, and keep past reasoning in the history for multi-turn tool use (truncate_history_thinking: false).