vLLM/Recipes
NVIDIA

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16

NVIDIA Nemotron-3-Super Mamba-hybrid latent-MoE (~120B total / ~12B active) with BF16, FP8, and NVFP4 variants

View on HuggingFace
moe120B / 12B262,144 ctxvLLM 0.17.1+text
Guide

Overview

NVIDIA Nemotron-3-Super-120B-A12B is a hybrid-Mamba latent-MoE model (~120B total, ~12B active per token) trained for general reasoning, tool use, and agentic workflows. It supports a 1M-token context window and Multi-Token Prediction (MTP). Variants ship in BF16, FP8, and NVFP4 (Blackwell). A pre-RL Base BF16 checkpoint is also available for downstream fine-tuning.

Prerequisites

  • Hardware: 4-8x H100/H200/B200/RTX Pro 6000, or DGX Spark
  • vLLM >= 0.17.1
  • Docker with NVIDIA Container Toolkit (recommended)

Launch commands

Reference command from the vLLM blog (BF16, 4x H100, FP8 KV cache):

vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --served-model-name nemotron \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3

FP8 weights:

vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3

NVFP4 (Blackwell only):

vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
  --tensor-parallel-size 2 \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3

DGX Spark (GB10)

The NVFP4 variant runs on a single DGX Spark (GB10) at TP=1. A few notes:

  • Set --gpu-memory-utilization 0.85 to avoid an OOM at startup — the default 0.92 is too aggressive here. A slightly higher value (e.g. 0.87) may also work.

  • --max-model-len 131072 caps the context so the KV cache is sized appropriately for DGX Spark's memory; the model's full 262144 context would leave too little room.

  • Backends. On GB10 (SM121) auto selection already lands on FlashInfer, faster than the Marlin fallback:

    • linear_backend='auto' → FLASHINFER_CUTLASS
    • moe_backend='auto' → FLASHINFER_CUTLASS
    • --attention-backend='auto' → FLASHINFER

    This recipe additionally opts into the newer flashinfer_b12x CuteDSL NVFP4 kernels via --moe-backend flashinfer_b12x --linear-backend flashinfer_b12x. These are excluded from auto-selection on SM121 pending an upstream CUTLASS SM121 MMA-op guard, so they are experimental/opt-in — drop both flags to fall back to the auto FLASHINFER_CUTLASS path if you hit issues.

vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
  --trust-remote-code \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.85 \
  --moe-backend flashinfer_b12x \
  --linear-backend flashinfer_b12x \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3

RTX Pro 6000 NVFP4

The NVFP4 variant runs on a single RTX Pro 6000 at TP=1. Requires vLLM v0.24.0 or later. A few notes:

  • Set --gpu-memory-utilization 0.92 to avoid an OOM at startup. If you encounter an OOM, try adjusting the value.
  • Set VLLM_FLOAT32_MATMUL_PRECISION="high" to use high precision for the matmul operations.
  • Set VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 to allow the model to use the full context length.
# Use container: vllm/vllm-openai:v0.24.0-ubuntu2404

VLLM_FLOAT32_MATMUL_PRECISION="high" \
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
    --max-model-len 262144 \
    --gpu-memory-utilization 0.92 \
    --max-num-seqs 8 \
    --max-num-batched-tokens 16384 \
    --kv-cache-dtype fp8 \
    --mamba-cache-mode align \
    --mamba_ssm_cache_dtype float32 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
    --enable-prefix-caching \
    --reasoning-parser nemotron_v3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml

Benchmarking

vllm bench serve \
  --model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 \
  --trust-remote-code \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 1024 \
  --num-warmups 20 \
  --ignore-eos \
  --max-concurrency 1024 \
  --num-prompts 2048

References