Gemma 4 31B NVFP4 + MTP on Dual RTX PRO 6000 Blackwell — Optimized vLLM Setup

Optimized vLLM configuration for nvidia/Gemma-4-31B-IT-NVFP4 on two RTX PRO 6000 Blackwell GPUs (no NVLink), with Gemma 4's native Multi-Token Prediction (MTP) drafter (google/gemma-4-31B-it-assistant) enabled.

NVFP4 — Blackwell (GB202) is the first GPU architecture with native FP4 Tensor Cores. NVFP4 is not a quality compromise: it is the format the hardware is designed to run at peak throughput. On Blackwell, NVFP4 GEMMs run at 2× the throughput of BF16. On any pre-Blackwell GPU, NVFP4 would require emulation and offer no benefit; here it is the correct operating mode.

MTP (Multi-Token Prediction) — Gemma 4 ships a dedicated ~0.5B drafter model (google/gemma-4-31B-it-assistant) that predicts multiple tokens ahead in parallel. With 4 speculative tokens, we measured a 96.5% acceptance rate and an acceptance length of 4.86 — meaning nearly every draft is fully accepted. The result is substantially higher throughput with no change to output quality, since rejected tokens are discarded and the main model's output is always authoritative.

This repo documents the working vLLM image, the non-obvious fixes required to get NVFP4 + MTP running on Blackwell, the GRUB flags needed for CUDA P2P on AMD, and benchmark results from sweeping speculative tokens, max concurrent sequences, batch sizes, and prefix cache effectiveness.

Hardware

CPU / platform	AMD Ryzen 9 7950X — AM5 socket
System RAM	2× 32 GB DDR5-6000 (64 GB dual-channel)
GPU 0	RTX PRO 6000 Blackwell Max-Q Workstation Edition (GB202GL, 96 GB GDDR7)
GPU 1	RTX PRO 6000 Blackwell Workstation Edition (GB202GL, 96 GB GDDR7)
Total VRAM	192 GB
GPU interconnect	Bifurcated PCIe 5.0 x8 — PHB topology, ~32 GB/s theoretical, ~24 GB/s measured CUDA P2P
CUDA compute capability	12.0 (Blackwell GB202)
vLLM image	vllm/vllm-openai:cu129-nightly (built May 9 2026, contains Gemma4 MTP PR #41745)

The Max-Q variant has a lower TDP and the same memory capacity as the full variant, but may have lower memory bandwidth in practice: our measurements show GPU0 (Max-Q) drawing ~295W vs GPU1's ~385W under decode load — a 30% power difference on the same die that likely implies reduced GDDR7 clock speeds. Both share the same die (GB202GL) and CUDA capability, so they behave identically to vLLM and NCCL for correctness; the power/clock difference may cause step-time asymmetry (GPU0 being the slower half).

PCIe bifurcation is handled at the CPU/platform level on AM5 — no discrete bifurcation card required. The two x16 slots are each wired at x8 from the CPU, giving each GPU a full PCIe 5.0 x8 lane (~32 GB/s theoretical per direction). Measured CUDA P2P throughput is ~24 GB/s: the gap is due to the PHB (host bridge) topology — GPU-to-GPU traffic traverses the CPU's root complex rather than a direct peer link, which adds latency and reduces sustained throughput below the raw link rate.

---

Host setup: GRUB flags for CUDA P2P

On AMD platforms, CUDA peer-to-peer memory access (required for NCCL all-reduce between the two GPUs) needs specific kernel parameters. Without these, NCCL cannot establish direct GPU-to-GPU transfers and falls back to a much slower CPU-bounce path.

Step 1 — GRUB kernel parameters. Add to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub:

iommu=pt amd_iommu=off pcie_aspm=off pci=realloc=off

sudo update-grub

Step 2 — Disable IOMMU in the NVIDIA UVM module. The GRUB flags set global IOMMU policy, but the nvidia_uvm kernel module has its own IOMMU handling that must be disabled separately:

echo "options nvidia_uvm uvm_disable_iommu=1" | sudo tee /etc/modprobe.d/nvidia-uvm.conf

Step 3 — Reboot (applies both changes):

sudo reboot

Flag	Reason
iommu=pt	Puts AMD IOMMU in passthrough mode — required for CUDA P2P DMA between GPUs
amd_iommu=off	Disables AMD IOMMU translation entirely; iommu=pt alone is sometimes insufficient for P2P
pcie_aspm=off	Disables PCIe Active State Power Management; ASPM can throttle inter-GPU bandwidth
pci=realloc=off	Prevents the kernel from reallocating PCIe BARs on boot, which can break GPU addressing

Security note: iommu=pt and amd_iommu=off disable the hardware memory isolation that normally prevents PCIe devices from performing unauthorized DMA reads and writes to system memory. On a dedicated workstation this is a reasonable trade-off, but it does mean a rogue or compromised PCIe device (including anything plugged into an exposed slot) could access arbitrary physical memory. Do not apply these flags on shared, multi-tenant, or internet-facing machines.

To verify P2P is working after boot:

import torch
print(torch.cuda.can_device_access_peer(0, 1))  # should be True
print(torch.cuda.can_device_access_peer(1, 0))  # should be True

And confirm NCCL uses CUDA P2P, not socket/network (run inside the container):

import os, torch, torch.distributed as dist, torch.multiprocessing as mp

def worker(rank, world_size):
    os.environ.update({"MASTER_ADDR": "localhost", "MASTER_PORT": "29500",
                       "NCCL_DEBUG": "INFO", "NCCL_DEBUG_SUBSYS": "ALL"})
    dist.init_process_group("nccl", rank=rank, world_size=world_size)
    t = torch.ones(1024*1024, device=f"cuda:{rank}")
    dist.all_reduce(t)

mp.spawn(worker, args=(2,), nprocs=2, join=True)

Look for isAllDirectP2p 1 and isAllCudaP2p 1 in the output. On this system:

Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1
Pattern: type PHB/PIX, bw 24.000000/24.000000

Model Downloads

Both models are gated. Accept the license terms on Hugging Face for each before downloading.

• nvidia/Gemma-4-31B-IT-NVFP4 — main model (~18 GB)
• google/gemma-4-31B-it-assistant — MTP drafter (~1 GB)

huggingface-cli login   # paste your HF token when prompted

huggingface-cli download nvidia/Gemma-4-31B-IT-NVFP4
huggingface-cli download google/gemma-4-31B-it-assistant

Models are cached to ~/.cache/huggingface by default, which is the path mounted into the container by the run command below.

---

Optimized Run Command

docker run --gpus '"device=0,1"' \
           --name gemma4-31b-it \
           --ipc=host \
           -p 8000:8000 \
           -v ~/.cache/huggingface:/root/.cache/huggingface \
           -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
           -e CUDA_VISIBLE_DEVICES=0,1 \
           -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
           -e VLLM_NVFP4_GEMM_BACKEND=cutlass \
           -e SAFETENSORS_FAST_GPU=1 \
           -e OMP_NUM_THREADS=12 \
           vllm/vllm-openai:cu129-nightly \
    nvidia/Gemma-4-31B-IT-NVFP4 \
    --quantization modelopt_fp4 \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 175000 \
    --max-num-seqs 32 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --kv-cache-dtype auto \
    --tool-call-parser gemma4 \
    --reasoning-parser gemma4 \
    --enable-auto-tool-choice \
    --disable-custom-all-reduce \
    --speculative-config '{"model": "google/gemma-4-31B-it-assistant", "num_speculative_tokens": 4}' \
    --host 0.0.0.0 \
    --port 8000

Flag reference

Flag	Reason
--quantization modelopt_fp4	NVFP4 — native Blackwell FP4 Tensor Cores, 2× BF16 throughput
--tensor-parallel-size 2	Splits model across both GPUs
--gpu-memory-utilization 0.9	10% headroom; expandable_segments handles allocation spikes
--max-model-len 175000	Practical limit chosen to fit within available VRAM — Gemma 4's maximum is 256K tokens but KV cache growth makes the full window expensive; 175K is the sweet spot for this hardware
--max-num-seqs 32	+18% throughput vs default 16; GPU was under-utilised at 16
--enable-chunked-prefill	Prevents long prefills from starving decode steps
--enable-prefix-caching	7.5× TTFT reduction for repeated context (see results below)
--kv-cache-dtype auto	vLLM resolves auto to fp8_e4m3 for this model; uncalibrated scale=1.0 — verified acceptable quality for coding use
--tool-call-parser gemma4	Gemma4-specific tool call parsing
--reasoning-parser gemma4	Gemma4 thinking/reasoning output
--enable-auto-tool-choice	Automatic tool selection
--disable-custom-all-reduce	Required: custom CUDA all-reduce kernel does not support cc 12.0 (Blackwell)
--speculative-config	4-token MTP drafter — 96.5% acceptance rate, +8% throughput vs 1-token spec (552 vs 510 tok/s at max-num-seqs=16); no 0-token baseline was measured
VLLM_NVFP4_GEMM_BACKEND=cutlass	Activates optimised CUTLASS NVFP4 kernel path
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True	Reduces CUDA OOM from allocator fragmentation
OMP_NUM_THREADS=12	Limits CPU thread contention from torch CPU parallelism

---

Image selection

Four images were tested. Only cu129-nightly works for this hardware + model combination:

Image	Result
vllm/vllm-openai:cu130-nightly	custom_all_reduce crash on cc 12.0 — no workaround in this build
vllm/vllm-openai:gemma4-0505-cu130	NVFP4 + MTP drafter weight shape mismatch — predates the MTP PR fix
vllm/vllm-openai:v0.20.2-cu129	NotImplementedError: Speculative Decoding with draft models does not support multimodal models yet — stable release missing MTP PR #41745
vllm/vllm-openai:cu129-nightly	Works — MTP merged to main post-May 6 2026

The cu129-nightly tag is moving. Pin to a digest for reproducible deploys once it stabilises.

---

Non-obvious fixes

--disable-custom-all-reduce — vLLM's custom all-reduce CUDA kernel (custom_all_reduce.cuh) does not support compute capability 12.0. The flag falls back to NCCL, which fully supports Blackwell. Performance impact is negligible: NCCL uses direct CUDA P2P (isAllDirectP2p 1, isAllCudaP2p 1, PHB topology) at ~24 GB/s measured — the expected throughput for PCIe 5.0 x8 via a host bridge, and optimal for this interconnect.

--kv-cache-dtype auto — vLLM resolves auto to fp8_e4m3 for this model (visible in the startup log: kv_cache_dtype=fp8_e4m3). The checkpoint has no pre-calibrated KV scaling factors, so vLLM uses scale=1.0. Quality is acceptable for coding use. If you observe quality regressions on sensitive tasks, switch to bfloat16 explicitly.

Entrypoint — The image ENTRYPOINT is ["vllm", "serve"]. Pass the model as a plain positional argument; prepending vllm serve doubles the command.

--speculative-config JSON — Gemma4 MTP uses a separately loaded draft model (google/gemma-4-31B-it-assistant). The older --num-speculative-tokens flag alone does not configure the drafter; the full JSON is required.

---

Benchmark results

Speculative decoding sweep (1–4 tokens)

Tested with --max-num-seqs 16, --max-concurrency 1 (single-request latency profile):

Spec tokens	tok/s	Mean TTFT	Acceptance rate	Accept length
1	510	3420ms	99.2%	1.99
2	499	3965ms	97.9%	2.96
3	530	4219ms	97.4%	3.92
4	552	4939ms	96.5%	4.86

4 tokens wins: throughput increases monotonically (spec=2 is noise), acceptance length 4.86 shows nearly every draft fully accepted. Per-position rates: 99% / 97% / 96% / 95% — high throughout.

8 tokens was tested and catastrophic: throughput collapsed to 24 tok/s (97% drop). The verification batch overwhelms the scheduler before acceptance rate can compensate.

max-num-seqs sweep (spec_tokens=4)

max-num-seqs	tok/s	Mean TTFT	Acceptance
16 (default)	552	4939ms	96.5%
32	654	3474ms	95.9%
64	663	3442ms	95.0%

32 is a significant win (+18% throughput, lower TTFT, only 0.6% acceptance drop). The GPU was under-utilised at 16 sequences. 64 offers diminishing returns (+1.4%) with further acceptance rate erosion.

--max-num-batched-tokens sweep (175k context, spec_tokens=4)

Tested with --max-concurrency 1, inputs of 512 / 2048 / 8192 tokens:

batch	512-tok TTFT	2048-tok TTFT	8192-tok TTFT	tok/s
8192 (default)	100ms	303ms	1209ms	173
16384	100ms	304ms	1207ms	173
32768	98ms	303ms	unstable	175
65536	103ms	306ms	1210ms	167 ↓

No meaningful effect up to 32768 (inputs fit in a single chunk). 65536 actively hurts throughput by stalling decode steps. Use the vLLM default (8192); do not set this flag.

Prefix cache effectiveness

4096-token input, 4 concurrency, 30 requests. High sharing = 87.5% shared prefix (simulates resending the same codebase context per turn). Low sharing = 25%.

Scenario	Mean TTFT	Median TTFT	tok/s	Token hit rate
High sharing (87.5%) — cold	777ms	251ms	303	83.9%
High sharing — warm	104ms	103ms	500	91.2%
Low sharing (25%) — cold	726ms	531ms	272	68.7%
Low sharing — warm	100ms	101ms	496	76.2%

7.5× TTFT reduction and +65% throughput on warm cache. Both sharing levels converge to ~100ms warm TTFT — once the cache is populated, TTFT is dominated by KV block lookup + first decode step, not prefill. For an iterative coding workflow (edit → question → edit), the cache warms on the first request for any given context and every subsequent turn benefits.

Raw result JSON files are in results/prefix-cache/.

Cross-GPU PCIe traffic: inference vs idle

nvidia-smi dmon -s tup captures PCIe RX/TX throughput, SM utilisation, and power per GPU. On this PHB topology system, GPU-to-GPU NCCL all-reduce traffic routes through the PCIe switch and is visible in these counters (unlike NVLink, which bypasses the PCIe bus and would not appear here).

What is being transferred

Every all-reduce is a direct card-to-card transfer between the two GPUs. GPU0 sends its partial result to GPU1 and GPU1 sends its partial result to GPU0 simultaneously; both then compute the element-wise sum and continue to the next layer. The host CPU and system RAM are not involved — it is DMA directly between the two GPU framebuffers through the PCIe fabric.

The physical route on this hardware is:

GPU0 → PCIe 5.0 x8 → CPU root complex (PHB) → PCIe 5.0 x8 → GPU1
                      (AMD Ryzen 9 7950X)

This is why the traffic appears in nvidia-smi's per-GPU PCIe counters at all. On NVLink systems, GPU-to-GPU transfers have their own dedicated bus and are invisible to those counters. Here there is no NVLink, so both cards communicate through the CPU's PCIe root complex. That relay is also the reason measured P2P throughput is 24 GB/s rather than the full 32 GB/s theoretical: the PHB adds latency and reduces sustained bandwidth compared to a direct peer link.

Measured with 2048-token input / 512-token output, --request-rate inf, 300 requests:

Phase	GPU0 rxPCIe	GPU0 txPCIe	GPU1 rxPCIe	GPU1 txPCIe	SM util	Power
Idle (server ready, no requests)	0–28 MB/s (~9 MB/s avg)	~2 MB/s	~1 MB/s	~1 MB/s	3% / 0%	93W / 92W
Prefill (2048-tok inputs, chunked 8192)	11–15 GB/s	10–16 GB/s	11–16 GB/s	11–14 GB/s	100% / 100%	~250W / ~350W
Decode (512-tok generation, steady-state)	6–9 GB/s	6–9 GB/s	6–9 GB/s	6–9 GB/s	96% / 96%	~295W / ~385W
Model loading (startup only, not captured)	host→GPU burst	—	host→GPU burst	—	low	low

Key observations:

• Prefill is ~1.5–2× heavier than decode. The all-reduce tensor per layer scales with tokens in flight: batch_tokens × hidden_size (5376) × BF16. A full 8192-token chunked prefill batch generates tensors ~51× larger than a full decode batch (32 sequences × 5 spec tokens = 160 tokens). Despite that size difference, sustained bandwidth is only 1.5–2× higher because each prefill step takes proportionally longer to compute — the all-reduce happens once per step regardless of step duration.
• Compute-limited, not PCIe-limited. Peak decode utilisation is ~7–9 GB/s against a 32 GB/s theoretical PCIe budget — 22–28% utilisation. SM stays at 96–100%. PCIe latency adds to TPOT (two all-reduce round-trips per layer), but bandwidth is not the bottleneck — there is headroom for higher concurrency or longer sequences.
• Model loading is a one-time unidirectional burst (host RAM → each GPU) that finishes at startup. Inference is the sustained ongoing cost.
• GPU1 runs ~100W hotter than GPU0 during decode (~385W vs ~295W). Both are the same die (GB202GL) — this reflects the Max-Q TDP ceiling on GPU0. If GPU0 is the throughput bottleneck due to lower clocks, swapping physical GPUs (or adjusting power limits) may reduce step-time jitter.

Raw sample data is in results/pcie/. The phase labels in results/pcie/active.txt (# --- Prefill-heavy ---, # --- Decode-heavy ---) were added by hand after collection; bench-pcie.sh produces unlabelled output. The script's summary averages all active samples together, blending both phases — compare against the per-phase rows in the table above for interpretation.

Why PCIe utilisation is low (22–28%) — expected behaviour

The measured 7–9 GB/s is correct and expected. The bottleneck chain at this concurrency level is:

GDDR7 bandwidth → compute FLOPS → PCIe latency → PCIe bandwidth (never reached)

All-reduce message size scales with active batch tokens. Each all-reduce carries batch_tokens × hidden_size (5376) × BF16. With 32 sequences × 5 spec tokens = 160 tokens that is 1.72 MB per all-reduce. But with chunked prefill interleaving, many of those 32 slots are mid-prefill at any given moment — if 8–10 sequences are in active decode, the message drops to ~420–525 KB, which produces exactly the 7–9 GB/s measured.

The GPU is GDDR7-bandwidth-bound, not PCIe-bandwidth-bound. At small batch sizes, the dominant cost per forward pass is loading model weights from GDDR7. The RTX PRO 6000 Blackwell uses a 512-bit GDDR7 bus at 1792 GB/s. NVFP4 weights are ~7.75 GB per GPU:

7.75 GB ÷ 1.792 TB/s (GB202 GDDR7 bandwidth) ≈ 4.3ms per step

Measured step time is ~5.8ms. The GPU spends most of that waiting on GDDR7 reads. The all-reduces are brief bursts between layers; nvidia-smi at 1-second intervals averages those microsecond pulses into a much lower sustained figure.

All-reduces are overlapped with the next layer's weight fetch. NCCL launches the all-reduce asynchronously. The GPU immediately starts prefetching the next layer's weights from GDDR7 while the PCIe transfer is in flight, so SM stays at 96–100% even though communication is happening concurrently.

PCIe saturation requires ~538+ active decode tokens (~107 sequences) simultaneously. Working backwards from 24 GB/s measured P2P bandwidth, using the observed token output rate of 173 tok/s and accounting for 4.86 accepted spec tokens per step (~35.6 model passes/second):

24 GB/s ÷ (120 all-reduces/pass × 10.5 KB/token × 35.6 passes/s) ≈ 538 tokens minimum

In practice this means ~107 fully-active decode sequences with 5 spec tokens each — well beyond --max-num-seqs 32. At our concurrency level PCIe bandwidth is never the constraint; compute FLOPS and KV cache capacity would bind first.

---

Potential optimizations not yet benchmarked

These are untested candidates. The current config is already compute-saturated (SM 96–100%), so gains will be incremental:

Option	What it does	Expected impact
--attention-config '{"enable_flashinfer_autotune": true}'	Benchmarks FlashInfer kernel variants at startup and selects the fastest per batch size	Small throughput gain; adds ~1 min to startup
--num-scheduler-steps N (try N=4)	Runs N decode steps per Python scheduler iteration, reducing Python/GIL overhead	5–10% throughput gain at high concurrency
GPU0 cooling / power limit parity	GPU0 (Max-Q) hits 88°C under load while GPU1 stays at 65°C — GPU0 may thermal-throttle	Improves step-time consistency; physical change

FULL_AND_PIECEWISE CUDAGraph mode is already active for Gemma4 (visible in startup logs). This is the optimal mode; no change needed here. (Qwen3 with embedded MTP is stuck at PIECEWISE due to a FlashInfer incompatibility — not an issue for Gemma4's external drafter architecture.)

---

Running the benchmarks

Prerequisites

1. Start the server using the Optimized Run Command above.

2. Fetch the sonnet dataset — sonnet.txt is Shakespeare's sonnets, the standard vLLM benchmark corpus, hosted in the vLLM repo:

curl -L -o sonnet.txt \
  https://raw.githubusercontent.com/vllm-project/vllm/main/benchmarks/sonnet.txt

Running

# Full benchmark suite (~20 min)
./bench.sh

# Quick sanity check (~3 min)
./bench.sh quick

# Specific suites
./bench.sh coding    # throughput with long coding-shaped prompts
./bench.sh spec      # speculative decode acceptance metrics
./bench.sh latency   # single-request TTFT profile

# Prefix cache effectiveness test (~8 min)
./bench-prefix-cache.sh

Results are saved under bench-results/<timestamp>/ on the host and inside the container at /tmp/bench/.

The bench scripts use docker exec against a running container named gemma4-31b-it. If you renamed the container, update the CONTAINER variable at the top of each script.

---

Acknowledgements

Thanks to Nickcolus Martin for his article How I Got vLLM Running on Dual RTX Pro 6000 Blackwell GPUs which served as the starting point for this setup — in particular identifying the IOMMU/UVM interference as the root cause of NCCL deadlocks on AMD platforms, and the uvm_disable_iommu=1 modprobe fix.

The local-inference-lab/rtx6kpro community wiki is an excellent companion resource — a community-sourced knowledge base covering large model deployments (Qwen3.5-397B, MiniMax M2.5, Kimi-K2.5, GLM-5) on RTX PRO 6000 Blackwell in multi-GPU PCIe configurations without NVLink.