[Roadmap] Blackwell MXFP8 and NVFP4 RL training

[zianglih]

@HumansAnd

Blog

• https://humansand.ai/blog/nvfp4-rl
• https://www.lmsys.org/blog/2026-07-29-mxfp8-nvfp4-rl

Miles

• MXFP8 & NVFP4

  • [feat] Expand mxfp8 quantization to support fine-grained layer precision config #614
  • Bump Megatron version to inlcude --te-precision-config-file #567
  • [Draft] [WIP] Support --fp8-param-gather for mxfp8 #919

• MXFP8

  • Implement Blackwell MXFP8 recipe #512
  • Support DeepSeek v3.2 #963
  • [feat] Deepseek V4 Mxfp8 #1340
  • Use TransformerEngine for MXFP8 quantization #1935

• NVFP4

  • Implement nvfp4 #546 retired attempt
  • Add nvfp4 quantizer files #907
  • Use TE NVFP4 reference quantizer and add bitwise exact tests #1054
  • NVFP4 RL #1261

SGLang

• online NVFP4 serving
  • Implement online nvfp4 quantization sgl-project/sglang#26083
• MXFP8 & NVFP4
  • [FlashInfer v0.6.6][RL] Support fp8-last-n-bf16 RL for flashinfer_trtllm_routed moe backend sgl-project/sglang#20214
• MXFP8
  • Add mxfp8 support for online quantization, Triton dense linear, and CUTLASS MoE sgl-project/sglang#17449
  • [RL] Support per-layer mixed FP8/BF16 serving for FP8 checkpoints sgl-project/sglang#18742
  • Expand deep_gemm entrypoint to support more FP8 recipes. sgl-project/sglang#17294
  • Add DeepGEMM MXFP8 MoE backend sgl-project/sglang#26342
  • [FlashInfer v0.6.4] [RL] Integrate FlashInfer mxfp8 gemm, MoE, and routed MoE sgl-project/sglang#19537
  • [RL] Support mxfp8 DeepSeek V3 sgl-project/sglang#21280
  • [FlashInver v0.6.7] Integrate flashinfer_trtllm mxfp8 gemm sgl-project/sglang#21576
  • [RL] Fix weight update for mxfp8 flashinfer_cutlass gemm backend sgl-project/sglang#22484
  • [RL] Fix FP8 skip matching for trailing-dot prefixes sgl-project/sglang#26287
  • [RL] Fix FlashInfer TRTLLM MXFP8 dense weight layout sgl-project/sglang#28459
  • [RL] fix deepseek v4 MXFP8 flashinfer_trtllm_routed MoE weight update sgl-project/sglang#28676
• NVFP4
  • [RL] Add an nvfp4 online input scale mode sgl-project/sglang#18012
  • Fix nvfp4 weight update sgl-project/sglang#18085
  • [RL] Refactor NVFP4 shuffling/swizzling to in-place replacement sgl-project/sglang#22204
  • [FlashInfer v0.6.11] [RL] Support FlashInfer per-token NVFP4 MoE sgl-project/sglang#22918
  • [FlashInfer v0.6.13] Use CuTe DSL backend for FlashInfer per-token NVFP4 quantization sgl-project/sglang#28220
  • [FlashInfer v0.6.16] Support FlashInfer CuTe DSL NVFP4 MoE quantization sgl-project/sglang#28354

TransformerEngine

• MXFP8 & NVFP4
  • Add NVTE_BACKWARD_OVERRIDE=high_precision|dequantized NVIDIA/TransformerEngine#2644
  • Optimizations for MXFP8/NVFP4 dequantize kernels NVIDIA/TransformerEngine#2865
  • [PyTorch] Preserve fprop operands for dequantized backward override NVIDIA/TransformerEngine#3141
• NVFP4
  • Implement row-scaled NVFP4 fprop recipe NVIDIA/TransformerEngine#2931
  • Implement 4over6 NVFP4 recipe NVIDIA/TransformerEngine#2972
  • Add row-scale support to grouped GEMM quant NVIDIA/cudnn-frontend#251
  • Support FP32 output and dynamic M in row-scaled FP4 grouped GEMM NVIDIA/cudnn-frontend#461
  • Use cuDNN for row-scaled NVFP4 grouped GEMM NVIDIA/TransformerEngine#3042
  • Optimize NVFP4 4over6 candidate error path NVIDIA/TransformerEngine#3068

FlashInfer

• MXFP8 & NVFP4
  • Make cute dsl mxfp8/nvfp4 quantizer bitwise exact flashinfer-ai/flashinfer#3387
• MXFP8
  • Implement cutlass_fused_moe mxfp8 flashinfer-ai/flashinfer#2581
• NVFP4
  • [feat] Trtllm-gen Per-token Nvfp4 MoE flashinfer-ai/flashinfer#3027
  • Support 4over6 nvfp4 for quantizer and fused MoE flashinfer-ai/flashinfer#3264
  • Add CuTe DSL NVFP4 quantization with 4over6 FP16 scoring flashinfer-ai/flashinfer#3448
  • Add per-token NVFP4 quantization to CuTe DSL MoE flashinfer-ai/flashinfer#3645
  • feat(moe): add deterministic CuTe DSL NVFP4 finalize mode flashinfer-ai/flashinfer#3976
  • feat(moe): support BF16 activations in CuTe DSL NVFP4 MoE flashinfer-ai/flashinfer#4048

Additional stablizer for sparse attention TopK

• [SGLang v0.5.13] Add configurable Miles DSA top-k backend #1058
• [FlashInfer v0.6.10] [RL] [DSv32] [GLM-5] Add --dsa-topk-backend and integrate FlashInfer and pytorch topk sgl-project/sglang#22851
• feat: implement configurable tie_break for filtered topk flashinfer-ai/flashinfer#3095
• feat: Add row_starts and dsa_graph_safe to topk flashinfer-ai/flashinfer#3133
• perf(topk): skip output index sort for tie-break selection flashinfer-ai/flashinfer#4295
• [RL] DSV4: dispatch indexer topk_transform_512 through DSATopKBackend sgl-project/sglang#31087
• fix(dsa): correct packed FlashInfer top-k and backend selection semantics sgl-project/sglang#32490

[hanxdmech-ship-it]

Great work! We're also working on something similar and would love to collaborate and exchange ideas. Looking forward to potential synergies between our projects.

[zianglih]

For accurate reproduction of #1261, need to make sure the flashinfer is after flashinfer-ai/flashinfer#3448 to include the bit-exact 4over6 NVFP4 quantization. That is at least https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.6.13rc1 .

# Replace FlashInfer with v0.6.13rc2:

   python3 -m pip uninstall -y flashinfer flashinfer-python flashinfer-cubin flashinfer-jit-cache
   python3 -m pip install --pre 'flashinfer-python[cu13]==0.6.13rc2' 'flashinfer-cubin==0.6.13rc2'
   python3 -m pip install --pre 'flashinfer-jit-cache==0.6.13rc2' \
     --index-url https://flashinfer.ai/whl/cu130
   flashinfer show-config

# Restore cudnn version after FlashInfer reinstall:

    pip install --no-deps nvidia-cudnn-cu13==9.16.0.29