Miles CI gap Between ROCm & CUDA

[indianspeedster]

Tracking ROCm CI parity with the NVIDIA PR Test workflow (.github/workflows/pr-test.yml): the same per-commit tests green on AMD Instinct MI300/MI355X (ROCm) as on NVIDIA.

Snapshot of the currently active (enabled) per-commit tests against origin/main @ c8e85df24. A ticked box = confirmed passing on MI355X (trailing #NNNN is the PR carrying the ROCm fix, where one was needed); a blank box = still to run (newly added/split files, or not yet green). Suites are bucketed by GPU type/count and selected at runtime by domain labels. List a suite's contents with:

python3 -m tests.ci.run_suite --hw cpu  --suite stage-a-cpu --list-only
python3 -m tests.ci.run_suite --hw cuda --suite <suite-name> --list-only

77 / 83 passing.

stage-a-cpu (CPU) — 44 active

• tests/fast/backends/megatron_utils/test_fp32_param_utils.py
• tests/fast/backends/megatron_utils/test_lora_checkpoint_helpers.py
• tests/fast/backends/megatron_utils/test_lora_hf_weight_iterator.py
• tests/fast/backends/megatron_utils/test_lora_model_branches.py
• tests/fast/backends/megatron_utils/test_lora_update_weight.py
• tests/fast/backends/megatron_utils/test_lora_utils.py
• tests/fast/backends/megatron_utils/test_lora_weight_sync_validation.py
• tests/fast/backends/megatron_utils/test_model_provider_true_on_policy.py
• tests/fast/backends/megatron_utils/test_qwen2_true_on_policy_conversion.py
• tests/fast/rollout/generate_hub/test_tool_call_utils.py
• tests/fast/rollout/generate_utils/test_openai_endpoint_utils.py
• tests/fast/rollout/generate_utils/test_sample_utils.py
• tests/fast/rollout/inference_rollout/test_compatibility.py
• tests/fast/rollout/rm_hub/test_deepscaler.py
• tests/fast/rollout/rm_hub/test_f1.py
• tests/fast/rollout/rm_hub/test_gpqa.py
• tests/fast/rollout/rm_hub/test_math_dapo_utils.py
• tests/fast/rollout/rm_hub/test_math_utils.py
• tests/fast/rollout/rm_hub/test_rm_hub.py
• tests/fast/router/test_linear_trajectory.py
• tests/fast/router/test_router.py
• tests/fast/router/test_session_pretokenized_e2e.py
• tests/fast/router/test_session_race_conditions.py
• tests/fast/router/test_sessions.py
• tests/fast/test_megatron_cli_flags.py
• tests/fast/utils/chat_template_utils/test_pretokenized_chat.py
• tests/fast/utils/chat_template_utils/test_pretokenized_via_tito.py
• tests/fast/utils/chat_template_utils/test_template.py
• tests/fast/utils/chat_template_utils/test_tito_tokenizer.py
• tests/fast/utils/chat_template_utils/test_token_seq_comparator.py
• tests/fast/utils/test_arguments.py
• tests/fast/utils/test_async_utils.py
• tests/fast/utils/test_dumper_utils.py
• tests/fast/utils/test_env_report.py
• tests/fast/utils/test_http_utils.py
• tests/fast/utils/test_logging_utils.py
• tests/fast/utils/test_lora_arguments.py
• tests/fast/utils/test_mask_utils.py
• tests/fast/utils/test_misc.py
• tests/fast/utils/test_types.py
• tests/fast/utils/test_utils/test_mock_sglang_server.py
• tests/fast/utils/test_utils/test_mock_tools.py
• tests/fast/utils/test_utils/test_session_verify_runner.py
• tests/utils/test_sglang_config.py

stage-b-cpu (CPU)

No tests registered yet.

stage-b-2-gpu-h200 (CUDA, 2-GPU) — 15 active

• tests/fast/rollout/generate_hub/test_multi_turn.py
• tests/fast/rollout/generate_hub/test_single_turn.py
• tests/fast/rollout/inference_rollout/integration/test_agent_metadata.py
• tests/fast/rollout/inference_rollout/integration/test_basic.py
• tests/fast/rollout/inference_rollout/integration/test_deterministic.py
• tests/fast/rollout/inference_rollout/integration/test_dynamic_filter.py
• tests/fast/rollout/inference_rollout/integration/test_group_rm.py
• tests/fast/rollout/inference_rollout/integration/test_multi_sample.py
• tests/fast/rollout/inference_rollout/integration/test_multi_turn.py
• tests/fast/rollout/inference_rollout/integration/test_over_sampling.py
• tests/fast/rollout/inference_rollout/integration/test_sample_filter.py
• tests/fast/rollout/inference_rollout/integration/test_semaphore.py
• tests/fast/utils/test_quantizer_ci.py
• tests/fast/utils/test_mxfp8_quantizer.py
• tests/fast/utils/test_nvfp4_quantizer.py (ROCm TE wheel has no NVFP4QuantizerRef)

stage-c-2-gpu-h200 (CUDA, 2-GPU) — 2 active

• tests/e2e/long/test_qwen2.5_0.5B_gsm8k.py (rocm: disable gradient_accumulation_fusion on gfx950 in test_qwen2.5_0.5B_gsm8k #1159)
• tests/e2e/long/test_qwen2.5_0.5B_gsm8k_async.py (rocm: disable gradient_accumulation_fusion on gfx950 in test_qwen2.5_0.5B_gsm8k_async #1160)

stage-c-4-gpu-h200 (CUDA, 4-GPU) — 12 active

• tests/e2e/lora/test_lora_qwen2.5_0.5B.py
• tests/e2e/megatron/test_mimo_7B_mtp_only_grad.py (tests/mimo-7B: switch ROCm path to async 4+4 to avoid 8-rank RCCL hang and no gradient accumulation fusion #1163)
• tests/e2e/sglang/test_chat_input_ids_equivalence.py
• tests/e2e/megatron/test_qwen3_30B_A3B/test_baseline.py ci: handle ROCm qwen3 30B A3B configs #1162
• tests/e2e/megatron/test_qwen3_30B_A3B/test_int4_rollout.py INT4 quant kernel is CUDA-only (no ROCm/HIP build of fake_int_quant_cuda)
• tests/e2e/megatron/test_qwen3_30B_A3B/test_r3_baseline.py rocm/ ci: support Qwen3-30B-A3B R3 test on ROCm #1165
• tests/e2e/sglang/test_r3_router_equivalence.py
• tests/e2e/sglang/test_session_server_multi_role/test_glm47.py
• tests/e2e/sglang/test_session_server_multi_role/test_minimax_m27.py
• tests/e2e/sglang/test_session_server_multi_role/test_nemotron3.py (modelopt_fp8 quantization is currently rejected by the SGLang ROCm guard)
• tests/e2e/sglang/test_session_server_multi_role/test_qwen3.py
• tests/e2e/sglang/test_session_server_multi_role/test_qwen35.py

stage-c-8-gpu-h100 (CUDA, 8-GPU) — 10 active

• tests/e2e/sglang_config/test_sglang_config.py (rocm: disable gradient_accumulation_fusion on gfx950 in test_sglang_config #1156)
• tests/e2e/sglang_config/test_sglang_config_mixed_offload.py (rocm: disable gradient_accumulation_fusion on gfx950 in test_sglang_config_mixed_offload #1157)
• tests/e2e/sglang_config/test_sglang_config_mixed_offload_ft.py (rocm: disable gradient_accumulation_fusion on gfx950 in test_sglang_config_mixed_offload_ft #1158)
• tests/e2e/short/test_qwen2.5_0.5B_gsm8k_async_short.py (rocm: disable gradient_accumulation_fusion on gfx950 in test_qwen2.5_0.5B_gsm8k_async_short #1155)
• tests/e2e/short/test_qwen2.5_0.5B_gsm8k_short.py (rocm: disable gradient_accumulation_fusion on gfx950 in test_qwen2.5_0.5B_gsm8k_short #1154)
• tests/e2e/short/test_run_megatron.py
• tests/e2e/megatron/test_glm47_flash/test_r3_mtp.py (ROCm/support GLM-4.7-Flash R3+MTP test  #1242 )
• tests/e2e/megatron/test_qwen3_30B_A3B/test_deepep_fp8.py (Deep-EP WIP on Rocm)
• tests/e2e/megatron/test_qwen3_30B_A3B/test_deepep_fp8_bridge.py (Deep-EP WIP on Rocm)
• tests/e2e/megatron/test_glm5_744b_a40b_4layer_ci.py Rocm changes to tests/e2e/megatron/test_glm5_744b_a40b_4layer.py #1172

[sreerohi]

tests/e2e/megatron/test_qwen3_4B_ppo.py passes with docker container from #1119