Increasing the IMAGE_MAX_TOKEN_NUM does not increase the number of steps.

[LukasArana]

Checklist / 检查清单

• I have searched existing issues, and this is a new question or discussion topic. / 我已经搜索过现有的 issues，确认这是一个新的问题与讨论。

torch 2.9.1a0+gitd38164a
ms_swift 4.1.0

Question Description / 问题描述

I am doing several training runs on the multimodal finevision dataset with different values of IMAGE_MAX_TOKEN_NUM.

Since I am using packing, increasing the number of tokens per image should increase the number of total tokens per training. Therefore, I was expecting the number of training steps to increase between different training runs. However, the number of steps logged for the training are always 220918. Debugging shows that packing works well and the number of tokens for each image is increased as expected. Increasing IMAGE_MAX_TOKEN_NUM decreases the train_speed(s/it), which is expected.

Here is the training script:

# --- Environment Activation ---
source $WORK/environments/env_torch_2_9_megratron/bin/activate

export SWIFT_PATCH_CONV3D=1

export video_max_token_num=0
export video_max_token_num=0
export FPS_MIN_FRAMES=0

MASTER_PORT=9327
MAIN_PROCESS_IP=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1)
export ACCELERATE_FSDP_SHARDING_STRATEGY="1"

SWIFT_USE_MCORE_GDN=1 
export IMAGE_MAX_TOKEN_NUM=400 # This i the value that is increase between the runs
IMAGE_MAX_TOKEN_NUM=400 \ # This i the value that is increase between the runs
megatron sft \
    --model $WORK/baseModels/Qwen3.5-0.8B \
    --save_safetensors true \
    --cached_dataset $WORK/datasets/train/preprocessed/multimodal_v1_eu_512/train/ \
    --load_from_cache_file true \
    --add_non_thinking_prefix true \
    --loss_scale ignore_empty_think \
    --split_dataset_ratio 0.01 \
    --tensor_model_parallel_size 1 \
    --pipeline_model_parallel_size 1 \
    --micro_batch_size 1 \
    --global_batch_size 4 \
    --packing true \
    --padding_free \
    --recompute_granularity full \
    --recompute_method uniform \
    --recompute_num_layers 1 \
    --num_train_epochs 1 \
    --finetune true \
    --cross_entropy_loss_fusion true \
    --lr 1e-5 \
    --lr_warmup_fraction 0.05 \
    --min_lr 1e-6 \
    --adam_beta1 0.9 \
    --adam_beta2 0.95 \
    --adam_eps 1e-8 \
    --lr_decay_style cosine \
    --output_dir $WORK/multimodalModels \
    --save_steps 250 \
    --max_length 4052 \
    --dataloader_num_workers 1 \
    --dataset_num_proc 1 \
    --sequence_parallel true \
    --attention_backend flash \
    --no_load_optim false \
    --no_load_rng false \
    --save_total_limit 2 \
    --overlap_param_gather true \
    --overlap_grad_reduce true \
    --logging_steps 5 \
    --no_save_optim false \
    --freeze_llm true \
    --freeze_vit true \
    --freeze_aligner false