Economize slurm RPC calls, omit srun from jobstep plugin?

[ckronlage]

Software Versions

$ snakemake --version
9.22.0
$ pip list | grep snakemake
snakemake                               9.22.0
snakemake-executor-plugin-slurm         2.7.1
snakemake-executor-plugin-slurm-jobstep 0.6.1
snakemake-interface-common              1.23.0
snakemake-interface-executor-plugins    9.4.0
snakemake-interface-logger-plugins      2.1.0
snakemake-interface-report-plugins      1.3.0
snakemake-interface-scheduler-plugins   2.0.2
snakemake-interface-storage-plugins     4.4.1
$ sinfo --version
slurm 24.11.7

Describe the bug
On a cluster with heavy slurmctld RPC rate limiting (rl_bucket_size=40, rl_refill_period=60, rl_refill_rate=4), running a workflow with several hundred jobs becomes nearly impossible because sbatch calls count towards the limit.
Manually creating array jobs works, because each sbatch is only one call. However, using snakemake's array functionality doesn't. I suspect that is because the jobstep plugin internally calls srun, which also counts as an RPC call so that the limit is quickly reached.

Is there a way to make snakemake save these calls? I haven't tried group jobs but would assume they internally use srun as well? Can srun calls in the jobstep plugin be disabled?

Minimal example
Snakefile

SAMPLES = range(100)

rule all:
    input:
        expand("results/{i}.done", i=SAMPLES)

rule sleep_job:
    output:
        "results/{i}.done"
    shell:
    """
        sleep 60
        touch {output}
    """

bash

snakemake \
    --executor slurm \
    --jobs 100
squeue --me # this will freeze straight away because job submission hits the rate limit

snakemake \
    --executor slurm \
    --slurm-array-jobs: all \
    --jobs 100
squeue --me # here, only after array jobs get scheduled and run, the limit is hit and squeue hangs

Additional context
slurm configuration limits can be checked with

$ scontrol show config | grep rl_enable
SlurmctldParameters     = [...],rl_enable,rl_bucket_size=40,rl_refill_period=60,rl_refill_rate=4,[...]

[cmeesters]

The jobstep plugin serves several purposes by using srun as the jobstep trigger:

• to gather efficiency information
• to support GPU, MPI and smp (shared memory processes) jobs
• to support Snakemake wrappers (inherently)
  Eventually, the plan is to enable pooling of smp jobs in one (potentially) multi-node job. The executor plugin limits the calls to the db deamon (or slurmctld, if you switch to squeue) by default.

Right now, there are several ways to further limit the strain:

• --slurm-array-jobs - do not use all, if this is unnecessary
• --slurm-array-limit - you may limit the array size
• --slurm-status-attempts - if you set this to a higher value, more status checks will have an interval of 180 s between them.
• --slurm-init-seconds-before-status-checks - you may delay the first status check (actually useful for developing, when you want to shorten the time for the first feedback, too)

And, of course, you can limit the number of concurrent jobs with -j/--jobs too.

However, this may be beside the point. If your bucket size is above the default it might not be the sheer number of calls, but rather the refill rate: waiting a minute instead of second, will of course delay everything. I wonder what the reason might be? An ill performing network or master node?

So, by using Snakemake (in contrast to ordinary scripts with SLURM dependency handling), you actually have the same number of RPC calls to the slurmctld (assuming you would write all job scripts with srun inside). The difference is some extra monitoring with extra sacct calls (by default) which query the slurmdbd.