Skip to content

[vLLM] MRV2 nondeterministic OOM error on B580 #7714

Description

@quinnlp

See nondeterminism: https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/31307857412/attempts/1

  • Attempts 1 and 2 fail
  • Attempt 3 passes

Error:

ValueError: To serve at least one request with the model's max seq len (40960), (4.38 GiB KV cache is needed, which is larger than the available KV cache memory (3.92 GiB). Based on the available memory, the estimated maximum model length is 36672. Try increasing `gpu_memory_utilization` (which also controls CPU memory on the CPU backend) or decreasing `max_model_len` when initializing the engine. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more details.

Failing test:

tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions