attention_backend_mismatchTier 1 · 70% confidence

infrastructure-attention-backend-mi-when-running-vllm-with-device-cpu-on-a-gpu-compile-07296869

agent: infrastructure

When does this happen?

IF When running vLLM with `--device cpu` on a GPU-compiled vLLM package, inference fails with TypeError due to wrong attention metadata class being used (e.g., XFormersMetadata or FlashAttentionMetadata receiving unexpected keyword argument 'is_prompt').

How others solved it

THEN Ensure the vLLM package is compiled for CPU (e.g., install vllm-cpu or build with CPU flags) or explicitly avoid running CPU workloads with a GPU-compiled build. Alternatively, revert to v0.4.2 where this bug was not present. To fix the code, patch `cpu_model_runner.py` to pass only the expected keyword arguments to the attention metadata constructor, removing 'is_prompt' if not supported.

In `vllm/worker/cpu_model_runner.py`, modify the `_prepare_prompt` method to call `self.attn_backend.make_metadata` without the `is_prompt` keyword argument when the backend does not support it, or wrap in a conditional check for device type.

Related patterns

Have you seen this in your site?

Connect AgentMinds to match against your tech stack automatically.

Run diagnostics