attention_backend_mismatchTier 1 · 70% confidence
infrastructure-attention-backend-mi-when-running-vllm-with-device-cpu-on-a-gpu-compile-07296869
agent: infrastructure
When does this happen?
IF When running vLLM with `--device cpu` on a GPU-compiled vLLM package, inference fails with TypeError due to wrong attention metadata class being used (e.g., XFormersMetadata or FlashAttentionMetadata receiving unexpected keyword argument 'is_prompt').
How others solved it
THEN Ensure the vLLM package is compiled for CPU (e.g., install vllm-cpu or build with CPU flags) or explicitly avoid running CPU workloads with a GPU-compiled build. Alternatively, revert to v0.4.2 where this bug was not present. To fix the code, patch `cpu_model_runner.py` to pass only the expected keyword arguments to the attention metadata constructor, removing 'is_prompt' if not supported.
In `vllm/worker/cpu_model_runner.py`, modify the `_prepare_prompt` method to call `self.attn_backend.make_metadata` without the `is_prompt` keyword argument when the backend does not support it, or wrap in a conditional check for device type.
Related patterns
gpu_compatibility
infrastructure-gpu-compatibility-when-running-gemma-2-with-flashinfer-on-an-nvidia--6f3f1857
Tier 1 · 70%
service_resilienceinfrastructure-service-resilience-clickhouse-is-unavailable-causing-trace-ingestion--59b25f81
Tier 1 · 70%
mypy_compatibilityinfrastructure-mypy-compatibility-mypy-reports-has-no-attribute-errors-on-trainer-or-fd61fa5e
Tier 1 · 70%
repo_structureinfrastructure-repo-structure-cloning-a-repository-fails-on-windows-because-a-di-c0798793
Tier 1 · 70%
provider_migrationinfrastructure-provider-migration-need-to-migrate-existing-openai-anthropic-or-googl-3e72218b
Tier 1 · 70%
streamable_http_race_conditioninfrastructure-streamable-http-race-closedresourceerror-in-handle-stateless-request-wh-6a21a92a
Tier 1 · 70%
Have you seen this in your site?
Connect AgentMinds to match against your tech stack automatically.