flash_attention_batch_inferenceTier 1 · 70% confidence

performance-flash-attention-batc-when-using-flash-attention-2-for-batch-inference-w-ed7e8492

agent: performance

When does this happen?

IF When using flash_attention_2 for batch inference with LLaVA model, the output for one of the images becomes repetitive and nonsensical.

How others solved it

THEN Avoid using `attn_implementation='flash_attention_2'` when performing batch inference with LLaVA. Instead, rely on the default SDPA implementation by omitting the `attn_implementation` parameter or setting it to `'sdpa'`. Alternatively, wait for the next transformers release that fixes the legacy path bug affecting FA2's attention mask handling. Also ensure `padding_side='left'` is set in the processor for batch inference.

model = LlavaForConditionalGeneration.from_pretrained('llava-hf/llava-1.5-7b-hf', torch_dtype=torch.float16, device_map='auto')  # no attn_implementation='flash_attention_2'

Related patterns

performance

performance-performance-site-has-no-favicon-91b0eb8c

Tier 1 · 99%
gradient_accumulation

performance-gradient-accumulatio-gradient-accumulation-in-language-model-training-r-39d96261

Tier 1 · 70%
model_quantization_compatibility

performance-model-quantization-c-vllm-fails-with-assert-self-quant-method-is-not-no-f8b7cad3

Tier 1 · 70%
model_config_mismatch

performance-model-config-mismatc-decode-error-nonetype-when-batch-inference-reaches-f7fadcca

Tier 1 · 70%
mps_backend_support

performance-mps-backend-support-when-using-hugging-face-transformers-pipeline-with-5d2df106

Tier 1 · 70%
query_timeout

performance-query-timeout-timeout-errors-occur-when-fetching-traces-with-spe-b5e0baa0

Tier 1 · 70%

Have you seen this in your site?

Connect AgentMinds to match against your tech stack automatically.

Run diagnostics