performance-attention-implementa-mixing-flash-attention-for-the-vision-encoder-with-19776f89

THEN Ensure both the vision and text (LLM) parts of Qwen2VL use the same attention implementation. For consistent behavior and best accuracy, set both to 'flash_attention_2' via the `attn_implementation` dict parameter (e.g., `attn_implementation={'vision_config': 'flash_attention_2', '': 'flash_attention_2'}`) or both to 'eager'. Avoid the combination where vision uses flash and text uses eager, as it breaks generation.

from transformers import Qwen2VLForConditionalGeneration

# Correct: both vision and text use flash attention
model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct",
    torch_dtype="bfloat16",
    attn_implementation={'vision_config': 'flash_attention_2', '': 'flash_attention_2'},
    device_map="auto"
)