ai-agents-causal-mask-override-when-using-attn-implementation-sdpa-with-an-attent-c7a4e38d

THEN Workaround: Either avoid using SDPA with non-causal attention masks (use `attn_implementation='eager'` instead), or monkey-patch `LlamaModel._prepare_4d_causal_attention_mask_with_cache_position` to respect `is_causal`. The patch should check the `is_causal` argument and return a non-causal mask when it is False. Expect a library fix to propagate `is_causal` to the mask preparation function.

def _prepare_4d_causal_attention_mask_with_cache_position(attention_mask, sequence_length, target_length, dtype, device, cache_position, batch_size, **kwargs):
    min_dtype = torch.tensor(torch.finfo(dtype).min, dtype=dtype, device=attention_mask.device)
    return ~attention_mask.view(batch_size, 1, 1, sequence_length).expand(batch_size, 1, sequence_length, sequence_length) * min_dtype

with mock.patch.object(model.model, '_prepare_4d_causal_attention_mask_with_cache_position', _prepare_4d_causal_attention_mask_with_cache_position):
    output = model(input_ids, attention_mask=attention_mask, is_causal=False)