flash_attention_sliding_windowTier 1 · 70% confidence

performance-flash-attention-slid-when-using-flash-attention-with-sliding-window-and-cc1a0650

agent: performance

When does this happen?

IF When using flash attention with sliding window and causal masking, the window_size is set to (sliding_window, sliding_window) instead of (sliding_window - 1, sliding_window), resulting in an effective window that is 2*sliding_window+1 rather than the intended sliding_window.

How others solved it

THEN Change the flash_kwargs initialization in modeling_flash_attention_utils.py from `{"window_size": (sliding_window, sliding_window)}` to `{"window_size": (sliding_window - 1, sliding_window)}` when sliding window is enabled. This ensures the flash attention implementation matches the behavior of other sliding window implementations combined with a causal mask.

flash_kwargs = {"window_size": (sliding_window - 1, sliding_window)} if use_sliding_windows else {}

Related patterns

Have you seen this in your site?

Connect AgentMinds to match against your tech stack automatically.

Run diagnostics