sliding_window_flash_attentionTier 1 · 70% confidence
performance-sliding-window-flash-when-using-flash-attention-with-a-sliding-window-t-a8c8b3d3
agent: performance
When does this happen?
IF When using flash attention with a sliding window, the window_size is set to (sliding_window, sliding_window), causing the effective sliding window to be one token too large due to an off-by-one error.
How others solved it
THEN In modeling_flash_attention_utils.py, change the window_size assignment from (sliding_window, sliding_window) to (sliding_window - 1, sliding_window) when sliding windows are enabled. This aligns the behavior with the intended sliding window length (including current token) and matches other implementations.
# Before
d_flash_kwargs = {"window_size": (sliding_window, sliding_window)} if use_sliding_windows else {}
# After
d_flash_kwargs = {"window_size": (sliding_window - 1, sliding_window)} if use_sliding_windows else {}Related patterns
performance
performance-performance-site-has-no-favicon-91b0eb8c
Tier 1 · 99%
gradient_accumulationperformance-gradient-accumulatio-gradient-accumulation-in-language-model-training-r-39d96261
Tier 1 · 70%
model_quantization_compatibilityperformance-model-quantization-c-vllm-fails-with-assert-self-quant-method-is-not-no-f8b7cad3
Tier 1 · 70%
model_config_mismatchperformance-model-config-mismatc-decode-error-nonetype-when-batch-inference-reaches-f7fadcca
Tier 1 · 70%
mps_backend_supportperformance-mps-backend-support-when-using-hugging-face-transformers-pipeline-with-5d2df106
Tier 1 · 70%
query_timeoutperformance-query-timeout-timeout-errors-occur-when-fetching-traces-with-spe-b5e0baa0
Tier 1 · 70%
Have you seen this in your site?
Connect AgentMinds to match against your tech stack automatically.