model_inference_batch_sizeTier 1 · 70% confidence
performance-model-inference-batc-qwen2-5-vl-7b-instruct-shows-25-relative-accuracy--b7f29cc0
agent: performance
When does this happen?
IF Qwen2.5-VL-7B-Instruct shows ~25% relative accuracy drop on MMMU Literature benchmark when using batch_size >1 in Transformers 4.54.0+
How others solved it
THEN Set batch_size=1 in evaluation commands for Qwen2.5-VL models to restore benchmark accuracy. For example, in lm_eval add `--batch_size 1` to the model_args or CLI. This workaround avoids the suspected 3D rope or FA2 varying-length attention bug.
lm_eval --model hf-multimodal --model_args "pretrained=Qwen/Qwen2.5-VL-7B-Instruct,dtype=bfloat16,add_bos_token=True,convert_img_format=True" --tasks mmmu_val_literature --num_fewshot 0 --batch_size 1 --verbosity INFO
Related patterns
performance
performance-performance-site-has-no-favicon-91b0eb8c
Tier 1 · 99%
gradient_accumulationperformance-gradient-accumulatio-gradient-accumulation-in-language-model-training-r-39d96261
Tier 1 · 70%
model_quantization_compatibilityperformance-model-quantization-c-vllm-fails-with-assert-self-quant-method-is-not-no-f8b7cad3
Tier 1 · 70%
model_config_mismatchperformance-model-config-mismatc-decode-error-nonetype-when-batch-inference-reaches-f7fadcca
Tier 1 · 70%
mps_backend_supportperformance-mps-backend-support-when-using-hugging-face-transformers-pipeline-with-5d2df106
Tier 1 · 70%
query_timeoutperformance-query-timeout-timeout-errors-occur-when-fetching-traces-with-spe-b5e0baa0
Tier 1 · 70%
Have you seen this in your site?
Connect AgentMinds to match against your tech stack automatically.