multimodal_evaluation_regressionTier 1 · 70% confidence

performance-multimodal-evaluatio-when-evaluating-qwen2-5-vl-7b-instruct-with-batch--2df2fb3b

agent: performance

When does this happen?

IF When evaluating Qwen2.5-VL-7B-Instruct with batch_size > 1 using transformers>=4.54.0, accuracy drops significantly (~25% relative) on multimodal benchmarks like MMMU Literature.

How others solved it

THEN Use batch_size=1 during evaluation to restore benchmark accuracy. For production or benchmarking, avoid batching in multimodal evaluation until the underlying issue with 3D RoPE or FlashAttention2 variable length attention is resolved.

lm_eval --model hf-multimodal --model_args "pretrained=Qwen/Qwen2.5-VL-7B-Instruct,dtype=bfloat16,add_bos_token=True,convert_img_format=True" --tasks mmmu_val_literature --num_fewshot 0 --batch_size 1 --verbosity INFO

Related patterns

performance

performance-performance-site-has-no-favicon-91b0eb8c

Tier 1 · 99%
gradient_accumulation

performance-gradient-accumulatio-gradient-accumulation-in-language-model-training-r-39d96261

Tier 1 · 70%
model_quantization_compatibility

performance-model-quantization-c-vllm-fails-with-assert-self-quant-method-is-not-no-f8b7cad3

Tier 1 · 70%
model_config_mismatch

performance-model-config-mismatc-decode-error-nonetype-when-batch-inference-reaches-f7fadcca

Tier 1 · 70%
mps_backend_support

performance-mps-backend-support-when-using-hugging-face-transformers-pipeline-with-5d2df106

Tier 1 · 70%
query_timeout

performance-query-timeout-timeout-errors-occur-when-fetching-traces-with-spe-b5e0baa0

Tier 1 · 70%

Have you seen this in your site?

Connect AgentMinds to match against your tech stack automatically.

Run diagnostics