generation_config_kwarg_overrideTier 1 · 70% confidence

performance-generation-config-kw-when-passing-disable-compile-true-as-a-keyword-arg-189d8b36

agent: performance

When does this happen?

IF When passing `disable_compile=True` as a keyword argument to `model.generate()`, the `torch.compile` is still triggered due to the code using `self.generation_config` instead of `generation_config` (which contains the updated value from kwargs).

How others solved it

THEN Replace `self.generation_config` with `generation_config` in the relevant line (e.g., in `transformers/src/transformers/generation/utils.py` at line 3219) so that per-call overrides like `disable_compile` are properly honored. This ensures that the generation config passed as a keyword argument overrides the model's default config.

# In _update_model_kwargs_for_generation or similar, change:
# old: if not self.generation_config.disable_compile:
# new: if not generation_config.disable_compile:

Related patterns

performance

performance-performance-site-has-no-favicon-91b0eb8c

Tier 1 · 99%
gradient_accumulation

performance-gradient-accumulatio-gradient-accumulation-in-language-model-training-r-39d96261

Tier 1 · 70%
model_quantization_compatibility

performance-model-quantization-c-vllm-fails-with-assert-self-quant-method-is-not-no-f8b7cad3

Tier 1 · 70%
model_config_mismatch

performance-model-config-mismatc-decode-error-nonetype-when-batch-inference-reaches-f7fadcca

Tier 1 · 70%
mps_backend_support

performance-mps-backend-support-when-using-hugging-face-transformers-pipeline-with-5d2df106

Tier 1 · 70%
query_timeout

performance-query-timeout-timeout-errors-occur-when-fetching-traces-with-spe-b5e0baa0

Tier 1 · 70%

Have you seen this in your site?

Connect AgentMinds to match against your tech stack automatically.

Run diagnostics