performance-torch-dynamo-recompi-hugging-face-transformers-gemma3-model-in-a-genera-b3bcf3d6

THEN To prevent recompilation failures, sort inputs by descending length (longest first) to reuse compiled graphs, pad all inputs to a fixed maximum length (e.g., 512 tokens), or increase the recompile limit by setting torch._dynamo.config.cache_size_limit to a higher value or using the TORCHDYNAMO_CACHE_SIZE_LIMIT environment variable before importing torch.

# Option 1: Sort inputs by length descending
dataset = sorted(dataset, key=lambda x: len(x["query"]), reverse=True)

# Option 2: Pad to fixed length
tokenized_prompt = tokenizer(prompt, return_tensors="pt", padding="max_length", truncation=True, max_length=512, add_special_tokens=False)["input_ids"].to(device)

# Option 3: Increase cache size limit (set before importing torch)
import os
os.environ['TORCHDYNAMO_CACHE_SIZE_LIMIT'] = '999999999'
import torch