ai-agents-causal-lm-caching-when-using-past-key-values-with-a-padded-batch-in--fcd4de2b

THEN To fix, either manually pass correct position_ids (e.g., position_ids = torch.arange(past_length, past_length + new_tokens).unsqueeze(0)) or set tokenizer.padding_side='left' so that padding appears at the beginning and past_length excludes padding correctly.

tokenizer = AutoTokenizer.from_pretrained('gpt2', padding_side='left')
tokenizer.pad_token = tokenizer.eos_token
# Then use past_key_values as usual