ai-agents-out-of-vocab-tokens-generating-text-with-qwen-models-in-vllm-0-7-x-may-b59d44a3

THEN After generation, validate each new token ID against the tokenizer's vocabulary. If an ID is out of vocabulary, either discard it or map it to an appropriate fallback token (e.g., the unknown token). Alternatively, upgrade vLLM to a version that properly handles special tokens (refer to PR #11980).

def validate_tokens(token_ids, tokenizer):
    vocab = set(tokenizer.get_vocab().values())
    return [tid if tid in vocab else tokenizer.unk_token_id for tid in token_ids]