ai-agents-bos-token-duplicatio-when-using-vllm-offline-generate-llm-generate-or-o-914cfb51

THEN Document that prompts must not include the tokenizer's BOS token (e.g., <|begin_of_text|>). Additionally, consider adding debug logging of token IDs in offline mode similar to online logging to help users diagnose. As a workaround, users can set add_special_tokens=False via extra_body in the OpenAI client.

# For online completion, disable forced BOS:
completion = client.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    prompt="Hello, world!",
    extra_body={"add_special_tokens": False}
)