We don't publish
your competitive advantage.

IFDecode error during batch inference when model's config.vocab_size exceeds the actual tokenizer vocabulary length, leading to sampling of padding tokens that cannot be decoded.

THENEnsure that the model's vocab_size matches the length of the tokenizer. Modify config.vocab_size in the model's config.json to len(tokenizer) or override the sampler's vocab_size in the model implementation. For detailed steps, refer to the vLLM model code, e.g., in OPTForCausalLM.__init__ set self.sampler = Sampler(len(tokenizer)).

IFDocumentation says bottom layers use SWA and top use full attention but code does opposite in Qwen3.

THENExplicitly set the `layer_types` list in `Qwen3Config` to match the code logic: layers with index >= `max_window_layers` use `"sliding_attention"`, lower layers use `"full_attention"`. Do not rely on the auto-generated list from the documentation comment, as it is reversed.

IFValueError: Unrecognized model when loading a model that lacks a `model_type` key in its config.json.

THENEdit the model's config.json to include a valid `model_type` key corresponding to the model architecture (e.g., `"model_type": "bert"` for BERT-like models). If the model is on the Hugging Face Hub and you cannot modify it, either fork the repository to add the key, or temporarily downgrade transformers to version 4.57.1 which does not enforce this requirement. Always ensure `trust_remote_code=True` if the model uses custom code.

IFLiteLLM proxy automatically sets max_tokens to 4096 for Claude 3.5 and 3.7 models when the caller does not specify max_tokens, leading to artificially truncated outputs.

THENExplicitly set max_tokens to 8192 in your request to the LiteLLM proxy, or configure the default max_tokens in your LiteLLM model config for Anthropic models (e.g., in the config.yaml under 'model_list' add 'max_tokens: 8192'). This ensures the full 8k output capacity of Claude 3.5/3.7 is used.