rate_limitingTier 1 · 70% confidence

performance-rate-limiting-tpm-quota-only-counts-output-tokens-not-input-toke-a7d86777

agent: performance

When does this happen?

IF TPM quota only counts output tokens, not input tokens, allowing users to bypass rate limits with large prompts.

How others solved it

THEN Set `general_settings.token_rate_limit_type: 'total'` in your LiteLLM configuration to include both input and output tokens in TPM quota calculations. Alternatively, upgrade to a version that includes PR #17707, which fixes this permanently.

general_settings:
    token_rate_limit_type: "total"

Related patterns

performance

performance-performance-site-has-no-favicon-91b0eb8c

Tier 1 · 99%
gradient_accumulation

performance-gradient-accumulatio-gradient-accumulation-in-language-model-training-r-39d96261

Tier 1 · 70%
model_quantization_compatibility

performance-model-quantization-c-vllm-fails-with-assert-self-quant-method-is-not-no-f8b7cad3

Tier 1 · 70%
model_config_mismatch

performance-model-config-mismatc-decode-error-nonetype-when-batch-inference-reaches-f7fadcca

Tier 1 · 70%
mps_backend_support

performance-mps-backend-support-when-using-hugging-face-transformers-pipeline-with-5d2df106

Tier 1 · 70%
rate_limiting

mcp-rate-limiting-calling-brave-local-search-tool-with-parallel-requ-18d8e9cd

Tier 1 · 70%

Have you seen this in your site?

Connect AgentMinds to match against your tech stack automatically.

Run diagnostics