We don't publish
your competitive advantage.

IFContainers running LiteLLM versions 1.76.1–1.79.1 crash with Out-Of-Memory errors, accompanied by logs of 'Unclosed client session' and 'Error in is_prompt_caching_valid_prompt'.

THENDisable Slack alerting in LiteLLM configuration as a temporary workaround to reduce memory pressure and prevent OOM crashes. Monitor for versions that fix the underlying aiohttp session leaks, and consider upgrading when available.

IFCPU memory continues to increase under load even when prefix caching is disabled, though at a slower rate.

THENTo further mitigate memory growth after disabling prefix caching, also disable the multimodal preprocessor cache using `--disable-mm-preprocessor-cache`. This reduces CPU memory usage but may increase latency. Verify that the flag works in your vLLM version (changed to `--disable-mm-preprocessor-cache` in newer versions).

IFHeavy RAM usage over time, requiring periodic restarts.

THENSet environment variables MAX_IN_MEMORY_QUEUE_FLUSH_COUNT and MAX_SIZE_IN_MEMORY_QUEUE to limit the in-memory queue size and flush frequency. For example, set MAX_IN_MEMORY_QUEUE_FLUSH_COUNT to 5000 and MAX_SIZE_IN_MEMORY_QUEUE to 500. This prevents unbounded growth of the queue, reducing memory usage.

IFMemory leak in the litellm pass-through endpoint causes repeated initialization and memory growth leading to OOM.

THENAvoid using the litellm pass-through endpoint, or delete it if already active. Monitor memory usage closely. The bug is reported but not yet patched; disabling the pass-through endpoint is a temporary workaround until a fix is released.

IFUnbounded CPU memory growth when prefix caching is enabled, causing server crashes due to out-of-memory under continuous load.

THENAs a temporary workaround, disable prefix caching by using `--no-enable-prefix-caching` or `--disable-mm-preprocessor-cache`. Be aware that this may reduce latency and throughput. For a permanent solution, consider implementing memory limits or automatic eviction policies for prefix caches to prevent unbounded growth.

IFHeavy RAM usage over time in LiteLLM proxy, not releasing memory until restart, eventually causing server crashes or alerts.

THENSet environment variables MAX_IN_MEMORY_QUEUE_FLUSH_COUNT to 5000 and MAX_SIZE_IN_MEMORY_QUEUE to 500 in the proxy configuration. This limits the in-memory queue sizes and prevents memory leak buildup.

IFHeavy RAM usage over time in LiteLLM proxy, requiring container restarts to free memory, often triggered by sustained request load.

THENSet environment variables MAX_IN_MEMORY_QUEUE_FLUSH_COUNT and MAX_SIZE_IN_MEMORY_QUEUE to limit the in-memory queue size. For example, set MAX_IN_MEMORY_QUEUE_FLUSH_COUNT to 5000 and MAX_SIZE_IN_MEMORY_QUEUE to 500. This prevents unbounded queue growth and stabilizes memory usage.

IFFastAPI service using LiteLLM proxy experiences memory leaks and CPU spikes over time, consuming all available memory (e.g., 12 GB) and causing container crashes.

THENSet the MAX_REQUESTS_BEFORE_RESTART environment variable to limit the number of requests before the LiteLLM proxy automatically restarts. This provides a temporary workaround to mitigate memory leaks. Ensure you are using LiteLLM v1.77.7 or later, as the feature works from that version onward.

IFLiteLLM proxy container gradually consumes all available memory (e.g., 12 GB) and CPU spikes to 100%, causing crashes after processing the first query.

THENMitigate by setting the MAX_REQUESTS_BEFORE_RESTART environment variable (available since v1.77.7) to limit the number of requests before automatic restart. Alternatively, implement a health-check that schedules container restart when memory utilization exceeds a threshold.

IFCalling Router.completion() with a Bedrock provider repeatedly creates new SSL connections that remain open and accumulate ssl objects, eventually causing OOM crash.

THENUse litellm.completion() instead of Router.completion() for Bedrock providers as a workaround, which reuses connections and avoids the memory leak. If Router functionality is required, monitor memory usage and consider periodic restarts until the underlying fix is applied.

IFCPU memory grows unboundedly under load when prefix caching is enabled in vLLM, especially with multimodal models like Qwen3-VL and Qwen3-Reranker.

THENMonitor CPU memory usage when serving models with prefix caching. Consider disabling prefix caching via `--no-enable-prefix-caching` if memory growth is unacceptable. Alternatively, implement a limit on the prefix cache size or automatic eviction under memory pressure. This issue is observed across vLLM versions 0.11.0 to 0.14.0.

IFContainers OOM (out-of-memory) with repeated 'Unclosed client session client_session: <aiohttp.client.ClientSession object at ...>' log messages.

THENDisable Slack alerting in LiteLLM configuration as a temporary workaround to reduce memory pressure. Ensure all aiohttp client sessions are properly managed (e.g., using async context managers) in custom code paths that call acompletion or Gemini wrappers.