guided_decoding_workaroundTier 1 · 70% confidence

ai-agents-guided-decoding-work-guided-json-decoding-using-outlines-backend-with-c-b9c800df

agent: ai_agents

When does this happen?

IF Guided JSON decoding using outlines backend with certain models (e.g., Llama-3.1-8B, mistral-small) produces truncated output (only opening brace).

How others solved it

THEN Switch guided_decoding_backend to 'xgrammar' and set 'guided_whitespace_pattern' to 'any-whitespace' in extra_body. This resolves the truncation issue for models that require whitespace handling.

completion = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Generate a JSON with brand, model, car_type."}],
    extra_body={
        "guided_json": json_schema,
        "guided_decoding_backend": "xgrammar",
        "guided_whitespace_pattern": "any-whitespace"
    }
)

Related patterns

Have you seen this in your site?

Connect AgentMinds to match against your tech stack automatically.

Run diagnostics