Add structured output with tools support for models with limited structured output

[devv-shayan]

Add structured output with tools support for models with limited structured output

Summary

Adds an opt-in structured output with tools feature to LitellmModel, enabling the use of tools and structured outputs simultaneously on models that don’t natively support both (specifically Google Gemini via LiteLLM).

Problem:
Models like Gemini return
BadRequestError: Function calling with a response mime type 'application/json' is unsupported
when trying to use tools together with structured outputs (via response_schema).

Solution:
A new enable_structured_output_with_tools parameter on the LitellmModel class that:

• Injects JSON formatting instructions into the system prompt
• Disables native response_format to avoid API errors
• Parses the model’s JSON response into the specified Pydantic model
• Defaults to False for backward compatibility

Test plan

Unit tests (14 tests added):

• tests/utils/test_prompts.py – Tests for prompt generation and injection logic

  • Validates JSON schema extraction from Pydantic models
  • Tests conditional injection based on tools/output_schema presence
  • Verifies opt-in behavior (only injects when explicitly enabled)

Integration test:

• tests/test_gemini_local.py – Manual test script for Gemini (requires API key)

  • Tests actual Gemini integration with tools + structured output
  • Verifies structured output with tools works end-to-end

Verification performed:

make lint     # ✅ All checks passed
make format   # ✅ 359 files formatted correctly
make mypy     # ✅ No type errors
make tests    # ✅ 995 passed, 3 skipped, 20 failed*

20 failures are pre-existing Windows-specific SQLite file locking issues (not related to this PR). All 14 new tests pass.

Test coverage:

• Updated test mock models for optional enable_structured_output_with_tools parameter
• Verified backward compatibility
• Tested both enabled and disabled modes
• Validated structured output parsing with nested Pydantic models

Issue number

#2032

Checks

• I've added new tests (if relevant)
• I've added/updated the relevant documentation
• I've run make lint and make format
• I've made sure tests pass

Documentation

Added comprehensive documentation:

• docs/models/structured_output_with_tools.md – Complete guide with examples

  • Problem explanation and sample errors
  • Step-by-step setup and code samples
  • “When to Use” guide for various model providers
  • Debugging tips and best practices

• docs/models/litellm.md – Added Gemini integration example

• mkdocs.yml – Added new doc page to navigation

Files Changed

Source code:

• src/agents/extensions/models/litellm_model.py – Added enable_structured_output_with_tools parameter and implementation
• src/agents/util/_prompts.py – Utility for JSON prompt generation
• src/agents/models/interface.py – Updated for consistency
• src/agents/models/openai_chatcompletions.py – Pass-through (ignores parameter)
• src/agents/models/openai_responses.py – Pass-through (ignores parameter)
• src/agents/run.py – No longer handles structured output logic

Tests:

• tests/utils/test_prompts.py – 14 new unit tests
• tests/test_gemini_local.py – Integration test
• Updated mock models for consistency

Documentation:

• docs/models/structured_output_with_tools.md – New comprehensive guide
• docs/models/litellm.md – Added Gemini example
• mkdocs.yml – Updated navigation

Implementation Details

Design decisions:

1. Opt-in by default – enable_structured_output_with_tools=False ensures no impact on existing integrations.
2. LiteLLM-focused – Implementation is isolated to LitellmModel; OpenAI models continue to use native support.
3. Interface consistency – All models accept the parameter for uniform signatures.
4. User control – Users enable it explicitly based on their model’s capabilities.

How it works:

1. User sets enable_structured_output_with_tools=True when initializing LitellmModel.
2. The model checks for both tools and an output schema.
3. If both are present, JSON instructions are generated from the Pydantic schema.
4. Instructions are injected into the system prompt.
5. Native response_format is disabled to avoid API errors.
6. The model returns JSON text output.
7. The SDK parses and validates it against the Pydantic model.

Example Usage

from agents.extensions.models.litellm_model import LitellmModel
from agents import function_tool
from pydantic import BaseModel, Field

class WeatherReport(BaseModel):
    city: str = Field(description="City name")
    temperature: float = Field(description="Temperature in Celsius")

@function_tool
def get_weather(city: str) -> dict:
    return {"city": city, "temperature": 22.5}

# Gemini now supports tools + structured output!
model = LitellmModel(
    "gemini/gemini-2.5-flash",
    enable_structured_output_with_tools=True,
)

agent = Agent(
    name="WeatherBot",
    model=model,
    tools=[get_weather],
    output_type=WeatherReport,
)

Backward Compatibility

✅ Fully backward compatible –

• Parameter is optional and defaults to False
• Existing integrations remain unchanged
• OpenAI models ignore it (use native structured output support)
• Only affects LitellmModel when explicitly enabled

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[seratch]

@devv-shayan Thanks for looking into this. However, as I mentioned at #2032 (comment), we prefer resolving this only by the changes on the LiteLLM or our LiteLLM adapter side. So, we don't accept this large diff for this.

[devv-shayan]

Sure I'll work on it to make it on litellm adapter side

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".