[router] web_search_preview tool basic implementation #12290

key4ng · 2025-10-28T18:19:57Z

Motivation

Add a simplified web search interface to the OpenAI router that abstracts MCP implementation details from end users.
In order to keep the spec unchanged, only implement basic web search function. The parameter filters, search_context_size, user_location is not supported yet

Modifications

Architecture:

Introduced ToolContext enum (Regular | WebSearchPreview) to replace boolean flags, improving type safety and maintainability
Added web search constants (WEB_SEARCH_PREVIEW_SERVER_NAME, STATUS_COMPLETED, STATUS_FAILED, ACTION_TYPE_SEARCH) in utils.rs
Reserved "web_search_preview" as a protected server name for request MCP tools

Request Handling (router.rs):

Detect web_search_preview tool type in requests and validate MCP server availability
Return user-friendly error messages without exposing MCP terminology

Response Transformation (responses.rs, streaming.rs):

Transform tools array to simplified format: [{"type": "web_search_preview"}]
Convert function_call items to web_search_call (instead of mcp_call)
Use ws_* ID prefix (instead of mcp_* or fc_*)
Remove internal fields: arguments, call_id, name, server_label
Add query field extracted from tool arguments to action object

Streaming Events (streaming.rs):

Emit web_search_call.* events instead of mcp_call.* events
Skip mcp_list_tools and mcp_call_arguments events entirely
Add unique response.web_search_call.searching event for better UX
Simplified output items with only essential fields: id, type, status, action

Tool Execution (mcp.rs):

Filter MCP tools to only include those from web_search_preview server
Build specialized web_search_call items with query and status
Handle both success and failure cases with appropriate status codes

Tests

Start router using --mcp-config-path

cargo run --bin sglang-router --  --worker-urls https://api.openai.com/ --backend openai --history-backend oracle --prometheus-port 30002 --mcp-config-path /home/ubuntu/sglang/mcp.yaml

# mcp.yaml
servers:
  - name: "web_search_preview"
    protocol: sse
    url: http://localhost:8001/sse
    required: false

Test non stream output

ubuntu@moirai-a10-exp:~/sglang/sgl-router$   curl http://localhost:30000/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-5-nano",
    "tools": [{ "type": "web_search_preview" }],
    "input": "show me one new opened restaurant in seattle"
  }'
{"id":"resp_0adafba5bcc77c8c01690151bfddd481909a039ef42b234a53","object":"response","created_at":1761694143,"status":"completed","background":false,"billing":{"payer":"developer"},"error":null,"incomplete_details":null,"instructions":null,"max_output_tokens":null,"max_tool_calls":null,"model":"gpt-5-nano-2025-08-07","output":[{"id":"ws_92ac662ea100e61061bdf6615016b25af6202818e9ab8153a0","type":"web_search_call","status":"completed","action":{"type":"search","query":"newly opened restaurant Seattle"}},{"id":"ws_3ab32ace14c590b17fe28339011ddf996ac62d01f9e0ecf456","type":"web_search_call","status":"completed","action":{"type":"search","query":"newly opened restaurant Seattle 2024 2025"}},{"id":"rs_0adafba5bcc77c8c01690151c033988190a6beb8e28187337e","type":"reasoning","summary":[]},{"id":"msg_0adafba5bcc77c8c01690151c6cb288190a8176776cd2555c1","type":"message","status":"completed","content":[{"type":"output_text","annotations":[],"logprobs":[],"text":"One new opened restaurant in Seattle: Sacro Bosco — a new pizzeria listed among Seattle’s best new restaurant openings in 2025 (and noted as the new pizzeria inside Temple Pastries). If you’d like, I can pull up a different nearby option or confirm more details like exact neighborhood and hours."}],"role":"assistant"}],"parallel_tool_calls":true,"previous_response_id":null,"prompt_cache_key":null,"reasoning":{"effort":"medium","summary":null},"safety_identifier":null,"service_tier":"default","store":false,"temperature":1.0,"text":{"format":{"type":"text"},"verbosity":"medium"},"tool_choice":"auto","tools":[{"type":"web_search_preview"}],"top_logprobs":0,"top_p":1.0,"truncation":"disabled","usage":{"input_tokens":903,"input_tokens_details":{"cached_tokens":0},"output_tokens":1221,"output_tokens_details":{"reasoning_tokens":1152},"total_tokens":2124},"user":null,"metadata":{}}

Test streaming response

ubuntu@moirai-a10-exp:~/sglang/sgl-router$   curl http://localhost:30000/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-5-nano",
    "tools": [{ "type": "web_search_preview" }],
    "input": "show me one new opened restaurant in seattle",
    "stream": true
  }'
event: response.created
data: {"type":"response.created","sequence_number":0,"response":{"id":"resp_0c99f99bc2699eb201690152c2a9bc8190b7e8815ead8209d4","object":"response","created_at":1761694402,"status":"in_progress","background":false,"error":null,"incomplete_details":null,"instructions":null,"max_output_tokens":null,"max_tool_calls":null,"model":"gpt-5-nano-2025-08-07","output":[],"parallel_tool_calls":true,"previous_response_id":null,"prompt_cache_key":null,"reasoning":{"effort":"medium","summary":null},"safety_identifier":null,"service_tier":"auto","store":false,"temperature":1.0,"text":{"format":{"type":"text"},"verbosity":"medium"},"tool_choice":"auto","tools":[{"type":"web_search_preview"}],"top_logprobs":0,"top_p":1.0,"truncation":"disabled","usage":null,"user":null,"metadata":{}}}

event: response.in_progress
data: {"type":"response.in_progress","sequence_number":1,"response":{"id":"resp_0c99f99bc2699eb201690152c2a9bc8190b7e8815ead8209d4","object":"response","created_at":1761694402,"status":"in_progress","background":false,"error":null,"incomplete_details":null,"instructions":null,"max_output_tokens":null,"max_tool_calls":null,"model":"gpt-5-nano-2025-08-07","output":[],"parallel_tool_calls":true,"previous_response_id":null,"prompt_cache_key":null,"reasoning":{"effort":"medium","summary":null},"safety_identifier":null,"service_tier":"auto","store":false,"temperature":1.0,"text":{"format":{"type":"text"},"verbosity":"medium"},"tool_choice":"auto","tools":[{"type":"web_search_preview"}],"top_logprobs":0,"top_p":1.0,"truncation":"disabled","usage":null,"user":null,"metadata":{}}}

event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":2,"output_index":0,"item":{"id":"rs_0c99f99bc2699eb201690152c320708190bd0542da6ccfaf30","type":"reasoning","summary":[]}}

event: response.output_item.done
data: {"type":"response.output_item.done","sequence_number":3,"output_index":0,"item":{"id":"rs_0c99f99bc2699eb201690152c320708190bd0542da6ccfaf30","type":"reasoning","summary":[]}}

event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":4,"output_index":1,"item":{"id":"ws_0c99f99bc2699eb201690152c42ac4819096132f8286009791","type":"web_search_call","status":"in_progress"}}

event: response.web_search_call.in_progress
data: {"type":"response.web_search_call.in_progress","sequence_number":5,"output_index":1,"item_id":"ws_0c99f99bc2699eb201690152c42ac4819096132f8286009791"}

event: response.web_search_call.searching
data: {"type":"response.web_search_call.searching","sequence_number":6,"output_index":1,"item_id":"ws_0c99f99bc2699eb201690152c42ac4819096132f8286009791"}

event: response.web_search_call.completed
data: {"type":"response.web_search_call.completed","sequence_number":8,"output_index":1,"item_id":"ws_36a99cd20517d24931a74f48c2ff38990a68bf5ed05e90182c"}

event: response.output_item.done
data: {"type":"response.output_item.done","sequence_number":9,"output_index":1,"item":{"id":"ws_36a99cd20517d24931a74f48c2ff38990a68bf5ed05e90182c","type":"web_search_call","status":"completed","action":{"type":"search","query":"newly opened restaurant Seattle"}}}

event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":10,"output_index":2,"item":{"id":"rs_0c8941745996768a01690152c5e9d481959405156db9530496","type":"reasoning","summary":[]}}

event: response.output_item.done
data: {"type":"response.output_item.done","sequence_number":11,"output_index":2,"item":{"id":"rs_0c8941745996768a01690152c5e9d481959405156db9530496","type":"reasoning","summary":[]}}

event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":12,"output_index":3,"item":{"id":"ws_0c8941745996768a01690152c848e8819584badf44d6503dd5","type":"web_search_call","status":"in_progress"}}

event: response.web_search_call.in_progress
data: {"type":"response.web_search_call.in_progress","sequence_number":13,"output_index":3,"item_id":"ws_0c8941745996768a01690152c848e8819584badf44d6503dd5"}

event: response.web_search_call.searching
data: {"type":"response.web_search_call.searching","sequence_number":14,"output_index":3,"item_id":"ws_0c8941745996768a01690152c848e8819584badf44d6503dd5"}

event: response.web_search_call.completed
data: {"type":"response.web_search_call.completed","sequence_number":16,"output_index":3,"item_id":"ws_af4ea2e422fd23c0d6450454ba5c5f426ea6fa36e99f0f6530"}

event: response.output_item.done
data: {"type":"response.output_item.done","sequence_number":17,"output_index":3,"item":{"id":"ws_af4ea2e422fd23c0d6450454ba5c5f426ea6fa36e99f0f6530","type":"web_search_call","status":"completed","action":{"type":"search","query":"newly opened restaurant Seattle 2025"}}}

event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":18,"output_index":4,"item":{"id":"rs_014492a60b19208401690152c9e5108193b8d601c4206257ab","type":"reasoning","summary":[]}}

event: response.output_item.done
data: {"type":"response.output_item.done","sequence_number":19,"output_index":4,"item":{"id":"rs_014492a60b19208401690152c9e5108193b8d601c4206257ab","type":"reasoning","summary":[]}}

event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":20,"output_index":5,"item":{"id":"msg_014492a60b19208401690152d1a26c8193bb680e1eee1aa7cd","type":"message","status":"in_progress","content":[],"role":"assistant"}}

event: response.content_part.added
data: {"type":"response.content_part.added","sequence_number":21,"item_id":"msg_014492a60b19208401690152d1a26c8193bb680e1eee1aa7cd","output_index":5,"content_index":0,"part":{"type":"output_text","annotations":[],"logprobs":[],"text":""}}

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":22,"item_id":"msg_014492a60b19208401690152d1a26c8193bb680e1eee1aa7cd","output_index":5,"content_index":0,"delta":"One","logprobs":[],"obfuscation":"pGrOEGpQshwHd"}

...

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":137,"item_id":"msg_014492a60b19208401690152d1a26c8193bb680e1eee1aa7cd","output_index":5,"content_index":0,"delta":" find","logprobs":[],"obfuscation":"RHrkFGACBVu"}

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":138,"item_id":"msg_014492a60b19208401690152d1a26c8193bb680e1eee1aa7cd","output_index":5,"content_index":0,"delta":" more","logprobs":[],"obfuscation":"OyPzk0zoG2u"}

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":139,"item_id":"msg_014492a60b19208401690152d1a26c8193bb680e1eee1aa7cd","output_index":5,"content_index":0,"delta":" options","logprobs":[],"obfuscation":"zVVGc2aT"}

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":140,"item_id":"msg_014492a60b19208401690152d1a26c8193bb680e1eee1aa7cd","output_index":5,"content_index":0,"delta":"?","logprobs":[],"obfuscation":"uk3Fj6Xtx144s9y"}

event: response.output_text.done
data: {"type":"response.output_text.done","sequence_number":141,"item_id":"msg_014492a60b19208401690152d1a26c8193bb680e1eee1aa7cd","output_index":5,"content_index":0,"text":"One newly opened Seattle spot to check out:\n- Salad Party — a new salad-focused restaurant from the Grillbird team, opened as an adjoining spot after Grillbird’s West Seattle expansion, in the Phinney Ridge area. It’s highlighted as part of Seattle’s latest new openings.\n\nSource: The Infatuation’s guide to Seattle’s new restaurant openings (link below)\nhttps://www.theinfatuation.com/seattle/guides/seattles-new-restaurant-openings\n\nWould you like me to pull the address, hours, and a quick review score, or find more options?","logprobs":[]}

event: response.content_part.done
data: {"type":"response.content_part.done","sequence_number":142,"item_id":"msg_014492a60b19208401690152d1a26c8193bb680e1eee1aa7cd","output_index":5,"content_index":0,"part":{"type":"output_text","annotations":[],"logprobs":[],"text":"One newly opened Seattle spot to check out:\n- Salad Party — a new salad-focused restaurant from the Grillbird team, opened as an adjoining spot after Grillbird’s West Seattle expansion, in the Phinney Ridge area. It’s highlighted as part of Seattle’s latest new openings.\n\nSource: The Infatuation’s guide to Seattle’s new restaurant openings (link below)\nhttps://www.theinfatuation.com/seattle/guides/seattles-new-restaurant-openings\n\nWould you like me to pull the address, hours, and a quick review score, or find more options?"}}

event: response.output_item.done
data: {"type":"response.output_item.done","sequence_number":143,"output_index":5,"item":{"id":"msg_014492a60b19208401690152d1a26c8193bb680e1eee1aa7cd","type":"message","status":"completed","content":[{"type":"output_text","annotations":[],"logprobs":[],"text":"One newly opened Seattle spot to check out:\n- Salad Party — a new salad-focused restaurant from the Grillbird team, opened as an adjoining spot after Grillbird’s West Seattle expansion, in the Phinney Ridge area. It’s highlighted as part of Seattle’s latest new openings.\n\nSource: The Infatuation’s guide to Seattle’s new restaurant openings (link below)\nhttps://www.theinfatuation.com/seattle/guides/seattles-new-restaurant-openings\n\nWould you like me to pull the address, hours, and a quick review score, or find more options?"}],"role":"assistant"}}

event: response.completed
data: {"type":"response.completed","sequence_number":144,"response":{"id":"resp_0c99f99bc2699eb201690152c2a9bc8190b7e8815ead8209d4","object":"response","created_at":1761694409,"status":"completed","background":false,"error":null,"incomplete_details":null,"instructions":null,"max_output_tokens":null,"max_tool_calls":null,"model":"gpt-5-nano-2025-08-07","output":[{"id":"ws_5b502e6a80b72fc641474da666f16c18a2418b099fdb5134f2","type":"web_search_call","status":"completed","action":{"type":"search","query":"newly opened restaurant Seattle"}},{"id":"ws_b75a218db1fb539129047414a54523602d359b931aa65d4026","type":"web_search_call","status":"completed","action":{"type":"search","query":"newly opened restaurant Seattle 2025"}},{"id":"rs_014492a60b19208401690152c9e5108193b8d601c4206257ab","type":"reasoning","summary":[]},{"id":"msg_014492a60b19208401690152d1a26c8193bb680e1eee1aa7cd","type":"message","status":"completed","content":[{"type":"output_text","annotations":[],"logprobs":[],"text":"One newly opened Seattle spot to check out:\n- Salad Party — a new salad-focused restaurant from the Grillbird team, opened as an adjoining spot after Grillbird’s West Seattle expansion, in the Phinney Ridge area. It’s highlighted as part of Seattle’s latest new openings.\n\nSource: The Infatuation’s guide to Seattle’s new restaurant openings (link below)\nhttps://www.theinfatuation.com/seattle/guides/seattles-new-restaurant-openings\n\nWould you like me to pull the address, hours, and a quick review score, or find more options?"}],"role":"assistant"}],"parallel_tool_calls":true,"previous_response_id":null,"prompt_cache_key":null,"reasoning":{"effort":"medium","summary":null},"safety_identifier":null,"service_tier":"default","store":false,"temperature":1.0,"text":{"format":{"type":"text"},"verbosity":"medium"},"tool_choice":"auto","tools":[{"type":"web_search_preview"}],"top_logprobs":0,"top_p":1.0,"truncation":"disabled","usage":{"input_tokens":551,"input_tokens_details":{"cached_tokens":0},"output_tokens":1661,"output_tokens_details":{"reasoning_tokens":1536},"total_tokens":2212},"user":null,"metadata":{}}}

data: [DONE]

Benchmarking and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

key4ng · 2025-10-28T18:20:08Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces web search capabilities as a preview feature. It accomplishes this by adding a special web_search_preview tool type that is handled differently from other MCP tools. The implementation adds a new is_web_search boolean flag that is passed through many layers of the application to control logic flow. While this works, it introduces a code smell that could impact maintainability. My review includes a suggestion to refactor this using a strategy pattern to make the design more robust and extensible. I've also pointed out a few places where magic strings should be replaced with constants to improve code quality, and a small bug in an example configuration file. Overall, this is a good start for the feature, and addressing these points will make it more solid.

sgl-router/src/routers/openai/mcp.rs

sgl-router/examples/mcp_config_web_search.yaml

sgl-router/src/routers/openai/web_search.rs

key4ng · 2025-10-28T22:59:14Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for a web_search_preview tool, which is handled as a simplified version of an MCP tool. The changes are extensive, touching MCP handling, response generation, and streaming logic. The introduction of the ToolContext enum is a good design choice to differentiate between regular MCP tools and the new web search tool.

My review focuses on improving maintainability by addressing magic strings and code duplication. I've pointed out several places where string literals should be replaced with constants and where repeated blocks of code could be refactored into helper functions. These changes will make the code easier to read, less prone to errors, and simpler to modify in the future.

sgl-router/src/routers/openai/mcp.rs

sgl-router/src/routers/openai/responses.rs

sgl-router/src/routers/openai/streaming.rs

key4ng · 2025-10-29T16:34:37Z

Hi @slin1237 this pr is ready for review

…-search

…b-search

key4ng added 6 commits October 28, 2025 06:08

initial implementation

905bd16

add utils change

594ce0d

resolve merge conflict

7a1f1b6

fix compile error

9aa372a

refactoring

3158a6e

clean code

757582a

sglang-bot added the run-ci label Oct 28, 2025

gemini-code-assist bot reviewed Oct 28, 2025

View reviewed changes

key4ng added 8 commits October 28, 2025 20:49

refactoring the code for better readability

09168c8

clean code

3ddb5bb

fix clippy

b36c82c

remove unrelated changes

54da291

update error message

8286c9a

add query field for web search

a16911a

fix streaming format

f9e8a03

fix fmt

33819ae

gemini-code-assist bot reviewed Oct 28, 2025

View reviewed changes

key4ng added 2 commits October 28, 2025 23:26

resolve comments

2b089e1

fix format

951853f

key4ng changed the title ~~Web search[wip]~~ [router] web_search_preview tool basic implementation Oct 28, 2025

key4ng marked this pull request as ready for review October 28, 2025 23:43

key4ng requested review from CatherineSue and slin1237 as code owners October 28, 2025 23:43

Merge branch 'main' into web-search

faa2ce7

key4ng added 3 commits October 29, 2025 20:30

Merge branch 'main' into web-search

6acfd88

Merge branch 'main' of https://github.com/sgl-project/sglang into web…

ac06018

…-search

Merge branch 'main' of https://github.com/sgl-project/sglang into web…

ddc709e

…-search

Merge branch 'web-search' of https://github.com/key4ng/sglang into we…

a7e4e65

…b-search

slin1237 approved these changes Oct 30, 2025

View reviewed changes

slin1237 merged commit 96ac24c into sgl-project:main Oct 30, 2025
46 of 48 checks passed

slin1237 mentioned this pull request Oct 30, 2025

[router] 0.2.2 release #11986

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[router] web_search_preview tool basic implementation #12290

[router] web_search_preview tool basic implementation #12290

key4ng commented Oct 28, 2025 •

edited

Loading

Uh oh!

key4ng commented Oct 28, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

key4ng commented Oct 28, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

key4ng commented Oct 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[router] web_search_preview tool basic implementation #12290

[router] web_search_preview tool basic implementation #12290

Conversation

key4ng commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Tests

Benchmarking and Profiling

Checklist

Uh oh!

key4ng commented Oct 28, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

key4ng commented Oct 28, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

key4ng commented Oct 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

key4ng commented Oct 28, 2025 •

edited

Loading