Collaborate on Reusable Kubernetes Inference Orchestration

[rootfs]

Hey Community!

I am exploring collaboration with different inference engines to create a reusable orchestration flows on Kubernetes. I have started a proposal at vLLM Production Stack.

The goal is to create a generic API so that different inference engines (vLLM or SGLang) can be deployed on Kubernetes for different performance, SLA, and resource usage goals.

Such API should support the following use cases:

• Creating a basic inference engine on Kubernetes, e.g. through deploying a Kubernetes deployment.
• Exposing the KV cache configs (e.g. speculative decoding, P/D disagg) on the API to support SLA and performance goals.
• Supporting routing algorithms such as GPU metrics aware, weighted round robin, prefix hash aware, session aware to multiple inference engines
• Other LLM gateway features such as semantic cache, prompt guard, etc.

Currently, there are quite a few efforts towards to this goal. However, they lacks the reusable support for the above use cases.

I am looking forward to hearing from the community and starting a productive collaboration to explore this direction.

Cheers!

Huamin Chen, Red Hat

[rootfs]

@zhaochenyang20 WDYT?

[zhaochenyang20]

I've asked our team. thanks!