Use case

Every App, Model, and Tenant Gets an SLO

Route every request by tenant, model, and prompt. Kinference picks the vLLM pod based on KV cache state and predicts TTFT, ITL, and end-to-end latency per tenant, so you can hold inference to real SLOs.

Let the platform handle performance

Kinference sits between the apps and the models, applying the platform team’s policies to every request. It decides which model or provider serves each one, based on predicted response time and live GPU load, so apps get the performance they asked for.

  • App teams self serve their model selection, response times and data needs
  • Platform teams encode those choices as routing rules in Kubernetes
  • Each request is routed to the right model and measured against its SLO

SLOs built around how AI apps run in production

For application teams

Self-serve your application’s service levels without writing them into the code

No code changes

Point your client at the OpenAI-compatible endpoint and the platform handles model selection, fallback, and rate limits as policies.

QoS for every request

Each request can carry the QoS your app needs, and the platform decides how to serve it.

Routing and SLO tracking

Each request goes to the pod predicted to deliver the fastest first token, and every response is tracked against your app’s SLO.

Talk to us

For platform teams

Policy-driven SLOs, managed from the prompt to the pod

Inference-native policy language

Use declarative, inference-native language for writing policies by tenant, model, prompt, and GPU state.

Intelligent routing

Kinference picks the endpoint with the lowest predicted time to first token, using each replica’s live queue and cache state.

Inference SLO dashbards

Monitor every app’s inference SLOs in Grafana, in the context of queue depth and KV cache state.

Learn more

Kinference is a policy-driven inference control plane for self-hosted GPUs

The fastest endpoint for every request, and an SLO for every app

The problem.Self-hosted models in production now belong to the platform team, but setting SLOs for each app and model is complex, and vLLM’s metrics don’t tell you which app a request came from.

SLO enforcement

The platform team declares an SLO for every combination of app, team, model, and service tier. Kinference routes each request to the vLLM replica with the fastest predicted response, using live queue and KV cache state, and measures response start time, time between tokens, and total completion time on every request. Teams report SLO compliance for inference just as they do for any other production service.

Writing inference policies ↗

Policies that set how every app, model, and tenant is served

Quickstart guide

Frequently asked questions

What does Kinference measure for each request?

Time to first token (TTFT), inter-token latency (ITL), and end-to-end completion time (E2E), reported at p95 for every app, team, model, and service tier.

Can different teams have different SLOs for the same model?

Yes. SLOs are set per combination of app, team, model, and service tier, so a premium chat agent and a batch job can share a model with different targets.

How does routing help meet an SLO?

Kinference sends each request to the vLLM replica predicted to meet its target, using live queue depth and KV cache state, instead of spreading requests evenly.

Where do we see SLO compliance?

In SLO dashboards for every app, model, and tenant, alongside the dashboards your team already uses for other production services.

Do developers need to change their apps?

No. Kinference sits in front of the vLLM pods you already run, so apps keep calling the same endpoint.

Is Kinference open source?

Yes. Kinference is open source and runs in your own Kubernetes cluster, so you can read the code, see exactly how routing and policy decisions are made, and run it alongside the tools you already use. It’s built by Buoyant, the creators of Linkerd, and access to the code is available on GitHub. Kinference on GitHub.