Prefix caching
Shared system prompts are hashed once and held warm across the fleet. Repeat traffic skips the prefill entirely and lands in single digit milliseconds.
Kestrel is the serving layer for teams that treat latency as a product decision. Prefix caching, speculative decoding and deterministic routing, behind one endpoint and one line of setup.
01import { Kestrel } from "@kestrel/sdk"02 03const kestrel = new Kestrel({ region: "us.east" })04 05const stream = await kestrel.infer({06 model: "kestrel/sable8b",07 input: transcript,08 cache: "prefix",09 stream: true,10})11 12for await (const token of stream) {13 process.stdout.write(token.text)14}The claim was approved on the second review, and the adjuster left a note about the roof
Serving production traffic for
/ platform
We rebuilt each of them. The result is a serving path with no cold starts, no queue you cannot see, and a p99 that stays inside two times the p50 under real load.
Shared system prompts are hashed once and held warm across the fleet. Repeat traffic skips the prefill entirely and lands in single digit milliseconds.
Requests carrying the same session key land on the same replica, so the KV cache stays hot for the length of a conversation instead of a single call.
A small draft model proposes, the target model verifies. Two thirds of tokens are accepted on the first pass, which is where the throughput comes from.
Prompts and completions leave nothing behind. No training on your traffic, no logs past the request, and a regional pin if your auditors need one.
/ architecture
The router holds an open connection per replica and knows which cache each one is carrying, so a request is placed rather than queued.
/ models
Open weights we host, plus two we trained. Every one of them speaks the same API, so swapping is a string change and a redeploy.
| Model | Context | First token | Throughput | Input, per million |
|---|---|---|---|---|
| kestrel/sable8b | 128k | 31 ms | 184 tok/s | $0.09 |
| kestrel/sable70b | 128k | 48 ms | 96 tok/s | $0.42 |
| kestrel/quill.vision | 64k | 62 ms | 71 tok/s | $0.55 |
| kestrel/tern.embed | 32k | 9 ms | 2,400 tok/s | $0.01 |
A key, a package and a model string. The quickstart is four minutes end to end, and everything past that is in the reference.