Unless you’ve been living under a rock this week, you’ve heard about the TypeSafe AI launch.
They raised $40M and shipped Jev, a “System One” model that returns a label, a probability and a span instead of a paragraph. No prompt to engineer, no output to parse, and it can’t hallucinate prose because it never writes any. You ask “is this refund request legitimate?” and you get true, 0.94.
We watched the launch video and had a strange feeling. We’d already built that. We just never gave it a name.
CloudRaker’s Paperwork API has been making structured decisions over documents for months. Classify a page. Check whether a statement is actually backed by the contract. Pull six named fields out of a form. The models behind those calls are small and answer in tens of milliseconds. We built them for paperwork, and it never occurred to us that they were a product on their own.
TypeSafe made the case for us. The reception they got proved two things: there’s a market for cheap, fast, typed decisions, and AI can be useful to software, not just to humans reading its output.
We had the models. We had everything around them too: API keys, login, per-organization rate limits and billing all live in the Paperwork API already. So the first attempt was obvious: expose the decision routes on the existing gateway and call it a day.
Where the time went
The Paperwork API was built for large, long-running work. It runs pipelines over two-hundred-page PDFs, keeps signature envelopes alive for weeks, and coordinates redaction jobs that talk to three workers and a GPU. Every request carries authentication, fine-grained authorization, tenant context, structured logging, tracing and metering. When a job runs for forty seconds, all of that is invisible.
When a job runs for a few milliseconds, all of that is the job.
So we benchmarked it. The model answered fast every single time. The request didn’t, because the authorization round trips, the context propagation, the logging and the network hops between workers all stacked up in front of it. Our traces in Dash0 made it painfully obvious: in the worst cases the overhead was twelve times the inference time. The model was, by a wide margin, the cheapest part of the request.
sequenceDiagram
participant C as client
participant G as gateway
participant A as auth worker
participant Z as authz worker
participant T as tenant / metering
participant M as model
C->>G: POST /classify
G->>A: verify key
A-->>G: ok
G->>Z: can this org call this route?
Z-->>G: ok
G->>T: load tenant context, open trace
T-->>G: ok
G->>M: inference
M-->>G: label, probability
G->>T: log, meter
G-->>C: answer
Note over G,T: hops, context propagation, logging
Note over M: ~10 msThat settled it. The decision API would not be a route on the gateway. It would be its own service, stripped to the bones, with nothing on the request path that didn’t absolutely need to be there.
Building Milliseconds.ai
, the new inference API for decision-machine-1, took two days, and the reason is simple: every piece of infrastructure already existed. We only had to rearrange it for speed.
The shape
A request has four jobs before it reaches a model: prove who’s asking, check that the organization is allowed to ask right now, find a GPU with a free slot, and leave a record for billing once it’s done. On our main gateway, each of those is a hop to another worker. Here, each of them is a binding on the one Cloudflare Worker that answers the request, and the two stateful pieces are Durable Objects that keep their state in memory.
flowchart LR
C[client] --> W["api.milliseconds.ai<br/>Worker · Hono"]
W -. "verdict cache<br/>60 s per colo" .-> W
W --> NS["namespace DO<br/>one per organization<br/>keys · buckets · ledger"]
W --> SCH["scheduler DO<br/>one per region<br/>leases · queue · breaker · spot broker"]
SCH --> VPC[Workers VPC] --> T[Cloudflare Tunnel] --> R["inference runner<br/>spot GPU"]
W --> D1["D1<br/>hashed API keys"]
W --> AE["Analytics Engine<br/>one point per request"]
NS --> M["metering Worker<br/>service binding"] --> SM[Schematic<br/>credits]The Worker itself is a Hono app. Two Durable Objects do most of the work. Everything else is a binding: D1 for the hashed keys, Analytics Engine for one data point per request, and a service binding to metering.
Two Durable Objects
The scheduler
A GPU runner can serve a bounded number of inferences at once, and past that number latency falls off a cliff. Someone has to hold the leases, queue the overflow and stop sending traffic to a box that just got preempted.
That someone is a Durable Object, one instance per GPU region, each pinned to its region with a location hint . It holds every slot in its region in memory, prefers GPU slots and falls back to CPU slots when the GPUs are full. A call takes a lease, POSTs through the tunnel and releases the lease when the answer comes back. If nothing frees up within the wait window, the client gets a 529 and a retry hint.
When a call fails, the retry goes to another GPU rather than to another slot on the same dead host, and the failed slot is skipped for thirty seconds.
It also brokers the spot fleet itself. It watches slot pressure, asks for more spot GPUs before the queue backs up and lets boxes go when demand drops, so capacity follows load instead of sitting at a fixed size we’d have to guess and pay for. This is the entire reason we can be this cheap without subsidizing inference.
That one object is the load balancer, the queue, the circuit breaker and the capacity broker for its region. It’s about a hundred lines of TypeScript. Nothing is persisted. An in-flight lease keeps the object alive, and a cold start just rebuilds the pool from scratch.
flowchart TD
REQ[request arrives] --> LEASE{free slot?}
LEASE -- GPU --> G[lease GPU slot]
LEASE -- GPUs full --> CPU[lease CPU slot]
LEASE -- none --> Q[queue]
Q -- slot frees --> LEASE
Q -- wait window over --> E["529 + retry hint"]
G --> CALL[POST through tunnel]
CPU --> CALL
CALL -- ok --> REL[release lease · return answer]
CALL -- fails --> SKIP["skip that slot 30 s<br/>retry on another GPU"] --> LEASE
LEASE -. "slot pressure" .-> BROKER["spot broker<br/>add GPUs before queue backs up<br/>release when demand drops"]
BROKER -. "new / dropped slots" .-> LEASEThe namespace
When speed matters you get smart about lookups, so we went as far as changing the shape of our API keys. A Milliseconds.ai key looks like sk-ms-{namespace}-{entropy}, where the namespace is the organization id. The key alone tells the edge which Durable Object owns it, before anything is looked up anywhere.
The namespace object runs one instance per organization. It mirrors that organization’s keys from the database into its own storage and keeps them in memory. It owns two token buckets, one for requests per minute and one for input tokens per minute, plus a usage ledger in fifteen-minute buckets. Every request asks the object to admit it, with the token count already computed from the body, and the object answers yes, rate-limited or billing-blocked, handing back the OpenAI-style x-ratelimit-* headers along the way.
The ledger ships to our metering Worker over a service binding , and the metering Worker burns credits in Schematic . A billing verdict comes back the same way and flips one boolean. The hot path never touches a database.
flowchart TD
K["sk-ms-{namespace}-{entropy}"] -- "namespace = org id" --> NS["namespace DO · one per organization<br/>keys · two token buckets · 15 min ledger<br/>all in memory"]
D1[("D1 · hashed keys")] -. "mirror" .-> NS
NS -- "yes · rate-limited · billing-blocked<br/>+ x-ratelimit-* headers" --> W[Worker]
NS -- "ledger · service binding" --> M[metering Worker]
M -- "burn credits" --> SM[Schematic]
SM -- "billing verdict" --> M
M -- "flip one boolean" --> NSSchematic deserves a proper shoutout here. The plans, the credits, the entitlements and the customer-facing usage page were not built for Milliseconds.ai. They were already there for Paperwork. Adding a new product with its own credit type and its own burn rate was a configuration change plus one call from the metering Worker. Billing is usually the part that pushes a launch back a week. This time it took an afternoon, and four hours of that afternoon went into arguing about what a token should cost.
The edge in front of both
A Durable Object call is a round trip, and paying for one on every request would double the latency floor of a model that answers in tens of milliseconds. So the Worker caches the verdict per key in the Cache API for sixty seconds, in every location that sees the key. The cached verdict holds the key id, the block reason if there is one, and the rate-limit headers. Accounting runs after the response, off the critical path, which means a burst can slip a few requests past the limit before the block lands. We’re fine with that. It’s the difference between an API that costs one round trip and one that costs two.
sequenceDiagram
participant C as client
participant W as Worker
participant CA as Cache API (per colo)
participant NS as namespace DO
participant M as model
C->>W: request with key
W->>CA: verdict for key?
alt cached (< 60 s)
CA-->>W: key id · block reason · headers
else miss
W->>NS: admit? (token count)
NS-->>W: verdict + headers
W->>CA: store 60 s
end
W->>M: inference
M-->>W: answer
W-->>C: answer + x-ratelimit-*
W--)NS: account usage (after response)The tunnel
The GPUs aren’t on Cloudflare. They’re spot instances spread across managed instance groups in several regions.
Each VM runs two containers: the runner and cloudflared
. The tunnel
is configured from Cloudflare’s side, so a VM only needs its token to join. Every VM registers a connector against the same tunnel. On the Worker side, that tunnel is a Workers VPC
binding, so the scheduler reaches the runners with a plain fetch() on that binding. No public IP, no public hostname, no firewall rule, no load balancer, no service token to rotate. The GPU hop never touches the public internet. When a VM gets preempted it drops its connector, and the tunnel keeps serving from the others.
flowchart LR
SCH["scheduler DO"] -- "fetch() on binding" --> VPC[Workers VPC] --> TUN[one Cloudflare Tunnel]
subgraph R1["region A · managed instance group"]
V1["VM · runner + cloudflared"]
V2["VM · runner + cloudflared"]
end
subgraph R2["region B · managed instance group"]
V3["VM · runner + cloudflared"]
V4["VM · runner + cloudflared<br/>(preempted · connector dropped)"]
end
TUN --- V1
TUN --- V2
TUN --- V3
TUN -.- V4Five years ago I would have spent a week on this part alone.
Same rails
The services powering Milliseconds.ai aren’t special. They started from the same worker template as every other service we run, with the same configuration conventions, the same three environments and the same CI. They publish their API spec the same way, so the rest of the platform gets a typed client without anyone writing one. Secrets come from the same vault through the same sync. Deploys go through the same pipeline and the same release cut.
Traces land in Dash0 next to everything else, so the day it went live we could already put a Milliseconds.ai request and a Paperwork request side by side on the same dashboard. The GPU hop stays inside Cloudflare’s network, and the same access policies that guard our admin surfaces sit in front of everything else.
That bought us two things.
The first is time. Nothing about keys, secrets, environments, deploys, logging or metering got decided during the build, because it had all been decided months ago for Paperwork. Every hour went into the two Durable Objects and the tunnel.
The second is compliance. CloudRaker is SOC 2 Type II, and the perimeter is defined by exactly these conventions: which accounts we deploy to, which secret store we read from, which access policies apply, which deploy path is allowed and where the logs go.
Because Milliseconds.ai runs on the same rails, it landed inside that perimeter on day one. No new vendor review, no new control, no exception to document. A separate stack for a side product would have been impossible to certify.
Learnings
A year ago the same product would have taken a team, a Kubernetes cluster and a quarter. The models didn’t get that much smaller in that time, but the infrastructure got closer.
We also learned something about our own platform. A gateway built for forty-second jobs is the wrong gateway for four-millisecond jobs, and no amount of tuning changes that. The right move was to rebuild the thin path, and that was affordable because the thin path is mostly bindings.
We’re a small, bootstrapped company, and we compete by being early, by being cheap to run, and by solving headaches for our clients’ engineering teams. That week proved that the way we built our platform, and the conventions around it, are not implementation details. They’re core to our strategy.
For months, this much ceremony for a small team looked like overengineering. A service template, generated configuration, a central release cut, secret sync per environment, infrastructure as code down to the DNS records. All of it was slower than it needed to be for any single service, but it turned out to be the only reason a whole new product could ship in two days.
So, yes, milliseconds.ai is live. The API docs are at docs.milliseconds.ai . We ended up marginally cheaper and faster than TypeSafe AI, which lets us offer a free plan.
Some personal shoutouts: the whole team at CloudRaker, who tore up the week’s schedule and worked extra hard to push this through. Nina, for being a much, much better launch video talking head than I am. And a certain rowing fanatic, who was sitting on one of the coolest domain names you could dream of and texted me the transfer auth code five minutes after I told him what we were up to.