I brought AI in-house: E2E ~500ms → 120ms, cost down to 1/3–1/10

Once I owned the AI myself, E2E went from ~500ms to ~120ms. What disappeared was the queue (dedicated capacity), the distance (proximity), and finally generation (quantization).

There’s a pitfall on the way, though. The moment you eliminate the wait and the distance, generation — until then the smallest term — becomes the new rate limiter, and left stock it stalls at about 190ms. Only the full set of three wins.

The economics flip too. The variable cost you pay as you use it becomes a fixed cost, and the per-DAU unit price drops to roughly 1/3–1/10. Past a few thousand DAU, self-hosting is cheaper. Speed, cost, and supply sovereignty all line up in one and the same move.

You can’t get rid of this double cost while you’re on a third-party API

In Part 4 I wrote about supply sovereignty — being free of retirements and price increases — as the reason to own your AI. But there was a second motive. And it’s also the head-on answer to the problem I got stuck on in Part 2.

By Part 2, the dominant term in the slowness had moved to the queue and the distance. Break down the ~500ms of E2E latency and generation is a mere 10%, against distance (round trip) 275ms + queue 167ms = about 89%. And this double cost is one I can’t touch with my own hands as long as I keep using a third-party API. The queue is decided by other people’s traffic and the distance by where the server sits — neither of them budges no matter how much you polish your code.

Own it yourself and you can get rid of both. But — the same pitfall is waiting.

Killing the queue and the distance at once

Self-hosting makes two things possible that a third-party API can’t.

• Dedicated capacity → the queue on shared infrastructure disappears. If it’s yours alone, there’s nobody to line up behind. Of course, if demand spikes you can still get a queue on your own hardware. But the difference that settles it is that the queue is now one you control yourself. Tune the GPU, quantize the model to raise the volume it can handle, add servers (I’ll come to each of these in turn). On a shared third-party API, the queue is decided by other people’s traffic. The long-tail risk of wait times stretching absurdly, which I saw in Part 2, was something I could only accept.
• Proximity → the physical distance (275ms round trip) disappears. Put the server in a region close to your users and the round trip is a few tens of milliseconds.

Here’s what I measured.

One premise up front — this is not a comparison between the same models. The third-party API side is running a far higher-quality large model on a purpose-built chip (Part 4). My side is a small model that fits on a cheap GPU. So this isn’t a proper quality-matched comparison; it’s strictly a latency and cost comparison. The recipe for bringing the small model’s quality closer to the large one is covered in the model-specialization series. The claim here isn’t “the same quality, cheaper” — it’s that you can hold on to sovereignty and still land speed and cost that are good enough for how it feels to use.

And that 167ms queue is only the median. As I saw in Part 2, under congestion it could jump to about 5 seconds at p99. What dedicated capacity removes isn’t just the 167ms median — it’s the whole “worst-case tail” you can never see coming. Which is why the impact on perceived speed is bigger than the table’s numbers let on.

E2E: ~500ms → ~190ms. The queue and the distance, 89% of the total, collapsed wholesale. …But look at the table one more time.

But then I notice generation has gotten heavier

Generation (compute) has gone from 50ms to 168ms — more than 3x heavier.

What the third-party API was running on was an ultra-fast chip purpose-built for workloads like translation (an LPU), and generation there was a mere 50ms. The cheap commodity GPU I rent for myself (an RTX 4090), left stock, takes 168ms to generate — more than three times as long. Which means — exactly the same trap as Part 2. I got rid of the queue and the distance, and now that generation (compute) is the new bottleneck. 168ms of the ~190ms E2E is generation — this risks being nothing more than “swapping a queue-and-distance problem for a compute problem.”

The local-optimization trap shows its face here one more time. Kill two of the three and you still don’t win while the third one is left standing.

Killing the third one with quantization

What solved this was quantization (INT4). You compress the model’s weights and sharply raise both the volume a cheap commodity GPU can handle and its speed. Generation came down 168ms → 95ms. (Why merely compressing makes it faster, and why quality barely drops — I broke that mechanism down in the quantization installment of the model-specialization series.)

• Generation compute 168ms → 95ms (speed on par with an expensive data-center-class GPU)
• Volume it can handle 2,200 → 9,600 requests/min (4x)
• Quality −1pt (essentially no degradation, n=100)

That 95ms doesn’t reach the third-party purpose-built chip (the LPU’s 50ms). But it doesn’t need to be that fast. The queue (0) and the distance (14–36ms) are already gone, so even at 95ms rather than 50ms, E2E is about 120ms — a blowout win over the third-party API’s 500ms. That’s how I killed all three at once: queue (dedicated), distance (proximity), generation (quantization) — in terms of Figure 1 up top, I’d reached the third row. Even with generation slower than a purpose-built chip, you win overall.

Cost flips from variable to fixed

Just as big as the speed is the flip in the cost structure.

A third-party API is a variable cost — you pay as you use it. And under congestion you hit a global cap (429 errors). Self-hosting is a fixed cost — rent one GPU and the price doesn’t change no matter how many times you hit it.

A consumer GPU costs about 1/8 as much per month as a data-center GPU in the same class, and put INT4 on it and you get performance on par with that premium card. Divide this fixed cost back out by the third-party API’s per-DAU price and somewhere past a few thousand users, self-hosting becomes the cheaper option (the heavier the usage, the sooner it crosses over). Below that the GPU sits idle, so the third-party API is the rational choice; above it, self-hosting wins on both cost and latency. Note that the “1/3–1/10 of the variable cost” in the table is the unit price at a scale where the GPU is comfortably full (just after break-even there’s slack on the GPU, so per user it comes out pricier).

And this “fixed cost” property changed the business decision itself. On a variable cost, the more it gets used the more cost swells, so your instinct becomes “keep the free features modest.” On a fixed cost, one more call is essentially free. Which flips the answer to “how much do we open up for free.”

But that “fixed cost” hides work wearing the face of free

That said, self-hosting doesn’t end at “rent a GPU.” All the operations the third-party API was carrying for me — incident response, monitoring, model updates, redundancy for demand swings (N+1), procuring GPUs — become my job, wholesale. The costs so far only look at “the GPU bill”; the fixed cost of people and operations rides on top of that. So self-hosting only becomes rational when both are in place: the scale (break-even) and an organization that can carry the operations. Cheap and sovereign don’t come for free.

Summary — what changed in speed and cost

Here are the results of killing all three at once — dedicated capacity (queue), proximity (distance), quantization (generation) — split into two. On speed, it wasn’t just the median that moved but the whole distribution; on cost, it wasn’t the amount but the “shape” of the cost itself that changed — those are the two that worked, and neither was reachable while I stayed on a third-party API.

Lessons

1. Self-hosting lets you kill the queue and the distance yourself. On a shared, remote third-party API, both were in someone else’s hands. Dedicated capacity, proximity and scale-out took E2E from ~500ms to ~120ms.
2. But it doesn’t hold up if you leave the “third one” standing. Run stock on a cheap commodity GPU and the bottleneck you thought you’d killed simply moves to compute (the Part 2 trap, all over again). You only win with the full set: dedicated + near + quantized.
3. Cost flips from variable to fixed, and supply sovereignty comes along in the same move. Past a certain scale it gets cheaper, the congestion cap goes away, and even the decision about how much to open up for free changes. But only when you’ve measured right and killed the “third one.”

---

Appendix: raw data