Edge Inference: Latency Budgets That Hold

"The model runs in 18 milliseconds." That sentence usually means: on a workstation, with a warm cache, on a single image, at p50, once. A production line cares about the ninety-ninth percentile at 3pm in a cabinet that has been running for nine hours.

After chasing too many mysterious slowdowns, I stopped optimising for average inference time and started writing a latency budget.

Budget the whole loop

Inference is one stage in a pipeline. Write down every stage with a ceiling, and make the sum smaller than the cycle time you were promised.

CopyAcquire + decode        4 ms
Resize / normalise      2 ms
Inference               9 ms
Post-process + NMS      3 ms
Decision + I/O          2 ms
Headroom                12 ms
------------------------------
Cycle time              32 ms

Headroom is not padding; it is the part of the budget that pays for thermal throttling, a busy host, and the next feature request. A budget with no headroom is a bug report waiting to happen.

Measure the tail

Report p50, p95 and p99 separately, over a window long enough to include garbage collection and thermal drift. Warm the model before you start the clock, and include the first batch after a restart — cold-start spikes are exactly what operators notice.

The usual suspects

• Memory copies. A zero-copy path between capture and inference often beats a faster kernel.
• Dynamic shapes. Fixed input sizes let runtimes pre-allocate; variable batches rarely do.
• Batching by hope. Batching improves throughput and quietly destroys tail latency. Batch only when the arrival rate justifies it.
• Thermal limits. Sustained load is a different benchmark from a burst. Test for an hour, not a minute.

Quantisation with a contract

INT8 is close to free performance, but only if accuracy is guarded by a frozen, representative evaluation set — including the rare defect classes from the field. I treat the quantised model as a separate release with its own acceptance criteria, not as a flag I flip on a Friday.

A latency budget is a contract with the person standing next to the machine. They do not care about your median.

A lesson from a very hot cabinet

Degrade on purpose

Decide in advance what happens when the budget is exceeded: drop to a lower resolution, skip every other frame, or route the frame to a slower secondary path. A system that degrades deliberately is trustworthy; one that silently queues forever is not.

Performance engineering at the edge is mostly the discipline of being specific about when, where, and for how long. Everything after that is measurement.