Almost every guide to edge AI model compression is organised the same way. Here is pruning. Here is quantization. Here is knowledge distillation. Pick one, or pick the fashionable combination, and see how small the model gets.
That ordering is backwards for anything that has to hit a deadline. A deployed model does not get to choose how fast it must answer. The process it is attached to already fixed that number, before anyone chose an architecture, and the only interesting engineering question is how to land underneath it with the accuracy still intact and the evidence still standing.
The budget comes first, and it is not yours to choose
Fix the deadline before touching the model. In industrial deployments it comes from the loop, not from the user: a study of latency-aware edge architectures for industrial IoT reports achievable budgets of roughly 1 ms for servo loops and 6 to 12 ms for machine vision tasks when edge compute is combined with deterministic networking, and models those budgets as partitions across communication, computation and execution layers. Your inference time is one term in that partition, not the whole of it.
Those numbers also settle the architecture question on their own. A practitioner account of latency-critical on-device AI puts a network round trip from a mid-tier data centre at 30 to 80 ms before a single token is generated. Against a 6 ms vision budget, the round trip alone is an order of magnitude over. No routing strategy recovers that. The model runs at the plant or it does not run.
Why parameter count and FLOPs lie about latency
The most useful recent finding in this literature is a negative one. The authors of Prune-Quantize-Distill state it directly: common compression proxies such as parameter count or FLOPs do not reliably predict wall-clock inference time. Their headline example is unstructured sparsity, which reduces model storage while failing to accelerate, and sometimes slightly slowing, standard CPU execution, because irregular memory access and sparse-kernel overhead eat the arithmetic saving.
Read that carefully, because it invalidates a common project plan. A team sets a compression target as a percentage of parameters removed, hits it, ships, and finds the deadline is still missed. Nothing went wrong in the compression. The target was measuring the wrong thing.
The rule that follows is unglamorous and load-bearing: evaluate compression choices in the joint accuracy, size and latency space using measured runtime on the target hardware, rather than proxy metrics. Every number below is a measured one, and every number below is also specific to the hardware it was measured on, which is the same caution running in the other direction.
The ordered pipeline: prune, quantize, then distil
The same work tests whether the order of the three standard operators matters. It does.
INT8 quantization-aware training carries the runtime. It is the stage that produces the dominant wall-clock benefit, because it changes the arithmetic the deployment backend actually executes rather than the shape of a graph the backend still has to walk.
Pruning earns its place as a pre-conditioner. Not primarily as an accelerator. Reducing effective capacity first improves the robustness of the subsequent low-precision optimisation, which is a different and more defensible reason to prune than the storage figure usually quoted for it.
Distillation goes last, to recover what the first two cost. Applied at the end, it restores accuracy inside the already constrained sparse INT8 regime without changing the deployment form, so the artefact you validated is the artefact you ship.
Across ResNet-18, WRN-28-10 and VGG-16-BN on CIFAR-10 and CIFAR-100, that fixed ordering reached 0.99 to 1.42 ms CPU latency with competitive accuracy and compact checkpoints, and beat any single technique alone on the joint frontier. Controlled ordering ablations under a fixed 20/40/40 epoch allocation confirmed the permutation matters, not just the ingredients.
For the two-technique version of this decision in isolation, including when the architecture itself is the reason you are missing the deadline, we covered knowledge distillation vs quantization separately.
Half your latency is in the runtime, not the model
Compression is one of two levers, and teams routinely pull it hard while leaving the other one untouched. A benchmark of edge inference strategies for industrial machine vision compared PyTorch, ONNX Runtime, OpenVINO and TensorRT across CPU and GPU platforms, and the spread between frameworks running the identical model is large enough to swallow a compression stage.
On the Intel desktop CPUs, OpenVINO gave the lowest inference time for every model tested, reducing YOLOv8 inference time by 30 to 60 percent. On NVIDIA GPUs, TensorRT was fastest for the CNNs, cutting YOLOv8 latency by roughly 30 to 56 percent against the next best framework.
Then the results stop being tidy, which is the part worth internalising. On the Jetson CPU the ranking inverts and ONNX Runtime wins, with Grounding DINO running at 13580 ms under ONNX Runtime against 102720 ms under OpenVINO, a gap of more than seven times on the same model and the same board. And on desktop GPUs, plain PyTorch beat TensorRT for that transformer model, with TensorRT between 1.6 and 2.1 times slower than the unoptimised baseline.
So framework selection depends on the target hardware and on the model family, and neither dependency is predictable from the datasheet. The practical consequence is a sequencing one. The five-stage deployment methodology derived from a PRISMA review of neural networks on embedded systems puts requirement definition, model selection, optimization, hardware alignment and deployment in that order. In projects that miss their budget, hardware alignment has usually happened last, after the compression work was already finished against a runtime nobody had benchmarked.
Budget the tail, not the mean
A mean latency comfortably inside the budget is not the same as meeting the budget. For a loop that runs continuously, the number that matters is the one that does not blow the deadline on the worst iteration, under whatever else the board is doing at that moment.
A review of timing guarantees for AI inference in embedded systems frames this as the open problem rather than a tuning detail: predictable execution of DNN inference under strict timing constraints, with limited memory capacity, energy consumption, multi-DNN scheduling and heterogeneous resource management as the named obstacles, and time-predictable scheduling frameworks for consistent inference latency as an unmet need. If two models share the board, their interaction is part of your latency, and it is not in either model's benchmark.
Which raises the question of how to produce an honest number at all. The industrial benchmark above used a protocol worth copying: repeat inference 1000 times, discard the first 10 runs so warm-up does not contaminate the result, hold batch size at 1 to match single-sample deployment, synchronise the device on GPU measurements, and report mean and standard deviation over the remaining 990. A single timed forward pass on a warm cache is not a measurement.
What changes when the model closes a loop
Everything above holds for any latency-critical deployment. One change turns it into a different problem: let the model act on a live physical or biological process.
A missed deadline is then not a slow response. It is a control action arriving after the state it was computed for has passed, applied to a system that has already moved. And the model is no longer just accurate or inaccurate. It is validated or it is not.
From 1.2 seconds to 285 milliseconds
Our own control stack, built through the OptiVX and IntelliBot contract research programmes, runs end-to-end edge inference at 285 ms, reduced from 1.2 seconds, with the safety layer answering in under 2 ms and growth-model accuracy at R-squared above 0.95. It was validated on a 500 L pilot basin with zero safety violations across 400 simulated years.
The two-tier structure is deliberate and it is the part that generalises. The safety layer holds its own budget, three orders of magnitude tighter than the model's, because a guarantee that has to wait for a neural network is not a guarantee. Compression pressure lands on the 285 ms tier. The 2 ms tier is not a place to put a compressed model at all.
Compression as an evidence question
The 400 simulated years are the expensive asset, not the weights. That validation campaign attaches to one specific model, and the choice of compression operator decides whether it survives.
Quantize, and the computational graph is unchanged. The perturbation is bounded and characterisable, so the existing evidence can be argued forward and re-checked rather than regenerated. Distil, and you have a different function approximator wearing the same job title. The evidence does not transfer, the sim to real transfer work restarts, and the safety argument is made again from the beginning.
For a controller acting on something physical, that asymmetry usually decides the operator before any accuracy delta is compared. It is also the argument for settling the target hardware, the runtime and the latency budget during the design of the safe reinforcement learning for industrial process control stack, rather than treating compression as a deployment-week task. Compress before you validate, and you validate the thing that ships. Compress after, and you pay for the campaign twice.
A sequence that survives contact with a plant
Take the five-stage methodology and fill in the stage the surveys leave abstract.
- Take the deadline from the process. Servo loop, vision cycle, actuator response. Then subtract the communication and execution terms, because the inference budget is what remains after them, not the whole figure.
- Fix the hardware and the runtime, and benchmark them before compressing. Framework choice on that exact board can be worth more milliseconds than a compression stage, and it moves with the model family.
- Measure the uncompressed baseline on that stack. In wall clock, with warm-up discarded, at deployment batch size. Now you know the gap you are closing.
- Compress in order, against measured runtime. Prune to pre-condition, INT8 quantize for the runtime win, distil last to recover accuracy, and re-measure after every stage rather than at the end.
- Re-validate on the capability the deployment depends on. Not on an aggregate benchmark score.
The review that produced those stages also flags what is still missing across the field: limited support for ultra-low-precision inference, wide variability in hardware toolchains, and the absence of standardised holistic benchmarking. That last gap is why the sequence above ends where it does. Nobody is going to hand you a number for your board. You measure it.
This is the method we bring to contract research and development services where a model has to leave the notebook and act on something that can be damaged.
FAQ
What is model compression in edge AI?
Reducing the compute, memory and precision demands of a trained network so it can run on the device rather than in a data centre. The standard operators are pruning, quantization and knowledge distillation, and the systematic review of embedded deployment treats them as answers to a constraint the target hardware imposes, alongside lightweight architectures and inference-level optimisation.
How much accuracy do you lose to INT8 quantization?
It depends on calibration and on the task, and the honest answer is that you measure it. As a target, the constraint-driven optimisation framework holds mission-critical systems to well-calibrated 8-bit with under 0.5 percent degradation on the chosen metric, and tells you to evaluate tail-risk failures and subgroup performance rather than trusting an aggregate score.
Can you compress a model without a training run?
Yes. Post-training quantization needs only a calibration set, which is why it is the default first move on any project with no compute to spare. Quantization-aware training and distillation both require a training pipeline, and distillation additionally requires a teacher and the data to distil against, as the edge deployment comparison sets out across its operators.
How should edge inference latency be measured?
On the target hardware, repeated, with warm-up discarded. The published industrial protocol runs 1000 iterations, drops the first 10 to avoid initialisation effects, holds batch size at 1, synchronises the device for GPU timing, and reports mean and standard deviation over the remaining 990 runs.
Does compressing a model invalidate an existing safety validation?
It depends on the operator. Quantization leaves the computational graph intact, so the perturbation is bounded and the existing evidence can be argued forward and re-checked. Distillation produces a different model, and the validation campaign is re-run against it.