Most comparisons of knowledge distillation vs quantization treat them as two answers to one question, then hand you a table of adjectives. They are not two answers to one question. Quantization changes how the numbers in your model are stored and leaves the computational graph exactly as it was. Distillation trains a different, smaller model to reproduce the behaviour of the one you have. One is a compression of an artefact you already validated. The other is a new artefact.
That distinction is invisible if the thing you are optimising is a serving bill. It is decisive if the model has to hit a deadline and you have to defend it afterwards.
The short answer
Reach for quantization first. It requires no training run, it is reversible because you keep the original weights, and on a graph that stays intact you can reason about what changed. Reach for distillation when the architecture itself is the reason you are missing the deadline, when no amount of coarser arithmetic on the current model will close the gap.
Then, if the budget allows, do both in that order. The measured evidence for the combination is stronger than the evidence for either alone.
What each one actually changes
Quantization: same graph, coarser numbers
Quantization reduces the numerical precision of weights, and sometimes activations, from 16 or 32 bit floats down to 8, 4 or fewer bits. Layer count, attention heads and vocabulary are untouched. Post-training quantization needs only a calibration set. Quantization-aware training simulates the effect during training and costs a training pipeline.
The reported figures for the post-training case are the reason it is the default. A survey of constraint-driven optimisation across production deployments records GPTQ quantizing billion-scale models to 3 or 4 bits for roughly a 4x reduction in weight-storage footprint against FP16, holding perplexity within 0.25 points of the FP16 baseline, and completing in under 4 GPU-hours for a 175B model on the hardware used in that study. AWQ, which protects roughly 1% of salient weight channels identified by activation magnitude, reported accuracy within 0.5% of FP16 and a 1.45x throughput advantage over GPTQ on the edge hardware tested. Both figures move with architecture, task and device, but the order of magnitude is the point: this is an afternoon of engineering, not a project.
Distillation: a new model that has to earn its own evidence
Distillation trains a small student to match a large teacher, typically against the teacher's full output distribution rather than against hard labels. The student learns more than the correct answer, which is why a distilled model of a given size can outperform the same architecture trained from scratch.
You pay for it three times. Once in compute, because it is a real training run. Once in dependency, because you need access to a teacher good enough to be worth copying. And once in data: the same constraint framework places large-scale distillation in the abundant-data regime of hundreds of thousands of examples, with under a thousand labelled examples putting you in parameter-efficient fine-tuning territory and zero task-specific data restricting you to training-free methods entirely.
That last point is worth stating plainly, because it settles a lot of arguments before they start. If you do not have the data, the comparison is over.
What the measurements say when you compare them directly
Two recent studies measure the trade-off, and they point in different directions. Both are correct.
The first is an empirical comparison of edge deployment operators across mobile, edge and cloud tiers covering quantization-aware training, pruning and knowledge distillation, two post-training operators, and their hybrids, on four vision and language models. Its finding on the non-hybrid operators: when lower latency is the goal, the distilled operator is the better alternative on both the mobile and edge tiers, at the cost of small to medium accuracy loss. Operators involving distillation also showed lower latency in resource-constrained tiers than operators involving model partitioning. On latency at the edge, distillation wins.
The second is UniComp, a unified evaluation of pruning, quantization and distillation over six techniques and more than 40 datasets, scored on performance, reliability and efficiency together. Its finding: quantization provides the best overall trade-off between retained performance and efficiency, while distillation yields strong runtime acceleration at high computational cost.
These disagree because they are answering different questions. The first optimises latency at a deployment tier and finds the smaller architecture wins. The second weighs retained capability against what you spent to get it, and finds the technique that costs nothing wins. If your budget is a millisecond figure, read the first. If your budget is engineering time, read the second.
The failure mode nobody benchmarks
The most useful result in this literature is not about speed at all. UniComp reports a consistent knowledge bias across compression methods: knowledge-intensive task performance is relatively preserved, while reasoning, multilingual and instruction-following capabilities degrade substantially, with quantization a partial exception.
Read that as a warning about your acceptance test. An aggregate accuracy number measured after compression is close to meaningless, because the capability compression takes first is usually not the one the benchmark is measuring. If the deployment depends on the model reasoning about a situation it has not seen, a preserved knowledge score tells you nothing about whether it still can.
The constraint survey reaches the same place from the deployment side: mission-critical systems should hold to well-calibrated 8-bit quantization and under 0.5% degradation on the chosen metric, and should evaluate tail-risk failures and subgroup performance rather than relying on an aggregate.
Choose by the constraint that binds
Run these checks in order. Most of the time the answer falls out before you reach the bottom.
- Retraining budget of zero. No compute to alter weights means distillation is not on the table. Post-training quantization and training-free sparsification are what you have. This eliminates the comparison entirely for a large share of real deployments.
- No task-specific data. Same outcome. Distillation needs something to distil against.
- Memory is the binding constraint. Edge and mobile targets typically offer under 4 GB of VRAM. Precision reduction is the direct lever on parameter storage, so start with quantization.
- Accuracy tolerance under half a percent. Calibrated 8-bit, and validate on the specific capability rather than the aggregate.
- A hard real-time budget under 200 ms. This is where distillation earns its cost. A smaller architecture, or early exiting, is what reaches that band. Coarser numbers on the same graph often will not.
- Interactive, 200 ms to 2 seconds. Quantization alone usually gets there.
The general form of this, and the reason it works better than a technique-first comparison, is that deployments are characterised by which constraint is actually binding, not by which algorithm is fashionable. Our own work on model compression for edge AI starts from the same place: fix the budget first, then pick the operator.
Compression under a control deadline
Everything above still assumes the model is answering questions. Change one thing, and the calculus changes with it: let the model close a loop on a live physical or biological process.
Now the deadline is not set by user patience. It is set by process dynamics, and missing it is not a slow response, it is a control action arriving after the state it was computed for has passed. In our own control stack, documented across the OptiVX and IntelliBot programmes, end-to-end edge latency runs at 285 ms, reduced from 1.2 seconds, with the safety layer answering in under 2 ms, growth-model accuracy at R-squared above 0.95, and validation on a 500 L pilot basin showing zero safety violations across 400 simulated years.
Those last two numbers are the ones that change the compression decision, and no page comparing these techniques for language-model serving has any reason to mention them. They are a validation record. They were expensive to produce, and they attach to a specific model.
Quantize that model, and the computational graph is unchanged. The perturbation is bounded and characterisable, so the existing evidence can be argued forward and re-checked rather than regenerated, and the safety layer, which is a separate guarantee, still holds its own budget. Distil it, and you have a different function approximator. The evidence does not transfer. The sim to real transfer work restarts, the safety argument is re-made, and the campaign that produced those 400 simulated years runs again.
For a controller that acts on something physical, that asymmetry usually decides it, and it decides it long before anyone compares accuracy deltas. It is also why the compression choice belongs in the design of the safe reinforcement learning for industrial process control stack rather than being bolted on at deployment. Choosing the target hardware and the latency budget before the controller is trained means the architecture that gets validated is the one that ships.
Use both, in the right order
The strongest measured configuration in the edge study is not either technique alone. It is the combination of distillation with post-training quantization, which the authors recommend over non-hybrid operators when lower latency is required at the edge, at small to medium accuracy drop.
Practitioner consensus agrees, and describes the same staged pipeline of distillation followed by quantization. The order is not arbitrary. Distil first, then quantize the student. Reversed, you distil from an already degraded teacher and carry that degradation into a model you cannot then un-train.
FAQ
Is knowledge distillation better than quantization for edge devices?
For latency, and only as a non-hybrid choice. The measured edge comparison finds the distilled operator the better non-hybrid option on both mobile and edge tiers when lower latency is the goal, at small to medium accuracy cost. If memory is the binding constraint rather than latency, quantization reaches the target with no training run at all.
Can you quantize a distilled model?
Yes, and it is the strongest measured configuration in the edge study. Distil to a smaller architecture, then quantize the student. Doing it in the other order means distilling from an already degraded teacher.
How much data do you need for knowledge distillation?
The constraint framework places large-scale distillation in the abundant-data regime, hundreds of thousands of examples. Under roughly a thousand labelled examples you are in parameter-efficient fine-tuning territory, and with no task-specific data at all you are restricted to training-free methods.
Does compression hurt reasoning more than it hurts accuracy?
Compression shows a consistent knowledge bias: knowledge-intensive performance holds up while reasoning, multilingual and instruction-following degrade substantially, with quantization a partial exception. Test the specific capability the deployment depends on, not an aggregate score.
Which is cheaper as a one-off engineering job?
Post-training quantization, by a wide margin. The reported GPTQ run quantized a 175B model in under 4 GPU-hours with no training pipeline. Distillation requires a full training run against a teacher, plus the data to do it with.