Quantization and Compression Techniques for Model Deployment: Optimizing Neural Networks for Production

Abstract

Compression research reports compression ratios, and deployment engineering pays for something else. This paper reports a CPU benchmark study in which the binding constraint was neither model size nor latency but key-value cache memory, which grows linearly with sequence length and comes to dominate the inference memory budget well before weight storage does. Post-training INT8 quantization was applied to reduce that footprint, with latency treated as a secondary and unguaranteed benefit. The central finding concerns the gap between the compression ratio and the realized gain. Memory reduction followed the quantization ratio reliably. Runtime did not, because the software stack did not sustain low-precision execution end to end: several operators lacked native INT8 kernels, so tensors were repeatedly upcast to floating point and downcast back to INT8 between operations, and the conversion overhead consumed much of the arithmetic saving. The practical consequence is that a compression decision on CPU is governed by operator coverage across the execution path rather than by the nominal precision of the weights, and a technique that halves arithmetic width while doubling conversions can be a net loss. INT4 was considered and not pursued, on grounds of accuracy risk and the same coverage limitation in a more severe form. The paper also proposes a promotion process for moving a compressed model toward production against an uncompressed reference baseline. This is a benchmark feasibility study. No model was deployed to production or pilot, no quantitative result is reported, and Section IX states precisely what was not measured.