Inference

INT4 Quantization

INT4 quantization uses 4-bit integers, achieving 8x compression from float32. This is aggressive—quality loss is more noticeable than INT8. INT4 is used for fitting large models on consumer hardware. Careful calibration mitigates quality loss. INT4 enables running 70B models on single 24GB GPUs. Quality is usually acceptable for many tasks. INT4 is practical for cost-sensitive deployment.