Use a smaller set of values

Instead of storing each weight at high precision, a quantized representation uses fewer possible values, plus information needed to interpret them. QLoRA demonstrates a training approach using a frozen 4-bit base model and trainable adapters.

A useful back-of-the-envelope estimate

WEIGHTS ONLY · DECIMAL UNITS

Eight billion weights at 16 bits take about 16 GB. At 4 bits, their raw values take about 4 GB.

That is not the total memory required to run the model. Quantization metadata, activations, runtime allocations and the cache for context consume additional space. A longer conversation can therefore raise memory use even when weights stay the same.

Smaller does not always mean faster

Quality and speed depend on hardware, kernels, precision choices and the task. Test a representative workload rather than choosing solely by file size. Running locally also requires checking the model license and the application's data handling.

KEEP EXPLORINGMake sense of another idea↗