Inference

Choosing precision (quantization)

Hardware requirements for LLaMA

8-bit

4-bit

General setup

8-bit setup

4-bit setup

Online inference server

Benchmark