llama-cpp-python

🦙 Python Bindings for `llama.cpp`

Installation

Installation Configuration

Supported Backends

Windows Notes

MacOS Notes

Upgrading and Reinstalling

High-level API

Pulling models from Hugging Face Hub

Chat Completion

JSON and JSON Schema Mode

JSON Mode

JSON Schema Mode

Function Calling

Multi-modal Models

Speculative Decoding

Embeddings

Adjusting the Context Window

OpenAI Compatible Web Server

Web Server Features

Docker image

Low-level API

Documentation

Development

FAQ

Are there pre-built binaries / binary wheels available?

How does this compare to other Python bindings of `llama.cpp`?

License