LLaMA.cpp HTTP Server

Usage

Common params

Sampling params

Server-specific params

Multimodal support

Built-in tools support

Build

Build with SSL

Quick Start

Unix-based systems (Linux, macOS, etc.)

Windows

Docker

Using with CURL

API Endpoints

GET `/health`: Returns health check result

POST `/completion`: Given a `prompt`, it returns the predicted completion.

POST `/tokenize`: Tokenize a given text

POST `/detokenize`: Convert tokens to text

POST `/apply-template`: Apply chat template to a conversation

POST `/embedding`: Generate embedding of a given text

POST `/reranking`: Rerank documents according to a given query

POST `/infill`: For code infilling.

GET `/props`: Get server global properties.

POST `/props`: Change server global properties.

POST `/embeddings`: non-OpenAI-compatible embeddings API

GET `/slots`: Returns the current slots processing state

GET `/metrics`: Prometheus compatible metrics exporter

POST `/slots/{idslot}?action=save`: Save the prompt cache of the specified slot to a file.

POST `/slots/{idslot}?action=restore`: Restore the prompt cache of the specified slot from a file.

POST `/slots/{idslot}?action=erase`: Erase the prompt cache of the specified slot.

GET `/lora-adapters`: Get list of all LoRA adapters

POST `/lora-adapters`: Set list of LoRA adapters

OpenAI-compatible API Endpoints

GET `/v1/models`: OpenAI-compatible Model Info API

POST `/v1/completions`: OpenAI-compatible Completions API

POST `/v1/chat/completions`: OpenAI-compatible Chat Completions API

POST `/v1/responses`: OpenAI-compatible Responses API

POST `/v1/embeddings`: OpenAI-compatible embeddings API

POST `/v1/messages`: Anthropic-compatible Messages API

POST `/v1/messages/counttokens`: Token Counting

Server built-in tools

Using multiple models

Model sources

Model presets

Routing requests

GET `/models`: List available models

POST `/models/load`: Load a model

POST `/models/unload`: Unload a model

API errors

Sleeping on Idle

More examples

Interactive mode

Custom default Web UI preferences

Legacy completion web UI

Extending or building alternative Web Front End