llama.cpp/tools/completion

Table of Contents

Quick Start

Unix-based systems (Linux, macOS, etc.):

Input prompt (One-and-done)

Conversation mode (Allow for continuous interaction with the model)

Conversation mode using built-in jinja chat template

One-and-done query using jinja with custom system prompt and a starting prompt

Infinite text from a starting prompt (you can use `Ctrl-C` to stop it):

Windows:

Input prompt (One-and-done)

Conversation mode (Allow for continuous interaction with the model)

Conversation mode using built-in jinja chat template

One-and-done query using jinja with custom system prompt and a starting prompt

Infinite text from a starting prompt (you can use `Ctrl-C` to stop it):

Usage

Common params

Sampling params

Completion-specific params

Common Options

Input Prompts

Interaction

Interaction Options

Reverse Prompts

In-Prefix

In-Suffix

Chat templates

Context Management

Context Size

Extended Context Size

Keep Prompt

Generation Flags

Number of Tokens to Predict

Temperature

Repeat Penalty

DRY Repetition Penalty

Top-K Sampling

Top-P Sampling

Min-P Sampling

Adaptive-P Sampling

Locally Typical Sampling

Mirostat Sampling

XTC Sampling

Top-nσ Sampling

Logit Bias

RNG Seed

Performance Tuning and Memory Options

Number of Threads

Mlock

No Memory Mapping

NUMA support

Batch Size

Prompt Caching

Grammars & JSON schemas

Quantization

LoRA (Low-Rank Adaptation) adapters

Additional Options