One-click-installer

The script uses Miniconda to set up a Conda environment in the

If you ever need to install something manually in the

• There is no need to run any of those scripts (
  start_
  ,
  update_wizard_
  , or
  cmd_
  ) as admin/root.
• To install the requirements for extensions, you can use the
  extensions_reqs
  script for your OS. At the end, this script will install the main requirements for the project to make sure that they take precedence in case of version conflicts.
• For additional instructions about AMD and WSL setup, consult the documentation.
• For automated installation, you can use the
  GPU_CHOICE
  ,
  USE_CUDA118
  ,
  LAUNCH_AFTER_INSTALL
  , and
  INSTALL_EXTENSIONS
  environment variables. For instance:
  GPU_CHOICE=A USE_CUDA118=FALSE LAUNCH_AFTER_INSTALL=FALSE INSTALL_EXTENSIONS=TRUE ./start_linux.sh
  .

Manual installation using Conda

Recommended if you have some experience with the command-line.

0. Install Conda

https://docs.conda.io/en/latest/miniconda.html

On Linux or WSL, it can be automatically installed with these two commands (source):

curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh"
bash Miniconda3.sh

1. Create a new conda environment

conda create -n textgen python=3.11
conda activate textgen

2. Install Pytorch

The up-to-date commands can be found here: https://pytorch.org/get-started/locally/.

For NVIDIA, you also need to install the CUDA runtime libraries:

conda install -y -c "nvidia/label/cuda-12.1.1" cuda-runtime

If you need

conda install -y -c "nvidia/label/cuda-12.1.1" cuda

3. Install the web UI

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
pip install -r <requirements file according to table below>

Requirements file to use:

Start the web UI

conda activate textgen
cd text-generation-webui
python server.py

Then browse to

AMD GPU on Windows

1. Use

   requirements_cpu_only.txt
   or
   requirements_cpu_only_noavx2.txt
   in the command above.

2. Manually install llama-cpp-python using the appropriate command for your hardware: Installation from PyPI.

   • Use the
     LLAMA_HIPBLAS=on
     toggle.
   • Note the Windows remarks.

3. Manually install AutoGPTQ: Installation.

   • Perform the from-source installation - there are no prebuilt ROCm packages for Windows.

Older NVIDIA GPUs

1. For Kepler GPUs and older, you will need to install CUDA 11.8 instead of 12:

pip3 install torch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 --index-url https://download.pytorch.org/whl/cu118
conda install -y -c "nvidia/label/cuda-11.8.0" cuda-runtime

2. bitsandbytes >= 0.39 may not work. In that case, to use
   --load-in-8bit
   , you may have to downgrade like this:
   • Linux:
     pip install bitsandbytes==0.38.1
   • Windows:
     pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl

Manual install

The

Alternative: Docker

For NVIDIA GPU:
ln -s docker/{nvidia/Dockerfile,nvidia/docker-compose.yml,.dockerignore} .
For AMD GPU: 
ln -s docker/{amd/Dockerfile,intel/docker-compose.yml,.dockerignore} .
For Intel GPU:
ln -s docker/{intel/Dockerfile,amd/docker-compose.yml,.dockerignore} .
For CPU only
ln -s docker/{cpu/Dockerfile,cpu/docker-compose.yml,.dockerignore} .
cp docker/.env.example .env
#Create logs/cache dir : 
mkdir -p logs cache
# Edit .env and set: 
#   TORCH_CUDA_ARCH_LIST based on your GPU model
#   APP_RUNTIME_GID      your host user's group id (run `id -g` in a terminal)
#   BUILD_EXTENIONS      optionally add comma separated list of extensions to build
# Edit CMD_FLAGS.txt and add in it the options you want to execute (like --listen --cpu)
# 
docker compose up --build

• You need to have Docker Compose v2.17 or higher installed. See this guide for instructions.
• For additional docker files, check out this repository.

Updating the requirements

From time to time, the

conda activate textgen
cd text-generation-webui
pip install -r <requirements file that you have used> --upgrade

Basic settings

Model loader

Accelerate/transformers

bitsandbytes 4-bit

⚠️ Requires minimum compute of 7.0 on Windows at the moment.

llama.cpp

ExLlamav2

AutoGPTQ

GPTQ-for-LLaMa

ctransformers

HQQ

DeepSpeed

RoPE (for llama.cpp, ExLlamaV2, and transformers)

Gradio

API

Multimodal