🚀 Colossal-Inference

Table of Contents

Introduction

Design

Architecture of inference:

Roadmap of our implementation

Get started

Installation

Requirements

Docker

Usage

Quick start

Example

Performance

environment:

Single GPU Performance:

Tensor Parallelism Inference

Llama

Bloom

Pipline Parallelism Inference

A10 7b, fp16

A10 13b, fp16

A800 7b, fp16

Quantization LLama