Table of Contents

News

Colossal-LLaMA-2-7B

Colossal-LLaMA-2-13B

Performance Evaluation

Model with ~7 Billion Parameters

Model with ~13 Billion Parameters

Examples

Training Logs

Colossal-LLaMA-2-7b-base

Colossal-LLaMA-2-13b-base

Inference

Import from HuggingFace

Import from Modelscope

Quick Start

Usage

Install

0. Pre-requisite

1. Install required packages

2. Install `xentropy`, `layernorm` and `rotary`

How to run

1. Init Tokenizer Preparation

2. Init Model Preparation

3. Data Preparation

3.1 Data for Pretraining

3.2 Data for Supervised Fine-tuning

4. Command Line Arguments for Training

4.1 Arguments for Pretraining

4.2 Arguments for Supervised Fine-tuning

5. Running Command

5.1 Command for Pretraining

5.2 Command for Supervised Fine-tuning

Technical Insights

Data

Tokenizer

Training Strategy

Multi-stage Training

Bucket-based Training

Bridging Any Domain-specific Large Models

Citations