Table of Contents

News

Colossal-LLaMA-2-7B

Colossal-LLaMA-2-13B

Performance Evaluation

Model with ~7 Billion Parameters

Model with ~13 Billion Parameters

Examples

Training Logs

Colossal-LLaMA-2-7b-base

Colossal-LLaMA-2-13b-base

Inference

Import from HuggingFace

Import from Modelscope

Quick Start

Usage

Install

0. Pre-requisite

1. Install required packages

2. Install Apex

How to run

1. Init Tokenizer Preparation

2. Init Model Preparation

3. Data Preparation

3.1 Data for Pretraining

3.2 Data for Supervised Fine-tuning

4. Command Line Arguments for Training

4.1 Arguments for Pretraining

4.2 Arguments for Supervised Fine-tuning

5. Running Command

5.1 Command for Pretraining

5.2 Command for Supervised Fine-tuning

Technical Insights

Data

Tokenizer

Training Strategy

Multi-stage Training

Bucket-based Training

Bridging Any Domain-specific Large Models

Citations