Examples

Table of Contents

Install requirements

Get Start with ColossalRun

Training Configuration

Parameter Efficient Finetuning (PEFT)

Lora Parameters

RLHF Training Stage1 - Supervised Instructs Tuning

Step 1: Data Collection

Step 2: Preprocessing

Step 3: Training

RLHF Training Stage2 - Training Reward Model

Step 1: Data Collection

Step 2: Preprocessing

Step 3: Training

Features and Tricks in RM Training

Note on Reward Model Training

RLHF Training Stage3 - Proximal Policy Optimization

Step 1: Data Collection

Step 2: Preprocessing

Step 3: Training

Sample Training Results Using Default Script

Reward

Note on PPO Training

Q1: My reward is negative

Q2: My actor loss is negative

Q3: My reward doesn't go up (decreases)

Q4: Generation is garbage

GRPO Training and DeepSeek R1 reproduction

GRPO Model Selection

Reinforcement Learning with Verifiable Reward

Step 1: Data Collection & Preparation

Step 2: Training

GRPO Result

Reward and Response Length

Response Length Distribution (After Training) and Sample response

Alternative Option For RLHF: Direct Preference Optimization

DPO Training Stage1 - Supervised Instructs Tuning

DPO Training Stage2 - DPO Training

Step 1: Data Collection & Preparation

Step 2: Training

DPO Result

Alternative Option For RLHF: Simple Preference Optimization

SimPO Result

Alternative Option For RLHF: Odds Ratio Preference Optimization

ORPO Result

Alternative Option For RLHF: Kahneman-Tversky Optimization (KTO)

KTO Result

SFT for DeepSeek V3

Dataset preparation

Model weights preparation

Usage

Hardware Requirements

Inference example

Attention