RLHF - Reinforcement Learning using Human Feedback

This directory contains notebooks that can be used to train a reward model, and then fine-tune the LLM using Reinforcement Learning. For a detailed overview of Reward Modeling and RLHF, refer to:

• RLHF Reward Model Training
• Measuring AI Alignment using Human Feedback for RAG Architecture

Architecture

Notebooks

1. rewardModelTraining.ipynb : This notebook takes in user preference data as input and trains a model of choice to output scaler reward.

2. RLHFImplementation.ipynb : This notebook takes a SFT LLM, a reward model and data as input. It then finetunes the LLM using PPO.

References

1. https://github.com/lvwerra/trl
2. https://argilla.io/blog/argilla-for-llms/