/
githubmirror
/
Open-Assistant
Обзор
Документация
Войти
/
githubmirror
/
Open-Assistant
Код
Запросы
0
Пакеты
0
Релизы
0
Аналитика
Безопасность
main
model/model_training/configs/deepspeed_rl_zero3.json
33 строки
766 B
sanagno
Rl training (#2206)
24 апр 2023, 10:49
Не верифицирован
24 апр 2023, 10:49
4f02095
Код
Авторство
О чём код?
{ "fp16": { "enabled": "true", "loss_scale": 0, "loss_scale_window": 1000, "initial_scale_power": 16, "hysteresis": 2, "min_loss_scale": 1 }, "zero_optimization": { "stage": 3, "allgather_partitions": true, "allgather_bucket_size": 1e9, "overlap_comm": true, "reduce_scatter": true, "reduce_bucket_size": 1e9, "contiguous_gradients": true, "offload_optimizer": { "device": "cpu", "pin_memory": true }, "offload_param": { "device": "cpu", "pin_memory": true } }, "gradient_accumulation_steps": "8", "train_micro_batch_size_per_gpu": "auto", "train_batch_size": "auto", "gradient_clipping": "auto", "steps_per_print": 2000, "wall_clock_breakdown": false }