Distributed Training using Gluon

Distributed Training Architecture:

Parameter Server:

Worker:

Scheduler:

Moving to distributed training:

Step 1: Use a distributed key-value store:

Step 2: Split the training data:

Step 3: Training with multiple GPUs

Final Step: Launching the distributed training