Typically, training a deep learning model starts with a forward pass where loss functions are evaluated followed by a backward pass where the loss-compensating gradients are generated, which are then pushed to servers and updated.
https://analyticsindiamag.com/dedloc-huggingface-language-model-training-distributed/

#ml #deep-learning