How do you get the most precise machine learning model? Through experiments, of course! Whether you’re testing which algorithm to use, changing variable values, or choosing features to include, ML experiments help you decide.

But, there’s a downside. They produce massive amounts of artifacts. The output could be a trained model, a model checkpoint, or a file created during the training process. Data scientists need a standardized way to manage these artifacts – otherwise it can become hectic very quickly. Here is just a _basic _list of all the variables and artifacts probably flowing through:

• Parameters: hyperparameters, model architectures, training algorithms
• Jobs: pre-processing job, training job, post-processing job — these consume other infrastructure resources such as compute, networking and storage
• Artifacts: training scripts, dependencies, datasets, checkpoints, trained models
• Metrics: training and evaluation accuracy, loss
• Debug data: weights, biases, gradients, losses, optimizer state
• Metadata: experiment, trial and job names, job parameters (CPU, GPU and instance type), artifact locations (e.g. S3 bucket)

If data scientists don’t store all this experimental metadata, they will not be able to achieve reproducibility or compare ML experiment results.

#ml experiment tracking #ml model management #ml tools #mlops