Working in Python, I often run data processing, transfer, and model training scripts. Now, with any reasonable degree of complexity and/or big data, this can take some time.

Although often enough we all have some other work to be done whilst waiting for these processing to complete, occasionally, we don’t.

For this purpose, I put together a set of Python scripts built for this exact problem. I use these scripts to send process updates, visualizations and completion notifications to my phone.

So, when we do occasionally have those moments of freedom. We can enjoy them without being worried about model progress.

What we need

Okay so the first thing we need to ask is — what do we need to know?

Now of-course, this really depends on the work you are doing. For me I have three main processing tasks that have the potential to take up time:

• Model training
• Data **processing **and/or transfer
• Financial modelling

For each of these, there are of-course different pieces of information that we need to stay informed about. Let’s take a look at an example of each.

Model Training

Update every n epochs, must include key metrics. For example, loss and accuracy for training and validation sets.

Notification of completion (of-course). For this I like to include:

• prediction outputs, for text generation, the generated text (or a sample of it) — for image generation, a (hopefully) cool visualization.
• visualization of key metrics during training (again, loss and accuracy for both training and validation sets)
• other, less essential but still useful information such as local model directories, training time, model architecture etc

Let’s take the example of training a neural network to reproduce a given artistic style.

For this, we want to see; generated images from the model, loss and accuracy plots, and current training time, and a model name.

import notify

START = datetime.now()  # this line would be placed before model training begins
MODELNAME = "Synthwave GAN"  # giving us our model name
NOTIFY = 100  # so we send an update notification every 100 epochs

# for each epoch e, we would include the following code
if e % notify_epoch == 0 and e != 0:
    # here we create the email body message
    txt = (f"{MODELNAME} update as of "
           f"{datetime.now().strftime('%H:%M:%S')}.")

    # we build the MIME message object with notify.message
    msg = notify.message(
        subject='Synthwave GAN',
        text=txt,
        img=[
            f'../visuals/{MODELNAME}/epoch_{e}_loss.png',
            f'../visuals/{MODELNAME}/epoch_{e}_iter_{i}.png'
        ]
    )  # note that we attach two images here, the loss plot and
    #    ...a generated image output from our model
           
    notify.send(msg)  # we then send the message

In this scenario, every 100 epochs, an email containing all of the above will be sent.

#python #programming