Intro

My experience with Luigi started a few years ago, when the sole person in charge of the company pipeline left, and I got “gifted” with a massive legacy codebase.

At the time, I had no knowledge about pipelines in general, or Luigi in particular, so I started looking for tutorials and reading the official documentation in full.

I didn’t find it particularly useful; it is, of course, factually correct, but it is not a tutorial. Nor, at the time, were many courses or blogs about it.

This article, therefore, is my attempt to fill this gap. I’ve collected what learnt, and it will hopefully help you during your first steps with the Luigi package

Pros and Cons

Before we dive into the code, let’s see the strengths and weaknesses of this package.

The Luigi pipeline library has been designed and still used by Spotify. Over the years it has been adopted by other companies, notably like Deliveroo. But, exactly, because it was tailor-made for Spotify, it might not be suited to solve your need and I advise you to consider carefully if this system can work for you.

#python #luigi #data-science #pipeline #tutorial