NOTE :- This article is meant to guide you with the thought process(Python Modules) and right kind of questions that you need to answer for creating an Effective Pipeline. Here is an more Advanced article with example technologies for creating a production level Pipeline, But I would suggest you to skim through this before going there.

To Understand about ML Pipeline in an Interactive way check out The****Explorer**.**

🌀There was a time when I thought ML pipelines are these complex design architectures that can only be done by Software Engineers with high coding and system design skills, But that’s not True at all with the help recent technologies such as Amazon SageMaker, Apache Kafka, Microsoft Azure Etc. Pipelines can be automated and maintained in a easy and cost effective way.🌀

In fact Pipelines are a way to organize and automate your code so that not only your future self but others can be able to understand and integrate with it easily. Some of the objectives of a ML Pipeline are

⦿_ Resistant to New Errors._

_⦿ Real-Time _Processing.

_⦿ _Handling efficient_ Computation _and_ WorkLoad _Balance.

_⦿ Scalable _and message driven.

---

Before you start diving in, building models and predicting the future ask yourself these questions.

① ML Problem Framing:

What is the Business Problem that you want to solve?

Do you really need a Machine Learning Approach to solve this Problem?

Here are some tips to know if you really need a machine learning approach.

• Are there any repeating patterns that you want to understand.
• Do you have required data to understand the pattrens.
• Understand the type of prediction that you want to make whether it is a Binary/Multi Classification problem or you need to predict a Continuous value(Regression) such as stock price.

Wait………………………………! you are not ready yet.

Now it’s time to ask Domain expects and test your assumptions (which can again be yourself if you are doing your own project)

The more questions you ask at this stage the better your model is going to be

• what are the important features that effect your predictions?
• Are they any feature overlaps?
• How to test your model (This is not as easy as spliting data)?
• Questions that can help you understand the domain and problem that you are solving.

Let’s get on with the boring part and dive in into actually predicting the Future.

② Data Collection & Integration:

No matter from where you or your team collects the data there might be some noise in it. So you should be equiped with tools which can help you clean and integrate the data properly and you should be able to handle all kinds of data types thrown at you.

Although it is not intresting to do but it is useful to produce intresting results. Becoz real data doesn’t consists of numerical and categorical features it consists of garbage and others Indolence.

#data-science #ml-pipeline #amazon-sagemaker #artificial-intelligence