And What a Data Scientist Can Learn to Face Changing Technology

What threatens a data scientist’s job?

As computers get faster and data science tools get better, less of a data scientists job will be focused on optimizing a traditional ML models (non neural network models). Many companies are pursuing AutoML frameworks that can perform a lot of feature engineering and model optimization for all sorts of problems. All the major cloud providers (and a bunch of start ups) offer out-of-the-box transfer learning models for computer vision, and many offer AutoML services for both tabular and NLP models. These are services where you upload your data and the best model gets spit out after you click train. Ironically enough, data scientists who are supposed to create ML/AI are finding these new tools are automating portions of their own jobs.

What is a data scientist good at besides ML?

As a data scientist myself, I’ve spent a lot of time thinking about what I can work on if my jobs is one day heavily automated. My conclusion is that while there are many non ML data science tasks, my core technical abilities — writing ML models, explaining them, deploying them, monitoring and updating them — may need diversification. So besides those skills what am I and most other data scientists good at? Python and functional programming. If you don’t know, functional programming puts an emphasis on writing functions as opposed to writing object oriented code. Object oriented programming (OOP) took off because people tend to think in terms of objects and their relationships to each other. Because a lot of OO code has no defined order of execution, focusing on the behavior of objects is paramount. Lots of time is spent to designing software and people have made whole careers of describing design patterns. Functional programming on the other hand aims to focus on functions, where we can more clearly see the output as a result of an input. I think most data scientists recognize that when they write code, they do it outside of a class definition and generally aim to have a consistent set of verifiable operations happen to the input. Functional programs should aim to bring the same level of precision that mathematical functions have. f(x) = 4x + 3 is just as valid as a piece of code as it is a mathematical function. This function is simple to verify, while OO code can end up in states that are hard to recreate and debug.

#data-science #careers #serverless