Editor’s note: The Towards Data Science podcast’s “Climbing the Data Science Ladder” series is hosted by Jeremie Harris. Jeremie helps run a data science mentorship startup called SharpestMinds. You can listen to the podcast below:

There’s been a lot of talk about the future direction of data science, and for good reason. The space is finally coming into its own, and as the Wild West phase of the mid-2010s well and truly comes to an end, there’s keen interest among data professionals to stay ahead of the curve, and understand what their jobs are likely to look like 2, 5 and 10 years down the road.

And amid all the noise, one trend is clearly emerging, and has already materialized to a significant degree: as more and more of the data science lifecycle is automated or abstracted away, data professionals can afford to spend more time adding value to companies in more strategic ways. One way to do this is to invest your time deepening your subject matter expertise, and mastering the business side of the equation. Another is to double down on technical skills, and focus on owning more and more of the data stack —particularly including productionization and deployment stages.

My guest for today’s episode of the Towards Data Science podcast has been down both of these paths, first as a business-focused data scientist at Spotify, where he spent his time defining business metrics and evaluating products, and second as a data engineer at Better.com, where his focus has shifted towards productionization and engineering. During our chat, Kenny shared his insights about the relative merits of each approach, and the future of the field.

Here were some of my favourite take-homes from our conversation:

• Even professional data scientists have a strong bias towards complexity. After all, most data scientists get into the space because they’re excited about the power of machine learning algorithms, and lean towards any opportunity to apply these systems, even when they’re not called for. For that reason, organizations need people whose jobs it becomes to ensure that data science solutions aren’t over-engineered, and that the actual needs of real users and customers are prioritized over crazy stacked models and neural networks. Those people are called product managers.
• When most people hear that “simple” models work best, they assume that means “logistic regression is what you should try first”. But in reality, even rule-based systems can often be good enough — and they have the added benefit of forcing the people who build them to understand the problem they’re solving deeply. For that reason, one skill that employers often look for is your ability to decide when machine learning is unnecessary.
• More recently, Kenny transitioned from his more business-centric and people-centric roles at Spotify to a full-on data engineering role. He did this in part to become a better leader: even though he tends to prefer people problems over technical ones, experience has taught him that the best ideas come from being in the technical weeds.
• Kenny hammers home the importance of building full-stack products even if you’re new and just trying to break into the space. We’ve talked about the importance of full-stack data science in previous episodes (and I’ve written a fair bit about that topic as well) but it’s worth re-emphasizing: even if you have no work experience, you can get your hands dirty deploying models to AWS or Google Cloud and develop an understanding of containerization using Docker, or version control using git. These are rapidly emerging as the new core survival skills for the data science job market.
• Better.com is hiring! You can check out their data science job posting here.

#machine-learning #data-engineering #data-science #editors-pick #tds-podcast #data analysis