How to beat Python’s pip: Software stack resolution pipelines

How to beat Python’s pip: Software stack resolution pipelines

Creating an API server using Flask, making data analysis in Jupyter notebooks or creating a neural network using TensorFlow, these all can be easily written in a few lines of code. Any performance-critical parts can be optimized thanks to CPython's C API. Python is a very effective weapon in anyone's inventory.

Following our previous article about reinforcement learning-based dependency resolution, we will take a look at actions taken during the resolution process. An example can be resolving intel-tensorflow instead of tensorflow following programmed rules based on the dependency graph.

Dependency graphs and software stack resolution

Users and maintainers have limited control with additional semantics when it comes to dependencies installed and resolved in Python applications. Tools, such as pipPipenv, or Poetry resolve a dependency stack to the latest possible candidate following dependency specification stated by application developers (direct dependencies) and by library maintainers (transitive dependencies). This can become a limitation, especially considering applications that were written months or years ago and require non-zero attention in maintenance. A trigger to revisit the dependency stack can be a security vulnerability found in the software stack or an observation the given software stack is no longer suitable for the given task and should be upgraded or downgraded (e.g. performance improvements in releases).

Even the resolution of the latest software can lead to issues that library maintainers haven’t spotted or haven’t considered. We already know that the state space of all the possible software stacks in the Python ecosystem is in many cases too large to explore and evaluate. Moreover, dependencies in the application stack evolve over time and underpinning or overpinning dependencies happen quite often.

Another aspect to consider is the human being itself. The complexity behind libraries and application stacks becomes a field of expertise on their own. What’s the best performing TensorFlow stack for the given application running on specific hardware? Should I use a Red Hat build, an Intel build, or a Google TensorFlow build? All of the companies stated have dedicated teams that focus on the performance of the builds produced and there is required certain manpower to quantify these questions. The performance aspect described is just another item in the vector coming to the application stack quality evaluation.

resolver reinforcement-learning python machine-learning

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Hire Machine Learning Developers in India

We supply you with world class machine learning experts / ML Developers with years of domain experience who can add more value to your business.

How To Plot A Decision Boundary For Machine Learning Algorithms in Python

How To Plot A Decision Boundary For Machine Learning Algorithms in Python, you will discover how to plot a decision surface for a classification machine learning algorithm.

How To Get Started With Machine Learning With The Right Mindset

You got intrigued by the machine learning world and wanted to get started as soon as possible, read all the articles, watched all the videos, but still isn’t sure about where to start, welcome to the club.

What is Supervised Machine Learning

What is neuron analysis of a machine? Learn machine learning by designing Robotics algorithm. Click here for best machine learning course models with AI

Python For Machine Learning | Machine Learning With Python

Python For Machine Learning | Machine Learning With Python, you will be working on an end-to-end case study to understand different stages in the Machine Learning (ML) life cycle. This will deal with 'data manipulation' with pandas and 'data visualization' with seaborn. After this an ML model will be built on the dataset to get predictions. You will learn about the basics of scikit-learn library to implement the machine learning algorithm.