Following our previous article about reinforcement learning-based dependency resolution, we will take a look at actions taken during the resolution process. An example can be resolving intel-tensorflow instead of tensorflow following programmed rules based on the dependency graph.

Dependency graphs and software stack resolution

Users and maintainers have limited control with additional semantics when it comes to dependencies installed and resolved in Python applications. Tools, such as pip, Pipenv, or Poetry resolve a dependency stack to the latest possible candidate following dependency specification stated by application developers (direct dependencies) and by library maintainers (transitive dependencies). This can become a limitation, especially considering applications that were written months or years ago and require non-zero attention in maintenance. A trigger to revisit the dependency stack can be a security vulnerability found in the software stack or an observation the given software stack is no longer suitable for the given task and should be upgraded or downgraded (e.g. performance improvements in releases).

Even the resolution of the latest software can lead to issues that library maintainers haven’t spotted or haven’t considered. We already know that the state space of all the possible software stacks in the Python ecosystem is in many cases too large to explore and evaluate. Moreover, dependencies in the application stack evolve over time and underpinning or overpinning dependencies happen quite often.

Another aspect to consider is the human being itself. The complexity behind libraries and application stacks becomes a field of expertise on their own. What’s the best performing TensorFlow stack for the given application running on specific hardware? Should I use a Red Hat build, an Intel build, or a Google TensorFlow build? All of the companies stated have dedicated teams that focus on the performance of the builds produced and there is required certain manpower to quantify these questions. The performance aspect described is just another item in the vector coming to the application stack quality evaluation.

#resolver #reinforcement-learning #python #machine-learning