Install the NumExpr library

First, we need to make sure we have the library numexpr. So, as expected,

pip install numexpr

The project is hosted here on Github. It is from the PyData stable, the organization under NumFocus, which also gave rise to Numpy and Pandas.

As per the source, “NumExpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays, are accelerated and use less memory than doing the same calculation in Python. In addition, its multi-threaded capabilities can make use of all your cores — which generally results in substantial performance scaling compared to NumPy.” (source)

Here is the detailed documentation for the library and examples of various use cases.

A simple vector-scalar operation

We start with the simple mathematical operation — adding a scalar number, say 1, to a Numpy array. For using the NumExpr package, all we have to do is to wrap the same calculation under a special method evaluate in a symbolic expression. The following code will illustrate the usage clearly,

Wow! That was magical! All we had to do was to write the familiar a+1 Numpy code in the form of a symbolic expression "a+1" and pass it on to the ne.evaluate() function. And we got a significant speed boost — from 3.55 ms to 1.94 ms on average.

Note that we ran the same computation 200 times in a 10-loop test to calculate the execution time. Now, of course, the exact results are somewhat dependent on the underlying hardware. You are welcome to evaluate this on your machine and see what improvement you got.

#overviews #numpy #pandas #python