Recently, I came across a use-case which involved running a PyTorch model for inference in real-time, along with a Node.js server which handles I/O. After an initial failed attempt of using torch.js, we decided to go with Tensorflow.js (tfjs). However, the layers model is a synchronous and a blocking process which forces the use of worker threads (this route is detailed in a separate story). This approach has its own difficulties: (1) a tfjs specific worker thread needs to be running separately. (2) the use of GPU via WebGL backend is not straightforward.
PyTorch is convenient for AI development, with the ease of using Python. Also, PyTorch provides a nice C++ frontend which is especially attractive for low-latency systems. For the above use-case, we decided to continue using PyTorch. While Node.js handles input and output, Python/PyTorch will handle the AI model inference. The two systems will communicate using named pipes.
Named Pipe
‘Pipe’ is a fundamental feature of Linux that allows separate processes to communicate and pass data to each other. A named pipe (uses the file system) is a file that can be accessed by two unrelated processes; one is a reader and and the other, writer. More details can be found here. Named pipes are sometimes referred to as FIFOs: first-in first-out, since the byte order (going in and coming out) is preserved.
Simplifying the problem statement…
The goal is to use Named Pipes for communication.
- A Node.js process will write some data to a pipe (A)
- A Python process reads from pipe (A) and manipulates the data
- Above python process then writes the data into a pipe (B)
- Node.js reads from pipe (B)
#nodejs #python
