Learning To Create Python Multi-Threaded And Multi-Process

Learning To Create Python Multi-Threaded And Multi-Process - This article is not useful for experienced Python developers. It is rather a superficial overview of Python multi-threaded features for those who recently started learning Python.

Unfortunately, you cannot find tons of material about multithreading in Python. Moreover, quite often I meet Python beginners who don’t know about GIL, for example. In this article, I want to cover the most basic features of a Python multi-threaded, discuss what does GIL stand for and how to act with/without it.

Python is a perfect programming language. It ideally includes many programming paradigms. Majority of the tasks that can be faced by a developer are solved easily, elegantly and concisely using Python. However, a single-threaded solution is quite often enough for all those tasks. Single-threaded programs are usually predictable and easy-to-debug. The same can’t be said for the Python multi-threaded and multi-process applications.

Python Multi-Threaded Application

Python has a threading module that includes everything for the multi-threaded programming: there you can find different types of locks, a semaphore, and an event mechanism. Thus, you get everything you need for the vast majority of Python multi-threaded applications. Moreover, it is extremely easy to use all those tools. To make sure, let’s discuss an example of an application that runs two threads. The first thread types ten “0”, the second – ten “1”, and strictly in turn.

import threading

def writer(x, event_for_wait, event_for_set):

    for i in xrange(10):

        event_for_wait.wait() # wait for event

        event_for_wait.clear() # clean event for future

        print x

        event_for_set.set() # set event for neighbor thread

# init events

e1 = threading.Event()

e2 = threading.Event()

# init threads

t1 = threading.Thread(target=writer, args=(0, e1, e2))

t2 = threading.Thread(target=writer, args=(1, e2, e1))

# start threads

t1.start()

t2.start()

e1.set() # initiate the first event

# join threads to the main thread

t1.join()

t2.join()

No magic and voodoo code. As you can see, the code is accurate and consistent. As you can see, we have created the thread out of a function which is highly convenient for small tasks. Moreover, this code is rather flexible. For example, you have created one more thread that types “2”. Thus, you will get the following:

import threading

def writer(x, event_for_wait, event_for_set):

    for i in xrange(10):

        event_for_wait.wait() # wait for event

        event_for_wait.clear() # clean event for future

        print x

        event_for_set.set() # set event for neighbor thread

# init events

e1 = threading.Event()

e2 = threading.Event()

e3 = threading.Event()

# init threads

t1 = threading.Thread(target=writer, args=(0, e1, e2))

t2 = threading.Thread(target=writer, args=(1, e2, e3))

t3 = threading.Thread(target=writer, args=(2, e3, e1))

# start threads

t1.start()

t2.start()

t3.start()

e1.set() # initiate the first event

# join threads to the main thread

t1.join()

t2.join()

t3.join()

Here, we have added a new event, a new thread, and changed the parameters passed to the threads for the start (it is also possible to create a more general solution with the help of MapReduce, for example, but is out of the article). Thus, as you can see, there are no complicated things and magic. Everything is simple and comprehensive. Let’s move forward.

Global Interpreter Lock

There are two the most widespread reasons for using threads. Firstly, it is useful for increasing the using of modern multi-core processor architecture. It means increasing application performance. Secondly, threads are of utmost importance in case we need to divide the application logic into parallel and fully or partly asynchronous sections (i.e. you need to have an opportunity to ping several servers simultaneously).

Considering the first situation, we face the following Python limitation called Global Interpreter Lock (GIL). The GIL concept means that at any specific time only one thread may be executed by a processor. It was designed to avoid the threads being competing for different variables. An executing thread gains access through the whole environment. This feature of Python thread implementation significantly simplifies the work with threads. Moreover, this feature provides you with certain thread safety.

However, you should pay attention to the following moment: it may seem that a Python multi-threaded application will work exactly the same time as a single-threaded, doing the same thing. However, here you will face the following unpleasant issue. Let’s consider the following code to understand what I mean:

with open('test1.txt', 'w') as fout:

    for i in xrange(1000000):

        print >> fout, 1

# This application just creates a million of strings ‘1’ for ~0.35s on my local machine.

# Now, let’s consider another program for comparison:

from threading import Thread

def writer(filename, n):

    with open(filename, 'w') as fout:

        for i in xrange(n):

            print >> fout, 1

t1 = Thread(target=writer, args=('test2.txt', 500000,))

t2 = Thread(target=writer, args=('test3.txt', 500000,))

t1.start()

t2.start()

t1.join()

t2.join()

The second application creates 2 threads. In each thread, the application creates a separate file for half a million lines “1”. In fact, the amount of work is the same. However, over time, you will see an interesting effect. The application performance is from 0.7 seconds to 7 seconds. What is the reason for this situation?

In fact, it happens due to the fact that when a thread does not need a CPU resource, it frees GIL. At this moment, both threads can try to get it. At the same time, the operating system knows that there are many cores. Thus, it can intensify this situation by trying to distribute the threads between the cores.

UPD: currently, Python 3.2 has an improved implementation of GIL, and this problem is partially solved. The solution is about the fact that each thread after losing control waits for a short period before it can capture GIL again.

“Thus, it’s extremely difficult to create an effective multi-threaded application in Python?” you may ask. However, keep calm, there is always a solution.

Python Multi-Process Applications

In order to solve the problem mentioned in the previous chapter, Python provides us with the subprocess module. We can create an application that should be executed in a parallel thread and execute it in several threads in other application. This solution would have significantly increased our application performance. The matter is that the threads created in GIL only wait for the launched process shutdown. However, this approach has many problems as well. The main issue is that it becomes difficult to transfer the data between the processes. This way, we would have to serialize objects, adjust the connection through PIPE or other tools. In fact, it results in additional expenses and the code becomes hard to understand.

For this reason, here comes another useful approach. Python also has multiprocessing module which is quite similar to threading. For example, the processes can be created in the same way using the functions. The methods of operation with the processes are almost the same as for threading. However, in order to synchronize the processes and provide data sharing – we need to use other tools. I am referring to Queues and Pipes. Alongside, the analogues to locks, events and semaphores mentioned in threading are also available here.

Additionally, the multiprocessing module also provides an operating principle of general memory. Thus, the module provides the class of Value and Array variables that can be shared between the processes. For the convenience of working with the variables, you can use Manager classes. They are more flexible and convenient, but slower.

Furthermore, the multiprocessing module provides an opportunity to create pools of processes. This mechanism is pretty convenient for implementing the Master-Worker template to build a parallel Map.

Among the basic problems of working with multiprocessing, I need to point out a module relative platform dependence. Due to the fact that different operating systems provide a different working process with the processes, the code receives several limitations. For example, Windows OS doesn’t have fork mechanism. Therefore, you need to wrap the processes separation point in the following:

if __name__ =='__main__':

Thus, this code construction is a good code style.

What’s More …

In order to create parallel applications using Python, there are also other libraries and approaches. For example, you can use Haddop+Python or different implementations of MPI and Python (pyMPI, mpi4py). Moreover, it’s even possible to use the wrappers of the existing libraries in C++ and Fortran. Here, we can mention such frameworks/libraries as Pyro, Twisted, Tornado and so on. However, all these things are the topics of other articles.

#python #web-development

What is GEEK

Buddha Community

Learning To Create Python Multi-Threaded And Multi-Process

Pyringe: Debugger Capable Of Attaching to & Injecting Code Into Python

DISCLAIMER: This is not an official google project, this is just something I wrote while at Google.

Pyringe

What this is

Pyringe is a python debugger capable of attaching to running processes, inspecting their state and even of injecting python code into them while they're running. With pyringe, you can list threads, get tracebacks, inspect locals/globals/builtins of running functions, all without having to prepare your program for it.

What this is not

A "Google project". It's my internship project that got open-sourced. Sorry for the confusion.

What do I need?

Pyringe internally uses gdb to do a lot of its heavy lifting, so you will need a fairly recent build of gdb (version 7.4 onwards, and only if gdb was configured with --with-python). You will also need the symbols for whatever build of python you're running.
On Fedora, the package you're looking for is python-debuginfo, on Debian it's called python2.7-dbg (adjust according to version). Arch Linux users: see issue #5, Ubuntu users can only debug the python-dbg binary (see issue #19).
Having Colorama will get you output in boldface, but it's optional.

How do I get it?

Get it from the Github repo, PyPI, or via pip (pip install pyringe).

Is this Python3-friendly?

Short answer: No, sorry. Long answer:
There's three potentially different versions of python in play here:

  1. The version running pyringe
  2. The version being debugged
  3. The version of libpythonXX.so your build of gdb was linked against

2 Is currently the dealbreaker here. Cpython has changed a bit in the meantime[1], and making all features work while debugging python3 will have to take a back seat for now until the more glaring issues have been taken care of.
As for 1 and 3, the 2to3 tool may be able to handle it automatically. But then, as long as 2 hasn't been taken care of, this isn't really a use case in the first place.

[1] - For example, pendingbusy (which is used for injection) has been renamed to busy and been given a function-local scope, making it harder to interact with via gdb.

Will this work with PyPy?

Unfortunately, no. Since this makes use of some CPython internals and implementation details, only CPython is supported. If you don't know what PyPy or CPython are, you'll probably be fine.

Why not PDB?

PDB is great. Use it where applicable! But sometimes it isn't.
Like when python itself crashes, gets stuck in some C extension, or you want to inspect data without stopping a program. In such cases, PDB (and all other debuggers that run within the interpreter itself) are next to useless, and without pyringe you'd be left with having to debug using print statements. Pyringe is just quite convenient in these cases.

I injected a change to a local var into a function and it's not showing up!

This is a known limitation. Things like inject('var = 2') won't work, but inject('var[1] = 1337') should. This is because most of the time, python internally uses a fast path for looking up local variables that doesn't actually perform the dictionary lookup in locals(). In general, code you inject into processes with pyringe is very different from a normal python function call.

How do I use it?

You can start the debugger by executing python -m pyringe. Alternatively:

import pyringe
pyringe.interact()

If that reminds you of the code module, good; this is intentional.
After starting the debugger, you'll be greeted by what behaves almost like a regular python REPL.
Try the following:

==> pid:[None] #threads:[0] current thread:[None]
>>> help()
Available commands:
 attach: Attach to the process with the given pid.
 bt: Get a backtrace of the current position.
 [...]
==> pid:[None] #threads:[0] current thread:[None]
>>> attach(12679)
==> pid:[12679] #threads:[11] current thread:[140108099462912]
>>> threads()
[140108099462912, 140108107855616, 140108116248323, 140108124641024, 140108133033728, 140108224739072, 140108233131776, 140108141426432, 140108241524480, 140108249917184, 140108269324032]

The IDs you see here correspond to what threading.current_thread().ident would tell you.
All debugger functions are just regular python functions that have been exposed to the REPL, so you can do things like the following.

==> pid:[12679] #threads:[11] current thread:[140108099462912]
>>> for tid in threads():
...   if not tid % 10:
...     thread(tid)
...     bt()
... 
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 524, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "./test.py", line 46, in Idle
    Thread_2_Func(1)
  File "./test.py", line 40, in Wait
    time.sleep(n)
==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> 

You can access the inferior's locals and inspect them like so:

==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> inflocals()
{'a': <proxy of A object at remote 0x1d9b290>, 'LOL': 'success!', 'b': <proxy of B object at remote 0x1d988c0>, 'n': 1}
==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> p('a')
<proxy of A object at remote 0x1d9b290>
==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> p('a').attr
'Some_magic_string'
==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> 

And sure enough, the definition of a's class reads:

class Example(object):
  cl_attr = False
  def __init__(self):
    self.attr = 'Some_magic_string'

There's limits to how far this proxying of objects goes, and everything that isn't trivial data will show up as strings (like '<function at remote 0x1d957d0>').
You can inject python code into running programs. Of course, there are caveats but... see for yourself:

==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> inject('import threading')
==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> inject('print threading.current_thread().ident')
==> pid:[12679] #threads:[11] current thread:[140108241524480]
>>> 

The output of my program in this case reads:

140108241524480

If you need additional pointers, just try using python's help (pyhelp() in the debugger) on debugger commands.

Author: google
Source Code: https://github.com/google/pyringe
License: Apache-2.0 License

#python 

Ray  Patel

Ray Patel

1619518440

top 30 Python Tips and Tricks for Beginners

Welcome to my Blog , In this article, you are going to learn the top 10 python tips and tricks.

1) swap two numbers.

2) Reversing a string in Python.

3) Create a single string from all the elements in list.

4) Chaining Of Comparison Operators.

5) Print The File Path Of Imported Modules.

6) Return Multiple Values From Functions.

7) Find The Most Frequent Value In A List.

8) Check The Memory Usage Of An Object.

#python #python hacks tricks #python learning tips #python programming tricks #python tips #python tips and tricks #python tips and tricks advanced #python tips and tricks for beginners #python tips tricks and techniques #python tutorial #tips and tricks in python #tips to learn python #top 30 python tips and tricks for beginners

Ray  Patel

Ray Patel

1625843760

Python Packages in SQL Server – Get Started with SQL Server Machine Learning Services

Introduction

When installing Machine Learning Services in SQL Server by default few Python Packages are installed. In this article, we will have a look on how to get those installed python package information.

Python Packages

When we choose Python as Machine Learning Service during installation, the following packages are installed in SQL Server,

  • revoscalepy – This Microsoft Python package is used for remote compute contexts, streaming, parallel execution of rx functions for data import and transformation, modeling, visualization, and analysis.
  • microsoftml – This is another Microsoft Python package which adds machine learning algorithms in Python.
  • Anaconda 4.2 – Anaconda is an opensource Python package

#machine learning #sql server #executing python in sql server #machine learning using python #machine learning with sql server #ml in sql server using python #python in sql server ml #python packages #python packages for machine learning services #sql server machine learning services

Harry Patel

Harry Patel

1614145832

A Complete Process to Create an App in 2021

It’s 2021, everything is getting replaced by a technologically emerged ecosystem, and mobile apps are one of the best examples to convey this message.

Though bypassing times, the development structure of mobile app has also been changed, but if you still follow the same process to create a mobile app for your business, then you are losing a ton of opportunities by not giving top-notch mobile experience to your users, which your competitors are doing.

You are about to lose potential existing customers you have, so what’s the ideal solution to build a successful mobile app in 2021?

This article will discuss how to build a mobile app in 2021 to help out many small businesses, startups & entrepreneurs by simplifying the mobile app development process for their business.

The first thing is to EVALUATE your mobile app IDEA means how your mobile app will change your target audience’s life and why your mobile app only can be the solution to their problem.

Now you have proposed a solution to a specific audience group, now start to think about the mobile app functionalities, the features would be in it, and simple to understand user interface with impressive UI designs.

From designing to development, everything is covered at this point; now, focus on a prelaunch marketing plan to create hype for your mobile app’s targeted audience, which will help you score initial downloads.

Boom, you are about to cross a particular download to generate a specific revenue through your mobile app.

#create an app in 2021 #process to create an app in 2021 #a complete process to create an app in 2021 #complete process to create an app in 2021 #process to create an app #complete process to create an app

Ray  Patel

Ray Patel

1619510796

Lambda, Map, Filter functions in python

Welcome to my Blog, In this article, we will learn python lambda function, Map function, and filter function.

Lambda function in python: Lambda is a one line anonymous function and lambda takes any number of arguments but can only have one expression and python lambda syntax is

Syntax: x = lambda arguments : expression

Now i will show you some python lambda function examples:

#python #anonymous function python #filter function in python #lambda #lambda python 3 #map python #python filter #python filter lambda #python lambda #python lambda examples #python map