Eleo Nona

Eleo Nona

1588224420

How can I Launch Parallel Tasks in Python

By reading this piece, you will learn how to use the concurrent.futures library to run tasks asynchronously in Python. It is a better alternative to the threading and multiprocessing classes in Python due to the fact that it implemented both Thread and Process with the same interface, which is defined by the abstract Executor class. The official documentation reveals one major problem with Thread:

Besides, the threading class does not allow you to return a value from the callable functions except null. The main concept of the concurrent.futures module lies with the Executor class. It is an abstract class that provides methods to execute calls asynchronously. Instead of using it directly, we will be using the subclasses that inherit from it:

  • ThreadPoolExecutor
  • ProcessPoolExecutor

Let’s proceed to the next section and start writing some Python code.

1. ThreadPoolExecutor

Import

Add the following import declaration at the top of your Python file:

from concurrent.futures import ThreadPoolExecutor
import time

Callable function (target)

Let’s define a new function that serves as the callable function for the asynchronous call. I will just define a simple function that sleeps for two seconds and returns the multiplication of both input parameters as a result after that:

def wait_function(x, y):
    print('Task(', x,'multiply', y, ') started')
    time.sleep(2)
    print('Task(', x,'multiply', y, ') completed')
    return x * y

Single task

The next step is to create a ThreadPoolExecutor object. It is highly recommended to wrap it inside the with context manager, as it will call the shutdown function on its own and free up the resources once it finishes the execution. It accepts the following input parameters.

  • max_workers — The number of workers for this instance. For version 3.5 onward, it will default to the number of processors on the machine, multiplied by five. From version 3.8 onward, the default value is changed to min(32, os.cpu_count() + 4).
  • thread_name_prefix — Allows users to control the threading.Thread names for worker threads created by the pool for easier debugging.
  • initializer — An optional callable that is called at the start of each worker process.
  • initargs — A tuple of arguments passed to the initializer.

In this tutorial, I will be using just the max_workers parameter. Let’s create a ThreadPoolExecutor and call the submit function with the wait_function as an input callable function. Remember that wait_function accepts two input parameters. I am going to pass them as separate parameters instead of a tuple:

with ThreadPoolExecutor(max_workers=1) as executor:
    future = executor.submit(wait_function, 3, 4)

The submit function will return a Future object that encapsulates the asynchronous execution of a callable. The most commonly used functions for the Future object are:

  • cancel — Attempts to cancel the execution. Returns a boolean that indicates if the call has been successfully canceled.
  • running — Checks if the call is being executed. Returns a boolean.
  • done — Checks if the call was canceled or completed. Returns a boolean.
  • result — Returns the value returned by the call. If the call hasn’t yet completed, then this method will wait up to n seconds given by the input timeout parameter. It is highly recommended to check using the done function before calling the result, as timeout will block the current execution.
  • add_done_callback — Attaches the callable function to the Future object. This function will be called with Future as its only argument when Future is canceled or finishes running.

Append the following code right below the submit function. It is just a simple loop that prints a string while the thread is running. When it is completed, it will print out the result:

while True:
    if(future.running()):
        print("Task 1 running")
    elif(future.done()):
        print(future.result())
        break

Check out the complete code on GitHub:

from concurrent.futures import ThreadPoolExecutor
import time

def wait_function(x, y):
    print('Task(', x,'multiply', y, ') started')
    time.sleep(2)
    print('Task(', x,'multiply', y, ') completed')
    return x * y

with ThreadPoolExecutor(max_workers=1) as executor:
    future = executor.submit(wait_function, 3, 4)

    while True:
        if(future.running()):
            print("Task 1 running")
        elif(future.done()):
            print(future.result())
            break

You should see the following result when you run the Python file:

This is image title

Multiple tasks

Next, we are going to add another task to it so that both of them will run in parallel. Change the code in your Python file to the following:

from concurrent.futures import ThreadPoolExecutor
import time

def wait_function(x, y):
    print('Task(', x,'multiply', y, ') started')
    time.sleep(2)
    print('Task(', x,'multiply', y, ') completed')
    return x * y

with ThreadPoolExecutor(max_workers=1) as executor: #change max_workers to 2 and see the results
    future = executor.submit(wait_function, 3, 4)
    future2 = executor.submit(wait_function, 8, 8)
    while True:
        if(future.running()):
            print("Task 1 running")
        if(future2.running()):
            print("Task 2 running")

        if(future.done() and future2.done()):
            print(future.result(), future2.result())
            break

For now, set the max_workers to one first. Run it and you should notice that the tasks are not running in parallel. It will run the first task and then the second task. This is mainly because you only have one worker in the pool. Let’s increase the max_workers to two and you should be able to see that both tasks are running in parallel.

Callback function

You can attach a callback function to the Future object. It will call the attached function once the execution is canceled or completed. This is extremely useful if you intend to proceed with an update to the UI after a successful connection to the database or completion of URL requests. Let’s create a simple callback function for now:

def callback_function(future):
    print('Callback with the following result', future.result())

Add the following code right below the submit function:

future.add_done_callback(callback_function)

Check out the complete code on GitHub:

The following result will be shown in the console when you run the Python file:

from concurrent.futures import ThreadPoolExecutor
import time

def wait_function(x, y):
    print('Task(', x,'multiply', y, ') started')
    time.sleep(2)
    print('Task(', x,'multiply', y, ') completed')
    return x * y

def callback_function(future):
    print('Callback with the following result', future.result())

with ThreadPoolExecutor(max_workers=1) as executor: #change max_workers to 2 and see the results
    future = executor.submit(wait_function, 3, 4)
    future.add_done_callback(callback_function)
    future2 = executor.submit(wait_function, 8, 8)
    while True:
        if(future.running()):
            print("Task 1 running")
        if(future2.running()):
            print("Task 2 running")

        if(future.done() and future2.done()):
            print(future.result(), future2.result())
            break

This is image title

2. ProcessPoolExecutor

The ProcessPoolExecutor class works exactly the same as ThreadPoolExecutor, but with a few minor differences. It uses the multiprocessing module, which allows it to sidestep the Global Interpreter Lock. However, this also means that only pickable objects can be executed and returned.

Besides, it does not work in an interactive interpreter and must have a __main__ function that is importable by worker subprocesses. max_workers will be the number of processes in the machine. On Windows operating system, max_workers must be equal to or lower than 61.

You have to import the ProcessPoolExecutor to use it:

from concurrent.futures import ProcessPoolExecutor

You can reuse the previous code and modify it to ProcessPoolExecutor instead of ThreadPoolExecutor. Wrap the code inside a function and call it directly from __main__. Check out the complete code in the following on GitHub:

from concurrent.futures import ProcessPoolExecutor
import time

def wait_function(x, y):
    print('Task(', x,'multiply', y, ') started')
    time.sleep(2)
    print('Task(', x,'multiply', y, ') completed')
    return x * y

def callback_function(future):
    print('Callback with the following result', future.result())
    
def main():
    with ProcessPoolExecutor(max_workers=2) as executor:
        future = executor.submit(wait_function, 3, 4)
        future.add_done_callback(callback_function)
        future2 = executor.submit(wait_function, 8, 8)
        while True:
            if(future.running()):
                print("Task 1 running")
            if(future2.running()):
                print("Task 2 running")

            if(future.done() and future2.done()):
                print(future.result(), future2.result())
                break

if __name__ == '__main__':
    main()

3. Keyboard Interrupt

If you intend to stop the execution via Ctrl+C while the process is running in a thread, the compiler will most likely hang and get stuck at the KeyboardInterupt exception. This is mainly because the Ctrl+C command generates SIGINT, which will not stop or break the execution. You need to generate SIGBREAK to end the execution and return to the terminal. Use the following command to generate SIGBREAK based on the operating system and computer model:

This is image title

4. Conclusion

Let’s recap what we have learned today.

We started off with a simple explanation of the concurrent.futures module.

After that, we explored in-depth the basic ThreadPoolExecutor class and the Future class. We tried running multiple tasks in parallel with a different number of max_workers. We also tested out setting up a callback function that will execute upon completion of the task.

We moved on to the ProcessPoolExecutor, which is similar to the ThreadPoolExecutor with a few minor differences.

Thanks for reading this piece. I hope to see you again in the next article!

#python #data science #devops #async

What is GEEK

Buddha Community

How can I Launch Parallel Tasks in Python
Ray  Patel

Ray Patel

1619510796

Lambda, Map, Filter functions in python

Welcome to my Blog, In this article, we will learn python lambda function, Map function, and filter function.

Lambda function in python: Lambda is a one line anonymous function and lambda takes any number of arguments but can only have one expression and python lambda syntax is

Syntax: x = lambda arguments : expression

Now i will show you some python lambda function examples:

#python #anonymous function python #filter function in python #lambda #lambda python 3 #map python #python filter #python filter lambda #python lambda #python lambda examples #python map

Shardul Bhatt

Shardul Bhatt

1626775355

Why use Python for Software Development

No programming language is pretty much as diverse as Python. It enables building cutting edge applications effortlessly. Developers are as yet investigating the full capability of end-to-end Python development services in various areas. 

By areas, we mean FinTech, HealthTech, InsureTech, Cybersecurity, and that's just the beginning. These are New Economy areas, and Python has the ability to serve every one of them. The vast majority of them require massive computational abilities. Python's code is dynamic and powerful - equipped for taking care of the heavy traffic and substantial algorithmic capacities. 

Programming advancement is multidimensional today. Endeavor programming requires an intelligent application with AI and ML capacities. Shopper based applications require information examination to convey a superior client experience. Netflix, Trello, and Amazon are genuine instances of such applications. Python assists with building them effortlessly. 

5 Reasons to Utilize Python for Programming Web Apps 

Python can do such numerous things that developers can't discover enough reasons to admire it. Python application development isn't restricted to web and enterprise applications. It is exceptionally adaptable and superb for a wide range of uses.

Robust frameworks 

Python is known for its tools and frameworks. There's a structure for everything. Django is helpful for building web applications, venture applications, logical applications, and mathematical processing. Flask is another web improvement framework with no conditions. 

Web2Py, CherryPy, and Falcon offer incredible capabilities to customize Python development services. A large portion of them are open-source frameworks that allow quick turn of events. 

Simple to read and compose 

Python has an improved sentence structure - one that is like the English language. New engineers for Python can undoubtedly understand where they stand in the development process. The simplicity of composing allows quick application building. 

The motivation behind building Python, as said by its maker Guido Van Rossum, was to empower even beginner engineers to comprehend the programming language. The simple coding likewise permits developers to roll out speedy improvements without getting confused by pointless subtleties. 

Utilized by the best 

Alright - Python isn't simply one more programming language. It should have something, which is the reason the business giants use it. Furthermore, that too for different purposes. Developers at Google use Python to assemble framework organization systems, parallel information pusher, code audit, testing and QA, and substantially more. Netflix utilizes Python web development services for its recommendation algorithm and media player. 

Massive community support 

Python has a steadily developing community that offers enormous help. From amateurs to specialists, there's everybody. There are a lot of instructional exercises, documentation, and guides accessible for Python web development solutions. 

Today, numerous universities start with Python, adding to the quantity of individuals in the community. Frequently, Python designers team up on various tasks and help each other with algorithmic, utilitarian, and application critical thinking. 

Progressive applications 

Python is the greatest supporter of data science, Machine Learning, and Artificial Intelligence at any enterprise software development company. Its utilization cases in cutting edge applications are the most compelling motivation for its prosperity. Python is the second most well known tool after R for data analytics.

The simplicity of getting sorted out, overseeing, and visualizing information through unique libraries makes it ideal for data based applications. TensorFlow for neural networks and OpenCV for computer vision are two of Python's most well known use cases for Machine learning applications.

Summary

Thinking about the advances in programming and innovation, Python is a YES for an assorted scope of utilizations. Game development, web application development services, GUI advancement, ML and AI improvement, Enterprise and customer applications - every one of them uses Python to its full potential. 

The disadvantages of Python web improvement arrangements are regularly disregarded by developers and organizations because of the advantages it gives. They focus on quality over speed and performance over blunders. That is the reason it's a good idea to utilize Python for building the applications of the future.

#python development services #python development company #python app development #python development #python in web development #python software development

Art  Lind

Art Lind

1602968400

Python Tricks Every Developer Should Know

Python is awesome, it’s one of the easiest languages with simple and intuitive syntax but wait, have you ever thought that there might ways to write your python code simpler?

In this tutorial, you’re going to learn a variety of Python tricks that you can use to write your Python code in a more readable and efficient way like a pro.

Let’s get started

Swapping value in Python

Instead of creating a temporary variable to hold the value of the one while swapping, you can do this instead

>>> FirstName = "kalebu"
>>> LastName = "Jordan"
>>> FirstName, LastName = LastName, FirstName 
>>> print(FirstName, LastName)
('Jordan', 'kalebu')

#python #python-programming #python3 #python-tutorials #learn-python #python-tips #python-skills #python-development

Art  Lind

Art Lind

1602666000

How to Remove all Duplicate Files on your Drive via Python

Today you’re going to learn how to use Python programming in a way that can ultimately save a lot of space on your drive by removing all the duplicates.

Intro

In many situations you may find yourself having duplicates files on your disk and but when it comes to tracking and checking them manually it can tedious.

Heres a solution

Instead of tracking throughout your disk to see if there is a duplicate, you can automate the process using coding, by writing a program to recursively track through the disk and remove all the found duplicates and that’s what this article is about.

But How do we do it?

If we were to read the whole file and then compare it to the rest of the files recursively through the given directory it will take a very long time, then how do we do it?

The answer is hashing, with hashing can generate a given string of letters and numbers which act as the identity of a given file and if we find any other file with the same identity we gonna delete it.

There’s a variety of hashing algorithms out there such as

  • md5
  • sha1
  • sha224, sha256, sha384 and sha512

#python-programming #python-tutorials #learn-python #python-project #python3 #python #python-skills #python-tips

Jupyter Notebook Kernel for Running ansible Tasks and Playbooks

Ansible Jupyter Kernel

Example Jupyter Usage

The Ansible Jupyter Kernel adds a kernel backend for Jupyter to interface directly with Ansible and construct plays and tasks and execute them on the fly.

Demo

Demo

Installation:

ansible-kernel is available to be installed from pypi but you can also install it locally. The setup package itself will register the kernel with Jupyter automatically.

From pypi

pip install ansible-kernel
python -m ansible_kernel.install

From a local checkout

pip install -e .
python -m ansible_kernel.install

For Anaconda/Miniconda

pip install ansible-kernel
python -m ansible_kernel.install --sys-prefix

Usage

Local install

    jupyter notebook
    # In the notebook interface, select Ansible from the 'New' menu

Container

docker run -p 8888:8888 benthomasson/ansible-jupyter-kernel

Then copy the URL from the output into your browser:
http://localhost:8888/?token=ABCD1234

Using the Cells

Normally Ansible brings together various components in different files and locations to launch a playbook and performs automation tasks. For this jupyter interface you need to provide this information in cells by denoting what the cell contains and then finally writing your tasks that will make use of them. There are Examples available to help you, in this section we'll go over the currently supported cell types.

In order to denote what the cell contains you should prefix it with a pound/hash symbol (#) and the type as listed here as the first line as shown in the examples below.

#inventory

The inventory that your tasks will use

#inventory
[all]
ahost ansible_connection=local
anotherhost examplevar=val

#play

This represents the opening block of a typical Ansible play

#play
name: Hello World
hosts: all
gather_facts: false

#task

This is the default cell type if no type is given for the first line

#task
debug:
#task
shell: cat /tmp/afile
register: output

#host_vars

This takes an argument that represents the hostname. Variables defined in this file will be available in the tasks for that host.

#host_vars Host1
hostname: host1

#group_vars

This takes an argument that represents the group name. Variables defined in this file will be available in the tasks for hosts in that group.

#group_vars BranchOfficeX
gateway: 192.168.1.254

#vars

This takes an argument that represents the filename for use in later cells

#vars example_vars
message: hello vars
#play
name: hello world
hosts: localhost
gather_facts: false
vars_files:
    - example_vars

#template

This takes an argument in order to create a templated file that can be used in later cells

#template hello.j2
{{ message }}
#task
template:
    src: hello.j2
    dest: /tmp/hello

#ansible.cfg

Provides overrides typically found in ansible.cfg

#ansible.cfg
[defaults]
host_key_checking=False

Examples

You can find various example notebooks in the repository

Using the development environment

It's possible to use whatever python development process you feel comfortable with. The repository itself includes mechanisms for using pipenv

pipenv install
...
pipenv shell

Author: ansible
Source Code:  https://github.com/ansible/ansible-jupyter-kernel
License: Apache-2.0 License

#jupyter #python