Python Multithreading and Multiprocessing Tutorial

Python Multithreading and Multiprocessing Tutorial

Discussions criticizing Python often talk about how it is difficult to use Python for multithreaded work, pointing fingers at what is known as the global interpreter ...

Discussions criticizing Python often talk about how it is difficult to use Python for multithreaded work, pointing fingers at what is known as the global interpreter ...

Note: By popular request that I demonstrate some alternative techniques---including async/await, only available since the advent of Python 3.5---I've added some updates at the end of the article. Enjoy!

Discussions criticizing Python often talk about how it is difficult to use Python for multithreaded work, pointing fingers at what is known as the global interpreter lock (affectionately referred to as the GIL) that prevents multiple threads of Python code from running simultaneously. Due to this, the Python multithreading module doesn’t quite behave the way you would expect it to if you’re not a Python developer and you are coming from other languages such as C++ or Java. It must be made clear that one can still write code in Python that runs concurrently or in parallel and make a stark difference in resulting performance, as long as certain things are taken into consideration. If you haven’t read it yet, I suggest you take a look at Eqbal Quran’s article on concurrency and parallelism in Ruby here on the Toptal Engineering Blog.

In this Python concurrency tutorial, we will write a small Python script to download the top popular images from Imgur. We will start with a version that downloads images sequentially, or one at a time. As a prerequisite, you will have to register an application on Imgur. If you do not have an Imgur account already, please create one first.

The scripts in these threading examples have been tested with Python 3.6.4. With some changes, they should also run with Python 2—urllib is what has changed the most between these two versions of Python.

Getting Started with Python Multithreading

Let us start by creating a Python module, named download.py. This file will contain all the functions necessary to fetch the list of images and download them. We will split these functionalities into three separate functions:

  • get_links
  • download_link
  • setup_download_dir

The third function, setup_download_dir, will be used to create a download destination directory if it doesn’t already exist.

Imgur’s API requires HTTP requests to bear the Authorization header with the client ID. You can find this client ID from the dashboard of the application that you have registered on Imgur, and the response will be JSON encoded. We can use Python’s standard JSON library to decode it. Downloading the image is an even simpler task, as all you have to do is fetch the image by its URL and write it to a file.

This is what the script looks like:

import json
import logging
import os
from pathlib import Path
from urllib.request import urlopen, Request

logger = logging.getLogger(__name__)

types = {'image/jpeg', 'image/png'}

def get_links(client_id):
    headers = {'Authorization': 'Client-ID {}'.format(client_id)}
    req = Request('https://api.imgur.com/3/gallery/random/random/', headers=headers, method='GET')
    with urlopen(req) as resp:
        data = json.loads(resp.read().decode('utf-8'))
    return [item['link'] for item in data['data'] if 'type' in item and item['type'] in types]

def download_link(directory, link):
    download_path = directory / os.path.basename(link)
    with urlopen(link) as image, download_path.open('wb') as f:
        f.write(image.read())
    logger.info('Downloaded %s', link)

def setup_download_dir():
    download_dir = Path('images')
    if not download_dir.exists():
        download_dir.mkdir()
    return download_dir


Next, we will need to write a module that will use these functions to download the images, one by one. We will name this single.py. This will contain the main function of our first, naive version of the Imgur image downloader. The module will retrieve the Imgur client ID in the environment variable IMGUR_CLIENT_ID. It will invoke the setup_download_dir to create the download destination directory. Finally, it will fetch a list of images using the get_links function, filter out all GIF and album URLs, and then use download_link to download and save each of those images to the disk. Here is what single.py looks like:

import logging
import os
from time import time

from download import setup_download_dir, get_links, download_link

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

def main():
    ts = time()
    client_id = os.getenv('IMGUR_CLIENT_ID')
    if not client_id:
        raise Exception("Couldn't find IMGUR_CLIENT_ID environment variable!")
    download_dir = setup_download_dir()
    links = get_links(client_id)
    for link in links:
        download_link(download_dir, link)
    logging.info('Took %s seconds', time() - ts)

if __name__ == '__main__':
    main()


On my laptop, this script took 19.4 seconds to download 91 images. Please do note that these numbers may vary based on the network you are on. 19.4 seconds isn’t terribly long, but what if we wanted to download more pictures? Perhaps 900 images, instead of 90. With an average of 0.2 seconds per picture, 900 images would take approximately 3 minutes. For 9000 pictures it would take 30 minutes. The good news is that by introducing concurrency or parallelism, we can speed this up dramatically.

All subsequent code examples will only show import statements that are new and specific to those examples. For convenience, all of these Python scripts can be found in this GitHub repository.

Concurrency and Parallelism in Python: Threading Example

Threading is one of the most well-known approaches to attaining Python concurrency and parallelism. Threading is a feature usually provided by the operating system. Threads are lighter than processes, and share the same memory space.

In this Python threading example, we will write a new module to replace single.py. This module will create a pool of eight threads, making a total of nine threads including the main thread. I chose eight worker threads because my computer has eight CPU cores and one worker thread per core seemed a good number for how many threads to run at once. In practice, this number is chosen much more carefully based on other factors, such as other applications and services running on the same machine.

This is almost the same as the previous one, with the exception that we now have a new class, DownloadWorker, which is a descendent of the Python Thread class. The run method has been overridden, which runs an infinite loop. On every iteration, it calls self.queue.get() to try and fetch a URL to from a thread-safe queue. It blocks until there is an item in the queue for the worker to process. Once the worker receives an item from the queue, it then calls the same download_link method that was used in the previous script to download the image to the images directory. After the download is finished, the worker signals the queue that that task is done. This is very important, because the Queue keeps track of how many tasks were enqueued. The call to queue.join() would block the main thread forever if the workers did not signal that they completed a task.

import logging
import os
from queue import Queue
from threading import Thread
from time import time

from download import setup_download_dir, get_links, download_link

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')

logger = logging.getLogger(__name__)

class DownloadWorker(Thread):

    def __init__(self, queue):
        Thread.__init__(self)
        self.queue = queue

    def run(self):
        while True:
            # Get the work from the queue and expand the tuple
            directory, link = self.queue.get()
            try:
                download_link(directory, link)
            finally:
                self.queue.task_done()

def main():
    ts = time()
    client_id = os.getenv('IMGUR_CLIENT_ID')
    if not client_id:
        raise Exception("Couldn't find IMGUR_CLIENT_ID environment variable!")
    download_dir = setup_download_dir()
    links = get_links(client_id)
    # Create a queue to communicate with the worker threads
    queue = Queue()
    # Create 8 worker threads
    for x in range(8):
        worker = DownloadWorker(queue)
        # Setting daemon to True will let the main thread exit even though the workers are blocking
        worker.daemon = True
        worker.start()
    # Put the tasks into the queue as a tuple
    for link in links:
        logger.info('Queueing {}'.format(link))
        queue.put((download_dir, link))
    # Causes the main thread to wait for the queue to finish processing all the tasks
    queue.join()
    logging.info('Took %s', time() - ts)

if __name__ == '__main__':
    main()


Running this Python threading example script on the same machine used earlier results in a download time of 4.1 seconds! That’s 4.7 times faster than the previous example. While this is much faster, it is worth mentioning that only one thread was executing at a time throughout this process due to the GIL. Therefore, this code is concurrent but not parallel. The reason it is still faster is because this is an IO bound task. The processor is hardly breaking a sweat while downloading these images, and the majority of the time is spent waiting for the network. This is why Python multithreading can provide a large speed increase. The processor can switch between the threads whenever one of them is ready to do some work. Using the threading module in Python or any other interpreted language with a GIL can actually result in reduced performance. If your code is performing a CPU bound task, such as decompressing gzip files, using the threading module will result in a slower execution time. For CPU bound tasks and truly parallel execution, we can use the multiprocessing module.

While the de facto reference Python implementation—CPython–has a GIL, this is not true of all Python implementations. For example, IronPython, a Python implementation using the .NET framework, does not have a GIL, and neither does Jython, the Java-based implementation. You can find a list of working Python implementations here.

Related: Python Best Practices and Tips by Toptal Developers](https://www.toptal.com/python/tips-and-practices) "https://www.toptal.com/python/tips-and-practices)")

Concurrency and Parallelism in Python Example 2: Spawning Multiple Processes

The multiprocessing module is easier to drop in than the threading module, as we don’t need to add a class like the Python threading example. The only changes we need to make are in the main function.

To use multiple processes, we create a multiprocessing Pool. With the map method it provides, we will pass the list of URLs to the pool, which in turn will spawn eight new processes and use each one to download the images in parallel. This is true parallelism, but it comes with a cost. The entire memory of the script is copied into each subprocess that is spawned. In this simple example, it isn’t a big deal, but it can easily become serious overhead for non-trivial programs.

import logging
import os
from functools import partial
from multiprocessing.pool import Pool
from time import time

from download import setup_download_dir, get_links, download_link

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logging.getLogger('requests').setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

def main():
    ts = time()
    client_id = os.getenv('IMGUR_CLIENT_ID')
    if not client_id:
        raise Exception("Couldn't find IMGUR_CLIENT_ID environment variable!")
    download_dir = setup_download_dir()
    links = get_links(client_id)
    download = partial(download_link, download_dir)
    with Pool(4) as p:
        p.map(download, links)
    logging.info('Took %s seconds', time() - ts)

if __name__ == '__main__':
    main()


Concurrency and Parallelism in Python Example 3: Distributing to Multiple Workers

While the threading and multiprocessing modules are great for scripts that are running on your personal computer, what should you do if you want the work to be done on a different machine, or you need to scale up to more than the CPU on one machine can handle? A great use case for this is long-running back-end tasks for web applications. If you have some long-running tasks, you don’t want to spin up a bunch of sub-processes or threads on the same machine that need to be running the rest of your application code. This will degrade the performance of your application for all of your users. What would be great is to be able to run these jobs on another machine, or many other machines.

A great Python library for this task is RQ, a very simple yet powerful library. You first enqueue a function and its arguments using the library. This pickles the function call representation, which is then appended to a Redis list. Enqueueing the job is the first step, but will not do anything yet. We also need at least one worker to listen on that job queue.

The first step is to install and run a Redis server on your computer, or have access to a running Redis server. After that, there are only a few small changes made to the existing code. We first create an instance of an RQ Queue and pass it an instance of a Redis server from the redis-py library. Then, instead of just calling our download_linkmethod, we call q.enqueue(download_link, download_dir, link). The enqueue method takes a function as its first argument, then any other arguments or keyword arguments are passed along to that function when the job is actually executed.

One last step we need to do is to start up some workers. RQ provides a handy script to run workers on the default queue. Just run rqworker in a terminal window and it will start a worker listening on the default queue. Please make sure your current working directory is the same as where the scripts reside in. If you want to listen to a different queue, you can run rqworker queue_name and it will listen to that named queue. The great thing about RQ is that as long as you can connect to Redis, you can run as many workers as you like on as many different machines as you like; therefore, it is very easy to scale up as your application grows. Here is the source for the RQ version:

import logging
import os

from redis import Redis

from rq import Queue

from download import setup_download_dir, get_links, download_link

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logging.getLogger('requests').setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

def main():
    client_id = os.getenv('IMGUR_CLIENT_ID')
    if not client_id:
        raise Exception("Couldn't find IMGUR_CLIENT_ID environment variable!")
    download_dir = setup_download_dir()
    links = get_links(client_id)
    q = Queue(connection=Redis(host='localhost', port=6379))
    for link in links:
        q.enqueue(download_link, download_dir, link)

if __name__ == '__main__':
    main()


However, RQ is not the only Python job queue solution. RQ is easy to use and covers simple use cases extremely well, but if more advanced options are required, other Python 3 queue solutions (such as Celery) can be used.

Python Multithreading vs. Multiprocessing

If your code is IO bound, both multiprocessing and multithreading in Python will work for you. Multiprocessing is a easier to just drop in than threading but has a higher memory overhead. If your code is CPU bound, multiprocessing is most likely going to be the better choice—especially if the target machine has multiple cores or CPUs. For web applications, and when you need to scale the work across multiple machines, RQ is going to be better for you.

Related: Become More Advanced: Avoid the 10 Most Common Mistakes That Python Programmers Make](https://www.toptal.com/python/top-10-mistakes-that-python-programmers-make) "https://www.toptal.com/python/top-10-mistakes-that-python-programmers-make)")

 

Update

Python concurrent.futures

Something new since Python 3.2 that wasn’t touched upon in the original article is the concurrent.futurespackage. This package provides yet another way to use concurrency and parallelism with Python.

In the original article, I mentioned that Python’s multiprocessing module would be easier to drop into existing code than the threading module. This was because the Python 3 threading module required subclassing the Thread class and also creating a Queue for the threads to monitor for work.

Using a concurrent.futures.ThreadPoolExecutor makes the Python threading example code almost identical to the multiprocessing module.

import logging
import os
from concurrent.futures import ThreadPoolExecutor
from functools import partial
from time import time

from download import setup_download_dir, get_links, download_link

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')

logger = logging.getLogger(__name__)

def main():
    client_id = os.getenv('IMGUR_CLIENT_ID')
    if not client_id:
        raise Exception("Couldn't find IMGUR_CLIENT_ID environment variable!")
    download_dir = setup_download_dir()
    links = get_links(client_id)

    # By placing the executor inside a with block, the executors shutdown method
    # will be called cleaning up threads.
    # 
    # By default, the executor sets number of workers to 5 times the number of
    # CPUs.
    with ThreadPoolExecutor() as executor:

        # Create a new partially applied function that stores the directory
        # argument.
        # 
        # This allows the download_link function that normally takes two
        # arguments to work with the map function that expects a function of a
        # single argument.
        fn = partial(download_link, download_dir)

        # Executes fn concurrently using threads on the links iterable. The
        # timeout is for the entire process, not a single call, so downloading
        # all images must complete within 30 seconds.
        executor.map(fn, links, timeout=30)

if __name__ == '__main__':
    main()


Now that we have all these images downloaded with our Python ThreadPoolExecutor, we can use them to test a CPU-bound task. We can create thumbnail versions of all the images in both a single-threaded, single-process script and then test a multiprocessing-based solution.

We are going to use the Pillow library to handle the resizing of the images.

Here is our initial script.

import logging
from pathlib import Path
from time import time

from PIL import Image

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')

logger = logging.getLogger(__name__)

def create_thumbnail(size, path):
    """
    Creates a thumbnail of an image with the same name as image but with
    _thumbnail appended before the extension.  E.g.:

    >>> create_thumbnail((128, 128), 'image.jpg')

    A new thumbnail image is created with the name image_thumbnail.jpg

    :param size: A tuple of the width and height of the image
    :param path: The path to the image file
    :return: None
    """
    image = Image.open(path)
    image.thumbnail(size)
    path = Path(path)
    name = path.stem + '_thumbnail' + path.suffix
    thumbnail_path = path.with_name(name)
    image.save(thumbnail_path)

def main():
    ts = time()
    for image_path in Path('images').iterdir():
        create_thumbnail((128, 128), image_path)
    logging.info('Took %s', time() - ts)

if __name__ == '__main__':
    main()


This script iterates over the paths in the images folder and for each path it runs the create_thumbnail function. This function uses Pillow to open the image, create a thumbnail, and save the new, smaller image with the same name as the original but with _thumbnail appended to the name.

Running this script on 160 images totaling 36 million takes 2.32 seconds. Lets see if we can speed this up using a ProcessPoolExecutor.

import logging
from pathlib import Path
from time import time
from functools import partial

from concurrent.futures import ProcessPoolExecutor

from PIL import Image

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')

logger = logging.getLogger(__name__)

def create_thumbnail(size, path):
    """
    Creates a thumbnail of an image with the same name as image but with
    _thumbnail appended before the extension. E.g.:

    >>> create_thumbnail((128, 128), 'image.jpg')

    A new thumbnail image is created with the name image_thumbnail.jpg

    :param size: A tuple of the width and height of the image
    :param path: The path to the image file
    :return: None
    """
    path = Path(path)
    name = path.stem + '_thumbnail' + path.suffix
    thumbnail_path = path.with_name(name)
    image = Image.open(path)
    image.thumbnail(size)
    image.save(thumbnail_path)

def main():
    ts = time()
    # Partially apply the create_thumbnail method, setting the size to 128x128
    # and returning a function of a single argument.
    thumbnail_128 = partial(create_thumbnail, (128, 128))

    # Create the executor in a with block so shutdown is called when the block
    # is exited.
    with ProcessPoolExecutor() as executor:
        executor.map(thumbnail_128, Path('images').iterdir())
    logging.info('Took %s', time() - ts)

if __name__ == '__main__':
    main()


The create_thumbnail method is identical to the last script. The main difference is the creation of a ProcessPoolExecutor. The executor’s map method is used to create the thumbnails in parallel. By default, the ProcessPoolExecutor creates one subprocess per CPU. Running this script on the same 160 images took 1.05 seconds—2.2 times faster!

Async/Await (Python 3.5+ only)

One of the most requested items in the comments on the original article was for an example using Python 3’s asynciomodule. Compared to the other examples, there is some new Python syntax that may be new to most people and also some new concepts. An unfortunate additional layer of complexity is caused by Python’s built-in urllib module not being asynchronous. We will need to use an async HTTP library to get the full benefits of asyncio. For this, we’ll use aiohttp.

Let’s jump right into the code and a more detailed explanation will follow.

import asyncio
import logging
import os
from time import time

import aiohttp

from download import setup_download_dir, get_links

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

async def async_download_link(session, directory, link):
    """
    Async version of the download_link method we've been using in the other examples.
    :param session: aiohttp ClientSession
    :param directory: directory to save downloads
    :param link: the url of the link to download
    :return:
    """
    download_path = directory / os.path.basename(link)
    async with session.get(link) as response:
        with download_path.open('wb') as f:
            while True:
                # await pauses execution until the 1024 (or less) bytes are read from the stream
                chunk = await response.content.read(1024)
                if not chunk:
                    # We are done reading the file, break out of the while loop
                    break
                f.write(chunk)
    logger.info('Downloaded %s', link)

# Main is now a coroutine
async def main():
    client_id = os.getenv('IMGUR_CLIENT_ID')
    if not client_id:
        raise Exception("Couldn't find IMGUR_CLIENT_ID environment variable!")
    download_dir = setup_download_dir()
    # We use a session to take advantage of tcp keep-alive
    # Set a 3 second read and connect timeout. Default is 5 minutes
    async with aiohttp.ClientSession(conn_timeout=3, read_timeout=3) as session:
        tasks = [(async_download_link(session, download_dir, l)) for l in get_links(client_id)]
        # gather aggregates all the tasks and schedules them in the event loop
        await asyncio.gather(*tasks, return_exceptions=True)

if __name__ == '__main__':
    ts = time()
    # Create the asyncio event loop
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(main())
    finally:
        # Shutdown the loop even if there is an exception
        loop.close()
    logger.info('Took %s seconds to complete', time() - ts)


There is quite a bit to unpack here. Let’s start with the main entry point of the program. The first new thing we do with the asyncio module is to obtain the event loop. The event loop handles all of the asynchronous code. Then, the loop is run until complete and passed the main function. There is a piece of new syntax in the definition of main: async def. You’ll also notice await and with async.

The async/await syntax was introduced in PEP492. The async def syntax marks a function as a coroutine. Internally, coroutines are based on Python generators, but aren’t exactly the same thing. Coroutines return a coroutine object similar to how generators return a generator object. Once you have a coroutine, you obtain its results with the await expression. When a coroutine calls await, execution of the coroutine is suspended until the awaitable completes. This suspension allows other work to be completed while the coroutine is suspended “awaiting” some result. In general, this result will be some kind of I/O like a database request or in our case an HTTP request.

The download_link function had to be changed pretty significantly. Previously, we were relying on urllib to do the brunt of the work of reading the image for us. Now, to allow our method to work properly with the async programming paradigm, we’ve introduced a while loop that reads chunks of the image at a time and suspends execution while waiting for the I/O to complete. This allows the event loop to loop through downloading the different images as each one has new data available during the download.

There Should Be One—Preferably Only One—Obvious Way to Do It

While the zen of Python tells us there should be one obvious way to do something, there are many ways in Python to introduce concurrency into our programs. The best method to choose is going to depend on your specific use case. The asynchronous paradigm scales better to high-concurrency workloads (like a webserver) compared to threading or multiprocessing, but it requires your code (and dependencies) to be async in order to fully benefit.

Hopefully the Python threading examples in this article—and update—will point you in the right direction so you have an idea of where to look in the Python standard library if you need to introduce concurrency into your programs.

Originally published by  MARCUS MCCURDY at* *toptal.com

=============================================================================

Thanks for reading :heart: If you liked this post, share it with all of your programming buddies! Follow me on Facebook | Twitter

Learn More

☞ Complete Python Bootcamp: Go from zero to hero in Python 3

☞ Complete Python Masterclass

☞ Learn Python by Building a Blockchain & Cryptocurrency

☞ Python and Django Full Stack Web Developer Bootcamp

☞ The Python Bible™ | Everything You Need to Program in Python

☞ Learning Python for Data Analysis and Visualization

☞ Python for Financial Analysis and Algorithmic Trading

☞ The Modern Python 3 Bootcamp

What's Python IDLE? How to use Python IDLE to interact with Python?

What's Python IDLE? How to use Python IDLE to interact with Python?

In this tutorial, you’ll learn all the basics of using **IDLE** to write Python programs. You'll know what Python IDLE is and how you can use it to interact with Python directly. You’ve also learned how to work with Python files and customize Python IDLE to your liking.

In this tutorial, you'll learn how to use the development environment included with your Python installation. Python IDLE is a small program that packs a big punch! You'll learn how to use Python IDLE to interact with Python directly, work with Python files, and improve your development workflow.

If you’ve recently downloaded Python onto your computer, then you may have noticed a new program on your machine called IDLE. You might be wondering, “What is this program doing on my computer? I didn’t download that!” While you may not have downloaded this program on your own, IDLE comes bundled with every Python installation. It’s there to help you get started with the language right out of the box. In this tutorial, you’ll learn how to work in Python IDLE and a few cool tricks you can use on your Python journey!

In this tutorial, you’ll learn:

  • What Python IDLE is
  • How to interact with Python directly using IDLE
  • How to edit, execute, and debug Python files with IDLE
  • How to customize Python IDLE to your liking

Table of Contents

What Is Python IDLE?

Every Python installation comes with an Integrated Development and Learning Environment, which you’ll see shortened to IDLE or even IDE. These are a class of applications that help you write code more efficiently. While there are many IDEs for you to choose from, Python IDLE is very bare-bones, which makes it the perfect tool for a beginning programmer.

Python IDLE comes included in Python installations on Windows and Mac. If you’re a Linux user, then you should be able to find and download Python IDLE using your package manager. Once you’ve installed it, you can then use Python IDLE as an interactive interpreter or as a file editor.

An Interactive Interpreter

The best place to experiment with Python code is in the interactive interpreter, otherwise known as a shell. The shell is a basic Read-Eval-Print Loop (REPL). It reads a Python statement, evaluates the result of that statement, and then prints the result on the screen. Then, it loops back to read the next statement.

The Python shell is an excellent place to experiment with small code snippets. You can access it through the terminal or command line app on your machine. You can simplify your workflow with Python IDLE, which will immediately start a Python shell when you open it.

A File Editor

Every programmer needs to be able to edit and save text files. Python programs are files with the .py extension that contain lines of Python code. Python IDLE gives you the ability to create and edit these files with ease.

Python IDLE also provides several useful features that you’ll see in professional IDEs, like basic syntax highlighting, code completion, and auto-indentation. Professional IDEs are more robust pieces of software and they have a steep learning curve. If you’re just beginning your Python programming journey, then Python IDLE is a great alternative!

How to Use the Python IDLE Shell

The shell is the default mode of operation for Python IDLE. When you click on the icon to open the program, the shell is the first thing that you see:

This is a blank Python interpreter window. You can use it to start interacting with Python immediately. You can test it out with a short line of code:

Here, you used print() to output the string "Hello, from IDLE!" to your screen. This is the most basic way to interact with Python IDLE. You type in commands one at a time and Python responds with the result of each command.

Next, take a look at the menu bar. You’ll see a few options for using the shell:

You can restart the shell from this menu. If you select that option, then you’ll clear the state of the shell. It will act as though you’ve started a fresh instance of Python IDLE. The shell will forget about everything from its previous state:

In the image above, you first declare a variable, x = 5. When you call print(x), the shell shows the correct output, which is the number 5. However, when you restart the shell and try to call print(x) again, you can see that the shell prints a traceback. This is an error message that says the variable x is not defined. The shell has forgotten about everything that came before it was restarted.

You can also interrupt the execution of the shell from this menu. This will stop any program or statement that’s running in the shell at the time of interruption. Take a look at what happens when you send a keyboard interrupt to the shell:

A KeyboardInterrupt error message is displayed in red text at the bottom of your window. The program received the interrupt and has stopped executing.

How to Work With Python Files

Python IDLE offers a full-fledged file editor, which gives you the ability to write and execute Python programs from within this program. The built-in file editor also includes several features, like code completion and automatic indentation, that will speed up your coding workflow. First, let’s take a look at how to write and execute programs in Python IDLE.

Opening a File

To start a new Python file, select File → New File from the menu bar. This will open a blank file in the editor, like this:

From this window, you can write a brand new Python file. You can also open an existing Python file by selecting File → Open… in the menu bar. This will bring up your operating system’s file browser. Then, you can find the Python file you want to open.

If you’re interested in reading the source code for a Python module, then you can select File → Path Browser. This will let you view the modules that Python IDLE can see. When you double click on one, the file editor will open up and you’ll be able to read it.

The content of this window will be the same as the paths that are returned when you call sys.path. If you know the name of a specific module you want to view, then you can select File → Module Browser and type in the name of the module in the box that appears.

Editing a File

Once you’ve opened a file in Python IDLE, you can then make changes to it. When you’re ready to edit a file, you’ll see something like this:

The contents of your file are displayed in the open window. The bar along the top of the window contains three pieces of important information:

  1. The name of the file that you’re editing
  2. The full path to the folder where you can find this file on your computer
  3. The version of Python that IDLE is using

In the image above, you’re editing the file myFile.py, which is located in the Documents folder. The Python version is 3.7.1, which you can see in parentheses.

There are also two numbers in the bottom right corner of the window:

  1. Ln: shows the line number that your cursor is on.
  2. Col: shows the column number that your cursor is on.

It’s useful to see these numbers so that you can find errors more quickly. They also help you make sure that you’re staying within a certain line width.

There are a few visual cues in this window that will help you remember to save your work. If you look closely, then you’ll see that Python IDLE uses asterisks to let you know that your file has unsaved changes:

The file name shown in the top of the IDLE window is surrounded by asterisks. This means that there are unsaved changes in your editor. You can save these changes with your system’s standard keyboard shortcut, or you can select File → Save from the menu bar. Make sure that you save your file with the .py extension so that syntax highlighting will be enabled.

Executing a File

When you want to execute a file that you’ve created in IDLE, you should first make sure that it’s saved. Remember, you can see if your file is properly saved by looking for asterisks around the filename at the top of the file editor window. Don’t worry if you forget, though! Python IDLE will remind you to save whenever you attempt to execute an unsaved file.

To execute a file in IDLE, simply press the F5 key on your keyboard. You can also select Run → Run Module from the menu bar. Either option will restart the Python interpreter and then run the code that you’ve written with a fresh interpreter. The process is the same as when you run python3 -i [filename] in your terminal.

When your code is done executing, the interpreter will know everything about your code, including any global variables, functions, and classes. This makes Python IDLE a great place to inspect your data if something goes wrong. If you ever need to interrupt the execution of your program, then you can press Ctrl+C in the interpreter that’s running your code.

How to Improve Your Workflow

Now that you’ve seen how to write, edit, and execute files in Python IDLE, it’s time to speed up your workflow! The Python IDLE editor offers a few features that you’ll see in most professional IDEs to help you code faster. These features include automatic indentation, code completion and call tips, and code context.

Automatic Indentation

IDLE will automatically indent your code when it needs to start a new block. This usually happens after you type a colon (:). When you hit the enter key after the colon, your cursor will automatically move over a certain number of spaces and begin a new code block.

You can configure how many spaces the cursor will move in the settings, but the default is the standard four spaces. The developers of Python agreed on a standard style for well-written Python code, and this includes rules on indentation, whitespace, and more. This standard style was formalized and is now known as PEP 8. To learn more about it, check out How to Write Beautiful Python Code With PEP 8.

Code Completion and Call Tips

When you’re writing code for a large project or a complicated problem, you can spend a lot of time just typing out all of the code you need. Code completion helps you save typing time by trying to finish your code for you. Python IDLE has basic code completion functionality. It can only autocomplete the names of functions and classes. To use autocompletion in the editor, just press the tab key after a sequence of text.

Python IDLE will also provide call tips. A call tip is like a hint for a certain part of your code to help you remember what that element needs. After you type the left parenthesis to begin a function call, a call tip will appear if you don’t type anything for a few seconds. For example, if you can’t quite remember how to append to a list, then you can pause after the opening parenthesis to bring up the call tip:

The call tip will display as a popup note, reminding you how to append to a list. Call tips like these provide useful information as you’re writing code.

Code Context

The code context functionality is a neat feature of the Python IDLE file editor. It will show you the scope of a function, class, loop, or other construct. This is particularly useful when you’re scrolling through a lengthy file and need to keep track of where you are while reviewing code in the editor.

To turn it on, select Options → Code Context in the menu bar. You’ll see a gray bar appear at the top of the editor window:

As you scroll down through your code, the context that contains each line of code will stay inside of this gray bar. This means that the print() functions you see in the image above are a part of a main function. When you reach a line that’s outside the scope of this function, the bar will disappear.

How to Debug in IDLE

A bug is an unexpected problem in your program. They can appear in many forms, and some are more difficult to fix than others. Some bugs are tricky enough that you won’t be able to catch them by just reading through your program. Luckily, Python IDLE provides some basic tools that will help you debug your programs with ease!

Interpreter DEBUG Mode

If you want to run your code with the built-in debugger, then you’ll need to turn this feature on. To do so, select Debug → Debugger from the Python IDLE menu bar. In the interpreter, you should see [DEBUG ON] appear just before the prompt (>>>), which means the interpreter is ready and waiting.

When you execute your Python file, the debugger window will appear:

In this window, you can inspect the values of your local and global variables as your code executes. This gives you insight into how your data is being manipulated as your code runs.

You can also click the following buttons to move through your code:

  • Go: Press this to advance execution to the next breakpoint. You’ll learn about these in the next section.
  • Step: Press this to execute the current line and go to the next one.
  • Over: If the current line of code contains a function call, then press this to step over that function. In other words, execute that function and go to the next line, but don’t pause while executing the function (unless there is a breakpoint).
  • Out: If the current line of code is in a function, then press this to step out of this function. In other words, continue the execution of this function until you return from it.

Be careful, because there is no reverse button! You can only step forward in time through your program’s execution.

You’ll also see four checkboxes in the debug window:

  1. Globals: your program’s global information
  2. Locals: your program’s local information during execution
  3. Stack: the functions that run during execution
  4. Source: your file in the IDLE editor

When you select one of these, you’ll see the relevant information in your debug window.

Breakpoints

A breakpoint is a line of code that you’ve identified as a place where the interpreter should pause while running your code. They will only work when DEBUG mode is turned on, so make sure that you’ve done that first.

To set a breakpoint, right-click on the line of code that you wish to pause. This will highlight the line of code in yellow as a visual indication of a set breakpoint. You can set as many breakpoints in your code as you like. To undo a breakpoint, right-click the same line again and select Clear Breakpoint.

Once you’ve set your breakpoints and turned on DEBUG mode, you can run your code as you would normally. The debugger window will pop up, and you can start stepping through your code manually.

Errors and Exceptions

When you see an error reported to you in the interpreter, Python IDLE lets you jump right to the offending file or line from the menu bar. All you have to do is highlight the reported line number or file name with your cursor and select Debug → Go to file/line from the menu bar. This is will open up the offending file and take you to the line that contains the error. This feature works regardless of whether or not DEBUG mode is turned on.

Python IDLE also provides a tool called a stack viewer. You can access it under the Debug option in the menu bar. This tool will show you the traceback of an error as it appears on the stack of the last error or exception that Python IDLE encountered while running your code. When an unexpected or interesting error occurs, you might find it helpful to take a look at the stack. Otherwise, this feature can be difficult to parse and likely won’t be useful to you unless you’re writing very complicated code.

How to Customize Python IDLE

There are many ways that you can give Python IDLE a visual style that suits you. The default look and feel is based on the colors in the Python logo. If you don’t like how anything looks, then you can almost always change it.

To access the customization window, select Options → Configure IDLE from the menu bar. To preview the result of a change you want to make, press Apply. When you’re done customizing Python IDLE, press OK to save all of your changes. If you don’t want to save your changes, then simply press Cancel.

There are 5 areas of Python IDLE that you can customize:

  1. Fonts/Tabs
  2. Highlights
  3. Keys
  4. General
  5. Extensions

Let’s take a look at each of them now.

Fonts/Tabs

The first tab allows you to change things like font color, font size, and font style. You can change the font to almost any style you like, depending on what’s available for your operating system. The font settings window looks like this:

You can use the scrolling window to select which font you prefer. (I recommend you select a fixed-width font like Courier New.) Pick a font size that’s large enough for you to see well. You can also click the checkbox next to Bold to toggle whether or not all text appears in bold.

This window will also let you change how many spaces are used for each indentation level. By default, this will be set to the PEP 8 standard of four spaces. You can change this to make the width of your code more or less spread out to your liking.

Highlights

The second customization tab will let you change highlights. Syntax highlighting is an important feature of any IDE that highlights the syntax of the language that you’re working in. This helps you visually distinguish between the different Python constructs and the data used in your code.

Python IDLE allows you to fully customize the appearance of your Python code. It comes pre-installed with three different highlight themes:

  1. IDLE Day
  2. IDLE Night
  3. IDLE New

You can select from these pre-installed themes or create your own custom theme right in this window:

Unfortunately, IDLE does not allow you to install custom themes from a file. You have to create customs theme from this window. To do so, you can simply start changing the colors for different items. Select an item, and then press Choose color for. You’ll be brought to a color picker, where you can select the exact color that you want to use.

You’ll then be prompted to save this theme as a new custom theme, and you can enter a name of your choosing. You can then continue changing the colors of different items if you’d like. Remember to press Apply to see your changes in action!

Keys

The third customization tab lets you map different key presses to actions, also known as keyboard shortcuts. These are a vital component of your productivity whenever you use an IDE. You can either come up with your own keyboard shortcuts, or you can use the ones that come with IDLE. The pre-installed shortcuts are a good place to start:

The keyboard shortcuts are listed in alphabetical order by action. They’re listed in the format Action - Shortcut, where Action is what will happen when you press the key combination in Shortcut. If you want to use a built-in key set, then select a mapping that matches your operating system. Pay close attention to the different keys and make sure your keyboard has them!

Creating Your Own Shortcuts

The customization of the keyboard shortcuts is very similar to the customization of syntax highlighting colors. Unfortunately, IDLE does not allow you to install custom keyboard shortcuts from a file. You must create a custom set of shortcuts from the Keys tab.

Select one pair from the list and press Get New Keys for Selection. A new window will pop up:

Here, you can use the checkboxes and scrolling menu to select the combination of keys that you want to use for this shortcut. You can select Advanced Key Binding Entry >> to manually type in a command. Note that this cannot pick up the keys you press. You have to literally type in the command as you see it displayed to you in the list of shortcuts.

General

The fourth tab of the customization window is a place for small, general changes. The general settings tab looks like this:

Here, you can customize things like the window size and whether the shell or the file editor opens first when you start Python IDLE. Most of the things in this window are not that exciting to change, so you probably won’t need to fiddle with them much.

Extensions

The fifth tab of the customization window lets you add extensions to Python IDLE. Extensions allow you to add new, awesome features to the editor and the interpreter window. You can download them from the internet and install them to right into Python IDLE.

To view what extensions are installed, select Options → Configure IDLE -> Extensions. There are many extensions available on the internet for you to read more about. Find the ones you like and add them to Python IDLE!

Conclusion

In this tutorial, you’ve learned all the basics of using IDLE to write Python programs. You know what Python IDLE is and how you can use it to interact with Python directly. You’ve also learned how to work with Python files and customize Python IDLE to your liking.

You’ve learned how to:

  • Work with the Python IDLE shell
  • Use Python IDLE as a file editor
  • Improve your workflow with features to help you code faster
  • Debug your code and view errors and exceptions
  • Customize Python IDLE to your liking

Now you’re armed with a new tool that will let you productively write Pythonic code and save you countless hours down the road. Happy programming!

Importance of Python Programming skills

Importance of Python Programming skills

Python is one among the most easiest and user friendly programming languages when it comes to the field of software engineering. The codes and syntaxes of python is so simple and easy to use that it can be deployed in any problem solving...

Python is one among the most easiest and user friendly programming languages when it comes to the field of software engineering. The codes and syntaxes of python is so simple and easy to use that it can be deployed in any problem solving challenges. The codes of Python can easily be deployed in Data Science and Machine Learning. Due to this ease of deployment and easier syntaxes, this platform has a lot of real world problem solving applications. According to the sources the companies are eagerly hunting for the professionals with python skills along with SQL. An average python developer in the united states makes around 1 lakh U.S Dollars per annum. In some of the top IT hubs in our country like Bangalore, the demand for professionals in the domains of Data Science and Python Programming has surpassed over the past few years. As a result of which a lot of various python certification courses are available right now.

Array in Python: An array is defined as a data structure that can hold a fixed number of elements that are of the same python data type. The following are some of the basic functions of array in python:

  1. To find the transverse
  2. For insertion of the elements
  3. For deletion of the elements
  4. For searching the elements

Along with this one can easily crack any python interview by means of python interview questions

Tkinter Python Tutorial | Python GUI Programming Using Tkinter Tutorial | Python Training

This video on Tkinter tutorial covers all the basic aspects of creating and making use of your own simple Graphical User Interface (GUI) using Python. It establishes all of the concepts needed to get started with building your own user interfaces while coding in Python.

This video on Tkinter tutorial covers all the basic aspects of creating and making use of your own simple Graphical User Interface (GUI) using Python. It establishes all of the concepts needed to get started with building your own user interfaces while coding in Python.

Learn More

☞ Complete Python Bootcamp: Go from zero to hero in Python 3

☞ Complete Python Masterclass

☞ Learn Python by Building a Blockchain & Cryptocurrency

☞ Python and Django Full Stack Web Developer Bootcamp

☞ The Python Bible™ | Everything You Need to Program in Python

☞ Learning Python for Data Analysis and Visualization

☞ Python for Financial Analysis and Algorithmic Trading

☞ The Modern Python 3 Bootcamp

Original video source: https://www.youtube.com/watch?v=VMP1oQOxfM0