Obtain Historical Weather Forecast data in CSV format using Python

Obtain Historical Weather Forecast data in CSV format using Python

Obtain Historical Weather Forecast data in CSV format using Python Recently, I worked on a machine learning project related to renewable energy, which required ****historical weather forecast data from multiple cities****. Despite intense research...

Obtain Historical Weather Forecast data in CSV format using Python
Recently, I worked on a machine learning project related to renewable energy, which required historical weather forecast data from multiple cities. Despite intense research, I had a hard time finding the good data source. Most websites restrict the access to only past two weeks of historical data. If you need more, you need to pay. In my case, I needed five years of data — hourly historical forecast, which can be costly.


My requirements are...

1. Free — at least during trial period

No need to provide credit card info.

2. Flexible

Flexible to change forecast interval, time periods, locations.

3. Reproducible

Easy to reproduce and implement in the production phase.

In the end, I decided to use data from World Weather Online. This took me less than two minutes to subscribe free trial premium API — without filling credit card info. (500 free requests/key/day for 60 days, as of 30-May-2019).

You can try out requests in JSON or XML format here. The result is nested JSON which needed a bit pre-processing work before feeding into ML models. Therefore, I wrote some scripts to parse them into pandas DataFrames and save as CSV for further use.


Introducing wwo-hist package

This wwo-hist package is used to retrieve and parse historical weather data from World Weather Online into pandas DataFrame and CSV file.

Input: api_key, location_list, start_date, end_date, frequency

Output: location_name.csv

Output column names: date_time, maxtempC, mintempC, totalSnow_cm, sunHour, uvIndex, uvIndex, moon_illumination, moonrise, moonset, sunrise, sunset, DewPointC, FeelsLikeC, HeatIndexC, WindChillC, WindGustKmph, cloudcover, humidity, precipMM, pressure, tempC, visibility, winddirDegree, windspeedKmph

Install and import the package:

pip install wwo-hist
# import the package and function
from wwo_hist import retrieve_hist_data

# set working directory to store output csv file(s)
import os
os.chdir(".\YOUR_PATH")

Example code:

Specify input parameters and call retrieve_hist_data(). Please visit my github repo for more info about parameters setup.

This will retrieve 3-hour interval historical weather forecast data for Singapore and California from 11-Dec-2018 to 11-Mar-2019, save output into hist_weather_data variable and CSV files.frequency = 3

FREQUENCY = 3
START_DATE = '11-DEC-2018'
END_DATE = '11-MAR-2019'
API_KEY = 'YOUR_API_KEY'
LOCATION_LIST = ['singapore','california']

hist_weather_data = retrieve_hist_data(API_KEY,
                                LOCATION_LIST,
                                START_DATE,
                                END_DATE,
                                FREQUENCY,
                                location_label = False,
                                export_csv = True,
                                store_df = True)

This is what you will see in your console

Result CSV(s) exported to your working directory.

Check the CSV output.

There you have it! The script detailed is also documented on GitHub.


Thank you for reading

Python Tutorial | Python Functions and Functional Programming

Python Tutorial | Python Functions and Functional Programming

In this post, we will introduce a functional programming model. We learn about lambda expressions in Python, important function functions and the concept of particles.

Most of us have been introduced to Python as an object-oriented language, but Python functions are also useful tools for data scientists and programmers alike. While classes, and objects, are easy to start working with, there are other ways to write your Python code. Languages like Java can make it hard to move away from object-oriented thinking, but Python makes it easy.

Given that Python facilitates different approaches to writing code, a logical follow-up question is: what is a different way to write code? While there are several answers to this question, the most common alternative style of writing code is called functional programming. Functional programming gets its name from writing functions which provides the main source of logic in a program.

In this post, we will:

  • Explain the basics of functional programming by comparing it to object-oriented programming.

  • Cover why you might want to incorporate functional programming in your own code.

  • Show you how Python allows you to switch between the two.

Comparing object-oriented to functional

The easiest way to introduce functional programming is to compare it to something we’re already aware of: object-oriented programming. Suppose we wanted to create a line counter class that takes in a file, reads each line, then counts the total amount of lines in the file. Using a class, it could look something like the following:

class LineCounter:
    def __init__(self, filename):
        self.file = open(filename, 'r')
        self.lines = []

        def read(self):
            self.lines = [line for line in self.file]

        def count(self):
            return len(self.lines)

While not the best implementation, it does provide an insight into object-oriented design. Within the class, there are the familiar concepts of methods and properties. The properties set and retrieve the state of the object, and the methods manipulate that state.

For both these concepts to work, the object’s state must change over time. This change of state is evident in the lines property after calling the read() method. As an example, here’s how we would use this class:

# example_file.txt contains 100 lines.
lc = LineCounter('example_file.txt')
print(lc.lines)
>> []
print(lc.count())
>> 0

# The lc object must read the file to
# set the lines property.
lc.read()
# The `lc.lines` property has been changed.
# This is called changing the state of the lc
# object.
print(lc.lines)
>> [['Hello world!', ...]]
print(lc.count())
>> 100

The ever-changing state of an object is both its blessing and curse. To understand why a changing state can be seen as a negative, we have to introduce an alternative. The alternative is to build the line counter as a series of independent functions.

def read(filename):
    with open(filename, 'r') as f:
        return [line for line in f]

def count(lines):
    return len(lines)

example_lines = read('example_log.txt')
lines_count = count(example_lines)

Working with pure functions

In the previous example, we were able to count the lines only with the use of functions. When we only use functions, we are applying a functional approach to programming which is, non-excitingly, called functional programming. The concepts behind functional programming requires functions to be stateless, and rely only on their given inputs to produce an output.

The functions that meet the above criteria are called pure functions. Here’s an example to highlight the difference between pure functions, and non-pure:

# Create a global variable `A`.
A = 5

def impure_sum(b):
    # Adds two numbers, but uses the
    # global `A` variable.
    return b + A

def pure_sum(a, b):
    # Adds two numbers, using
    # ONLY the local function inputs.
    return a + b

print(impure_sum(6))
>> 11

print(pure_sum(4, 6))
>> 10

The benefit of using pure functions over impure (non-pure) functions is the reduction of side effects. **Side effects **occur when there are changes performed within a function’s operation that are outside its scope. For example, they occur when we change the state of an object, perform any I/O operation, or even call print():

def read_and_print(filename):
    with open(filename) as f:
        # Side effect of opening a
        # file outside of function.
        data = [line for line in f]
    for line in data:
        # Call out to the operating system
        # "println" method (side effect).
        print(line)

Programmers reduce side effects in their code to make it easier to follow, test, and debug. The more side effects a codebase has, the harder it is to step through a program and understand its sequence of execution.

While it’s convienent to try and eliminate all side effects, they’re often used to make programming easier. If we were to ban all side effects, then you wouldn’t be able to read in a file, call print, or even assign a variable within a function. Advocates for functional programming understand this tradeoff, and try to eliminate side effects where possible without sacrificing development implementation time.

The Lambda Expression

Instead of the def syntax for function declaration, we can use a lambda expression to write Python functions. The lambda syntax closely follows the def syntax, but it’s not a 1-to-1 mapping. Here’s an example of building a function that adds two integers:

# Using `def` (old way).
def old_add(a, b):
    return a + b

# Using `lambda` (new way).
new_add = lambda a, b: a + bold_add(10, 5) == new_add(10, 5)
>> True

The lambda expression takes in a comma seperated sequences of inputs (like def). Then, immediately following the colon, it returns the expression without using an explicit return statement. Finally, when assigning thelambda expression to a variable, it acts exactly like a Python function, and can be called using the the function call syntax: new_add().

If we didn’t assign lambda to a variable name, it would be called an anonymous function. These** anonymous functions** are extremely helpful, especially when using them as an input for another function. For example, the sorted() function takes in an optional key argument (a function) that describes how the items in a list should be sorted.

unsorted = [('b', 6), ('a', 10), ('d', 0), ('c', 4)]

# Sort on the second tuple value (the integer).
print(sorted(unsorted, key=lambda x: x[1]))
>> [('d', 0), ('c', 4), ('b', 6), ('a', 10)]

The Map Function

While the ability to pass in functions as arguments is not unique to Python, it is a recent development in programming languages. Functions that allow for this type of behavior are called** first-class functions**. Any language that contains first-class functions can be written in a functional style.

There are a set of important first-class functions that are commonly used within the functional paradigm. These functions take in a Python iterable, and, like sorted(), apply a function for each element in the list. Over the next few sections, we will examine each of these functions, but they all follow the general form of function_name(function_to_apply, iterable_of_elements).

The first function we’ll work with is the map() function. The map() function takes in an iterable (ie. list), and creates a new iterable object, a special map object. The new object has the first-class function applied to every element.

# Pseudocode for map.
def map(func, seq):
    # Return `Map` object with
    # the function applied to every
    # element.
    return Map(
        func(x)
        for x in seq
    )
		

Here’s how we could use map to add 10 or 20 to every element in a list:

values = [1, 2, 3, 4, 5]

# Note: We convert the returned map object to
# a list data structure.
add_10 = list(map(lambda x: x + 10, values))
add_20 = list(map(lambda x: x + 20, values))

print(add_10)
>> [11, 12, 13, 14, 15]

print(add_20)
>> [21, 22, 23, 24, 25]

Note that it’s important to cast the return value from map() as a list object. Using the returned map object is difficult to work with if you’re expecting it to function like a list. First, printing it does not show each of its items, and secondly, you can only iterate over it once.

The Filter Function

The second function we’ll work with is the filter() function. The filter() function takes in an iterable, creates a new iterable object (again, a special map object), and a first-class function that must return a bool value. The new map object is a filtered iterable of all elements that returned True.

# Pseudocode for filter.
def filter(evaluate, seq):
    # Return `Map` object with
    # the evaluate function applied to every
    # element.
    return Map(
        x for x in seq
        if evaluate(x) is True
    )
		

Here’s how we could filter odd or even values from a list:

values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Note: We convert the returned filter object to
# a list data structure.
even = list(filter(lambda x: x % 2 == 0, values))
odd = list(filter(lambda x: x % 2 == 1, values))

print(even)
>> [2, 4, 6, 8, 10]

print(odd)
>> [1, 3, 5, 7, 9]

The Reduce Function

The last function we’ll look at is the reduce() function from the functools package. The reduce() function takes in an iterable, and then reduces the iterable to a single value. Reduce is different from filter() and map(), because reduce() takes in a function that has two input values.

Here’s an example of how we can use reduce() to sum all elements in a list.

from functools import reduce

values = [1, 2, 3, 4]

summed = reduce(lambda a, b: a + b, values)
print(summed)
>> 10

An interesting note to make is that you do not have to operate on the second value in the lambda expression. For example, you can write a function that always returns the first value of an iterable:

from functools import reduce

values = [1, 2, 3, 4, 5]

# By convention, we add `_` as a placeholder for an input
# we do not use.
first_value = reduce(lambda a, _: a, values)
print(first_value)
>> 1

Rewriting with list comprehensions

Because we eventually convert to lists, we should rewrite the map() and filter() functions using list comprehension instead. This is the more pythonic way of writing them, as we are taking advantage of the Python syntax for making lists. Here’s how you could translate the previous examples of map() and filter() to list comprehensions:

values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Map.
add_10 = [x + 10 for x in values]
print(add_10)
>> [11, 12, 13, 14, 15, 16, 17, 18, 19, 20]

# Filter.
even = [x for x in values if x % 2 == 0]
print(even)
>> [2, 4, 6, 8, 10]

From the examples, you can see that we don’t need to add the lambda expressions. If you are looking to add map(), or filter() functions to your own code, this is usually the recommended way. However, in the next section, we’ll provide a case to still use the map() and filter() functions.

Writing Function Partials

Sometimes we want to use the behavior of a function, but decrease the number of arguments it takes. The purpose is to “save” one of the inputs, and create a new function that defaults the behavior using the saved input. Suppose we wanted to write a function that would always add 2 to any number:

def add_two(b):
    return 2 + b

print(add_two(4))
>> 6

Theadd_two function is similar to the general function, $f(a,b) = a + b$, only it defaults one of the arguments ($a = 2$). In Python, we can use the partial module from the functools package to set these argument defaults. The partial module takes in a function, and “freezes” any number of args (or kwargs), starting from the first argument, then returns a new function with the default inputs.

from functools import partialdef add(a, b):
    return a + b

add_two = partial(add, 2)
add_ten = partial(add, 10)

print(add_two(4))
>> 6
print(add_ten(4))
>> 14

Partials can take in any function, including ones from the standard library.

# A partial that grabs IP addresses using
# the `map` function from the standard library.
extract_ips = partial(
    map,
    lambda x: x.split(' ')[0]
)
lines = read('example_log.txt')
ip_addresses = list(extract_ip(lines))

Summary

In this post, we introduced the paradigm of functional programming. We learned about the lambda expression in Python, important functional functions, and the concept of partials. Overall, we showed that Python provides a programmer with the tools to easily switch between functional programming and object-oriented programming.

Game Development with Python: Snake Game

Game Development with Python: Snake Game

Today, I’m taking you along for a journey in game development. We are making it with the classic game of Snake with Kivy, Python

Hello, there.

A lot of people want to start programming apps for Android, but they prefer not to use Android Studio and/or Java. Why? Because it's an overkill. "I just wanna create Snake and nothing more!"

Let's snake without java! (with a bonus at the end)

Get familiarized

First app

Please confirm that you have already installed Kivy (if not, follow the instructions) and ran buildozer init in the project directory.

Let's run our first app:

# main.py
from kivy.app import App
from kivy.uix.widget import Widget

class WormApp(App):
    def build(self):
        return Widget()

if __name__ == '__main__':
    WormApp().run()

_First run of the application_

We created a Widget. Analogously, we can create a button or any other UI element:

from kivy.app import App
from kivy.uix.widget import Widget
from kivy.uix.button import Button

class WormApp(App):
    def build(self):
        self.but = Button()
        self.but.pos = (100, 100)
        self.but.size = (200, 200)
        self.but.text = "Hello, cruel world"

        self.form = Widget()
        self.form.add_widget(self.but)
        return self.form

if __name__ == '__main__':
    WormApp().run()

_Creating a button_

Wow! Congratulations! You've created a button!

.kv files

However, there's another way to create UI elements. First, we implement our form:

from kivy.app import App
from kivy.uix.widget import Widget
from kivy.uix.button import Button

class Form(Widget):
    def __init__(self):
        super().__init__()
        self.but1 = Button()
        self.but1.pos = (100, 100)
        self.add_widget(self.but1)

class WormApp(App):
    def build(self):
        self.form = Form()
        return self.form

if __name__ == '__main__':
    WormApp().run()

Then, we create a «worm.kv» file.

# worm.kv
<Form>:
    but2: but_id

    Button:
        id: but_id
        pos: (200, 200)

What just happened? We created another Button and assigned id as but_id. Then, but_id was matched to but2 of the form. It means that now we can refer to this button by but2.

class Form(Widget):
    def __init__(self):
        super().__init__()
        self.but1 = Button()
        self.but1.pos = (100, 100)
        self.add_widget(self.but1)   #
        self.but2.text = "OH MY"

_Creating a new button_

Graphics

What we do next is creating a graphical element. First, we implement it in worm.kv:

<Form>:

<Cell>:
    canvas:
        Rectangle:
            size: self.size
            pos: self.pos

We linked the rectangle's position to self.pos and its size to self.size. So now, those properties are available from Cell, for example, once we create a cell, we can do:


class Cell(Widget):
    def __init__(self, x, y, size):
        super().__init__()
        self.size = (size, size)   # As you can see, we can change self.size which is "size" property of a rectangle
        self.pos = (x, y)

class Form(Widget):
    def __init__(self):
        super().__init__()
        self.cell = Cell(100, 100, 30)
        self.add_widget(self.cell)

_Creating a cell_

Ok, we have created a cell.

Necessary Methods

Let's try to move it. To do that, we should add Form.update function and schedule it.

from kivy.app import App
from kivy.uix.widget import Widget
from kivy.clock import Clock

class Cell(Widget):
    def __init__(self, x, y, size):
        super().__init__()
        self.size = (size, size)
        self.pos = (x, y)

class Form(Widget):
    def __init__(self):
        super().__init__()
        self.cell = Cell(100, 100, 30)
        self.add_widget(self.cell)

    def start(self):
        Clock.schedule_interval(self.update, 0.01)

    def update(self, _):
        self.cell.pos = (self.cell.pos[0] + 2, self.cell.pos[1] + 3)

class WormApp(App):
    def build(self):
        self.form = Form()
        self.form.start()
        return self.form

if __name__ == '__main__':
    WormApp().run()

The cell will move across the form. As you can see, we can schedule any function with Clock.

Next, let's make a touch event. Rewrite Form:

class Form(Widget):
    def __init__(self):
        super().__init__()
        self.cells = []

    def start(self):
        Clock.schedule_interval(self.update, 0.01)

    def update(self, _):
        for cell in self.cells:
            cell.pos = (cell.pos[0] + 2, cell.pos[1] + 3)

    def on_touch_down(self, touch):
        cell = Cell(touch.x, touch.y, 30)
        self.add_widget(cell)
        self.cells.append(cell)

Each touch_down creates a cell with coordinates = (touch.x, touch.y) and size of 30. Then, we add it as a widget of the form AND to our own array (in order to easily access them).

Now you can tap on your form and generate cells.

_Generating multiple cells_

Neat settings

Because we want to get a nice snake, we should distinguish the graphical positions and the actual positions of cells.

Why?

_A lot of reasons to do so. All logic should be connected with the so-called actual data, while the graphical data is the result of the actual data. For example, if we want to make margins, the actual pos of the cell will be (100, 100) while the graphical pos of the rectangle — (102, 102).

P. S. We wouldn't do it if we dealt with classical on_draw. But here, we don't have to program on_draw._

Let's fix the worm.kv file:

<Form>:

<Cell>:
    canvas:
        Rectangle:
            size: self.graphical_size
            pos: self.graphical_pos

and main.py:

...
from kivy.properties import *
...
class Cell(Widget):
    graphical_size = ListProperty([1, 1])
    graphical_pos = ListProperty([1, 1])

    def __init__(self, x, y, size, margin=4):
        super().__init__()
        self.actual_size = (size, size)
        self.graphical_size = (size - margin, size - margin)
        self.margin = margin
        self.actual_pos = (x, y)
        self.graphical_pos_attach()

    def graphical_pos_attach(self):
        self.graphical_pos = (self.actual_pos[0] - self.graphical_size[0] / 2, self.actual_pos[1] - self.graphical_size[1] / 2)
...
class Form(Widget):
    def __init__(self):
        super().__init__()
        self.cell1 = Cell(100, 100, 30)
        self.cell2 = Cell(130, 100, 30)
        self.add_widget(self.cell1)
        self.add_widget(self.cell2)
...

_Connecting cells_

The margin appeared, so it looks pretty although we created the second cell with X = 130 instead of 132. Later, we will make smooth motion based on the distance between actual_pos and graphical_pos.

Coding the Worm

Implementation

Init config in main.py

class Config:
    DEFAULT_LENGTH = 20
    CELL_SIZE = 25
    APPLE_SIZE = 35
    MARGIN = 4
    INTERVAL = 0.2
    DEAD_CELL = (1, 0, 0, 1)
    APPLE_COLOR = (1, 1, 0, 1)

(Trust me, you'll love it!)

Then, assign config to the app:

class WormApp(App):
    def __init__(self):
        super().__init__()
        self.config = Config()
        self.form = Form(self.config)

    def build(self):
        self.form.start()
        return self.form

Rewrite init and start:

class Form(Widget):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.worm = None

    def start(self):
        self.worm = Worm(self.config)
        self.add_widget(self.worm)
        Clock.schedule_interval(self.update, self.config.INTERVAL)

Then, the Cell:

class Cell(Widget):
    graphical_size = ListProperty([1, 1])
    graphical_pos = ListProperty([1, 1])

    def __init__(self, x, y, size, margin=4):
        super().__init__()
        self.actual_size = (size, size)
        self.graphical_size = (size - margin, size - margin)
        self.margin = margin
        self.actual_pos = (x, y)
        self.graphical_pos_attach()

    def graphical_pos_attach(self):
        self.graphical_pos = (self.actual_pos[0] - self.graphical_size[0] / 2, self.actual_pos[1] - self.graphical_size[1] / 2)

    def move_to(self, x, y):
        self.actual_pos = (x, y)
        self.graphical_pos_attach()

    def move_by(self, x, y, **kwargs):
        self.move_to(self.actual_pos[0] + x, self.actual_pos[1] + y, **kwargs)

    def get_pos(self):
        return self.actual_pos

    def step_by(self, direction, **kwargs):
        self.move_by(self.actual_size[0] * direction[0], self.actual_size[1] * direction[1], **kwargs)

Hopefully, it's more or less clear.

and finally the Worm:

class Worm(Widget):
    def __init__(self, config):
        super().__init__()
        self.cells = []
        self.config = config
        self.cell_size = config.CELL_SIZE
        self.head_init((100, 100))
        for i in range(config.DEFAULT_LENGTH):
            self.lengthen()

    def destroy(self):
        for i in range(len(self.cells)):
            self.remove_widget(self.cells[i])
        self.cells = []

    def lengthen(self, pos=None, direction=(0, 1)):
        # If pos is set, we put the cell in pos, otherwise accordingly to the specified direction
        if pos is None:
            px = self.cells[-1].get_pos()[0] + direction[0] * self.cell_size
            py = self.cells[-1].get_pos()[1] + direction[1] * self.cell_size
            pos = (px, py)
        self.cells.append(Cell(*pos, self.cell_size, margin=self.config.MARGIN))
        self.add_widget(self.cells[-1])

    def head_init(self, pos):
        self.lengthen(pos=pos)

Let's give life to our wormie.

_IT'S ALIVE!_

Motion

Now, we will make it move.

It's simple:

class Worm(Widget):
...
    def move(self, direction):
        for i in range(len(self.cells) - 1, 0, -1):
            self.cells[i].move_to(*self.cells[i - 1].get_pos())
        self.cells[0].step_by(direction)
class Form(Widget):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.worm = None
        self.cur_dir = (0, 0)

    def start(self):
        self.worm = Worm(self.config)
        self.add_widget(self.worm)
        self.cur_dir = (1, 0)
        Clock.schedule_interval(self.update, self.config.INTERVAL)

    def update(self, _):
        self.worm.move(self.cur_dir)

_He moving!_

It's alive! It's alive!

Controlling

As you could judge by the preview image, the controls of the snake will be the following:

class Form(Widget):
...
    def on_touch_down(self, touch):
        ws = touch.x / self.size[0]
        hs = touch.y / self.size[1]
        aws = 1 - ws
        if ws > hs and aws > hs:
            cur_dir = (0, -1)         # Down
        elif ws > hs >= aws:
            cur_dir = (1, 0)          # Right
        elif ws <= hs < aws:
            cur_dir = (-1, 0)         # Left
        else:
            cur_dir = (0, 1)           # Up
        self.cur_dir = cur_dir

_Even better!_

Cool.

Creating the Fruit

First, we initialize it.

class Form(Widget):
...
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.worm = None
        self.cur_dir = (0, 0)
        self.fruit = None
...
    def random_cell_location(self, offset):
        x_row = self.size[0] // self.config.CELL_SIZE
        x_col = self.size[1] // self.config.CELL_SIZE
        return random.randint(offset, x_row - offset), random.randint(offset, x_col - offset)

    def random_location(self, offset):
        x_row, x_col = self.random_cell_location(offset)
        return self.config.CELL_SIZE * x_row, self.config.CELL_SIZE * x_col

    def fruit_dislocate(self):
        x, y = self.random_location(2)
        self.fruit.move_to(x, y)
...
    def start(self):
        self.fruit = Cell(0, 0, self.config.APPLE_SIZE, self.config.MARGIN)
        self.worm = Worm(self.config)
        self.fruit_dislocate()
        self.add_widget(self.worm)
        self.add_widget(self.fruit)
        self.cur_dir = (1, 0)
        Clock.schedule_interval(self.update, self.config.INTERVAL)

The current result:

_Creating the fruit_

Now, we should implement some Worm methods:

class Worm(Widget):
...
    # Here we get all the positions of our cells
    def gather_positions(self):
        return [cell.get_pos() for cell in self.cells]
    # Just check if our head has the same position as another Cell
    def head_intersect(self, cell):
        return self.cells[0].get_pos() == cell.get_pos()

...and add this check to update().

class Form(Widget):
...
    def update(self, _):
        self.worm.move(self.cur_dir)
        if self.worm.head_intersect(self.fruit):
            directions = [(0, 1), (0, -1), (1, 0), (-1, 0)]
            self.worm.lengthen(direction=random.choice(directions))
            self.fruit_dislocate()

Detection of Self Tile Hitting

We want to know whether the head has the same position as one of the worm's cells.

class Form(Widget):
...
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.worm = None
        self.cur_dir = (0, 0)
        self.fruit = None
        self.game_on = True

    def update(self, _):
        if not self.game_on:
            return
        self.worm.move(self.cur_dir)
        if self.worm.head_intersect(self.fruit):
            directions = [(0, 1), (0, -1), (1, 0), (-1, 0)]
            self.worm.lengthen(direction=random.choice(directions))
            self.fruit_dislocate()
       if self.worm_bite_self():
            self.game_on = False

    def worm_bite_self(self):
        for cell in self.worm.cells[1:]:
            if self.worm.head_intersect(cell):
                return cell
        return False

_Losing game if snake runs into itself_

Coloring, Decorating, and Code Refactoring

Let's start with code refactoring.

Rewrite and add

class Form(Widget):
...
    def start(self):
        self.worm = Worm(self.config)
        self.add_widget(self.worm)
        if self.fruit is not None:
            self.remove_widget(self.fruit)
        self.fruit = Cell(0, 0, self.config.APPLE_SIZE)
        self.fruit_dislocate()
        self.add_widget(self.fruit)
        Clock.schedule_interval(self.update, self.config.INTERVAL)
        self.game_on = True
        self.cur_dir = (0, -1)

    def stop(self):
        self.game_on = False
        Clock.unschedule(self.update)

    def game_over(self):
        self.stop()
...
    def on_touch_down(self, touch):
        if not self.game_on:
            self.worm.destroy()
            self.start()
            return
        ...

Now, if the worm is dead (frozen), and you tap again, the game will be reset.
Now, let's go to decorating and coloring.

worm.kv

<Form>:
    popup_label: popup_label
    score_label: score_label

    canvas:
        Color:
            rgba: (.5, .5, .5, 1.0)

        Line:
            width: 1.5
            points: (0, 0), self.size

        Line:
            width: 1.5
            points: (self.size[0], 0), (0, self.size[1])

    Label:
        id: score_label
        text: "Score: " + str(self.parent.worm_len)
        width: self.width

    Label:
        id: popup_label
        width: self.width

<Worm>:

<Cell>:
    canvas:
        Color:
            rgba: self.color
        Rectangle:
            size: self.graphical_size
            pos: self.graphical_pos

Rewrite WormApp:

class WormApp(App):
    def build(self):
        self.config = Config()
        self.form = Form(self.config)
        return self.form

    def on_start(self):
        self.form.start()

_Adding a score_

Let's color it. Rewrite Cell in .kv:

<Cell>:
    canvas:
        Color:
            rgba: self.color

        Rectangle:
            size: self.graphical_size
            pos: self.graphical_pos

Add this to Cell.init

self.color = (0.2, 1.0, 0.2, 1.0)    # 

and this to Form.start

self.fruit.color = (1.0, 0.2, 0.2, 1.0)

Great, enjoy your snake

_The finished product!_

Finally, we will make a «game over» label

class Form(Widget):
...
    def __init__(self, config):
    ...
        self.popup_label.text = ""
...
    def stop(self, text=""):
        self.game_on = False
        self.popup_label.text = text
        Clock.unschedule(self.update)

    def game_over(self):
        self.stop("GAME OVER" + " " * 5 + "\ntap to reset")

and make the hit cell red:

instead of

    def update(self, _):
    ...
        if self.worm_bite_self():
            self.game_over()
    ...

write

    def update(self, _):
        cell = self.worm_bite_self()
        if cell:
            cell.color = (1.0, 0.2, 0.2, 1.0)
            self.game_over()

_GAME OVER_

Are you still paying attention? Coming next is the most interesting part.

Bonus Section — Smooth Motion

Because the worm's step is equal to the cell_size, it's not that smooth. But we want to make it step as frequently as possible, without rewriting the entire logic of the game. So, we need to create a mechanism moving our graphical poses but not our actual poses. So, I wrote a simple file:

smooth.py

from kivy.clock import Clock
import time

class Timing:
    @staticmethod
    def linear(x):
        return x

class Smooth:
    def __init__(self, interval=1.0/60.0):
        self.objs = []
        self.running = False
        self.interval = interval

    def run(self):
        if self.running:
            return
        self.running = True
        Clock.schedule_interval(self.update, self.interval)

    def stop(self):
        if not self.running:
            return
        self.running = False
        Clock.unschedule(self.update)

    def setattr(self, obj, attr, value):
        exec("obj." + attr + " = " + str(value))

    def getattr(self, obj, attr):
        return float(eval("obj." + attr))

    def update(self, _):
        cur_time = time.time()
        for line in self.objs:
            obj, prop_name_x, prop_name_y, from_x, from_y, to_x, to_y, start_time, period, timing = line
            time_gone = cur_time - start_time
            if time_gone >= period:
                self.setattr(obj, prop_name_x, to_x)
                self.setattr(obj, prop_name_y, to_y)
                self.objs.remove(line)
            else:
                share = time_gone / period
                acs = timing(share)
                self.setattr(obj, prop_name_x, from_x * (1 - acs) + to_x * acs)
                self.setattr(obj, prop_name_y, from_y * (1 - acs) + to_y * acs)
        if len(self.objs) == 0:
            self.stop()

    def move_to(self, obj, prop_name_x, prop_name_y, to_x, to_y, t, timing=Timing.linear):
        self.objs.append((obj, prop_name_x, prop_name_y, self.getattr(obj, prop_name_x), self.getattr(obj, prop_name_y), to_x,
                          to_y, time.time(), t, timing))
        self.run()

class XSmooth(Smooth):
    def __init__(self, props, timing=Timing.linear, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.props = props
        self.timing = timing

    def move_to(self, obj, to_x, to_y, t):
        super().move_to(obj, *self.props, to_x, to_y, t, timing=self.timing)

This module is not the most elegant solution ©. It's a bad solution and I acknowledge it. It is an only-hello-world solution.

So you just create smooth.py and copy-paste this code to the file.
Finally, let's make it work:

class Form(Widget):
...
    def __init__(self, config):
    ...
        self.smooth = smooth.XSmooth(["graphical_pos[0]", "graphical_pos[1]"])

Then, we replace self.worm.move() with

class Form(Widget):
...
    def update(self, _):
    ...
        self.worm.move(self.cur_dir, smooth_motion=(self.smooth, self.config.INTERVAL))

And this is how methods of Cell should look like:

class Cell(Widget):
...
    def graphical_pos_attach(self, smooth_motion=None):
        to_x, to_y = self.actual_pos[0] - self.graphical_size[0] / 2, self.actual_pos[1] - self.graphical_size[1] / 2
        if smooth_motion is None:
            self.graphical_pos = to_x, to_y
        else:
            smoother, t = smooth_motion
            smoother.move_to(self, to_x, to_y, t)

    def move_to(self, x, y, **kwargs):
        self.actual_pos = (x, y)
        self.graphical_pos_attach(**kwargs)

    def move_by(self, x, y, **kwargs):
        self.move_to(self.actual_pos[0] + x, self.actual_pos[1] + y, **kwargs)

That's it, thank you for your attention!

How the final result works, My final code

Thank for reading ! Originally published on dzone.com

Learn more

Beginning Game Development with Python

Python Online Multiplayer Game Development Tutorial

Buggy Python Code: The 10 Most Common Mistakes That Python Developers Make

Buggy Python Code: The 10 Most Common Mistakes That Python Developers Make

Buggy Python Code: The 10 Most Common Mistakes That Python Developers Make - Python's simple, easy-to-learn syntax can mislead Python developers — especially those who are newer to the language...

Buggy Python Code: The 10 Most Common Mistakes That Python Developers Make - Python's simple, easy-to-learn syntax can mislead Python developers — especially those who are newer to the language...

About Python

Python is an interpreted, object-oriented, high-level programming language with dynamic semantics. Its high-level built in data structures, combined with dynamic typing and dynamic binding, make it very attractive for Rapid Application Development, as well as for use as a scripting or glue language to connect existing components or services. Python supports modules and packages, thereby encouraging program modularity and code reuse.

About this article

Python’s simple, easy-to-learn syntax can mislead Python developers – especially those who are newer to the language – into missing some of its subtleties and underestimating the power of the diverse Python language.

With that in mind, this article presents a “top 10” list of somewhat subtle, harder-to-catch mistakes that can bite even some more advanced Python developers in the rear.

(Note: This article is intended for a more advanced audience than Common Mistakes of Python Programmers, which is geared more toward those who are newer to the language.)

Common Mistake #1: Misusing expressions as defaults for function arguments

Python allows you to specify that a function argument is optional by providing a default value for it. While this is a great feature of the language, it can lead to some confusion when the default value is mutable. For example, consider this Python function definition:

>>> def foo(bar=[]):        # bar is optional and defaults to [] if not specified
...    bar.append("baz")    # but this line could be problematic, as we'll see...
...    return bar

A common mistake is to think that the optional argument will be set to the specified default expression each time the function is called without supplying a value for the optional argument. In the above code, for example, one might expect that calling foo() repeatedly (i.e., without specifying a bar argument) would always return 'baz', since the assumption would be that each time foo() is called (without a bar argument specified) bar is set to [] (i.e., a new empty list).

But let’s look at what actually happens when you do this:
>>> foo()
["baz"]
>>> foo()
["baz", "baz"]
>>> foo()
["baz", "baz", "baz"]

Huh? Why did it keep appending the default value of "baz" to an existing list each time foo() was called, rather than creating a new list each time?

The more advanced Python programming answer is that the default value for a function argument is only evaluated once, at the time that the function is defined. Thus, the bar argument is initialized to its default (i.e., an empty list) only when foo() is first defined, but then calls to foo() (i.e., without a bar argument specified) will continue to use the same list to which bar was originally initialized.

FYI, a common workaround for this is as follows:

>>> def foo(bar=None):
...    if bar is None:		# or if not bar:
...        bar = []
...    bar.append("baz")
...    return bar
...
>>> foo()
["baz"]
>>> foo()
["baz"]
>>> foo()
["baz"]

Common Mistake #2: Using class variables incorrectly

Consider the following example:

>>> class A(object):
...     x = 1
...
>>> class B(A):
...     pass
...
>>> class C(A):
...     pass
...
>>> print A.x, B.x, C.x
1 1 1

Makes sense.

>>> B.x = 2
>>> print A.x, B.x, C.x
1 2 1

Yup, again as expected.

>>> A.x = 3
>>> print A.x, B.x, C.x
3 2 3

What the $%#!&?? We only changed A.x. Why did C.x change too?

In Python, class variables are internally handled as dictionaries and follow what is often referred to as Method Resolution Order (MRO). So in the above code, since the attribute x is not found in class C, it will be looked up in its base classes (only A in the above example, although Python supports multiple inheritances). In other words, C doesn’t have its own x property, independent of A. Thus, references to C.x are in fact references to A.x. This causes a Python problem unless it’s handled properly. Learn more about class attributes in Python.

Common Mistake #3: Specifying parameters incorrectly for an exception block

Suppose you have the following code:

>>> try:
...     l = ["a", "b"]
...     int(l[2])
... except ValueError, IndexError:  # To catch both exceptions, right?
...     pass
...
Traceback (most recent call last):
  File "", line 3, in 
IndexError: list index out of range

The problem here is that the except statement does not take a list of exceptions specified in this manner. Rather, In Python 2.x, the syntax except Exception, e is used to bind the exception to the optional second parameter specified (in this case e), in order to make it available for further inspection. As a result, in the above code, the IndexError exception is not being caught by the except statement; rather, the exception instead ends up being bound to a parameter named IndexError.

The proper way to catch multiple exceptions in an except statement is to specify the first parameter as a tuple containing all exceptions to be caught. Also, for maximum portability, use the as keyword, since that syntax is supported by both Python 2 and Python 3:

>>> try:
...     l = ["a", "b"]
...     int(l[2])
... except (ValueError, IndexError) as e:  
...     pass
...
>>>

Common Mistake #4: Misunderstanding Python scope rules

Python scope resolution is based on what is known as the LEGB rule, which is shorthand for Local, Enclosing, Global, Built-in. Seems straightforward enough, right? Well, actually, there are some subtleties to the way this works in Python, which brings us to the common more advanced Python programming problem below. Consider the following:

>>> x = 10
>>> def foo():
...     x += 1
...     print x
...
>>> foo()
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in foo
UnboundLocalError: local variable 'x' referenced before assignment

What’s the problem?

The above error occurs because, when you make an assignment to a variable in a scope, that variable is automatically considered by Python to be local to that scope and shadows any similarly named variable in any outer scope.

Many are thereby surprised to get an UnboundLocalError in previously working code when it is modified by adding an assignment statement somewhere in the body of a function. (You can read more about this here.)

It is particularly common for this to trip up developers when using lists. Consider the following example:

>>> lst = [1, 2, 3]
>>> def foo1():
...     lst.append(5)   # This works ok...
...
>>> foo1()
>>> lst
[1, 2, 3, 5]

>>> lst = [1, 2, 3]
>>> def foo2():
...     lst += [5]      # ... but this bombs!
...
>>> foo2()
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in foo
UnboundLocalError: local variable 'lst' referenced before assignment

Huh? Why did foo2 bomb while foo1 ran fine?

The answer is the same as in the prior example problem but is admittedly more subtle. foo1 is not making an assignment to lst, whereas foo2 is. Remembering that lst += [5] is really just shorthand for lst = lst + [5], we see that we are attempting to assign a value to lst (therefore presumed by Python to be in the local scope). However, the value we are looking to assign to lstis based on lst itself (again, now presumed to be in the local scope), which has not yet been defined. Boom.

Common Mistake #5: Modifying a list while iterating over it

The problem with the following code should be fairly obvious:

>>> odd = lambda x : bool(x % 2)
>>> numbers = [n for n in range(10)]
>>> for i in range(len(numbers)):
...     if odd(numbers[i]):
...         del numbers[i]  # BAD: Deleting item from a list while iterating over it
...
Traceback (most recent call last):
  	  File "", line 2, in 
IndexError: list index out of range

Deleting an item from a list or array while iterating over it is a Python problem that is well known to any experienced software developer. But while the example above may be fairly obvious, even advanced developers can be unintentionally bitten by this in code that is much more complex.

Fortunately, Python incorporates a number of elegant programming paradigms which, when used properly, can result in significantly simplified and streamlined code. A side benefit of this is that simpler code is less likely to be bitten by the accidental-deletion-of-a-list-item-while-iterating-over-it bug. One such paradigm is that of list comprehensions. Moreover, list comprehensions are particularly useful for avoiding this specific problem, as shown by this alternate implementation of the above code which works perfectly:

>>> odd = lambda x : bool(x % 2)
>>> numbers = [n for n in range(10)]
>>> numbers[:] = [n for n in numbers if not odd(n)]  # ahh, the beauty of it all
>>> numbers
[0, 2, 4, 6, 8]

Common Mistake #6: Confusing how Python binds variables in closures

Considering the following example:

>>> def create_multipliers():
...     return [lambda x : i * x for i in range(5)]
>>> for multiplier in create_multipliers():
...     print multiplier(2)
...

You might expect the following output:

0
2
4
6
8

But you actually get:

8
8
8
8
8

Surprise!

This happens due to Python’s late binding behavior which says that the values of variables used in closures are looked up at the time the inner function is called. So in the above code, whenever any of the returned functions are called, the value of i is looked up in the surrounding scope at the time it is called (and by then, the loop has completed, so i has already been assigned its final value of 4).

The solution to this common Python problem is a bit of a hack:

>>> def create_multipliers():
...     return [lambda x, i=i : i * x for i in range(5)]
...
>>> for multiplier in create_multipliers():
...     print multiplier(2)
...
0
2
4
6
8

Voilà! We are taking advantage of default arguments here to generate anonymous functions in order to achieve the desired behavior. Some would call this elegant. Some would call it subtle. Some hate it. But if you’re a Python developer, it’s important to understand in any case.

Common Mistake #7: Creating circular module dependencies

Let’s say you have two files, a.py and b.py, each of which imports the other, as follows:

In a.py:

import b

def f():
    return b.x
	
print f()

And in b.py:

import a

x = 1

def g():
    print a.f()

First, let’s try importing a.py:

>>> import a
1

Worked just fine. Perhaps that surprises you. After all, we do have a circular import here which presumably should be a problem, shouldn’t it?

The answer is that the mere presence of a circular import is not in and of itself a problem in Python. If a module has already been imported, Python is smart enough not to try to re-import it. However, depending on the point at which each module is attempting to access functions or variables defined in the other, you may indeed run into problems.

So returning to our example, when we imported a.py, it had no problem importing b.py, since b.py does not require anything from a.py to be defined at the time it is imported. The only reference in b.py to a is the call to a.f(). But that call is in g() and nothing in a.py or b.pyinvokes g(). So life is good.

But what happens if we attempt to import b.py (without having previously imported a.py, that is):

>>> import b
Traceback (most recent call last):
  	  File "", line 1, in 
  	  File "b.py", line 1, in 
    import a
  	  File "a.py", line 6, in 
	print f()
  	  File "a.py", line 4, in f
	return b.x
AttributeError: 'module' object has no attribute 'x'

Uh-oh. That’s not good! The problem here is that, in the process of importing b.py, it attempts to import a.py, which in turn calls f(), which attempts to access b.x. But b.x has not yet been defined. Hence the AttributeError exception.

At least one solution to this is quite trivial. Simply modify b.py to import a.py within g():

x = 1

def g():
    import a	# This will be evaluated only when g() is called
    print a.f()

No when we import it, everything is fine:

>>> import b
>>> b.g()
1	# Printed a first time since module 'a' calls 'print f()' at the end
1	# Printed a second time, this one is our call to 'g'

Common Mistake #8: Name clashing with Python Standard Library modules

One of the beauties of Python is the wealth of library modules that it comes with “out of the box”. But as a result, if you’re not consciously avoiding it, it’s not that difficult to run into a name clash between the name of one of your modules and a module with the same name in the standard library that ships with Python (for example, you might have a module named email.py in your code, which would be in conflict with the standard library module of the same name).

This can lead to gnarly problems, such as importing another library which in turns tries to import the Python Standard Library version of a module but, since you have a module with the same name, the other package mistakenly imports your version instead of the one within the Python Standard Library. This is where bad Python errors happen.

Care should, therefore, be exercised to avoid using the same names as those in the Python Standard Library modules. It’s way easier for you to change the name of a module within your package than it is to file a Python Enhancement Proposal (PEP) to request a name change upstream and to try and get that approved.

Common Mistake #9: Failing to address differences between Python 2 and Python 3

Consider the following file foo.py:

import sys

def bar(i):
    if i == 1:
        raise KeyError(1)
    if i == 2:
        raise ValueError(2)

def bad():
    e = None
    try:
        bar(int(sys.argv[1]))
    except KeyError as e:
        print('key error')
    except ValueError as e:
        print('value error')
    print(e)

bad()

On Python 2, this runs fine:

$ python foo.py 1
key error
1
$ python foo.py 2
value error
2

But now let’s give it a whirl on Python 3:

$ python3 foo.py 1
key error
Traceback (most recent call last):
  File "foo.py", line 19, in 
    bad()
  File "foo.py", line 17, in bad
    print(e)
UnboundLocalError: local variable 'e' referenced before assignment

What has just happened here? The “problem” is that, in Python 3, the exception object is not accessible beyond the scope of the except block. (The reason for this is that, otherwise, it would keep a reference cycle with the stack frame in memory until the garbage collector runs and purges the references from memory. More technical detail about this is available here).

One way to avoid this issue is to maintain a reference to the exception object outside the scope of the except block so that it remains accessible. Here’s a version of the previous example that uses this technique, thereby yielding code that is both Python 2 and Python 3 friendly:

import sys

def bar(i):
    if i == 1:
        raise KeyError(1)
    if i == 2:
        raise ValueError(2)

def good():
    exception = None
    try:
        bar(int(sys.argv[1]))
    except KeyError as e:
        exception = e
        print('key error')
    except ValueError as e:
        exception = e
        print('value error')
    print(exception)

good()

Running this on Py3k:

$ python3 foo.py 1
key error
1
$ python3 foo.py 2
value error
2

Yippee!

(Incidentally, our Python Hiring Guide discusses a number of other important differences to be aware of when migrating code from Python 2 to Python 3.)

Common Mistake #10: Misusing the __del__ method

Let’s say you had this in a file called mod.py:

import foo

class Bar(object):
   	    ...
    def __del__(self):
        foo.cleanup(self.myhandle)

And you then tried to do this from another_mod.py:

import mod
mybar = mod.Bar()

You’d get an ugly AttributeError exception.

Why? Because, as reported here, when the interpreter shuts down, the module’s global variables are all set to None. As a result, in the above example, at the point that [__del__](https://docs.python.org/2/reference/datamodel.html#object.__del__ "__del__") is invoked, the name foo has already been set to None.

A solution to this somewhat more advanced Python programming problem would be to use [atexit.register()](https://docs.python.org/2/library/atexit.html "atexit.register()") instead. That way, when your program is finished executing (when exiting normally, that is), your registered handlers are kicked off before the interpreter is shut down.

With that understanding, a fix for the above mod.py code might then look something like this:

import foo
import atexit

def cleanup(handle):
    foo.cleanup(handle)


class Bar(object):
    def __init__(self):
        ...
        atexit.register(cleanup, self.myhandle)

This implementation provides a clean and reliable way of calling any needed cleanup functionality upon normal program termination. Obviously, it’s up to foo.cleanup to decide what to do with the object bound to the name self.myhandle, but you get the idea.

Wrap-up

Python is a powerful and flexible language with many mechanisms and paradigms that can greatly improve productivity. As with any software tool or language, though, having a limited understanding or appreciation of its capabilities can sometimes be more of an impediment than a benefit, leaving one in the proverbial state of “knowing enough to be dangerous”.

Familiarizing oneself with the key nuances of Python, such as (but by no means limited to) the moderately advanced programming problems raised in this article, will help optimize use of the language while avoiding some of its more common errors.

You might also want to check out our Insider’s Guide to Python Interviewing for suggestions on interview questions that can help identify Python experts.

We hope you’ve found the pointers in this article helpful and welcome your feedback.