Getting Started with Python working on Google Cloud Functions

Getting Started with Python working on Google Cloud Functions

Writing Python cloud functions is easy and fun, as I will now demonstrate. To follow this tutorial you’ll need a Google Cloud Project, a local Python development environment and the gcloud SDK.

I’ve been a fan of Cloud Functions for a while and a fan of Python since forever, so when support for Python functions was finally announced at Google NEXT’18 I did a small dance of celebration (or at least, I fist-bumped the air when no one was looking). I’m sure you know what Python is, but if you’re new to the serverless world we’re now living in, a quick re-cap on Cloud Functions:

Cloud Functions are small pieces of code that execute in an event-driven manner. They do one small thing very efficiently in reaction to a trigger — usually an HTTP request. The neat thing is you manage zero infrastructure and usually only pay for the execution of your function and a few seconds of compute. You may have heard of other Functions-as-a-Service offerings such as AWS Lambda.

Writing Python cloud functions is easy and fun, as I will now demonstrate. To follow this tutorial you’ll need a Google Cloud Project (you can sign up and get free credits here), a local Python development environment and the gcloud SDK.

Hello, World

We can create a simple function by creating a main.py with just 2 lines:

def hello_world(request):
    return 'Hello, World!\n'

You can deploy this as a cloud Function with the following command. Note that the cloud function name matches the name of the function we defined in code: hello_world

gcloud beta functions deploy hello_world --runtime python37 --trigger-http

After a couple of minutes, the command will return the httpsTrigger or URL for your function. Try accessing this URL and you’ll see the message “Hello, World!” returned. You can also see your function in the Cloud Console UI:

But this is pretty boring stuff, let’s make the function do something a bit more interactive:

def hello_world(request):
    request_json = request.get_json()
    if request_json and 'name' in request_json:
        name = request_json['name']
    else:
        name = 'World'
    return 'Hello, {}!\n'.format(name)

You can use the same gcloud beta functions deploy command as before to update your function. After a couple of minutes, try accessing the URL again and you’ll get the same message. But now try sending it a POST with some data:

curl -X POST <your function URL> -H "Content-Type:application/json" -d '{"name": "Derek"}'

And you should receive a custom message in return! The runtime for Cloud Functions uses the Flask library and provides us with a Request object that we can manipulate however we like.

A More Useful Example

This is all great fun, but now let’s do something useful to show a real world example of where you might employ a Cloud Function. As we’ve seen, Functions are typically triggered by HTTPS endpoints, however you can also make use of Background triggers. Functions deployed this way have no HTTPS endpoint, and are effectively hidden from the Internet, but will instead execute in response to an event such as a file being uploaded to a storage bucket, or a message being sent to a Pub/Sub topic.

Let’s say we have a web service that writes images to a Cloud Storage Bucket. Every time an image is uploaded, we’d like to create a corresponding thumbnail image. Arguably this could be part of our main application code, but for the sake of argument let’s say we want to run this as a separate function, because there may be other ways that images end up in the bucket (maybe an external service sends them to us in batches).

Let’s write a function that reacts when a new file is written to a Cloud Storage bucket. You’ll need 2 GCS buckets for this: You’ll upload images into the first one, and your function will write thumbnails into the second one. In a new directory, create a new main.py file:

from wand.image import Image
from google.cloud import storage

client = storage.Client()

THUMBNAIL_BUCKET = '<your thumbnail bucket>'

def make_thumbnail(data, context):
    # Get the file that has been uploaded to GCS
    bucket = client.get_bucket(data['bucket'])
    blob = bucket.get_blob(data['name'])
    imagedata = blob.download_as_string()
    # Create a new image object and resample it
    newimage = Image(blob=imagedata)
    newimage.sample(200,200)
    # Upload the resampled image to the other bucket
    bucket = client.get_bucket(THUMBNAIL_BUCKET)
    newblob = bucket.blob('thumbnail-' + data['name'])     
    newblob.upload_from_string(newimage.make_blob())

Notice that we’re importing modules, just like we would do with any other Python application. Cloud Functions uses pip, which means we can specify the dependencies we need in a requirements.txt file:

google-cloud-storage
Wand

Just keep your requirements.txt file in the same location as your main.py when you deploy your function.

Don’t forget to replace in the above code with the name of your own thumbnail bucket, and with your source bucket name in the following example. You can now deploy the cloud function like this:

gcloud beta functions deploy make_thumbnail --runtime python37 --trigger-resource <your source bucket>  --trigger-event google.storage.object.finalize

Note: You can’t use the same bucket for both. Why? Because every time the function writes a thumbnail, it would trigger a new invocation of itself!

When you deploy this function you’ll notice we don’t get given a URL. Instead we’ve told the function to run when the google.storage.object.finalize event occurs in the bucket we specified. Let’s upload a picture of Derek the Dog to the source bucket:

Check the thumbnails bucket, and you should see that the function has done its job and created a thumbnail for us!

You can inspect the magic under the hood by revisiting the Cloud Functions part of Cloud Console, where you can track the invocations of your function and even watch its logs in real-time.

Cloud Functions are perfect when you need small pieces of code to tie together larger pieces of a stack. Now that Google has added Python support, we have a huge and diverse ecosystem of functionality available to us to write functions. For more information, read about the Python runtime or follow all of Google’s How To guides.

Thanks for reading!

How to Create Python Development Environment In Google Cloud Platform

How to Create Python Development Environment In Google Cloud Platform

In this post, you will learn how to create python application development environment in google cloud platform

In this post, you will learn how to create python application development environment in google cloud platform

Set up a Python development environment on Google Cloud Platform, using Google Compute Engine to create a virtual machine (VM) and installing software libraries for software development.

You perform the following tasks,

  • Provision a Google Compute Engine instance.
  • Connect to the instance using SSH.
  • Install a Python library on the instance.
  • Verify the software installation.

Prerequisites

  • Google cloud account

Follow the below steps to create a Python Application development environment in Google Cloud Platform.

Step 1

Login https://console.cloud.google.com/.

Step 2

Create and connect to a virtual machine,

  1. In the Console, click Navigation menu > Compute Engine > VM Instances.

  1. On the VM Instances page, click Create.

  2. On the Create an instance page, for Name type dev-instance, and select a Region as us-central1 (Iowa) and Zone as us-central1-a.

  3. In the Identity and API access section, select Allow full access to all Cloud APIs.

  4. In the Firewall section, enable Allow HTTP traffic.

  5. Leave the remaining settings as their defaults, and click Create**.**

It takes about 20 seconds for the VM to be provisioned and started

On the VM instances page, in the dev-instance row, click SSH.

This launches a browser-hosted SSH session. If you have a popup blocker, you may need to click twice.

There's no need to configure or manage SSH keys.

Step 3

Install software on the VM instance

In the SSH session, to update the Debian package list, execute the following command,

sudo apt-get update

To install Git, execute the following command,

sudo apt-get install git

When prompted, enter Y to continue, accepting the use of additional disk space.

To install Python, execute the following command,

sudo apt-get install python-setuptools python-dev build-essential

Again, when prompted, enter Y to continue, accepting the use of additional disk space.

To install pip, execute the following command,

_sudo easy_install pip

Step 4

Configure the VM to Run Application Software.

In this section, you verify the software installation on your VM and run some sample code.

Verify Python installation

Still in the SSH window, verify the installation by checking the Python and pip version,

python --version
pip --version

The output provides the version of Python and pip that you installed.

Clone the class repository,

git clone

Change the working directory,

cd ~/training-data-analyst/courses/developingapps/python/devenv/

Run a simple web server,

sudo python server.py

Step 5

Return to the Cloud Console VM instances list (Navigation menu > Compute Engine > Virtual Instances), and click on the External IP address for the dev-instance.

A browser opens and displays a Hello GCP dev! message from Python.

Step 6

  1. Return to the SSH window, and stop the application by pressing Ctrl+c.

  2. Install the Python packages needed to enumerate Google Compute Engine VM instances,

sudo pip install -r requirements.txt

Step 7

Now list your instance in Cloud Shell. Enter the following command to run a simple Python application that lists Compute Engine instances. Replace with your GCP Project ID and is the region you specified when you created your VM. Find these values on the VM instances page of the console,

python list-gce-instances.py <PROJECT_ID> --zone=<YOUR_VM_ZONE>

Your instance name should appear in the SSH terminal window.

Output

Summary

I hope you understood how to create a Python Application development environment in Google Cloud Platform. Stay tuned for more GCP environment articles.

Running Serverless Node.js on Google Cloud Functions

Running Serverless Node.js on Google Cloud Functions

Running Serverless Node.js on Google Cloud Functions

Google Cloud Functions is a service that allows you to run your code in the cloud. You don't need to manage server, just code for the application logic (serverless). It's also highly available and fault tolerant, as well as very automatically scales. There are some ways to trigger function execution, some of which are the result of integration with other Google Cloud services. In this tutorial, I'm going to show how to run serverless Node.js in Google Cloud Functions.

Preparation
  1. Create or select a Google Cloud project

A Google Cloud project is required to use this service. Open Google Cloud console, then create a new project or select existing project

  1. Enable billing for the project

Like other cloud platforms, Google requires you to enable billing for your project. If you haven't set up billing, open billing page.

  1. Enable Cloud Functions API

To use an API, you must enable it first. Open this page to enable Cloud Functions API.

  1. Set up service account for authentication

You need to install gcloud on your computer and get authenticated to access your account.

Writing a Function

First, create a folder for this project. Inside the folder, create a file with the following content.

index.js

  exports.helloWorld = (req, res) => res.send("Hello, World!");

Running Google Cloud Functions Emulator

Before running functions on the real Google Cloud servers, we can test whether the code works as expected by running Google Cloud Functions Emulator on your local machine.

  • First, you need to install the emulator using npm install -g @google-cloud/functions-emulator
  • Run it using functions start.
  • To deploy the helloWorld function we have created on the previous step, run functions deploy helloWorld --trigger-http
  • If the function successfully deployed, you will see a resource URL - open it to call the function
Deploying to Google Cloud Functions

Run the following command in order to deploy the functionn to Google Cloud Functions.

gcloud functions deploy helloAsync --trigger-http --project PROJECT_NAME

Below is the list of supported flags while running gcloud functions deploy command.

gcloud functions deploy (NAME : --region=REGION)
	        [--entry-point=ENTRY_POINT] [--memory=MEMORY] [--retry]
	        [--runtime=RUNTIME] [--source=SOURCE] [--stage-bucket=STAGE_BUCKET]
	        [--timeout=TIMEOUT] [--update-labels=[KEY=VALUE,...]]
	        [--clear-env-vars | --env-vars-file=FILE_PATH
	          | --set-env-vars=[KEY=VALUE,...]
	          | --remove-env-vars=[KEY,...] --update-env-vars=[KEY=VALUE,...]]
	        [--clear-labels | --remove-labels=[KEY,...]]
	        [--trigger-bucket=TRIGGER_BUCKET | --trigger-http
	          | --trigger-topic=TRIGGER_TOPIC
	          | --trigger-event=EVENT_TYPE --trigger-resource=RESOURCE]
	        [GCLOUD_WIDE_FLAG ...]

NAME is the ID of the function or fully qualified identifier for the function. Description for each flags is on the table below

Installing Dependencies

To install a module, add it to the dependencies of package.json. Google will automatically install all dependencies at the deployment.

Alternatively you can run npm install MODULE_NAME, which will update package.json content.

Using Node.js 8 Environment

Currently Google still uses Node.js 6 as the default version. With that version, you can't use some new features such as async/await. But Google makes it possible for use to use Node.js 8. First, add a function that won't run on Node.js 6, as examplified below.

index.js

 const request = require('request-promise');
 
  exports.helloAsync = async (req, res) => {
    const result = await request('https://www.example.com');
 
    return res.send(result);
  };

To use Node.js 8 runtime environment, add --runtime nodejs8 parameter to the command.

gcloud functions deploy helloAsync --trigger-http --project PROJECT_NAME --runtime nodejs8

Python Tutorial | Python Functions and Functional Programming

Python Tutorial | Python Functions and Functional Programming

In this post, we will introduce a functional programming model. We learn about lambda expressions in Python, important function functions and the concept of particles.

Most of us have been introduced to Python as an object-oriented language, but Python functions are also useful tools for data scientists and programmers alike. While classes, and objects, are easy to start working with, there are other ways to write your Python code. Languages like Java can make it hard to move away from object-oriented thinking, but Python makes it easy.

Given that Python facilitates different approaches to writing code, a logical follow-up question is: what is a different way to write code? While there are several answers to this question, the most common alternative style of writing code is called functional programming. Functional programming gets its name from writing functions which provides the main source of logic in a program.

In this post, we will:

  • Explain the basics of functional programming by comparing it to object-oriented programming.

  • Cover why you might want to incorporate functional programming in your own code.

  • Show you how Python allows you to switch between the two.

Comparing object-oriented to functional

The easiest way to introduce functional programming is to compare it to something we’re already aware of: object-oriented programming. Suppose we wanted to create a line counter class that takes in a file, reads each line, then counts the total amount of lines in the file. Using a class, it could look something like the following:

class LineCounter:
    def __init__(self, filename):
        self.file = open(filename, 'r')
        self.lines = []

        def read(self):
            self.lines = [line for line in self.file]

        def count(self):
            return len(self.lines)

While not the best implementation, it does provide an insight into object-oriented design. Within the class, there are the familiar concepts of methods and properties. The properties set and retrieve the state of the object, and the methods manipulate that state.

For both these concepts to work, the object’s state must change over time. This change of state is evident in the lines property after calling the read() method. As an example, here’s how we would use this class:

# example_file.txt contains 100 lines.
lc = LineCounter('example_file.txt')
print(lc.lines)
>> []
print(lc.count())
>> 0

# The lc object must read the file to
# set the lines property.
lc.read()
# The `lc.lines` property has been changed.
# This is called changing the state of the lc
# object.
print(lc.lines)
>> [['Hello world!', ...]]
print(lc.count())
>> 100

The ever-changing state of an object is both its blessing and curse. To understand why a changing state can be seen as a negative, we have to introduce an alternative. The alternative is to build the line counter as a series of independent functions.

def read(filename):
    with open(filename, 'r') as f:
        return [line for line in f]

def count(lines):
    return len(lines)

example_lines = read('example_log.txt')
lines_count = count(example_lines)

Working with pure functions

In the previous example, we were able to count the lines only with the use of functions. When we only use functions, we are applying a functional approach to programming which is, non-excitingly, called functional programming. The concepts behind functional programming requires functions to be stateless, and rely only on their given inputs to produce an output.

The functions that meet the above criteria are called pure functions. Here’s an example to highlight the difference between pure functions, and non-pure:

# Create a global variable `A`.
A = 5

def impure_sum(b):
    # Adds two numbers, but uses the
    # global `A` variable.
    return b + A

def pure_sum(a, b):
    # Adds two numbers, using
    # ONLY the local function inputs.
    return a + b

print(impure_sum(6))
>> 11

print(pure_sum(4, 6))
>> 10

The benefit of using pure functions over impure (non-pure) functions is the reduction of side effects. **Side effects **occur when there are changes performed within a function’s operation that are outside its scope. For example, they occur when we change the state of an object, perform any I/O operation, or even call print():

def read_and_print(filename):
    with open(filename) as f:
        # Side effect of opening a
        # file outside of function.
        data = [line for line in f]
    for line in data:
        # Call out to the operating system
        # "println" method (side effect).
        print(line)

Programmers reduce side effects in their code to make it easier to follow, test, and debug. The more side effects a codebase has, the harder it is to step through a program and understand its sequence of execution.

While it’s convienent to try and eliminate all side effects, they’re often used to make programming easier. If we were to ban all side effects, then you wouldn’t be able to read in a file, call print, or even assign a variable within a function. Advocates for functional programming understand this tradeoff, and try to eliminate side effects where possible without sacrificing development implementation time.

The Lambda Expression

Instead of the def syntax for function declaration, we can use a lambda expression to write Python functions. The lambda syntax closely follows the def syntax, but it’s not a 1-to-1 mapping. Here’s an example of building a function that adds two integers:

# Using `def` (old way).
def old_add(a, b):
    return a + b

# Using `lambda` (new way).
new_add = lambda a, b: a + bold_add(10, 5) == new_add(10, 5)
>> True

The lambda expression takes in a comma seperated sequences of inputs (like def). Then, immediately following the colon, it returns the expression without using an explicit return statement. Finally, when assigning thelambda expression to a variable, it acts exactly like a Python function, and can be called using the the function call syntax: new_add().

If we didn’t assign lambda to a variable name, it would be called an anonymous function. These** anonymous functions** are extremely helpful, especially when using them as an input for another function. For example, the sorted() function takes in an optional key argument (a function) that describes how the items in a list should be sorted.

unsorted = [('b', 6), ('a', 10), ('d', 0), ('c', 4)]

# Sort on the second tuple value (the integer).
print(sorted(unsorted, key=lambda x: x[1]))
>> [('d', 0), ('c', 4), ('b', 6), ('a', 10)]

The Map Function

While the ability to pass in functions as arguments is not unique to Python, it is a recent development in programming languages. Functions that allow for this type of behavior are called** first-class functions**. Any language that contains first-class functions can be written in a functional style.

There are a set of important first-class functions that are commonly used within the functional paradigm. These functions take in a Python iterable, and, like sorted(), apply a function for each element in the list. Over the next few sections, we will examine each of these functions, but they all follow the general form of function_name(function_to_apply, iterable_of_elements).

The first function we’ll work with is the map() function. The map() function takes in an iterable (ie. list), and creates a new iterable object, a special map object. The new object has the first-class function applied to every element.

# Pseudocode for map.
def map(func, seq):
    # Return `Map` object with
    # the function applied to every
    # element.
    return Map(
        func(x)
        for x in seq
    )
		

Here’s how we could use map to add 10 or 20 to every element in a list:

values = [1, 2, 3, 4, 5]

# Note: We convert the returned map object to
# a list data structure.
add_10 = list(map(lambda x: x + 10, values))
add_20 = list(map(lambda x: x + 20, values))

print(add_10)
>> [11, 12, 13, 14, 15]

print(add_20)
>> [21, 22, 23, 24, 25]

Note that it’s important to cast the return value from map() as a list object. Using the returned map object is difficult to work with if you’re expecting it to function like a list. First, printing it does not show each of its items, and secondly, you can only iterate over it once.

The Filter Function

The second function we’ll work with is the filter() function. The filter() function takes in an iterable, creates a new iterable object (again, a special map object), and a first-class function that must return a bool value. The new map object is a filtered iterable of all elements that returned True.

# Pseudocode for filter.
def filter(evaluate, seq):
    # Return `Map` object with
    # the evaluate function applied to every
    # element.
    return Map(
        x for x in seq
        if evaluate(x) is True
    )
		

Here’s how we could filter odd or even values from a list:

values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Note: We convert the returned filter object to
# a list data structure.
even = list(filter(lambda x: x % 2 == 0, values))
odd = list(filter(lambda x: x % 2 == 1, values))

print(even)
>> [2, 4, 6, 8, 10]

print(odd)
>> [1, 3, 5, 7, 9]

The Reduce Function

The last function we’ll look at is the reduce() function from the functools package. The reduce() function takes in an iterable, and then reduces the iterable to a single value. Reduce is different from filter() and map(), because reduce() takes in a function that has two input values.

Here’s an example of how we can use reduce() to sum all elements in a list.

from functools import reduce

values = [1, 2, 3, 4]

summed = reduce(lambda a, b: a + b, values)
print(summed)
>> 10

An interesting note to make is that you do not have to operate on the second value in the lambda expression. For example, you can write a function that always returns the first value of an iterable:

from functools import reduce

values = [1, 2, 3, 4, 5]

# By convention, we add `_` as a placeholder for an input
# we do not use.
first_value = reduce(lambda a, _: a, values)
print(first_value)
>> 1

Rewriting with list comprehensions

Because we eventually convert to lists, we should rewrite the map() and filter() functions using list comprehension instead. This is the more pythonic way of writing them, as we are taking advantage of the Python syntax for making lists. Here’s how you could translate the previous examples of map() and filter() to list comprehensions:

values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Map.
add_10 = [x + 10 for x in values]
print(add_10)
>> [11, 12, 13, 14, 15, 16, 17, 18, 19, 20]

# Filter.
even = [x for x in values if x % 2 == 0]
print(even)
>> [2, 4, 6, 8, 10]

From the examples, you can see that we don’t need to add the lambda expressions. If you are looking to add map(), or filter() functions to your own code, this is usually the recommended way. However, in the next section, we’ll provide a case to still use the map() and filter() functions.

Writing Function Partials

Sometimes we want to use the behavior of a function, but decrease the number of arguments it takes. The purpose is to “save” one of the inputs, and create a new function that defaults the behavior using the saved input. Suppose we wanted to write a function that would always add 2 to any number:

def add_two(b):
    return 2 + b

print(add_two(4))
>> 6

Theadd_two function is similar to the general function, $f(a,b) = a + b$, only it defaults one of the arguments ($a = 2$). In Python, we can use the partial module from the functools package to set these argument defaults. The partial module takes in a function, and “freezes” any number of args (or kwargs), starting from the first argument, then returns a new function with the default inputs.

from functools import partialdef add(a, b):
    return a + b

add_two = partial(add, 2)
add_ten = partial(add, 10)

print(add_two(4))
>> 6
print(add_ten(4))
>> 14

Partials can take in any function, including ones from the standard library.

# A partial that grabs IP addresses using
# the `map` function from the standard library.
extract_ips = partial(
    map,
    lambda x: x.split(' ')[0]
)
lines = read('example_log.txt')
ip_addresses = list(extract_ip(lines))

Summary

In this post, we introduced the paradigm of functional programming. We learned about the lambda expression in Python, important functional functions, and the concept of partials. Overall, we showed that Python provides a programmer with the tools to easily switch between functional programming and object-oriented programming.