Demystifying the MQTT maze: clients, servers, connection, publish, subscribe and its applications.

Demystifying the MQTT maze: clients, servers, connection, publish,  subscribe and its applications.

I take you through MQTT protocol and the best MQTT broker to use for machine to machine communiation (M2M).

What is MQTT?


MQTT is an ISO standard publish - subscribe based message protocol. It stands for Message Queuing Telemetry Transport and it works on top of TPC/IP protocol. MQTT is designed for connection with remote locations. Furthermore, its small size, low power usage, minimized data packets and ease of implementation make it ideal for machine-to-machine communication.

How MQTT works


Before we dive deep into publishing message to MQTT broker using IoT development board ESP8266-12 in my next article, we must understand the publish-subscribe pattern known as the pub-sub pattern.

In publish-subscribe pattern, a client that publishes a message is decoupled from the other client or clients that receive the message. The clients don't know about the existence of the other clients. A client can publish messages of a specific type “Topic” and only the clients that are interested in specific topics of messages will receive the published messages.


 Publish-subscribe pattern requires a broker, also known as server.  The MQTT broker is the central hub of the publish/subscribe model we previously analyzed. The MQTT server is responsible of the authentication and authorization of the MQTT clients that will be able to become publishers and/or subscribers after they are authenticated and authorized. So, the first thing that an MQTT client must do is to establish a connection with the MQTT server.

All the clients establish a connection with the broker. The client that sends a message through the broker is known as the publisher. The broker filters the incoming messages and distributes them to the clients that are interested in the type of received messages. The clients that register to the broker as interested in specific types of messages are known as subscribers. Hence, both publishers and subscribers establish a connection with the broker.There are many MQTT servers available for the most popular platforms, including Linux, Windows and macOS like mosquitto http://www.mosquitto.org , HiveMQ http://hivemq.com and cloudMQTT http://www.cloudmqtt.com

Why MQTT?


MQTT was designed to be suitable to support the following typical challenges in IoT, M2M, embedded, and mobile applications:

  •       Be lightweight to make it possible to transmit high volumes of data without huge overheads.
  •       Distribute minimal packets of data in huge volumes
  •        Support an event-oriented paradigm with asynchronous bidirectional low latency push delivery of
  • messages.
  •       Easily emit data from one client to many clients
  •       Make it possible to listen for events whenever they happen (event-oriented architecture).
  •        Support always-connected and sometimes-connected models
  •       Publish information over unreliable networks and provide reliable deliveries over fragile
  • connections.
  •        Work very well with battery-powered devices or require low power consumption.
  •       Provide responsiveness to make it possible to achieve near real-time delivery of information.


MQTT applications


 MQTT is suitable for the following application domains in which data exchange is required:

  •        Asset tracking and management
  •         Automotive telematics
  •        Chemical detection
  •        Environment and traffic monitoring
  •        Field force automation
  •        Fire and gas testing
  •        Home automation
  •        In-Vehicle Infotainment (IVI)
  •        Medical
  •        Messaging
  •        Railway
  •        Radio-Frequency Identification (RFID)
  •       Supervisory Control and Data Acquisition (SCADA).


Which broker to use?


CloudMQTT is one of the best and easiest cloud-based Mosquitto broker.

CloudMQTT has a free plan that allows you to set up your own CloudMQTT broker instance that will run on their hardware servers. Hence, you can have an online broker that’s ready to use in your IoT project.

It also has a well designed GUI to monitor the publishing and subscribing processes and topics through an easy to use WebSocket UI.


How to use CloudMqtt ?

Let us figure out how CloudMQTT works:

1- Go to http://www.cloudmqtt.com , register an account and select the free option.

2- Then press on the “+ create new instance” green button to get an instance of the MQTT based mosquito cloud broker that cloudMQTT offer.


3- Just write the broker instance name, like: “maximiliano” and press “create new instance”

4- I have created a broker instance before and named it “maximiliano”, but on your account, you will find only the one you have created. Press on its name to see the details.


6- CloudMQTT offers a pretty feature which is the WebSocket UI that allows you to monitor and publish and subscribe happens within this broker.

Now you have a free broker that you can connect any devices. You won't get any messages in your websocket ui.

In my next article, I will show you how to publish and subscribe to the cloudMQTT broker.

Conclusion


We started our journey toward understanding the MQTT protocol. We understood convenient scenarios for this protocol, the details of the publish/subscribe pattern and message filtering. We learned basic concepts related to MQTT and understood the different components: clients, brokers, publish, subscribe and cloudMQTT. In my next article, we are going to learn how to publish and subscribe data to cloudMQTT broker.





Google Cloud Bigtable vs Google Cloud Datastore

What is the difference between&nbsp;<a href="http://googlecloudplatform.blogspot.ro/2015/05/introducing-Google-Cloud-Bigtable.html" target="_blank">Google Cloud Bigtable</a>&nbsp;and Google Cloud Datastore / App Engine datastore, and what are the main practical advantages/disadvantages? AFAIK Cloud Datastore is build on top of Bigtable.

What is the difference between Google Cloud Bigtable and Google Cloud Datastore / App Engine datastore, and what are the main practical advantages/disadvantages? AFAIK Cloud Datastore is build on top of Bigtable.

Cloud Application Development | Cloud Computing Solutions

The advantages of cloud computing have been widely touted. Running your applications on cloud platform helps make your great ideas a reality. Cloud app development helps customers take benefit of the growing number of public and private Paas providers. 

Getting Started with Python working on Google Cloud Functions

Getting Started with Python working on Google Cloud Functions

Writing Python cloud functions is easy and fun, as I will now demonstrate. To follow this tutorial you’ll need a Google Cloud Project, a local Python development environment and the gcloud SDK.

I’ve been a fan of Cloud Functions for a while and a fan of Python since forever, so when support for Python functions was finally announced at Google NEXT’18 I did a small dance of celebration (or at least, I fist-bumped the air when no one was looking). I’m sure you know what Python is, but if you’re new to the serverless world we’re now living in, a quick re-cap on Cloud Functions:

Cloud Functions are small pieces of code that execute in an event-driven manner. They do one small thing very efficiently in reaction to a trigger — usually an HTTP request. The neat thing is you manage zero infrastructure and usually only pay for the execution of your function and a few seconds of compute. You may have heard of other Functions-as-a-Service offerings such as AWS Lambda.

Writing Python cloud functions is easy and fun, as I will now demonstrate. To follow this tutorial you’ll need a Google Cloud Project (you can sign up and get free credits here), a local Python development environment and the gcloud SDK.

Hello, World

We can create a simple function by creating a main.py with just 2 lines:

def hello_world(request):
    return 'Hello, World!\n'

You can deploy this as a cloud Function with the following command. Note that the cloud function name matches the name of the function we defined in code: hello_world

gcloud beta functions deploy hello_world --runtime python37 --trigger-http

After a couple of minutes, the command will return the httpsTrigger or URL for your function. Try accessing this URL and you’ll see the message “Hello, World!” returned. You can also see your function in the Cloud Console UI:

But this is pretty boring stuff, let’s make the function do something a bit more interactive:

def hello_world(request):
    request_json = request.get_json()
    if request_json and 'name' in request_json:
        name = request_json['name']
    else:
        name = 'World'
    return 'Hello, {}!\n'.format(name)

You can use the same gcloud beta functions deploy command as before to update your function. After a couple of minutes, try accessing the URL again and you’ll get the same message. But now try sending it a POST with some data:

curl -X POST <your function URL> -H "Content-Type:application/json" -d '{"name": "Derek"}'

And you should receive a custom message in return! The runtime for Cloud Functions uses the Flask library and provides us with a Request object that we can manipulate however we like.

A More Useful Example

This is all great fun, but now let’s do something useful to show a real world example of where you might employ a Cloud Function. As we’ve seen, Functions are typically triggered by HTTPS endpoints, however you can also make use of Background triggers. Functions deployed this way have no HTTPS endpoint, and are effectively hidden from the Internet, but will instead execute in response to an event such as a file being uploaded to a storage bucket, or a message being sent to a Pub/Sub topic.

Let’s say we have a web service that writes images to a Cloud Storage Bucket. Every time an image is uploaded, we’d like to create a corresponding thumbnail image. Arguably this could be part of our main application code, but for the sake of argument let’s say we want to run this as a separate function, because there may be other ways that images end up in the bucket (maybe an external service sends them to us in batches).

Let’s write a function that reacts when a new file is written to a Cloud Storage bucket. You’ll need 2 GCS buckets for this: You’ll upload images into the first one, and your function will write thumbnails into the second one. In a new directory, create a new main.py file:

from wand.image import Image
from google.cloud import storage

client = storage.Client()

THUMBNAIL_BUCKET = '<your thumbnail bucket>'

def make_thumbnail(data, context):
    # Get the file that has been uploaded to GCS
    bucket = client.get_bucket(data['bucket'])
    blob = bucket.get_blob(data['name'])
    imagedata = blob.download_as_string()
    # Create a new image object and resample it
    newimage = Image(blob=imagedata)
    newimage.sample(200,200)
    # Upload the resampled image to the other bucket
    bucket = client.get_bucket(THUMBNAIL_BUCKET)
    newblob = bucket.blob('thumbnail-' + data['name'])     
    newblob.upload_from_string(newimage.make_blob())

Notice that we’re importing modules, just like we would do with any other Python application. Cloud Functions uses pip, which means we can specify the dependencies we need in a requirements.txt file:

google-cloud-storage
Wand

Just keep your requirements.txt file in the same location as your main.py when you deploy your function.

Don’t forget to replace in the above code with the name of your own thumbnail bucket, and with your source bucket name in the following example. You can now deploy the cloud function like this:

gcloud beta functions deploy make_thumbnail --runtime python37 --trigger-resource <your source bucket>  --trigger-event google.storage.object.finalize

Note: You can’t use the same bucket for both. Why? Because every time the function writes a thumbnail, it would trigger a new invocation of itself!

When you deploy this function you’ll notice we don’t get given a URL. Instead we’ve told the function to run when the google.storage.object.finalize event occurs in the bucket we specified. Let’s upload a picture of Derek the Dog to the source bucket:

Check the thumbnails bucket, and you should see that the function has done its job and created a thumbnail for us!

You can inspect the magic under the hood by revisiting the Cloud Functions part of Cloud Console, where you can track the invocations of your function and even watch its logs in real-time.

Cloud Functions are perfect when you need small pieces of code to tie together larger pieces of a stack. Now that Google has added Python support, we have a huge and diverse ecosystem of functionality available to us to write functions. For more information, read about the Python runtime or follow all of Google’s How To guides.

Thanks for reading!