I created a home IoT setup with AWS, Raspberry Pi

I created a home IoT setup with AWS, Raspberry Pi

I created a home IoT setup with AWS, Raspberry Pi. Hobby projects are a fun way to try and learn new things. This time, I decided to build a simple IoT setup for home, to collect and visualise information like temperature, humidity and pressure.

While learning by doing was definitely one of the reasons I decided to embark the project, I for example wanted to control the radiators located in the attic: Not necessarily by switching power on/off, but getting alarms if I’m heating it too much or little, so that I can tune the power manually. Saving some money, in practice. Also, it is nice the get reminders from humidor that the cigars are getting dried out 😉

I personally learned several things while working on it, and via this blog post, hopefully you can too!

Overview

Idea of the project is relatively simple: Place a few RuuviTag -sensors around the house, collect the data and push it into AWS cloud for permanent storage and additional processing. From there, several solutions can be built around the data, visualisation and alarms being being only few of them.

Solution is built using AWS serverless technologies that keeps the running expenses low while requiring almost non-existing maintenance. Following code samples are only snippets from the complete solution, but I’ve tried to collect the relevant parts.

Collect data with RuuviTags and Raspberry Pi

Tag sensors broadcasts their data (humidity, temperature, pressure etc.) via Bluetooth LE periodically. Because Ruuvi is an open source friendly product, there are already several ready-made solutions and libraries to utilise. I went with node-ruuvitag, which is a Node.js module (Note: I found that module works best with Linux and Node 8.x but you may be successful with other combinations, too).

Raspberry Pi runs a small Node.js application that both listens the incoming messages from RuuviTags and forwards them into AWS IoT service. App communicates with AWS cloud using thingShadow client, found in AWS IoT Device SDK module. Application authenticates using X.509 certificates generated by you or AWS IoT Core.

const sensors = [
  { id: 'f16bcba62cbd', name: 'Room', notified: 0, color: 'green', tag: null },
  { id: 'f9cc863b43f1', name: 'Attic', notified: 0, color: 'cyan', tag: null },
  { id: 'c64981244133', name: 'Humidor', notified: 0, color: 'magenta', tag: null },
  { id: 'fb33d5b20a3d', name: 'Outside', notified: 0, color: 'blue', tag: null },
]

async function registerService(shadow, sensors, updateInterval) {
  const tags = await ruuvi.findTags()
  console.log(colors.gray('Found', tags.length, 'tags'))

  tags.forEach(tag => {
    const sensor = sensors.find(sensor => sensor.id === tag.id)
    // Register if not subscribed yet 
    if (!sensor.tag) {
      sensor.tag = tag 
      sensor.tag.on('updated', getTagDataHandler(shadow, sensor, sensor.tag, updateInterval))
    }
  })
}

function startService(shadow, sensors, registerInterval, updateInterval) {
  // Run registration to ensure all tags get collected
  registerService(shadow, sensors, updateInterval)
  setInterval(() => {
    registerService(shadow, sensors, updateInterval)
  }, registerInterval)
}

// Create and run service 
const shadow = createShadow()
shadow.client.on('connect', () => {
  console.log(colors.bgCyan('Connected to AWS IoT'))
  shadow.client.register(shadow.path, {}, () => {
    startService(shadow, sensors, TAG_REGISTER_INTERVAL, TAG_UPDATE_INTERVAL)
  })
})

app.js

The scripts runs as a Linux service. While tags broadcast data every second or so, the app in Raspberry Pi forwards the data only once in 10 minutes for each tag, which is more than sufficient for the purpose. This is also an easy way to keep processing and storing costs very low in AWS.

When building an IoT or big data solution, one may initially aim for near real-time data transfers and high data resolutions while the solution built on top of it may not really require it. Alternatively, consider sending data in batches once an hour and with 10 minute resolution may be sufficient and is also cheaper to execute.

When running the broadcast listening script in Raspberry Pi, there are couple things to consider:

  • All the tags may not appear at first reading: (Re)run ruuvi.findTags() every 30mins or so, to ensure all the tags get collected
  • Raspberry Pi can drop from WLAN: Setup a script to automatically reconnect in a case that happens

With these in place, the setup have been working without issues, so far.

Process data in AWS using IoT Core and friends

Once the data hits the AWS IoT Core there can be several rules for handling the incoming data. In this case, I setup a lambda to be triggered for each message. AWS IoT provides also a way to do the DynamoDB inserts directly from the messages, but I found it more versatile and development friendly approach to use the lambda between, instead.

import 'source-map-support/register'
import * as omit from 'lodash.omit'
import { IoTEvent } from '../interfaces'
import { ScrewedTable, getMapper } from '../tables'

export const handle = async (event: IoTEvent, _context) : Promise<void> => {
  console.log('Handling event:', event)
  const mapper = getMapper()

  // Store data into table, drop id and time from data field
  const entry = Object.assign(new ScrewedTable, {
    type: 'tag',
    id: `${event.id}-${event.time}`, // Unique identifier for single tag and time
    time: event.time,
    data: JSON.stringify(omit(event, ['id', 'time'])),
  })
  console.debug('Writing entry:', entry);
  await mapper.put(entry)
}

iot.ts

DynamoDB works well as permanent storage in this case: Data structure is simple and service provides on demand based scalability and billing. Just pay attention when designing the table structure and make sure it fits with you use cases as changes done afterwards may be laborious. For more information about the topic, I recommend you to watch a talk about Advanced Design Patterns for DynamoDB.

Visualise data with React and Highcharts

Once we have the data stored in semi structured format in AWS cloud, it can be visualised or processed further. I set up a periodic lambda to retrieve the data from DynamoDB and generate CSV files into public S3 bucket, for React clients to pick up. CSV format was preferred over for example JSON to decrease the file size. At some point, I may also try out using the Parquet -format and see if it suits even better for the purpose.

The React application fetches the CSV file from S3 using custom hook and passes it to Highcharts -component.

During my professional career, I’ve learnt the data visualisations are often causing various challenges due to limitations and/or bugs with the implementation. After using several chart components, I personally prefer using Highcharts over other libraries, if possible.

export function useTagData(tagId: string): IoTData {
  const initialData: IoTData = []
  const [data, setData] = useState(initialData)

  useEffect(() => {
    axios.get(`https://iot.mydomain.com/tag-${tagId}.csv`)
      .then(csvData => {
        const output = parse(csvData.data, {
          skip_empty_lines: true,
        })

        setData(output
          .filter((entry: any, index: number) => index > 0)
          .sort((entry: any) => parseInt(entry[0]))
          .map((entry: any) => {
            // Map data into format suitable for Highcharts
          })
        )
      })
  })
  
  return data
}

hooks.ts

Send notifications with Telegram bots

Visualisations works well to see the status and how the values vary by the time. However, in case something drastic happens, like humidor humidity gets below preferred level, I’d like to get an immediate notification about it. This can be done for example using Telegram bots:

  1. Define the limits for each tag for example into DynamoDB table
  2. Compare limits with actual measurement whenever data arrives in custom lambda
  3. If value exceeds the limit, trigger SNS message (so that we can subscribe several actions to it)
  4. Listen into SNS topic and send Telegram message to message group you’re participating in
  5. Profit!

export async function notifyBot(token: string, chatId: string, message: string): Promise<void> {
   await axios.post(`https://api.telegram.org/bot${token}/sendMessage`, {
    chat_id: chatId,
    text: message,
    parse_mode: 'Markdown'
  })
  return
}

telegram.ts

Summary

By now, you should have some kind of understanding how one can combine IoT sensor, AWS services and outputs like web apps and Telegram nicely together using serverless technologies. If you’ve built something similar or taken very different approach, I’d be happy hear it!

Price tag

Building and running your own IoT solution using RuuviTags, Raspberry Pi and AWS Cloud does not require big investments. Here are some approximate expenses from the setup:

  • 3-pack of RuuviTags: 90e (ok, I wish these were a little bit cheaper so I’d buy these more since the product is nice)
  • Raspberry Pi with accessories: 50e
  • Energy used by RPi: http://www.pidramble.com/wiki/benchmarks/power-consumption
  • Lambda executions: $0,3/month
  • SNS notifications: $0,01/month
  • S3 storage: $0,01/month
  • DynamoDB: $0,01/month

And after looking into numbers, there are several places to optimise as well. For example, some lambdas are executed more often than really needed.

Next steps

I’m happy say this hobby project has achieved that certain level of readiness, where it is running smoothly days through and being valuable for me. As a next steps, I’m planning to add some kind of time range selection. As the amount of data is increasing, it will be interesting to see how values vary in long term. Also, it would be a good exercise to integrate some additional AWS services, detect drastic changes or communication failures between device and cloud when they happen. This or that, at least now I have a good base for continue from here or build something totally different next time 🙂

References, credits and derivative work

This project is no by means a snowflake and has been inspired by existing projects and work:

At Nordcloud we are always looking for talented people. If you enjoy reading this post and would like to work with public cloud projects on a daily basis — check out our open positions here.

Scratch Programming for Raspberry Pi

Raspberry Pi 3 Day Project: Retro Gaming Suite

Raspberry Pi Projects : Build a Media Centre Computer

Build Your Own Super Computer with Raspberry Pis

PiBot: Build Your Own Raspberry Pi Powered Robot

Raspberry Pi 4 on the Raspberry Pi 4 - Computerphile

Raspberry Pi 4 on the Raspberry Pi 4 - Computerphile

A quick tour of the Raspberry Pi 4 edited on the Raspberry Pi 4. Dr Steve Bagley gets out his knife.dll to unbox Sean's purchases! ☞ [I created a home IoT setup with AWS, Raspberry...

A quick tour of the Raspberry Pi 4 edited on the Raspberry Pi 4. Dr Steve Bagley gets out his knife.dll to unbox Sean's purchases!

I created a home IoT setup with AWS, Raspberry Pi

Benchmarking the Raspberry Pi 4

The easy way to set up Docker on a Raspberry Pi – freeCodeCamp.org

Creating a Rogue Wi-Fi Access Point using a Raspberry Pi

Building a Smart Garden With Raspberry Pi 3B+

Learn Raspberry Pi for Image Processing Applications

Learn Raspberry Pi for Image Processing Applications

New to the newly launched Raspberry Pi 3? Learn all the components of Raspberry Pi, connecting components to Raspberry Pi, installation of NOOBS operating system, basic Linux commands, Python programming and building Image Processing applications on Raspberry Pi. At just $9.

Description
Image Processing Applications on Raspberry Pi is a beginner course on the newly launched Raspberry Pi 3 and is fully compatible with Raspberry Pi 2 and Raspberry Pi Zero.

The course is ideal for those who are new to the Raspberry Pi and want to explore more about it.

You will learn the components of Raspberry Pi, connecting components to Raspberry Pi, installation of NOOBS operating system, basic Linux commands, Python programming and building Image Processing applications on Raspberry Pi.

This course will take beginners without any coding skills to a level where they can write their own programs.

Basics of Python programming language are well covered in the course.

Building Image Processing applications are taught in the simplest manner which is easy to understand.

Users can quickly learn hardware assembly and coding in Python programming for building Image Processing applications. By the end of this course, users will have enough knowledge about Raspberry Pi, its components, basic Python programming, and execution of Image Processing applications in the real time scenario.

The course is taught by an expert team of Electronics and Computer Science engineers, having PhD and Postdoctoral research experience in Image Processing.

Anyone can take this course. No engineering knowledge is expected. Tutor has explained all required engineering concepts in the simplest manner.

The course will enable you to independently build Image Processing applications using Raspberry Pi.

This course is the easiest way to learn and become familiar with the Raspberry Pi platform.

By the end of this course, users will build Image Processing applications which includes scaling and flipping images, varying brightness of images, perform bit-wise operations on images, blurring and sharpening images, thresholding, erosion and dilation, edge detection, image segmentation. User will also be able to build real-world Image Processing applications which includes real-time human face eyes nose detection, detecting cars in video, real-time object detection, human face recognition and many more.

The course provides complete code for all Image Processing applications which are compatible on Raspberry Pi 3/2/Zero.

Who is the target audience?

Anyone who wants to explore Raspberry Pi and interested in building Image Processing applications

To read more:

Raspberry Pi: Dummy tutorial on port forwarding and SSH

Raspberry Pi: Dummy tutorial on port forwarding and SSH

Raspberry Pi: Dummy tutorial on port forwarding and SSH .This is a continuation of my series on setting up Raspberry Pi to be a remote jupyter notebook code editor. In the last chapter Raspberry Pi was set up and could be accessed by SSH at your home network

This is a continuation of my series on setting up Raspberry Pi to be a remote jupyter notebook code editor. In the last chapter Raspberry Pi was set up and could be accessed by SSH at your home network. In this chapter I will guide you how to set up port forwarding and access the Pi remotely from the internet, so you could write command under CLI just like at home.

First time Pi user and first time reader? Check out the first part here.

But once your Pi is accessible from the internet, you need to be 100x cautious about potential security vulnerability. Most of the content you could find on Google Search should guide you to a term “Port Forwarding”, while in the Raspberry Pi official documentation it explicitly told you that port forwarding exposes a known security problem and you should consider altnerative ways to do it. I am going to introduce both ways in this series. And I promise we will get to host website soon, but let’s make sure it’s safe to do so first.

Table of Content
  • Increase your security before exposing your Pi to the internet
  • Port Fowarding
  • What’s next: Cloud Proxy Connections
Increase your security before exposing your Pi to the internet

There are two things you should do before exposing your Pi to the internet.

  1. Change the password of default user Pi

To change the password, simply type in passwd and follow the text instructions by re-typing your old password, your new password and re-tpying your new password.

  1. Create a new user without sudo access and use that user to access your pi remotely

sudo: similar to system admin preveilage in windows, which user can run command that is defined as superuser level, including rebooting your computer and installing any software. You can check out more here.
A good practice, for example, is to use sudo at home to install the software needed and give normal user read/write access to only some sub-folders, then use normal user to remotely login the Pi and only work on these sub-folders.

Let’s create a new user without sudo access, type in sudo adduser normal_user.

You will be asked a series of questions, including the password, basic information of the user and enter Y to confirm all information is correct.

Now we have created a new user normal_user.

To check whether this user have sudo access, we could simply type in sudo -l -U normal_user. And the text should show you ‘User normal_user is not allowed to run sudo on raspberrypi.’

So next time, you could try out SSH connection by not using pi as the login, but normal_user .

Want to know more about user access and organizing user? Check out the tutorial on digitalocean.

Port Forwarding

First, I want to reiterate that setting up port forwarding without any safety add-on to block malicious traffic is not recommended, and this article does not provide enough guidance to set that up. What I would recommend is to set up a cloud proxy server, which we will go through in the next chapter.

With all said, let’s go through a dummy example.

Your home network is known as Private LAN in that no external device could connect to your devices at home. This is controlled by firewall, which by default denies all incoming traffic.

But you don’t always spend your life in Raspberry Pi (me neither, just to be clear), you also love to play Diablo II, a classic rpg game which you could coop with other players. And you play it on another home device, PC. Turns out Diablo II is an old game which requires a direct connection between you and other players in order to hack-and-slash-and-loot the monsters. When you guys are in the game, data package including player’s location, level and action will need to be continuously streamed between players.

Your home PC has an internal IP address of 192.168.1.4. Diablo II automatcially configures a port 1033 for other players to get your data pacakge.

As I mentioned in last chatper, each device at your home will be automatically assigned an IP address by the router (e.g. my Pi’s address is at 192.168.1.50) and each internet application will use up one port number (e.g. 22). A quick refresher example: 192.168.1.50:22 represents <my Raspberry Pi>:<SSH Application>.

Finally, you do a google search and know that your external IP address is 50.247.207.5.

Turns out, if you were able to open a port on the external IP address, and map(point) that port to one set of <Internal IP Address>:<Port>, then you can tell your friends to enter your <External IP Address>:<Port your opened> to route their connection to your <PC's internal IP Address>:<Dialbo II data package application (1033)>.

This is what referred as Port Fowarding and you basically pinhole on your firewall to finally allow incoming traffic to a specific internal IP address and port.

Risk with Port Fowarding

Before I teach you how to forward a port, I want to crystalize what could happen at the worst case.

First, Port Forward won’t expose all your devices at your home. It only allow external user to connect to device you’ve pointed it at. In Diablo II, that’s perfectly fine. As the only application you opened is for other player to receive data package from your game. But for Raspberry Pi, the SSH application represents full access to your Pi’s CLI, and able to run any command including communication with your other home devices. And this is why opening Port for Diablo and XBox are generally safe and trivial but for Raspberry Pi it could be very dangerous.

The chance for any hacker to hide in the brush and wait til you open a port is nearly impossible, let alone to say that hacker needs to know a user / password pair in order to login. So it isn’t really a huge security problem for most of home users. But the possibility is there.

Port Forwarding your Pi to external network

Let’s clarify things we know before we started. From last chatper, we know how to check the internal IP address of our Pi, external IP address of our network and we also know that port 22 is opened for SSH from our Pi.

1~ Enter your router configuration page

Router configuration page is usually a website hosted inside your home network. The url should be printed on the router itself, and definitely shown in the router manual. If you still cannot find it, simply try http://192.168.1.1/ as most of the routers occupied this IP as their configuration page.

2~ Go to port forwarding configuration page

The page location will vary for different brands. Try your best to look for keywords including

  • Advanced Setting, Security Setting, Port Forwading, Virtual Server, Wan Setting

And you should be able to get to the port forwarding page.

3~ Configure rules

For some routers, they will require an extra step to enable port forwarding. Make sure to enable that!

Then, you will probably see a list of inputs, including

  • Service Name: text to describle the port foward service.
  • Source Target [optional]: Whitelist of external IP. Only IPs that are whitelisted could be able to connect to this port. Therefore, if you already know the IP where you are going to make the connection. Adding the whitelist IP here could tremendously increase your security. But this is only available for newer models of router. Leave it blank if you allow all IP to make connection.
  • Port Range: Port that is opened up from your router. In the above example, I open a port on 10300. Let’s say my external IP is 50.247.207.5, using 50.247.207.5:10300 could redirect me to the Pi later.
  • Local IP: IP address of your Pi in your private LAN. Check out last chapter if you don’t know how to find it. In this case it’ 192.168.1.50.
  • Port: Port / Application you want to open from the device specified in the local IP. In this case its 22, since Pi configure port 22 as the SSH application.
  • Protocol: TCP / UDP are usually available. You should also be able to select both. If only one of them can be selected. Please select TCP.

Once you apply the change, you are able to ssh to your Pi anywhere in the world! In this example, we can access our Pi by visiting 50.247.207.5:10300.

Accessing your Pi with windows

Same as last chatper, we could use Putty to connect to our Pi. But this time you could finally try this in a safe network (aka not Starbucks) outside your home wifi.

On Putty, type in your external IP address and Port you open up from the port forwarding setting. In this example it is 50.247.207.5:10300. Keep the connection type as SSH.

And you should be able to login with the CLI pop-up! This time let’s try out our normal_user username and password, so even if hackers get your credential, it’s just a non-sudo user.

And…. we are in! Now you can write and run any program in your Pi from anywhere!

What’s next: Cloud Proxy Connections

Port forwarding is always associated with risk. And we should consider other options like cloud proxy connections. Unfortunately on Raspberry Pi documentation there is no concreted instruction on how to do so, though it does recommend some free services we could use.

In next story, I am going to demonstrate how to use one of the cloud proxy connection service listed in the official documentation to connect to your Pi with SSH, remotely and securely. Stay tuned!