How Python can help you learn ES6

How Python can help you learn ES6

In this post, we’ll look at how Python can help you learn ES6.

“Have you learned ES6 yet?”

Oof. I used to feel a sense of pressure when people would ask me that. In the end, I learned ES6 with the help of Python. Strange, huh? Turns out, a bunch of syntax is shared between the two languages, so they go hand in hand in a way.

Basic differences between Python and ES6

Before we get into how similar JavaScript and Python are, I first want to talk about some key differences. For example, whitespace in JavaScript doesn’t matter at compile time, but it sure does in Python. Python relies on indentation to determine the grouping of statements.

Primitives in JavaScript and **Python **are also quite different. Check out the table below which details the primitives of both languages. You’ll see that they overlap a bit with their representations of Booleans and nothing, but beyond that, they’re distinct.

Table of primitives in JavaScript** **versus Python

One last, basic difference to note about JavaScript** and Python **is that JavaScript allows type coercion. The following code blocks demonstrate coercing a number to a string in JavaScript, but an impossibility in Python!

// You can coerce an integer into string in JavaScript
let coerced = 1;
let concatenated = coerced + 'string';

# You can't coerce an integer into a string in Python
not_coerced = 1
concatenated = str(not_coerced) + 'string'

Functions or.. methods?

Functions and conditionals have extremely similar structures in both **JavaScript **and Python. For example:

function drSeuss(catInTheHat, thing1, thing2) {
  if (catInTheHat == true &&
    thing1 == true &&
    thing2 == true) {
    console.log('is cray');
  } else if (catInTheHat != true) {
    console.log('boring');
  } else {
    console.log('so boring');
  }
}

def dr_seuss(cat_in_the_hat, thing1, thing2):
  if cat_in_the_hat == True and
    thing2 == True and
    thing2 == True:
    print 'is cray'
  elif cat_in_the_hat != True:
    print 'boring'
  else:
    print 'so boring'

I hadn’t thought too much about this, but with JavaScript, the idea of “methods” often refers to methods that are built into the specification of the language, e.g. Function.prototype.apply().

From MDN:

In most respects functions and methods are identical except for two key differences:> A method is implicitly passed the object on which it was called.> A method is able to operate on data that is contained within the class.
Because classes don’t truly exist in JavaScript, the function and method example below is represented in **Python **only (more on ES6 classes later in this article).

def fish_and_chips():
  ingredients = ['fish', 'potatoes', 'batter']
  print 'cooking %s together' % (', '.join(ingredients))

# cooking fish, potatoes, batter

class Baking(object):
  def __init__(self, supplies):
    self.supplies = supplies

  def bakewell_tart(self):
    ingredients = ['almonds', 'raspberry', 'icing sugar']
    print self
    print 'baking %s' % (', '.join(ingredients))

# <__main__.Baking object at 0x10d6e0510>

A function versus a method in Python

Ok, onto some of the ways **Python **pushed me to learn more about ES6!

Block scope

When I first started learning JavaScript (back in “ancient” ES5 days), I thought many structures in the language created scope. I thought that blocks inside of conditionals statements created scope. I found that only functions create scope in JavaScript.

With the addition of const and let to ES6, we get block scope!

function simpleExample(value) {
  if (value) {
    var varValue = value;
    let letValue = value;
    console.log(varValue, letValue); // value value
  }
 
  // varValue is available even though it was defined
  // in if-block because it was "hoisted" to function scope
  console.log(varValue); // value
 
  // letValue is a ReferenceError because 
  // it was defined within the if-block
  console.log(letValue); // Uncaught ReferenceError: letValue is not defined

What else creates scope in JavaScript, ES6, and Python? And what type of scope do they use? Check out the following table:

Template literals

I often think of template literals as Mad Libs. In case you didn’t grow up with Mad Libs, they were short stories which you could fill in with your own content. Sentences were missing words, and you could write anything you wanted into those spaces. You only had to conform to the specified word type: noun, pronoun, verb, adjective, exclamation.

Mad Libs that read things like:

mothers sit around burping. Last summer, my little brother fell in a/an hairdo and got poison palmtree all over his butt. My family is going to Winsconsin, and I will..
Similar to Mad Libs, template literals are string literals that allow embedded expressions. They were originally called “template strings” in prior editions of the ES2015 specification.

Yup, these already existed in **Python **before **ES6 **was released. I had actually learned about literal string interpolation in Python, which made it that much easier for me to understand template literals in ES6. They are great because you no longer need the concatenation found in older versions of JavaScript which could get a bit ridiculous and coerce other types into strings.

let exclamation = 'Whoa!';
let sentence = `They are really similar to Python.`;
 
console.log(`Template Literals: ${exclamation} ${sentence}`);
// Template Literals: Whoa! They are really similar to Python.

print '.format(): {} {}'.format('Yup.', 'Quite!')
# .format(): Yup. Quite!

Default parameters

Yup, Python has had these forever too. Default parameters set a default for function parameters. This is most effective for avoiding bugs that arise when arguments are missing. And with the advent of ES6, JavaScript gained default parameters too.

function nom(food="ice cream") {
  console.log(`Time to eat ${food}`);
}
 
nom(); // Time to eat ice cream

def nom(food="ice cream"):
  print 'Time to eat {}'.format(food)
 
nom() # Time to eat ice cream

*Rest parameters & args

Rest parameter syntax allows us to represent an indefinite number of arguments as an array. In Python, they’re called *args, which again, I’d already learned before ES6! Are you sensing a pattern here?

Check out how each of the languages bundles parameters up in neat little packages:

function joke(question, ...phrases) {
  console.log(question);
  for (let i = 0; i > phrases.length; i++) {
    console.log(phrases[i]);
  }
}

let es6Joke = "Why does JS single out one parameter?"
joke(es6Joke, "Because it doesn't", 'really like', 'all the REST of them!');
 
// Why does JS single out one parameter?
// Because it doesn't
// really like
// all the REST of them!

def pirate_joke(question, *args):
  print question
  for arg in args:
    print arg
 
python_joke = "What's a Pyrate's favorite parameter?"
 
pirate_joke(python_joke, "*args!", "*arrgs!", "*arrrgs!")
 
# What's a Pyrate's favorite parameter?
# *args!
# *arrgs!
# *arrrgs!

Classes

Now, let’s look at prototypal inheritance! ES6 classes are actually syntactic sugar and based on the prototype chain found in ES5 and previous iterations of JavaScript. So, what we can do with **ES6 **classes is not much different from what we do with ES5 prototypes.

Python has classes built in, allowing for quick and easy object-oriented programming. I always found the prototype chain extremely confusing in JavaScript, but looking at **Python **and **ES6 **classes side by side really made sense to me.

To explain what the JavaScript prototype is, here’s a quote from MDN:

When it comes to inheritance, JavaScript only has one construct: objects. Each object has an internal link to another object called its prototype. That prototype object has a prototype of its own, and so on until an object is reached with null as its prototype. null, by definition, has no prototype, and acts as the final link in this prototype chain.
Let’s take a look at these ES6 “classes” based on the prototype chain:

class Mammal {
  constructor() {
    this.neocortex = true;
  }
}
 
class Cat extends Mammal {
  constructor(name, years) {
    super();
    this.name = name;
    this.years = years;
  }
 
  eat(food) {
    console.log('nom ' + food);
  }
}
 
let fryCat = new Cat('Fry', 7);
fryCat.eat('steak');

class Mammal(object):
  neo_cortex = True
 
class Cat(Mammal):
  def __init__(self, name, years):
    self.name = name
    self.years = years
 
  def eat(food):
    print 'nom %s' % (food)
 
fry_cat = Cat('Fry', 7)
fry_cat.eat('steak')

A big difference between ES6 Classes and ES5 Prototypes: you can inherit more easily with classes than with the prototype chain. This is very similar to Python’s structure. Neat!

So there you have it. A bunch of examples of how Python helped me learn all about ES6. Often with programming languages, many differences exist, but so do many similarities. And it’s in those similarities that we can more easily learn new languages!

Introduction New Features in TypeScript 3.7 and How to Use Them

Introduction New Features in TypeScript 3.7 and How to Use Them

The TypeScript 3.7 release is coming soon, and it's going to be a big one.

The target release date is November 5th, and there are some seriously exciting headline features included:

  • Assert signatures.
  • Recursive type aliases.
  • Top-level await.
  • Null coalescing.
  • Optional chaining.

Personally, I'm super excited about this, they're going to whisk away all sorts of annoyances that I've been fighting in TypeScript while building HTTP Toolkit.

If you haven't been paying close attention to the TypeScript development process though, it's probably not clear what half of these mean, or why you should care. Let's talk through all of them.

Assert Signatures

This is a brand-new and little-known TypeScript feature, which allows you to write functions that act like type guards as a side-effect, rather than explicitly returning their boolean result.

It's easiest to demonstrate this with a JavaScript example:

function assertString(input) { 
  if (typeof input === 'string') 
    return; 
  else 
    throw new Error('Input must be a string!'); 
} 
function doSomething(input) { 
  assertString(input); 
  // ... Use input, confident that it's a string 
} 
doSomething('abc'); 
// All good doSomething(123); // Throws an error

This pattern is neat and useful, and you can't use it in TypeScript today.

TypeScript can't know that you've guaranteed the type of input after it's run assertString. Typically, people just make the argument input: string to avoid this, and that's good. But, it also just pushes the type checking problem somewhere else, and in cases where you just want to fail hard, it's useful to have this option available.

Fortunately, soon we will:

// With TS 3.7 
function assertString(input: any): 
	asserts input is string { 
      // <-- the magic 
      if (typeof input === 'string') 
        return; 
      else 
        throw new Error('Input must be a string!'); 
    } 
function doSomething(input: string | number) { 
  assertString(input); 
  // input's type is just 'string' here }

Here assert input is string means that if this function ever returns, TypeScript can narrow the type of input to string, just as if it was inside an if block with a type guard.

To make this safe, that means if the assert statement isn't true then your assert function must either throw an error or not return at all (kill the process, infinite loop, you name it).

That's the basics, but this actually lets you pull some really neat tricks:

// With TS 3.7 
// Asserts that input is truthy, throwing immediately if not: 
function assert(input: any): 
	asserts input { // <-- not a typo 
      if (!input) 
        throw new Error('Not a truthy value'); 
    } 
declare const x: number | string | undefined; 
assert(x); // Narrows x to number | string 
// Also usable with type guarding expressions! 
assert(typeof x === 'string'); 
// Narrows x to string // -- Or use assert in your tests: -- 
const a: Result | Error = doSomethingTestable(); 
expect(a).is.instanceOf(result); 
// 'instanceOf' could 'asserts a is Result' 
expect(a.resultValue).to.equal(123); 
// a.resultValue is now legal // -- Use as a safer ! that throws immediately if 
// you're wrong -- 
function assertDefined<T>(obj: T): 
	asserts obj is NonNullable<T> { 
      if (obj === undefined || obj === null) { 
        throw new Error('Must not be a nullable value'); 
      } 
    } 
declare const x: string | undefined; 
// Gives y just 'string' as a type, could throw elsewhere later: 
const y = x!; 
// Gives y 'string' as a type, or throws immediately if you're wrong: 
assertDefined(x); const z = x; 
// -- Or even update types to track a function's side-effects -- 
type X<T extends string | {}> = { value: T }; 
// Use asserts to narrow types according to side effects: 
function setX<T extends string | {}>(x: X<any>, v: T): 
	asserts x is X<T> { 
      x.value = v; 
    } 
	declare let x: X<any>; 
// x is now { value: any }; 
setX(x, 123); 
// x is now { value: number };

This is still in flux, so don't take it as the definite result, and keep an eye on the pull request if you want the final details.

There's even a discussion there about allowing functions to assert something and return a type, which would let you extend the final example above to track a much wider variety of side effects, but we'll have to wait and see how that plays out.

Top-Level Await

Async/await is amazing and makes promises dramatically cleaner to use.

Unfortunately, though, you can't use them at the top level. This might not be something you care about much in a TS library or application, but if you're writing a runnable script or using TypeScript in a REPL, then this gets super annoying.

It's even worse if you're used to frontend development, since top-level await has been working nicely in the Chrome and Firefox console for a couple of years now.

Fortunately though, a fix is coming. This is actually a general stage-3 JS proposal, so it'll be everywhere else eventually too, but for TS devs 3.7 is where the magic happens.

This one's simple, but let's have another quick demo anyway:


// Your only solution right now for a script that does something async: 
async function doEverything() { 
  ... 
  const response = await fetch('http://example.com'); 
  ... 
} 
  
doEverything(); // <- eugh (could use an IIFE instead, but even more eugh)

With top-level await:

// With TS 3.7: 
// Your script: ... 
const response = await fetch('http://example.com'); 
// ...

There's a notable gotcha here: if you're not writing a script, or using a REPL, don't write this at the top level, unless you really know what you're doing!

It's totally possible to use this to write modules that do blocking async steps when imported. That can be useful for some niche cases, but people tend to assume that their import statement is a synchronous, reliable, and fairly quick operation, and you could easily hose your codebase's startup time if you start blocking imports for complex async processes (even worse, processes that can fail).

This is somewhat mitigated by the semantics of imports of async modules: they're imported and run in parallel, so the importing module effectively waits for Promise.all(importedModules) before being executed.

Rich Harris wrote an excellent piece on a previous version of this spec before that change when imports ran sequentially and this problem was much worse), which makes for good background reading on the risks here if you're interested.

It's also worth noting that this is only useful for module systems that support asynchronous imports. There isn't yet a formal spec for how TS will handle this, but that likely means that a very recent target configuration, and, either ES Modules or Webpack v5 (whose alphas have experimental support), will be used at runtime.

Recursive Type Aliases

If you're ever tried to define a recursive type in TypeScript, you may have run into StackOverflow questions like this: https://stackoverflow.com/questions/47842266/recursive-types-in-typescript.

Right now, you can't. Interfaces can be recursive, but there are limitations to their expressiveness, and type aliases can't. That means right now, you need to combine the two: define a type alias and extract the recursive parts of the type into interfaces. It works, but it's messy, and we can do better.

As a concrete example, this is the suggested type definition for JSON data:

type JSONValue = | string | number | boolean | JSONObject | JSONArray; 
interface JSONObject { [x: string]: JSONValue; } 
interface JSONArray extends Array<JSONValue> { }

That works, but the extra interfaces are only there because they're required to get around the recursion limitation.

Fixing this requires no new syntax; it just removes that restriction, so the below compiles:

// With TS 3.7: 
type JSONValue = | string | number | boolean | { [x: string]: JSONValue } | Array<JSONValue>;

Right now, that fails to compile with Type alias 'JSONValue' circularly references itself. Soon though, soon...

Null Coalescing

Aside from being difficult to spell, this one is quite simple and easy. It's based on a JavaScript stage-3 proposal, which means it'll also be coming to your favorite vanilla JavaScript environment soon (if it hasn't already).

In JavaScript, there's a common pattern for handling default values, and falling back to the first valid result of a defined group. It looks something like this:

// Use the first of firstResult/secondResult which is truthy: 
const result = firstResult || secondResult; 
// Use configValue from provided options if truthy, or 'default' if not: 
this.configValue = options.configValue || 'default';

This is useful in a host of cases, but due to some interesting quirks in JavaScript, it can catch you out. If firstResult or options.configValue can meaningfully be set to false, an empty string or 0, then this code has a bug. If those values are set, then when considered as booleans they're falsy, so the fallback value (secondResult / 'default') is used anyway.

Null coalescing fixes this. Instead of the above, you'll be able to write:

// With TS 3.7: 
// Use the first of firstResult/secondResult which is *defined*: 
const result = firstResult ?? secondResult; 
// Use configSetting from provided options if *defined*, or 'default' if not: 
this.configValue = options.configValue ?? 'default';

?? differs from || in that it falls through to the next value only if the first argument is null or undefined, not falsy. That fixes our bug. If you pass false as firstResult, that will be used instead of secondResult because, while it's falsy, it is still defined, and that's all that's required.

It's simple but super-useful, as it takes away a whole class of bugs.

Optional Chaining

Last but not least, optional chaining is another stage-3 proposal that is making its way into TypeScript.

This is designed to solve an issue faced by developers in every language: how do you get data out of a data structure when some or all of it might not be present?

Right now, you might do something like this:

// To get data.key1.key2, if any level could be null/undefined: 
let result = data ? (data.key1 ? data.key1.key2 : undefined) : undefined; 
// Another equivalent alternative: 
let result = ((data || {}).key1 || {}).key2;

Nasty! This gets much much worse if you need to go deeper, and although the second example works at runtime, it won't even compile in TypeScript, since the first step could be {}, in which case key1 isn't a valid key at all.

This gets still more complicated if you're trying to get into an array, or there's a function call somewhere in this process.

There's a host of other approaches to this, but they're all noisy, messy & error-prone. With optional chaining, you can do this:

// With TS 3.7: 
// Returns the value is it's all defined & non-null, or undefined if not. 
let result = data?.key1?.key2; 
// The same, through an array index or property, if possible: 
array?.[0]?.['key']; 
// Call a method, but only if it's defined: 
obj.method?.(); 
// Get a property, or return 'default' if any step is not defined: 
let result = data?.key1?.key2 ?? 'default';

The last case shows how neatly some of these dovetails together: null coalescing + optional chaining is a match made in heaven.

One gotcha: this will return undefined for missing values, even if they were null, e.g. in cases like (null)?.key (returns undefined). A small point, but one to watch out for if you have a lot of null in your data structures.

That's the lot! That should outline all the essentials for these features, but there are lots of smaller improvements, fixes, and editor support improvements coming too, so take a look at the official roadmap if you want to get into the nitty-gritty.

How to building a stable Node.js project architecture.

How to building a stable Node.js project architecture.

Often product development process which involves JavaScript, is accompanied by the use of Node.js, a JavaScript runtime environment. The birth of this technology has certainly turned the use of JS upside-down. Today, [JavaScript...

Often product development process which involves JavaScript, is accompanied by the use of Node.js, a JavaScript runtime environment. The birth of this technology has certainly turned the use of JS upside-down. Today, JavaScript is in the category of the most preferred languages to build apps thanks to Node.js.

What is so special about this technology? To answer this, let’s reflect on not only this technology’s benefits but also its architecture limitations and the ways to deal with cons.

Node JS brief history

Node.js was introduced by Ryan Dahl in 2009. The technology is mostly used for building app’s server side/ back-end development. What’s special about Node.js is that the technology is asynchronous.

This means that server continues to process other client requests without urging the client to wait till another previously sent request is processed. Let’s say, it’s Node.js “value proposition” for all who would like to create reliable JS-based apps.

What is Node JS commonly used for?

The non-blocking I/O machine behind the current framework is a great way to build real-time web app with NodeJS, mobile products, chats, data streaming apps, browser games, APIs and medium-performance JS apps.

Node JS real-time applications examples and showcase

Among the companies who rely their tech part to Node.js are LinkedIn, Yahoo, IBM, Netflix, PayPal, Uber and others.

Let’s see where else Node is used apart from back-end (2017 data):
This is image title

Source
As of business point of view, Node.js is used for:
This is image title

If you have worked with Node.js, you probably already know that with Node.js you can gain the following:

  • Since Node.js is created with the C++ help, you can call C functions

  • Asynchronous nature which accelerates app’s functioning and provides the ability to multitask. Apart from this, non-closing i/o method suits for high-traffic, real-time websites, resources creation.

  • Stable multiplatform app development

  • Lots of Node.js development tools for better workflow: npm-s, Express, Socket.io, etc. (we’ll touch upon these a bit later in the article)

  • Clear and flexible learning curve

But with Node.js architecture limitations you lose the opportunities to:

  • Create heavy-computed apps with the elements of 3D projection; calculation apps.
    As Node.js is single-threaded, it is not the right fit for such projects. All the actions happen on a single thread, and hence, overload CPU. For such types of apps or software, it’s better to utilize multithreading languages, like C, C#. For the full-scale video games, you can use Unity.

  • Use some npm-s.

Not all npm-s, we’ve mentioned in the pros, are of high quality and stable. Thus, you have to filter them properly and choose only the reliable ones. (Author’s note: Think of npm-s, like of plugins in Wordpress).

  • Taking in consideration Node.js architecture, you can’t utilize relational databases at full power.

Node.js come best with such document-oriented databases, like MongoDB

Let’s get into tech details and best practices for Node.js development and more detailed practical tips on working with this platform

  • Application Specifics
    First and foremost, think about what type of application you plan to release. To further proceed with Node.js project architecture building, ask yourself some of the following questions:

  • Are you going to build a real-time web app with Node.js? Is it meant to be a mobile or a console one? Or maybe it's a multiplatform app?

  • What data should the app operate with? Are these databases, files, or remote storages (like Amazon S3)?

  • Do you plan to use special software in your application? Are sophisticated data processing algorithms such as face detection or text recognition enlisted in your app's functionality business plan?

  • Does your application need an extra hardware, like camera, microphone, various sensors, or any other related devices?

  • What are the architectural specifics of the future app? Is this meant to be a client server, MVC or maybe any other type of architecture?

If you've answered all of these questions, make sure that Node.js is able to fully meet all requirements set for your project development. For example, if you need an API server that works with several types of databases, Node.js might be a good choice. But if you need an application that is designed to build 3D graphics using Directx, you might want to get acquainted with C ++ a bit closer.

Let's assume that your application uses special temperature and contamination sensors. You can pair such features with RaspberryPi, Arduino or any other special device to go further and create a 'smart' functionality model. But before you start, make sure that the driver of any mentioned device is compatible with Node.js.

Best practices for Node.js development workflow

Typically, an application is written by a team of developers. Everyone in the team is unique as well as their own code style. Therefore, it’s recommended to settle and take into account the following code organization nuances before the development stage.

  • Functional development style or object-oriented programming patterns usage?

Since JS is a weakly-typed language and allows you to write your code in a freestyle, it's still better to agree upon a single code writing rules in your team. This will keep most of the misunderstandings away and will help your colleagues to get the better understanding of the project’s code.

  • Code style
    Discuss with your teammates code writing do-s and don’t-s. Check the quality of what is written, using such Node.js development tools, like lint

  • Your team’s experiencewith the integration of third-party means and devices in your application (i.e. Google Maps, data collection, analytics tools, e-communication means,etc.)

  • Data models you’re up to work with (files, databases or third-party APIs)

  • Communication and data exchange tools you’re going to use (REST API, Blouse Protocol, Socket io, GraphQl, DDP protocol)

  • Possibility to utilize 3rd party libraries (hardware libraries, special algorithms)

The scope of work and its specifics can be as random as possible. Let's say one of your teammates works with SQL database, someone else deals with Amazon API. Thus, each of your colleagues is assigned to do the particular tasks.

Don't reinvent the bike

Currently Node.js community is up and thriving. A lot of neat features are invented and written by other developers already. So before you create a particular functionality for your application, make sure if someone else has not encountered the exact problem before.

If you’re not the only one who has experienced a certain issue, you might find the solution in npm packages. This is an entire catalogue of many ready-made useful libraries that will make life much easier for you.

The same applies to frameworks. Think about whether to use any of them to speed up the process to build a real-time web app with Node JS or a mobile one.

Let's say if you’re dealing with REST API, you can try out Express js. If you need to interact with a particular database type, you can refer to such frameworks, as Mongoose js or SQL depending on which database type you need.

Although packages can benefit your project, there are some significant dangers to be aware of. Given the fact, that these solutions are open source there are several threats to bear in mind:

  • Duplicates

There are too many packages already and some of them clone the others. Unfortunately, this mainstream is only growing. Be careful and make sure you choose the right and unique npm.

  • Malicious code
    Since these packages are not supervised, anyone can write anything they want. Read more about security issues in the article by David Gilbertson ‘I’m harvesting credit card numbers and passwords from your site. Here’s how’. So if your product has to provide AAA-security type check each code snippet of any package you instal meticulously.
Always stay ahead of the time

JS and Node.js community is constantly growing. ES standards are frequently updated. Old features are being replaced by the new, better ones and implemented in Node.js. Thus it’s important to always monitor the technology’s state of art.

For instance,

[calbackHell](http://callbackhell.com/) 
fs.action(source, function (err, res) {
  if (err) {
    console.log('Error: ' + err)
  } else {
     res.acton(function(err, res) {
       console.log('Error s: ' + err)
     })
    })
  }
})

Was replaced with

[promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise)
fs.action(source)
  .then(res => res.action())
  .then(res => res.action())
  .then(res => res.action())
  .catch(err =>  console.log('Error : ' + err))
	

Now we can use async/await as the alternative to Promises:

[async/await](https://blog.risingstack.com/async-await-node-js-7-nightly)
try {
  const res = await fs.action(source);
  const res1 = await res.action(source);
  const res2 = await res1.action(source);
  const res3 = await res2.action(source);
} catch(error) {
   console.log('Error s: ' + err)
}

As for now, community’s opinion whether to use promises or async/await vary.

Try to update your knowledge base with the new Node.js and ES releases on the regular basis. This will help you keep a modernized development process.

Node.js app development techniques and tips

“Deep in the human unconscious is a pervasive need for a logical universe that makes sense. But the real universe is always one step beyond logic.”
― Frank Herbert.

To overcome Node.js architecture limitations and trivial-to-challenging issues, keep to the clear development structure.

Since Node.js project might consist of only one file, this does not mean that you need to pile everything up into one great mess.

Make your code as readable and understandable for others as it could be. The following recommendations :

  • Follow the instructions on the structure development for the framework you are using. If you do not use any, try to place your code in the directories and subdirectories in the most logical way possible.

  • File naming

Keep to a single agreed file naming. For example, choose one of these: ErrorHandler or errorHandler orerror_handle or error-handler. Try to name the files according to their purpose but not according to their functionality. For example, it’s better to name the File NotifyAllUserByEmaisSMSLocal as Notifier.

  • The entry point must not contain unnecessary code lines

The entry point is the main.js,app.js and www files that are requested to launch your application. Such files contain only certain methods or classes calls, but not more.

  • index.js.

Keeping only import / export in these files is in general considered to be a justified practice.

Tips for better Node.js project architecture

The way the code is written signifies the ‘face’ (reputation) of the programmer. It also shows how the entire development team deals with the app’s creation using a certain technology or language. Node.js in our case. Therefore, always try to keep it up-to-date and structured (and comprehensive for other developers). Code readability is one of the main ways to build stable, real time web app with Node.js.

  • Use code quality control tools, like Lint

This tool will help you not to slip out/keep a keen eye on any trivial small error./Give a trivial error no chance. It will also allow you to keep the code in one unified form.

  • Keep track of your files size

Too large files are difficult in guiding and understanding. The optimal file size is of 300-500 lines or less. So if you have noted that the code is constantly growing within the same file, turn it to the directory with several files inside.

  • Comment on your code
    When you write a universal module that will be used in several places, don’t forget to create a quick guide on how to utilize this code.
/**
 * Provide sending notification for users by Local, Email , SMS
 *
 * @example
 *          new Notifier(user).notify(['sms', 'email'], ...)
 **/
class Notifier {

You can also leave instructions for methods in your code

/**
 * Provide parse date for single format on project
 * @param Date
 * @example
 *        dateToString(date) => String
 * **/
 

If you’re developing a particular REST API, you can embed instructions on its usage in the code itself. It’s recommended though, to create the complete documentation/guidelines and store it on a single resource.

/**
* Provide Api for Account
  Account Register  POST /api/v1/account/
  @params
         email {string}
         password {string}
  Account Login  POST /api/v1/account/login
  @params
         email {string}
         password {string}

  Account Logout  GET /api/v1/account/logout
  @header
         Authorization: Bearer {token}
 **/
 

There are 2 sides of a coin though.

To ensure the use of Node.js architecture best practices, keep your code as descriptive and organized as possible but don’t overdo.

Don’t comment each code string. It will do more harm than good and nothing but only make the development process more complex. The better tip is to comment the code snippet’s purpose but not its functionality (what does it do). Depending on the development style (callback, promise, async/await) write and use only one (if it’s possible) general error handler/processor.

Errors handling

Errors handling is another important aspect among other best practices for Node.js development to bear in mind.

Since JavaScript is not as strict as Java all the responsibility lies on the development team.

First and foremost, always try to process the errors. Otherwise it can lead to app’s uncontrolled behavior.

  • With callback
const withoutErrors = calback => (err, updatedTank) => {
  if (err) {
    return // do something
  }
  return calback(updatedTank);
};
fs.action(withoutErrors(data => ...))

With Promise

const handlError = error => {
  if (err) {
    return // do something
  }
};
fs.action()
    .then(data => )
    .then(data => )
    .cattch(handlError)
  • With async / await
class Actions
  async action1 (data) {
    return fs.action(data)
  }
  async action2 (data) {
    return fs.action(data)
  }
  .... 
}

try {
  await new Actions().action1();
  await new Actions().action1();
} catach(error) {
  return handlError(error)
}

Node JS development tools
  • Gulp

A toolkit which allows to launch several apps simultaneously. It might be useful if you’d like to run several services at the same time with one command/request.

  • Nodemon

Hot reload feature for Node.js. This tool automatically updates/ resets your project after any code change is made. A quite handy tool during the Node.js project architecture development.

  • Forever, pm2
    These two packages ensure app’s launch during the (OC) system’s start.

  • Winston
    Provides with the opportunity to record app’s logs to the primary source (file or database). The package comes to help, when you need the app to work remotely and don’t have the full access to it.

  • Threads
    A tool designed for the better work with threads.

To sum up with

We hope that Node.js architecture best practices represented in this article will help you reach the most desireable result when looking for the way to build performant real-time web or mobile apps with Node.js.

Python Tutorial: Image processing with Python (Using OpenCV)

Python Tutorial: Image processing with Python (Using OpenCV)

In this tutorial, you will learn how you can process images in Python using the OpenCV library.

In this tutorial, you will learn how you can process images in Python using the OpenCV library.

OpenCV is a free open source library used in real-time image processing. It’s used to process images, videos, and even live streams, but in this tutorial, we will process images only as a first step. Before getting started, let’s install OpenCV.

Table of Contents

Install OpenCV

To install OpenCV on your system, run the following pip command:

 pip install opencv-python

Now OpenCV is installed successfully and we are ready. Let’s have some fun with some images!

Rotate an Image

First of all, import the cv2 module.

 import cv2

Now to read the image, use the imread() method of the cv2 module, specify the path to the image in the arguments and store the image in a variable as below:

 img = cv2.imread("pyimg.jpg")

The image is now treated as a matrix with rows and columns values stored in img.

Actually, if you check the type of the img, it will give you the following result:

>>>print(type(img))
 
<class 'numpy.ndarray'>

It’s a NumPy array! That why image processing using OpenCV is so easy. All the time you are working with a NumPy array.

To display the image, you can use the imshow() method of cv2.

cv2.imshow('Original Image', img) 
 
cv2.waitKey(0)

The waitkey functions take time as an argument in milliseconds as a delay for the window to close. Here we set the time to zero to show the window forever until we close it manually.

To rotate this image, you need the width and the height of the image because you will use them in the rotation process as you will see later.

 height, width = img.shape[0:2]

The shape attribute returns the height and width of the image matrix. If you print img.shape[0:2] , you will have the following output:

Okay, now we have our image matrix and we want to get the rotation matrix. To get the rotation matrix, we use the getRotationMatrix2D() method of cv2. The syntax of getRotationMatrix2D() is:

 cv2.getRotationMatrix2D(center, angle, scale)

Here the center is the center point of rotation, the angle is the angle in degrees and scale is the scale property which makes the image fit on the screen.

To get the rotation matrix of our image, the code will be:

 rotationMatrix = cv2.getRotationMatrix2D((width/2, height/2), 90, .5)

The next step is to rotate our image with the help of the rotation matrix.

To rotate the image, we have a cv2 method named wrapAffine which takes the original image, the rotation matrix of the image and the width and height of the image as arguments.

 rotatedImage = cv2.warpAffine(img, rotationMatrix, (width, height))

The rotated image is stored in the rotatedImage matrix. To show the image, use imshow() as below:

cv2.imshow('Rotated Image', rotatedImage)
 
cv2.waitKey(0)

After running the above lines of code, you will have the following output:

Crop an Image

First, we need to import the cv2 module and read the image and extract the width and height of the image:

import cv2
 
img = cv2.imread("pyimg.jpg")
 
height, width = img.shape[0:2]

Now get the starting and ending index of the row and column. This will define the size of the newly created image. For example, start from row number 10 till row number 15 will give the height of the image.

Similarly, start from column number 10 until column number 15 will give the width of the image.

You can get the starting point by specifying the percentage value of the total height and the total width. Similarly, to get the ending point of the cropped image, specify the percentage values as below:

startRow = int(height*.15)
 
startCol = int(width*.15)
 
endRow = int(height*.85)
 
endCol = int(width*.85)

Now map these values to the original image. Note that you have to cast the starting and ending values to integers because when mapping, the indexes are always integers.

 croppedImage = img[startRow:endRow, startCol:endCol]

Here we specified the range from starting to ending of rows and columns.

Now display the original and cropped image in the output:

cv2.imshow('Original Image', img)
 
cv2.imshow('Cropped Image', croppedImage)
 
cv2.waitKey(0)

The result will be as follows:

Resize an Image

To resize an image, you can use the resize() method of openCV. In the resize method, you can either specify the values of x and y axis or the number of rows and columns which tells the size of the image.

Import and read the image:

import cv2
 
img = cv2.imread("pyimg.jpg")

Now using the resize method with axis values:

newImg = cv2.resize(img, (0,0), fx=0.75, fy=0.75)
 
cv2.imshow('Resized Image', newImg)
 
cv2.waitKey(0)

The result will be as follows:

Now using the row and column values to resize the image:

newImg = cv2.resize(img, (550, 350))
 
cv2.imshow('Resized Image', newImg)
 
cv2.waitKey(0)

We say we want 550 columns (the width) and 350 rows (the height).

The result will be:

Adjust Image Contrast

In Python OpenCV module, there is no particular function to adjust image contrast but the official documentation of OpenCV suggests an equation that can perform image brightness and image contrast both at the same time.

 new_img = a * original_img + b

Here a is alpha which defines contrast of the image. If a is greater than 1, there will be higher contrast.

If the value of a is between 0 and 1 (smaller than 1 but greater than 0), there would be lower contrast. If a is 1, there will be no contrast effect on the image.

b stands for beta. The values of b vary from -127 to +127.

To implement this equation in Python OpenCV, you can use the addWeighted() method. We use The addWeighted() method as it generates the output in the range of 0 and 255 for a 24-bit color image.

The syntax of addWeighted() method is as follows:

 cv2.addWeighted(source_img1, alpha1, source_img2, alpha2, beta)

This syntax will blend two images, the first source image (source_img1) with a weight of alpha1 and second source image (source_img2).

If you only want to apply contrast in one image, you can add a second image source as zeros using NumPy.

Let’s work on a simple example. Import the following modules:

import cv2
 
import numpy as np

Read the original image:

 img = cv2.imread("pyimg.jpg")

Now apply the contrast. Since there is no other image, we will use the np.zeros which will create an array of the same shape and data type as the original image but the array will be filled with zeros.

contrast_img = cv2.addWeighted(img, 2.5, np.zeros(img.shape, img.dtype), 0, 0)
 
cv2.imshow('Original Image', img)
 
cv2.imshow('Contrast Image', contrast_img)
 
cv2.waitKey(0)

In the above code, the brightness is set to 0 as we only want to apply contrast.

The comparison of the original and contrast image is as follows:

Make an image blurry

Gaussian Blur

To make an image blurry, you can use the GaussianBlur() method of OpenCV.

The GaussianBlur() uses the Gaussian kernel. The height and width of the kernel should be a positive and an odd number.

Then you have to specify the X and Y direction that is sigmaX and sigmaY respectively. If only one is specified, both are considered the same.

Consider the following example:

import cv2
 
img = cv2.imread("pyimg.jpg")
 
blur_image = cv2.GaussianBlur(img, (7,7), 0)
 
cv2.imshow('Original Image', img)
 
cv2.imshow('Blur Image', blur_image)
 
cv2.waitKey(0)

In the above snippet, the actual image is passed to GaussianBlur() along with height and width of the kernel and the X and Y directions.

The comparison of the original and blurry image is as follows:

Median Blur

In median blurring, the median of all the pixels of the image is calculated inside the kernel area. The central value is then replaced with the resultant median value. Median blurring is used when there are salt and pepper noise in the image.

To apply median blurring, you can use the medianBlur() method of OpenCV.

Consider the following example where we have a salt and pepper noise in the image:

import cv2
 
img = cv2.imread("pynoise.png")
 
blur_image = cv2.medianBlur(img,5)

This will apply 50% noise in the image along with median blur. Now show the images:

cv2.imshow('Original Image', img)
 
cv2.imshow('Blur Image', blur_image)
 
cv2.waitKey(0)

The result will be like the following:

Another comparison of the original image and after blurring:

Detect Edges

To detect the edges in an image, you can use the Canny() method of cv2 which implements the Canny edge detector. The Canny edge detector is also known as the optimal detector.

The syntax to Canny() is as follows:

 cv2.Canny(image, minVal, maxVal)

Here minVal and maxVal are the minimum and maximum intensity gradient values respectively.

Consider the following code:

import cv2
 
img = cv2.imread("pyimg.jpg")
 
edge_img = cv2.Canny(img,100,200)
 
cv2.imshow("Detected Edges", edge_img)
 
cv2.waitKey(0)

The output will be the following:

Here is the result of the above code on another image:

Convert image to grayscale (Black & White)

The easy way to convert an image in grayscale is to load it like this:

 img = cv2.imread("pyimg.jpg", 0)

There is another method using BGR2GRAY.

To convert a color image into a grayscale image, use the BGR2GRAY attribute of the cv2 module. This is demonstrated in the example below:

Import the cv2 module:

 import cv2

Read the image:

 img = cv2.imread("pyimg.jpg")

Use the cvtColor() method of the cv2 module which takes the original image and the COLOR_BGR2GRAY attribute as an argument. Store the resultant image in a variable:

 gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

Display the original and grayscale images:

cv2.imshow("Original Image", img)
 
cv2.imshow("Gray Scale Image", gray_img)
 
cv2.waitKey(0)

The output will be as follows:

Centroid (Center of blob) detection

To find the center of an image, the first step is to convert the original image into grayscale. We can use the cvtColor() method of cv2 as we did before.

This is demonstrated in the following code:

import cv2
 
img = cv2.imread("py.jpg")
 
gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

We read the image and convert it to a grayscale image. The new image is stored in gray_img.

Now we have to calculate the moments of the image. Use the moments() method of cv2. In the moments() method, the grayscale image will be passed as below:

 moment = cv2.moments(gray_img)

Finally, we have the center of the image. To highlight this center position, we can use the circle method which will create a circle in the given coordinates of the given radius.

The circle() method takes the img, the x and y coordinates where the circle will be created, the size, the color that we want the circle to be and the thickness.

 cv2.circle(img, (X, Y), 15, (205, 114, 101), 1)

The circle is created on the image.

cv2.imshow("Center of the Image", img)
 
cv2.waitKey(0)

The original image is:

After detecting the center, our image will be as follows:

Apply a mask for a colored image

Image masking means to apply some other image as a mask on the original image or to change the pixel values in the image.

To apply a mask on the image, we will use the HoughCircles() method of the OpenCV module. The HoughCircles() method detects the circles in an image. After detecting the circles, we can simply apply a mask on these circles.

The HoughCircles() method takes the original image, the Hough Gradient (which detects the gradient information in the edges of the circle), and the information from the following circle equation:

 (x - xcenter)2 + (y - ycenter)2 = r2

In this equation (xcenter , ycenter) is the center of the circle and r is the radius of the circle.

Our original image is:

After detecting circles in the image, the result will be:

Okay, so we have the circles in the image and we can apply the mask. Consider the following code:

import cv2
 
import numpy as np
 
img1 = cv2.imread('pyimg.jpg')
 
img1 = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

Detecting the circles in the image using the HoughCircles() code from OpenCV: Hough Circle Transform:

gray_img = cv2.medianBlur(cv2.cvtColor(img, cv2.COLOR_RGB2GRAY), 3)
 
circles = cv2.HoughCircles(gray_img, cv2.HOUGH_GRADIENT, 1, 20, param1=50, param2=50, minRadius=0, maxRadius=0)
 
circles = np.uint16(np.around(circles))

To create the mask, use np.full which will return a NumPy array of given shape:

masking=np.full((img1.shape[0], img1.shape[1]),0,dtype=np.uint8)
 
for j in circles[0, :]:
 
    cv2.circle(masking, (j[0], j[1]), j[2], (255, 255, 255), -1)

The next step is to combine the image and the masking array we created using the bitwise_or operator as follows:

 final_img = cv2.bitwise_or(img1, img1, masking=masking)

Display the resultant image:

Extracting text from Image (OCR)

To extract text from an image, you can use Google Tesseract-OCR. You can download it from this link

Then you should install the pytesseract module which is a Python wrapper for Tesseract-OCR.

The image from which we will extract the text from is as follows:

Now let’s convert the text in this image to a string of characters and display the text as a string on output:

Import the pytesseract module:

 import pytesseract

Set the path of the Tesseract-OCR executable file:

 pytesseract.pytesseract.tesseract_cmd = r'C:\Program Files (x86)\Tesseract-OCR\tesseract'

Now use the image_to_string method to convert the image into a string:

 print(pytesseract.image_to_string('pytext.png'))

The output will be as follows:

Works like charm!

Detect and correct text skew

In this section, we will correct the text skew.

The original image is as follows:

Import the modules cv2, NumPy and read the image:

import cv2
 
import numpy as np
 
img = cv2.imread("pytext1.png")

Convert the image into a grayscale image:

 gray_img=cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

Invert the grayscale image using bitwise_not:

 gray_img=cv2.bitwise_not(gray_img)

Select the x and y coordinates of the pixels greater than zero by using the column_stack method of NumPy:

 coordinates = np.column_stack(np.where(gray_img > 0))

Now we have to calculate the skew angle. We will use the minAreaRect() method of cv2 which returns an angle range from -90 to 0 degrees (where 0 is not included).

 ang=cv2.minAreaRect(coordinates)[-1]

The rotated angle of the text region will be stored in the ang variable. Now we add a condition for the angle; if the text region’s angle is smaller than -45, we will add a 90 degrees else we will multiply the angle with a minus to make the angle positive.

if ang<-45:
 
    ang=-(90+ang)
 
else:
 
    ang=-ang

Calculate the center of the text region:

height, width = img.shape[:2]
 
center_img = (width / 2, height / 2)

Now we have the angle of text skew, we will apply the getRotationMatrix2D() to get the rotation matrix then we will use the wrapAffine() method to rotate the angle (explained earlier).

rotationMatrix = cv2.getRotationMatrix2D(center, angle, 1.0)
 
rotated_img = cv2.warpAffine(img, rotationMatrix, (width, height), borderMode = cv2.BORDER_REFLECT)

Display the rotated image:

cv2.imshow("Rotated Image", rotated_img)
 
cv2.waitKey(0)

Color Detection

Let’s detect the green color from an image:

Import the modules cv2 for images and NumPy for image arrays:

import cv2
 
import numpy as np

Read the image and convert it into HSV using cvtColor():

img = cv2.imread("pydetect.png")
 
hsv_img = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

Display the image:

 cv2.imshow("HSV Image", hsv_img)

Now create a NumPy array for the lower green values and the upper green values:

lower_green = np.array([34, 177, 76])
 
upper_green = np.array([255, 255, 255])

Use the inRange() method of cv2 to check if the given image array elements lie between array values of upper and lower boundaries:

 masking = cv2.inRange(hsv_img, lower_green, upper_green)

This will detect the green color.

Finally, display the original and resultant images:

 cv2.imshow("Original Image", img)

cv2.imshow("Green Color detection", masking)
 
cv2.waitKey(0)

Reduce Noise

To reduce noise from an image, OpenCV provides the following methods:

  1. fastNlMeansDenoising(): Removes noise from a grayscale image
  2. fastNlMeansDenoisingColored(): Removes noise from a colored image
  3. fastNlMeansDenoisingMulti(): Removes noise from grayscale image frames (a grayscale video)
  4. fastNlMeansDenoisingColoredMulti(): Same as 3 but works with colored frames

Let’s use fastNlMeansDenoisingColored() in our example:

Import the cv2 module and read the image:

2
3
	
import cv2
 
img = cv2.imread("pyn1.png")

Apply the denoising function which takes respectively the original image (src), the destination (which we have kept none as we are storing the resultant), the filter strength, the image value to remove the colored noise (usually equal to filter strength or 10), the template patch size in pixel to compute weights which should always be odd (recommended size equals 7) and the window size in pixels to compute average of the given pixel.

 result = cv2.fastNlMeansDenoisingColored(img,None,20,10,7,21)

Display original and denoised image:

cv2.imshow("Original Image", img)
 
cv2.imshow("Denoised Image", result)
 
cv2.waitKey(0)

The output will be:

Get image contour

Contours are the curves in an image that are joint together. The curves join the continuous points in an image. The purpose of contours is used to detect the objects.

The original image of which we are getting the contours of is given below:

Consider the following code where we used the findContours() method to find the contours in the image:

Import cv2 module:

 import cv2

Read the image and convert it to a grayscale image:

img = cv2.imread('py1.jpg')
 
gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

Find the threshold:

 retval, thresh = cv2.threshold(gray_img, 127, 255, 0)

Use the findContours() which takes the image (we passed threshold here) and some attributes. See findContours() Official.

 img_contours, _ = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

Draw the contours on the image using drawContours() method:

  cv2.drawContours(img, img_contours, -1, (0, 255, 0))

Display the image:

cv2.imshow('Image Contours', img)
 
cv2.waitKey(0)

The result will be:

Remove Background from an image

To remove the background from an image, we will find the contours to detect edges of the main object and create a mask with np.zeros for the background and then combine the mask and the image using the bitwise_and operator.

Consider the example below:

Import the modules (NumPy and cv2):

import cv2
 
import numpy as np

Read the image and convert the image into a grayscale image:

img = cv2.imread("py.jpg")
 
gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

Find the threshold:

 _, thresh = cv2.threshold(gray_img, 127, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)

In the threshold() method, the last argument defines the style of the threshold. See Official documentation of OpenCV threshold.

Find the image contours:

 img_contours = cv2.findContours(threshed, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)[-2]

Sort the contours:

img_contours = sorted(img_contours, key=cv2.contourArea)
 
for i in img_contours:
 
    if cv2.contourArea(i) > 100:
 
        break

Generate the mask using np.zeros:

 mask = np.zeros(img.shape[:2], np.uint8)

Draw contours:

 cv2.drawContours(mask, [i],-1, 255, -1)

Apply the bitwise_and operator:

 new_img = cv2.bitwise_and(img, img, mask=mask)

Display the original image:

 cv2.imshow("Original Image", img)

Display the resultant image:

cv2.imshow("Image with background removed", new_img)
 
cv2.waitKey(0)

Image processing is fun when using OpenCV as you saw. I hope you find the tutorial useful. Keep coming back.

Thank you.

How to Set up an SMS Notification With Python

How to Set up an SMS Notification With Python

How to Set up an SMS Notification With Python. oday I am beginning a new series of posts specifically aimed at Python beginners.

Hi everyone :) Today I am beginning a new series of posts specifically aimed at Python beginners. The concept is rather simple: I'll do a fun project, in as few lines of code as possible, and will try out as many new tools as possible.

For example, today we will learn to use the Twilio API, the Twitch API, and we'll see how to deploy the project on Heroku. I'll show you how you can have your own "Twitch Live" SMS notifier, in 30 lines of codes, and for 12 cents a month.

Prerequisite: You only need to know how to run Python on your machine and some basic commands in git (commit & push). If you need help with these, I can recommend these 2 articles to you:

Python 3 Installation & Setup Guide

The Ultimate Git Command Tutorial for Beginners from Adrian Hajdin.

What you'll learn:

  • Twitch API
  • Twilio API
  • Deploying on Heroku
  • Setting up a scheduler on Heroku

What you will build:

The specifications are simple: we want to receive an SMS as soon as a specific Twitcher is live streaming. We want to know when this person is going live and when they leave streaming. We want this whole thing to run by itself, all day long.

We will split the project into 3 parts. First, we will see how to programmatically know if a particular Twitcher is online. Then we will see how to receive an SMS when this happens. We will finish by seeing how to make this piece of code run every X minutes, so we never miss another moment of our favorite streamer's life.

Is this Twitcher live?

To know if a Twitcher is live, we can do two things: we can go to the Twitcher URL and try to see if the badge "Live" is there.

Screenshot of a Twitcher live streaming.

This process involves scraping and is not easily doable in Python in less than 20 or so lines of code. Twitch runs a lot of JS code and a simple request.get() won't be enough.

For scraping to work, in this case, we would need to scrape this page inside Chrome to get the same content like what you see in the screenshot. This is doable, but it will take much more than 30 lines of code. If you'd like to learn more, don't hesitate to check my recent web scraping guide.

So instead of trying to scrape Twitch, we will use their API. For those unfamiliar with the term, an API is a programmatic interface that allows websites to expose their features and data to anyone, mainly developers. In Twitch's case, their API is exposed through HTTP, witch means that we can have lots of information and do lots of things by just making a simple HTTP request.

Get your API key

To do this, you have to first create a Twitch API key. Many services enforce authentication for their APIs to ensure that no one abuses them or to restrict access to certain features by certain people.

Please follow these steps to get your API key:

  • Create a Twitch account
  • Now create a Twitch dev account -> "Signing up with Twitch" top right
  • Go to your "dashboard" once logged in
  • "Register your application"
  • Name -> Whatever, Oauth redirection URL -> http://localhost, Category -> Whatever

You should now see, at the bottom of your screen, your client-id. Keep this for later.

Is that Twitcher streaming now?

With your API key in hand, we can now query the Twitch API to have the information we want, so let's begin to code. The following snippet just consumes the Twitch API with the correct parameters and prints the response.

# requests is the go to package in python to make http request
# https://2.python-requests.org/en/master/
import requests

# This is one of the route where Twich expose data, 
# They have many more: https://dev.twitch.tv/docs
endpoint = "https://api.twitch.tv/helix/streams?"

# In order to authenticate we need to pass our api key through header
headers = {"Client-ID": "<YOUR-CLIENT-ID>"}

# The previously set endpoint needs some parameter, here, the Twitcher we want to follow
# Disclaimer, I don't even know who this is, but he was the first one on Twich to have a live stream so I could have nice examples
params = {"user_login": "Solary"}

# It is now time to make the actual request
response = request.get(endpoint, params=params, headers=headers)
print(response.json())

The output should look like this:

{
   'data':[
      {
         'id':'35289543872',
         'user_id':'174955366',
         'user_name':'Solary',
         'game_id':'21779',
         'type':'live',
         'title':"Wakz duoQ w/ Tioo - GM 400LP - On récupère le chall après les -250LP d'inactivité !",
         'viewer_count':4073,
         'started_at':'2019-08-14T07:01:59Z',
         'language':'fr',
         'thumbnail_url':'https://static-cdn.jtvnw.net/previews-ttv/live_user_solary-{width}x{height}.jpg',
         'tag_ids':[
            '6f655045-9989-4ef7-8f85-1edcec42d648'
         ]
      }
   ],
   'pagination':{
      'cursor':'eyJiIjpudWxsLCJhIjp7Ik9mZnNldCI6MX19'
   }
}

This data format is called JSON and is easily readable. The data object is an array that contains all the currently active streams. The key type ensures that the stream is currently live. This key will be empty otherwise (in case of an error, for example).

So if we want to create a boolean variable in Python that stores whether the current user is streaming, all we have to append to our code is:

json_response = response.json()

# We get only streams
streams = json_response.get('data', [])

# We create a small function, (a lambda), that tests if a stream is live or not
is_active = lambda stream: stream.get('type') == 'live'
# We filter our array of streams with this function so we only keep streams that are active
streams_active = filter(is_active, streams)

# any returns True if streams_active has at least one element, else False
at_least_one_stream_active = any(streams_active)

print(at_least_one_stream_active)

At this point, at_least_one_stream_active is True when your favourite Twitcher is live.

Let's now see how to get notified by SMS.

Send me a text, NOW!

So to send a text to ourselves, we will use the Twilio API. Just go over there and create an account. When asked to confirm your phone number, please use the phone number you want to use in this project. This way you'll be able to use the $15 of free credit Twilio offers to new users. At around 1 cent a text, it should be enough for your bot to run for one year.

If you go on the console, you'll see your Account SID and your Auth Token , save them for later. Also click on the big red button "Get My Trial Number", follow the step, and save this one for later too.

Sending a text with the Twilio Python API is very easy, as they provide a package that does the annoying stuff for you. Install the package with pip install Twilio and just do:

from twilio.rest import Client
client = Client(<Your Account SID>, <Your Auth Token>)
client.messages.create(
	body='Test MSG',from_=<Your Trial Number>,to=<Your Real Number>)

And that is all you need to send yourself a text, amazing right?

Putting everything together

We will now put everything together, and shorten the code a bit so we manage to say under 30 lines of Python code.

import requests
from twilio.rest import Client
endpoint = "https://api.twitch.tv/helix/streams?"
headers = {"Client-ID": "<YOUR-CLIENT-ID>"}
params = {"user_login": "Solary"}
response = request.get(endpoint, params=params, headers=headers)
json_response = response.json()
streams = json_response.get('data', [])
is_active = lambda stream:stream.get('type') == 'live'
streams_active = filter(is_active, streams)
at_least_one_stream_active = any(streams_active)
if at_least_one_stream_active:
    client = Client(<Your Account SID>, <Your Auth Token>)
	client.messages.create(body='LIVE !!!',from_=<Your Trial Number>,to=<Your Real Number>)

Avoiding double notifications

This snippet works great, but should that snippet run every minute on a server, as soon as our favorite Twitcher goes live we will receive an SMS every minute.

We need a way to store the fact that we were already notified that our Twitcher is live and that we don't need to be notified anymore.

The good thing with the Twilio API is that it offers a way to retrieve our message history, so we just have to retrieve the last SMS we sent to see if we already sent a text notifying us that the twitcher is live.

Here what we are going do to in pseudocode:

if favorite_twitcher_live and last_sent_sms is not live_notification:
	send_live_notification()
if not favorite_twitcher_live and last_sent_sms is live_notification:
	send_live_is_over_notification()

This way we will receive a text as soon as the stream starts, as well as when it is over. This way we won't get spammed - perfect right? Let's code it:

# reusing our Twilio client
last_messages_sent = client.messages.list(limit=1)
last_message_id = last_messages_sent[0].sid
last_message_data = client.messages(last_message_id).fetch()
last_message_content = last_message_data.body

Let's now put everything together again:

import requests
from twilio.rest import Client
client = Client(<Your Account SID>, <Your Auth Token>)

endpoint = "https://api.twitch.tv/helix/streams?"
headers = {"Client-ID": "<YOUR-CLIENT-ID>"}
params = {"user_login": "Solary"}
response = request.get(endpoint, params=params, headers=headers)
json_response = response.json()
streams = json_response.get('data', [])
is_active = lambda stream:stream.get('type') == 'live'
streams_active = filter(is_active, streams)
at_least_one_stream_active = any(streams_active)

last_messages_sent = client.messages.list(limit=1)
if last_messages_sent:
	last_message_id = last_messages_sent[0].sid
	last_message_data = client.messages(last_message_id).fetch()
	last_message_content = last_message_data.body
    online_notified = "LIVE" in last_message_content
    offline_notified = not online_notified
else:
	online_notified, offline_notified = False, False

if at_least_one_stream_active and not online_notified:
	client.messages.create(body='LIVE !!!',from_=<Your Trial Number>,to=<Your Real Number>)
if not at_least_one_stream_active and not offline_notified:
	client.messages.create(body='OFFLINE !!!',from_=<Your Trial Number>,to=<Your Real Number>)

And voilà!

You now have a snippet of code, in less than 30 lines of Python, that will send you a text a soon as your favourite Twitcher goes Online / Offline and without spamming you.

We just now need a way to host and run this snippet every X minutes.

The quest for a host

To host and run this snippet we will use Heroku. Heroku is honestly one of the easiest ways to host an app on the web. The downside is that it is really expensive compared to other solutions out there. Fortunately for us, they have a generous free plan that will allow us to do what we want for almost nothing.

If you don't already, you need to create a Heroku account. You also need to download and install the Heroku client.

You now have to move your Python script to its own folder, don't forget to add a requirements.txt file in it. The content of the latter begins:

requests
twilio

This is to ensure that Heroku downloads the correct dependencies.

cd into this folder and just do a heroku create --app <app name>.

If you go on your app dashboard you'll see your new app.

We now need to initialize a git repo and push the code on Heroku:

git init
heroku git:remote -a <app name>
git add .
git commit -am 'Deploy breakthrough script'
git push heroku master

Your app is now on Heroku, but it is not doing anything. Since this little script can't accept HTTP requests, going to <app name>.herokuapp.com won't do anything. But that should not be a problem.

To have this script running 24/7 we need to use a simple Heroku add-on call "Heroku Scheduler". To install this add-on, click on the "Configure Add-ons" button on your app dashboard.

Then, on the search bar, look for Heroku Scheduler:

Click on the result, and click on "Provision"

If you go back to your App dashboard, you'll see the add-on:

Click on the "Heroku Scheduler" link to configure a job. Then click on "Create Job". Here select "10 minutes", and for run command select python <name_of_your_script>.py. Click on "Save job".

While everything we used so far on Heroku is free, the Heroku Scheduler will run the job on the $25/month instance, but prorated to the second. Since this script approximately takes 3 seconds to run, for this script to run every 10 minutes you should just have to spend 12 cents a month.

Ideas for improvements

I hope you liked this project and that you had fun putting it into place. In less than 30 lines of code, we did a lot, but this whole thing is far from perfect. Here are a few ideas to improve it:

  • Send yourself more information about the current streaming (game played, number of viewers ...)
  • Send yourself the duration of the last stream once the twitcher goes offline
  • Don't send you a text, but rather an email
  • Monitor multiple twitchers at the same time

Do not hesitate to tell me in the comments if you have more ideas.

Conclusion

I hope that you liked this post and that you learned things reading it. I truly believe that this kind of project is one of the best ways to learn new tools and concepts, I recently launched a web scraping API where I learned a lot while making it.

Please tell me in the comments if you liked this format and if you want to do more.

I have many other ideas, and I hope you will like them. Do not hesitate to share what other things you build with this snippet, possibilities are endless.

Happy Coding.

Pierre

Don't want to miss my next post:

You can subscribe here to my newsletter.

Introduction to Python Hex() Function for Beginners

Introduction to Python Hex() Function for Beginners

Python hex() function is used to convert any integer number ( in base 10) to the corresponding hexadecimal number. Notably, the given input should be in base 10. Python hex function is one of the built-in functions in Python3, which is used to convert an integer number into its corresponding hexadecimal form.

Python hex() function is used to convert any integer number ( in base 10) to the corresponding hexadecimal number. Notably, the given input should be in base 10. Python hex function is one of the built-in functions in Python3, which is used to convert an integer number into its corresponding hexadecimal form.

##Python Hex Example
The hex() function converts the integer to the corresponding hexadecimal number in string form and returns it.

The input integer argument can be in any base such as binary, octal, etc. Python will take care of converting them to hexadecimal format.

Syntax

hex(number)

number: It is an integer that will be converted into hexadecimal value.
This function converts the number into the hexadecimal form, and then it returns that hexadecimal number in a string format.

Please note that the return value always starts with ‘0x’ (without quotes), which proves that the number is in hexadecimal format.

# app.py

print("Enter the number: ")

# taking input from user
num = int(input())

# converting the number into hexadecimal form
h1 = hex(num)

# Printing hexadecimal form
print("The ", num, " in hexadecimal is: ", h1)

# Converting float number to hexadecimal form
print("\nEnter a float number")
num2 = float(input())

# converting into hexadecimal form
# for float we have to use float.hex() here
h2 = float.hex(num2)

# printing result
print("The ", num2, " in hexadecimal is: ", h2)

In the above example, we used the Python input() function to take the input from the user.

See the output.

Enter the number:
541
The  541  in hexadecimal is:  0x21d
    
Enter a float number
123.54
The  123.54  in hexadecimal is:  0x1.ee28f5c28f5c3p+6

Python hex() without 0x

See the following program.

# app.py

print("Enter the number: ")

# taking input from user
num = int(input())

# converting the number into hexadecimal form
h1 = hex(num)

# Printing hexadecimal form
# we have used string slicing here
print("The ", num, " in hexadecimal is: ", h1[2:])

# Converting float number to hexadecimal form
print("\nEnter a float number")
num2 = float(input())

# converting into hexadecimal form
h2 = float.hex(num2)

# printing result
print("The ", num2, " in hexadecimal is: ", h2[2:])

See the output.

Enter the number:
541
The  541  in hexadecimal is:  21d

Enter a float number
123.65
The  123.65  in hexadecimal is:  1.ee9999999999ap+6

On the above program, we have used string slicing to print the result without ‘0x’.

We have started our index from position 2 to the last of the string, i.e., h1[2:]; this means the string will print characters from position 2 to the last of the string.

Hexadecimal representation of float in Python

See the following program.

# app.py

numberEL = 11.21
print(numberEL, 'in hex =', float.hex(numberEL))

numberK = 19.21
print(numberK, 'in hex =', float.hex(numberK))

See the output.

➜  pyt python3 app.py
11.21 in hex = 0x1.66b851eb851ecp+3
19.21 in hex = 0x1.335c28f5c28f6p+4
➜  pyt

Python hex() with object

See the following code.

# app.py

class AI:
    id = 0

    def __index__(self):
        print('__index__() function called')
        return self.rank


stockfish = AI()
stockfish.rank = 2900

print(hex(stockfish))

In the above example, we have used the index() method so that we can use it with hex() function.

See the output.

➜  pyt python3 app.py
__index__() function called
0xb54
➜  pyt

How to convert hex string to int in Python

Without the 0x prefix, you need to specify the base explicitly. Otherwise, it won’t work.

See the following code.

# app.py

data = int("0xa", 16)
print(data)

With the 0x prefix, Python can distinguish hex and decimal automatically.

You must specify 0 as the base to invoke this prefix-guessing behavior, omitting the second parameter means to assume base-10.)
If you want to convert the string to an int, pass the string to int along with a base you are converting from. Both strings will suffice for conversion in this way.

# app.py

hexStrA = "0xffff"
hexStrB = "ffff"

print(int(hexStrA, 16))
print(int(hexStrB, 16))

See the output.

➜  pyt python3 app.py
65535
65535
➜  pyt

In the all above examples, we have used Python int() method.

Thanks for reading !