JavaScript for Machine Learning using TensorFlow.js

JavaScript for Machine Learning using TensorFlow.js

JavaScript for Machine Learning using TensorFlow.js - Although Python or R programming language has a relatively easy learning curve, web developers are just happy to do everything within their comfort zone of JavaScript.

Originally published by Priyesh Patel at blog.bitsrc.io

Considering the trend that Node.js has started, of applying JavaScript to every field, I decided to understand the concepts of machine learning using JS. Python became popular because of the abundance of packages available, but the JS community is not far behind. In this article, we will build a simple classifier in a beginner friendly process.

What you will build

You will make a webpage that uses TensorFlow.js to train a model in the browser. Given “AvgAreaNumberofRooms” for a house, the model will learn to predict “price” of the house.

To do this you will:

  • Load the data and prepare it for training.
  • Define the architecture of the model.
  • Train the model and monitor its performance as it trains.
  • Evaluate the trained model by making some predictions.

Step 1: Let us start with the basics.

<!DOCTYPE html>
<html>
<head>
  <title>TensorFlow.js Tutorial</title>
  <!-- Import TensorFlow.js -->
  <script src="https://cdn.jsdelivr.net/npm/@tensorflow/[email protected]/dist/tf.min.js"></script>
  <!-- Import tfjs-vis -->
  <script src="https://cdn.jsdelivr.net/npm/@tensorflow/[email protected]/dist/tfjs-vis.umd.min.js"></script>
  <!-- Import the main script file -->
  <script src="script.js"></script>
</head>
<body>
</body>
</html>

Create the JavaScript file for the code

In the same folder as the HTML file above, create a file called script.js and put the following code in it.

console.log('Hello TensorFlow');

Test it out

Now that you’ve got the HTML and JavaScript files created, test them out. Open up the index.html file in your browser and open up the devtools console.

If everything is working, there should be two global variables created and available in the devtools console.:

  • tf is a reference to the TensorFlow.js library
  • tfvis is a reference to the tfjs-vis library

You should see a message that says Hello TensorFlow. “If so, you are ready to move on to the next step.


Step 2: Load the data, format the data and visualize the input data.

We will load the “house” data-set, which can be found here. It contains many different features for a given house. For this tutorial, we only want data about average area number of rooms and the price of each house.

Add the following code to your script.js file.

async function getData() {
  const houseDataReq = await fetch('https://raw.githubusercontent.com/meetnandu05/ml1/master/house.json');  
  const houseData = await houseDataReq.json();  
  const cleaned = houseData.map(house => ({
    price: house.Price,
    rooms: house.AvgAreaNumberofRooms,
  }))
  .filter(house => (house.price != null && house.rooms != null));

return cleaned;
}

This will also remove any entries that do not have either price or number of rooms defined. Let’s also plot this data in a scatterplot to see what it looks like.

Add the following code to the bottom of your script.js file.

async function run() {
// Load and plot the original input data that we are going to train on.
const data = await getData();
const values = data.map(d => ({
x: d.rooms,
y: d.price,
}));
tfvis.render.scatterplot(
{name: 'No.of rooms v Price'},
{values},
{
xLabel: 'No. of rooms',
yLabel: 'Price',
height: 300
}
);
// More code will be added below
}
document.addEventListener('DOMContentLoaded', run);

When you refresh the page. You should see a panel on the left-hand side of the page with a scatterplot of the data. It should look something like this.

Generally, when working with data it is a good idea to find ways to take a look at your data and clean it if necessary. Visualizing the data can give us a sense of whether there is any structure to the data that the model can learn.

We can see from the plot above that there is a positive correlation between no. of rooms and price, i.e. as no. of rooms increases, prices of houses generally increase.

Step 3: Build the model to be trained.

Here, we will write the code to build our machine learning model. The model will be trained based on this code and hence this is an important step. Machine Learning models take inputs and then produce outputs. In the case of Tensorflow.js, we have to build neural networks.

Add the following function to your script.js file to define the model.

function createModel() {
// Create a sequential model
const model = tf.sequential();

// Add a single hidden layer
model.add(tf.layers.dense({inputShape: [1], units: 1, useBias: true}));

// Add an output layer
model.add(tf.layers.dense({units: 1, useBias: true}));
return model;
}

This is one of the simplest models we can define in tensorflow.js, let us break-down each line a bit.

Instantiate the model

const model = tf.sequential();

This instantiates a [tf.Model](https://js.tensorflow.org/api/latest/#class:Model) object. This model is [sequential](https://js.tensorflow.org/api/latest/#sequential) because its inputs flow straight down to its output. Other kinds of models can have branches, or even multiple inputs and outputs, but in many cases your models will be sequential.

Add layers

model.add(tf.layers.dense({inputShape: [1], units: 1, useBias: true}));

This adds a hidden layer to our network. As this is the first layer of the network, we need to define our inputShape. The inputShape is [1] because we have 1number as our input (the number of rooms of a given house).

units sets how big the weight matrix will be in the layer. By setting it to 1 here we are saying there will be 1 weight for each of the input features of the data.

model.add(tf.layers.dense({units: 1}));

The code above creates our output layer. We set units to 1 because we want to output 1 number.

Create an instance

Add the following code to the run function we defined earlier.

// Create the model
const model = createModel();
tfvis.show.modelSummary({name: 'Model Summary'}, model);

This will create an instance of the model and show a summary of the layers on the webpage.

Step 4: Prepare the data for training

To get the performance benefits of TensorFlow.js that make training machine learning models practical, we need to convert our data to tensors.

Add the following code to your script.js file

function convertToTensor(data) {

return tf.tidy(() => {
// Step 1. Shuffle the data
tf.util.shuffle(data);
// Step 2. Convert data to Tensor
const inputs = data.map(d => d.rooms)
const labels = data.map(d => d.price);
const inputTensor = tf.tensor2d(inputs, [inputs.length, 1]);
const labelTensor = tf.tensor2d(labels, [labels.length, 1]);
//Step 3. Normalize the data to the range 0 - 1 using min-max scaling
const inputMax = inputTensor.max();
const inputMin = inputTensor.min();
const labelMax = labelTensor.max();
const labelMin = labelTensor.min();
const normalizedInputs = inputTensor.sub(inputMin).div(inputMax.sub(inputMin));
const normalizedLabels = labelTensor.sub(labelMin).div(labelMax.sub(labelMin));
return {
inputs: normalizedInputs,
labels: normalizedLabels,
// Return the min/max bounds so we can use them later.
inputMax,
inputMin,
labelMax,
labelMin,
}
});
}

Let’s break down what’s going on here.

Shuffle the data

// Step 1. Shuffle the data
tf.util.shuffle(data);

During training the model, the dataset is divided into smaller sets, each set known as a batch. These batches are then fed to the model for training. Shuffling the data is important because the model should not get the same data over and over again. If the model gets the same data over and over again, the model will not be able to generalize the data and give outputs specified to the inputs it received during training. Shuffling will help by having a variety of data in each batch.

Convert to tensors

// Step 2. Convert data to Tensor
const inputs = data.map(d => d.rooms)
const labels = data.map(d => d.price);
const inputTensor = tf.tensor2d(inputs, [inputs.length, 1]);
const labelTensor = tf.tensor2d(labels, [labels.length, 1]);

Here we make two arrays, one for our input examples (the no. of rooms entries), and another for the true output values (which are known as labels in machine learning, in our case the price of each house). We then convert each array data to a 2d tensor.

Normalize the data

//Step 3. Normalize the data to the range 0 - 1 using min-max scaling
const inputMax = inputTensor.max();
const inputMin = inputTensor.min();
const labelMax = labelTensor.max();
const labelMin = labelTensor.min();
const normalizedInputs = inputTensor.sub(inputMin).div(inputMax.sub(inputMin));
const normalizedLabels = labelTensor.sub(labelMin).div(labelMax.sub(labelMin));

Next, we normalize the data. Here we normalize the data into the numerical range 0-1 using min-max scaling.Normalization is important because the internals of many machine learning models you will build with tensorflow.js are designed to work with numbers that are not too big. Common ranges to normalize data to include 0 to 1 or -1 to 1.

Return the data and the normalization bounds

return {
inputs: normalizedInputs,
labels: normalizedLabels,
// Return the min/max bounds so we can use them later.
inputMax,
inputMin,
labelMax,
labelMin,
}

We want to keep the values we used for normalization during training so that we can un-normalize the outputs to get them back into our original scale and to allow us to normalize future input data the same way.

Step 5: Train the model

With our model instance created and our data represented as tensors we have everything in place to start the training process.

Copy the following function into your script.js file.

async function trainModel(model, inputs, labels) {
// Prepare the model for training.
model.compile({
optimizer: tf.train.adam(),
loss: tf.losses.meanSquaredError,
metrics: ['mse'],
});

const batchSize = 28;
const epochs = 50;

return await model.fit(inputs, labels, {
batchSize,
epochs,
shuffle: true,
callbacks: tfvis.show.fitCallbacks(
{ name: 'Training Performance' },
['loss', 'mse'],
{ height: 200, callbacks: ['onEpochEnd'] }
)
});
}

Let’s break this down.

Prepare for training

// Prepare the model for training.
model.compile({
optimizer: tf.train.adam(),
loss: tf.losses.meanSquaredError,
metrics: ['mse'],
});

We have to ‘compile’ the model before we train it. To do so, we have to specify few very important things:

const batchSize = 28;
const epochs = 50;

Next, we pick a batchSize and a number of epochs:

Start the train loop

return model.fit(inputs, labels, {
batchSize,
epochs,
callbacks: tfvis.show.fitCallbacks(
{ name: 'Training Performance' },
['loss', 'mse'],
{
height: 200,
callbacks: ['onEpochEnd']
}
)
});

model.fit is the function we call to start the training loop. It is an asynchronous function so we return the promise it gives us so that the caller can determine when training is complete.

To monitor training progress we pass some callbacks to model.fit. We use [tfvis.show.fitCallbacks](https://github.com/tensorflow/tfjs-vis#showfitcallbackscontainer-surface--metrics-string-opts---key-string-iteration-number-log-logs--promise) to generate functions that plot charts for the ‘loss’ and ‘mse’ metric we specified earlier.

Put it all together

Now we have to call the functions we have defined from our run function.

Add the following code to the bottom of your run function.

// Convert the data to a form we can use for training.
const tensorData = convertToTensor(data);
const {inputs, labels} = tensorData;

// Train the model
await trainModel(model, inputs, labels);
console.log('Done Training');

When you refresh the page, after a few seconds you should see the following graphs updating.

These are created by the callbacks we created earlier. They display the loss (on the most recent batch) and mse (on the whole dataset) at the end of each epoch.

When training a model we want to see the loss go down. In this case, because our metric is a measure of error, we want to see it go down as well.

Step 6: Make Predictions

Now that our model is trained, we want to make some predictions. Let’s evaluate the model by seeing what it predicts for a uniform range of numbers of low to a high number of rooms.

Add the following function to your script.js file

function testModel(model, inputData, normalizationData) {
const {inputMax, inputMin, labelMin, labelMax} = normalizationData;

// Generate predictions for a uniform range of numbers between 0 and 1;
// We un-normalize the data by doing the inverse of the min-max scaling
// that we did earlier.
const [xs, preds] = tf.tidy(() => {

const xs = tf.linspace(0, 1, 100);      
const preds = model.predict(xs.reshape([100, 1]));      

const unNormXs = xs
  .mul(inputMax.sub(inputMin))
  .add(inputMin);

const unNormPreds = preds
  .mul(labelMax.sub(labelMin))
  .add(labelMin);

// Un-normalize the data
return [unNormXs.dataSync(), unNormPreds.dataSync()];

});

const predictedPoints = Array.from(xs).map((val, i) => {
return {x: val, y: preds[i]}
});

const originalPoints = inputData.map(d => ({
x: d.rooms, y: d.price,
}));

tfvis.render.scatterplot(
{name: 'Model Predictions vs Original Data'},
{values: [originalPoints, predictedPoints], series: ['original', 'predicted']},
{
xLabel: 'No. of rooms',
yLabel: 'Price',
height: 300
}
);
}

A few things to notice in the function above.

const xs = tf.linspace(0, 1, 100);
const preds = model.predict(xs.reshape([100, 1]));

We generate 100 new ‘examples’ to feed to the model. Model.predict is how we feed those examples into the model. Note that they need to have a similar shape ([num_examples, num_features_per_example]) as when we did training.

// Un-normalize the data
const unNormXs = xs
.mul(inputMax.sub(inputMin))
.add(inputMin);

const unNormPreds = preds
.mul(labelMax.sub(labelMin))
.add(labelMin);

To get the data back to our original range (rather than 0–1) we use the values we calculated while normalizing, but just invert the operations.

return [unNormXs.dataSync(), unNormPreds.dataSync()];

[.dataSync()](https://js.tensorflow.org/api/latest/#tf.Tensor.dataSync) is a method we can use to get a typedarray of the values stored in a tensor. This allows us to process those values in regular JavaScript. This is a synchronous version of the [.data()](https://js.tensorflow.org/api/latest/#tf.Tensor.data) method which is generally preferred.

Finally, we use tfjs-vis to plot the original data and the predictions from the model.

Add the following code to your run function.

testModel(model, data, tensorData);

Refresh the page and you should see something like the following once the model finishes training.

Congratulations! You have just trained a simple machine learning model using Tensorflow.js! Here is the GitHub repository for reference.

Conclusion

I started doing this because the concept of Machine Learning intrigued me very much and wanted to see if there was any way this could be done in front end development. I was very happy to learn that the Tensorflow.js library could help me achieve my objective. This is just the beginning of Machine Learning in front end development. There is a lot more which can be done and has been already done by the Tensorflow.js team. Thanks for reading!

Originally published by Priyesh Patel at https://blog.bitsrc.io

=============================

Thanks for reading :heart: If you liked this post, share it with all of your programming buddies! Follow me on Facebook | Twitter


Learn More

☞ Machine Learning A-Z™: Hands-On Python & R In Data Science

☞ Python for Data Science and Machine Learning Bootcamp

☞ Data Science, Deep Learning, & Machine Learning with Python

☞ Deep Learning A-Z™: Hands-On Artificial Neural Networks

☞ Artificial Intelligence A-Z™: Learn How To Build An AI

Machine Learning Full Course - Learn Machine Learning

Machine Learning Full Course - Learn Machine Learning

This complete Machine Learning full course video covers all the topics that you need to know to become a master in the field of Machine Learning.

Machine Learning Full Course | Learn Machine Learning | Machine Learning Tutorial

It covers all the basics of Machine Learning (01:46), the different types of Machine Learning (18:32), and the various applications of Machine Learning used in different industries (04:54:48).This video will help you learn different Machine Learning algorithms in Python. Linear Regression, Logistic Regression (23:38), K Means Clustering (01:26:20), Decision Tree (02:15:15), and Support Vector Machines (03:48:31) are some of the important algorithms you will understand with a hands-on demo. Finally, you will see the essential skills required to become a Machine Learning Engineer (04:59:46) and come across a few important Machine Learning interview questions (05:09:03). Now, let's get started with Machine Learning.

Below topics are explained in this Machine Learning course for beginners:

  1. Basics of Machine Learning - 01:46

  2. Why Machine Learning - 09:18

  3. What is Machine Learning - 13:25

  4. Types of Machine Learning - 18:32

  5. Supervised Learning - 18:44

  6. Reinforcement Learning - 21:06

  7. Supervised VS Unsupervised - 22:26

  8. Linear Regression - 23:38

  9. Introduction to Machine Learning - 25:08

  10. Application of Linear Regression - 26:40

  11. Understanding Linear Regression - 27:19

  12. Regression Equation - 28:00

  13. Multiple Linear Regression - 35:57

  14. Logistic Regression - 55:45

  15. What is Logistic Regression - 56:04

  16. What is Linear Regression - 59:35

  17. Comparing Linear & Logistic Regression - 01:05:28

  18. What is K-Means Clustering - 01:26:20

  19. How does K-Means Clustering work - 01:38:00

  20. What is Decision Tree - 02:15:15

  21. How does Decision Tree work - 02:25:15 

  22. Random Forest Tutorial - 02:39:56

  23. Why Random Forest - 02:41:52

  24. What is Random Forest - 02:43:21

  25. How does Decision Tree work- 02:52:02

  26. K-Nearest Neighbors Algorithm Tutorial - 03:22:02

  27. Why KNN - 03:24:11

  28. What is KNN - 03:24:24

  29. How do we choose 'K' - 03:25:38

  30. When do we use KNN - 03:27:37

  31. Applications of Support Vector Machine - 03:48:31

  32. Why Support Vector Machine - 03:48:55

  33. What Support Vector Machine - 03:50:34

  34. Advantages of Support Vector Machine - 03:54:54

  35. What is Naive Bayes - 04:13:06

  36. Where is Naive Bayes used - 04:17:45

  37. Top 10 Application of Machine Learning - 04:54:48

  38. How to become a Machine Learning Engineer - 04:59:46

  39. Machine Learning Interview Questions - 05:09:03

Learn TensorFlow.js - Deep Learning and Neural Networks with JavaScript

Learn TensorFlow.js - Deep Learning and Neural Networks with JavaScript

This full course introduces the concept of client-side artificial neural networks. We will learn how to deploy and run models along with full deep learning applications in the browser! To implement this cool capability, we’ll be using TensorFlow.js (TFJS), TensorFlow’s JavaScript library.

By the end of this video tutorial, you will have built and deployed a web application that runs a neural network in the browser to classify images! To get there, we'll learn about client-server deep learning architectures, converting Keras models to TFJS models, serving models with Node.js, tensor operations, and more!

⭐️Course Sections⭐️

⌨️ 0:00 - Intro to deep learning with client-side neural networks

⌨️ 6:06 - Convert Keras model to Layers API format

⌨️ 11:16 - Serve deep learning models with Node.js and Express

⌨️ 19:22 - Building UI for neural network web app

⌨️ 27:08 - Loading model into a neural network web app

⌨️ 36:55 - Explore tensor operations with VGG16 preprocessing

⌨️ 45:16 - Examining tensors with the debugger

⌨️ 1:00:37 - Broadcasting with tensors

⌨️ 1:11:30 - Running MobileNet in the browser

Machine Learning in JavaScript with TensorFlow.js

Machine Learning in JavaScript with TensorFlow.js

Machine Learning in JavaScript with TensorFlow.js

Google releases and showcases TensorFlow.js at the TensorFlow Dev Summit 2018. Machine learning in the browser has never been better. Here is the presentation:

ML in JS with TensorFlow.js Video Transcript

All right, hi everyone thanks for coming today. My name is Daniel, my name is Nikhil. We're from the Google brain team and today we're delighted to talk about JavaScript. So, Python has been one of the mainstream languages for scientific computing and it's been like that for a while and there's a lot of tools and libraries around Python but it's that where it ends. We're here today to take the job to convince you that JavaScript and the browser have a lot to offer and the TensorFlow playground is a great example of that. Curious, how many people have seen TensorFlow playground before?

Oh wow, quite a few. I'm very glad. So, for those of you that haven't seen it you can check it out after our talk at https://playground.tensorflow.org - it is an in-browser visualization of a small
neural network and it shows in real time all the internals of the network as it's training and this was a lot of fun to make and had a huge educational success.

We've been getting emails from high schools and universities that have been using this to teach students about machine learning. After we launched playground we were wondering why was it so successful and we think one big reason was that it was in the browser and the browser is this unique platform where with the things you build you can share with anyone with just a link and those people that open your app, don't have to install any drivers or any software it just works.

Another thing is it's the browser is highly interactive and so the user is going to be engaged with whatever you're building. Another big thing is that browsers, we didn't take advantage of this in the playground, but browsers have access to sensors like the microphone and the camera and the accelerometer and all of these sensors are behind standardized APIs that work on all
browsers, and the last thing, the most important thing, is the data that comes from these sensors doesn't ever have to leave the client, you don't have to upload anything to the server, which preserves privacy.

Now the playground that we built is powered by a small neural network 300 lines of vanilla JavaScript that we wrote as a one-off library. It doesn't scale, is just simple for loops and it wasn't engineered to be reusable. But it was clear to us that if we were to open the door to for people to merge machine learning and the browser we had to build a library and we did it.

We released deeplearn.js, a JavaScript library that is GPU accelerated, it does that via WebGL, which is a standard in the browser that allows you to render 3D graphics. We utilize it to do linear algebra for us and deeplearn.js allows you to both run inference in the browser and training entirely in the browser. When we released it we had an incredible momentum the community took
the plunges and took existing models in Python and ported it to the browser and build interactive fun things with it.

So one example is the style transfer. Another person ported the character RNN and then built a novel interface that allows you to explore all the different possible endings of a sentence, all generated by the model in real time.

Another example is a font generative model. There was a post about this that the person that built it allowed users to explore the hidden dimensions the interesting dimensions in the embedding space and you can see how they relate to boldness and slanted nosov the font and there was even educational examples like teachable machine that build this fun little game that taught people how computer vision models work so people could interact directly with the webcam now all the examples I showed you point to the incredible momentum we have with deep lunges and building on that - we're very excited today to announce that deep learn Jas is joining the tensorflow family and with that we are releasing a new ecosystem of libraries and tools for machine learning with JavaScript called tensorflow Jas now before we get into the details I want to go over three main use cases of how you can use tensorflow Jas today with the tools and libraries that we're releasing so one use case is you can write models directly in the browser and this has huge educational implications think of the playground that I just showed a second use case is a major use case is you can take a pre-existing model pre-trained model in python use a script and you can import it into the browser to do inference and a related use case is the same model that you take to do inference you can retrain it potentially with private data that comes from those sensors of the browser in the browser itself now to give you more of a schematic view we have the browser that utilizes WebGL to do fast linear algebra on top of it tensorflow jeaious has two sets of api's the apps API which used to be deep learn Jas and we worked hard to align the API with tensorflow Python it is powered by an automatic differentiation library that is built analogous to eager mode and on top of that we have a high level API layers API that allow you to use best practices and high level building blocks to write models what I'm also very excited today to announce is that we're releasing tools that can take a existing Kerris model or tensorflow safe model and port it automatically free for execution in the browser now to show you an example of our API we're going to go over a small program that tries to learn the coefficients of a quadratic function so the coefficients we're trying to learn our a B and C from data so we have our import TF from tensorflow J's for those of you that don't know this is a standard yes six important JavaScript very common we have our three tensors a B and C we mark them as variables which means that they are mutable and the optimizer can change them we have our f of X function that does the polynomial computation you can see here familiar API like TF ad + TF square like tensorflow in addition to that API we also have a chaining API which allows you to call these math operations on tensors itself and this leads to better readable code that is closer to how we write math chaining is very popular in in JavaScript world so that's the feed-forward part of the model now for the training part we need a loss function so here is a loss function there is just a mean squared error from between the prediction and the label we have our optimizer and SJ edge the optimizer and we train the model we call optimizer that minimize for some number of epochs and here I want to emphasize for those of you that have used Thea figure before or the talk before us Alexes talk the API in tens of ojs is aligned with the eager API in Python alright so clearly that's not how most people write machine learning because because those a low-level linear algebra ops can be quite verbose and for that we have our layers API and to show you an example of that we're gonna we're going to build a recurrent neural network that learns to sum two numbers but the complicated part is that those numbers like the number 90 + 10 are being fed character by character and then the neural network has to maintain an internal state with an Alice TM cell that that state then gets passed into a decoder and then the decoder has to output 100 character by characters so it's a sequence to sequence model now this may sound a little complicated but with the layers API is not that much lines of code we have our import TF from tensorflow J's the four we have our sequential model which just means it's a stack of layers for those of you that are familiar with TF layers in Python or Kerris this API looks very familiar we have the first two layers of the encoder the last three layers are the decoder and that's our model we then compile it with a loss and optimizer and a metric we want a monitor like accuracy and we call model dot fit with our data now what I want to point out here is the await keyword so model dot fit is an asynchronous call which means because in practice that can take
about 30 or 40 seconds in a browser and in those 30 or 40 seconds you don't want the main UI thread of the browser to be locked and this is why you get a callback with a history object after that's done and in between the GPU is going to do the work now the code I showed you is when you are trying to write models directly when you want to write models directly in the browser but as I said before a major use case where even with deep learning is with people importing models that were pre trained and they just want to do inference in the browser and before we jump into the details of that I want to show you a fun little game that our friends at Google brand studio built that takes advantage of an automatically pre trained model poured it into the browser and the game is called emoji scavenger hunt and the way it works I'm going to show you here a real demo with the phone it's in the browser let me see I see here so you can see I have a Chrome browser open up on a pixel phone you can see the URL on the top and the game uses the webcam and shows me an emoji and then I have some time some number of seconds to find the real version item of that emoji before the time runs out before we play the kill here is going to help me identify the objects that this game asks you ready I'm ready all right let's go alright alright watch watch have a watch didn't I spot a velvet come on yay we got that let's see what's our next item is choo-choo you gotta help me out here buddy oh yeah we got the shoe alright what's next right nice a banana banana yes use any whenever but this guy's got a banana oh come over here alright yeah high score here beer yeah it's 10:30 in the morning Daniel let's get back to the talk alright alright so I'm gonna jump into some of the technical details of how we actually built that game so what we did was we trained a model in tensorflow to be an object recognizer for the scavenger hunt game we chose about 400 different classes that would be reasonable for a game like this you know watches and bananas and and and beer so what we did was we use the tensorflow for poets code lab and in that code lab what you essentially do is you take a pre trained mobile net model and if you don't know what mobile net is it's a state-of-the-art computer vision model for edge devices so what we effectively did was we took that model and we retrained it for these classes now we have an object detector in the Python world how do we actually now get this into the browser well we provide a set of tools today that help you actually do that once it's in you skin the game and you you know make the computer talk and all that kind of fun stuff let's jump into how we actually convert that model so as Daniel mentioned earlier today we actually support two types of models so we support tensorflow saved models we have a convertor for that and we also have a convertor for Karis saved models so you define your model and you save it with a saved model this is a standard way to do that similarly for similarly this is code that you would do that for Karis is our important the next piece is that we actually convert it to the web today we're releasing a pip package it's tensorflow Jess you can install that there there's a script in there that lets you point to your tensorflow save model and point to an output directory and that output directory will be where those static build artifacts will go for the web caris is the same exact flow point to your hdf5 input and you have an output directory where those Bulldog effects will go now you statically host those on your website somewhere you know just simple static hosting and on the JavaScript side we provide an API that lets you load that model so this is what it looks like for tensor flow and the tensor flow state model you'll notice that it's a frozen model we don't right now support continuing training of this model well in the Karis case we actually let you continue training and we're working hard to keep these api's aligned in the future okay so under the cover what are we actually doing so we're doing some graph optimization which essentially means that we prune out nodes that you don't need to make the prediction you don't need that in the web we optimize waits for browser Auto caching so we pack in shard in chunks of four megabytes which helps your browser be quick the next time your page reloads today we support about 90 of the most commonly used tensorflow ups and we're working very hard to support more like control flops and we support 32 of the most commonly as Kara Slayers today and as I mentioned we let you continue training for Karis models and we let you do evaluation as well as make predictions from that mole okay so obviously there's a lot you can do with just porting your models to the web for inference but since the beginning of deep learned s we've made it a high priority to make sure that you can train directly in the browser you know this opens up the door for education and interactive tools like we saw with a playground as well as let you train with data that never leaves your client so this is huge for privacy so to show off what you can do with something like this we built another little game so the goal of the game is to play pac-man with your webcam now Daniel is gonna be my helper here he is much much better at this game that I am for some reason to say hi so there are three phases of the game phase one we're going to collect frames from the webcam and we're going to associate those with up down left and right these classes now Daniel is gonna do you know move his head up down left and right and he's just simply gonna play the game like that and you'll notice as he's collecting frames he's kind of moving around a little bit this kind of helps the model see different angles for that class and generalize a little bit better so after he's done collecting these frames we're gonna go and train our model so we're not actually training from scratch here when we hit that train button we're taking a pre train mobile net again porting that to the web and doing a retraining phase with that data that's local and we're using the layers API to do that in the browser here you want to press that train button alright our loss is going down it looks like we're learning something that's great so as soon as we press that play button what's gonna happen is we're gonna make predictions from the webcam those are gonna get plugged into those controls and it's going to control the Plaquemine pac-man game ready all right so you can see in the bottom right it's highlighting the class that it thinks it is and down if he moves his head around you'll see it you'll see a change like class and he's off. So so all of this code is online and you can go for kit we invite you to do so and obviously this is just a game but

You can imagine other types of applications of this. Like make a browser extension that lets you control the page for accessibility purposes so again all this code is online please go for kit and play and make something else with it okay Daniel I know this is fine I know .all right