Shooting Hoops with Machine Learning

Shooting Hoops with Machine Learning

<br>

In this article, we’ll dive into using Unity3D](https://unity3d.com/) "https://unity3d.com/)")** and TensorFlow](http://tensorflow.org) "http://tensorflow.org)") to teach an AI to perform a simple in-game task: shooting balls into a hoop. The complete source code is available on Github,](https://github.com/abehaskins/tf-jam), "https://github.com/abehaskins/tf-jam),") if you have any questions reach out to me on Twitter.****](https://twitter.com/abeisgreat).** "https://twitter.com/abeisgreat).****")

An Introduction to our Game

There is a game where players have one main goal: get a ball into a basket. This doesn’t sound that hard, but when your blood is pumping, your heart is racing, the crowd is cheering — well, it gets pretty tough to make that shot. Am I talking about the classic American game of Basketball? No, never heard of it. I’m talking about the classic Midway arcade game NBA Jam.

If you’ve ever played NBA Jam or any of the games it inspired (including the real life NBA league, which I think came after NBA Jam) then you know the mechanic to shoot a ball, from the player’s perspective, is fairly simple. You hold and release the shoot button with perfect timing. Have you ever wondered how this shot takes place from the game’s perspective though? How is the arc of the ball chosen? How hard is the ball thrown? How does the computer know the angle to shoot at?

If you were a smart, math-inclined person you may be able to figure out these answers with pen and paper, however the author of this blog post failed 8th grade algebra, so… those “smart person” answers are out of the question. I’ll need to go about this in a different way.

Instead of taking the simpler, faster, more efficient route of actually doing the math required to make a shot we’re going to see how deep the rabbit hole goes, learn some simple TensorFlow, and try to shoot some darn hoops.

Getting Started

We’ll need a handful of things to walk through this project.

If you’re not an expert in any of these technologies, that is totally okay! (I’m definitely not an expert in all this!) I’ll do my best to explain how these pieces all fit together. One disadvantage of using so many varied technologies is that I will not be able to explain everything in full detail, but I’ll try to link out to educational resources as much as possible!

Download the Project

I will not attempt to recreate this project step-by-step, so I suggest pulling down the source code on Github and following along as I explain what is happening.

Note: You will need to download an import the ML-Agents Unity asset package for Tensorflow to be usable in C#. If you get any errors regarding Tensorflow not being found in Unity, then make sure you’ve followed the Unity setup docs for TensorflowSharp.

What is our goal?

To keep things simple, our desired outcome for this project will be incredibly simple. We want to solve: if the shooter is X distance away from the hoop, shoot the ball with Y force. That’s it! We will not try to aim the ball or anything fancy. We are only trying to figure out how hard to throw the ball to make the shot.

If you’re interested in how to make more complex AIs in Unity, you should check out the much more complete ML-Agents project from Unity. The methods I’ll talk about here are designed to be simple, approachable, and not necessarily indicative of best practices (I’m learning, too!)

My limited knowledge of TensorFlow, machine learning, and math will not be subtle. So take it with a grain of salt and understand that this is all for fun.

The Basket and the Ball

We already discussed the gist of our goal: to shoot a basket. To shoot a ball into a basket, you need a basket and… well, a ball. This is where Unity comes in.

If you’re not familiar with Unity, just know it’s a game engine which lets you build 2D and 3D games for all platforms. It has built in physics, basic 3D modeling, and a great scripting runtime (Mono) which lets us write our game in C#.

I’m not an artist, but I dragged around some blocks and put together this scene.

That red block is obviously our player. The hoops have been set up with invisible triggers which allow us to detect when an object (the ball) passes through the hoop.

In the Unity editor you can see the invisible triggers outlined in green. You’ll notice there are two triggers. This is so we can ensure that we only count baskets where the ball falls from the top to the bottom.

If we take a look at the OnTriggerEnter method in /Assets/BallController.cs (the script that each instance of our basketball will have), you can see how these two triggers are used together.

tf-jam-snippet-1.cs

private void OnTriggerEnter(Collider other)
{
  if (other.name == "TriggerTop")
  {
     hasTriggeredTop = true;
  } else if (other.name == "TriggerBottom") {
     if (hasTriggeredTop  && !hasBeenScored)
     {
        GetComponent<Renderer>().material = MaterialBallScored;
        Debug.Log(String.Format("{0}, {1}, {2}", SuccessCount++, Distance, Force.y));
     }
     hasBeenScored = true;
  }
}

This function does a few things. First, it ensures that both the top and bottom triggers are hit, then it changes the ball’s material so we can visually see that the ball has made the shot, and finally it logs out the two key variables we care about distance and force.y.

Taking Shots

Open up /Assets/BallSpawnerController.cs. This is a script that lives on our shooter and does the job of spawning Basketballs and attempting to make shots. Check out this snippet near the end of the DoShoot() method.

tf-jam-snippet-2.cs

var ball = Instantiate(PrefabBall, transform.position, Quaternion.identity);
var bc = ball.GetComponent<BallController>();
bc.Force = new Vector3(
  dir.x * arch * closeness,
  force,
  dir.y * arch * closeness
);
bc.Distance = dist;

This code Instantiates a new instance of a ball, then sets the force with which we’ll shoot and distance from the goal (so we can log this out more easily later, as we showed in the last snippet).

If you still have /Assets/BallController.cs open, you can take a look at our Start() method. This code is invoked when we create a new basketball.

tf-jam-snippet-3.cs

void Start ()
{
  var scaledForce = Vector3.Scale(Scaler, Force);
  GetComponent<Rigidbody>().AddForce(scaledForce);
  StartCoroutine(DoDespawn(30));
}

In other words, we create a new ball, give it some force, then automatically destroy the ball after 30 seconds because we’re going to be dealing with a lot of balls and we want to make sure we keep things reasonable.

Let’s try running all this and see how our all-star shooter is doing. You can hit the ▶️ (Play) button in the Unity editor and we’ll see…

Our player, affectionately known as “Red”, is almost ready to go up against Steph Curry.

So why is Red so darn awful? The answer lies in one line in Assets/BallController.cs which says float force = 0.2f. This line makes the bold claim that every shot should be exactly the same. You’ll notice that Unity takes this “exactly the same” thing very literally. The same object, with the same forces, duplicated again and again will always bounce in the exact same way. Neat.

This, of course, isn’t what we want. We’ll never learn to shoot like Lebron if we never try anything new, so let’s spice it up.

Randomizing Shots, Gathering Data

We can introduce some random noise by simply changing the force to be something random.

tf-jam-snippet-4.cs

float force = Random.Range(0f, 1f);

This mixes up our shots so we can finally see what it looks like when a basket is successfully scored, even if it takes it a while to guess right.

Red is very dumb, he’ll make a shot occasionally, but it’s pure luck. That’s okay though. At this point, any shot made is a data point we can use. We’ll get to that in a moment.

In the meantime, we don’t want to be able to make shots from only one spot. We want Red to successfully shoot (when he’s lucky enough) from any distance. In Assets/BallSpawnController.cs, look for these lines and uncomment MoveToRandomDistance().

tf-jam-snippet-5.cs

yield return new WaitForSeconds(0.3f);
// MoveToRandomDistance();

If we run this, we’ll see Red enthusiastically jumping around the court after each shot.

This combination of random movement and random forces is creating one very wonderful thing: data. If you look at the console in Unity, you’ll see data getting logged out for each shot as the successful attempts trickle in.

Each successful shot logs out the # of successful shots so far, the distance from the hoop, and the force required to make the shot. This is pretty slow though, lets ramp this up. Go back to where we added the MoveToRandomDistance() call and change 0.3f (a delay of 300 milliseconds per shot) to 0.05f (a delay of 50 milliseconds).

tf-jam-snippet-6.cs

yield return new WaitForSeconds(0.05f);
MoveToRandomDistance();

Now hit play and watch our successful shots pour in.

Now that is a good training regime! We can see from the counter in the back that we’re successfully scoring about 6.4% of shots. Steph Curry, he ain’t. Speaking of training, are we actually learning anything from this? Where is TensorFlow? Why is this interesting? Well, that’s the next step. We’re now prepared to take this data out of Unity and build a model to predict the force required.

Predictions, Models, and Regression

Checking our data in Google Sheets

Before we dive into TensorFlow, I wanted to take a look at the data so I let Unity run until Red successfully completed about 50 shots. If you look in the root directory of the Unity project, you should see a new file successful_shots.csv. This is a raw dump, from Unity, of each successful shot we made! I have Unity export this so that I can easily analyze it in a spreadsheet.

The .csv file has only three rows index, distance and force. I imported this file in Google Sheets and created a Scatterplotwith a trendline which will allow us to get an idea of the distribution of our data.

Wow! Look at that. I mean, look at that. I mean, wow… Alright fine, I’ll admit, I wasn’t sure what this meant at first either. Let me break down what we’re seeing

This graph shows up a series of points which are positioned along the Y axis based on the force of the shot and X axis based on the distance the shot was made from. What we see is a very clear correlation between force required and distance the shot is taken from (with a few random exceptions that had crazy bounces).

Practically you can read this as “TensorFlow is going to be very good at this.”

Although this use case is simple, one of the great things about TensorFlow is that we could build a more complex model if we wanted to, using similar code. For example, in a full game, we could include features — like the positions of the other plays, and stats about how often they’ve blocked shots in the past — to determine if our player should take a shot, or pass.

Creating our model TensorFlow.js

Open the tsjs/index.js file in your favorite editor. This file is unrelated to Unity and is just a script to train our model based on the data in successful_shots.csv.

Here’s the entire method which trains and saves our model…

tf-jam-snippet-7.js

(async () => {
   /*
       Load our csv file and get it into a properly shaped array of pairs likes...
       [
           [distanceA, forceB],
           [distanceB, forceB],
           ...
       ]
    */
   var pairs = getPairsFromCSV();
   console.log(pairs);

   /*
       Train the model using the data.
    */
   var model = tf.sequential();
   model.add(tf.layers.dense({units: 1, inputShape: [1]}));
   model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});

   const xs = tf.tensor1d(pairs.map((p) => p[0] / 100));
   const ys = tf.tensor1d(pairs.map((p) => p[1]));

   console.log(`Training ${pairs.length}...`);
   await model.fit(xs, ys, {epochs: 100});

   await model.save("file://../Assets/shots_model");
})();

As you can see, there’s just not much to it. We load our data from the .csv file and create a series of X and Y points (sounds a lot like our Google Sheet above!). From there we ask the model to “fit” to this data. After that, we save our model for future consumption!

Sadly, TensorFlowSharp isn’t expecting a model in the format that Tensorflow.js can save to. So we need to do some magical translating to be able to pull our model into Unity. I’ve included a few utilities to help with this. The general process is that we’ll translate our model from TensorFlow.js Format to [Keras]([https://keras.io/)](https://keras.io/) "https://keras.io/)") Format where we can make a checkpoint which we merge with our [Protobuf Graph Definition]([https://www.tensorflow.org/extend/tool_developers/)](https://www.tensorflow.org/extend/tool_developers/) "https://www.tensorflow.org/extend/tool_developers/)") to get a [Frozen Graph Definition]([https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py)](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py) "https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py)") which we can pull into Unity.

Luckily, if you want to play along you can skip all that and just run tsjs/build.sh and if all goes well it’ll automatically go through all the steps and plop the frozen model in Unity.

Inside of Unity, we can look at GetForceFromTensorFlow() in Assets/BallSpawnController.cs to see what interacting with our model looks like.

tf-jam-snippet-8.cs

float GetForceFromTensorFlow(float distance)
{
  var runner = session.GetRunner ();

  runner.AddInput (
     graph["shots_input"][0],
     new float[1,1]{{distance}}
  );
  runner.Fetch (graph ["shots/BiasAdd"] [0]);
  float[,] recurrent_tensor = runner.Run () [0].GetValue () as float[,];
  var force = recurrent_tensor[0, 0] / 10;
  Debug.Log(String.Format("{0}, {1}", distance, force));
  return force;
}

When you make a graph definition, you’re defining a complex system which has multiple steps. In our case, we defined our model as a single dense layer (with an implicit input layer) which means our model takes a single input and gives us some output.

When you use model.predict in TensorFlow.js, it will automatically supply your input to the correct input graph node and provide you with the output from the correct node once the calculation is complete. However, TensorFlowSharp works differently and requires us to interact directly with the graph nodes via their names.

With that in mind, it’s a matter of getting our input data into the format which our graph expects and sending the output back to Red.

It’s game day!

Using the system above, I created a few variations on our model. Here is Red shooting using a model trained on only 500 successful shots.

We see about a 10x increase in baskets made! What happens if we train Red for a couple hours and collect 10k or 100k successful shots? Surely that will improve his game even further! Well, I’ll leave that up to you.

I highly recommend you check source code on Github and tweet at me if you can beat a 60% success rate (spoiler: beating 60% is 100% possible, go back and look at the first gif to see just how good you can train Red!)

In this article, we’ll dive into using Unity3D](https://unity3d.com/) "https://unity3d.com/)")** and TensorFlow](http://tensorflow.org) "http://tensorflow.org)") to teach an AI to perform a simple in-game task: shooting balls into a hoop. The complete source code is available on Github,](https://github.com/abehaskins/tf-jam), "https://github.com/abehaskins/tf-jam),") if you have any questions reach out to me on Twitter.****](https://twitter.com/abeisgreat).** "https://twitter.com/abeisgreat).****")

An Introduction to our Game

There is a game where players have one main goal: get a ball into a basket. This doesn’t sound that hard, but when your blood is pumping, your heart is racing, the crowd is cheering — well, it gets pretty tough to make that shot. Am I talking about the classic American game of Basketball? No, never heard of it. I’m talking about the classic Midway arcade game NBA Jam.

If you’ve ever played NBA Jam or any of the games it inspired (including the real life NBA league, which I think came after NBA Jam) then you know the mechanic to shoot a ball, from the player’s perspective, is fairly simple. You hold and release the shoot button with perfect timing. Have you ever wondered how this shot takes place from the game’s perspective though? How is the arc of the ball chosen? How hard is the ball thrown? How does the computer know the angle to shoot at?

If you were a smart, math-inclined person you may be able to figure out these answers with pen and paper, however the author of this blog post failed 8th grade algebra, so… those “smart person” answers are out of the question. I’ll need to go about this in a different way.

Instead of taking the simpler, faster, more efficient route of actually doing the math required to make a shot we’re going to see how deep the rabbit hole goes, learn some simple TensorFlow, and try to shoot some darn hoops.

Getting Started

We’ll need a handful of things to walk through this project.

If you’re not an expert in any of these technologies, that is totally okay! (I’m definitely not an expert in all this!) I’ll do my best to explain how these pieces all fit together. One disadvantage of using so many varied technologies is that I will not be able to explain everything in full detail, but I’ll try to link out to educational resources as much as possible!

Download the Project

I will not attempt to recreate this project step-by-step, so I suggest pulling down the source code on Github and following along as I explain what is happening.

Note: You will need to download an import the ML-Agents Unity asset package for Tensorflow to be usable in C#. If you get any errors regarding Tensorflow not being found in Unity, then make sure you’ve followed the Unity setup docs for TensorflowSharp.

What is our goal?

To keep things simple, our desired outcome for this project will be incredibly simple. We want to solve: if the shooter is X distance away from the hoop, shoot the ball with Y force. That’s it! We will not try to aim the ball or anything fancy. We are only trying to figure out how hard to throw the ball to make the shot.

If you’re interested in how to make more complex AIs in Unity, you should check out the much more complete ML-Agents project from Unity. The methods I’ll talk about here are designed to be simple, approachable, and not necessarily indicative of best practices (I’m learning, too!)

My limited knowledge of TensorFlow, machine learning, and math will not be subtle. So take it with a grain of salt and understand that this is all for fun.

The Basket and the Ball

We already discussed the gist of our goal: to shoot a basket. To shoot a ball into a basket, you need a basket and… well, a ball. This is where Unity comes in.

If you’re not familiar with Unity, just know it’s a game engine which lets you build 2D and 3D games for all platforms. It has built in physics, basic 3D modeling, and a great scripting runtime (Mono) which lets us write our game in C#.

I’m not an artist, but I dragged around some blocks and put together this scene.

That red block is obviously our player. The hoops have been set up with invisible triggers which allow us to detect when an object (the ball) passes through the hoop.

In the Unity editor you can see the invisible triggers outlined in green. You’ll notice there are two triggers. This is so we can ensure that we only count baskets where the ball falls from the top to the bottom.

If we take a look at the OnTriggerEnter method in /Assets/BallController.cs (the script that each instance of our basketball will have), you can see how these two triggers are used together.

tf-jam-snippet-1.cs

private void OnTriggerEnter(Collider other)
{
  if (other.name == "TriggerTop")
  {
     hasTriggeredTop = true;
  } else if (other.name == "TriggerBottom") {
     if (hasTriggeredTop  && !hasBeenScored)
     {
        GetComponent<Renderer>().material = MaterialBallScored;
        Debug.Log(String.Format("{0}, {1}, {2}", SuccessCount++, Distance, Force.y));
     }
     hasBeenScored = true;
  }
}

This function does a few things. First, it ensures that both the top and bottom triggers are hit, then it changes the ball’s material so we can visually see that the ball has made the shot, and finally it logs out the two key variables we care about distance and force.y.

Taking Shots

Open up /Assets/BallSpawnerController.cs. This is a script that lives on our shooter and does the job of spawning Basketballs and attempting to make shots. Check out this snippet near the end of the DoShoot() method.

tf-jam-snippet-2.cs

var ball = Instantiate(PrefabBall, transform.position, Quaternion.identity);
var bc = ball.GetComponent<BallController>();
bc.Force = new Vector3(
  dir.x * arch * closeness,
  force,
  dir.y * arch * closeness
);
bc.Distance = dist;

This code Instantiates a new instance of a ball, then sets the force with which we’ll shoot and distance from the goal (so we can log this out more easily later, as we showed in the last snippet).

If you still have /Assets/BallController.cs open, you can take a look at our Start() method. This code is invoked when we create a new basketball.

tf-jam-snippet-3.cs

void Start ()
{
  var scaledForce = Vector3.Scale(Scaler, Force);
  GetComponent<Rigidbody>().AddForce(scaledForce);
  StartCoroutine(DoDespawn(30));
}

In other words, we create a new ball, give it some force, then automatically destroy the ball after 30 seconds because we’re going to be dealing with a lot of balls and we want to make sure we keep things reasonable.

Let’s try running all this and see how our all-star shooter is doing. You can hit the ▶️ (Play) button in the Unity editor and we’ll see…

Our player, affectionately known as “Red”, is almost ready to go up against Steph Curry.

So why is Red so darn awful? The answer lies in one line in Assets/BallController.cs which says float force = 0.2f. This line makes the bold claim that every shot should be exactly the same. You’ll notice that Unity takes this “exactly the same” thing very literally. The same object, with the same forces, duplicated again and again will always bounce in the exact same way. Neat.

This, of course, isn’t what we want. We’ll never learn to shoot like Lebron if we never try anything new, so let’s spice it up.

Randomizing Shots, Gathering Data

We can introduce some random noise by simply changing the force to be something random.

tf-jam-snippet-4.cs

float force = Random.Range(0f, 1f);

This mixes up our shots so we can finally see what it looks like when a basket is successfully scored, even if it takes it a while to guess right.

Red is very dumb, he’ll make a shot occasionally, but it’s pure luck. That’s okay though. At this point, any shot made is a data point we can use. We’ll get to that in a moment.

In the meantime, we don’t want to be able to make shots from only one spot. We want Red to successfully shoot (when he’s lucky enough) from any distance. In Assets/BallSpawnController.cs, look for these lines and uncomment MoveToRandomDistance().

tf-jam-snippet-5.cs

yield return new WaitForSeconds(0.3f);
// MoveToRandomDistance();

If we run this, we’ll see Red enthusiastically jumping around the court after each shot.

This combination of random movement and random forces is creating one very wonderful thing: data. If you look at the console in Unity, you’ll see data getting logged out for each shot as the successful attempts trickle in.

Each successful shot logs out the # of successful shots so far, the distance from the hoop, and the force required to make the shot. This is pretty slow though, lets ramp this up. Go back to where we added the MoveToRandomDistance() call and change 0.3f (a delay of 300 milliseconds per shot) to 0.05f (a delay of 50 milliseconds).

tf-jam-snippet-6.cs

yield return new WaitForSeconds(0.05f);
MoveToRandomDistance();

Now hit play and watch our successful shots pour in.

Now that is a good training regime! We can see from the counter in the back that we’re successfully scoring about 6.4% of shots. Steph Curry, he ain’t. Speaking of training, are we actually learning anything from this? Where is TensorFlow? Why is this interesting? Well, that’s the next step. We’re now prepared to take this data out of Unity and build a model to predict the force required.

Predictions, Models, and Regression

Checking our data in Google Sheets

Before we dive into TensorFlow, I wanted to take a look at the data so I let Unity run until Red successfully completed about 50 shots. If you look in the root directory of the Unity project, you should see a new file successful_shots.csv. This is a raw dump, from Unity, of each successful shot we made! I have Unity export this so that I can easily analyze it in a spreadsheet.

The .csv file has only three rows index, distance and force. I imported this file in Google Sheets and created a Scatterplotwith a trendline which will allow us to get an idea of the distribution of our data.

Wow! Look at that. I mean, look at that. I mean, wow… Alright fine, I’ll admit, I wasn’t sure what this meant at first either. Let me break down what we’re seeing

This graph shows up a series of points which are positioned along the Y axis based on the force of the shot and X axis based on the distance the shot was made from. What we see is a very clear correlation between force required and distance the shot is taken from (with a few random exceptions that had crazy bounces).

Practically you can read this as “TensorFlow is going to be very good at this.”

Although this use case is simple, one of the great things about TensorFlow is that we could build a more complex model if we wanted to, using similar code. For example, in a full game, we could include features — like the positions of the other plays, and stats about how often they’ve blocked shots in the past — to determine if our player should take a shot, or pass.

Creating our model TensorFlow.js

Open the tsjs/index.js file in your favorite editor. This file is unrelated to Unity and is just a script to train our model based on the data in successful_shots.csv.

Here’s the entire method which trains and saves our model…

tf-jam-snippet-7.js

(async () => {
   /*
       Load our csv file and get it into a properly shaped array of pairs likes...
       [
           [distanceA, forceB],
           [distanceB, forceB],
           ...
       ]
    */
   var pairs = getPairsFromCSV();
   console.log(pairs);

   /*
       Train the model using the data.
    */
   var model = tf.sequential();
   model.add(tf.layers.dense({units: 1, inputShape: [1]}));
   model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});

   const xs = tf.tensor1d(pairs.map((p) => p[0] / 100));
   const ys = tf.tensor1d(pairs.map((p) => p[1]));

   console.log(`Training ${pairs.length}...`);
   await model.fit(xs, ys, {epochs: 100});

   await model.save("file://../Assets/shots_model");
})();

As you can see, there’s just not much to it. We load our data from the .csv file and create a series of X and Y points (sounds a lot like our Google Sheet above!). From there we ask the model to “fit” to this data. After that, we save our model for future consumption!

Sadly, TensorFlowSharp isn’t expecting a model in the format that Tensorflow.js can save to. So we need to do some magical translating to be able to pull our model into Unity. I’ve included a few utilities to help with this. The general process is that we’ll translate our model from TensorFlow.js Format to [Keras]([https://keras.io/)](https://keras.io/) "https://keras.io/)") Format where we can make a checkpoint which we merge with our [Protobuf Graph Definition]([https://www.tensorflow.org/extend/tool_developers/)](https://www.tensorflow.org/extend/tool_developers/) "https://www.tensorflow.org/extend/tool_developers/)") to get a [Frozen Graph Definition]([https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py)](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py) "https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py)") which we can pull into Unity.

Luckily, if you want to play along you can skip all that and just run tsjs/build.sh and if all goes well it’ll automatically go through all the steps and plop the frozen model in Unity.

Inside of Unity, we can look at GetForceFromTensorFlow() in Assets/BallSpawnController.cs to see what interacting with our model looks like.

tf-jam-snippet-8.cs

float GetForceFromTensorFlow(float distance)
{
  var runner = session.GetRunner ();

  runner.AddInput (
     graph["shots_input"][0],
     new float[1,1]{{distance}}
  );
  runner.Fetch (graph ["shots/BiasAdd"] [0]);
  float[,] recurrent_tensor = runner.Run () [0].GetValue () as float[,];
  var force = recurrent_tensor[0, 0] / 10;
  Debug.Log(String.Format("{0}, {1}", distance, force));
  return force;
}

When you make a graph definition, you’re defining a complex system which has multiple steps. In our case, we defined our model as a single dense layer (with an implicit input layer) which means our model takes a single input and gives us some output.

When you use model.predict in TensorFlow.js, it will automatically supply your input to the correct input graph node and provide you with the output from the correct node once the calculation is complete. However, TensorFlowSharp works differently and requires us to interact directly with the graph nodes via their names.

With that in mind, it’s a matter of getting our input data into the format which our graph expects and sending the output back to Red.

It’s game day!

Using the system above, I created a few variations on our model. Here is Red shooting using a model trained on only 500 successful shots.

We see about a 10x increase in baskets made! What happens if we train Red for a couple hours and collect 10k or 100k successful shots? Surely that will improve his game even further! Well, I’ll leave that up to you.

I highly recommend you check source code on Github and tweet at me if you can beat a 60% success rate (spoiler: beating 60% is 100% possible, go back and look at the first gif to see just how good you can train Red!)

Learn TensorFlow.js - Deep Learning and Neural Networks with JavaScript

Learn TensorFlow.js - Deep Learning and Neural Networks with JavaScript

This full course introduces the concept of client-side artificial neural networks. We will learn how to deploy and run models along with full deep learning applications in the browser! To implement this cool capability, we’ll be using TensorFlow.js (TFJS), TensorFlow’s JavaScript library.

By the end of this video tutorial, you will have built and deployed a web application that runs a neural network in the browser to classify images! To get there, we'll learn about client-server deep learning architectures, converting Keras models to TFJS models, serving models with Node.js, tensor operations, and more!

⭐️Course Sections⭐️

⌨️ 0:00 - Intro to deep learning with client-side neural networks

⌨️ 6:06 - Convert Keras model to Layers API format

⌨️ 11:16 - Serve deep learning models with Node.js and Express

⌨️ 19:22 - Building UI for neural network web app

⌨️ 27:08 - Loading model into a neural network web app

⌨️ 36:55 - Explore tensor operations with VGG16 preprocessing

⌨️ 45:16 - Examining tensors with the debugger

⌨️ 1:00:37 - Broadcasting with tensors

⌨️ 1:11:30 - Running MobileNet in the browser

Machine Learning in JavaScript with TensorFlow.js

Machine Learning in JavaScript with TensorFlow.js

Machine Learning in JavaScript with TensorFlow.js

Google releases and showcases TensorFlow.js at the TensorFlow Dev Summit 2018. Machine learning in the browser has never been better. Here is the presentation:

ML in JS with TensorFlow.js Video Transcript

All right, hi everyone thanks for coming today. My name is Daniel, my name is Nikhil. We're from the Google brain team and today we're delighted to talk about JavaScript. So, Python has been one of the mainstream languages for scientific computing and it's been like that for a while and there's a lot of tools and libraries around Python but it's that where it ends. We're here today to take the job to convince you that JavaScript and the browser have a lot to offer and the TensorFlow playground is a great example of that. Curious, how many people have seen TensorFlow playground before?

Oh wow, quite a few. I'm very glad. So, for those of you that haven't seen it you can check it out after our talk at https://playground.tensorflow.org - it is an in-browser visualization of a small
neural network and it shows in real time all the internals of the network as it's training and this was a lot of fun to make and had a huge educational success.

We've been getting emails from high schools and universities that have been using this to teach students about machine learning. After we launched playground we were wondering why was it so successful and we think one big reason was that it was in the browser and the browser is this unique platform where with the things you build you can share with anyone with just a link and those people that open your app, don't have to install any drivers or any software it just works.

Another thing is it's the browser is highly interactive and so the user is going to be engaged with whatever you're building. Another big thing is that browsers, we didn't take advantage of this in the playground, but browsers have access to sensors like the microphone and the camera and the accelerometer and all of these sensors are behind standardized APIs that work on all
browsers, and the last thing, the most important thing, is the data that comes from these sensors doesn't ever have to leave the client, you don't have to upload anything to the server, which preserves privacy.

Now the playground that we built is powered by a small neural network 300 lines of vanilla JavaScript that we wrote as a one-off library. It doesn't scale, is just simple for loops and it wasn't engineered to be reusable. But it was clear to us that if we were to open the door to for people to merge machine learning and the browser we had to build a library and we did it.

We released deeplearn.js, a JavaScript library that is GPU accelerated, it does that via WebGL, which is a standard in the browser that allows you to render 3D graphics. We utilize it to do linear algebra for us and deeplearn.js allows you to both run inference in the browser and training entirely in the browser. When we released it we had an incredible momentum the community took
the plunges and took existing models in Python and ported it to the browser and build interactive fun things with it.

So one example is the style transfer. Another person ported the character RNN and then built a novel interface that allows you to explore all the different possible endings of a sentence, all generated by the model in real time.

Another example is a font generative model. There was a post about this that the person that built it allowed users to explore the hidden dimensions the interesting dimensions in the embedding space and you can see how they relate to boldness and slanted nosov the font and there was even educational examples like teachable machine that build this fun little game that taught people how computer vision models work so people could interact directly with the webcam now all the examples I showed you point to the incredible momentum we have with deep lunges and building on that - we're very excited today to announce that deep learn Jas is joining the tensorflow family and with that we are releasing a new ecosystem of libraries and tools for machine learning with JavaScript called tensorflow Jas now before we get into the details I want to go over three main use cases of how you can use tensorflow Jas today with the tools and libraries that we're releasing so one use case is you can write models directly in the browser and this has huge educational implications think of the playground that I just showed a second use case is a major use case is you can take a pre-existing model pre-trained model in python use a script and you can import it into the browser to do inference and a related use case is the same model that you take to do inference you can retrain it potentially with private data that comes from those sensors of the browser in the browser itself now to give you more of a schematic view we have the browser that utilizes WebGL to do fast linear algebra on top of it tensorflow jeaious has two sets of api's the apps API which used to be deep learn Jas and we worked hard to align the API with tensorflow Python it is powered by an automatic differentiation library that is built analogous to eager mode and on top of that we have a high level API layers API that allow you to use best practices and high level building blocks to write models what I'm also very excited today to announce is that we're releasing tools that can take a existing Kerris model or tensorflow safe model and port it automatically free for execution in the browser now to show you an example of our API we're going to go over a small program that tries to learn the coefficients of a quadratic function so the coefficients we're trying to learn our a B and C from data so we have our import TF from tensorflow J's for those of you that don't know this is a standard yes six important JavaScript very common we have our three tensors a B and C we mark them as variables which means that they are mutable and the optimizer can change them we have our f of X function that does the polynomial computation you can see here familiar API like TF ad + TF square like tensorflow in addition to that API we also have a chaining API which allows you to call these math operations on tensors itself and this leads to better readable code that is closer to how we write math chaining is very popular in in JavaScript world so that's the feed-forward part of the model now for the training part we need a loss function so here is a loss function there is just a mean squared error from between the prediction and the label we have our optimizer and SJ edge the optimizer and we train the model we call optimizer that minimize for some number of epochs and here I want to emphasize for those of you that have used Thea figure before or the talk before us Alexes talk the API in tens of ojs is aligned with the eager API in Python alright so clearly that's not how most people write machine learning because because those a low-level linear algebra ops can be quite verbose and for that we have our layers API and to show you an example of that we're gonna we're going to build a recurrent neural network that learns to sum two numbers but the complicated part is that those numbers like the number 90 + 10 are being fed character by character and then the neural network has to maintain an internal state with an Alice TM cell that that state then gets passed into a decoder and then the decoder has to output 100 character by characters so it's a sequence to sequence model now this may sound a little complicated but with the layers API is not that much lines of code we have our import TF from tensorflow J's the four we have our sequential model which just means it's a stack of layers for those of you that are familiar with TF layers in Python or Kerris this API looks very familiar we have the first two layers of the encoder the last three layers are the decoder and that's our model we then compile it with a loss and optimizer and a metric we want a monitor like accuracy and we call model dot fit with our data now what I want to point out here is the await keyword so model dot fit is an asynchronous call which means because in practice that can take
about 30 or 40 seconds in a browser and in those 30 or 40 seconds you don't want the main UI thread of the browser to be locked and this is why you get a callback with a history object after that's done and in between the GPU is going to do the work now the code I showed you is when you are trying to write models directly when you want to write models directly in the browser but as I said before a major use case where even with deep learning is with people importing models that were pre trained and they just want to do inference in the browser and before we jump into the details of that I want to show you a fun little game that our friends at Google brand studio built that takes advantage of an automatically pre trained model poured it into the browser and the game is called emoji scavenger hunt and the way it works I'm going to show you here a real demo with the phone it's in the browser let me see I see here so you can see I have a Chrome browser open up on a pixel phone you can see the URL on the top and the game uses the webcam and shows me an emoji and then I have some time some number of seconds to find the real version item of that emoji before the time runs out before we play the kill here is going to help me identify the objects that this game asks you ready I'm ready all right let's go alright alright watch watch have a watch didn't I spot a velvet come on yay we got that let's see what's our next item is choo-choo you gotta help me out here buddy oh yeah we got the shoe alright what's next right nice a banana banana yes use any whenever but this guy's got a banana oh come over here alright yeah high score here beer yeah it's 10:30 in the morning Daniel let's get back to the talk alright alright so I'm gonna jump into some of the technical details of how we actually built that game so what we did was we trained a model in tensorflow to be an object recognizer for the scavenger hunt game we chose about 400 different classes that would be reasonable for a game like this you know watches and bananas and and and beer so what we did was we use the tensorflow for poets code lab and in that code lab what you essentially do is you take a pre trained mobile net model and if you don't know what mobile net is it's a state-of-the-art computer vision model for edge devices so what we effectively did was we took that model and we retrained it for these classes now we have an object detector in the Python world how do we actually now get this into the browser well we provide a set of tools today that help you actually do that once it's in you skin the game and you you know make the computer talk and all that kind of fun stuff let's jump into how we actually convert that model so as Daniel mentioned earlier today we actually support two types of models so we support tensorflow saved models we have a convertor for that and we also have a convertor for Karis saved models so you define your model and you save it with a saved model this is a standard way to do that similarly for similarly this is code that you would do that for Karis is our important the next piece is that we actually convert it to the web today we're releasing a pip package it's tensorflow Jess you can install that there there's a script in there that lets you point to your tensorflow save model and point to an output directory and that output directory will be where those static build artifacts will go for the web caris is the same exact flow point to your hdf5 input and you have an output directory where those Bulldog effects will go now you statically host those on your website somewhere you know just simple static hosting and on the JavaScript side we provide an API that lets you load that model so this is what it looks like for tensor flow and the tensor flow state model you'll notice that it's a frozen model we don't right now support continuing training of this model well in the Karis case we actually let you continue training and we're working hard to keep these api's aligned in the future okay so under the cover what are we actually doing so we're doing some graph optimization which essentially means that we prune out nodes that you don't need to make the prediction you don't need that in the web we optimize waits for browser Auto caching so we pack in shard in chunks of four megabytes which helps your browser be quick the next time your page reloads today we support about 90 of the most commonly used tensorflow ups and we're working very hard to support more like control flops and we support 32 of the most commonly as Kara Slayers today and as I mentioned we let you continue training for Karis models and we let you do evaluation as well as make predictions from that mole okay so obviously there's a lot you can do with just porting your models to the web for inference but since the beginning of deep learned s we've made it a high priority to make sure that you can train directly in the browser you know this opens up the door for education and interactive tools like we saw with a playground as well as let you train with data that never leaves your client so this is huge for privacy so to show off what you can do with something like this we built another little game so the goal of the game is to play pac-man with your webcam now Daniel is gonna be my helper here he is much much better at this game that I am for some reason to say hi so there are three phases of the game phase one we're going to collect frames from the webcam and we're going to associate those with up down left and right these classes now Daniel is gonna do you know move his head up down left and right and he's just simply gonna play the game like that and you'll notice as he's collecting frames he's kind of moving around a little bit this kind of helps the model see different angles for that class and generalize a little bit better so after he's done collecting these frames we're gonna go and train our model so we're not actually training from scratch here when we hit that train button we're taking a pre train mobile net again porting that to the web and doing a retraining phase with that data that's local and we're using the layers API to do that in the browser here you want to press that train button alright our loss is going down it looks like we're learning something that's great so as soon as we press that play button what's gonna happen is we're gonna make predictions from the webcam those are gonna get plugged into those controls and it's going to control the Plaquemine pac-man game ready all right so you can see in the bottom right it's highlighting the class that it thinks it is and down if he moves his head around you'll see it you'll see a change like class and he's off. So so all of this code is online and you can go for kit we invite you to do so and obviously this is just a game but

You can imagine other types of applications of this. Like make a browser extension that lets you control the page for accessibility purposes so again all this code is online please go for kit and play and make something else with it okay Daniel I know this is fine I know .all right

Introducing TensorFlow.js: Machine Learning in Javascript

Introducing TensorFlow.js: Machine Learning in Javascript

Introducing TensorFlow.js: Machine Learning in Javascript: In this post, we’ll give you a quick overview of TensorFlow.js, and getting started resources you can use to try it out.

We’re excited to introduce TensorFlow.js, an open-source library you can use to define, train, and run machine learning models entirely in the browser, using Javascript and a high-level layers API. If you’re a Javascript developer who’s new to ML, TensorFlow.js is a great way to begin learning. Or, if you’re a ML developer who’s new to Javascript, read on to learn more about new opportunities for in-browser ML. In this post, we’ll give you a quick overview of TensorFlow.js, and getting started resources you can use to try it out.

In-Browser ML

Running machine learning programs entirely client-side in the browser unlocks new opportunities, like interactive ML! If you’re watching the livestream for the TensorFlow Developer Summit, during the TensorFlow.js talk you’ll find a demo where @dsmilkov and @nsthorat train a model to control a PAC-MAN game using computer vision and a webcam, entirely in the browser. You can try it out yourself, too, with the link below — and find the source in the examples folder.

If you’d like to try another game, give the Emoji Scavenger Hunt a whirl — this time, from a browser on your mobile phone.

ML running in the browser means that from a user’s perspective, there’s no need to install any libraries or drivers. Just open a webpage, and your program is ready to run. In addition, it’s ready to run with GPU acceleration. TensorFlow.js automatically supports WebGL, and will accelerate your code behind the scenes when a GPU is available. Users may also open your webpage from a mobile device, in which case your model can take advantage of sensor data, say from a gyroscope or accelerometer. Finally, all data stays on the client, making TensorFlow.js useful for low-latency inference, as well as for privacy preserving applications.

What can you do with TensorFlow.js?

If you’re developing with TensorFlow.js, here are three workflows you can consider.

  • You can import an existing, pre-trained model for inference. If you have an existing TensorFlow or Keras model you’ve previously trained offline, you can convert into TensorFlow.js format, and load it into the browser for inference.
  • You can re-train an imported model. As in the Pac-Man demo above, you can use transfer learning to augment an existing model trained offline using a small amount of data collected in the browser using a technique called Image Retraining. This is one way to train an accurate model quickly, using only a small amount of data.
  • Author models directly in browser. You can also use TensorFlow.js to define, train, and run models entirely in the browser using Javascript and a high-level layers API. If you’re familiar with Keras, the high-level layers API should feel familiar.
Let’s see some code

If you like, you can head directly to the samples or tutorials to get started. These show how-to export a model defined in Python for inference in the browser, as well as how to define and train models entirely in Javascript. As a quick preview, here’s a snippet of code that defines a neural network to classify flowers, much like on the getting started guide on TensorFlow.org. Here, we’ll define a model using a stack of layers.

import * as tf from ‘@tensorflow/tfjs’;
const model = tf.sequential();
model.add(tf.layers.dense({inputShape: [4], units: 100}));
model.add(tf.layers.dense({units: 4}));
model.compile({loss: ‘categoricalCrossentropy’, optimizer: ‘sgd’});

The layers API we’re using here supports all of the Keras layers found in the examples directory (including Dense, CNN, LSTM, and so on). We can then train our model using the same Keras-compatible API with a method call:

await model.fit(
  xData, yData, {
    batchSize: batchSize,
    epochs: epochs
});

The model is now ready to use to make predictions:

// Get measurements for a new flower to generate a prediction
// The first argument is the data, and the second is the shape.
const inputData = tf.tensor2d([[4.8, 3.0, 1.4, 0.1]], [1, 4]);

// Get the highest confidence prediction from our model
const result = model.predict(inputData);
const winner = irisClasses[result.argMax().dataSync()[0]];

// Display the winner
console.log(winner);

TensorFlow.js also includes a low-level API (previously deeplearn.js) and support for Eager execution. You can learn more about these by watching the talk at the TensorFlow Developer Summit.

An overview of TensorFlow.js APIs. TensorFlow.js is powered by WebGL and provides a high-level layers API for defining models, and a low-level API for linear algebra and automatic differentiation. TensorFlow.js supports importing TensorFlow SavedModels and Keras models.

How does TensorFlow.js relate to deeplearn.js?

Good question! TensorFlow.js, an ecosystem of JavaScript tools for machine learning, is the successor to deeplearn.js which is now called TensorFlow.js Core. TensorFlow.js also includes a Layers API, which is a higher level library for building machine learning models that uses Core, as well as tools for automatically porting TensorFlow SavedModels and Keras hdf5 models. For answers to more questions like this, check out the FAQ.

Where’s the best place to learn more?

To learn more about TensorFlow.js, visit the project homepage, check out the tutorials, and try the examples. You can also watch the talk from the 2018 TensorFlow Developer Summit, and follow TensorFlow on Twitter.

Thanks for reading, and we’re excited to see what you’ll create with TensorFlow.js!