Nat  Kutch

Nat Kutch


How to understand LSTM by playing football

In the series of articles, we have studied the limits of Feed Forward Neural Network (FFNN) when it comes to understanding contexts. In order to solve the problem, we introduced the concept of Recurrent Neural Networks (RNN) and we defined the different types there exist. However, our previous analysis did not adequately address key questions on RNN such as what does it make a RNN different? What kind of calculations take place inside a cell? How does the RNN learn?

The purpose of this article is to address all these questions and, in order to do so, it might be beneficial to approach them by starting with an historical perspective. One of the first attempts in creating models capable of understanding context was developed by Jeffrey L. Elman in the early 90’s. This first RNN, sometimes called Vanilla RNN, was one of the first cells in which two units of the same layer would share information, the ‘state’ would pass from one to the other.

As suggested by the title of this article, the aim is to understand LSTM by playing football not talking about history, where is the ball?

The match kicks off

Here it comes, the model below represents a simplification of the actions you must take as a football player. The input for the model is an image of what your eyes would see in the field. Then a RNN layer makes its proper calculation of what your next action should be, it can be ‘pass’, ‘wait’, ‘run’…

At time t you have the ball and you see one of your colleagues is not being defended, so the RNN decides to pass the ball. The next image is your colleague trying to control the ball, you don’t know if he is going to manage to do so, then you wait before going to attack or defend. At time t+2 he has the ball and you offer yourself if he wants to pass the ball back. He is a very good player, so he decides to run towards the football goal, then your action is to run too. At time t+4 he is shooting the ball, so your first thought is to be ready if there is any rebound. At time t+5 you see the ball entering the goal, so you are with your colleagues celebrating the goal and having and awesome moment.

Image for post

Once we felt the energizing moment of scoring and celebrating the goal, is time to see some math. We represent below one of the cells in the RNN layer, what are the calculations that occurs inside?

Image for post

At time t+1 the cell receives two inputs, the state from the previous cell S̅ₜ and the event x̄ₜ₊₁ , in our example would be the image. To calculate the next state of the cell S̅ₜ₊₁ we add the result from the dot product of the event and the state with its respective weight matrices and we apply an activation function φ. Don’t you know what an activation function is? The concept of activation functions will be explored in greater detail in the next section. The resulting vector enter the cell at time t+2 as one of its inputs. Finally, the prediction y̅ₜ₊₁ is obtained by the dot product of S̅ₜ₊₁ with its weight matrix Wᵧ .

So, are we saying that human memory can be replicated with just two calculations? Is it that simple for a Neural Network to understand context? The answer is no, the approach of Elman was a very nice work, but it has one obstacle called vanishing gradient problem and therefore we also know this cell as short memory. Computers have not become wiser than human yet.

1- Who is in my team? Or Vanishing Gradient Problem

In the example of the representation of the RNN for football decision we have just focused on one move. But what is the result? How much time is left? Is it a League or a Champions game? Is the player I am going to pass the ball to even in my team?

The Elman structure suffers from vanishing gradient problem: this means that the cell is not able to learn any temporal dependency from 8–10 time steps away. We can find a more explicit representation of the problem thanks to the previous image of the RNN model. In this case, we are going to change the RNN layer color (green bubble) by a representation of how much data it can remember from previous steps. At each time step new data is added to the RNN represented by different colors. For the first step, we just have the blue colour displayed whilst for the second step the colour orange is added. Then as we continue through time, the information received from previous time steps starts to disappear while the information given by more recent steps becomes more visible.Our model has changed to the following:

#artificial-intelligence #data-science #machine-learning #ai #deep-learning #deep learning

What is GEEK

Buddha Community

How to understand LSTM by playing football
Jones Brianna

Jones Brianna


Fantasy Football App Development I Fantasy Football Software Development

As far as we know football is certainly the most popular and highest earning sport in the world. Investing in Fantasy Football is a great business opportunity. Millions of young sports enthusiasts are now hooked on fantasy sports websites.

#fantasy football app development #fantasy football software development #fantasy football website development #fantasy football app development services #create fantasy football app #hire fantasy football app developers

Jones Brianna

Jones Brianna


Leading Football Betting App Development Company

Are you struggling to design a robust football betting app? Mobiweb Technologies guaranteed you to deliver a secure football betting app with some advanced features. Our mobile app includes a variety of inbuilt features to keep the application up to date. Team of betting app developers work together to design an interactive and user-friendly application that can grab the attention of the client. We have command on potential technology to create customized betting apps to satisfy the needs of clients.

#football betting software development #football betting software #football betting app development #football betting app developers #football betting software developers #football betting game development

College Football Betting Software Development | College Football Betting Solutions

Mobiweb Technologies offers ultra-modern and feature-rich College Football Betting Software that covers all the aspects and features of enhancing the betting experience of the clients. We have a team of qualified and experienced sports betting software developers from different walks of industry that design robust and user-centric betting software for newbie and hi-tech enterprises. We offer on-demand customization services and integrate our skills and expertise to design high-quality betting software.
We develop unbeatable software solutions to increase the user-experience and reduce the risk of fraud and insecurity.

#college football betting software #college football betting app development #college football betting app developers #college football betting software developers #college football betting website developers

Jones Brianna

Jones Brianna


Things to Consider While Hiring a Football Betting App Developer

Betting is always seen as an entertaining mode to earn extra money from a domain of your interest. Sports betting app development is emerging as a great business and is enticing many people and football is also getting popular day by day.

#football betting software development #football betting software #football betting app development #football betting app developers #football betting game development

Fantasy Football Software Development

Check out the below listed Top 10 Fantasy Football Software and App Development Companies 2021–22, that can furnish you with a fabulous fantasy football mobile application, as per your precise requirements. These companies not only offer expert consulting services but are also proficient in creating the best soccer applications.

#fantasy football app development #develop fantasy football app #fantasy football software development #fantasy football website development #create fantasy football app development