Recurrent Neural Networks (RNN) are a class of Artificial Neural Networks that can process a sequence of inputs in deep learning and retain its state while processing the next sequence of inputs. Traditional neural networks will process an input and move onto the next one disregarding its sequence. Data such as time series have a sequential order that needs to be followed in order to understand. Traditional feed-forward networks cannot comprehend this as each input is assumed to be independent of each other whereas in a time series setting each input is dependent on the previous input.

Illustration 1: source

In Illustration 1 we see that the neural network (hidden state) _A _takes an _xt _and outputs a value _ht. _The loop shows how the information is being passed from one step to the next. The inputs are the individual letters of ‘JAZZ’ and each one is passed on to the network A in the same order it is written (i.e. sequentially).

Recurrent Neural Networks can be used for a number of ways such as:

• Detecting the next word/letter
• Forecasting financial asset prices in a temporal space
• Action modeling in sports (predict the next action in a sporting event like soccer, football, tennis etc)
• Music composition
• Image generation

RNN vs Autoregressive Models

An autoregressive model is when a value from data with a temporal dimension are regressed on previous values up to a certain point specified by the user. An RNN works the same way but the obvious difference in comparison is that the RNN looks at all the data (i.e. it does not require a specific time period to be specified by the user.)

Yt = β0 + β1yt-1 + Ɛt

The above AR model is an order 1 AR(1) model that takes the immediate preceding value to predict the next time period’s value (yt). As this is a linear model, it requires certain assumptions of linear regression to hold–especially due to the linearity assumption between the dependent and independent variables. In this case, **_Yt _**and **_yt-1 _**must have a linear relationship. In addition there are other checks such as autocorrelation that have to be checked to determine the adequate order to forecast Yt.

An RNN will not require linearity or model order checking. It can automatically check the whole dataset to try and predict the next sequence. As demonstrated in the image below, a neural network consists of 3 hidden layers with equal weights, biases and activation functions and made to predict the output.

Source

These hidden layers can then be merged to create a single recurrent hidden layer. A recurrent neuron now stores all the previous step input and merges that information with the current step input.

Source

Advantages of an RNN

• It can model non-linear temporal/sequential relationships.
• No need to specify lags to predict the next value in comparison to and autoregressive process.

Disadvantages of an RNN

• Vanishing Gradient Problem
• Not suited for predicting long horizons

#deep learning #neural networks #artificial inteligence #rnn #deep learning