Introduced in the 1970s, Hopfield networks were popularised by John Hopfield in 1982. Hopfield networks, for the most part of machine learning history, have been sidelined due to their own shortcomings and introduction of superior architectures such as the Transformers (now used in BERT, etc.).
Co-creator of LSTMs, Sepp Hochreiter with a team of researchers, have revisited Hopfield networks and came up with surprising conclusions. In a paper titled, ‘Hopfield networks Is All You Need’, the authors introduce a couple of elements that make Hopfield networks interchangeable with the state-of-the-art transformer models.
What’s New About Hopfield Networks
The above figure depicts the relation between binary modern Hopfield networks — the new Hopfield network has continuous states, a new update rule, and the transformer.
The standard binary Hopfield network has an energy function that can be expressed as the sum of interaction functions F with F(x) = x^2. Modern Hopfield networks called “dense associative memory” (DAM) models use an energy function with interaction functions of form F(x) = x^n and, thereby, achieve a storage capacity proportional to d^(n−1).
The main contributions of the paper can be summarised as follows:
1| Introduction of a new energy function using the log-sum-exp function
#developers corner #hopfield network #lstm network #sepp hochreiter #neural networks
