Working through attention in deep learning — I’ve started to go through classic papers in machine learning, inventions that shifted the state of the art or created an entirely new application. These are my notes on the Transformer introduced in Attention is All You Need. …

An AI alternative history with natural language processing — My top priority is to reduce carbon emissions. The more we can reduce our dependence on foreign oil, the more energy we can produce. This is why I want the U.S. to keep growing as fast as possible, to do whatever it takes to bring about the transformation of our…

In my last article on RNNs, we looked at how the purpose of RNNs, how an RNN cell is built, and how to implement RNNs in PyTorch for sequence prediction. …

Neural networks with memory — Introduction A typical feed-forward neural network maps inputs to outputs with no consideration of previous computations or where the current input fits in relation to others. The network applies the same function to each input regardless of sequence. This is fine for many applications, but often the context of an input…