Neural Network

In Part 1, we built a single-layer neural network – no hidden layer at all – that generates words one character at a time. It worked well enough: the network reproduced exactly what the simple counting model had produced. But a bigram model is limited by construction, since it assumes each character depends only on the one immediately before it. If a character starts just one bigram, the model will always emit that same next character, no matter what came earlier or how likely the alternatives are. That missing context is what holds bigram models back. In this lecture, Andrej Karpathy shows how a deeper network fixes it. ...

This is a series of learning notes for the excellent online course Neural Networks: Zero to Hero created by Andrej Karpathy. Andrej’s official notebook for this lecture is on GitHub. In this lecture, Andrej shows two different approaches to generating characters. The first samples characters from a probability distribution built by counting; the second trains a neural network from scratch. The interesting part is where they end up, so let’s prepare the data first. ...