https://gejun.name/tags/decoder/ Recent content in Decoder on Gejun's Blog https://gejun.name/ https://gejun.name/ Hugo -- 0.165.0 en-us Wed, 15 Mar 2023 00:00:00 +0000 https://gejun.name/natural-language-processing/understanding-attention-nlp/ Wed, 15 Mar 2023 00:00:00 +0000 https://gejun.name/natural-language-processing/understanding-attention-nlp/ <div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"> <iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube-nocookie.com/embed/kCc8FmEb1nY?autoplay=0&amp;controls=1&amp;end=0&amp;loop=0&amp;mute=0&amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"></iframe> </div> <p>In <a href="https://gejun.name/natural-language-processing/building-makemore-mlp/">Part 2</a>, we built a simple MLP that generates names one character at a time, trained on 32k popular names. In this lecture, <a href="https://karpathy.ai">Andrej Karpathy</a> walks through the transformer architecture one piece at a time. We will start by refactoring that model, then add each transformer component in turn and watch the loss drop as we go.</p>