Writing

Modern deep learning is a weird place. We have these amazing tools that are constantly redefining what can be done with computers (go take a look at AlphaFold), but we basically have no idea how these things actually work. Clearly someone knows what they are doing, because surely we aren't driving all those cars on a hunch? (*cough*) But from the outside looking in, it's witchcraft.

After five years of actually working on this stuff, I think I've finally developed enough intuition to figure out how to actually do something interesting with this technology in a consistent way. And one of the first things I learned is that intuition is the cornerstone of modern deep learning practice.

I think of ML scientists as the modern equivalent of Roman bridge builders. Humanity wouldn't figure out how to formalize bridge building until Newton came along...but the Romans still made some damn good bridges. The individual engineers all had these mental models of how bridge building worked, and that allowed them to innovate and optimize. Even though every model was wrong, some of those models were useful. You even see the same kind of rise and fall of 'architectures'. For years everyone builds Clapper bridges, and then one day an engineer wakes up and goes 'Hey maybe arches would be better' and everyone starts doing that.

The Newton of deep learning will show up eventually, but it might take a few hundred years. In the meantime, we have to make due with intuition. Luckily, I think it's possible to build a pretty good mental model of deep ML models through practice. Yea, the first few times a model fails, it'll be a frustrating experience. But eventually you should be able to pattern match and generalize, doing a human form of gradient descent. In that sense, Deep learning is a bit like sculpture or photography -- after a while, you get an eye for it.

There are a lot of ML guides floating around out there, so let me start by saying what SimpleDL is not. It's not an implementation guide -- if I'm lucky I'll stay pretty far from actual code. It's not a dive into literature -- in a few places I might link to papers, and I might do a separate series on individual papers, but not here. It's not a highly rigorous mathematical approach to DL. It's not a history of ML. It's not a course on ML. It's definitely not a social commentary on ML.

This series is, first and foremost, about building an intuition for deep learning. The goal is for the reader to come away with an instinct for sniffing out what a model is doing, and why, and how.

This series is titled SimpleDL because I want to make things simple. But some familiarity with basic computing concepts will be helpful. If you've never touched a computer before, this series probably isn't for you. I'll be aiming for 'intermediate programmer, never worked in ML before' as my audience. That said, I think this will be most useful for folks who have maybe taken one or two classes on ML, and know some of the basic terms. If you know that a ML model has two modes, and that they are called training and inference, but you don't know much more beyond that, you're in the sweet spot.

Before diving in, I want to highlight a few resources that really helped me get off the ground. First, the Oxford ML Class by Nando de Freitas. If you're interested in doing an actual course on this stuff, I think it's hard to beat this class. Second, the deep learning Youtube Videos also by Nando de Freitas. Nando is an amazing lecturer, and I think he does an excellent job mixing theory and practice.

Also, thanks for coming by to read some of the things I write! I hope it's useful/interesting. I don't know how frequently I'll update here, but when I do I'll post on Twitter.