It Runs on the Computer in My Basement

And That’s Just Crazy

Exactly 21 years ago, as a final project for a Mathematical Methods for Computation course, I figured out how to trick the newly-released ATI 9700 graphics card (which I had just bought) to simulate cloth… doing the math for a few million springs per second on hardware designed to do no such thing.

This was one of my favorite open-ended class projects at Stanford – figuring out how to implement a variety of different ‘solvers’ (Runge-Kutta, Verlet, Adams-Bashforth) within the tight limits of the newly-released 2nd-generation DirectX shader model. Full paper here:

I lamented many of the limitations of the GPU (64 instruction limit, etc.) in that paper. When I returned to Microsoft months later as PM for DirectX graphics, we’d work with the leading GPU hardware architects at NVidia, ATI (eventually AMD), Intel and others to eliminate those limitations.

Yesterday, I ran a few 80-billion-parameter LLMs in real-time my desktop GPU (RTX 4090). It gave me the same feeling. On this chip on my desk, I’m running a hyper-intelligent ChatBot that passes the Turing Test with flying colors. I can talk to it. It writes code for me. It understands the images I show it.

Simulating a few million springs per second in 2003 gave me gorgeous, lifelike cloth with features like collision and friction with arbitrary, moving objects that wouldn’t become available in commercial videogame cloth simulators for over a decade.

A running cloth simulator looks surprisingly similar to an Artificial Neural Network. More or less – you replace the springs with neural connections, and scale it from a few hundred thousand connections to a few billion. To make it work, I had to shuffle data to and from CPU and GPU memory (which, ironically, is the same thing I need to do to run an LLM on a consumer graphics card designed for videogames).

Moore’s Law is a powerful force.