Why AI Needs So Much Energy: Chips, Data Centers and Cooling Explained

Where the electricity behind AI actually goes: training versus everyday use, what the chips draw, why a data center spends energy on more than computing, how one company measured the energy of a single prompt, and the engineering levers that cut the footprint by large factors.

0:36 Two kinds of work: training and answering

AI uses energy in two phases: training a model once, and inference every time someone uses it; the compute used for the largest training runs grew exponentially.

Sources: OpenAI (original) · Google Cloud blog (original)

1:52 Where the watts go: the machines

Accelerator chips draw hundreds of watts each, but host processors, memory and machines kept idle for spikes also use energy.

Sources: Nvidia (original) · Google Cloud blog (original)

3:21 The building: cooling and PUE

Power usage effectiveness compares a data center's total energy with what reaches the computers; cooling is one of the biggest overheads, and machine learning cut Google's cooling energy by up to forty percent.

Sources: Google DeepMind (original) · Google Data Centers (original) · Google Cloud blog (original)

4:41 The energy of one question

Google estimated the median Gemini Apps text prompt at about a quarter of a watt-hour when counting the whole system, more than double the chip-only figure.

Sources: Google Cloud blog (original)

5:59 The levers that cut it

Sparse models, efficient data centers and accelerators, carbon-aware location, and serving tricks like distillation and speculative decoding can cut energy by large factors.

Sources: arXiv (original) · Google Cloud blog (original)

Follow the show in your podcast app to get each new episode as soon as it is out.

Produced by Juan Pablo.