Every time I watch a large language model spin out a story, or an image generator conjure a painting in seconds, I can’t help but think of the invisible stage crew. We rave about the actors — the models, the algorithms — but the real drama is happening backstage, where chips, fabs, and electricity bills decide what’s possible.

If LLMs are the rockstars, then GPUs are the guitars. TPUs are the custom-built violins. And foundries in Taiwan, Korea, and Arizona are the factories where those instruments are carved, sanded, and polished to impossible precision.

Without the hardware, the magic doesn’t happen. And right now, hardware is not just a technical detail — it’s a geopolitical chess match, an economic bottleneck, and maybe the hidden key to who leads in AI.

The Surprising Origin Story of GPUs

It’s almost poetic: the chips that made generative AI possible weren’t built for AI at all. They were built for video games.

In the late 1990s, GPUs (graphics processing units) became the secret sauce of smoother rendering, faster explosions, more realistic shadows. But the same qualities that made them good for games — massively parallel operations, lots of small calculations at once — turned out to be perfect for neural networks.

In 2012, when Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton trained AlexNet, the deep learning model that crushed ImageNet, they used two NVIDIA GTX 580 GPUs — gaming cards (Krizhevsky et al., 2012). That moment is widely credited with kicking off the deep learning revolution.

Since then, GPUs have become the beating heart of AI.

Why GPUs Are Perfect for AI

Training a transformer model is basically endless matrix multiplication. Multiply vectors. Add vectors. Repeat billions of times. CPUs, the chips in your laptop, are generalists — good at juggling different tasks. GPUs are specialists: thousands of cores crunching the same operation in parallel.

Jensen Huang, NVIDIA’s co-founder, loves to say GPUs are “the new industrial tool of our time.” Hyperbolic, maybe — but also true. In 2023, NVIDIA briefly became a trillion-dollar company, riding the AI wave (Financial Times, 2023).

The Rise of TPUs and Custom Silicon

Of course, GPUs aren’t the only game in town. Google built TPUs (Tensor Processing Units), custom accelerators tuned specifically for deep learning (Jouppi et al., 2017). TPUs handle matrix multiplications with terrifying efficiency, making them ideal for Google’s internal workloads.

Other companies followed:

• Apple bakes neural engines into iPhones for local AI.
• Amazon has its Inferentia and Trainium chips.
• Tesla built its own Dojo supercomputer for self-driving.

The logic is simple: if AI is your business, why rent your future from NVIDIA?

Moore’s Law Is Wheezing, But Packaging Saves the Day

For decades, we counted on Moore’s Law: transistors doubling every two years. But at 3nm, 2nm, and soon 1.4nm, physics is fighting back. Leakage, heat, quantum effects — you can only shrink silicon so far.

The workaround is packaging. Instead of making one giant chip, companies now stitch together “chiplets” with advanced interconnects. AMD’s MI300 accelerator and NVIDIA’s Grace Hopper superchip are prime examples (AMD, 2023).

It’s less about making transistors smaller, and more about making lots of them work together.

The Energy Problem

AI eats electricity. Training GPT-3 reportedly consumed ~1.3 GWh of power — about what 120 U.S. homes use in a year (Patterson et al., 2021). Training GPT-4? Likely much more, though OpenAI hasn’t disclosed.

Datacenters are being built near rivers, hydro plants, even in colder climates to manage cooling. Some researchers warn that if scaling continues unchecked, energy costs could become the limiting factor. It’s not just about chips; it’s about watts.

The Geopolitics of Silicon

Here’s where it gets spicy. Almost all advanced chips — NVIDIA, AMD, Apple — are manufactured by TSMC in Taiwan. That island has become the single point of failure for global AI.

Chris Miller’s Chip War (2022) makes it stark: if TSMC sneezes, the world’s tech industry catches pneumonia. The U.S. knows it. China knows it. Which is why chip export bans, subsidies, and CHIPS Act funding dominate headlines. AI isn’t just a tech story. It’s a national security story.

In 2023, the U.S. restricted exports of NVIDIA’s most advanced AI chips to China (Reuters, 2023). China responded by doubling down on domestic fabs. The arms race is no longer theoretical.

Supercomputers as Cultural Icons

Frontier at Oak Ridge, Aurora at Argonne, Japan’s Fugaku, Europe’s Leonardo — these supercomputers aren’t just research tools. They’re national mascots. Each represents a government’s bet that AI power equals economic and cultural power.

NVIDIA’s DGX servers are becoming status symbols in Silicon Valley startups the way corner offices used to be. Owning a rack of H100s is the new badge of seriousness.

Cloud vs. On-Prem

Most people never see the hardware. They rent it from the cloud: AWS, Azure, Google Cloud. This democratizes access — but also centralizes power. If only a handful of companies own the compute, they control the pace of AI.

Some researchers are pushing back with open compute cooperatives — shared datacenters for universities and nonprofits. It’s a reminder that hardware access isn’t just a technical issue; it’s an equity issue.

The Human Side of Chips

It’s easy to get lost in teraflops. But I keep thinking about the factory workers in Taiwan polishing wafers, or the engineers in Oregon debugging a photolithography machine the size of a bus. Each transistor, each GPU rack, is the result of a global web of labor, politics, and risk.

The AI revolution is being powered by invisible human hands. Maybe remembering that helps ground the conversation: chips aren’t magic, they’re made.

Voices in the Debate

• Optimists like Jensen Huang argue we’re only at the dawn of accelerated computing, and that AI workloads will justify ever-bigger chips.
• Skeptics warn that energy costs, supply chain fragility, and physics limits will cap growth (Neil Thompson, MIT, 2022).
• Pragmatists see hybrid futures: some workloads done in the cloud, some on edge devices with smaller accelerators.

The disagreements echo the ones we saw with models: scale forever vs. rethink the architecture.

Closing Thought

Every demo of an AI model is really a demo of a chip. GPUs and TPUs are the unsung protagonists of this story. They decide how big we can train, how fast we can iterate, and who gets to play in the first place.

The fight for AI is, in many ways, the fight for hardware. And hardware is messy, political, resource-intensive, human.

I think about it this way: if LLMs are the brains of AI, then GPUs are the heartbeat. And the heartbeat, right now, is racing.