Welcome Back to XcessAI

For most of the AI boom, one piece of hardware has dominated the conversation: the GPU.

Originally designed for graphics, GPUs turned out to be remarkably good at the parallel mathematics required to train neural networks. As generative AI exploded, demand followed.

The formula became simple: More GPUs. More compute. Bigger models. Better AI.

But as AI workloads become larger, more predictable and more expensive, a different question is emerging.

If we increasingly know exactly what AI needs its hardware to do, why keep using chips designed to do almost anything?

Google, Amazon, Microsoft and Meta have all developed their own AI accelerators. OpenAI has unveiled Jalapeño, a processor developed with Broadcom specifically around large-language-model inference. Meanwhile, researchers and semiconductor companies are exploring increasingly unconventional approaches, including computing inside memory and even using analog circuitry to perform AI calculations.

Something is changing. The first phase of the AI hardware revolution was about getting as much computing power as possible.

The next may be about getting exactly the right computing power. We may be approaching the end of the general-purpose era of AI compute.

Bottom line: As AI workloads become larger, more predictable and more expensive, hardware is increasingly being designed around specific AI tasks rather than asking general-purpose processors to do everything.

GPUs remain extraordinarily important because of their flexibility and mature software ecosystems. But inference is creating an enormous, repetitive workload where custom hardware can offer compelling economics.

The optimisation is also moving beyond individual chips toward the entire stack: models, memory, networking, software, cooling and power.

Further out, analog, in-memory and other unconventional architectures could change how AI computation itself is performed.

The next AI hardware race may therefore be less about more compute and more about better-matched compute.

To understand what is changing, we first need to understand why GPUs became so successful.

Their great advantage isn't that they were originally designed for AI. They weren't.

It is flexibility.

A modern GPU is an extraordinarily capable programmable machine. The same underlying hardware can train a language model, generate an image, perform scientific calculations or run inference across thousands of different neural networks.

That flexibility is incredibly valuable when a technology is evolving quickly.

If nobody knows exactly what tomorrow's AI architecture will look like, you want hardware capable of adapting to almost anything.

And for years, that was precisely where AI was.

The industry was experimenting rapidly. Model architectures changed. Training techniques evolved. Workloads expanded in unexpected directions.

In that environment, flexibility wasn't an inefficiency. It was an advantage.

Imagine buying a Swiss Army knife.

It contains scissors, screwdrivers, a bottle opener, a saw and several blades. That's incredibly useful when you don't know what problem you'll encounter.

But if your job is to cut the same piece of material ten million times every day, eventually you build a machine specifically for cutting it.

AI may be reaching that point.

Models are still changing rapidly, but many of the fundamental workloads involved in operating them are becoming better understood.

And critically, as we discussed many times, the economics of AI are increasingly being shaped by inference.

Training happens when a model is created. Inference happens every time somebody uses it.

Every chatbot question. Every coding suggestion. Every AI-generated image. Every agentic task. Every API request.

As AI spreads through the economy, inference becomes an enormous, continuous industrial workload. And enormous repetitive workloads invite optimisation.

This is where specialised chips enter the story.

ASIC stands for Application-Specific Integrated Circuit.

The name explains the concept.

Instead of designing a processor capable of doing many different things, you design it around a narrower set of tasks.

Google's TPUs are an established example of specialised AI acceleration. Other hyperscalers have followed with their own silicon strategies.

OpenAI's Jalapeño pushes the idea further. The chip was designed around modern LLM inference, optimising the architecture around the calculations, memory movement, networking and serving patterns these systems actually use.

That's important. Because once you understand the workload well enough, you can begin removing capabilities you don't need and optimising aggressively around those you do.

You sacrifice some flexibility. You gain efficiency.

This could become one of the defining economic battles in AI.

Training a frontier model may cost billions. But once millions, and potentially billions, of people use those models continuously, the cumulative cost of inference becomes equally important.

A small improvement in efficiency multiplied across billions of queries becomes enormous.

Lower power consumption. Lower latency. Higher throughput. Lower cost per token. Less cooling. Better utilisation of expensive data centres.

At sufficient scale, even modest efficiency gains can become worth designing silicon around. And the more AI is used, the stronger that incentive becomes.

This creates an interesting paradox. The more successful AI becomes, the greater the incentive to stop treating AI computation as a generic workload.

And this is where the story becomes more interesting than chips.

Imagine you control the model. You know exactly how it operates. You control the software serving it. You know which calculations happen most frequently. You understand how memory is accessed. You know where latency appears. You operate the data centres.

Eventually, you ask an obvious question:

Why shouldn't the hardware itself be designed around all of that knowledge?

This creates what we might call full-stack AI optimisation.

Instead of optimising each layer independently, the entire system can increasingly be engineered around the workload. That changes the nature of competition.

The AI infrastructure race becomes less about owning the fastest individual component and more about designing the most efficient system.

The temping conclusion is that specialised chips replace GPUs. That's too simplistic.

GPUs retain an enormous advantage precisely because they are flexible.

AI remains a rapidly changing field. New architectures emerge. Models evolve. Training requirements change. Developers need mature software ecosystems.

General-purpose acceleration remains extremely valuable under those conditions.

The more interesting possibility is fragmentation.

Frontier training may continue to favour highly flexible accelerators. Large-scale inference may increasingly migrate toward custom silicon.

Phones and PCs use specialised NPUs. Cars and robots may develop different processors again.

Individual AI workloads begin finding the hardware architecture best suited to them.

Instead of one AI chip market, we get many. And that may ultimately be the more important change.

Specialisation can go much further. Almost every mainstream AI accelerator today remains fundamentally digital.

Information moves between memory and processors, calculations happen, and the results move again.

That movement consumes energy.

For modern AI systems, moving data can itself become a major bottleneck. Analog in-memory computing asks a radical question:

What if some calculations happened inside the memory itself?

Rather than repeatedly moving model weights to a processor, analog systems can encode information using physical properties of memory devices and perform certain mathematical operations directly there.

The potential advantage is significant: dramatically reducing the amount of data that needs to move around the system.

This could make certain AI workloads considerably more energy efficient.

But analog computing isn't simply a better ASIC. It is a different approach to computation. And significant challenges remain, including precision, noise, device variation and integration with conventional digital systems.

Analog chips therefore shouldn't be confused with specialised chips generally. They represent one possible destination at the more radical end of the specialisation trend.

Alongside them are other approaches, including in-memory computing, photonic computing and neuromorphic architectures.

We may eventually redesign more than the processor. We may redesign how computation itself happens.

This pattern appears repeatedly in technological history.

Early technologies optimise for capability. Mature technologies increasingly optimise for efficiency.

The first cars were about making automobiles work. Later generations optimised engines, aerodynamics, transmissions, manufacturing and fuel efficiency.

The first commercial aircraft proved that mass aviation was possible. Decades of engineering then optimised every kilogram, component and litre of fuel.

AI may be following the same trajectory. The first challenge was:

Can we build machines capable of this intelligence?

The next becomes:

How efficiently can we deliver that intelligence billions of times?

That is a fundamentally different engineering problem. And it suggests that AI may gradually be moving from an era dominated by brute-force scaling toward one increasingly defined by optimisation.

There is another implication. If specialised hardware dramatically reduces the cost of inference, we probably won't simply spend less money running today's AI.

We'll use more AI. Cheaper intelligence makes previously uneconomic applications viable.

Agents can run longer. Models can reason for more steps. AI can operate continuously in the background.

More sophisticated models can run locally on phones, computers, cars and industrial equipment. Entire workflows that currently require expensive cloud inference may become commonplace.

This is a familiar phenomenon in technology. Efficiency doesn't necessarily reduce consumption. Sometimes it expands the market.

If the cost of delivering intelligence falls dramatically, we may discover thousands of applications that don't make economic sense today.

In that sense, specialised chips don't merely change the semiconductor market. They could change where AI becomes economically possible.

There is an interesting parallel with what is happening elsewhere in AI.

Models themselves are becoming increasingly specialised. Some are optimised for reasoning. Others for coding. Others for images, video, robotics or edge devices.

The software layer is becoming more specialised too, with agents increasingly designed around particular workflows and industries.

Hardware may simply be following the same trajectory.

As we understand AI better, we can increasingly optimise each layer around what it actually needs to do.

AI started as a relatively general technological capability. Its industrialisation may produce an increasingly specialised ecosystem.

The first era of generative AI was defined by scale.

That era isn't ending. Frontier models will continue demanding extraordinary amounts of compute, and flexible accelerators will remain essential.

But another era is beginning alongside it. Specialisation.

Once we understand what AI systems actually spend their time doing, we can start designing infrastructure specifically around those tasks.

Custom ASICs are already part of that transition. Analog and in-memory computing show where it might eventually lead. And the destination may be an AI ecosystem where there is no single "AI chip" at all.

There may instead be different architectures for training, inference, reasoning, agents, edge devices, robotics and workloads we haven't invented yet.

The first AI race was about building enough compute.

The next may be about building the right compute.

Until next time,
Stay adaptive. Stay strategic.
And keep exploring the frontier of AI.

Fabio Lopes
XcessAI

💡Next week: I’m breaking down one of the most misunderstood AI shifts happening right now. Stay tuned. Subscribe above.

Read our previous episodes online!