One of the most significant, yet underestimated, changes in AI over the last year has been that, for the first time, software is dictating how hardware is built, meaning understanding how AI looks inside automatically gives you “predicting powers” to foresee the future of AI hardware.

This is no joke, considering AI hardware is by far the most dynamic and opportunity-rich market in AI these days because, well, it’s the only one where money is being made and thus where most investors flock to.

But if I put you on the spot today, would you be able to answer how AI workloads are structured, how hardware works, why software now calls the shots, and ultimately, what will hardware look like two years from now?

If the answer is no, you can't consider yourself someone who truly understands AI. Luckily for you, by the end of this piece, you will:

Let’s dive in.

We could go as detailed as we wanted to explain the intricacies of Artificial Intelligence, or AI, but we don’t need to get into the gradient descents, embeddings, and optimizers today.

Luckily, to make successful inferences about AI’s future, to understand what will happen next, and why Hyperscalers feel the urge to spend another $500 billion next year on AI, we just need to learn the first principles, by starting from the most basic question almost no one can answer correctly: How and what do AIs learn?

Most AI models today, at least the most popular ones leading the biggest corporations on the planet to invest a combined trillions of dollars, are artificial neural networks (ANNs), a particular type of AI software.

ANNs are compressors, algorithms that are capable of seeing a lot of data, learning from it, and making predictions.

These predictions are then compared with the correct answers, and we use this signal to improve the model over time by pulling them closer and closer to the ground truth.

The key is that, by the way they are structured, these models can capture the patterns that govern the behavior of data. They cut through the noise to answer: from the data I’m seeing, what moves the needle and what doesn’t?

Using a simple example, say we want to predict the price of a random house. A totally uncompressed process (or ‘unintelligent’) would be a database where we store housing data, storing every piece of information we can, along with the price, and then do some statistical analysis ourselves to identify which variables affect the most, maybe even by running a regression.

Databases store data that you can retrieve, but they are unopinionated about what data really has predictive power (i.e., out of all the datapoints, which ones are most indicative of the price of a home?)

Instead, a neural network doesn’t store every single possible detail, but only those that matter. A database may store thousands of data points, while the NN will store only those that actually explain its price (postal code, size, number of bathrooms, etc.).

We call this compression: the capability of eliminating the noise and just storing what’s essential. This pattern-identifying capability allows them to be much smaller than the dataset they learn.

For instance, when we look at Large Language Models (LLMs), they are sometimes three or even four orders of magnitude smaller than the datasets they can represent (i.e., 1,000 or even 10,000 times smaller).

But how? Again, capturing patterns. In LLMs, which are mostly trained on text, they eventually pick up things like grammar and how words correctly follow one another according to language rules.

And from the moment they understand grammar, the infinite possible combinations of words in English get narrowed down to only those that are correct. But it’s not just grammar; they can pick various patterns that refer to knowledge too (capital city → country), maths (how to add up numbers without memorizing), etc.

However, there’s a problem: models, even modern generative ones, are mostly trained by imitation. That is, they learn data by seeing it repeatedly and copying it.

And although this has started not to be the case over the last year, as AI Labs are experimenting with trial-and-error learning (i.e., instead of showing them the solution to copy, they force models to find it themselves, a process known as Reinforcement Learning, or RL), it’s safe to say models are still heavily dependent on this imitation phase.

But what is the issue with imitation learning? Well, because it’s clearly not the best approach to incentivize the emergence of real intelligence, as it makes it too easy for models to simply memorize the data.

Memorization is not intelligence; it’s memorization. Intelligence can be framed as the ability to learn patterns that can be applied to new data or new scenarios, a behavior we formally describe as out-of-distribution generalization.

Which is to say, we are still being mostly fooled by AI models that appear intelligent when, most of the time, they are mostly regurgitating patterns they’ve seen multiple times.

Using our earlier database comparison, despite there being definitely some compression going onf, they are still closer to databases than to real intelligent entities like humans.

So what do Frontier AI Labs do when they run into this wall? Easy, make datasets larger and thus make models larger too.

And that is the key thing I wanted to get to; all this explanation was to tell you that models are getting bigger and will only get much bigger.

Okay, so what?

Well, simple: Because this is the first reason why hardware is being governed by software; what matters is not the chip, it’s the system.

To understand what an AI data center is and how software changes are modifying how we build AI data centers (and how big they get), we first need to explain once and for all what an AI workload is by describing how most frontier models look like today: sparse MoEs.

What is an AI workload? That question is obvious to ask, yet you’d be surprised how almost no one can actually explain it in detail. Let’s fix that.

The most important thing to understand about AI workloads is that they are always distributed because, well, models are big. We’ll take a look at training and inference workloads in a second, but in both cases, the model is distributed across several GPUs, which we define as ‘workers’.

As I’ve explained countless times, and most recently here a few days ago, the type of AI models everyone is thinking about when discussing this massive AI buildout are Transformers, which are just concatenations of ‘Transformer blocks’ formed by two types of layers: attention and MLPs.

For example, for the sequence “The capital of France is…”, attention layers identify what the question is, and that we are speaking about France, and the MLP layers tap into the model’s knowledge, helping the word ‘Paris’ emerge.

We don’t have to go any deeper than this (in the previous link, I go into great detail explaining how all this happens).

Instead, it’s enough if we just understand one thing: ‘most frontier models today are sparse MoEs’. That is, most frontier models today are a variation of this architecture known as an ultra-sparse mixtures-of-experts (MoE) autoregressive LLM.

Yeah, I know, that was a mouthful. Luckily, it’s pretty intuitive, actually.

If we picture the MLP layers as the model’s long-term memory, in MoEs, we break these layers into smaller pieces from the very beginning, forcing the model to allocate knowledge in smaller compartments instead of having everything stored everywhere, which become ‘experts’ at different topics.

The advantage of using MoEs is that, during inference, an internal router decides which experts to query (i.e., which part of the model’s long-term memory is best for answering that question) and prevents the rest from activating.

This introduces the idea of ‘activated parameters’, which refers to the number of model parameters that are actually activated for any given prediction.

This is crucial because it allows us to run huge models at small-model compute requirements. For example, if you take Chinese Kimi K2, it’s a one trillion parameter model with 36 billion active parameters, so only 3.6% of the total model activates for any given prediction.

But why do this?

There’s more to this (there’s a reason we don’t just train a small model and call it quits), but think of this as running a super smart model at the compute requirements of a smaller one, or vice versa, running a small model with the “intelligence” of a large one.

However, despite being capable of running models much faster, MoEs don’t address the bigger issue: memory bottlenecks, as both the model and the sequence’s cache get pretty big either way.

Luckily, there’s a solution: breaking the model and sequences into parts across several GPUs.

We already know AI workloads are distributed. All of them. At this point, engineers have to decide how. The primary partition techniques used today are:

But which ones do we use? Well, all of them at the same time. If we look at how Meta trained the Llama 3 models, one of the seminal open-source releases, they use all the above except for expert parallelism (Llama 3 for a dense model, meaning it was not an MoE).

As you can see below, one of the options Meta considered for its GPU cluster was grouping into 64-GPU blocks, each of which was subsequently divided into 16 4-GPU blocks. But how do we make this graph make sense?

More specifically:

All in all, they trained Llama 3 by breaking models in both dimensions (across the model’s length and width) and by partitioning data and sequences (meaning a text sequence would go to only half of the GPU network and would be further broken down into pieces).

This is just one example, as today's workloads, which are all MoEs, are partitioned even further with expert parallelism (EP).

For instance, in the case of DeepSeek’s v3 training run (and subsequent 3.1 and 3.2 models, the latter of which I recently covered), they perform 64-way EP, meaning the model’s experts are broken down into groups of 64 (e.g., if we have 128 experts per model, each GPU out of the 64 gets two experts).

But even if we now understand parallelization techniques much better, we would still fall short of understanding how AI data centers are built today if we don’t explain the differences between training and inference.

The parallelization techniques are what they are, but we use them differently depending on the workload.

When training AI models, the amount of training data is so unbelievably large that our biggest bottleneck is getting through the entire data in a “human lifetime”. Wait, what? An example will make it make sense.

For instance, say we want to train a small model that, unlike Llama 3, fits in a single GPU. Let’s also suppose we want to give it training dataset with a total compute budget of 10²⁵ FLOPs (meaning the model has to perform 10²⁵ operations to go through that training data).

This sounds like a lot, but it is fairly small by modern standards (it’s approximately GPT-4’s training budget). If we took the most powerful chip in the world today, NVIDIA’s B300, with 9 PetaFLOPs of training performance (peak), assuming we could achieve such peak (we can’t), that GPU would require ~35 years to train a model on that “modest” dataset, and that’s assuming untenable peak performance.

Therefore, it’s not only a memory issue, but it’s also about sweeping through the dataset in a manageable time, at most a few months to avoid obsolescence risks (i.e., you could deliver a model that, by the time it is deployed, is already obsolete).

Consequently, we need large training clusters for a single model training run, with modern workloads requiring hundreds of thousands of GPUs per run.

Inference is another story. Funnily enough, quite the opposite, actually. Unlike in training, where you’re bottlenecked by total training run length, in inference, per-sequence latency is the primary concern, because you have a user waiting on the other side.

Here, slow inter-GPU communication is a significant problem, so we want to ensure the GPUs working on a user’s request are closer together, ideally in the same physical rack.

But do not confuse this with having small AI servers. Actually, as we’ll see in a minute, we want them to be huge! At this point, you may feel confused, but that’s because we need to differentiate between servers, clusters, and data centers to make sense of it all. This introduces us to the three primary metrics you need to know about AI data centers:

For instance, NVIDIA’s Oberon architecture, with the Blackwell Ultra chip, has 72 GPUs connected in a single physical GPU rack. This is the crucial metric for inference, where we want to keep workloads inside the server.

The problem with scaling out is that the networking hardware is slower than intra-server speeds, so you can grow to thousands of servers in total, but you take a hit in speed (this is why this is something to avoid in inference). As you can guess, this moves you from servers to clusters, which can be as large as the entire data center.

But here we run into a “problem”: RL training. As you already know by now, if you’re a regular of this newsletter, frontier models today are all “RL-trained”, meaning they were trained using Reinforcement Learning on top of an LLM trained via imitation learning.

And the “problem” here is that RL mixes both worlds of training and inference. But how?

Let me put it this way: in RL training, all training is inference. Wait, what?

Here, the model is no longer imitating the exact sequence you want it to learn. Instead, you give it a problem, and the model has to find the solution by itself (we guide it using rewards that score good actions and punish bad ones, hence the name of RL, but the model is literally “trial-and-erroring its way into the solution”).

Okay, so?

Well, it’s pretty obvious; if you want to train the model, you first have to let it run (inference) until it finds the solution, an entirely inference-driven workload, and once it finds it, then perform the learning phase.

In practice, as you can see below in Mistral’s diagram of how they trained their reasoning model ‘Magistral’, this means we have to separate your GPU cluster into groups of learners and inference workers, where:

And to enrich our confusion even more, generators (inference workers) can also be divided into prefill and decoding workers, because inside the inference stage itself, workloads are different enough between the two inference stages (prefill and decode) that you specialize GPUs for that, too. This is called ‘disaggregated inference’ and will become very important later in this piece.

And why am I telling you all this? With all we’ve learned so far, we can finally understand how hardware will change in 2026 and throughout the decade.

That is, we are now going to explain the five changes which I call the ‘Fiver Horses of Change’ happening in hardware and infrastructure, a beautiful executive summary that, in one shot, allows us to understand… basically the entire industry.

Understanding this is understanding the business strategies of NVIDIA, AMD, Amazon, Meta, Broadcom, Marvell, or Google, among others, and will help you position yourself better than most (including Wall Street analysts) for 2026.

After that, as a bonus, we’ll explain the biggest takeaway for me:

There are five ways in which hardware is changing based on software influences. Understanding these five ways enables you to predict what companies like NVIDIA will do over the next few years (at least throughout the decade).

The first big change comes, ironically, from the lack of change. I’m referring to the Transformer, the underlying architecture that, as we’ve discussed, has remained essentially unchanged for the best part of the last decade and shows no signs of changing anytime soon.

Yes, we are seeing some cosmetic changes, like MoEs and, more recently, DeepSeek’s DSA attention, but it’s still mostly the same architecture.

In this regard, some companies are literally printing this architecture into the chip. What this means is that, unlike other accelerators like GPUs (more flexible, runtime scheduler) or TPUs (very specific to AI matrix multiplications, but still architecture-agnostic), this Application Specific Integrated Circuit (ASIC), like Etched.ai, is the most ASIC example you can fathom, a chip that is designed for models running this specific architecture and nothing more. If the architecture changes, this chip doesn’t work at all.

This is a very risky take despite the pervasiveness of this architecture, but extremely dangerous for incumbents like NVIDIA if the architecture does resist the test of time.

For those reasons, this is the most asymmetrical bet: it’s huge for one company, bad for the rest. TSMC, memory, and networking players still benefit, but we’ll cover them in more detail in the other four sections.

And here we get to the most significant change and the one that could make you the most money in 2026.

This one feels somewhat predictable, but it’s unmistakably not priced in by markets. Models are getting bigger and will only get bigger, with some voices saying modern frontier AIs have up to 10 trillion parameters.

So what?

Well, because it has caused the two most significant visible changes in AI hardware in years: increasing scale-up and per-GPU HBM allocations, both of absolutely immense importance in this industry.

It’s understanding AMD and NVIDIA’s strategies and potential stock performances.

It’s understanding who wins and loses the chip wars.

It’s understanding why Broadcom and other notable players matter a lot, too.

And, crucially, it’s understanding why I have chosen three companies as the big, big winners of 2026.