Last week I wrote my longest newsletter ever, 8,500 words long, which covered the essentials of AI software and hardware, from the very basic key intuitions about AI models (what they are, how they learn), to the key intuitions in AI hardware that helped readers understand why GPUs and other accelerators are used, why memory is so important, and other key ideas the industry holds dear.

And finally, we discussed the “interesting” world of AI finance, particularly some numbers to understand how this entire industry aims to make money.

Today, I bring you the second part of the guide. In this one, I’ve focused more on the product, detailed AI inference maths, and future trends.

Let’s dive in.

Definitions of an agent are like opinions; everyone has one. Interestingly, despite everyone talking about it, most people don’t even understand what an agent is, when, in fact, it’s quite simple.

I’ve talked about agents in detail in the past, so I won’t dwell too much on definitions, only the essentials. An agent has three components:

At this point, it’s mandatory that we ask: why do agents need external memory in the first place?

We can understand the need for external tools (AIs can’t use tools directly; they are simply next-word predictions), but they do have internal memory and knowledge (known as parametric memory), so why the need for external systems providing extra context?

Doesn’t the AI know it all?

When you think about it, it’s pretty ironic that AIs, which have “seen it all,” need support from an external memory component; much like a human has Google search a smartphone unlock away, AIs need external memory too.

But why? And the answer is twofold: AIs are stateless and have a continual learning problem.

The first one is crucial to understanding the dynamics of what is going on under the hood when you interact with Gemini, Claude, or any LLM for that matter.

LLMs don’t have state. Every single interaction is like a new blank page in a book. Unless they’ve been trained on you, if the ChatGPT app doesn’t provide some context about you, when you click ‘New chat’ and you send something to the model, it’s like the first interaction it’s ever had with you. And if you click ‘New chat’ again, the model loses context from the previous conversation and starts anew with you.

What this means is that it doesn’t have an ‘internal state’ that carries from one conversation to the next. It knows everything, but can’t remember what happened 5 seconds ago. It’s as if an encyclopedia and Dory from Finding Nemo, who forgets everything every three seconds, had a baby.

That lack of state forces AI Labs to add it externally. When the previous conversation ends, unbeknownst to you, a background process may run, identify key details you shared in the conversation, and add them to all future prompts.

If you mentioned you liked Tom Brady a lot, the harness will catch this, add a “the user loves Tom Brady” snippet to future prompts, and if you ever ask it “Who’s my favorite player?” the LLM will “magically” know despite not really knowing it; if the harness decides to eliminate that memory and no longer add it to the prompt, suddenly, the model doesn’t know any more who’s your favorite sports star.

Of course, that also means that the memory system must work well. That process (which, by the way, is called ‘Dreaming’, which I wrote about here in case you want to understand it further) isn’t perfect and may not pick the right memories.

But the takeaway I want you to, well, take away, is that this is very much an external process, and no matter how much better an AI becomes at “knowing you”, it doesn’t really know you at all unless it’s trained on your data, which is never the case as things stand today.

Therefore, to this day, LLMs do not have state, and the quality of their knowledge about you depends entirely on the harness. But I know what you’re thinking: can’t we just train AIs on the user’s outputs so that they become knowledgeable about us by default?

And this, my dear reader, is the continual learning problem.

While we have cracked the code for training excellent models, we have yet to crack the code for retraining them at scale.

In other words, we don’t know yet how to retrain general models without breaking them. Careful, I’m not saying we don’t know how to fine-tune them. We do, of course, but we’re doing so knowing that we’re breaking them in one way or another due to what the industry calls “catastrophic forgetting”.

But companies like OpenAI or Anthropic do care about breaking the model (i.e., training on something while making it forget other stuff), because consumers can one day ask about cake recipes and the next about quantum physics.

Worse, it’s not predictable; the only thing we can predict is that it will happen, but not where. As shown below by research from Thinking Machines, training a model on internal docs improved the performance for that use case, while also catastrophically degrading the model’s ability to follow instructions, essentially killing the model.

But why does this happen?

Because you’re modifying the model and neurons are polysemantic, a certain neuron may be in charge of storing information about maths principles and also about turtles, so modifying that neuron could make it ‘drop’ the maths principles to absorb more ‘turtle data’ even if you didn’t intend that outcome.

In the meantime, our best bet is to simply have a good harness on top of models, providing that additional context in the prompt.

And what about the third component of the modern AI harness, tools? It’s important that you understand this often-neglected part of the AI story.

As the name suggests, the ‘tools’ component of the LLM harness “offers” tools to the AI to choose and leverage towards the assigned task. As we have discussed, AIs can’t use tools; they don’t have either a physical or a digital body (the harness is the body, in a way); they just predict tokens.

Therefore, when it wants to use a tool, it doesn’t respond with words; it responds with a ‘tool call,’ a declaration of intent for the harness to run that tool with those specifications and return the result to the AI.

Tools have numerous implications.

Regarding the fourth, I discussed it at length in this piece (I also compare CPU vendors), but the short explanation, as mentioned above, is that tools shift latency from the GPU to the CPU.

As shown by Intel below, for workloads that include tool calling, most of the model’s latency is attributable to the CPU running tools, not to the AIs generating tokens on the GPUs.

Today, with much heavier tool-heavy workloads, the picture is way worse. This is one of the primary reasons NVIDIA introduced an NVLink-type interconnect between the GPU and the CPU for Blackwell onward (which are traditionally connected via a PCI bus), effectively increasing the communication bandwidth between the host and the device by seven times (CPU and GPU).

And talking about tools lets us perfectly bridge the conversation with the effect all of this is having on AI as a business.

And it’s a lot.