As promised a few days ago, this post is going to focus on the nothing-short-of-crucial release of DeepSeek v4 from the homonymous company, because it reveals a lot about the state of Chinese AI and, overall, the future of the industry.

How much behind are they? Are they behind at all? What’s the long-term plan?

We’ll answer all these questions today by presenting the main insights from the paper. If you read this piece in full, you’re going to:

This piece will leave you with a bittersweet taste, especially if you’re concerned about US supremacy. Not because China has “caught up”, we are way past that simplistic conversation that judges who’s ahead based on who scores higher in contaminated benchmarks.

The real battle is being fought elsewhere, and I can’t help but feel quite worried about the US's chances if things don’t change before it’s too late.

You’re about to learn a ton about frontier AI today, so let’s dive in.

First, let’s start with the official release numbers, which, in themselves, are very impressive for an open-source model.

DeepSeek has released two new extremely cheap open models: DeepSeek-V4-Pro and DeepSeek-V4-Flash.

Both support context windows of up to 1 million tokens (i.e., they can handle ~750k words in context, so you can feed them a lot of information), making the release especially well-suited for long-context reasoning, coding, search, and agentic workflows.

But let’s dissect its claims more. On general knowledge and factuality benchmarks, DeepSeek-V4-Pro-Max, the highest-reasoning version of V4-Pro, shows a major jump over previous open models but still trails the strongest proprietary systems in some categories.

One of the strongest parts of the release is the models’ reasoning, coding, and maths capabilities.

But the model’s strongest suit is, undeniably, its long-context results.

On agentic benchmarks, DeepSeek-V4-Pro-Max is competitive with frontier models but not consistently ahead of them.

So, overall, we’re talking about an almost frontier-level model that’s free to download. Not quite Opus 4.7 or GPT-5.5-level, but pretty incredible considering that puts it a few weeks behind the absolute frontier.

The important thing to note here is that this is a radically different model relative to the others. It’s one that presents an extremely novel architecture, unique enough to warrant this entire newsletter.

But to capture DeepSeek’s genius, we need to understand how frontier AI models are built in first principles.

Inside every single model you’ve used lately, and I mean every single one, lies a piece of software known as a Transformer. Conceived in 2017, it has become synonymous with AI.

Just as no one doubts the use of sugar in any sweet recipe, no one doubts the use of the Transformer as the foundation of any model aspiring to be good.

As the name suggests, it’s a model architecture that works by ‘transforming’ the meaning of data, be that words, images, or audio.

But what do I mean by that?

In AI, or in any software, data is represented as numbers (recall that digital machines can only work with numbers), where each number can be seen as an attribute that, when combined, defines what a data point represents.

For example, say we have a toy model that does nothing but classify food by its ‘dessertness’ or ‘sandwichness.’ An apple strudel can be represented as [0.5, 1], a food with a decent degree of sandwichness and, undoubtedly, a dessert. This means that, in the model's eyes, an apple strudel is nothing but two numbers: 0.5 and 1.

Modern AIs have thousands upon thousands of dimensions, meaning they can get extremely granular in how they categorize each concept (e.g., applying different levels of dessertness, such as sweetness, texture, etc.).

Of course, the more attributes two concepts share, the more “alike” they are. This allows the model to know that ‘hot dog’ is closer to ‘shawarma’ than it is to ‘apple strudel’.

More importantly, it’s key to understand that meaning is not static; it can be modified. Words like ‘king’ and ‘queen’ are closely related, mostly separated by the ‘sex’ attribute.

Thus, if you give an AI model the word ‘king’, it can represent a ‘queen’ by taking out the ‘man’ attribute and adding the ‘woman’ one, giving you ‘queen’.

The third and final insight is that meaning is contextual. Say we have the word ‘bank’. It has not one, but several intrinsic meanings depending on the context.

Thus, we need AIs to do all three things:

And the Transformer is the perfect architecture for this. When a Transformer architecture like ChatGPT receives a piece of text, it processes its meaning and adds relevant information to predict the next word. In other words, it’s mostly a meaning-update process.

And the reason it works so well is that it breaks down processing data into two dimensions: context and knowledge.

As I always say, AI models are just ‘maps’; they map an input to an output.

If my experience tells me 5-bedroom homes rarely go below one million dollars, whenever I see a 6-bedroom home, I’ll infer that it’s very likely more than one million dollars.

But at heart, AI is mostly statistics, but on steroids. Hence, at their core, they all do the same thing: capture patterns in data and use them to make inferences about new data.

And the tool they all overwhelmingly use to capture such patterns is the Transformer.

However, they differ in how they “work” with the input. In the housing scenario, the quality of the model’s prediction is mostly about knowledge: how much housing data it had seen previously.

Large Language Models (LLMs) add a twist: context matters just as much. The model has not only seen what’s basically all public digitized content the world has ever created, plus proprietary data that AI Labs create, but it also receives real-time context.

For the sequence “Jimmy ate some peanuts, what do I do?” that seems like a harmless, and quite frankly hard-to-answer, question without more context, but if the model knew ‘Jimmy’ is the user’s peanut-allergy-maximizer son, that question is clearly a medical emergency.

Alternatively, even if the model doesn’t know Jimmy has a peanut allergy, if the user had written “Jimmy ate some peanuts by accident, what do I do?” the model can already guess Jimmy probably shouldn’t be eating peanuts.

And here’s where the Transformer truly starts to shine. A Transformer model, a model like GPT-5.5 or Opus 4.7, is a concatenation of ‘Transformer blocks’ where each block is made up of two things:

Intuitively, you can think of a block as a transformation (hence the name) over the words. Each word is updated contextually (with respect to previous words and the model’s knowledge on the matter).

For instance, earlier blocks might focus on capturing how nouns and adjectives relate to each other, while deeper layers can represent much more complex structures, such as compositional reasoning.

As you may guess, the more blocks you concatenate, the more transformations you can apply, the better results (hence why bigger models are ‘smarter’).

And how do we scale this architecture besides just concatenating more blocks? In MLP layers, one way to make them more efficient is to split them into ‘experts’.

We’ll talk about it later because it will become important, but for now, in terms of innovating over the quintessential Transformer architecture, it’s the attention mechanism where DeepSeek is redefining how we do it.

The most impressive thing about these models is two things: cost, and long-context performance, both of which can be considered absolutely at the forefront amongst all models, closed and open.

With regard to the cost of the DeepSeek API, the numbers are almost embarrassing, with some multiples reaching ~80-100 times cheaper.

And the culprit behind this incredibly cheap numbers, besides the fact DeepSeek isn’t nearly as much in debt to investors as OpenAI or Anthropic (their key costs are not running AIs, but securing hardware and training) is this new attention mechanism, a first in this industry that beautifully explains why these models are technological marvels that represent an order of magnitude decrease in costs relative to the previous generation.

And to explain this, we first need to understand why the attention mechanism can play such a larger role in costs.

As explained earlier, attention is the mechanism that the operator models use to process the input sequence, which requires words to “talk” to one another. This ‘talking’ is a very effortful computational process we call attention, which grows proportionally to sequence length.

Think of it like this: if you grow a sequence from 100,000 words to 1,000,000, the one millionth word has to attend to 999,999 previous words instead of 99,999. That’s not free.

Of course, an ideal model would not require attending to all 999,999 previous words, only those that matter, meaning attention should be dynamic and input-dependent, not a fixed computational cost per word.

By ‘matter’, we can intuit that, for a given word, recent words provide more meaningful information, called ‘recency bias’ (e.g., a noun is more likely to be affected by adjectives close by) and that some words are higher quality than others by default (e.g., somewhere in the 999,999 previous words there’s probably a ‘meh’ or ‘uhm’; words that provide little or no value relative to words like a character’s name in a novel).

This is what we call ‘sparse attention’, reducing computational costs by making attention more computationally efficient.

Despite this, most models make no assumptions about attention, and instead use global attention (assuming all previous words are worth attending to), which, for very long sequences, can make each new prediction require an overwhelming amount of compute.

This sounds awesome, but it’s not only expensive, but it’s also hard to “get right”. This is something known as context rot, where the model simply can’t handle too much context.

In fact, most of the long-context claims AI Labs make are bullshit: one thing is for Google to claim Gemini can process one million words; the other is that it does so well.

As you can see in the table below, the MRCR long-context benchmark shows that model performance degrades significantly from 128k words onward, let alone 1 million, with most models becoming unusable, while only GPT-5.5, released last week, maintains good quality, besides DSv4.

Nonetheless, if you look at the sequence below, you see that DeepSeek models handle long context best by far, and only the recent release of GPT-5.5 saves the US from total humiliation.

So, how did DeepSeek pull this off? And the answer is hybrid attention.

A typical trick to make attention more cost-effective, popularized by Mistral and adopted here by DeepSeek, is sliding-window attention.

For a given word, it can only attend to a certain number of previous words, a value that is way smaller than the actual sequence length.

In the image below, ‘on’ can only attend to ‘cat’ and ‘sat’, but not to ‘The’, which is too far away (out of the window), because the window is three tokens (including the token itself). Cumulatively across all tokens, this implies significant cost savings because each word imposes much lower computational requirements.

However, this decision doesn’t come free.

By enforcing strict recency bias, the model is guaranteed to miss key long-horizon patterns. For example, in a ten-chapter crime novel, a key detail to uncover the mystery might have been mentioned in the first chapter. If the model can only see three chapters back, it’s going to forget that key detail.

This is a toy example, but in practice, this seriously undermines model performance. An alternative is to use hybrid attention, mixing local attentions (short-range) with global attention layers in a pattern that’s usually 5:1 (5 short-range attentions for every dense one).

And one of these hybrid attention mechanisms is precisely what DeepSeek is proposing. And it’s a big deal.

In late 2025, DeepSeek released DeepSeek Sparse Attention. The gist is to add a ‘selector’ that lets each word decide which previous tokens it wants to attend to, considerably reducing computation requirements.

The idea is that, instead of forcing the word to look back to only a certain number of immediately previous words, it can “skim” the entire sequence and choose what words it wants to look closely at.

Theoretically, this would be the ideal solution to grow to arbitrarily large sequences because we keep attention computation in check, and the model can still decide what’s important.

However, the skimming part, the part where the model still has to skim through the entire sequence to select the words that matter, is still very taxing. So DeepSeek has taken a next step and provided the most aggressive attention mechanism we’ve ever seen.

The idea is to break the sequence into chunks of words, compress the information in each chunk into a single representation, and then share that representation with the word. I know this is confusing, so let’s imagine a simple example.

We have a 1,000,000-word sequence. How does DeepSeek v4 process such a humongous amount of data? For reference, this is a couple of hundred thousand more words than Harry Potter’s entire saga.

The answer is Compressed Sparse Attention, or CSA.

Recall that the naive approach is for the millionth word to attend to 999,999 words. The 999,999th word has to attend to 999,998 words, and so on. This is incredibly taxing computationally.

Instead, the DeepSeek models break the sequence into chunks of, say, 50,000 words (20 chunks) and compress the information in each chunk into a single representation, which can be thought of as a summary of that chunk.

This is, of course, a considerable reduction in information, but at least the model can look way back with ease; even if it doesn’t see each individual work, it sees summaries of all that happened previously.

Additionally, each word gets dense access to the most recent words using SWA. If the SWA window is 5,000 words, that means that a word, say the 500,000th, receives information from 10 compressed 50,000-word chunks, plus individual access to the last 5,000 words (from the 495,000th onward).

Then, the model selects “what chunks matter” using the selector explained earlier, so that each word can see everything that happened recently, along with selected summaries of the past.

The DeepSeek team then added an even sparser attention step called Heavily Compressed Attention (HCA), which, as you may guess, is an even more radical compression of what we were discussing just now (meaning each chunk is larger, basically).

And considering that DeepSeek scores SOTA in long context, we are safe to claim that this is literally a SOTA attention mechanism; the first time we have a fully hybrid attention mechanism taking the first spot in expressiveness.

But this isn’t close to all the new stuff these cracked Chinese guys have presented to us.

Next, we’ll cover the remaining research breakthroughs, both algorithmic and in terms of infrastructure, and cover why this is a much bigger deal than meets the eye because while the US frontier is still focused on ‘big number go up at all costs’, Chinese Labs, severerly compute constrained, are dong ‘big number go up, but only if it does so efficiently’, which is music to the ears of the big customers in the AI game: enterprises.

At the end of the piece, we’ll turn to geopolitics and explain why DeepSeek v4 might be the first model the Chinese Communist Party can actively weaponize against the US.

Another of the big algorithmic breakthroughs in this paper is perhaps the most interesting.

Why?