From Stateless to Sticky AI

AI’s journey is like a blockbuster movie trilogy. The first installment of 2022-23 was all about building monster, skyscraper-sized brains packed with billions of parameters. Last year, it got clever with “reasoning” models like a math whiz solving a puzzle. However, it is somewhat bizarre so far that the models, which are capable of retaining the history of everything, struggle to incorporate your chat history. 

Quietly but surely, this is changing. AI is turning personal now.

A string of announcements in recent weeks provides the first signs of a significant shift. AI’s using your chats, files, and habits to give answers that feel tailor-made. It’s like going from a dusty encyclopedia to a friend who knows your favorite pizza topping.

This new era’s shiny star is a concept often referred to as persistent memory. It’s AI’s way of keeping a diary of your interactions, so it doesn’t forget what you said last week. Google’s all in with the announcements a week ago. It has begun the process of weaving the information it has collected over decades through emails, calendars, browser and search history, photos, and other applications into its AI to provide an experience that others cannot. Assuming it is not halted by court orders, Google - and to a lesser extent Meta - can personalize the way others have no chance of emulating. 

OpenAI is working hard to retain context across sessions, remember user preferences, and deliver increasingly tailored experiences that have roots in users’ other interactions with the models. According to leaked documents mentioned in an article over the weekend, its blueprint is to be everyone’s assistant for any access to the Internet. The recognition of how this could be the best way to prevent users from continuously switching to a more efficient or cheaper model should prompt others to follow suit soon. 

Persistent Memory: Changes What We Need from Models 

Let’s clear up a common mix-up first. “Persistent memory” in the AI modelling context is different from what the term has meant in the memory industry. For those familiar with “PMEM” (like Intel’s Optane, a type of non-volatile memory that sits between RAM and storage), AI persistent memory has no connection: here, it’s about AI models holding onto your personal context—your chats, preferences, or files—across sessions, so the AI feels like it knows you. 

For readers familiar with the GenAI landscape of the last few quarters, “persistent memory” may appear like a slam-dunk for things like Retrieval-Augmented Generation (RAG) or vectorization, and a reason to pine for the likes of Pinecone and MongoDB. While RAG and vector databases are brilliant at fishing out the right piece of information from your vast data ocean, true persistent memory demands more from the AI models themselves. It’s not enough for a model to just glance at a retrieved document; it needs to weave that context, along with the history of your interactions, into its very "understanding." This means models are evolving from being stateless prompt-processors to becoming stateful, adaptive agents. Imagine an AI tutor: RAG might pull up the relevant textbook page, but persistent memory allows the tutor to remember which concepts you struggled with last week, your preferred learning style, and actively adjust its approach. This requires new model architectures capable of dynamically updating and referencing an evolving "state" or internal memory about each user, going deeper than just a longer context window.

The Hardware Squeeze – It's Not Just About Gigabytes, It's About Giga-Access

This evolution towards deeply contextual AI naturally brings up the question of memory capacity. Does every user now require their own substantial allocation of dedicated RAM? Not exactly. While active user contexts will need to be cached in fast memory, the primary hardware challenge spurred by persistent memory isn't just about the total volume of RAM. Instead, it's about the speed, bandwidth, and intelligence of the entire memory system. When millions of users, each with their unique persistent context, interact with AI simultaneously, the underlying infrastructure must be able to identify, fetch, and deliver these individualized "context slices" to the AI models with breathtaking speed. 

This new hunger for instant data access means the old rulebook for AI hardware got tossed out. It used to be a simpler world, all about FLOPs – basically, how many raw calculations a chip could do per second. The AI chip with the most FLOPs was akin to a car with the most horsepower; it was often declared the winner, making the "fastest GPU" the primary prize. But when your AI needs to constantly fetch and use your specific memories to be useful, raw chip speed alone doesn't cut it if the data can't get there on time.

Suddenly, the spotlight swings to the "interconnects" – the super-highways that data travels on. And these aren't just simple roads; they're a complex network at every level. There are connections within the AI server linking multiple processors together (think NVIDIA's NVLink), pathways from processors to that crucial fast HBM, and even bigger network pipes (like InfiniBand or ultra-fast Ethernet) lashing thousands of servers into colossal AI supercomputers. If any of these data highways get jammed, the whole AI factory slows to a crawl, no matter how powerful the individual chips are.

So, it might seem like figuring out who has the strongest "moat" – the biggest competitive advantage – in AI hardware just got a whole lot more complicated, right? We've gone from just counting FLOPs to juggling memory speeds, HBM capacities, and a half-dozen types of interconnects. Yet, here’s the twist: while the understanding of what makes a winning AI system has become more complex, the top player in this game hasn't really changed. In fact, because they've been building for this integrated, system-level future all along, their moat has arguably only gotten wider and deeper. And for now, that’s perhaps the most crucial thing to grasp.

The rise of AI factories—massive, coordinated systems powering personalized AI—has thrust hardware integration into the spotlight. While we could dive deep into the technical weeds, we’ll spare you the jargon overload. It has been known for years that NVIDIA doesn’t just build fast chips; they architect entire ecosystems. In the era of persistent memory-based AI services, this vision and the competitive gaps it has created will begin to emerge.

Subjective Ratings on a scale of 1-to-10

NVIDIA's Expanding Moat vs. The Open Rebellion 

NVIDIA's dominance was once primarily attributed to CUDA, its proprietary software ecosystem that effectively locked developers in. While CUDA remains a powerful barrier, the rise of AI factories has seen NVIDIA strategically erect even more formidable proprietary walls with its interconnect technologies, such as NVLink (for chip-to-chip communication) and InfiniBand (for cluster-scale networking). These technologies are crucial for the high-speed data movement required in persistent memory and large model training, and NVIDIA's tight integration across its stack provides a significant performance edge.

The rest of the industry isn't standing still, of course. A coalition of major tech players, including AMD, Intel, Google, and Microsoft, is championing open standards to counter NVIDIA's proprietary grip. Efforts like UALink (Ultra Accelerator Link) aim to provide an open alternative to NVLink for chiplet and direct chip-to-chip connections, while CXL (Compute Express Link) offers an open standard for high-speed CPU-to-device and CPU-to-memory connections, fostering more heterogeneous computing.

However, the gap between these emerging open standards and NVIDIA's deployed, optimized, and integrated proprietary solutions is substantial. NVIDIA's power of integration – how its GPUs, NPUs, DPUs, networking, and software all work in concert – creates a performance and efficiency advantage that simple numerical comparisons often fail to capture. In many ways, this integrated approach means NVIDIA's lead is not just maintained but potentially widening, despite the industry's push for open alternatives.

The Bottom Line: Navigating the AI Gold Rush Just Got Trickier

Let’s cut through the noise. Persistent memory’s rise makes AI personal, but it’s also turning the AI investment game into a complex puzzle. Gone are the days when picking winners was as simple as betting on the fastest chip. Now, it’s a tangle of interconnects, memory systems, and software ecosystems. Understanding who truly benefits from the AI spending boom now demands a much deeper dive into how these intricate systems – from silicon to software to the data itself – actually click.

You might think, "Easy, I'll just bet on the data kings like Google or Meta!" Their treasure troves of user information are indeed potent fuel for persistent, personalized AI. However, the world's regulators and courts are increasingly likely to have a significant say about those data moats as they recognize the significantly increased power of data collected, potentially redrawing the competitive landscape overnight. And while NVIDIA’s integrated fortress looks imposing today, the tech battlefield is relentless. Expect a constant barrage of disruptive innovations in AI algorithms, nimble edge computing solutions, and new connectivity paradigms aiming to challenge the status quo. No one, not even the current champion, gets to rest easy.

Ultimately, the sheer pace and evolving intricacy of AI might seem overwhelming. However, perhaps there is a silver lining to all this complexity, in that the tools that aid us in navigating it are improving at a faster rate.