Nemotron 3: per què NVIDIA barreja Mamba, MoE i MTP

Over the past six months, Nvidia has been releasing Neymotron 3 in 3 variants, but why exactly 3? What criteria did Nvidia use for sizing these models? We try and size them based on the most common compute for print available for many videos in install days. And we do that, so there's a variety of both accuracy and cost latency throughput trade-off, essentially. This, by the way, is joy from Nvidia during our interview. So the size of nano and supervayrins target the chip layer and the ultra targets the infrastructure. For example, NIMO-TRAON 3 Nano is a 30-billion-primator model meant to run on consumer hardware that we could get at home. The Nano from December is a 30-billion-total parameter active 3 billion. The MVFP4 allows us to run that model at essentially half FP8 memory capacity. So instead of 30-billion total, say gigabytes, we can now run it around 15. Now, the super variant is 120 billion in size similar to OpenAS-UPTOS as 120B. Now, you could run this on

or DJX spark, but the Compute Profile mirrors closer to the server-grade GPUs like the H100 and A100. And super, it's 120 billion active 10 billion. And inside of there, say like on 1, H180 gig, we can run an FP8, the NVFP4 runs great on the Blackwell architecture. And so say like B200, B300, there's not just a memory reduction but a computer acceleration there. That's also what is in DJX spark and allows the super monitor run on a smart footprint and DJX Spark. And we'll see more of that going forward with things like DJX Station as well. Finally, we have the Ultra, which is massive. 550 billion parameter model with 50B active, similar sizing to models like DeepSig V3. At this scale, you're going to need more than one GPU to run models like this, which is why the Ultra is fitted for AI factor level at the infrastructure layer. On the ultra side, we're looking at

around 550 billion total. And as we went through and looked at some of the base model accuracy valuations. And so these are essentially kind of next token predictions. So it's not, it's not reasoning and it's not destruction falling. But even looking at previous base models out in the community, we were able to see the best accuracy scores we've ever seen of any base model published. As you can see, Nvidia is slowly moving up the AI 5 layer cake and pushing out models like Neymotron 3. And because Nvidia practically dominates in the lower layers, they haven't advantage in fitting their model architecture to be fully optimized at the model layer. In other words, Nvidia can leverage their hardware to make the best fit for the model given various hardware profiles. So the natural question here is how exactly does NemoTron 3 stand out against?

other models in the model layer. What architectural decision did Nvidia make at the model layer to fully optimize the hardware on the Nith. When it comes to the model architecture, there are three things worth noting about Neumotron III. Hybrid Mamba Transformer, Lates & Moui, and MTP. Now most of you might have never heard of these terms before. So at first glance, it can be really difficult to contextualize the significance of these. But I'll walk through them one by one. Let's start with the hybrid Mamba Transformer architecture, which directly targets the biggest challenge when it comes to scaling large language models, which is attention. Yep, we've all heard it before. Attention is expensive and this and that, and we also hear things like attention makes compute scale quadratically as the context window grows. All of these are definitely true, and we certainly are need of a good solution for it. Especially if we're going to offer a context window in the size of 1 million.

Nobody in the right mind would release a model with one million context we know without doing something about this quadratic scaling in attention. And for Nvidia's case, they decided to take a shot at this by changing the model's architecture itself to use what's called Mamba 2 and then interleaving them with full attention layer. Some of the more traditional attention mechanisms can be a bit more quadratic and they grow significantly the longer the input. Our concern with that is that often in an enterprise setting, you'll have existing knowledge bases or experts and there's a lot of context to feed a model and it can give a much more accurate answer if it has a thorough understanding of what the question's about. Even something simple like, you know, what's the tire pressure, you know, in a car, being able to go through the owners manual and pull out the sections relevant to how to measure the tire pressure and what the recommended number is, being able to feed those into context are concerned is that it would grow the memory significantly, which then reduces the memory we have for concurrent queries. And so in thinking about things as an AI factory where a lot

should be able to handle many queries concurrently. We were getting frustrated by the amount of memory used. So given this scaling problem when it comes to attention, commonly labs would target this problem by swapping out the attention head from multi-head attention to different ways like GQA and MLA to essentially minimize the KV cache. There are of course other ways like turbo quant where you quantize a KV cache itself to lower precision, but Mamba 2 works completely different than any of the ways above. And so one of the exciting things about Mamba and kind of the state space mechanism there is that it's more linear than it is quadrat. And in that sense, we can be able to take these longer input sequences and use less memory. The actual mechanics of Mamba to is quite technical, but unlike traditional transform or architecture, Mamba or state space models don't actually use KV-Cache at all. Instead, memory requirement is constant since the representations are stored in a fixed size matrix that updates as tokens are brought.

And this might sound really familiar because it sounds a lot like how recurrent neural networks actually work given their hidden state that updates as tokens are processed. And Mamba2 essentially compresses the past into a state then uses a hardware friendly algorithm so those state updates can be done as a big matrix multiplication and paralyzed efficiently. And we couldn't really do this with RNNs. And since we risk losing broader dependencies when using mechanism like this interliving Mamba2 But two layers with full attention is how Nimotron was able to keep their inference so tight and offer up to one million context window without too much sweat. Now let's go to the second point in the architecture. Late in MOE, and this is a really interesting concept and here's why. Quick shout out to Code Rabbit sponsoring this video.

Ever since I started relying on codex and Cloudcode to write code for my projects, not only did I start making more pull requests. I find that my focus tends to be more on the flow state to develop things than actually reading the code that was generated with coding agents. CodeRavid works besides me that reviews my pull request automatically with the entire repository mind. And part of its review includes fixing changes, security and quality checks and most importantly gives actionable changes that's ready to be merged right away. You can also use CodeRavid agent in Slack. So you and your team don't have to leave your usual environment to talk with your agent and bring the context of your repository directly into your workflow. My favorite feature is that code rabbit will remember you and your team's preferences. So as you continue to work with code rabbit, you will get better over time. Check out code rabbit using the link in the description below. Now we all know how mixture of experts or MOE works were instead of activating the entire model's weights, you sparsely activate the model. Let's say 10% of the models weights are activated instead of a hud

Now on the hardware side, you see a huge improvement here since only 10% of the model's weights needs to travel from HBM to S-RM for compute to happen. Now imagine we disperse these experts across a node of GPUs through what's called expert parallelism. Spreading the experts like this actually speeds things up because you now have multiple funnels that carry the model's weights from HBM to S-RM. And since the funnel which is the bandwidth that typically gives you three to 8 terabytes per second you can have models like Dimitron III Ultra which is a 500 billion model which is roughly 275 gigabytes in V-RAM in 4-bit precision takes anywhere between 91 milliseconds to 34 milliseconds using just one funnel but having 8 funnels that each have 3 to 8 terabytes per second essentially speeds up the read rate

theoretically, since each GPU should only have roughly 35 gigabytes to stream from HBM to S-Rim. And because we're only activating 10% of the model on top through mixture of experts, we can go ahead and slash that number down even that much more. Now of course, this is only looking at the hardware side, which means the model architecture also needs to make sure that the quality and accuracy isn't dropped from this kind of setup, because clearly it's a benefit on the hardware side. The company that really took mixture of experts to another level was none other than DeepSeek. DeepSeek is essentially proof to the world that we can add load balancing to make sure that you don't underutilize or overutilize experts and they also find a way to really specialize each expert to make this kind of parallelism work in quality while having one shared expert for general knowledge. So what about Neemotron? What does this really look like in Neemotron? Turns out, beyond DeepSeek's breakthrough when it comes to making mixture of experts action.

work, there's still a lot of room left on the table when it comes to hardware optimization. And Nvidia introduced what's called latent MOE targeting this very gap. And the core idea behind it is this. What if we have a down projection on the dimension so that each token being processed is reduced in footprint? This way we can get away with activating our experts on a much smaller latent representation, which cuts down memory bandwidth and also cuts down on compute that's typically needed to routes and process each token as it comes. For specifically for the late Moui, every query goes for the router, and we wanted to think how do we make that router more efficient? How do we think about getting more experts exposed to pick the right one? And so, you know, in that scenario, what we thought about was for the late Moui, we wanted to be able to essentially kind of reduce the footprint that compute in the memory it takes and be able to get more experts to weigh in on a query to make sure we're always picking the best one. So what you find is that this kind of

architecture actually creates a more room since you're doing math on latent representation. An Nvidia made a design choice here to use their surplus that we have gained to pack in more experts which gives each token ability to see more experts than it would have otherwise. Pretty clever right? Now we finally get to MTP or multi-token prediction. And this has to do with how output tokens are generated by the model. And these are the tokens that would typically see on screen as model. Generates tokens. Traditionally, tokens are generated auto-regressively, which means the model will generate the next token and takes that into account to generate the next one.

one at a time. An MTP is meant to address this very specific region of the model by changing up what the model can see during auto regression. Now I wish I could cut here to Joey talking about MTP, but I totally forgot to ask Joey about Nehotrans incorporation of MTP. So I guess you're stuck with me trying to explain it, but I'll do my best. Multi-token prediction, let's the model predict not just the next token, but the tokens after that. We can have the model produce, let's say next five tokens. which theoretically gives a model deeper intuition on how the next five tokens would look like if the model could generate them.

Now, MTP is typically used for training to make the model more expressive and theoretically have a better forward thinking ability as it generates token after token. But it can also be used during inference as well in the form of speculative decoding where the model drafts few tokens ahead then checks them in one pass and keeps ones that match what it would have generated anyway. And Neymotron III allows speculative decoding to be done using MTP which theoretically should speed up your token production side, since it could skip over multiple tokens as it predicts and accepts tokens beyond the next coming token. Now that we covered with a broad stroke, what Neymotron architecture looks like, let's briefly talk about licensing. We all know that labs tend to flip flop between how open they really are when it comes to their model releases. And what's even more confusing is licenses like,

patch II, which can be confusing when it comes to scope. And that's largely because Apache 2.0 is meant to be used for software, not for AI models. For example, we know that many open labs release the models' weights, but they also release code that's used to run them for inference, as well as few other artifacts, like documents, oral environments, and training recipe. So the question really becomes, when a lab says their model is open, what exactly does that mean? Recently, Linux Foundation revised their open MDW license from 1 to 1.1. To help reform the language around AI model licensing and Nvidia adopted this very license to clarify their stance on Neymotron model along with some of the other things they're working on like Cosmos, Isaac, Ruts and more.