Gemma 4 12B QAT: més IA local amb menys memòria

Hey everybody, Timothy Carambat, founder of Anything LLM, where we believe that you should own your intelligence as opposed to renting it from a cloud provider. Today, I want to make a video about Google's new Gemma 12B model. Now, there's two versions I'm going to talk about today, but the 12B version by itself is already impressive, but there's this derivative of the model called the 12B QAT, or Quantization Aware Training. This is a really cool concept, something that this channel is not super unfamiliar with already. It's already been talked about in a way in some of my other videos. So, the whole idea with Gemma, just to come back to this, was building intelligent models that can run on edge devices. And what people are talking about with edge devices is a laptop, and more importantly, a phone. And when this series just came out, it was the E2B and E4B models, which were really cool because they were optimized for multimodal stuff. So, it could support audio input and video input with some limitations, but more importantly, it could run on a phone. Compare that to the more desktop form factors, which require a good amount more of compute, where initially we got a 26B MOE, which was good, but a 31B, that was much better. So, initially people were kind of gravitating towards this 26B active 4B model. Now, this was a good model, and you can see it has 103 downloads this month for the safe tensor version of it. And it was a good model. It was meant to be essentially a stopgap for people who can't really run the 31B model fully. But if you go to the 31B

model, you can see it has far more activity, and also just a lot more, I guess you would say appreciation in the community. Both models are respectively good. It just depends on your use case and what you're doing, but the consensus did seem to come to, if you can run the MOE, it's actually worth it to just run a higher quantization version of the 31B model. And what I mean by higher quantization is instead of running the Q4, even going down to Q3 or Q2, just to run the model for 31B, it was just a much better experience across the board. So, when this model family came out, in my opinion, it was a bit all over the place, but it did hit a lot of targets at the same time. You had E2B and E4B. Super tiny, you could run it on a phone, and seriously multimodal with both audio and video understanding with some limits, but still. And then on the other side, on the more like compute-heavy kind of versions, you had the MOE and the 31B. And the 31B is truly great and intelligent. But, it felt like there was a weird gap in between the phone models and these kind of like desktop form factor models. If you compare this to a Qwen launch, it it did seem a little weird because Qwen typically releases a series of almost any parameter you could imagine, 0.8 all the way up to 325, right? And so, this felt a little weird, and so it was kind of thought that the set was incomplete for the Gemma series. And that is when they dropped a 12B model. Now, them dropping a 12B version

of the Gemma 4 series probably would have been just a nothingburger, honestly, had they not done some really interesting things with that model, and then doubled down on it and did something even more interesting. And let's talk about both. So, the 12B model was interesting because it is bridging the gap in not only size in terms of offering for this family, but also capability. Whereas, if you look at the E2B and E4B, it could understand all sorts of things, text, audio, video, and of course, image. But, it was using kind of of a smaller active parameter count layer inside of those models, which made them just a little bit slower, but they were more flexible. They also had a smaller context window than the MOE and the 31B. And the 12B has come into the picture and instead of just being a dense model that's 12B, they actually created a unified model which dropped the encoders for the image and video understanding. It's just in the model now and it doesn't take up like additional compute, which is really cool. So maybe to put like words to a visual here, you can see we have the E2B, E4B, and 12B, and the 31B dense. Now notice they call this unified and not dense and I think there is nuance there. And so you can see with the E2B and E4B, text, image, audio. Text, image, audio. 12B has text, image, and audio, whereas even the dense model did not even process audio. And you can see that the vision encoder parameters for the E2B and E4B, they were actually built into the model, baked into the model. The unified doesn't even have

them. That's just been dropped entirely, which is really, really incredible because this is actually typical. If you want these kind of modalities, there is typically an encoder and that has been that is just gone now from the 12B model, making it more lightweight and honestly easier to run across a large amount of devices. Now if you look at benchmarks, you can see that the 31B dense model is clearly going to take it away, right? At least that's what we would expect. Second to that being the MOE. But now we have this 12B unified that is honestly bridging the gaps in terms of capability and performance to the E2B, E4B models and these more compute heavy MOE and 31B dense. And an interesting detail is that the 12B unified is scoring significantly better than the MOE without thinking. And that's really great because now you can have this really great experience with a much lower kind of resource demand and have still a unified model where you are doing text, image, audio, and video. Still really cool. There is still, however, a limitation with the 12B and it's the same as the E2B and E4B. Audio is limited to 30 seconds and video is limited to 60. Frankly, I don't think this is a big deal. It's kind of also expected and all this means is that if you want to give it audio or video, you just got to pre-process it. So, chop up the video and the audio to its required duration and then send it to the model. This is pretty trivial. And then, of course, really great thing, also bridging the gap, is the E2B and E4B were at 128K

tokens, whereas the 12B and MOE and everybody else had a much higher 256K maximum context size. So, this is really great. You've got bigger context window, you've dropped the encoders, you still get the multi-modality, and it runs in a lower resource demand. That's all really cool. Now, that being said, I am still unsure how this model fits with respect to like a Qwen 3.5 9B, which is a much smaller, denser parameter model. Sure, it doesn't do audio and video, but most times that's not like a top priority thing for me, at least. It may be for your workload, and in which case, great, you should use it. Most people that I've seen using and having a lot of success with this series actually already do a lot of media-related work. So, a tool that maybe generates YouTube Shorts out of a YouTube video. That would be a great fit for this model because the vision encoder and the video encoder are already baked into the model and it's not additional parameter or even an additional model altogether, which is typically the case, honestly. And someone published this kind of like, you know, benchmark mashing between the 12B unified and the Quinn 3 and 1/2 9B. And Quinn does do better on these benchmarks. So, if you are a regular Quinn driver and you can run this 9B Quinn 3 and 1/2 version, it actually might be worth loading in this Gemma 4 unified model because there is a version of this unified model that actually demands even less compute than a typical 12B model would. And Google has accomplished this by introducing a not new concept, but maybe new to a I guess

you would say traditional lab release with a concept called quantization aware training, QAT. Now, as I've mentioned, this is not like some novel concept Google has invented. It just hasn't been done with a big frontier model lab release like this. And QAT kind of changes the mental math that it takes to run these models. QAT or quantization aware training is not a brand new concept. In fact, it's been around for quite some time. And in fact, actually one of the earliest papers I can find is when the Meta researchers were actually doing this with the Llama series of models, not even Llama 2, just the original Llama models. Where they saw that if you took a model that was already trained and applied this additional post-training step where you give the model essentially more data under the assumption that it will be operating with essentially less compute. It's a very gross simplification, but it's essentially training the model to be like, "Hey, you need to really operate more resource constrained in so that in the future when we do quantize you, you don't become dumb." Again, I'm overly simplifying that. But that's the idea. I mean, it's in the name, quantization aware training. If you've ever watched Dragon Ball Z, the hyperbolic time chamber is a decent analogy. You're essentially taking your model, which would I guess be these guys right here, sending them into the chamber, and basically saying, "Hey, when I deploy you, it's going to be much tougher out there than you maybe expect. So, go into here, train for an additional really hard kind of constraint, and then when you come out, you'll be ready to handle that

constraint." That's That's basically what's happening. I'm trying to make it as easy to digest as possible. Uh if you didn't watch Dragon Ball Z, uh the hyperbolic time chamber is basically like you spend uh a year in there, and it's a day outside. That's it. So, you can basically train really, really hard, and when you come out, it's like no time passed. So, it's like the same concept. Now, is Bit Net the same thing as QAT? I would hazard to say no. Uh I think it is maybe a subset of QAT because it is very similar, for sure. Whereas, you basically train a model under the assumption that, "Hey, your weights will be, you know, 1, 0, and -1." Uh this is something that actually might be more similar to how these models from Prism ML have been trained, where it's like almost quantization aware, right? It's an additional step after the model's been trained, uh where you can basically tell it, "Hey, you need to still be smart, but you're going to operate under more constraint in a real deployment fashion." Now, is this how they do it? I don't know, but I'm just saying that like one-bit models, ternary models, maybe are similar to QAT. They're definitely not the same as what Google has done with the 12-bit. So, why even do QAT? Well, it allows you to run a model that's just as smart, hopefully just as smart, uh with very little loss compared to its But, you run it on way, way, way less power than its non-QAT sibling. So, why even do parameters. So, let's test that model intelligence because a 12B model should be very capable of instruction

following. So, let's test that in Anything LLM. So, in Anything LLM, I have the Gemma 4 12B QAT version loaded. And I'm actually using LM Studio to do the inference for this. And you can see, I mean, it's it's a basic model, right? Yeah, I hope it can reply to a singular message. So, let's ask it to do the most basic of information where we can tell it to scrape a website. Just do some tool calling. And you can see we start getting output here. Keep in mind, I am on a MacBook 48 GB M4 Pro, so your mileage may vary depending on your device. Now, I did profile this model while running it just to see what my resource demands were, and it does take just a little bit more, uh, I guess you would say physical memory or shared memory than a Qwen 3 and 1/2 9B, which I feel like is not unexpected. It's a smaller model, but compared to a dense 12B anywhere else or anything in that ballpark, this is actually way more memory efficient. Uh, I didn't actually save the profiling for this, but again, it will also vary on your system. So, again, you should just test this model if you're interested. In order to really test a 12B model, for me, I mess with local models all the time. I typically hover around an 8 or a 9B simply because it's a good mix of intelligence plus functionality and also doesn't really like crush my computer's resources. But for a 12B model, I tend to be a lot more demanding. And one of the things I really like to test is kind of like multi-chain tool calling. Now,

this is an example prompt. You could try something else, but I wanted to go to Hacker News, find three AI related related links, jump into those links and summarize them, and then save that as a PDF. But don't only do that, style each element with a graphic from the pages that you scraped with something that maybe was on the page if it existed. Now, this is a simple prompt. If you were to send this to OpenAI or Claude or something like that or a Claude model, you would hope that it would handle it because it's a trillion parameters or hundreds of billions at least. For local models, that game doesn't really work that well. And you can see actually, I'm going to send this so we can talk about it while it works. I'm asking it to do a web scrape and then take that information, do an additional deeper web scrape, and then save that as a PDF. But then the instruction for the PDF actually comes after telling it to do that. So, we're looking at multi-chain tool calling, reasoning, and also kind of just keeping all of this in context at the same time. So, I'm going to let this run and we're going to see what the output is. And I'm just going to fast-forward through this section so we get an idea of maybe how long we're waiting. Okay. So, it looks like it came out okay. Um I have multiple resources that it went over. It looks like maybe it decided to cut one out of those cuz it I was full of only supposed to get three results, and you can see we get these

three links here. Um but let's open up the PDF that it actually went through. If we want to look at the entire tool call stack, uh we have it here. Um but let's just open the PDF cuz that's the thing I actually asked for. And so you can see we get TechCrunch, we get the source here, which is nice. We got the full link. We got an image that came from the page, which is pretty cool. We have the brief summary. And then we get Apple reveals the new AI architecture built on Gemini, which is funny enough we're talking about Google. We get a summary and then we get the Xiaomi Mi mobile blog talking probably about a new model release that came out. So, in general, I'm actually quite happy with this PDF and genuinely impressed. I've actually struggled to generate a PDF like this with much smaller models and this is kind of running in a similar memory ballpark. And I will say maybe something like this like this kind of prompt is maybe a little bit contrived, but there's actually a feature of anything LLM where doing a kind of like complex multi-tool call stack like this would thrive and that's actually this idea of scheduled jobs. So, we have this idea where you can, you know, have basically a cron job that you define. So, you would type in a name, we would paste in that same prompt and maybe we want to run it daily at 9:00 p 9:00 a.m. Um we could add the tools that we want to limit it to. In this case, it would be web scraping, web browsing and creating a PDF file just to keep the

context limited and we could create that job and then run it in the background autonomously. All that being said, I'm actually very impressed with this model. I've actually seen Qwen 3.59B fail pretty spectacularly on a task like this and so it does seem that this 12B unified is quite good at instruction following even with the QAT. Keep in mind, I'm not running the regular 12B. I'm running this quantization aware trained version of it. So, it technically should perform marginally worse than its full parameter unquantized, I guess you would say, counterpart. In general, I'm excited about these models, specifically the QAT version. The 12B unified is cool on its own dropping the encoders or the vision encoders for that, but the QAT version really drives it home because you get the larger context window with the 12B, you get the multi-modality with the 12B, but then you run it all on a much lower memory footprint with QAT. That's really cool. The fact that this is also similar, but maybe not the same as BitNet or the one-bit models or the ternary models, which we've seen promise with already, where we have, I think it was Flux 2 Klein image models from Prisma ML, the bonsai image gen, which I did a video I'll link here as well. I'm really excited about these models, and it seems like Google is actually doing the most for on-device. This makes sense. They do own Android, and however you might feel about Google owning Android, they have actually started to push some decent models, especially for the on-device segment. Now, obviously, I'm aware that they also downloaded, I think it was like some form of Gemma 4

on every Chrome installation. I think that's a bit egregious, but that being said, to really kind of evangelize for local models, the regular everyday person needs to not know that it is even local. It should just feel like magic to them because most people are never going to know what QAT stands for, and they shouldn't have to to use a model on their device. That being said, I'm super excited about these models. I'm still going to daily drive it for at least a little bit of time just to see how I like it in Anything LLM. We have some really interesting tools coming out with Anything LLM, where the speed and kind of like its ability to understand and reason about what's on your screen will be very important. And I'm interested to see how this model performs there. I'll also be making some more feature highlight videos about Anything LLM soon, specifically the one I just showed you with scheduled jobs. I still haven't made a video about that. So, expect some Anything LLM specific feature highlight videos very soon cuz we have a whole slew of features coming out in preparation for V2. Other than that, that's all I had to say. Thanks for watching. If you like local models or hearing about local models from a more, I guess you would say layperson perspective, then consider subscribing or leaving a like on this video. Other than that, have a great day. Thanks for watching.