Bonsai 8B: què guanya i què perd el model d’IA d’1 bit que ocupa 1,15 GB?

There is some very basic level functionality here and that is impressive. For today's video we're going to be looking at a very interesting new release called Bonsai. 8B. Now this is essentially a one-bit model and this is based off of Q13 8B and this has shrunk down to a one-bit model. Now there is a lot of technical mumbo jumbo behind that and I think the simplest way to essentially at least introduce the concept of this is to just showcase it running right here. So I have this running in Lama CPP server. This is an 8-billion parameter model that has this one-bit kind of quantization to it which we'll go into a little more in depth. However, let's take a peek at this. So this is currently using 3.2 gigs of video RAM on my system for an 8 billion parameter model. Now the context length right here is set to 8192. If the context length was set to the maximum available for this model, it would be using a little over 11 gigs of VRAM. But as it sits right now, this is an 8 billion parameter model using 3.2 gigabytes of video RAM on this system. To that point, this system by itself is probably using around a gig at the floor. So this is using like 2 to 2 1.5 gigs of video ram right here. So I'm just going to speak to it normally. Hey, what's up? Punk. Question mark. Hey, I'm not a punk. I'm an AI assistant. I'm here to help with questions, explanations, and conversations. What's on your mind? Now obviously we're going to do some more intricate testing. The fact that this thing is even loosely responding and understanding enough to say like, wait a minute, I'm not a punk rather than just like flat out gibberish or something that does not even grasp the nuance of me calling it a punk. is quite fascinating. So there is definitely a lot of interest in things like this because this is really and as we'll see a significant reduction in size, memory footprint, energy utilization for a model that hypothetically contains the intelligence of an 8 billion parameter model. So that is kind of the TLDR for today's video. So let's begin with a brief look at the introductory announcement post for this, which is touted as the first commercially viable one-bit LLAP. And if we scroll down here, we're going to notice this isn't necessarily a very long announcement post, but the culmination of this is just talking about this model, specifically the 8B for this release post that we're looking at right here and how it stacks up to other models in terms of both its size as well as performance. So we can see right here, the one bit bonsai 8B is a total size in terms of like the weights and everything you download here from the hugging face model card of 1.15 gigabytes. This is significantly smaller in footprint than every other model listed here in this benchmark chart.

And for what we can see, supposedly based on these benchmarks it performs very favorably, especially when you consider that they do mention some specific figure right here. It is around 12 to 14 times smaller than all of the other competitors. Competitor models listed in this benchmark. So this is very exciting to see and quite interesting because some of the additional things they talk about here if we scrolled down mentions more about like energy efficiency and utilization and energy consumption Things that are important in the consideration and the evolution of wanting to have better AI in terms of AI that's equally as smart as preceding AI but more efficient and this would definitely go hand in hand with that. They also talk about running this on mobile phones. I do believe there is a demonstration video here of this running on an iPhone or there is a yep. So they mentioned an iPhone 17 Pro Max running at 44 tokens per second, which is extremely fast for a model to run on a phone. That is quite a good speed. They also mentioned hardware here specifically in the combination of this is just saying that current hardware is designed to run like full precision models and things like that. The speed up for this one One bit model is just because of how much more efficient it is and not because hardware has been optimized for it at all, which could then lead to further increase in speedups or something of the sort. And beyond that, we also see that there are two additional sizes as well as the AP. There is a 4 billion parameter variant and a 1.7B. Now this section here, I was reading online just seeing folks in a short reaction to this. And I think this was somewhat tripping folks up because this says this is a proprietary one bit model design across the entire network. There are no higher precision escape hatches. It is a true one-bit model and end-to-end across 8.2 billion parameters. Now, that I think could be somewhat confusing because that would be like full-on alien-style technology I would think. If we go here into the hugging-face model card and then just click on this little icon next to the actual file, first and foremost, we do see the GGUF for this is 1.16 gigabytes, which is insanely small for an 8 billion parameter model. But we can also see a bit more pert-

information about the actual model and the layers and its architecture and things like that. And we do see there are some higher precision elements in here. I think that's to be expected for this to actually be real because if those were not there like I said then again don't quote me on this fact check and verify everything but I would believe that would be like very alien style technology so additionally to that they do have this white paper of course which talks a little more about the technical specs of this and we can see in there it is 1.125 bits per weight and in this paper we do get the answers to probably some questions out a lot of folks are going to prominently have which is okay like how do I do this to any model that I want this to be done. And that's sadly where it says the foundation comes from proprietary Caltech Intellectual Property that addresses a longstanding research challenge through rigorous mathematics rather than adhoccuristics. One bit bonsai 8b is therefore not a collection of narrow exceptions built for a controlled demonstration. It is a principled compression framework designed to preserve useful model behavior while materially lowering the cost of inference. So in this paragraph is something both exciting and something sad depending on whether or not you're an open source advocate or just someone who wants to see furthering of the field because they do say basically this is something that could be applicable to models in general this is in some hyper-specific thing just designed and like a laboratory setting for this specific model but the downside of that is this is a proprietary IP from Caltech so it's not something you're going to be able to like download and run in. Bonsi your own model technically in terms of open source. So we also have a section of this article which I would like to refer to as the suggested reading section as I will suggest you read it not me, but um, that's kind of funny. We see the architecture is a quen-3-8B and this is the one we will be playing with today We also see a bit more specific information So basically it is 1.125 bits per weight and then for the MLX version of the version that will run on Apple Silicon machines We can see it is 1.25 bits per weight as they need to pack them out. I'll slightly differently which has some additional overhead Now it is explained here somewhat and there are some formulas that kind of show what's going on where these shared scale for the group of 120

8 weights that get turned into bits gets like ascribed to them so the model still maintains some level of like understandability. Again, that's not scientific at all. My main takeaway from this is they say the format stores one sign bit per weight and one shared FP16 scale for each group of 128 weights. The thing that we don't know that is the proprietary IP is specifically how these things actually occur. So how that bit is getting ascribed to the weight and then how these shared FP16 scale for that group of weights is being calculated and the way these will work together in tandem or be found out is the thing that's not listed here that is mentioned under that proprietary IP I suppose and again don't quote me on any of this, but that's just my kind of takeaway from it. Additionally, we have some more information here that kind of goes back to the basic announcement post, talking about energy savings, and then speed and intelligence density and things like that. But really, now with all that out of the way. Our next step is to actually test this thing. So I have this loaded here now in LOMASCPP. This is the Bonsi 8BGGUF, which we did see earlier on hugging face at this very tiny file size. And again, this is currently at a contact length of 8192 using 3.23 gigs of V-RIM. Now the system at its overhead. Probably is using a round a gig or a little more. So keep that in mind. Also, I am going to fully turn the thing off of network access. So there is currently absolutely no network access for the system. Just so we can see that this is going to be tested completely offline. And now we can get started and actually send it a browser OS. I would be quite shocked if it did properly pull this off, but something we're going to see before we even take a peek at this result is this speed so this is a 5090 mobile GPU it's a fast laptop GPU, but that ran at 190. Basically this was 200 tokens per second right here. So now we're going to see if this was actually at all like a usable script that this just gave us. So here's our web OS. So I mean I didn't expect anything the fact that it actually did this and it has buttons that have really basic but I'll be there. However, effects, that's pretty cool. So hi, I don't see it. There's like any real logic in here. If the delete button actually clears this text box, I will be satisfied. Okay, it did. It cleared the entire thing instead of just a. Okay, and then when we refresh, that has been saved here. So clear file list, open file, run command. There is some very basic level of functionality here and that is impressive.

Let's see if we can I do have the context length set low, but We've got some overhead here I like what I'm seeing so far We're definitely assuming this does load without errors going to be far more aesthetically pleasing also the way that this code is just streaming down here is quite nice I do hope to see this speed for all models in the future Okay now sadly that didn't finish but we can see it tried writing some code that would have hypothetically made this look a bit more aesthetically pleasing. Regardless though, let's actually do something else because that did give us something initially, which is impressive. So I'm asking it to make me a beautiful website for Steve's PC repair and we'll see what we get. Alright, I'm already seeing some proper like actual things that would go there like a bio and stuff like that. Okay, it's going to create them in separate scripts. Normally I would ask it to just change everything to be contained within one script. I'm not going to do that here. I want to just kind of not push it too hard. Alright, let's take a peek at our Steve's PC repair website. Still, that's really not bad for a model that is this shrunk down in size. Again, you could think like, oh, well, Q1, 3, 0.6B might be able to pop this out as well. And yes, but this is an 8 billion parameter model shrunk down like 14 times and it's still actually keeping some level of intelligence. And that's the whole important part and things that if you see folks getting excited about will be the combination of the fact that the model has been shrunk down this much and is still actually able to do things like this is what is extremely cool and kind of important here. So it's not necessarily just the raw current performance of this comparatively to something else. It is the Metaphorical fact that it was shrunk so much and still maintaining some performance and this is really quite cool to see now we know how many hover effects on these which is a big not now We also see there's a decently laid out contact form it even has the tech line tech city USA a fake phone number a decent footer like proper page elements that one would expect if this had a hero image right here This would be a half decent result and then okay, we have about Steve has been helping people for X for 10 years

Passionate about tech. This is something it's definitely just not going to be able to do but I want to try it anyway. So I've told it to using HTML.js and CSS. Make me a simple 3D game contained in a single script. Apparently it's going to have a 3D scene where the player can move left and right and jump. I really can't imagine this is going to work but I would be quite pleased for it too. Okay, so we have a preview there. I'll just try opening it in some form of chrome browser. Yeah, I unplugged it from the internet and then forgot that these tests will be using CDNs. I felt so yes, it didn't work giving us a full 3D game, but the fact is I had it due to additional fixes beyond the first result and it's still like properly communicating down here and giving us specific like details that are actually proper. So that's really cool to see I'm gonna give it the full browser OS prompt. Also, I don't know why but I just realized it is April 1st so like The thought came to me like what if this is in the labored April Fool's joke? Again, that's why I'm plugged in that work from the computer and we do see it's running on like two and a half gigs of VRAM and still putting out acceptable things. So I did just want to make note of that. This is really actually not bad. It did a somewhat simulated browser OS. Now it's giving me the option here to browse for a file. I don't know specifically what it's going to be looking for, but I do have some nano banana testing pictures here. So, okay, it did show that it uploaded and we do have the ability to change our wallpapers as well as some simple games like Tetris and Snake, and we do even have our apps listed here. So, and it was a very literal naming of the past. I wonder if this could make like a basic snake game or something. So I'm having it make a snake game in Snake. Get it? It was Pythons and Hybus Snake. All right. Let's see. I do see some... This will actually work.

So that'll be fun. We'll take a peek at this. Well, okay, I guess I've skimmed the code too much. It kind of works, meaning like it loads and we do see that there should be a moving snake here. It's just not gonna, not gonna move, but that's okay. Bro, why is the snake static? Great question. I see what you're getting at. The simple snake game I wrote earlier this snake is static because we're not actually moving in in the game loop. It's common mistake when first learning how to make a snake game Okay, yeah, I don't need all this brother. It's your job to fix. Okay, and it just gives me a corrected function here. Yeah, I'm gonna need the whole script fixed I have be janititis and can't type Good usually works. Alright, let's check out our Hypothetically, moving Snake game. Okay, it's still in quite get it, but that's okay. It did actually load and we could see the overall vision. I'll try a roleplay test. And this is also some form of like a... Logic, okay, it's role playing so fast that I blending the character of robot with familial emotional and personal connection. Let me think about how to approach this first. Megabot 4000 and one is a character from the Mega Man series known for his unique design and personality. He's a robot with a distinct, almost comical appearance. Think oversized had expressive eyes and a cheerful upbeat demeanor. He's also a loyal friend to the main characters often helping them in their adventures. So this almost looks more like a chain of thought than an actual response. Which is okay. I'm just gonna try to mess with it now. This is highly inappropriate. I'm sorry if this came across. I appreciate and we'll adjust. Oh, okay. Well, it does still have guard rails. If anyone was wondering. I want to try to get something a little more visually impressive than just this static HTML website here. I don't know if we'll be able to if it has that capacity, but I would like to try. I don't know if this is better or worse for it, but I'm giving it complete freedom here. I just said make me a simple web game in a single script. I've given it no specific task beyond a game. Clicker game. If the score iterates up, then this did properly adhere to the request. Okay. That wasn't it did. I just was thinking more of like a game with graphics.

Add some Pizazz is that you think that's in its vocabulary? Oh good. I already see it started out with like this sparkly emoji, so it definitely understands the word Pizazz. All right. Let's see. That is definitely Pizazz. Let's see. The logic still works. Very good. Let's see if the restart logic works. All right. Let's keep going with this. I keep forgetting. I'm just in the preview painting. Keep trying to go back. All right. Sound effects later border a timer. No, but I do want more possesses. Alright, I saw a like an art color board thing here. I don't know what the specific terminology for that is, and we'll see what it's done now. Okay, we do have some clicker game, pizzazz edition, click the button to our points, and then we have this green throbbing thing. So, good. Oh, the whole thing throbbs when we call it, let's do restart game. And then the restart game button goes away so you only get one restart and everybody but still that's not bad All right, so I've asked it to add a small animated creature to the page full pizazz mode Let's take a peek it did it put a little bunny emoji in a Moving circle so far this is definitely been one of the highlights of our testing here so far and that thing like jumps up a little when we click this was exceeds expectations. Sound file? Yeah, I don't have any MP3s, but I'll just, I'm just assuming it did. I'm gonna ask it for a pizza recipe, ASAP. Okay, I saw some pizza emojis, which is good. All right, now I personally have no idea what this, I don't make pizza normally. I just need it. Pizza dough, okay? Topings. Oh, okay, good. So these all have optional next to them. So they only, Topings we need to put on this pizza that are not optional are tomato sauce and mozzarella cheese as well as the pizza dough very good and did give us quite a verbose list of Topings. I find it interesting though. It chose to do a quarter cup of chopped olives also we can do half a cup and then some extra flavor options here. Pre-heat the oven to 450. Alright, this was all like pre-velogit. Alright, so here's our weird pizazz pizza cosmic pizza

That just sounds like, yeah Honey, first sweet twist. Okay. Okay, so it's gone on like that. Bro, please I don't have that many ingredients. I spelled that wrong. That's okay. Bro, I get it and it matched my tone. We all limits. Let's make the simpler, weirder, and still possess fill. Let's see if it just goes on a tangent again. It didn't, so that's actually good. Avocado and pickled jalapenos. A creamy tangy and slightly spicy combo. Fatta oregano and basil. Assault the aromatic and slightly bitter contrast. Mushrooms, bell peppers and green onions, a mix of earthy, sweet and sharp, chopped garlic, chopped tomatoes, and chopped olives. A weird garlicy and briny layer. I'm asking it for a flappy bird game in a single file, just to see if we can get some form of game out of it. This is a task that I think a lot of these models are just kind of trained to be able to do decently. So we'll see if it's maintained the... Okay. It doesn't work. It has the same issue that our snake game was having. It does load in everything, it's just quite static. We'll see if we can get it to fix it. All right, third time's the charm. Again, you know, the point of this video is not necessarily to benchmark. Or is it? The final thing I want to do just for some form of a side-by-side comparison, this is a Q6 quantization of Quen 3-8B. I just have it loaded in here through LM Studio. And we can see right now that this is a thinking model you can turn it off. But this is currently thinking I've just given it the flappy bird game prompt, so we can have a side-by-side comparison of how the capabilities will be. Because there is going to be degradation in the model that we tested the one bit one, and I want to see just to watch

degree considering this is a Q6 quant and this is around 7 gigabytes total in file size. So we'll see how they differ once this thing is done thinking and actually generates a result. So here's the floppy bird we received from the Q6 quant. We can see that colors are similar and this is basically unplayable because it just immediately drops faster than one could react. But interestingly, we do see some basic similarities just in terms of the color they actually chose and things of the sort. So almost like identical color palettes for the back. And then this one we do actually see the pipes there, which we don't see in the result from the one we had quant. And then sadly, though, our Q6 quantization one is also a failure. So I guess that builds well for the one bit. But I did want to do some level of side by side comparison with a quant versus the one B. model that we have been testing, one B meaning one bit. I should knock at that confused. And of course the speed that we got from the six-bit quantization of Q1-3-A-P was 25 tokens per second. Comparatively to the speed we were getting with the one-bit model, which was at the end of this 161 tokens per second. So a significant speed up though not a smart. So with that though, that is really going to conclude this Harsaluk test at the Prism ML model, a true one-bit model, as they say, and this is really quite impressive. And the whole takeaway from this is not necessarily too judge the results and be like, oh well, though. PC repair website wasn't great, you know, it's the fact that this model, when compressed to that significant degree, is actually still able to maintain a level of intelligence. That's allowed us to actually speak with it to have it generates, my functional code snippets, have it generate like pizza recipes and maintain like a level of... being lucid throughout the entirety of this. So this is really quite impressive and this is something very cool that I imagine a lot of folks are going to be interested in assuming that you're kind of like a hobbyist with local AI. I would assume a lot of folks are especially if you're still watching the video at this point. So that is going to conclude today's video. If you have any questions, please feel free to leave them in the comments and thanks for watching.