Clúster de dos Ryzen AI Halo: així executa un model de 397B en local

You might have seen this device before. This is the AMD Ryzen AI Halo. Shares the name with a stricks Halo chip that's inside there. It's an APU, so it's got the CPU, the GPU with 128 GB of memory that can be shared between the two to do AI work. And in my previous video, I already covered the details about this, and I ran a bunch of benchmarks comparing it [music] to the Nvidia DJX Spark, as well as the previous machines that had this chip in it. But this one is the smallest. And according to AMD, on a tiny box like this that you can fit into your pocket, well, maybe you can run models that are up to 200 billion parameters in size. In the last year that this chip has been out, AMD has done a lot of work to not only get support going for it, but they also came out with these playbooks that you can easily follow along to generate images, videos, text, and use them as coding assistance, something you'd all be interested in probably. And in this advanced section, you can now cluster two of them. So you'll be able to potentially run models that are about 400 billion parameters.

Now two computers don't automatically become one larger computer. The model must be split somehow. The GPUs that are processing that data must continually exchange data while generating. And on this machine, that communication happens over Ethernet cuz this 10 GB port is the only thing we have to connect them to. So the question is, can these two actually function as one practical AI system or will the network between them ruin the entire idea? According to AMD's playbooks here, you can use two different methods to cluster them together. One is using llama CPP with RPC and that's basically remote procedure call and llama CPP is a very popular tool. O Lama, if you've heard of that, LM Studio all use Llama CPP under the hood and you can run Llama CPP directly which is what this will do. Well, it's got RPC built into it. We'll go into that. And Rickle, RCCL. It's pronounced Rickle. I don't know. It's kind of like Nvidia has nickel NCCL. So, this is Rickle. Rickle uses VLM and Ray. Another two key pieces of technology with tensor parallelism. Basically, it's a way to split up the model. I've discussed this in many videos in the past. I won't go into the details of that, but this will use that technology. So, there's two different ways of doing it. And both methods combine the 128 GB here and 128 GB here to give you 256 GB of available memory on the two machines. It's actually a little bit less than that because you still need a little bit of memory for the operating system. But both of these methods [music] call for proper 10 GB Ethernet network. However, you can't just take a cable [music] and plug them in together like that and have that work. That's just not going to do it. even though each one of these has a 10 gig port. AMD's instructions for both of these require a 10 GB Ethernet switch to be used between the machines. So yeah, that's not going to work. So before I can build this cluster, I need to sort out that network switch between them. These Ryzen Halo machines are now available at the MicroEnter, which is their exclusive retailer. [music] >> This is Dan, by the way. You've met Dan before in previous videos. >> Nice to meet you guys. Say something about his hair. >> Yeah, please do. I changed it up this time. >> Dan, take us to the switches if you don't mind. >> Yeah, I don't mind at all. Probably going to be in a little bit of a fancier spot. In theory, this is the switch area. In practice, like I said, I think the switch you want is probably going to be hidden somewhere. I can make it appear. >> Let's make it appear. >> Movie magic. >> I know where it is. >> Oh, you do? >> Yeah. >> What? When are you submitting your application? >> Dude, it's in the Unifi section. >> Yeah, but I mean, I figured, but I thought it was going to be like one of those boxes that we hide up in another aisle. But we'll find it. Don't worry. >> We will find it. Movie magic is happening. I promise. >> In the back of the scenes. >> Oh, was it just out there? Oh, >> okay. Thank you. They put it in a different box now. >> But it works. That's what matters, right? >> Do you want to like hold it? >> See, as a size comparison, so they can see that I'm actually bigger than the box. >> You are bigger than the box. You couldn't fit in that box. >> I could not. >>

This is that Flex XG switch that I picked up for MicroEnter, but actually it's just an empty box. I picked that up uh last year and I already had it. It's right here. Thing is, back then I didn't understand why it had four 10 GB ports and one 1 GB port. Well, now I do. It's actually perfect for clustering because you would plug in your cluster up to four machines. In this case, we're doing just two into the 10 GB ports and the 1 GB port that's just going out to your network. So, you can manage your cluster. Plus, it's completely silent because there's no active cooling here. It's all a giant heat sink. It's pretty heavy, too.

That would take care of the network, but there's still one little issue. I can't cluster these machines the way they are now. They don't match. The Halo I tested in the previous video came with Windows, and the second one had Linux on it. AMD's documentation for beginner and intermediate. All these different playbooks you can do on Windows or Linux, but the clustering playbook is only on Linux. So, I need both machines set up the same way. Instead of configuring the second machine from scratch and hoping I reproduced the environment exactly, I just simply cloned the Linux installation onto the Windows machine's SSD. That gave me two identical starting points with the same drivers, the same AMD software, and the same exact configuration. Of course, I knew we needed to download models onto it. So before I cloned it, I made sure that one of the machines had models on it, so I don't need to download the models twice. Also, cloning the drives also clone the machine's identity. So before I connected them up, I had to make sure that the host names were different. One became Halo 1 and one became Halo 2. Turn that on. Yes, it's Ubuntu. That used to be Windows. And we're in, baby. Nice. Now I have two matching Linux machines. And yeah, I'm not going to put them on top of each other because the vents are actually on top and on the sides. So these kind of have to be spread apart a little bit.

Just a quick little test. I got IPF 3 on server 2, Halo 2, and Iperf 3 on Halo 1 just to test the network bandwidth between the two. And it looks like it's coming back pretty much to its maximum capacity. 9.41, 9.44. Yeah, that's as much as we're going to get for a 10 gig network. Network is ready. software is ready. Now it's time to try this playbook.

So, first we're going to try RPC, which is under advanced. And right here, clustering two Ryzen AI halos with RPC. I already went through some of this installation here. I did this step, this step, this step, and this is interesting. Right here, very, very interesting. AMD TTM. This lets you find out how much shared memory there is. And I got bitten by this on one of the machines. I'll tell you why. If we run this, you'll see that I have a total system memory of 125. It's 128, but you know, there's some for the system. And total current TTM pages limit is 120 GB. This is the reason why we're using Linux instead of Windows. Windows limits this to 96 as maximum. Linux can go up to 120, but not by default. You have to run this command right here. as part of the playbook in order to give it or allocate 120 gigabytes, which means the GPU can use up to that much, allowing you to fit models theoretically up to 240 GB in size. Now, the issue that I ran into was really funny because, well, it wasn't that funny. [laughter] I didn't think so at the time cuz it took me a little while to figure it out. The Windows version automatically sets it to 64 GB. And this is not something that you can discover in the BIOS. In fact, I went into the BIOS and try to find that setting like some other Ryzen machines allow you to do, but it wasn't there. It's not exposed in the BIOS. You can only modify that setting using the AMD Ryzen AI developer center. Even after I put Linux on that box, I couldn't change it in the BIOS. I couldn't change it using this AMD TTM set 120. In fact, I ran that and it did not work. The system was limited to 64 and Llama CPP RPC was detecting only 64. But I'm getting ahead of myself. So, in order to fix that, you need to go into the settings tab of the AMD Ryzen AI developer center, which comes with the Linux installation here. And that software allows you to tweak the GPU memory allocation. And as far as I know, that's the only way to do it. Now, you also need to determine the name of the interface, unlike the DJX Spark, which has a lot of interfaces. This one only has one. There it is right there. And most likely, yours will also have that same name. Having the interface name will allow you to confirm the speed that's available to you. So that's 10 GB right there. So we're good with the connections.

Now, this gives you two installation options. Lemonade SDK or manual source build. I actually like doing the manual source build. It's not that hard. You just go to Llama CPP's GitHub and it's good to poke around in here. What's good about this project is that it's so popular that it kind of had to be really well documented. And if you're doing any sort of AI work on your machines, you should be familiar with these steps. So just go on the GitHubs and use the quick start there and do the build. Do it right now. [laughter] And you can just keep going through the playbook. I don't want to bore you with all the steps to go through this because it's pretty much verbatim. You just go through these steps one by one. I did that and I want to share the results.

One more thing to notice here is the playbook uses GLM 4.7. See, I told you it was verbatim. I'm reading it right here. It's a 358 billion parameter model in the Q4 KXL quantization from Onslaught. That's actually not what it is because here they tell you to include the UDQ4 KXL and that's what I use, the UD version, but the Q4 KXL and Q4 KM will also work. The KM is what I usually use. So I was wondering why do they have UDQ4? Well, this model is unsllo GGUF guff. And the difference with UD that just means unsloth dynamic. So if I back up here under GLM 4.7 GGUF, you have all these different quantizations that Unsloth was nice enough to quantize for you. So all you have to do is just download them. Instead of Q4KM, Q4KS, we're getting this UG Q4K XL. Unlike the other ones where it's uniform 4bit quantization like the classic Q4KM, this one spends more bits on the tensors that matter. attention the router embedded and fewer bits on the huge rarely hit mixture of experts layers. And that's actually a good thing for a 358 billion mixture of experts model like this one. It's supposed to be better and have better payoff for the quality per bite. Yet, it's exactly the same size. I've talked about this on the channel before and Barttowski is another quantizer pretty famous on the internets. They all use like little techniques to tweak the quantizations to get more juice out of it. It's not just about running a bigger model. It's also running a higher quality quantization of a big model. But neither one of these quants will fit on a single 128 GB box. The second Halo is what lets you serve a good quant instead of dropping down, I don't know, one of these Q IQ2M or IQ1S quants, which are probably going to mess up your output. I made a whole separate video on the ladders going from different quantizations and what kind of output they produce. I'll link to that down below.

I'm running it now. And with either one of these approaches, it takes a little while for the system to start up. But it's a lot easier with the RPC server the way we're doing it right now. This is basically llama CPP borrowing the second Halo's memory. And it's a much simpler approach. Halo 2 simply exposes his GPU as an RPC worker. And Halo 1 is like the root node that points to it and uses it. And it's up. It's doing the prefill. And there it goes. It's generating the text. We're getting about uh 8.2 tokens per second here. Remember folks, this is GLM 4.7. It's pretty big. I'm going to run SMI here. And wow, interesting. Okay, so VRAMm is at 35% and the power is only at about 43 watts. GPU is only about halfway utilized. This is only on one machine, by the way. So, you'd imagine it's the same thing is happening on the second machine. Oh, now it's going. Okay, so the GPU is back up to 100%. That's good. It's kind of bouncing around probably between the decode stage and the uh processing stage. All right, it's almost done. By the way, I I I got to take a moment to say this. Uh you can't complain in the comments below that I don't ever do big models. Hey, big model. Another big model coming up shortly here. An even bigger model. Oh, here we go. 7.93 7.69 7.6 within that range for the concurrency of one. By the way, this is like basically chatting with this thing. And that's how many tokens per second we're getting out of this. Now, if we take a look at concurrency 4, which I also ran, we're almost doubling. That's not bad, actually. But it only goes up to a certain point with Llama CPP. With concurrency of four for TG128, we're getting 13. And for TG 256, we're getting 13.5. But then if you raise your prompt up a little bit, a little higher to 2048. Our concurrency of four falls back to 5.5 tokens per second. This is where this workflow breaks down. And with Llama CPP, you need a slightly different approach. I made a whole different video about that. How to handle Llama CPP with lots of concurrencies. Why do you need that? Well, if you're doing things like multiple requests at the same time with that kind of setup, if you're in a workspace or with multiple people using it or if you have an agent and if you're designing some kind of multi- aent architecture, then that matters and I'll link to that video down below. But here we have a different approach to handling concurrency and that's where the other workflow comes in. And that one's a little bit more difficult and you say, well, why do I want a more difficult one, right? when this works perfectly fine. And if it works for your use case, go ahead and use it. RPC still lets you fit a giant model and run one conversation at a time, maybe two. But VLM with Rickle, it's a different story. [music]

Your Ryzen AI Halo is already capable of running large language models locally, but clustering takes it a further by combining GPU memory. You already know all this from watching the rest of my video, but in case you skipped around, this is uh this is a little refresher. But this playbook teaches you how to cluster two of them using Rickle Rock Communication Collectives Library. It's it's an acronym within an acronym with VLM. And we're going to run Quen 3.5 397B, which is a much bigger model than the previous one. Most of this documentation is the same as the one we just went through. You're getting the network link speeds, which we're fine with. You're installing a bunch of dependencies, setting your memory to the highest possible. Here's where things get different. VLM containers and this is going to use a Podman container and tells you exactly what to run to launch it. Now, this is going to take a while. In my case, for this large model, it took probably 15 minutes to actually launch it. It uses another technology called Ray, which is responsible for doing the multi-node communication. Yeah, there's a lot of stuff going on here. Let's see the results. Oh, on machine one, you start Ray. On machine two, you join the cluster. And then you serve the model using this right here. If you get one flag wrong, you have to start over and wait another 15 minutes. Don't ask me how I know. And this is why I use smaller models to try it with. Don't start with the large models first. Use a model with similar architecture. In this case, it'll be mixture of experts quen 3.5, but a smaller one. All right. So, here we go. Llama beni. I'm going to point it to this machine. I'm going to run a couple of PPS prompt processing 512 and 2048 and a couple of TGS token generation 128 and 256 for concurrency 1 and concurrency 4.

Here we go. This is the big one. Quen 3.5 397B. And it's going. I can actually hear both machines. It's not breaking. It's going. You can also run rockom SMI on each one of these machines. Here it is showing that 36% of VRAM is being used and 100% of the GPU. This also monitors the temperature for you and the power output. So, we're using 60 W, 52° C. It's a full amount of GPU usage, but not a crazy amount of VRAM usage. This is actually tensor parallelism at work here. TP equals 2. And we're using about 110 GB per box here for a total of 220 pulled. Can you hear the machines? [laughter] I can hear it. Rickle talking over the 10 GB link at about 7.81 tokens per second and up to almost 18 tokens per second for concurrency of four. That's what I'm talking about.

So, if you remember group three from my previous video, the one that I kind of set up, well, this is for those of you that are in group three. If you're going to be setting this up to do work, you have a couple of different approaches. And if you're going to be clustering multiple machines, then AMD gives you a couple of different options to do that. Lomar RPC is simple. Rickle is a little bit more complicated, but it scales a little bit better and it's like running a little mini data center at home or in your office. This is the technology stack that you'll use if you transition into bigger AMD infrastructure. Now, AMD also alludes to having four node cluster as well. And if you want to see my attempts at another machines in the foreign node cluster with this very chip, watch this video here. Thanks for watching and I'll see you next time. [music]