AMD MI50 de 32 GB per a IA local: molta VRAM, però compatibilitat comunitària

If you saw the dual R9 700 video back in December, you might remember I mentioned picking up a refurbished AMD MI50 from eBay. I paid around 370 quid for it. And the main reason I got it was simple. It has 32 GB of VRAM. Now, normally getting that much memory for local inference means spending a lot more, especially in 2026. So, I wanted to see if we could actually build a working AI setup out of cheaper refurbished data center hardware. So, in this video, we'll see how to run LLMs with llama.cpp and vLLM on the MI50 using recent model families like Qwen 3.6 and Gemma 4. And we'll compare the performance with a card with a modern architecture, that is the AMD R9 700 AI Pro. So, you can get an idea of the tradeoffs. First, let's take a look at what this card actually is. The MI50 is built on the Vega 20 architecture from 2018, also called GFX 906. If you compare it to the modern R9 700 I tested recently, both give you 32 GB of VRAM. The MI50 has 60 compute unit compared to the 64 on the R9 700. A compute unit is just a cluster of streaming processors, which are AMD's equivalent to CUDA cores. Basically, they are the cores that do the actual computations. So, the MI50 has 3,840 of these streaming processors in total, while the R9 700 has 4,096. Another important difference is that the

newer R9 700 is based on the RDNA 4 architecture, and it includes hardware support for modern data types like BF16 and FP8. BF16 is a 16-bit format optimized for machine learning, and it is the native format most models are trained on today. FP8 is an 8-bit format used to quantize model weights while preserving good quality. Having native hardware support means that the GPU can load these formats directly into its registers. The calculations are fast, and the card does not have to waste time and memory unpacking numbers into different data formats, losing precision, etc. But, and that's the caveat to be aware of with older cards, the MI50 doesn't support any of these formats. However, what the MI50 does have is HBM2, which stands for high bandwidth memory. Typically, consumer GPUs have GDDR memory chips that are spread out around the board. HBM2, instead, stacks the memory directly next to the GPU core. Because of how it's wired, it gives this card memory bandwidth of 1 TB per second. For running LLMs, which are heavily memory bound, having 1 TB of bandwidth on a cheap 2018 card is what makes it interesting even in 2026. For reference, the newer R9 700 has 640 GB of memory bandwidth, still considerable, but nearly not as much as

the MI50. Another difference between the Vega 20 and the RDNA 4 architecture that will matter for LLMs is the presence of MFMA units, which stands for matrix fused multiply add. These allow to perform math operations needed for flash attention in a hardware native efficient way. Attention is the most expensive operation in transformers, the underlying architecture of modern LLMs. And flash attention is the algorithm that speeds up attention computation and reduces memory usage for long context operations. Because the MI50 lacks these units, the software stack cannot implement flash attention in the optimized hardware native way, and we'll see how this impacts performance later on. But, throwing a data center card into a regular desktop case comes with problems. The main one is cooling. These cards are entirely passive. They don't have fans because they expect the kind of airflow you find in a data center chassis. To cool it down, you need to attach a fan so that you can simulate the airflow that the card would experience in a data center. The card I bought actually came with an attachment for a fan, which is uh common with all of these eBay listings. But, if yours doesn't, there are plenty of designs online, so you can just 3D print your own. For example, I printed this version, which allows to use a larger fan and has a different design. Just make sure you do not print in PLA, or it will likely melt. The other point

to be aware of is that this setup is noticeably louder than the R9 700 that comes with its built-in and optimized cooling solution. I also flashed the card with the V420 BIOS. The default BIOS runs at 225 W, but flashing the V420 version drops that power limit to 178 W while also enabling resizable bar and fixing some PCI-E Gen4 weirdness. People report it makes the card more stable and efficient for a home lab environment. But, I leave the links in the description if you want to flash another BIOS and see if you can get more performance out of this. The main issue with hardware this old is the software stack. The card is based on the GFX 906 architecture, which AMD has officially deprecated in its latest ROCm releases. If you just install the stock drivers, it likely won't work out of the box. To make this work, I am relying heavily on a GitHub repository from user named mixa3607. They actively maintain Docker images with all of the patched ROCm libraries needed to keep GFX 906 alive. They also have an active Discord server, so if you already own an MI50 or thinking about picking one up, I highly recommend joining that server. That's where all of the experience and discussion for this specific hardware lives. All right. So, to get started, as usual, head to my GitHub repository. In this case, it's the MI50 GFX 906 toolboxes. As I said, these are heavily, heavily based on the containers maintained by

this project, and they also have a Discord channel, so make sure to check that out. But, as usual, I prefer toolboxes. They're just more approachable, and I add some tools that make it easier to get started. So, if you want to use the same system that I have here, I have Fedora, if I can type. It's not making it easy. I have Fedora 43 on a very recent kernel. You could have Ubuntu and other things. If you are using Ubuntu and you want to use my toolboxes, typically, you want to use distrobox instead of toolbox. This is just because of incompatibilities. But, if you want to follow my setup, uh this is the same machine that I used for the R9 700, hence why the hostname is R9 700. Now, one thing I do, which I document here, is disable SE Linux, cuz it might create some problems when you are uh trying to access memory and share um the GPUs with the containers. So, I recommend to just uh disable that for this. Now, to get started, we are going to clone that MI50 repository, and in here, you can see there are different folders. I'll just make this a little bit bigger. And we want to go into the llama.cpp folder. As I said many times on this channel, llama.cpp has got the best ecosystem to get started with LLMs. They support an incredible number of models, an incredible number of different quantization methods, and if llama.cpp supports a model, it is supported across

all of the different GPUs and different type of hardware. If you look at vLLM, and we'll look at that later, it is much more complex and a lot of things will break. So, llama.cpp is the best starting point and here I give you a script which you can call called refresh llama.cpp toolbox. Now, for this I have built one on ROCm 6.4.4 and one on 7.2.1. There is a 7.2.2 Actually, I might rebuild it, but there will not be a big difference here. So, let's pick the most recent one available. This will download the image. It was very quick for me because I already had that image locally and now I can do toolbox enter. And the name of the toolbox is llama GFX906 and the version of ROCm. I remind you, if you are on Ubuntu, you will want to use this tool box instead of tool box. Now, once you're inside you want to immediately do export HIP visible devices and set it to zero. Cuz in my case, I have multiple GPUs here and one of the first steps you want to do is just expose the GPU or the GPUs that you want llama.cpp to see, otherwise it will become messy. Actually, let me show you if we run AMD GPU top, you can see that the first GPU at index zero is the GFX906 Vega 20 architecture, but I also have here the

AI Pro. And obviously, I don't want them to be mixed up. So, once I've set HIP visible devices to the first one, I can run llama CLI list devices and this correctly identifies our GPU with 32 GB of VRAM. Now, you are going to need to download some models and I've given you a very easy get models script that you can call and these are some of the modern LLMs that I tend to use. The one that I find today's most useful for general purpose things is Qwen Qwen 3.6 and probably by the time you download these, I will have already downloaded these weights over here. But basically, you can choose these and you can choose the quantization. Obviously, you need to choose something that fits in the amount of memory you've got. Now, here I only have one of these, so I will choose a model that fits in there. And I have these models in my home directory which lives outside of the toolbox under models and these are all the models that I've got downloaded. What we are going to do and as you can see here, I already got Qwen 3.6. Let's try to run that particular that particular model. The 35B A3B mixture of expert model is one of the best today and you are going to be able to run a Q4 quant. This is a very good Unsloth dynamic quant. Again, if you've been on the channel, you've seen this many times.

And then we can just do llama server. We enable flash attention although in this particular case, as we'll see, it doesn't really make a difference because as we said, there is no real way of implementing flash attention efficiently on this, so you can skip that. It won't do anything, I guess. Then we pass the model weights that we downloaded with the script or you can download from Hugging Face in GGUF format. You can pass a context window for example, I don't know, 60,000 tokens and that now goes ahead and allocates memory and as you can see, we see the memory being allocated and now llama server is up and running on port 8080. And obviously, if you're familiar with llama server, you can now connect anything, a chatbot, you can connect a coding agent or literally anything that can use an OpenAI compatible API. In this particular case, what I am going to do, I'm going to SSH into this. You can see it's port 8080, so I'm just going to SSH by forwarding port 8080 to localhost 8080 so that I can access that in the browser. That's how I typically deal with all of this. And I will still have this up and running and then we can go in the browser. And there it is. That's the llama.cpp web UI. You can see the model has been loaded here and you can obviously query the model

and it's very, very responsive. So, let's take a quick look at some benchmarks on recent models. As you can see here, I have taken the Qwen family, the Gemma models and GPT-OSS, Mistral and Nemotron and I've done some comparisons. So, there are two types of benchmarks that matter here, the prompt processing and the token generation. Let's start with the prompt processing which, I remind you, if you're not familiar with it, it's the time it takes in terms of tokens per second for the GPU to process the context or the prompt that you send to the LLM. And obviously, the longer the context, the longer this is going to take because of the attention mechanism and everything else which is involved in the transformer. But this is an important metric, especially with things like coding agents that have to consume a lot of context. Now, as you can see here, I have put the MI50 next to the R9 700 cuz I want to give you a real feeling, a real comparison of what an older architecture can do and is capable of doing versus a newer architecture. Now, what's immediately clear is that newer architecture, the RDNA 4, because of the built-in optimizations that we spoke about before, can do attention much faster in hardware than the MI50 can. So, you can see the difference here is almost on some of these models 2,000%. So, here we have, for example, with Mistral 14B in BF16 format, the MI50 can process 128 tokens per second in your prompt and the R9 700 can

process 2,652. So, there is a considerable difference and depending on the model architecture, this can go up and down, but you can see that even in the worst case, the R9 700 is considerably better. For example, on Qwen 3.6 Q4 quantization, 35 billion, this is a mixture of expert model. So, I remind you, mixture of expert models are the models where you have a certain number of parameters, in that case, 35 billion, but at every time, for every token, only a subset is activated. I think in this case, 3 billion parameters and that's obviously much faster to compute. And now you can see that difference is striking. And again, this is because of the native support for some of the operations needed by flash attention that the R9 700 has. So, even with 1 TB of memory bandwidth compared to the R9 700 that we said only has 640, the R9 700 still does much better. Now, typically, you want to look at long context, but even on long context, you can see that the difference is not as striking because in this case, the memory bandwidth of the MI50 does help it a little bit more. So, the R9 700 is still considerably better, but you can see that that gap reduces a little bit as the context becomes longer. And again, this is just because of the memory bandwidth. Now, let's look at token generation. So, this is after the prompt has been processed, how long it takes for each token to be generated or how many tokens per second can it generate in its response. Again, the R9 700 performs

better than the MI50, but it is not that dramatic. And here, the memory bandwidth of the MI50 really does help it compete. As you go to longer context, you can see that the difference is not incredible and actually for a model like llama 2 7 billion parameters we can see that actually the r9 700 on long context doesn't do as well as the MI 50 but obviously in all other cases the r9 700 is still better but not by a dramatic number. So I'm quickly jumping in as an editing cuz I thought it'd be interesting to also compare the MI 50 with these tricks Halo APU and as you can see here in prompt processing on short context tricks Halo does really well compared to the MI 50 and again I think this is due to the hardware support for Matrix cores wmma for the rdna architecture in tricks Halo which is simply not available in the MI 50 so you can see at least on short context this is a better performance as the context becomes longer prompt processing is still better in tricks Halo but you can see that the gap closes. So jumping to text generation you can see that the MI 50 is generally more performant than tricks Halo and I guess this is because the memory bandwidth is so much higher I think four to five times more than tricks Halo has so this is one case where the MI 50 at text generation will

perform better than tricks Halo. Sometimes at long context tricks Halo manages to perform with some architectures quite close to the MI 50 but again the MI 50 will be better at token generation. All right now let's take a look at the vllm so what we want to do we want to move into the vllm folder and as usual I have here a refresh vllm toolbox script it will download the container and it will instantiate the toolbox and then you can enter that toolbox just like this. Again I have only based these on rock and 7.2.1. Now this welcomes you with a little welcome screen and I just realized I haven't updated this so it says MI 25 this is MI 50 and I give you this little start vllm script that you might have seen for in some of the other videos and you can pick one of these models for example quant 3.6 that we discussed before. Now this is in 4-bit quantization awq which is a very common quantization format for vllm I think it stands for activation aware quantization you can launch it you can see all of the CLI and here we're using Triton for attention I I've explained other times what Triton is but this is very much used in vllm Triton is a way to write kernels so programs for

any GPU in this kind of like python language which then can get compiled even on the fly to run on a specific type of GPU architecture so it's a very simple way and versatile way to try and make your GPU programs compatible with as many GPUs as possible. Now this will take some time to load and we can see already here that the memory has been allocated and actually as this is starting I will show you some benchmarks of the vllm throughput and again I have compared on these models the MI 50 and the r9 700 as expected the r9 700 beats the MI 50 everywhere but because vllm optimizes parallel computation and concurrent requests and it uses a lot of architectural tricks to make it more efficient to serve many requests at the same time what we tend to measure is throughput so we send a lot of concurrent requests in this case I think I did 64 and we see how many tokens it can generate. Now the MI 50 does actually pretty well in llama.cpp in spite of again not being able to use those native operations for attention so on some very good models like quant 3.6 35b in awq quantization it can it has a throughput of 119 tokens per second which is a lot of

tokens being generated if you look at these you can see that the number of tokens on llama.cpp is less but again this is one request at a time whereas here we have we're stretching the GPUs to generate many tokens but again as usual r9 700 does better here the r9 700 also has access to some rock and attention flash attention kernels so it will perform considerably better for many reasons but the MI 50 defends itself as as you can see. Donato from the future again here we have the comparison with tricks Halo as well and as you can see the MI 50 because of the memory bandwidth ultimately is better than tricks Halo but not by a lot even here the rdna 3.5 architecture and the native operations like wmma or the native Matrix cores do give tricks Halo quite a big advantage in spite of a much much smaller memory bandwidth. Now it is taking some time to load this is something you really need to be aware of it does take time and it needs to compile some kernels especially the first time you're running these and I typically clear the cache because I'm running so many different GPUs and architectures I don't want to mess things up but this will load at some point and it will start an API server which again then you can connect to a coding agent or anything else that you want to use for inference. So you can see here we are computing the CUDA graphs and it should come up

momentarily hopefully. There it is this is up and running on port 8000 so as usual if you're not running this toolbox in a graphical environment or locally on your system but as I'm doing on a remote server you are going to need to SSH into that server and forward that port locally so that you can then connect your coding agent or whatever you want to vllm. So that's all for now I did not include image and video generation in this video to keep it focused on LLMs but if you want to see how this card handles those workflows let me know by leaving a comment down below. If there is enough interest I will definitely do a dedicated video on comfy UI later on. As for the upcoming videos in this series I will be collaborating with the folks at bargain hardware who kindly made a server with four AMD MI 25 cards available to me for experimentation and they also give me access to one server with four Nvidia P100s. I'm setting those up in the next few episodes. If there is any other specific refurbished data center card you want me to test just let me know in the comments. Before I go just a quick reminder that this channel is a hobby project. If you find any of these useful you can support me on buy me a coffee or just by liking and commenting on this video to appease the YouTube algorithm. Thank you for watching and I'll see you in the next one.