Fable 5 accelera llama.cpp un 64,5%? Com funciona la prova

I used Fable to improve Lama.CPP and it did. When Fable came out, people built a lot of eye catching stuff, cool marbles, but almost nobody tried the one thing Fable is actually best at. Improving the technology we already have. Taking what already exists and making it better, more robust. So I thought, why not point Fable at local AI itself? I let Fable lose on the Lama.CPP codebase. Freehand do any analysis, find any optimization that makes local AI faster. And it found real once. It made Lama.CPP almost 65% faster. Four optimizations in total. Some worked, some didn't. We're going to dissect all four, what Fable tried, what worked, and what it actually did. Now that Fable is safe to use, I thought, Instead of making flashy experiments and flashy demos, why not use Fable to improve the existing tools we already have for local AI?

And the most popular and crucial one, the tool that runs on most consumer grade hardware, is none other than Lama.CPP. And the changes here compound. If I can get even a 1% improvement, it reflects on every model Lama.CPP can run, and all the future optimizations other people ship will stack right on top of it. But there's a catch. The Lama.cpp repo doesn't accept pull request that are fully or predominantly AI generated. There words, fully AI generated PRs provide no value, maintainers have AI tools too. And it's a fair call, someone has to understand and maintain every line. But it does set the bar high. One of the most tuned pieces of software on Earth is basically saying an AI has nothing to add here. It's a decision I respect, and they say it right there, private folks are exempt. So that's what I did. I forked it to see if that's actually true.

If any changefable makes can really improve the engine. I didn't want to give Fable any direction, any instruction. I didn't want my own biases influencing its decisions or its thought process. So I just asked it to pull the Lama.cpp repo, read everything thoroughly, and find the places where we could improve or optimize for mid range local LLMs. For the setup, I'm running a 12GB 3060 and 32GB of DDR4. And yeah, that's the big brother of my old 6GB 1060 setup. So I let Fable lose on the Lama.cbB code base. And here's the interesting part. It asked if it could SSH into my rig. I gave it permission and it SSH into my AI rig. Found all the model folders, the TurboQuand setup I was running, and my Lama.cbB preset config. It saw I lean heavily on an MOE model. So it went looking for a way to optimize the MOE side of the engine. The model's 35 billion parameter MOE only about 3 billion active per token.

It doesn't fit in 12GB, so the experts for most of the layers live in system RAM, the exact big model's small GPU setup a lot of Urun. And one thing about how it worked start to finish. For every change, it benchmarked before and after, compared the output with and without, and confirmed it worked with no regression, all on its own. Now, while it was exploring and giving me suggestions, some of its explanations gave me doubts, so I asked a bunch of clarifying questions, and I found out I had something wrong. I used to think that when Lama.CPP offloads the MOE layers, the experts sitting in system RAM get executed by the CPU, and the ones in VRAM get executed by the GPU. So you never move weights back and forth across PCIe. But surprisingly that's not the case, that's only half the story. It's true in D-code when you're generating one token at a time that's exactly how Lama.cp does it. The CPU runs the experts right there in RAM. But pre-fill is different. In pre-fill, you're processing thousands of tokens at once.

Every token pulls 8 experts from a layer. of tokens only 128 or 256 experts, so almost every expert gets selected. The CPU would have to run the entire layer, every single expert, which is brutally compute heavy for a CPU. So what Lama.CPP does is past a certain batch threshold, instead of running it on the CPU from RAM, it pulls those weights back into VRAM and lets the GPU execute the experts in parallel and that's where the bottleneck is. The weird part, the GPU runs the whole attention and the router first. Then the system takes the expert ideas it needs to pull from RAM to VRAM and while that copy happens the GPU just sits idle. Everything streams from RAM to VRAM over PCIe and only then does the GPU execute the experts. When you actually profile a pre-filled pass, the GPU spends more time receiving weights than computing with them. About 4.4 seconds copying versus 2.5 seconds of real math. The villain isn't the GPU. It's the bus. It's PCIe. Before we go any further, let me set the baseline. On that 35B MOE at UBADS 2048, I'm getting about 1143 tokens per second on pre-fill, prompt processing.

That's the starting line for everything Fable does next. So after Fable explored the code base, it found a dead section of code, a piece that lets you pin the page memory. And nothing was calling it. Someone had already written the feature. There's an environment variable to turn it on and off, but the place where it's supposed to be called just doesn't call it. Maybe that was intentional, but if it was, I think they should have removed the dead code. To explain what that code does, I need to give you some background. In a lot of my previous videos, I've mentioned the no-MAP flag. Normally, Lama.cpp uses MAP, page memory that the OS can evict. No MAP instead loads the model into a private memory section that won't get evicted. But there's a drawback to that. With MM model loading is basically instant. It feels like zero time almost negligible to load the model.

With no MM app, it has to pull all those weights into that private section. So you weight a lot longer for a model load or a model swap. Now the dead code fabl found lets you pin the existing MM app page memory. So you get the same transfer performance as no MM app, but you keep the perks of MM app like instant model loading. And it only pins the expert weights, not the entire model, so it doesn't take that much extra space. Our baseline was 1143 tokens per second on pre-fill. With this change, we are getting about 13.85 around 21% faster. Now, you will get roughly the same number if you just use no MAP, so I won't pretend it's a huge The point is we got there without no MAPS drawback. You keep the instant model loading and swapping. Real quick, the next optimization is the one that actually gets us to the headline number and it's where Fable did something I didn't think it could do. If you want local AI to keep getting faster, subscribe helps. Alright, back to it. So in pre-fill, remember, Lama.cpp pulls the experts back to the GPU and lets the GPU do the compute because with hundreds of tokens at once almost every expert gets selected

and it's way too much work for the CPU. But here's the plumbing problem. First the GPU computes the attention then it runs the router which gives you the IDs of the experts that need to execute. Then the schedulers can use the job to pull those experts from system gram back to VRAM and while that copy is happening the GPU just sits idle, Waiting for the experts to arrive before it can compute anything. And this happens around 40 times per batch. And feeble, profile all of this, what you're seeing on screen is the exact dump from that profiling. This is not an estimated diagram, it's the real data. The first stretch is the attention computing. Then it drains the weights from the previous layer, gets the new expert IDs and pulls those experts from system ramp to the GPU. and during that copy the compute is idle. Only once the copy is done does the GPU calculate. Add it all up and the GPU is sitting idle about 41.7% of the time. Effort and time we could have spent computing the next patch. The fix is actually simple and it's a pattern used all over the place. Two lanes running in parallel. One lane computes the attention, the other pulls the expert layers from system RAM up to the GPU. But here's the catch.

Until the router runs, we don't know which experts to pull. Except, at pre-filled when you're processing 500 to 2000 tokens in one batch, each pulling 8 experts, it turns out almost all of the 128 or 256 experts get selected anyway. So why wait for the router to hand us the IDs? While the attention is still computing, we just pull the whole layer back to V-RAM. So by the time the router is done, the weights are already sitting on the GPU ready. No waiting for the copy, it all happens in parallel. And that's what you're seeing below. Again, the actual profiling of the patched code, real data, across those 35 to 45 milliseconds. The compute isn't idle anymore. The GPU is always fed, always working. And that takes the GPU idle time from 41.7% down to 2.8%. The best part, we lose zero quality. The output is exactly what it would be without the optimization. This is purely a fix on the pipeline. And it did one more thing that genuinely impressed me.

It found the bug in its own patch and fixed it before I even saw it. So far everything fable did, sped up pre-fil, prompt processing. And that's only half the story. The other half is D-code, actually generating the tokens. And this is the harder half. If you watched my D flash video, this is that same territory. When Fable profile decode, it killed one of my assumptions. I always thought D-code on an offloaded MOE was CPU bound. The experts run on the CPU so the CPU has to be the bottleneck. It's not. It's a near 5050 split, the GPU and the CPU trading off back and forth the whole time. trap hiding in here. Speculative decoding is where you draft a few tokens ahead and verify them all in one pass. And normally longer drafts mean most speed up.

But on an offloaded MOE that flips. The cost of verifying is in the experts for one token. It's the union of the experts for every token you drafted. Draft more tokens and that combined set balloon towards all the experts and the CPU has to read every one of them. So past a point, longer drafts actually get slower. Drafting 8 tokens ahead was slower than not speculating at all. And that's the thing that finally explains the D flash collapse on my old 1060 that I could never figure out. So Fable had an idea for this. Build an adaptive controller. Something that watches the workload and picks the right draft length on the fly. Instead of a fixed number, that's the clever move. It should win. built it, then rebuilt it five times over. Each version it measured found exactly why it was losing, fixed that measured again. And every single time it lost to a dumb fixed cap.

The decision space was just too flat, the learning cost more than it saved. This is the rare honest one, the smart thing lost to the dumb thing. But it wasn't a total loss, it handed me a real tip. On an offloaded MOE, just cap your speculative draft at 2 or 3 tokens and you're about 23% faster with none of the slowdown. Then came Fable's bulldest idea. What of the CPU and the GPU worked on the same layer at the same time, split the expert work between them. On paper, that's free performance. In practice, it ran 14 times slower. And here's the part I want to be straight about. The mechanism actually worked every piece of it measured correctly. The reason it lost wasn't fable. It was my hardware. PCIe Gen 4 is just too slow next to reading straight from system RAM. So shipping that work across the bus cost more than it saves. On a machine with PCIe 5 or NV link, it would pay off. The wall was physics. So, 4 optimizations. Did they work? 2 of them did. Pinning the memory and overlapping the uploads. Together, that's the 64.5% on pre-fill, same GPU, same settings. The other 2, the adaptive controller and the CPU GPU split didn't. They lost and I showed you exactly why. And I want to be precise about that number because it's easy to inflate. The 64.5% is the code gain. What fables patches did. 1143 to 1880, nothing else touched.

That separate from any conflict tuning. And like I said at the start, every single change, Fable benchmarked before and after, and dipped the output itself. The result is token identical to mainline, same answers just faster. And one more thing, even on the ideas that failed, Fable surfaced a couple of genuine latent bugs in mainline lama.cp, the kind that have just been sitting there that nobody is hit yet. And I've got all the changes up on my GitHub fork. So if you want to test any of this yourself, it's all there. So step back for a second, an AI radar code base, profile late, found real bottlenecks, wrote the patches, benchmark them, and even caught a bug in its own code. And it made one of the most tuned pieces of software

on Earth, measureably faster, on a desk, not in a frontier lab. That's the loop. The thing we build AI with can now improve the thing we build AI with. And here's the part I didn't expect. The loop didn't just make the code faster. It made me better. That whole picture of where the expert math actually runs. CPU for one token and uploaded to the GPU for a big batch. I didn't know that until Fables profiling walked me through it. So the loop improves the code and it improves the person running it. Now I'll be honest about one thing. The paper isn't a local model. This was a cloud model improving your local stack. But think about where this is heading. The moment a model you can run locally on Lama.CPP gets good enough to improve Lama.CPP itself.

That's when the loop closes all the way. A model on your own hardware, making the engine it runs on faster, which makes that same model run better. No cloud, no frontier lab, just you and your machine improving each other. That's the version I actually want to see, and that's when we'll truly bring the power back to us. So that's the experiment. I genuinely love to know what you point-fable at, drop it in the comments. And if you want to see where this goes next, you know what to do. I'll see you in the next one.