Scenario and context

It sounds provocative, but this is what Colibrì is demonstrating: a new inference engine written in C that can run GLM-5.2 on a consumer machine with around 25 GB of RAM.

The interesting point is not that the hardware requirements have suddenly disappeared. What has changed is the way memory is managed.

GLM-5.2 is a Mixture-of-Experts model: it has 744 billion parameters in total, but activates around 40 billion for each token. Colibrì keeps the shared part of the model in RAM and loads only the “experts” needed at that moment from the NVMe drive.

In practice, VRAM, RAM and SSD storage are treated as a single memory hierarchy.

Operational implications

The result is technically remarkable:
• no mandatory GPU;
• around 20 GB of RAM during execution;
• an INT4-quantised model occupying around 370 GB on disk;
• fully local inference.

But be careful with the hype: “it can run” does not mean “it can be used comfortably”.

On a configuration without a GPU and with a cold cache, the project reports around 0.05–0.1 tokens per second. That means waiting as long as 10–20 seconds for a single token. It is therefore not a practical alternative to ChatGPT or a good, smaller local model.

The real news is different: local inference will not depend solely on the amount of VRAM available.

What changes for the company

MoE architectures, quantisation, expert streaming and intelligent memory management are expanding the range of models that can run on accessible hardware.

For companies and professionals, the message is clear: the future of local AI will not necessarily be “an enormous model on an enormous GPU”, but the ability to distribute computation efficiently across the CPU, RAM, storage and accelerators.

Today, Colibrì is primarily a technical demonstration. But it points in a very practical direction.

ai-academyitalia.it