Running a 284B parameter model across two machines matches the cloud

The pursuit of running massive artificial intelligence models on consumer-grade hardware often feels like an exercise in wrestling with physics. Just a few weeks ago, I took on this challenge by attempting to run DeepSeek V4 Flash on a single Lenovo ThinkStation PGX.

While the experiment ultimately succeeded, it came packed with significant caveats. Getting such a behemoth of a model onto limited resources required clever engineering. The backbone of this operation was the ds4 engine, a sophisticated C project meticulously crafted for a single family of models.

Despite these limitations, the setup performed admirably. It managed to handle a million-token context window at speeds ranging between 10 to 14 tokens per second. However, the performance still demanded careful management. To fit a model boasting 284 billion parameters into just 128 GB of VRAM, the ds4 engine had to employ aggressive quantization, reducing the routed experts down to just 2 bits.

This necessary optimization introduced a trade-off: while the system could run, the resulting model was both slow and measurably less effective compared to what DeepSeek hosts on dedicated infrastructure. The experience perfectly illustrates the reality of pushing the boundaries between computational ambition and physical constraints in the world of AI hardware.

Ultimately, every high-performance experiment comes down to shared limitations. In this case, success hinges entirely on managing the bottleneck of 128 GB of VRAM and the relentless flow of 273 GB/s of memory bandwidth. These physical constraints define the limits of what can be achieved when trying to run models of this magnitude.

By Jackson Turner