Compress 80TB Minecraft map down to 15TB with custom format

In a world grappling with chronic shortages of memory and storage, a dedicated team of Minecraft players embarked on an epic digital archaeology project that redefined the boundaries of data preservation. They successfully mapped the entirety of 2b2t, arguably Minecraft’s oldest and most notorious server, culminating in a monumental 15TB download.

This wasn’t just a casual game session; it was a massive undertaking that took 22 months of relentless effort. To achieve this feat, the team deployed 28 bot accounts, tirelessly traversing the vast landscape of 2b2t to meticulously map every single block across the three primary dimensions: the Overworld, the Nether, and the End. The sheer scale of this digital expedition is staggering, encompassing well over one million blocks.

The scope of the data they sought was immense. The full footprint of the 2b2t server exceeds 80 TB. Simply storing this volume of information would require an astronomical amount of physical storage—think eight 8TB M.2 SSDs or five 16TB SATA SSDs, plus several massive hard drives.

But tackling the challenge posed by today’s scarcity was only half the battle. The team faced the reality that sharing this data would quickly push file sizes to impractical levels given current storage limitations and pricing. To solve this infrastructural headache, they engineered a custom solution: a proprietary compression format named .zvcr (Zstd-compressed Version Controlled Regions).

Unlike standard compression methods like Zlib or Gzip, the team utilized the superior Zstd algorithm. This choice was strategic, as Zstd delivers better compression ratios while simultaneously offering faster compression and decompression speeds than its competitors. The result? They managed to compress the entire 80 TB of server data into a highly efficient 15TB archive, saving up to 81.25% of space.

This ingenious approach dramatically altered the storage requirement. Instead of needing multiple massive drives, enthusiasts now only require two or one drive to store the compressed archive. While the team successfully made the entire history of the anarchic 2b2t server available for digital archaeology, the effort was driven by a deeper motivation: a desire to preserve the original, chaotic essence of the world before it shifts away from its roots.