Alternative region file format

[MelonHell]

Is your feature request related to a problem?

the world in minecraft weighs indecently a lot, large servers take up hundreds of gigabytes of space. I think it is worth embedding in a popular server software such as paper an alternative world storage format with better compression

Describe the solution you'd like.

solution examples are Slime and Linear
https://github.com/hugmanrique/Slime
https://github.com/xymb-endcrystalme/LinearPaper

Describe alternatives you've considered.

build Paper with Linear by yourself

Other

I do not propose to replace MCA with Slime, Linear or some kind of PaperRegionFormat, but I think that it should be possible to enable world compression in the config

upd. the slime example was bad, it's really not suitable for regular worlds

[Malfrador]

The maintenance effort of this would be immense, only to benefit a very small percentage of servers. World size usually isn't an issue, considering how cheap (SSD) disk space is nowadays.
A different compression was considered at some point, but ultimately not done due to several reasons, see #5029.

If you wish to use the slime world format - which isn't well suited for normal sized minecraft worlds, so wouldn't really be an option as a replacement anyways - you can just use https://github.com/InfernalSuite/AdvancedSlimePaper

[MelonHell]

The maintenance effort of this would be immense, only to benefit a very small percentage of servers. World size usually isn't an issue, considering how cheap (SSD) disk space is nowadays. A different compression was considered at some point, but ultimately not done due to several reasons, see #5029.

If you wish to use the slime world format - which isn't well suited for normal sized minecraft worlds, so wouldn't really be an option as a replacement anyways - you can just use https://github.com/InfernalSuite/AdvancedSlimePaper

most projects cannot afford their own server and use hosting services that have very inflexible rates. All infinite world projects that I know use their own servers. I'm sure this is not a useless feature, many projects will be able to expand their worlds several times, and there will be more projects with an infinite world

[electronicboy]

The burden of supporting and maintaining this sorta stuff often exceeds it's worth

The way that patch is designed is far from ideal for inclusion into paper, would be somewhat easy to make it more viable but it's far from a priority, not to mention the headaches around messing with this stuff. Relying on 3rd party tooling to convert worlds, this is especially the type of stuff which makes this inaccessible to the people who are running in shared hosting which means that the only people who get a benefit are people using their own hardware and/or are able to eat excessive downtime to convert worlds over, or people who go into this from scratch knowing this is a thing. Just feels like a tremendously niche type of deal and would be really low down on any form of priorities list

[lynxplay]

In my honest opinion I doubt shipping different region formats in paper is a good idea.

Something has impactful as the region format would need even more upkeep than different redstone implementations (see eigencraft and alternate, which paper does ship). Redstone engines can be turned on and off without much harm, changing region formats is a rather permanent change unless we also support migration from one into the other, which again is just more upkeep.

Slime, as stated above, is very specific to what worlds it would benefit.
I have yet to look into linear, which seems to be a very young format. As I have not looked into it I cannot comment much on it, however it is not particularly well known judging by the github stars at least, which brings me to my next point.

Including non standard region formats that are not based on some wildly known and maintained community standard makes future maintenance even harder. A format developed and maintained by a single person may just die at any moment and we'd have to retire a region format, breaking servers. (or develop the format ourselves, which is even more work on us).

[xymb-endcrystalme]

I think this is the best page that describes .linear with words: https://github.com/xymb-endcrystalme/LinearRegionFileFormatTools

I've been running it for more than a year on a multi-TB world, with no issues specific to the format itself. After all, it's only a region file format, it doesn't interact with any other code.

It saves about 50% of disk space in OW and Nether and 95% in The End at the cost of increased CPU and RAM usage.

A format developed and maintained by a single person may just die at any moment and we'd have to retire a region format, breaking servers. (or develop the format ourselves, which is even more work on us).

The saving grace here is that a region file doesn't do anything else than just encapsulate the NBT of chunks.

How much maintenance did .mca require over the years? Only one to implement support for chunks bigger than 1MB. Other than that it's basically running the same code for the last 10 years.

Thus I don't think that "may just die at any moment" is an issue at all to implementing a second, high-compression region format for open-world servers.

[xymb-endcrystalme]

The way that patch is designed is far from ideal for inclusion into paper

Totally agree. For example I made it use a thread for every region file, which does get out of hand when reading crash logs.

Just feels like a tremendously niche type of deal and would be really low down on any form of priorities list

Yep, everyone who runs a big server is experimenting a lot with forks and tools, so downloading a fork for a region file format is not a big deal.

I think that just having forks that use another region file format is good enough of a compromise. There is no need to include in Paper things that only 1/1000 servers will use.

[RealTriassic]

I'd personally find a lot of use in an alternative region file format as my world file is nearing 1TB.

[lynxplay]

I've been running it for more than a year on a multi-TB world, with no issues specific to the format itself. After all, it's only a region file format, it doesn't interact with any other code.

I am glad it is working out for you. Again I have not used your codebase, at least the stats you provide sound great!
Does not take away from the fact that including alternative region formats requires more than just the logic for it.
Conversion from and to anvil, upkeep on version changes down the line (who is to say the 1.17 changes to chunk/entity loading were the last of it) etc.

Just to make sure I don't come off as insulting, I believe there is definitely improvements to be made to the format and if your code is working well, then that sound like a very cool project that people with giant worlds should certainly check out.
But having additional maintenance cost in the paper project for the small number of users that actually end up with such large worlds, large enough that just getting more disk space is not an option, seems less than ideal.

I think that just having forks that use another region file format is good enough of a compromise. There is no need to include in Paper things that only 1/1000 servers will use.

Sounds like a good compromise in my honest opinion. It gives you (or other format developers) more freedom in their development and also just yields a bit more credit to your name for putting the work into designing such a format :)

[xymb-endcrystalme]

I think this is the best page that describes .linear with words: https://github.com/xymb-endcrystalme/LinearRegionFileFormatTools

I've been running it for more than a year on a multi-TB world, with no issues specific to the format itself. After all, it's only a region file format, it doesn't interact with any other code.

It saves about 50% of disk space in OW and Nether and 95% in The End at the cost of increased CPU and RAM usage.

A format developed and maintained by a single person may just die at any moment and we'd have to retire a region format, breaking servers. (or develop the format ourselves, which is even more work on us).

The saving grace here is that a region file doesn't do anything else than just encapsulate the NBT of chunks.

How much maintenance did .mca require over the years? Only one to implement support for chunks bigger than 1MB. Other than that it's basically running the same code for the last 10 years.

Thus I don't think that "may just die at any moment" is an issue at all to implementing a second, high-compression region format for open-world servers.

[xymb-endcrystalme]

The way that patch is designed is far from ideal for inclusion into paper

Totally agree. For example I made it use a thread for every region file, which does get out of hand when reading crash logs.

Just feels like a tremendously niche type of deal and would be really low down on any form of priorities list

Yep, everyone who runs a big server is experimenting a lot with forks and tools, so downloading a fork for a region file format is not a big deal.

I think that just having forks that use another region file format is good enough of a compromise. There is no need to include in Paper things that only 1/1000 servers will use.

[RealTriassic]

I'd personally find a lot of use in an alternative region file format as my world file is nearing 1TB.

[lynxplay]

I've been running it for more than a year on a multi-TB world, with no issues specific to the format itself. After all, it's only a region file format, it doesn't interact with any other code.

I am glad it is working out for you. Again I have not used your codebase, at least the stats you provide sound great!
Does not take away from the fact that including alternative region formats requires more than just the logic for it.
Conversion from and to anvil, upkeep on version changes down the line (who is to say the 1.17 changes to chunk/entity loading were the last of it) etc.

Just to make sure I don't come off as insulting, I believe there is definitely improvements to be made to the format and if your code is working well, then that sound like a very cool project that people with giant worlds should certainly check out.
But having additional maintenance cost in the paper project for the small number of users that actually end up with such large worlds, large enough that just getting more disk space is not an option, seems less than ideal.

I think that just having forks that use another region file format is good enough of a compromise. There is no need to include in Paper things that only 1/1000 servers will use.

Sounds like a good compromise in my honest opinion. It gives you (or other format developers) more freedom in their development and also just yields a bit more credit to your name for putting the work into designing such a format :)

[lordofpipes]

There is currently an experimental region file implementation (SpottedLeaf's SectorFile) here - #10231

[MelonHell]

haha, looks like I was right 🙂

[lordofpipes]

haha, looks like I was right 🙂

Maybe. It's not yet confirmed whether it will be maintained in Paper, only in Folia, or out of tree, or if the idea will be dropped altogether and left as an experiment. Adding new region formats is a potentially risky move.