Do we want to add support for resumable Creator?

[benoit74]

For now, it is not possible to stop a "writer" process (Creator) and resume later. ZIMs have to be created in one go.

While this is ok for small ZIMs, it becomes to be a kind of hurdle for big ZIMs like the one of stackoverflow.com_en_all or wikipedia_en_all_maxi; the whole run takes about many days (around a week typically, or even more), and many kind of intermittent problem can occur during this week and cause the scraper to stop.

Currently, when the scraper stops, we have no solution in libzim to resume, so the current strategy is a kind of brute-force approach: we restart the scraper and hope no problem will occur again.

This just happened to https://farm.openzim.org/pipeline/2baf25b3-4838-47a5-9199-b9c278f90f1d where communication between worker and Zimfarm API went down for a long time and Zimfarm decided to cancel the task.

Current strategy incurs:

• a significant waste of computing resources
• a potential inability to create ZIMs for longer periods than necessary: since we always restart from scratch, we give more chances for an intermittent problem to occur again
• too much drain of attention for content team
• frustration for our users which keep waiting for some ZIMs to get updated and barely see any progress (if we resume, progress can only increase ; for now, we start over so progress restart from 0 at each attempt)
• difficulties to drain a worker for maintenance: see e.g. Fix custom location of container data on non-k8s machines kiwix/operations#683 (comment)
• hard work in scraper and Zimfarm to increase resiliency to rare intermittent issues (where we could just not mind too much and benefit from resume capabilities)
• difficulties to fix bugs: since it takes long elapse to create these ZIMs, it takes long time to realize the ZIM has a bug not seen in other ZIMs of same scraper, and then we start the whole loop again once fix has been (hopefully) implemented

I feel like it might be the right moment to start discussing feasibility and strategy to have the capability to resume a Creator process. Downside of such a move is increased complexity in libzim, scraper and Zimfarm.

I obviously don't yet have much details of what this move can entail, but I can imagine something around:

• savepoint operation in Creator to save current memory state needed to resume
• restore operation in Creator to restart operation from a savepoint
• "read" capability on Creator to know what is already present in a ZIM under creation

WDYT? Is this a move which makes sense to you? Should this be planned for somewhere in late 2026 / early 2027?

[kelson42]

On my end:

• I'm not convinved this is something which would really help because either the scraper is buggy or weak and then this won't really help OR the remote resource to get content is not reliable and this won't probably that much because we will have call "resume" so many times.
• Seems a pretty complex feature to implement

[benoit74]

either the scraper is buggy or weak and then this won't really help OR the remote resource to get content is not reliable and this won't probably that much because we will have call "resume" so many times

When a task takes days to complete (and we have many in that situation), multiple kind of situation can happen which is hard for scraper to deal with in an automated way because it is hard for him to differentiate a definitive error from a transient problem which takes a bit long to resolve. The resource can be down for few hours. The worker might have a long network issue (like more than few minutes). The worker might suddenly reboot (just happened few days ago). These are not so rare situations, I don't expect to have to "resume" the same task multiple times, and restarting days long task over and over did happened significantly recently (I'm thinking at least about stackoverflow and wikipedia_en_all_maxi but others did fail and were not restarted because too boring).

[benoit74]

I agree it is probably a complex feature to implement, both libzim side and scraper side. But responsible infrastructure use both on worker side and resource side makes it worth it IMHO. As well as simplified Zimfarm usage.

[veloman-yunkan]

A will dare ask a silly question.

Why don't we simply change the scrapers to first download all resources to a directory and then run ZIM creation from data available locally?

[rgaudin]

I aged 👴 so much in the seconds it took to read your comment @veloman-yunkan because, if you don't already know, that's exactly what we were doing back in the days and invested a lot of effort to switch to the on-the-fly mode. At that time, main reason was performance, to escape the I/O bottleneck because network was so much faster than storage.

[veloman-yunkan]

BTW, having a resumable ZIM creator makes sense regardless of where the data comes from (network or local storage). Any long running task that can be ruined by a single glitch is not worth starting unless it is resumable from a saved checkpoint state.

Curiously, I am now (re-)reading a book by Arthur C. Clarke about the laying of the first transatlantic telegraph cable which was an endeavour facing the same type of problems. The first attempt in 1857 failed when the cable broke over depths of about 3200 meters with 480 kilometers of the (already laid) cable lost. It's just staggering how those people made another attempt without having a way of recovering from a similar situation and just hoping that it won't happen again. They were unlucky with three more attempts carried out the next year. The fifth attempt was successful, but the cable deteriorated (and stopped transmitting any signals) within a month. And the chief driver of the project - the Great American Cyrus Field - was committed to this project to an extent that he kept looking for more funding while US was fighting its Civil War and finally had the two continents connected in 1866 (though another failed attempt in 1865 suggested that with the right tools and technique it should be possible to recover a broken cable lying on the ocean bed).