btrfs balance doesn't resume on reboot

[CorrosiveTruths]

Hi, the documentation for mount options, balance itself, and my failing memory suggest that on reboot, a started balance will resume.

This doesn't seem to be the case anymore, the kernel cancels the balance as part of reboot, and resume only works on paused balances, whereas I think it previously worked on both cancelled and paused balances (but that could be a trick of my memory).

Not sure what the bug is. Either the documentation should be updated to reflect the current state of things, balances should pause rather than cancel on reboot, or resume should work with cancelled balances.

[Zygo]

Is it cancelling the balance, or does the balance just finish very quickly due to the usage filter injected by the thing in issue #550? Do you have kernel logs showing cancellation messages?

You might be mixing scrub and balance in your memory. Scrub can be resumed after cancellation because there's no "paused" state in the kernel for scrub--userspace starts a brand new scrub based on progress data it saved in /var/lib/btrfs/scrub.status.*, but tells the kernel to start at the point where the previous scrub left off. Balance cancel deletes the balance status item from the filesystem, making resume after cancel impossible.

Balance pause + resume arguably never worked because these operations do not place the balance in its former pre-pause state, and cannot do so because the balance item doesn't record progress. See commentary in #550 for details.

[CorrosiveTruths]

Hey Zygo, it is cancelling the balance according to logs from the kernel and no balance is running on the next start-up.

This is the penultimate kernel message before the reboot.

kernel: BTRFS info (device sdf): balance: canceled

Current boot has nothing from the kernel for balance.

Wasn't thinking of scrub in this case, was thinking of the last time I had to skip_balance, back in April 2021. Looking at the logs it also has 'kernel: BTRFS info (device sda): balance: canceled', but that then crashed, so didn't actually cancel?

Could it be the case that the kernel has for a long period tried to cancel the balance before reboot, but would frequently fail before all the changes making balance easier / quicker to cancel?

If balance being resumed (kinda) after reboot is now the rare case, then this is just a problem with documentation?

[Zygo]

The balance process can be cancelled with a fatal signal now (since kernel 5.9). If the message appears before reboot, then the balance process is likely being killed (and therefore cancelled) before the filesystem is umounted, e.g. by shutdown scripts sending SIGTERM to all processes. On previous kernel versions, the shutdown script would time out and finish shutdown without unmounting the filesystem.

If I set up a test VM running 6.1.0-rc7, start a balance, kill the VM (so nothing gets a chance to send SIGTERM to btrfs balance), and start the VM again, btrfs balance resumes on the following mount.

So there is a recent behavior change here, but the effect is not in the same place as the cause.