Long running migrations can cause systemd to terminate daemon on start

[Stebalien]

Version information:

0.5.0

Description:

Now that go-ipfs supports systemd's "notification" system, we need to tell systemd to repeatedly extend time startup timeout while performing repo migrations. Otherwise, systemd may kill the daemon thinking it timed out on startup.

We can do this by repeatedly sending EXTEND_TIMEOUT_USEC=... to systemd's notification socket using github.com/coreos/go-systemd/v22/daemon. See cmd/ipfs/daemon_linux.go for how we interact with systemd's notification service.

[RubenKelevra]

Can we notify systemd the same way while we open the database on startup (which is basically the reason for the slow startup documented here and on shutdown when we for example run database compaction jobs, flush data to disk, or clean up sockets etc.?

[Stebalien]

Yep. We can wrap all of https://github.com/ipfs/go-ipfs/blob/538bff085abb18a16fbf14386358c00c7078c222/cmd/ipfs/daemon.go#L257-L290.

[Stebalien]

However, we don't do database compactions on stop (explicitly disabled for this reason) so we shouldn't delay there. If IPFS gets stuck on shutdown, we should just die and cleanup when we next restart.

[RubenKelevra]

I've seen ipfs hit the one minute 30-second limit dozens of times on different machines, without badger db. My guess is, that it's a TCP cleanup issue.

I think it's still nicer to clean up everything cleanly and as long as we make progress we notify systemd.

Maybe add a hard limit, after which we stop notifying on shutdown, to make sure we don't hang indefinitely.

[Stebalien]

Odd. We shouldn't be spending any time cleaning up TCP connections or anything like that, unless we have a bug somewhere. Can you reproduce this?

[RubenKelevra]

Odd. We shouldn't be spending any time cleaning up TCP connections or anything like that, unless we have a bug somewhere. Can you reproduce this?

Well, I converted all my nodes to badgerds with the 0.5.0 release.

I need to convert one back to see how it can be reproduced.

[RubenKelevra]

@Stebalien

I've seen ipfs hit the one minute 30-second limit dozens of times on different machines, without badger db. My guess is, that it's a TCP cleanup issue.

I can report this bug seems to be gone. Not sure when, but I haven't noticed it at all on 0.8.

Apart from this, wouldn't it make sense to send go-ipfs a SIGABRT instead of a SIGKILL by systemd by default if this happens? This gives the user a stack trace to share with us?

[Stebalien]

I agree. Want to file a PR?

[Stebalien]

Ok, I'm actually just going to disable the startup timeout. It's not helping.