Trying to start non-disposable node fails on second attempt

[negamaxi]

Electron app. Trying to start ipfs daemon in a renderer process with this piece of code:

    const factory = IPFSFactory.create({ type: "go" })
    const startDaemon = (ipfsd) => {
      ipfsd.start(daemonFlags, (err, api) => {
        if (err) {
          reject(err)
        }
        resolve(api)
      })
    }
    factory.spawn({ disposable: false, repoPath: '/my/repo/path/' }, (err, ipfsd) => {
      if (err) {
        reject(err)
      }
      if (ipfsd.initialized) {
        startDaemon(ipfsd)
      } else {
        ipfsd.init((err) => {
          if (err) {
            resolve(err)
          }
          startDaemon(ipfsd)
        })
      }
    })

If repo doesn't exists it works as expected: new repo get initialized, daemon starts, I can see it in my processes list and in subprocess field of ipfsd. But the next time demon does not start, despite the fact that initialized and _started fields are all true. No errors, subprocess field is null. Am I missing something?

[dryajov]

Thanks for reporting this - it should indeed work for your use case. This seems like a bug.

[travisperson]

Most likely the reason for this is because under $IPFS_PATH there exists an api file. In this case, ipfsd-ctl will skip setting up the daemon and simply return an ipfs-api instance for the address within the file.

We are only allowing 10500ms before we issue a SIGKILL. This will not allow the daemon to properly clean up and will leave around the api and repo.lock files under $IPFS_PATH.

This would be my guess to why you are having this issue. We probably should clean up these files ourselves if we ever forcefully kill the daemon.

[dryajov]

Thanks @travisperson! That could indeed be the reason.

@negamaxi could you check if the api file is being left behind after ipfsd-ctl shutdowns?

We did experiment with the 10500 (increased from ~7000 ms AFAIR), way back when we we're adding js-ipfs support and found it to be sufficient, but it could fail on certain envs I'm guessing.

@hacdias Have you experienced anything like this in ipfs-desktop?

[dryajov]

@negamaxi I was able to reproduce this behavior by terminating the ipfsd-ctl process abruptly, which doesn't give ipfs a chance to shutdown gracefully. This could happen under a number of scenarios, but a few obvious ones I can think off are:

• kill -9 the ipfsd-ctl or ipfs process
• if running under an ide and/or debugger, stopping the process will usually kill it abruptly
• slower system might take longer than the specified (10500ms) graceful timeout given to ipfs to shutdown before ipfsd-ctl kills it

[negamaxi]

@dryajov both repo.lock and api files confirmed. Removing api file solves the issue. I'm not sure how exactly ipfs terminates right now, it's just happens when I close electron window. Suppose I should intercept close event and use ipfsd.stop() method to be sure ipfs shuts down gracefully?

[negamaxi]

@dryajov also should I check for those two files and remove them on app start? There are definitely will be cases when app won't be able to shut down gracefully.

[travisperson]

If you are not shutting down the daemon it self through ipfsd.stop() there is a high chance of the daemon not shutting down gracefully.

Take a look at ipfs-shipyard/ipfs-desktop and how it handles the daemon management. It seems to do a fairly good job.

You may not want to remove those files on startup, as there could be a daemon running. However, if you are still having issues after implementing ipfsd.stop() it might be an okay workaround till ipfsd-ctl itself does this cleanup when it forcefully shutdowns daemons.

[dryajov]

If you are not shutting down the daemon it self through ipfsd.stop() there is a high chance of the daemon not shutting down gracefully.

One approach might be allowing the user to choose which behavior suits him/her more? For example, ipfsd-ctl is used extensively in our tests across the various projects (ipfs, ipfs-api, interop, etc...). When running in test mode, you might not care much about waiting for the process to exit at all, as we've made sure that we're running with disposable repos anyways - we care more about speed in that scenario. On the other hand, applications that reuse the repo do care about exiting cleanly and we should wait for ipfs to exit and never timeout (or increase the timeout to something that will be enough to work in all scenarios).

So my proposal is:

• disposable repos do what they do now when calling stop
• non disposable repos have a longer timeout by default
• modify ipfsd.stop to allo passing a timeout that allows the user controlling the behavior

@dryajov also should I check for those two files and remove them on app start? There are definitely will be cases when app won't be able to shut down gracefully.

Give what I said above, in user facing applications, the flow might be as follows:

• see if there is an already running daemon (you can send a command with the returned api, say to list the connected peers or something)
  • if there is one use it
  • if not
    • clean up the api and repo.lock files if they exist
      • attempt to start the daemon again
      • if it fails, the repo is probably messed up and the user should take some manual action

Take a look at ipfs-shipyard/ipfs-desktop and how it handles the daemon management. It seems to do a fairly good job.

👍

[negamaxi]

@dryajov @travisperson big thanks to both of you.