program should daemonized itself

[dwerder]

So this program will probably most of the time run as a daemon it should have a flag like -daemonize so it disconnects itself from the start shell. At the moment on redhat system you have to use dirty workarounds like using nohup or & in startup scripts.

[driskell]

Unfortunately I think this would be difficult - since go itself does not implement fork nor guarantee any safety when using it. Some OS provide a daemon wrapper to do the demonising too but some don't and that's where the nohup fun begins.

[jordansissel]

Ignoring the difficulty (it's doable in Go with the same hacks that Java uses for this), I don't think it's useful.

My intention is to provide upstart/systemd/sysvinit configs that allow you to run it as a service. Simply adding a 'background this' flag doesn't solve the 'run this as a service' problem - and additionally, things like upstart, systemd, runit, supervisord, and more, do not require "daemonizing" programs.

If you need an init script right now you can try generating one with https://github.com/jordansissel/pleaserun - pleaserun --install --name logstash-forwarder /path/to/logstash-forwarder -f /path/to/config

[driskell]

Hmmm, thinking of LogStash though @jordansissel, fork behaviour can be quite useful, because you can leave stderr attached and not fork until you've initialised, checked config, opened log files etc, and only then fork.

That lets you catch startup errors (such as ruby compilation problems) in stderr during a 'service X start' command (or equivalent) - from Logstash perspective it only logs to syslog after its initialised so errors during startup get lost in the ether. Definitely the thing that could save someone time and not have to run the command on cmd line directly to see the output.

I guess the way around that though is to just sent stderr/stdout to another log file heh - but lots of stuff don't do that currently! :(

[driskell]

Last msg: Problem without fork is the service start always succeeds, even if the daemon fails to start. So can be difficult to check its going to stay alive.

[dwerder]

This is my point. I will try the pleaserun command the next days to see how it starts the service. But my point with nohup solutions and co is that you always get a positive result from the start scripts even if the command fails and you never see why.

[jordansissel]

Most non-sysv systems (upstart, systemd, runit, daemontools, supervisord, etc) solve this simply by automatically restarting things when they die and also providing an interface to ask if a program is running (initctl status, systemctl status, sv stat, etc)

It's also unclear what "fail to start" means. What errors would be considered a failure to start? Can we simply provide a way to check for those errors? What would you do when it fails to start?

Would simply starting and then asking if it's alive be sufficient? (that's the point of 'status' queries in most sysv init scripts, upstart and all the others also provide 'status' semantics)

[jordansissel]

For example, I would consider it a "failure to start" if there were a syntax error (syntactically incorrect flags or configuration). But it's also a "failure to start" more semantically if I provide a syntactically correct config but I have a typo in a file or a field name in the config. In this semantic failure, lsf will still be running, but it wouldn't be behaving correctly because of a human error.

[driskell]

I guess another way to explain, is during startup an app can "normal exit" for any reason it wish - config error etc. But once it forks or enters background it should never be a "normal" exit and there should always be logs.

Failing to find the binary or failing to compile/interpret/load it (which happens once only usually) would be a normal exit, but by whatever executor, and likely not a bug but a config issue with OS possibly.

Logstash failing cause jruby couldn't compile the program for example - that would be normal exit and should be immediately visible and logged, and would be due to OS or install issue outside the program itself, not a software bug.

The point a daemon forks would be a statement to config management or scripts, "I reasonably expect I will run fine from here-in - if I don't - it's a bug"

[jordansissel]

Right, but what I"ms aying is taht even with daemonizing, everything outside of sysv init (which is the future of service management, things like systemd, etc) doesn't really have this concept of daemonizing-after-start. It's also difficult to provide these "fork after start" semantics in languages like Java or Go (or anything that uses threads of any kind) because forking destroys all the work done since the start of program execution.

So with respect to "abnormal early termination" you can detect this by asking for status shortly after starting; if it's not running, it died and there's probably a problem ;)

[driskell]

If we look at Windows it's interesting there. A service is responsible for notifying the SCM (service control manager) it has started successfully. This allows SCM to detect startup failure such as hangs etc and force restart. Maybe sysv needs a similar interface. :)

[dwerder]

Ok this starts to be a basic discussion of the known problems of the sysv init system. I agree that the modern service manager have better solutions at hand. But at the moment the majority of Redhat based distributions use the old sysv init system. It was just a questions if it is possible to get a "native" sysv behavior for this service. If the answers is: No, we will not blow up the code for legacy support, then I can accept that. Redhat/CentOS 7 is near at hand so.

I think the pros and cons of sysv a discussed many times. ;-)

[jordansissel]

@dwerder No code changes are required for sysv support. /bin/sh has supported backgrounding tasks for a loooong time and should be sufficient - where "should be sufficient" is based only on my personal experience in operations and dealing with sysv init systems and that your experiences may vary.