I spent the last several hours troubleshooting any annoying error with the mongodb init script on Ubuntu. The script would start the daemon easy enough, but would report a failure. When subsequently trying to stop the daemon, it would then then say it was successful, but in-fact, still be running.
Fortunately, I found that somebody else had already reported the bug, but the comments pointing developers to the wrong place – looking inside the mongod code instead of in the init script.
After digging in to it for longer than I’d like to have spent, I found the cause was the –make-pidfile option being used in the init script.
My understanding of this process is that the start-stop-daemon command was creating the pidfile (as root), before spawning the actual mongod process (as the mongodb user). mongod in some cases (at least when not configsvr=true) must fork again before saving its own pidfile. Since the file created by start-stop-daemon is being run as root, the less-privileged mongodb user can not overwrite it (perhaps this should be logged, or logged at a less verbose level?), leaving the pidfile containing a pid that is no longer correct.
On my machine, the pidfile created with the –make-pidfile options was consistently exactly three less than the PID shown in the output of ‘ps’
After making that change to the init file, I can now reliably start/stop the mongod process using the expected commands.
Hopefully that bug will be closed soon and released so that I don’t have to customize the init script on every mongo server I have.