Running a Node.js app (ghost) in the background continuously with Supervisor (Supervisord)

Pre-requisites

Root access to your server or have access to an account that has sudo powers on your servers – Instructions here.
nodejs and npm installed – Instructions here.
ghost installed and a ghost user made to run ghost – Instructions here.

What is supervisor

From the official supervisor page,

Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.

It shares some of the same goals of programs like launchd, daemontools, and runit. Unlike some of these programs, it is not meant to be run as a substitute for init as â€œprocess id 1â€. Instead it is meant to be used to control processes related to a project or a customer, and is meant to start like any other program at boot time.

An important note here is it is meant to start like any other program at boot time. That means once supervisor is running our app we don’t have to worry about it surviving a reboot. It turns out that we accomplish that using an init script.

Installing supervisor

Do not use the package manager to install. The version they currently have is waaay behind:

[root@amayem supervisor]# yum list | grep supervisor
supervisor.noarch                       2.1-8.el6                      @epel    
nodejs-supervisor.noarch                0.5.2-5.el6                    epel

Wow it’s version 2.1. According to Pypi that version was released in 2007. Here is what happens when you try to use it.

Instead let’s install using easy_install.

Installing python

In order to start we need to have python installed. Supervisor’s github page says,

Supervisor is known to work with Python 2.4 or later but will not work under any version of Python 3.

Let’s check if we have python installed to begin with and if it installed, which version it is:

[ahmed@amayem ~]$ python --version
Python 2.6.6

Looks like I’m good. However if you find that you don’t have python installed then you can install it using yum

[ahmed@amayem ~]$ sudo yum install python

Make sure the version is acceptable (between 2.4 and 3). If you don’t see the right version number then enter n at the prompt and try this:

[ahmed@amayem ~]$ sudo yum install python 2.6.6-29.el6_3.3

Installing setup_tools (which has easy_install)

Next we need to install setup_tools as follows:

[ahmed@amayem ~]$ wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py -O - | sudo python

Now we can finally install supervisor.

Installing Supervisor

[ahmed@amayem ~]$ sudo easy_install supervisor
Searching for supervisor
Reading https://pypi.python.org/simple/supervisor/
Best match: supervisor 3.0

Notice the version number of supervisor. 3.0 is the latest stable version so we are good to go.

It should end with the following:

Installed /usr/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg

Let’s start using supervisor.

Using Supervisor

Easy way

supervisord and supervisorctl should have been installed in /usr/bin

[ahmed@amayem ~]$ ls /usr/bin | grep supervisor
echo_supervisord_conf
supervisorctl
supervisord

This allows us to use them from anywhere as follows:

[ahmed@amayem ~]$ sudo supervisord

For the sake of education we will also take a look at a lower level way of making it work by using python explicitly.

Harder way

Let’s go to where supervisor was installed.

[ahmed@amayem ~]$ /usr/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg
[ahmed@amayem supervisor-3.0-py2.6.egg]$ ls
EGG-INFO  supervisor
[ahmed@amayem supervisor-3.0-py2.6.egg]$ cd supervisor/
[ahmed@amayem supervisor]$ ls
childutils.py   dispatchers.py   http.py       medusa        process.pyc        socket_manager.pyc  supervisord.pyc  xmlrpc.py
childutils.pyc  dispatchers.pyc  http.pyc      options.py    rpcinterface.py    states.py           tests            xmlrpc.pyc
confecho.py     events.py        __init__.py   options.pyc   rpcinterface.pyc   states.pyc          ui
confecho.pyc    events.pyc       __init__.pyc  pidproxy.py   scripts            supervisorctl.py    version.txt
datatypes.py    http_client.py   loggers.py    pidproxy.pyc  skel               supervisorctl.pyc   web.py
datatypes.pyc   http_client.pyc  loggers.pyc   process.py    socket_manager.py  supervisord.py      web.pyc

We found the treasure. What we are most interested in is supervisord.py. Let’s try to run it. It has a .py extension, which means it needs python to run.

[ahmed@amayem supervisor]$ python ./supervisord.py

Making the configuration file

Whichever way you choose to start you will get this error:

Error: No config file found at default paths (etc/supervisord.conf, supervisord.conf, supervisord.conf, etc/supervisord.conf, /etc/supervisord.conf); use the -c option to specify a config file at a different path
For help, use ./supervisord.py -h

Looks like we need to make the configuration file.

We can get a sample configuration file with the following command:

[ahmed@amayem supervisor]$ echo_supervisord_conf

Let’s redirect the output into a config file that supervisor can use:

[ahmed@amayem supervisor]$ sudo echo_supervisord_conf > /etc/supervisord.conf 
-bash: /etc/supervisord.conf: Permission denied

We need to switch to root.

[ahmed@amayem supervisor]$ sudo su
[root@amayem supervisor]# echo_supervisord_conf > /etc/supervisord.conf

Now we give it a try

[root@amayem supervisor]# supervisord
usr/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/options.py:295: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '

It seems to be working but we got an annoying warning message. I guess we should use the -c argument from now on like the following:

[ahmed@amayem supervisor]$ sudo supervisord -c /etc/supervisord.conf

Let’s check the status of supervisor:

[root@amayem supervisor]# supervisorctl
supervisor> status
supervisor>

That means that supervisor is not supervising any processes at the moment. Time to add our node.js app’s configuration. The following configuration is from ghost’s deployment docs.

[program:ghost]
command = node /path/to/ghost/index.js
directory = /path/to/ghost
user = ghost
autostart = true
autorestart = true
stdout_logfile = /var/log/supervisor/ghost.log
stderr_logfile = /var/log/supervisor/ghost_err.log
environment = NODE_ENV="production"

We need to append the above to the supervisord.conf file. You need the sudo to be able to modify the file. Go to the bottom by pressing G, enter into insert mode i and paste the above. On my system the path to ghost is /var/www/ghost/. Also notice that we are using the ghost user. This should have been made when installing ghost. Please check these instructions for more information on how to do that. Exit insert mode with Esc or ctrl+c then save with :x and enter. One way of getting supervisor to read the new configuration is by restarting it.

Restarting supervisor

We can restart using supervisorctl as follows

[root@amayem supervisor]# supervisorctl
supervisor> help

default commands (type help <topic>):
=====================================
add    clear  fg        open  quit    remove  restart   start   stop  update 
avail  exit   maintail  pid   reload  reread  shutdown  status  tail  version

restart looks promising:

supervisor> restart
Error: restart requires a process name
restart <name>      Restart a process
restart <gname>:*   Restart all processes in a group
restart <name> <name>   Restart multiple processes or groups
restart all     Restart all processes
Note: restart does not reread config files. For that, see reread and update.

Nope! Let’s check some other commands:

supervisor> help shutdown
shutdown    Shut the remote supervisord down.
supervisor> help reload
reload      Restart the remote supervisord.

Looks like reload is the one I want.

supervisor> reload
Really restart the remote supervisord process y/N? y
Restarted supervisord
supervisor> status
ghost                            RUNNING    pid 6215, uptime 0:00:06

Great! Now that we have ghost running, let’s check that it is working properly. Visit your site at port 2368, in my case it was http://ahmed.amayem.com:2368. Success.

Making supervisor survive a reboot.

Testing it survive a reboot

First, let’s check if it is running.

[ahmed@amayem supervisor]$ ps aux | grep supervisor
root      1488  0.0  0.1  17516  7840 ?        Ss   18:26   0:00 python ./supervisord.py -c /etc/supervisord.conf
ahmed     1564  0.0  0.0   5508   728 pts/1    S+   19:14   0:00 grep supervisor

If you don’t see it running then go to the section above to see how to start it. Now we reboot

[ahmed@amayem supervisor]$ sudo reboot

and check again

[ahmed@amayem ~]$ ps aux | grep supervisor
ahmed      553  0.0  0.0   5508   728 pts/0    S+   19:16   0:00 grep supervisor

Argh! It’s not there. To make it survive a reboot we need an init script.

Making the init script

Some community init scripts for supervisor are available here. I am running on CentOS, which is supposed to be upwards compatible with red hat, so I chose to go with redhat-init-mingalevme, because it was the most comprehensive. Copy all the contents of the file. Now we need to paste it into a new init script.

[ahmed@amayem ~]$ cd /etc/init.d/
[ahmed@amayem init.d]$ sudo vi supervisord

Enter into insert mode with i and paste. For more about pasting in vi check here. Exit insert mode with Esc or ctrl+c then save with :x and enter.

We have an init file but we have to tell the OS to use it as a service. Let’s compare its permissions with other init scripts.

[ahmed@amayem init.d]$ ls -l
-rwxr-xr-x 1 root root  4683 Jun 22  2012 sshd
-rw-r--r-- 1 root root  3631 Apr 16 19:57 supervisord

I’ve truncated the output to only two items. Basically I need to add x or execute permissions to the owner, group and others as follows:

[ahmed@amayem init.d]$ sudo chmod ugo+x supervisord 
[ahmed@amayem init.d]$ ls -l 
-rwxr-xr-x 1 root root  4683 Jun 22  2012 sshd
-rwxr-xr-x 1 root root  3631 Apr 16 19:57 supervisord

They are exactly the same. Now I need to add the file to chkconfig list.

Adding the init script to chkconfig

[ahmed@amayem init.d]$ sudo chkconfig --add supervisord
[ahmed@amayem init.d]$ chkconfig --list
sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
supervisord     0:off   1:off   2:off   3:on    4:on    5:on    6:off

Looks like it’s been added and it is set to be start at run levels 3, 4 and 5. This is acceptable. For more about run levels check here. We can see our default run level at /etc/inittab:

[ahmed@amayem init.d]$ less /etc/inittab

Mine was set at:

id:3:initdefault:

So supervisor should run properly. Let’s confirm.

Testing starting on reboot

[ahmed@amayem init.d]$ sudo reboot

After waiting a few seconds login again and check if it is running:

[ahmed@amayem ~]$ ps aux | grep supervisor
root       513  0.0  0.1  17500  7768 ?        Ss   00:02   0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
[ahmed@amayem ~]$ sudo supervisorctl
ghost                            RUNNING    pid 515, uptime 0:03:25
supervisor>

Looks like it is. Check on your browser that ghost is running. All done!

References

The official supervisor page
Pypi’s setup_tools instructions
Supervisor’s github page
Ghost’s deployment docs