logging - Get notification from supervisord when a job exits

Is there any way supervisord can automatically restart a failed/exited/terminated job and send me a notification email with a dump of the last x lines of log file?

Answer

There is a plugin called superlance.

You install it with pip install superlance or download it at: http://pypi.python.org/pypi/superlance

The next thing you do is you go into your supervisord.conf and add the following lines:

[eventlistener:crashmail]
command=/usr/local/bin/crashmail -a -m email1@example.com
events=PROCESS_STATE

This should be followed by a "supervisorctl update". When a process "exits" you will now get a notification sent to email1@example.com.

If you only want to listen to some selected apps you can exchange the -a for a -p program1 or if it is a group group1:program2 One example would be

[eventlistener:crashmail]

command=/usr/local/bin/crashmail -p program1 -p group1:program2  -m email1@example.com
events=PROCESS_STATE

Regarding the automatic restart:
you should make sure that autorestart is set to true (it is set to unexpected by default). This way the package will be restarted 3 times. If after that it still exits, it gives up, but you can change that with startretries.

Example program:

[program:cat]

command=/bin/cat
autorestart=true
startretries=10