This post discusses one way we monitor our background process – which is different than how we monitor our web services. It’s difficult when you can’t send a request/poll the service. I wanted something more than checking if the process is alive. One solution we came up with is is using syslog with a log aggregration service.
Most log aggregration platforms (i.e “Splunk, Log Entries, Sumologic,”) can send an alert if some string (or regex) is found in the the logs. This is pretty common; what about alerting when logs are NOT found?
Basic Setup
Our process is managed by supervisor and it logs to stdout. Supervisor logging plugin sends logs to our centralized syslog collector, which forwards the logs to Sumologic. We then configure alerts based off of search results.
Configuring the alert
First, create your search (here’s a cheatsheet). Next, configure it:
- Library -> Edit Search
- Scheduled this Search
- Number of results Equal to = 0
Screenshot:
That’s it. You can integrate this with your Pager Duty account or just have it send directly to an e-mail address.
