Simple Distributed Process and System Monitoring

View project onGitHub


Watchy is a distributed system for Application and Server Monitoring, I've always found most solutions be very complicated and hard to manage. Making applications watch process id's then they stop and the pid has updated and you start the whole thing over again.

Watchy has a distributed architecture and communication between the daemon on each server to the dashboard is over udp, meaning if something goes down and comes back up again it just simply doesn't matter about handling connection issues. It seems to be working very well in my tests at work and i am sure you will love it.

alt text

How it works

Watchy has a very application/event driven model and it can be seen in this diagram:

alt text

This can be all scripted or manipulatied by Python or C/C++ but first we need to start the server:

$ /usr/local/bin/ --help
Usage: [options]

  -h, --help            show this help message and exit
  -v, --version         Print version
  -c CONFIG, --config=CONFIG
                        Config file location
  -F, --fork            Fork as daemon

You can run this server via:

$ /usr/local/bin/ -c /etc/watchy/example-watchy.cfg 
WATCHY INFO - Starting StatsAggregator on
WATCHY INFO - Starting Async Backend handler
WATCHY INFO - WSGIServer:[gevent] starting

Now we can start posting stuff to watchy:

>>> import pywatchy
# hostname and port of watchy server
>>> daemon = pywatchy.WatchyDaemon ('localhost', 7878) 
# Creates or attaches to existing daemon on the server                                                                                                                                                              
>>> daemon.watchHost ('hostname')
# tell the daemon to watch the host server for statistics and post under specified key ('hostname')

You can even do more stop and start watching pid's or even post logs:

>>> daemon.watchPid ('watchyserver', 14293) # watch the python web app
>>> daemon.postMessage ('test', 'Hello World')
>>> daemon.stopWatchPid (14293) # stop watching in

This is pretty fun its all realtime and dynamic, but probably not helpful for existing setups so it comes with more tools for example if i want to watch syslog on a server i can do:

$ tail -f /var/log/syslog

To post this to watchy i can do:

$ # hostname and port of watchy server, this will attach or create the daemon as expected
$ tail -f /var/log/syslog | /opt/watchy/bin/wtail -k syslog -b localhost -p 7878

alt text

And if i want to watch some pids from the command line i can do:

$ /opt/watchy/bin/watcher -b localhost -p 7878 key:pid key:pid...
# if you specify -k <keyname> this will then do: watchHost (keyname)

This can even integrate with C/C++ applications:

#include <watchy/watchy.h>

static int watchy_fd;

// super enterprise logging library
int mylogginginit (...)
    int ret = watchy_cAttachRuntime (WTCY_DEFAULT_FIFO, "localhost", 7878, &watchy_fd);
    if (ret != WTCY_NO_ERROR) {
        fprintf (stderr, "Watchy failed [%s]\n", watchy_strerror (ret));
        return -1;
   /// done...

void mylogwrapper (int level, const char * fmt, ...)
    char buffer [BUF_SIZE];
    vsnprintf (buffer, sizeof (buffer)...);
    // log the message

    struct watchy_data data;
    memset (&data, 0, sizeof (data));
    // buffer expects to be null terminated
    watchy_logPacket (&data, buffer);
    watchy_writePacket (&data, watchy_fd);

But whats interesting is you can then Federate all Watchy Servers onto a common backend such as Ganglia or MongoDB.:

alt text

To put stats to a backend such as mongo simply edit the configuration file to:

backends = mongo

type = mongodb
uri = mongodb://localhost:27017

This is great because you can then reuse your existing infrastructure. Watchy comes with a realtime web dashboard and you can view realtime graphs:

alt text