Graham’s Two Second History of Webhooks [non technical]

Well, I’m sure you all know what webhooks are but I only found out last month in preparation for another Helion OpenStack integration.

Here’s my simplified understanding of them – they’re “Push Notifications”.

With the advent of the intertubes we got very clever very fast and started using all these crazy udp and tcp protocols for M2M communication. As bandwidth was never an issue (I’m joking)  efficiency was never the first requirement.

Many monitoring solutions settled on http as the protocol of choice and simply polled the resources they were interested in and waited for a response. Monasca provides you with a RESTful API if you wish to use this approach.

However, with cloud comes the challenges of scale. And when you’re trying to poll hundreds of thousands of services, in seconds, efficiencies are very important. Along comes webhooks to save the day.

Unlike the ‘old-fashioned’ and inefficient polling which requires an http GET followed by and http POST even when there’s nothing to report, webhooks require a single http POST and they only trigger when there’s something worth POSTing home about!!!

So to summarise – it’s frequent HTTP GET/POSTs versus infrequent HTTP POSTs. There’s an urban myth floating about the net that when Github introduced Webhooks it reduced the load on their servers from users monitoring for new code changes by 66X . [ I don’t have any facts]

How do we set them up – simples, especially if you read my previous SMTP integration blog. Let’s do the GUI piece first – this could all be done from the CLi but that would require me reading more.

Note: The Operations Console GUI is specific to the Helion OpenStack distribution. If you’re not using Helion you will need to use the Monasca CLi.

Webhook Configuration in HOS Monasca

Log in to the Helion Ops Console as a cloud admin user

Select the Notification Methods option from the ‘hamburger’ menu on the top left hand side of the screen.

Create a new notification method, selecting Webhook and enter the details of the URL that you wish to http POST notifications to.

And that’s all there is to webhooks. But wait, ‘how do we test and verify this’ I hear you think…well read on.
Create a Test Alarm

Back to our ‘hamburger’ menu and this time select Alarm Creation and create a new alarm as follows – we want false alerts here – so I’ll use cpu.idle as the metric and set silly values –

Initially 2 Unknown Alarms may appear as this is the first time this alarm has run and it doesn’t have a previous state. However, be patient, in a few more minutes you should see the dashboard light up as shown below.

You can now select the alarms for more details.

And of course, what we’ve all been waiting for – webhook alerts!!!

Doh! How do we test webhook Alerts…

Many enterprise grade monitoring solutions still don’t support webhooks today – these guys just don’t get agile and wonder why the smaller fish over take them.

Once again I don’t recommend doing any of what you’re about to see on production systems – this is purely for testing and verifying webhooks. Also please note that I’ve opened a port on the HOS control-plane firewall to allow me to use my deployer node as the webhook server running on port 3001. [See the previous SMTP post if you wish to test this IN YOUR LAB]

Note: To all coders out there I apologies now for all rules I’m breaking, poor coding etc. be gentle I hacked these examples together thanks to all of your publishings 🙂

I have provided two example scripts below one for NodeJS and one for Python. Python because many openstack users will be familiar with it. NodeJS because I wanted an excuse to play with it – it’s nice! I’m not covering how to setup your python or node environments here there’s tons of blogs on these topics. Virtual Environment (virtualenv) is your friend if using python – use it.

Python Example

Copy the following python script to a file and save it as webhook.py . This script waits for  webhook notifications and when it receives them it writes them to a logfile. This could be used by legacy monitoring solutions to parse the alerts – it’s not enterprise grade though – just an example.

webhook.py

# Author: Graham Joseph Land
# Blog: allthingscloud.eu
# Twitter: @allthingsclowd
# Email: graham@the above domain
# Date: 24/03/2016
# 
# Name: webhook.py
# Version: 0.1
# Purpose: This script listens for webhook notifications and then writes them to a logfile
#
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import json
import logging
import logging.handlers

# log file for this script
log_file = '/home/stack/monitoring/venv/webhook/webhook.log'

log_max_size = 25165824         # 24 MB
log_level = logging.INFO
#log_level = logging.DEBUG      # DEBUG is quite verbose


log = logging.getLogger('log')
log.setLevel(log_level)
log_handler = logging.handlers.RotatingFileHandler(log_file,
                                                   maxBytes=log_max_size,
                                                   backupCount=4)
f = logging.Formatter("%(asctime)s %(filename)s %(levelname)s %(message)s",
                      "%B %d %H:%M:%S")
log_handler.setFormatter(f)
log.addHandler(log_handler)


class webhookReceiver(BaseHTTPRequestHandler):

    def do_POST(self):
        """
            receives post, handles it
        """
        log.info('<<<<<<<<< Alert Start >>>>>>>>')
        message = 'OK'
        self.rfile._sock.settimeout(5)
        data_string = self.rfile.read(int(self.headers['Content-Length']))
        self.send_response(200)
        self.send_header("Content-type", "text")
        self.send_header("Content-length", str(len(message)))
        self.end_headers()
        self.wfile.write(message)

        # parse data
        text = json.loads(data_string)
        text = json.dumps(text, indent=2)
        log.info(text)
        log.info('<<<<<<<<< Alert End >>>>>>>>')


    def log_message(self, formate, *args):
        """
            disable printing to stdout/stderr for every post
        """
        return


def main():
    """
        the main event.
    """
    try:
        server = HTTPServer(('', 3001), webhookReceiver)
        log.info('started web server...')
        server.serve_forever()
    except KeyboardInterrupt:
        log.info('ctrl-c pressed, shutting down.')
        server.socket.close()

if __name__ == '__main__':

Run the script and tail the logfile – now go back and generate the alerts again and see what happens..

And there you have it – webhooks – “simples”!

To rule out Monasca issues you could also test this script using a curl statement as follows:

curl -H "Content-Type: application/json" -X POST -d '{"alarm_id": "1234","alarm_definition_id": "freddef","alarm_name": "fire alarm","alarm_description": "alarmDescription","alarm_timestamp": "timestamp","state": "Critical","old_state": "oldState","message": "this is the message","tenant_id": "tenant1234","metrics": "1234"}' --trace-ascii debugdump.txt http://172.16.60.9:3000/openstack/alerts

NodeJS Example

Note that I’m using a different webhook endpoint than what was used in the python example.

There’s a slightly more complex file structure required for the NodeJS example application.

We use the main app server.js which then calls out to the ‘route’ app webhook.js  which will process the webhook upon receipt.

(venv)stack@helion-cp1-c0-m1-mgmt:~/monitoring/nodejs$ tree -I node_modules
.
|-- routers
|   `-- webhook.js
`-- server.js

Copy the two apps below into matching files named using the same directory structure shown above.

server.js

var bodyParser  = require( 'body-parser' ),
    express     = require( 'express' ),
    app         = express();

var webhook        = require( './routers/webhook' );

app.use( bodyParser.json() );
app.use( bodyParser.urlencoded({
  extended: true
}) );

app.get( '/', function ( req, res, next ) {
    res.send( 'Hooks listener running' );
});
app.use( '/openstack/alerts', webhook );

app.listen( 3001 );

and in the routers subdirectory

webhook.js

var express = require( 'express' ),
    bodyParser = require('body-parser'),
    router  = express.Router();

router.use(bodyParser.json());

router.post( '/', function ( req, res, next ) {
    console.log(req.body);
        
    res.status(200).json({message: 'Monasca Hook received!'})
});

module.exports  = router;

Launch the app and test with curl first –

stack@helion-cp1-c0-m1-mgmt:~$ curl -H "Content-Type: application/json" -X POST -d '{"alarm_id": "1234","alarm_definition_id": "freddef","alarm_name": "fire alarm","alarm_description": "alarmDescription","alarm_timestamp": "timestamp","state": "Critical","old_state": "oldState","message": "this is the message","tenant_id": "tenant1234","metrics": "1234"}' --trace-ascii debugdump.txt http://172.16.60.9:3001/openstack/alerts
{"message":"Monasca Hook received!"}stack@helion-cp1-c0-m1-mgmt:~$

The result should look like this

(venv)stack@helion-cp1-c0-m1-mgmt:~/monitoring/nodejs$ nodejs server.js
{ alarm_id: '1234',
  alarm_definition_id: 'freddef',
  alarm_name: 'fire alarm',
  alarm_description: 'alarmDescription',
  alarm_timestamp: 'timestamp',
  state: 'Critical',
  old_state: 'oldState',
  message: 'this is the message',
  tenant_id: 'tenant1234',
  metrics: '1234' }

And finally, send an alert from Monasca as detailed earlier to see the webhook notifications arriving and being processed by the app.

[Notes : Troubleshooting Notification Delivery]

• If you’re still not seeing any traffic now AND your /var/log/monasca/notification/notification.log file is empty try re-configuring monasca

cd ~/scratch/ansible/next/hos/ansible
ansible-playbook -i hosts/verb_hosts monasca-reconfigure.yml

• When reconfiguring existing alarm definitions ensure that you select BOTH the notifications checkbox AND the individual notification methods required.

• Use TCPDUMP to check if webhook traffic is leaving any of the three controller nodes interfaces [search on the webhook port number].