Graham’s Two Second History of Webhooks [non technical]
Well, I’m sure you all know what webhooks are but I only found out last month in preparation for another Helion OpenStack integration.
Here’s my simplified understanding of them – they’re “Push Notifications”.
With the advent of the intertubes we got very clever very fast and started using all these crazy udp and tcp protocols for M2M communication. As bandwidth was never an issue (I’m joking) efficiency was never the first requirement.
Many monitoring solutions settled on http as the protocol of choice and simply polled the resources they were interested in and waited for a response. Monasca provides you with a RESTful API if you wish to use this approach.
However, with cloud comes the challenges of scale. And when you’re trying to poll hundreds of thousands of services, in seconds, efficiencies are very important. Along comes webhooks to save the day.
Unlike the ‘old-fashioned’ and inefficient polling which requires an http GET followed by and http POST even when there’s nothing to report, webhooks require a single http POST and they only trigger when there’s something worth POSTing home about!!!
So to summarise – it’s frequent HTTP GET/POSTs versus infrequent HTTP POSTs. There’s an urban myth floating about the net that when Github introduced Webhooks it reduced the load on their servers from users monitoring for new code changes by 66X . [ I don’t have any facts]
How do we set them up – simples, especially if you read my previous SMTP integration blog. Let’s do the GUI piece first – this could all be done from the CLi but that would require me reading more.
Note: The Operations Console GUI is specific to the Helion OpenStack distribution. If you’re not using Helion you will need to use the Monasca CLi.
Webhook Configuration in HOS Monasca
Log in to the Helion Ops Console as a cloud admin user
Select the Notification Methods option from the ‘hamburger’ menu on the top left hand side of the screen.
Create a new notification method, selecting Webhook and enter the details of the URL that you wish to http POST notifications to.
And that’s all there is to webhooks. But wait, ‘how do we test and verify this’ I hear you think…well read on.
Create a Test Alarm
Back to our ‘hamburger’ menu and this time select Alarm Creation and create a new alarm as follows – we want false alerts here – so I’ll use cpu.idle as the metric and set silly values –
Initially 2 Unknown Alarms may appear as this is the first time this alarm has run and it doesn’t have a previous state. However, be patient, in a few more minutes you should see the dashboard light up as shown below.
You can now select the alarms for more details.
And of course, what we’ve all been waiting for – webhook alerts!!!
Doh! How do we test webhook Alerts…
Many enterprise grade monitoring solutions still don’t support webhooks today – these guys just don’t get agile and wonder why the smaller fish over take them.
Once again I don’t recommend doing any of what you’re about to see on production systems – this is purely for testing and verifying webhooks. Also please note that I’ve opened a port on the HOS control-plane firewall to allow me to use my deployer node as the webhook server running on port 3001. [See the previous SMTP post if you wish to test this IN YOUR LAB]
Note: To all coders out there I apologies now for all rules I’m breaking, poor coding etc. be gentle I hacked these examples together thanks to all of your publishings 🙂
I have provided two example scripts below one for NodeJS and one for Python. Python because many openstack users will be familiar with it. NodeJS because I wanted an excuse to play with it – it’s nice! I’m not covering how to setup your python or node environments here there’s tons of blogs on these topics. Virtual Environment (virtualenv) is your friend if using python – use it.
Copy the following python script to a file and save it as webhook.py . This script waits for webhook notifications and when it receives them it writes them to a logfile. This could be used by legacy monitoring solutions to parse the alerts – it’s not enterprise grade though – just an example.
Run the script and tail the logfile – now go back and generate the alerts again and see what happens..
And there you have it – webhooks – “simples”!
To rule out Monasca issues you could also test this script using a curl statement as follows:
Note that I’m using a different webhook endpoint than what was used in the python example.
There’s a slightly more complex file structure required for the NodeJS example application.
We use the main app server.js which then calls out to the ‘route’ app webhook.js which will process the webhook upon receipt.
Copy the two apps below into matching files named using the same directory structure shown above.
and in the routers subdirectory
Launch the app and test with curl first –
The result should look like this
And finally, send an alert from Monasca as detailed earlier to see the webhook notifications arriving and being processed by the app.
[Notes : Troubleshooting Notification Delivery]
- If you’re still not seeing any traffic now AND your /var/log/monasca/notification/notification.log file is empty try re-configuring monasca
- When reconfiguring existing alarm definitions ensure that you select BOTH the notifications checkbox AND the individual notification methods required.
- Use TCPDUMP to check if webhook traffic is leaving any of the three controller nodes interfaces [search on the webhook port number].