Recently I  had a request to configure the SMTP integraton with Monasca as part of the installation.  Having already configured the SMTP server settings in the cloudConfig.yml file during the model building phase it should have been a simple case of configuring the notification details in the Operations Console or via the Monasca CLi. This didn’t work however, so here’s the steps I took to debug the SMTP integration:

Verify SMTP Server Communication

From each of the controllers run a simple curl email test to ensure the controllers can communicate with the email server as follows:

echo "Hello Helion OpenStack World" > mail.txt

curl --url "smtp://172.16.60.9:3050" --mail-from "graham@allthingscloud.eu" --mail-rcpt "ops@alert.com" --upload-file mail.txt --user "graham@allthingscloud.eu:password"

[Note: Ensure to amend the SMTP server and IP to match your requirements]

If this fails then the issue exists between the controller nodes and the SMTP server – verify that  the SMTP details are correct,no firewalls blocking the ports etc. Check with the SMTP admin if they’re dropping the mails etc.

TCPDUMP and NETCAT (nc) are very useful utilities to verify this fundamental connectivity. Previous post here and all over the intertubes explain their use if required.

Use a Temporary ‘pop-up’ SMTP Server

If you don’t have an SMTP server or you wish to rule out you main SMTP server you could also follows these steps to quickly setup a testing SMTP server – you’ll see from the non standard ports I used above this is what I did initially.

I actually put my pop-up SMTP server on the deployer node [in a lab scenario – don’t do this in production] so I needed to open a port on the control-plane firewall to allow this the server to listen on port 3050 as follows:

vi /home/stack/helion/my_cloud/definition/data/firewall_rules.yml
.
.
.
.
   - name: TestWebHook
      network-groups:
      - MANAGEMENT
      rules:
      - type: allow
        remote-ip-prefix:  0.0.0.0/0
        port-range-min: 3000
        port-range-max: 3100
        protocol: tcp

So I’ve opened a range of ports here as I’ve other plans but you could simply open the single port required.

Once you’ve made these changes to the firewall cloud model file it’s the usual HLM process

cd ~/helion/hos/ansible
git add -A
git commit -m "firewall rule update"
cd ~/helion/hos/ansible
ansible-playbook -i hosts/localhost config-processor-run.yml
cd ~/helion/hos/ansible
ansible-playbook -i hosts/localhost ready-deployment.yml

cd ~/scratch/ansible/next/hos/ansible
ansible-playbook -i hosts/verb_hosts osconfig-iptables-deploy.yml

Now copy the following file to the deployer node and save it as smtp.py Replace the ip address of the server and port you wish to use in the file.

import smtpd
import asyncore

class CustomSMTPServer(smtpd.SMTPServer):
    
    def process_message(self, peer, mailfrom, rcpttos, data):
        print 'Receiving message from:', peer
        print 'Message addressed from:', mailfrom
        print 'Message addressed to  :', rcpttos
        print 'Message length        :', len(data)
        print 'Message               :', data
        return

server = CustomSMTPServer(('172.16.60.9', 3050), None)

asyncore.loop()

Open a screen session (you may need to sudo apt-get install screen)
and run the script as follows:

screen -h 10000
sudo python smtp.py

Now you have your very own SMTP server to test the integration against.

With SMTP all verified we now need to configure Monasca.

Monasca Configuration

Log in to the Helion Ops Console as a cloud admin user

Select the Notification Methods option from the ‘hamburger’ menu on the top left hand side of the screen.

Enter your the email address where you would like the emails to be send by creating a new email entry.

Now let’s create a test alert to see if we’re getting SMTP traffic.

Create a Test Alarm

Back to our ‘hamburger’ menu and this time select Alarm Creation and create a new alarm as follows – we want false alerts here – so I’ll use cpu.idle as the metric and set silly values –

Initially 2 Unknown Alarms may appear as this is the first time this alarm has run and it doesn’t have a previous state. However, be patient, in a few more minutes you should see the dashboard light up as shown below.

You can now select the alarms for more details.

And of course, what we’ve all been waiting for – email alerts!!!

[Notes : Troubleshooting Gotchas]

• If you’re still not seeing any traffic now AND your /var/log/monasca/notification/notification.log file is empty try re-configuring monasca

cd ~/scratch/ansible/next/hos/ansible
ansible-playbook -i hosts/verb_hosts monasca-reconfigure.yml

• If you have reused the Default email and the logfiles are complaining about an invalid ’email from’ address or the SMTP server requires a specific sender modify the default sender set in the /home/stack/helion/hos/ansible/roles/monasca-notification/defaults/main.yml . Ensure to do the full git commit…monasca-reconfigure cycle  -identified above

• When reconfiguring existing alarm definitions ensure that you select BOTH the notifications checkbox AND the individual notification methods required.