The Circonus API and Raspberry PI

Building a Raspberry PI Circonus AlertBot

To a Site Reliability Engineer, one of the most important things is making sure that you get alerts as soon as events happen. Circonus provides many contact options, such as SMS, email, Slack, PagerDuty, or VictorOps. For me, if I’m not actively watching for alerts,I rely on my phone to do something to get my attention. On occasion, I may bump the mute switch on my iPhone, or the battery might run out, etc, etc. It’s rare, but it happens.

Folks that know me also know that I get a kick out of the Raspberry Pi. They run linux, they’re cheap, and they provide all those nice GPIO pins. So it was a good practical exercise to have a Raspberry Pi do some work for me. I decided to build a back up alerting tool for those times when I don’t see the alerts on my phone.

First I attached a pair of LEDs to the GPIO (one red, one green), along with appropriate resistors, one to GPIO pin 7, and one to pin 11. Then I attached a super obnoxious buzzer with a resistor to GPIO pin 24, and a momentary switch (button) to GPIO pin 26 along with a pullup resistor.

Then I wrote up a script in Perl that will poll the Circonus API for any sev1 alerts. The green light means no alerts, and the red light means something broke. And since I’m usually sleeping when it goes off, the buzzer accompanies the red light. This is as annoying as intended, and therefore the momentary switch will signal to the script to keep quiet.

I have a few improvements planned for this set up. My next task is to implement a cheap OLED display to provide some information about an alert. I’ve also thought about replacing the LEDs with a relay that could switch on something more robust, like a lamp or a motor, which might be useful in an office setting.

I’ll admit the script leaves room for a lot of improvement, and pull requests are welcome! Either way, pairing the Raspberry Pi with the Circonus API has been fun and productive. I imagine there are plenty other applications that could make use the two! Join the Circonus Labs slack community and you can share and discuss your ideas with our engineers and other Circonus users.

Discovering circonusvi

Folks who know Circonus and use it regularly for operations also know that its API is an important part of efficient management for your monitoring facility.

For example, after building out a new datacenter with our SaaS application, we wanted to apply tagging to our checks to make searching more effective. The UI is easy to use, but I needed to tag batches of checks rather than one at a time, which is a job for the API.

I could also write a small program to do a search and populate tags. That’s when a co-worker suggested I use circonusvi (https://github.com/circonus-labs/circonusvi).

Circonusvi is a neat little tool contributed by Ben Chapman to the Circonus Labs github repo. It’s a natural tool to use for most folks who work with unix or unix-like platforms. Blend that with the JSON input/output of the Circonus API and you have a quick way to make adhoc changes.

So after installing the python requirements for circonusvi, I generated a normal API token from the browser, ran circonusvi once, and validated the token in the browser user interface.

My first run of circonusvi without arguments returned everything, allowing me look over things and understand the JSON structure.

Now for the business.

This returns a list of common servers JSON output that I can now edit in vi:

./circonusvi.py 'display_name=servers([0-9]).foo.net json:nad'

And this example finds all the empty tags and populates it with something useful:

%s/\"tags\"\:\ \[\]/\"tags\":\ [\"component:http\",\"datacenter:ohio\",\"os:plan9\"]/g

After saving the changes and verifying the results, circonusvi prompts you one last time about updating the server. Then it updates and you’re done!