Metrics from Custom Apps…Easy!

Many developers are looking for a platform to which they can send arbitrary data from their custom applications to collect and visualize those metrics, as well as alert on specific thresholds. With Circonus’s ability to accept and parse raw JSON, it’s easy to send metrics from custom applications into the system. More information on JSON parsing can be found here or in the User Docs, but the steps below will get you up and running quickly.

1.) The first step for sending JSON data to Circonus is to create an HTTPTrap check. Under the Checks page, click on “New Check +” in the upper right corner, then expand the JSON option and choose “Push (HTTPTrap)”. Select the HTTPTrap broker from the list, then set up the host and secret. Click on “Test Check”, then “Finish”, even though there are no metrics selected.

2.) Now that the check is created, find that check and go into the details. There will be a “Data Submission URL” listed, which is the URL to which you will PUT the data from your application. Once the data is being submitted at regular intervals (either as frequently as you have it, or every 30 seconds if it is a sample), you can go back into the check to enable the metrics. Alternatively, you can use the Check Bundle API to manage the metrics and enable any metrics that are present but disabled. You can also enable histogram collection using the same methods.

Once data is being collected, you can then start graphing, alerting, streaming to dashboards, and performing analytics on your data. Additionally, if you add other check types to your Circonus account, you can compare these custom metrics to other data to get the full picture of what is really going on at any given time.

JSON Over HTTP – Data Collection Made Simple

At Circonus, one of our goals is to try to make it as easy as possible to monitor your data. One of the ways we do this is to allow data formatted in JSON to be pushed or pulled over HTTP into Circonus. Since HTTP is spoken everywhere, and JSON is understood everywhere, this allows for easy metric submission so you can collect, store, graph, and analyze everything that you care about.

The HTTPTrap check type accepts JSON payloads via HTTP PUT requests. This allows you to push data from your devices or applications directly into Circonus. This is useful for data that happens sporadically, instead of at a regular or constant interval. HTTPTraps also let you send histogram data into Circonus, so you can see the whole picture instead of one aspect of your data.

The JSON check type gets data from an HTTP endpoint at the interval you select. This allows you to make applications that expose metrics in a JSON format that can be polled regularly from Circonus. These checks allow you to specify a username/password, port, and any additional headers, which gives you security and flexibility in what you allow to connect to your hosts.

One of the major shortcomings with JSON in most languages is the ability to deal with large numbers. Our parser works around that by allowing you to send the number as a string. This means there is no data that you’re interested in that we can’t collect or accept.

The ability to use JSON as a format for data also allows you to write your own data collector. For instance, Gollector was written by the folks at Triggit who wanted to have an agent that relied on the proc filesystem and C POSIX calls. Additionally, both Panoptimon (written in Ruby) and our very own nad agent (written in Node.js) utilize JSON to send system information. Customized agents like these allow you to adapt Circonus to your infrastructure and monitoring needs.

To show just how easy it is to format data so Circonus can read it, this is an example Python script that runs once per minute to generate some randomized data. Once you create an HTTPTrap check in Circonus, you can look at the check to get the URL that should be used in the PUT call. The example includes submitting strings, small numbers, large numbers, and a set of numbers that can be used for histogram data. Similar setups can be used in other languages and in your own custom applications.

import json
import urllib2
import time
import random

# Use the URL provided in the UI from the Circonus HTTPTrap check
httptrapurl = "https://trap.noit.circonus.net/module/httptrap/01234567-89ab-cdef-0123-456789abcdef/mys3cr3t"

while(1):
    # Make up the data
    data = {
            "number": random.uniform(1.0, 2.0),
            "test": "a text string",
            "bignum_as_string": "281474976710656", 
            "container": { "key1": random.randint(1200, 1300) },
            "array": [
                random.randint(1200, 1300),
                "string",
                { "crazy": "like a fox" }
            ],
            "testingtypedict": { "_type": "L", "_value": "12398234" },
            # Set the type to "n" for histogram-enabled data
            "histogramdata": { "_type": "n", "_value": [int(1000*random.betavariate(1,3)) for i in xrange(10000)] }
    }
    jsondata = json.dumps(data)

    # Form the PUT request
    requestHeaders = {"Accept": "application/json"}
    req = urllib2.Request(httptrapurl, jsondata, headers = requestHeaders)
    req.get_method = lambda: 'PUT'
    opener = urllib2.urlopen(req)
    putresponse = json.loads(opener.read())

    # Print the data we get back to the screen so we can make sure it's working
    print putresponse
    print jsondata
    print

    # Wait a minute
    time.sleep(60)

This will show up in Circonus as:

You can refer to the Circonus User Manual for more details about the HTTPtrap check. Also, please refer to the information there to import our certificate if you see the following error while following these instructions:
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

Ways to Collect Systems Data in Circonus

When you decide to monitor your systems with Circonus, there’s quite a few options on how to collect your metrics. We believe Circonus should be a tool that does what you need, when you need it. Circonus does not force you into a specific approach or method. Since there are so many different ways to gather telemetry via Circonus, we thought we would take a moment to outline some of the different approaches.

In addition to application-specific checks, you might like to get baseline information about things like memory, CPU, file systems, and interfaces of your servers and network equipment. We’ve listed out the main options that can be used for system performance metrics below, along with a brief description and our recommendations for each. We’ve roughly ordered these based on our best practices, but the tool that should be used depends on many variables that you’ll need to take into account.

For instance, some users may prefer to use a single agent on all of their devices, which may mean that some options won’t be available. Available plugins and ability to expand should also be considered. Some agents allow Circonus to reach out to the endpoint and gather metrics, while others require the data to be pushed (these agents mention push requirements in the description below). In some cases, the language that the agent was written in can have an effect on your decision.

Standard protocols

SNMP – SNMP is a standard that has been around for years, and allows monitoring of many types of network equipment, servers, and appliances. There is a good chance you already have SNMP configured on most of your hosts, which would significantly lower the up-front setup time. You’ll need to know the OIDs you want to monitor, but check bundle templates can make this process a little easier for you.

HTTPTrap – Circonus can accept JSON payloads via an HTTP PUT or POST request. This data is not polled regularly from the Circonus Broker, but is pushed to the Broker from the monitored target. This is the easiest way to get arbitrary data into Circonus, but you’ll have to figure out where to get the data.

Third-Party agents

collectd – Collectd is a lightweight C-based tool that has a variety of plugins available for data collection. There are 2 main ways to use collectd with Circonus, either to push the information from your device over UDP (similar to statsd and HTTP Traps) or via the write_http plugin.

Gollector – Gollector is a new monitoring agent that relies on the proc filesystem and C POSIX calls such as sysconf to determine your machine’s profile. This alleviates any performance penalty from shelling out the collection work that some other agents can have.

NRPE – Circonus can utilize existing NRPE checks from your Nagios or Icinga installation. NRPE allows you to remotely call Nagios scripts to collect information. If you want to monitor a non-standard metric, there’s probably a Nagios script for it.

statsd – Similar to an HTTP Trap, statsd allows your hosts to send information to Circonus Enterprise Brokers, rather than the Broker reaching out to the host to poll it. One downside is that this information cannot be played in real-time, but it can be useful for metrics that may not have regular intervals of available information or are particularly high volume.

Internally-developed agents

nad – Nad is a lightweight, simply managed host agent written in Node.js. Nad is the first choice of Circonus due to its easy extensibility and its ability to work on almost any platform, including Windows, RHEL, Ubuntu, and illumos derivatives. Nad comes with enough plugins to let you monitor any of the basics, while allowing you to add your own checks to fit your environment.

Resmon – Resmon is a Perl-based agent created by OmniTI. New modules can be created quickly and easily, but must be written in Perl; that’s a make it or break it factor for many.

Windows Agent – If you’d rather not use nad on your Windows servers, there is a Windows agent that can be used to collect performance metrics from Windows servers.

Which is right for me?

The choice of agent to use depends on many factors. Current operating system, existing monitoring setup, and network layout can all have an effect on which agent you choose. You may also need to incorporate several choices in order to best monitor your environment.

That covers the main ways to get system information into Circonus. There’s plenty of other methods of getting data, such as Google Analytics, a variety of database connections, Memcached, Varnish, NewRelic, and more. A combination of these collection types can enable you to have data on every piece of your infrastructure, so you can always find the information you need.