Quantifying WordPress Performance Improvements with circonus-logwatch

Deriving meaningful insights from third-party logs has always been a difficult yet necessary task. Most analysis occurs after-the-fact, when something has gone wrong. Very few tools allow real-time monitoring of logs, so SREs have become accustomed to backfilling log data into various analysis tools. Postmortem log analysis is the de facto standard, yet should it be? Why shouldn’t you be able to monitor your server logs in real-time?

As an operator of large-scale systems, we’ve often felt the pain of postmortem analysis. These experiences have led us to develop circonus-logwatch.

Flexible configuration allows you to easily pinpoint the values you want to extract and transform into metrics, changing the nature of log analysis from “we’ll do better next time” to “let’s look at what’s happening right now.” And best of all, metrics extracted with circonus-logwatch are first-class citizens in Circonus, so you can analyze and alert on them just like you would any other metric.

circonus-logwatch allows you to:

  • Track precise values from third-party logs as first-class Circonus metrics
  • Perform analytics and predictions on those values
  • Set rules and be alerted on those values
  • Store your new log-sourced metrics using the same multi-year data retention policies as the rest of your collected data

Setting Up circonus-logwatch

circonus-logwatch can be installed on most Unix-based servers and is designed to be configurable, making it easy to fit into your existing stack. It supports multiple configuration file formats, and can submit its metrics via StatsD, through a local Circonus Agent, or directly to a Circonus broker.

After installing circonus-logwatch, setup is simple: first add a main config file to define some basic settings and to specify how the metrics will be submitted (StatsD, agent, or broker). Then set up a config file for each third-party log source you want to monitor. You can monitor as many different third-party logs as you wish; simply create a new config file for each log source and place it in the log config directory. Each of these logs will be tailed by circonus-logwatch, which will process the lines as soon as the logging system writes them to the file. When a new line is written to a watched file, circonus-logwatch will take the regular expressions from the config and apply them to the new line, extracting the metric(s) from it and formatting them for submission to Circonus. If you’re collecting these new metrics via a Circonus Agent, then the new values will be submitted to Circonus every sixty seconds. If you’re collecting via direct broker submission, they will be sent every ten seconds, and if you’re collecting via StatsD they will be sent immediately. (Please see the repository for full installation instructions.)

circonus-logwatch comes with many sample log source configs, such as:

  • audit.yaml
    This example allows you to monitor system login and logout events; for instance, you could monitor successful logins compared to login attempts.
  • messages.json
    This example tracks transmit timeouts on ethernet interfaces, so you can monitor potential networking problems from that system’s perspective.
  • ssh.json
    This example monitors when users make SSH connections to that system and when they run sudo commands (for security purposes).
  • system.yaml
    This is a contrived example, tracking macOS system events; it is part of the auth system, when an app uses the user’s credentials to “unlock the user” (potentially accessing/modifying something that maybe the app shouldn’t).
  • yum.yaml
    This example allows you to track Yum package activity on the system…when packages are installed, updated, or erased.

Example Use Case: Monitoring Apache Request Latency

Let’s look at a real-world example: a web-facing system with some number of pages. We want to monitor how long it takes Apache to handle page requests such that we have a good handle on how our web server is performing.

First, we need to get the relevant metrics into Circonus. Apache has the associated information readily available, we just need to expose it by specifying the following LogFormat:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" tm:%D" combined

This is a sample of what the resulting logs should look like:

174.223.8.145 - - [18/Sep/2018:18:45:57 +0000] "GET / HTTP/2.0" 200 3266 "-" "Mozilla/5.0 (Linux; Android 7.1.1; Z839 Build/NMF26V; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/68.0.3440.85 Mobile Safari/537.36" tm:87902
104.225.9.155 - - [18/Sep/2018:18:46:00 +0000] "GET / HTTP/1.1" 200 9797 "-" "libmtev/0" tm:12651
97.105.47.162 - - [18/Sep/2018:18:47:47 +0000] "GET / HTTP/2.0" 200 9797 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36" tm:12046

Next, we install circonus-logwatch on a system with access to these logs, and configure it to submit our metrics to Circonus. To extract the relevant metrics from this format, we use the sample Apache log config below:

log_file: /var/log/httpd/access_log
metrics:
# since latency is not included in common log format, it was added to 'combined' format:
# e.g. `LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" tm:%D" combined`

# aggregate latency histogram
- match: 'tm:(?P<value>[0-9.]+)'
name: latency_µs
type: h
# latency histogram by request path
- match: ' (?P<path>/[^ ]*) HTTP.+tm:(?P<value>[0-9.]+)'
name: '{{.path}}`latency_µs'
type: h
# latency histogram by request path and specific request methods
- match: '(?P<method>(GET|POST|PUT)) (?P<path>/[^ ]*) HTTP.+tm:(?P<value>[0-9.]+)'
name: '{{.path}}`{{.method}}`latency_µs'
type: h

This configuration provides us with several different metrics. First, it aggregates the latency across all requests into a single latency_µs metric. Second, it collects a separate metric for each request path, allowing you to track latency for each individual path. Third, it collects a metric for each request method on each path, allowing us to differentiate latency between GET and PUT requests.

Now when we go into Circonus, we have many metrics with names like /my/path`latency_µs.

We continue by picking the metrics corresponding to the requests we want to monitor; for this example let’s say there are twenty. For each request path, we want to know if its latency goes above a half second (500,000 µs) and stays there for some extended period of time. To do so, we create a ruleset for each of those twenty metrics declaring “If the value is present and higher than 500000, trigger a sev 5 alert and wait 10 mins before notifying.” Very straightforward.

We’re not quite done, however. We’re interested in the bigger picture. To comprehensively monitor these requests we will now create a ruleset group. A ruleset group allows us to alert on all requests, as opposed to requests for a specific path. We add all twenty rulesets as members of the group and then add a formula with a threshold of “5 of 20.”

Finally, we can add a contact to the ruleset group, and here’s where the power of circonus-logwatch shines: we would be alerted when 25% of our watched request paths’ latencies ever rise above half a second and stay there for ten minutes. And all this is happening in real-time, based solely on our Apache logs.

circonus-logwatch ruleset group
Our ruleset group would look roughly like this

Real-World Benefits From Log Monitoring

We installed circonus-logwatch on the host serving irondb.io and configured it to watch our request handling latency. Upon doing so, we saw latencies between 250 – 350 ms per request. That’s not horrible, but we expected better. Since the site is run on WordPress, we installed the Super Cache plugin to generate and serve static pages, hoping that would improve our page request latency. As you can see in the histogram graph below, our request handling dropped to a respectable 10 – 20 ms per request after caching was enabled.

WordPress Request Handling Latency, Before and After Super Cache
WordPress Request Handling Latency, Before and After Super Cache

Start Making Better Use of Your Logs

There are many useful metrics hidden hidden in your organization’s logs. These logs contain rich information that can help you make better business decisions, right now, in real-time. Best of all, this capability comes free with any Circonus account.

circonus-logwatch is open source and is available at https://github.com/circonus-labs/circonus-logwatch.

A New Day for Navigation and Search

Today we finally rolled out the new navigation menu and search bar that we’ve been working on for a while. We had been getting feedback on the poor usability of the old horizontal menu system and knew that a “pain point” had been reached—it was time to revisit how we treated navigation and search in Circonus.

We heard numerous times from users that our old navigation menus were difficult to use, and a recent survey we performed simply underscored that feedback. The horizontal nature of the menus made them tricky to navigate, especially when combined with the fact that they were not very tall. Also, we had outgrown them; after the recent addition of Metric Clusters and Blueprints, we were feeling cramped and were running out of room in the menu system. The last problem (which we started hearing recently from users) is that the location of the search field made it seem like a global search despite the placeholder hint text it contained. Some users who were new to Circonus hadn’t even noticed the search field; it just blended into the interface too well.

In this redesign we’ve shifted paradigms dramatically to alleviate these three problems. We’ve done away with the notion of showing all the menu all the time, and have implemented a large “sitemap” style menu. When the menu is collapsed, you see the current section name and page title beside a hamburger menu icon. This offers a large trigger area and easy-to-use menu with very few “moving parts.” The menu appears when hovering anywhere over the trigger area, making clicking unnecessary (clicking does work, however, for tablet and other touch-based users). This offers plenty of room both horizontally and vertically for future expansion, and it frees up room to the right for more page-related buttons.

our newly redesigned navigation menu

On pages which are searchable, the search bar now sits immediately beside the menu trigger area (containing the page section and title). This makes it easier for users to recognize the contextual nature of the search, and also increases the visibility of search in general. This new search bar provides a dedicated space to show any current search string in operation on the page, and also offers a “minus” button to clear it with a single click. To enter a search string or edit an existing search string, you can click the magnifying glass or click the existing search string, if present. To commit your search string after typing, simply hit enter on your keyboard.

You’ll also notice that we’ve slightly reorganized the menu structure. The main goal of this was to make things more logical; to provide a better model upon which users can base their own mental models, making it easier to navigate Circonus. As such, the sections have been renamed with verbs pertaining to the general tasks related to each section. First is “Collect,” where you’ll find pages related to collecting and organizing data with checks, metrics, metric clusters, templates, and beacons. Next is “Monitor,” where you’ll go to see your hosts’ statuses, set rules, follow up on alerts, and work with contact groups and maintenance windows. Last is “Visualize.” This is where you work with graphs, worksheets, events, and dashboards. Hopefully this will make it easier for new users to get acquainted with the Circonus workflow of collecting data, setting rules to monitor that data, and working with visualizations.

One last benefit of this new menu design is that we now have the opportunity to highlight some secondary links at the bottom of the menu (documentation links, mobile site and changelog links, as well as keyboard shortcuts help). These have been present in the site footer, but many users are unaware of their existence. We wanted to pull some of these links up into a more prominent position since they’re helpful for users.

Thank you to all of our users whose feedback helps us shape Circonus into a better and more useful tool. We couldn’t do this without you!

Tags: A Long Time Coming

Ok, we know a lot of you have been asking for tags in Circonus for a long time. Well, they’re finally here! The tags feature is currently in beta, and will be released to all customers very soon. (Tags have actually been available to API users for a while, just without UI support in the application.) Let’s jump right in and I’ll give you a quick overview of how tags will work in Circonus.

First Things First: What’s Taggable?

For this initial implementation of tags, you can tag Graphs, Worksheets, Check Bundles, Templates, and Maintenance Windows. You will also see tags on some other pages, such as Alerts, Rulesets, Hosts, and Metrics, but these items aren’t taggable. The tags you see on those pages are inherited from the associated Check Bundles.

In the near future, we’ll be adding to these lists. We are planning on making Annotations and Dashboards taggable, and have some other unique ways we’re planning on using tags to help categorize and group items in the UI.

So, How Does This Work?

First, you’ll need to add tags to some items. All tags are categorized in the system, but if you don’t want to categorize your tags, that’s ok. Simply use the “uncategorized” category for all your tags and the tags will be presented solo (without category) throughout the UI. We have a couple of categories which are already created in the system, but you can create as many custom categories as you wish.

Let’s go to one of the list pages containing taggable items (e.g. the Checks list page) and look for the tags toolbar under an item (it will have an outlined tag with a plus icon). Click the “+” tag to open the “Add Tag” dialog. First choose a category or use the “+ ADD Category” option to enter a new one, then the tags dropdown will be populated with the tags under that category. Choose a tag or enter a new one by choosing the “+ ADD Tag” option, then use the “Add Tag +” button to add the tag to the item.


When the tag is added to the UI, you’ll notice right away that each tag category has its own color. There is a limited set of pre-selected colors which will be assigned to categories as they are created. These particular colors have been chosen to maximize your eye’s ability to distinguish the categories at a glance, and also because they work well under both light and dark icons. So you’ll also notice that the tag you added has its own icon. There’s a set of twelve icons which will be assigned to the first twelve tags in each category. Once a category has twelve tags, any further tags added to that category will receive blank icons. This system of colors and tags will create fairly unique combinations that should help you recognize tags at a glance without needing to read the tag every time. Note: taggable items can have unlimited tags.

After you add a tag to an item, you’ll also notice that a small set of summary tags is added (usually off to the right of the item). This shows the first few tags on the item, providing a way for you to quickly scan down the page and get a glimpse of the tags that are assigned to each item on the page.


One more note about tags and categories. Although you select them separately in the UI, when using the API the categories and tags are joined with a colon (“:”) as the separator. So a tag “windows” in the category of “os” would be represented as “os:windows” in the API.

Tag Filtering

The power of tags is apparent once you start using tag filters. Look in the upper right corner of the page and you’ll see an outlined tag with a funnel icon and beside it a similar menu button. These are for setting tag filters and saving tag filter sets for easy application later. Click the funnel tag to open the “Tag Filters” dialog, and click the “Add +” button to add a filter to the dialog. You may add as many filters as you wish, and in each one all you have to do is choose a category and tag from the choices (you may not enter new tags or categories here; these are simply the tags you’ve already added to the system). Use the “x” buttons at the right to remove filters, or use the “Clear” button to remove all filters and start with a clean slate. Note: none of your changes in this dialog are applied until you click the “Apply” button. After clicking “Apply,” the page will refresh and you’ll see your newly applied filters at the top of the page. You can then use the “Tag Filters” dialog to change these filters, or you can use the menu button on the right to open the “Tag Filter Sets” dialog, where you may save & apply sets of tag filters for easy switching.


One important feature to note is the “sticky” checkbox that appears when you have one or more tag filters applied. By default (with “sticky” turned off), the tag filters you apply are only visible in the current tab. If you close the tab or open a new one, it will not retain the current tag filters. The benefit of this is that we’ve developed a system to allow you to have multiple concurrent tag filter views open side-by-side. So with the “sticky” setting off, you can open several tabs, use different tag filters in all of them, and each tab will retain its own tag filters as you navigate Circonus in that tab. If at any point you turn the “sticky” setting on, the tag filters from that tab will be applied universally and will override all the other tabs. And not only are “sticky” tag filters applied across all tabs, they’re remembered across all of your user sessions, so they will remain applied until you choose to change or remove them.


Host Grouping

One unique feature we’ve already completed is Host Grouping. Head on over to the Hosts page and open the “Layout Options” by clicking on the grid icon at the right side of the page. You’ll see a new option labeled “Group By Tag Category.” If you choose a tag category there, the page will reorganize itself. You’ll now see a subtitle for each tag in the selected category, and under each subtitle you’ll see the Hosts which have Check Bundles with that tag. Because each Host can have many tags, including more than one tag in the same category, you may see a Host appear in more than one group. At the bottom of the page you’ll also see a grouping subtitled “Not in Category.” Under this group you’ll see all the Hosts which don’t have any Check Bundles with tags in the chosen category.


On the Move: Circonus Mobile

It’s been a long time coming, but it’s finally here: a Circonus that’s optimized for low-resolution devices like your smartphone or tablet! For the past few weeks, eagle-eyed Circonus customers may have noticed links to the mobile site creeping into various places, such as the login page (login.circonus.com) and the application footer. Although we officially “flipped the switch” a few weeks ago to enable the new Circonus Mobile experience, it is definitely a work in progress. To visit Circonus Mobile, visit m.circonus.com or use the “Switch to Mobile Site” link in the main site footer.

The most common need when using Circonus on a smartphone is being able to easily and quickly respond to alerts, so we made alerting functionality a priority when developing this new interface. The Current Alerts and Alert History pages are complete, already having feature parity with the main site. Also fully implemented are the Account Profile and Account Brokers pages, along with the User Profile and User Preferences pages, but most other pages are being implemented in a phased manner, beginning with read-only versions. TIP: when viewing a paginated list of items in Circonus Mobile, you may swipe left and right to page through the list (if you’re using a browser which supports touch events).

Our long-term goal for Circonus Mobile is not only achieving feature parity with the main site, but going further with push notifications and other native app functionality. But to give you a peek into our short-term goals, we’re currently working on making the Maintenance page fully functional, and after that we will move on to read-only versions of the Graphs and Worksheets pages.

If you use Circonus Mobile and don’t see a feature from the main site that you want, please let us know! We’re continually implementing features on the mobile site, but we’re more than willing to rearrange our implementation roadmap based on our customers’ needs.

Dashboards: Redux (or What to Look for in a Performance Monitoring Dashboard)

Last autumn we launched our customizable dashboards for Circonus, and we happen to think they’re pretty sweet. In this post, I’m not going to get into specifics about our dashboards (for more on that, you can check out my previous post, “One Dashboard to Rule Them All”), but instead I’ll talk more generally about what you should look for in your performance monitoring dashboard of choice.

Your dashboard shouldn’t limit its focus to technical data; it should help you track what really matters: business success.

A lot of data analysis done today is technical analysis for technical benefit. But the real value comes when we are able to take this expertise and start empowering better business decisions. As such, a performance monitoring dashboard which is solely focused on networks, systems, and applications is limiting because it doesn’t address what really matters: business.

While your purpose for monitoring may be to make your company’s web business operate smoothly, you can influence your overall business through what you operate and control, including releases, performance, stability, computing resources, networking, and availability. Thus, your dashboard should be designed to enable this kind of cross-pollination. By understanding which of your business metrics are critical to your success, you will be able to effectively use a dashboard to monitor those elements that are vital to your business.

Your dashboard should be able to handle multiple data sources.

There are many technologies in use across the web today. Chances are good that you have many different data sources across your business, so you need a dashboard that can handle them. It?s no good for a dashboard to only be able to gather part of your business data, because you’ll be viewing an incomplete picture. You need a dashboard that can handle all of your data sources, preferably on a system that’s under active development—continuing to integrate the best new technologies coming down the pike.

Your dashboard should provide access to real-time data.

The value of real-time data should not be underestimated; having real-time data capabilities on your dashboard is critical. Rather than requiring you to hit the refresh button, it should use real-time data to show you what is going on right now. Having this up-to-date picture makes your analysis of the data more valuable because it’s based on what’s happening in the moment. Some sectors already embracing this type of real-time analysis include finance, stock trading, and high-frequency trading.

Your dashboard should provide visualizations to match different types of data.

Your dashboard should provide different visualizations, because the visualization method you choose should fit the data you’re displaying. It’s easy to gravitate towards the slickest, shiniest visualizations, but they don’t always make the most sense for your data.

One popular visualization design is the rotary needle (dial) gauge. Gauges look cool, but they can be misleading if you don’t know their limits. Also, because of their opaque nature, the picture they give you of the current state is without context. Gauges can be great for monitoring certain data like percentages, temperature, power per rack, or bandwidth per uplink, but visualizations like graphs are generally better because they can give you context and history in a compact space. Graphs not only show you what’s going on now but also what happened before, and they allow you to project historic data (e.g. last day/week) alongside current data or place multiple datasets side-by-side so you can compare trends.

It’s also easy to forget that sometimes you may not need a visualization at all. Some data is most easily understood in text form (perhaps formatted in a table). Your dashboard should provide different ways of viewing data so you can choose the best method for your particular data sets.

Your dashboard’s interface shouldn’t be over-designed.

Designers tend to show off their design chops by creating slick, shiny user interfaces for their dashboards, but these are frequently just eye-candy and can get in the way of “scannability.” You need to be able to understand your dashboard at a glance, so design should stay away from being too graphics-heavy and should not have too much information crammed into tiny spaces. These lead to visual clutter and make you have to “work too hard” whenever you try to read your dashboard. The design should help you focus on your data, not the interface.

Everybody’s idea of a “perfect dashboard” will vary somewhat, but by following these guidelines you will be well on your way to selecting a dashboard that lets you monitor your data however you want. Remember, the goal is informed, data-driven decision-making, and it’s not unreachable.

Graph Annotations and Events

This feature has been a long time in coming: the ability to annotate your graphs! With the new annotations timeline sitting over the graph, not only can you create custom events to mark points in time, but you can also view alerts and see how they fit (or don’t fit) your metric data.

Annotations Timeline


First, let’s go to a graph and take a look at the annotations timeline to see how it works. When you choose a graph and view it, you will immediately see the new Annotation controls to the left side of the date tools, and the timeline itself will render in between the date tools and the graph itself. The timeline defaults to collapsed mode and by default will only show alerts from metrics on the current graph, so you may have an empty timeline at first. If you take a look at the controls, however, you will see three items: the Annotation menu, the show/hide toggle button, and the expand/collapse toggle button. The show/hide button does just what it says: it shows or hides the timeline. The expand/collapse button toggles between the space-saving collapsed timeline view and the more informative expanded timeline view.

If you open the Annotation menu, you will see a list of all the items you can possibly show in your timeline (or hide from it). Any selections you make here (as well as your show/hide and expand/collapse state changes) will be saved as site-wide user preferences in your current browser. All the items are separated into three groups:

Event Categories

This is a list of all the Event categories under the current account (these are seen and managed in the Events section of the site?we’ll get to that new section in a minute). If you have uncategorized events (due to deleting a category that was still in use), they will appear grouped under the “–” pseudo-category label.

Alerts

By default, the only alerts that will be shown will be alerts of all severity (sev) levels triggered by metrics on the current graph. If you wish, you may also show all alerts, and both categories of alerts may be filtered by sev levels. To do so, click one of the alert labels to expand a sev filter row with more checkboxes.

Text Metrics

This third group is not shown by default, but is represented by the checkbox at the bottom labeled “Include text metrics.” If you check this box, the page will refresh, and any text metrics on the current graph will then be rendered as a part of the timeline (and will be excluded from the graph plot and legend).

Once you have some annotations rendering on the timeline, take a look at the timeline itself. Hovering over a point will show a detail tooltip with the annotation title, date, and description, and hovering over either a point or a line segment will highlight the corresponding date range on the graph itself.

Now for the question on everyone’s minds: “Can I create events here, or do I have to go to the Events section to do that?” The answer is, yes, you can create events straight from the view graph page! To do so, simply use your right mouse button to drag-select a time range on the graph itself. A dialog will then popup for you to input your info and create the event.

Events Section

Now let’s head over to the Events section where you can manage your events and event categories. Simply click on the new Events tab (below the Graphs tab) and you’re there! To create an event, click the standard “+” tab at the upper left of the page. This will give you the New Event dialog. Most of the dialog inputs are pretty straightforward, with the exception of the category dropdown. This is a new hybrid “editable” dropdown input.


You may select any of its options if you’d like, or you can add new ones. To add a new option, simply select the last option (it’s labeled “+ ADD Category”). Your cursor will immediately be placed in a standard text input where you can enter your new category. When you’re finished, hit enter to create the new option and have it selected as your category of choice.

After you have created your event, you may need to edit it later. To edit any of its details, simply click on the pertinent detail of the event (when changing the event category, you will see it also has the new hybrid “editable” dropdown input which works exactly like the one in the New Event dialog).

In addition to start and end points (which may be the same date if you don’t want more than a single point), you may also add midpoints to your event. Click the Show details button for an event (the arrow button at the right end of an event row), and you will see the Midpoints list taking up the right half of the event details panel. Simply click the Add Midpoint button to get the New Midpoint dialog where you enter a title, description and choose a date for your point.

The one last element of the Events section that’s good to know about is the Categories menu at the upper right of the page. This allows you to delete categories as well as filter the Events list to only show a single category of events at a time. To do this, just click the name of a category in the Categories menu.

Template Web UI

Back in October we released the first version of our new Templating API, allowing you to easily replicate sets of bundles across multiple hosts. Now we bring you the time-saving sweetness of Templates in the web interface as well; if you have multiple servers that you want to monitor in exactly the same way, Templates are your friend. The idea behind them is pretty simple: you choose your master host, and select one or more of its check bundles to be used as master bundles. Then when you select your target hosts, the master bundles are copied and applied to the target hosts.

Creating A Template

So let’s look at how the Templating process works. Before you create a template, you first need to ensure that you have your master check bundles set up and active on your master host. Once that’s the case, start by going to the new “data” section of Circonus and visiting the “Templates” tab at the left. Create a new template via the “+” tab (or the “Create A Template” button in the middle of the page if you have no templates yet). In the resulting dialog, type a name for your template and choose a master host, and when you click “OK” you will see the templates table appear with a row for your new template (as usual, click the summary row to view the expanded details of the template).

When you first create a template, it’s in “draft” mode. This means that it’s only saved in your browser’s memory until you apply it. Nothing has been saved to the system yet, and the master bundles haven’t been replicated. This allows you to lay out templates and modify them or discard them before making any changes to the system. If you wish to save changes to a draft you may do so via the “Save” button; the draft is not applied as a regular Template until you click the “Apply” button. To aid in visually scanning the list of Templates for drafts, drafts will always appear at the top of your list, and will always be green. (If at any point you wish to change your template name or master host, you may click them in the summary row to edit them in-place. Please note: when changing your master host, you may only choose among the target hosts currently saved in the Template.)

Choosing Bundles

Once you’ve created your draft, you need to choose your master check bundles. Under the “Check Bundles” section at the left, click “Add Bundle” to bring up the new bundle dialog. All the bundles available for your master host will be shown here in a scrollable list. This is a selectable list, so when you select a bundle, it’s shown as selected in the list until you remove it from the Template. If you have a long list of bundles and are having a difficult time finding the ones you want, you may use the field above the list to filter the shown bundles by a filter string or regular expression (if you’re using a regular expression, don’t include the leading and trailing slashes, just use the desired RegEx syntax). After you have chosen a bundle, you may change its name by clicking on it in the list of chosen bundles. (Please Note: the reserved string “{target}” will be replaced by the current hostname/IP as the bundle is replicated across the target hosts.)

Choosing Hosts

Choosing your target hosts works mostly the same way as choosing the master check bundles. The “Add Host” button brings up a dialog with a scrollable, selectable, filterable list of available hosts on your account, and you may choose one or more of those hosts. There is an additional feature, however, which is the “Enter a new host” field below the list. This allows you to enter new hosts (either IP addresses or domain names are acceptable) that aren’t currently used on your account. When you enter a new host and hit return/enter, the new host will be subject to a DNS check to ensure that it really exists; if it passes the DNS check, it will then be added to your list of target hosts.

Once you’re satisfied with your bundle and host choices, clicking the “Apply” button will replicate your master bundles across each target host and will save the template in the database.

Modifying A Template

the action dropdown select

Once your Template is saved you will see several things change in the details panel. Each bundle and host will get checkboxes, and two “Action” dropdown selects will appear, one above the check bundles list and one above the target hosts list. Now that the bundles and hosts are a part of the template, if you wish to modify or remove them, you will need to check their checkboxes and choose an action from the appropriate dropdown before saving. There are four actions available:

Remove
When used on a bundle, it will delete the target bundles and remove them from the Template. When used on a host, it will delete the host’s bundles and show the host as inactive in the host list.
Unbind
When used on a bundle, it will leave the target bundles in place but will break their synchronization with the template and show them as inactive in the bundle list. When used on a host, it will leave the host’s bundles in place but will break their synchronization with the template and show the host as inactive in the host list.
Deactivate
When used on a bundle, it will deactivate the target bundles and show them as inactive in the bundle list. When used on a host, it will deactivate the host’s bundles and show the host as inactive in the host list.
Restore
When used on a bundle, it will reactivate, rebind, or recreate target bundles as necessary, to restore them to active status and synchronization with the template. When used on a host, it will reactivate, rebind, or recreate the host’s bundles as necessary, to restore them to active status and synchronization with the template.

Staying In-Sync

re-sync button

After creating and applying a Template, you are still allowed to edit the master check bundles. If you do so, any Templates using those check bundles as master bundles will be out-of-sync. When you go to your Templates page, the out-of-sync Templates will have their sync buttons activated and the buttons will say “Re-Sync.” Simply click the “Re-Sync” button to replicate the bundle changes across all the target bundles, and the Template will be in-sync again.

(Please Note: if at any point you wish to delete the template, any active bundles that are still a part of the template will be deleted from the target hosts. If you wish to keep the bundles on the target hosts but just delete the template, you will need to unbind all the bundles you wish to keep on the target hosts and then delete the template.)

One Dashboard to Rule Them All

<

four icons representing a dashboard

Ever dream of having a systems monitoring dashboard that was actually useful? One where you could move things around, resize them, and even choose what information you wanted to display? Large enterprise software packages may have decent dashboards, but what if you’re not a large enterprise or you don’t want to pay an arm and a leg for bloatware? Perhaps you have a good dashboard that came with a specific server or piece of hardware, but it’s narrowly-focused and inflexible. You’ve probably thought about (or even tried) creating your own dashboard, but it’s a significant undertaking that’s not for the faint-of-heart. What’s the solution? Should we just learn to live with sub-optimal monitoring tools?

Here at Circonus, we decided that this was one problem we could eliminate. Since we’ve built a SaaS offering that’s flexible enough to handle multiple different data sources, why shouldn’t we build a dashboard that’s flexible enough to display them? So we created a configurable dashboard that lets you monitor your data however you want. Do you want to show graphs side-by-side but at different sizes? Done. Want an up-to-date list of alerts beside those graphs? Easy. How about some real-time metric charts that automatically refresh? No problem. Our new configurable dashboards allow you to add all these items and more. Let’s dig in and see how these new dashboards work.

Dashboard Basics

Start by going to the standard ‘Dashboard’ and clicking the new ‘My Dashboards’ tab. These dashboards are truly yours; any dashboards you create are only visible to you (by default) and are segregated by account. If you want to share a custom dashboard with everyone else on an account, check that dashboard’s ‘share’ checkbox in your list of custom dashboards.

After you have created a custom dashboard, you may set it to be your default dashboard by using the radio buttons down the left side of your custom dashboards list. If you do this, you will be greeted with your selected dashboard when you login to Circonus. By selecting the ‘Standard Circonus Dashboard’ as your default dashboard, you will revert to being greeted with the old dashboard you’re already used to seeing.

part of the interface for creating a new dashboard layout

To create a new custom dashboard, click the ‘+’ tab and choose a layout. At first you will see only a couple predefined layouts available, but after you create a dashboard, its layout will then be available to choose when creating other new dashboards.

Now a note about working with these dashboards: every action auto-saves so you never have to worry about losing changes you’ve made. However, if you haven’t given your dashboard a title, the dashboard isn’t permanently saved yet. If you forget to title your dashboard and go off to do other things, don’t worry, the dashboard you created is saved in your browser’s memory. All you have to do is visit the ‘My Dashboards’ page and your dashboard will be listed there. With two clicks you can give your dashboard a title and save it permanently. (Please note our minimum browser requirements ‘Firefox 4+ or Chrome’ which are especially applicable for these new custom dashboards, since we’re using some features which are not available in older browsers.)

So let’s create a dashboard. Choose a layout, click ‘Create Dashboard,’ and you will be taken to the new dashboard with the ‘Add A Widget’ panel extended. To begin, let’s check out the title area. Notice that when you hover over the title, a dropdown menu appears. This lists your other dashboards on the current account (as well as dashboards shared by other account members) and is useful for quickly switching between dashboards.

the dashboard interface showing the dashboard controls icons

To the right of the title are some icons. The first icon opens the grid options dialog, which lets you change the dimensions of the dashboard grid, hide the grid (it’s still active and usable, though), enable or disable text scaling, and choose whether or not to auto-hide the title bar in fullscreen mode. The second icon toggles fullscreen mode on and off. Once you enter fullscreen mode a third icon will appear, and this icon toggles the ‘Black Dash’ theme (this theme is only available in fullscreen mode). The current states of both fullscreen mode and the ‘Black Dash’ theme are saved with your dashboard.

One other note about the dashboard interface: if you leave a dashboard sitting for more than ten or fifteen seconds and notice that parts of the interface disappear (along with the mouse cursor), don’t worry, it’s just gone to sleep! A move of the mouse will make everything visible again. (If there are any widget settings panels open, though, the sleep timer will not activate.)

Widgets

Now for the meat of it all: widgets. We currently have ten widgets which can be added to the dashboard grid to show various types of data, and we’ll be adding more widget types and contents in the future. Following is a quick rundown of the currently available widgets:

Graph

Graph widgets let you add existing graphs to your dashboard. You may choose any graph from the “My Graphs” section under your current account. Graph widgets are refreshed every few minutes to ensure they’re always up-to-date.

Beacon Map

Map widgets let you add existing Beacon maps to your dashboard. You may choose any map query from the “Beacons” page (under the “Checks” section of your current account). Map widgets are updated in real-time.

Beacon Table

Table widgets let you add existing Beacon tables to your dashboard. You may choose any table query from the “Beacons” page (under the “Checks” section of your current account). Table widgets are updated in real-time.

Chart

Chart widgets let you select multiple metrics to monitor and compare in a bar or pie chart. Chart widgets are updated in real-time.

Gauge

Gauge widgets let you monitor the current state of a single numeric metric in a graphical manner, displaying the most recent value on a bar gauge (dial gauges are coming soon). Gauge widgets are updated in real-time.

Status

Status widgets let you monitor the current state of one or more metrics, displaying the most recent value with custom formatting. This is most useful for text metrics, but it may be used for numeric metrics as well. Status widgets are updated in real-time.

HTML

HTML widgets let you embed arbitrary HTML content on your dashboard. It can be used for just about anything, from displaying a logo or graphic to using an iframe to embed more in-depth content. Everything is permissible except Javascript. HTML widgets are refreshed every few minutes to ensure they’re always up-to-date.

List

List widgets let you add lists of graphs and worksheets to your dashboard, ordered by their last modified date. You may specify how many items to list and (optionally) a search string to limit the list. List widgets are refreshed every few minutes to ensure they’re always up-to-date.

Alerts

Alerts widgets let you monitor your checks by showing the most recent alerts on your current account. You may filter the alerts by their age (how long ago they occurred), by particular search terms, by severity levels, or other status criteria. Alerts widgets are refreshed every few minutes to ensure they’re always up-to-date.

Admin

Admin widgets let you monitor selected administrative information, including the status of all Circonus agents on your current account. Admin widgets are refreshed every few minutes to ensure they’re always up-to-date.

icons representing some of the current widget types

To add widgets to the dashboard grid, there are two methods: you may use the ‘drag-and-drop’ method (dragging from the “Add a Widget” panel), or you may first click the target grid cell and then select the widget you want to place there. (Note: in fullscreen mode only the latter method is available.) After a widget has been added, some types of widgets will automatically activate with default settings, but most will be inactive. If the widget is inactive, click it to open the settings panel and get started. Once the widget is activated, the settings panel is available by clicking the settings icon in the upper right corner of the widget. In the lower right corner of the widget is the resize handle, so you can resize the widget as frequently as you want. And let’s not forget being able to rearrange the widgets, every widget has a transparent ‘title bar’ at its top which you can use to drag it around. I won’t get into the details of settings for every type of widget, because they should be self-explanatory (and that would make this one super-long blog post). But suffice it to say, there are plenty of options for everyone.

We’ve been working hard to create a configurable dashboard that will be as flexible as Circonus itself is, and we believe we’ve hit pretty close to the mark. Here’s a sample dashboard showing the power of these new dashboards:


A Lotta Love for Keyboard Users

All web users who bemoan the general lack of support for keyboard accessibility in web apps, take heart! Circonus has some great features for keyboard lovers. We know there are many web users out there for whom keyboard shortcuts are a quicker and easier way to use applications, particularly web apps. This is especially true if you use a specific app heavily, or are a full-time computer user in general.

Anywhere in Circonus, you can always see the keyboard help screen by typing “?” so you’ll have an ever-present “cheat sheet” as you learn the shortcuts. As soon as new keyboard functionality is added, the keyboard help screen will be updated immediately to reflect the new shortcuts, thanks to the magic of continuous deployment.

Jump Navigation

To jump to a particular section in Circonus, all you have to do is type the proper keyword and you will jump there immediately. For example, type the keyword “dash” (d-a-s-h) and you will jump to the current account’s dashboard. It’s that easy! Here’s a list of the current jump keywords:

  • “dash” (jump to the dashboard)
  • “alerts” (jump to the fault dashboard)
  • “rules” (jump to rules)
  • “checks” (jump to checks)
  • “metrics” (jump to metrics)
  • “trends” (jump to the trending dashboard)
  • “graphs” (jump to graphs)
  • “worksheets” (jump to worksheets)

The shortcut for opening the feedback dialog also works the same way: simply type “feedback” and the feedback dialog will open for you. Another quick shortcut is the forward-slash (/), which focuses on any search field that may be on the page.

Graph & Worksheet Shortcuts

Here’s where we get to the good stuff. We’ve added some great shortcuts to work with graphs and an enhanced zoom tool which is only available via keyboard shortcuts.

To start off, you can now see the legend on any thumbnail graph view (on “My Graphs,” “Trending Dashboard,” and all worksheets) not only on the large graph views as before. To do so, simply hold down the shift key, and the legend will appear for whichever graph you’re hovering over. On a worksheet, the shift key also inverts the legend hover option. So if you have enabled the new worksheet option to show legends upon graph hovering, holding down the shift key will disable the hovering legends.


Back in January we launched an enhanced graph zoom toolbar that relies on keyboard shortcuts. Normally the zoom toolbar is labeled “Past” because its buttons will set the graph zoom level to view data from the past one week, two weeks, etc. However, if you hold down either comma or period, the zoom tool will be enhanced and the label will change to “shift.” You will also see an orange bar at the end of the graph(s) which indicates the end that will be shifted (and if you hold both keys, you will get two orange bars, indicating that you can pan the entire graph date range into the past or the future). While holding one or both keys, click one of the new arrow icons that appear inside the “shift” buttons—the graph date(s) will be shifted by the specified amount in the specified direction. Not only does this work when viewing or editing graphs, it works almost everywhere there is one or more graphs, whether large or thumbnail sized.

One last set of useful shortcuts applies when viewing a worksheet. Among the newly added worksheet options is the ability to resize worksheet graphs to one of three sizes. In addition to being able to do this by clicking the buttons in the worksheet options dialog, you can instantly change the size of your worksheet graphs by pressing alt+1, alt+2, or alt+3.

Being avid keyboard users ourselves, we are excited to build keyboard support into more areas of Circonus as we are able to do so. Keep watching for more keyboard info and if you have ideas for some useful shortcuts, please let us know!