Expand the Power of Munin: Write Your Own Plugins!

Introduction


Munin comes with many plugins, which work out of the box after the installation. Even though, you may still want to monitor values for which no plugin has been created yet. For this chapter, we will create a plugin which monitors the CPU usage for a defined set of users; I created it to help identify which applications were using most of the CPU in a fastCGI environment, on one of our Ubuntu-eu servers.

You can create plugins in the language of your choice; most of the ones munin comes with are either in Perl or shell scripts, but you should be able to use whatever language you are familiar with. Although, as the plugin I will describe here is in sh, this chapter requires a basic knowledge of shell programming and awk.

Retrieve the Data



First, you need to define which data you want to graph, and how to retrieve it. In our example, it would be the CPU usage for a specified user, at the time the plugin is run. I noticed the output of ps on my system, with it's BSD syntax, printed out the CPU usage for every process. It is definitely not very precise, as the values are rounded up; but I believe it gives a fairly good overview of which users are using most of the CPU.


~$ ps aux | head

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root             1        0.0        0.2       2952  1856 ?                Ss    09:21        0:02    /sbin/init
root            2        0.0        0.0          0        0    ?                S<   09:21        0:00   [kthreadd]
root           3        0.0        0.0          0        0    ?                S<   09:21        0:00   [migration/0]
root          4        0.0        0.0          0        0    ?                SN   09:21       0:00   [ksoftirqd/0]
[...]

So, there are three things to do: filter out the processes that belong to a specific user, isolate the CPU column, and add all the values. To filter out processes that belong to a user, we will use the -u option of ps; to isolate the CPU column, the easiest way is to use the -o option of ps:

~$ ps -u "yann" -o "%C"

%CPU
 0.0
 0.0
 1.5
 0.3
 0.0
 0.0
 [...]

We can now add the values with a small awk script (here in a single line, to make it easy to try out):


~$ ps -u "yann" -o "%C" | awk 'BEGIN {FS=" ";CPU_USER=0}{CPU_USER+=$0}END{print CPU_USER}'
8.6

Now that looks interesting... But we don't want to monitor only a single user, but several, and the variables need to be presented under the form var_name.value:



#!/bin/sh
export USERS="root user"
for USER in $USERS ; do {


        ps -u "$USER" -o "%C" |
        awk '
        BEGIN {
                FS=" "
                CPU_USER=0
        }


        {
                CPU_USER+=$0
        }


        END { 
                print  "'$USER'.value "CPU_USER
        }'
}


done;


Save this in /usr/share/munin/plugins/cpubyuser, add a soft link to it in /etc/munin/plugins/, and try it out:


~$ sudo vi /usr/share/munin/plugins/cpubyuser
~$ sudo ln -s /usr/share/munin/plugins/cpubyuser /etc/munin/plugins/cpubyuser
~$ sudo chmod 755 /usr/share/munin/plugins/cpubyuser
~$ /etc/munin/plugins/cpubyuser

root.value 3.6
user.value 4.2

Now, we don't want the usernames to be hardcoded in the plugin, but set in one of munin's configuration files. Edit the file /etc/munin/plugin-conf.d/munin-node, and add the following lines at the end:



[cpubyuser]
env.USERS root user


... and remove the line defining $USERS in our script. Reload munin-node (sudo /etc/init.d/munin-node restart).

~$ /etc/munin/plugins/cpubyuser 
~$ munin-run cpubyuser

root.value 3

user.value 3

If you run the script directly, the variable $USERS won't be defined, and the script will exit. If you run it via munin-run, munin-node will create the environment variable for the script, so it will work.


Autoconf


As we have seen in the previous chapter, when munin-node installs, it runs every script with the autoconf parameter, to decide if it should or not activate it. Therefore, our plugin also need to handle this.



if [ "$1" = "autoconf" ] ; then
        if [ -n "$USERS" ] ; then
                echo "yes"
        else
                echo "no (\$USERS not defined - edit your /etc/munin/plugin-conf.d/!)."
        fi
        exit
fi

Add this at the top of the plugin, before the main loop. If the plugin is run with the autoconf parameter, it will check for the environment variable $USERS; if it is defined, our plugin with reply with "yes", meaning that it can be used. Else, it will reply "no". Your plugin is supposed to check in this section if everything is ok for the plugin to run correctly; like checking if apache is running if your plugin is about apache.


Config


When run with the "config" parameter, your plugin is supposed to output data for RRDTool to tell it how to graph the different values given by our plugin, and metadata like the title of the graphs, the title of the axes...



if [ "$1" = "config" ] ; then
        echo "graph_args --base 1000 -r --lower-limit 0";
        echo "graph_title CPU usage, by user";
        echo "graph_category system";
        echo "graph_info This graph shows CPU usage, for monitored users.";
        echo 'graph_vlabel %'
        echo 'graph_scale no'
        echo 'graph_period second'
        echo "graph_order $USERS"


        FIRSTUSER=1;
        for USER in $USERS; do
                echo "${USER}.label $USER"
                echo "${USER}.info CPU used by user $USER"
                echo "${USER}.type GAUGE"
                if [ $FIRSTUSER -eq 1 ] ; then
                        echo "${USER}.draw AREA"
                        export FIRSTUSER=0;
                else
                        echo "${USER}.draw STACK"
                fi
        done ;


        exit
fi


The "base" parameter on the second line states if we consider that 1M=1024K or 1M=1000K. One of the most important part of this is the loop in the end, which tells rrdtool how it should process the data we feed it with; for every variable monitored, we need to tell rrdtool what to do with it.

For example, the variable "user" should be labelled "user" (you can change it to "CPU usage for $USER", or whatever you think is better). The label is what will appear in the legend, on the graph, it should be reasonably short. For a longer description, you should use the "info" section.

For the "type" of the value, you should read man rrdcreate, there is a whole part on the different types of data sources. A "gauge" is described like this: " GAUGE: is for things like temperatures or number of people in a room or the value of a RedHat share.".

Add informations to your plugin

Well, our plugin is nearly done. Now add these two lines at the top, between the shebang and the autoconf part:


#%# family=auto
#%# capabilities=autoconf


This will tell munin that it should attempt to automatically install and configure your plugin. Now you just need to add some comments, installation instructions, license, history, author contact, etc.... and your first munin plugin is ready!

Now, this was a very simple plugin; depending on what you want to do, it can be a lot harder (look at the memory plugin!). If you want to create complex graphs, you should definitely complete your knowledge by reading rrdtool's documentation. On the other hand, if you only want to create simple plugins, reading plugins in /usr/share/munin/plugins and trying to adapt them is a very good way to get started.