Getting Output from jps with NRPE

2025-03-22T00:00:00+00:00

One of the tools our hosting team uses for server and site monitoring is Icinga (which is based on Nagios). When monitoring host resources, one of the tools we use is Nagios Remote Plugin Executor or NRPE.

We encountered an issue when executing NRPE: though NRPE runs on the server being monitored, it wasn’t giving the same output as a script which was executed on the server itself. The NRPE-related call should have no issues be executed on the target server, as it is declared in the sudoers file (commonly /etc/sudoers). In this post, I will explain how to get the output from jps (Java Virtual Machine Process Status Tool), which can only be executed as root.

Getting process information with jps

Let’s say we have a “hello world” program named Hello.java. How do we get the process’s state from Icinga’s head server?

First, let’s compile and run the program.

public class Hello {
    public static void main(String[] args) {
        System.out.println("Hello, World!");

        try {
            Thread.sleep(3000);  // Sleep for 3 seconds
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

java Hello.java
Hello, World!

The program will run until it times out.

First, let’s check the process by running ps on the server:

# ps aux | grep Hello.java | grep -v grep
najmi      61609 16.7  0.3 11541184 109360 pts/5 Sl+  01:19   0:00 java Hello.java

We can also get the process ID with jps. Here’s the output from the “najmi” user view:

# sudo -s -u najmi jps -l
61609 jdk.compiler/com.sun.tools.javac.launcher.SourceLauncher
61695 jdk.jcmd/sun.tools.jps.Jps

And from the “root” user view:

# whoami
root
# jps -l
61609 jdk.compiler/com.sun.tools.javac.launcher.SourceLauncher
61852 jdk.jcmd/sun.tools.jps.Jps
# ps aux | grep Hello.java | grep -v grep
najmi      61609  1.2  0.3 11541184 109360 pts/5 Sl+  01:19   0:01 java Hello.java

However, the “nrpe” user does not see it:

# sudo -s -u nrpe jps -l
61790 jdk.jcmd/sun.tools.jps.Jps

Instead of running the jps command directly as Nagios, we will let the system run jps as root and dump the results into a file. An NRPE-based script will later read the output and feed the result to the dashboard.

Running jps as root, with output to a file

For this example, I started the jetty process with sudo.

sudo /usr/share/jetty/bin/jetty.sh start

Now let’s check whether the “jetty” service is running or not.

First we’ll check the status as a non-root user. It sees the process as “not running”.

$ /usr/share/jetty/bin/jetty.sh status
** WARNING: JETTY_LOGS is Deprecated. Please configure logging within the jetty base.
Jetty NOT running

JAVA                  =  /usr/bin/java
JAVA_OPTIONS          =
JETTY_HOME            =  /usr/share/jetty
JETTY_BASE            =  /usr/share/jetty
START_D               =  /usr/share/jetty/start.d
START_INI             =  /usr/share/jetty/start.ini
JETTY_START           =  /usr/share/jetty/start.jar
JETTY_CONF            =  /usr/share/jetty/etc/jetty.conf
JETTY_ARGS            =  jetty.state=/run/jetty/jetty.state jetty.pid=/run/jetty/jetty.pid --module=pid,state
JETTY_RUN             =  /run/jetty
JETTY_PID             =  /run/jetty/jetty.pid
JETTY_START_LOG       =  /run/jetty/jetty-start.log
JETTY_STATE           =  /run/jetty/jetty.state
JETTY_START_TIMEOUT   =  60
JETTY_SYS_PROPS       =
RUN_ARGS              =  -Djava.io.tmpdir=/tmp -Djetty.home=/usr/share/jetty -Djetty.base=/usr/share/jetty --class-path /etc/jetty/resources:/usr/share/jetty/lib/logging/slf4j-api-2.0.16.jar:/usr/share/jetty/lib/logging/jetty-slf4j-impl-12.0.16.jar:/usr/share/jetty/lib/jetty-http-12.0.16.jar:/usr/share/jetty/lib/jetty-server-12.0.16.jar:/usr/share/jetty/lib/jetty-xml-12.0.16.jar:/usr/share/jetty/lib/jetty-util-12.0.16.jar:/usr/share/jetty/lib/jetty-io-12.0.16.jar org.eclipse.jetty.xml.XmlConfiguration java.version=23.0.1 jetty.base=/usr/share/jetty jetty.base.uri=file:///usr/share/jetty jetty.home=/usr/share/jetty jetty.home.uri=file:///usr/share/jetty jetty.pid=/run/jetty/jetty.pid jetty.state=/run/jetty/jetty.state jetty.webapp.addHiddenClasses=org.eclipse.jetty.logging.,file:///usr/share/jetty/lib/logging/,org.slf4j. runtime.feature.alpn=true slf4j.version=2.0.16 /etc/jetty/jetty-bytebufferpool.xml /etc/jetty/jetty-pid.xml /etc/jetty/jetty-threadpool.xml /etc/jetty/jetty.xml /etc/jetty/jetty-state.xml
ID                    =  uid=1000(user) gid=1000(user) groups=1000(user),90(network),98(power),984(users),987(storage),991(lp),994(input),996(audio),998(wheel)
JETTY_USER            =  jetty
USE_START_STOP_DAEMON =  0
START_STOP_DAEMON     =  0

Now, let’s check the status of the jetty service with sudo:

$ sudo /usr/share/jetty/bin/jetty.sh status
** WARNING: JETTY_LOGS is Deprecated. Please configure logging within the jetty base.
Jetty running pid=89969

JAVA                  =  /usr/bin/java
JAVA_OPTIONS          =
JETTY_HOME            =  /usr/share/jetty
JETTY_BASE            =  /usr/share/jetty
START_D               =  /usr/share/jetty/start.d
START_INI             =  /usr/share/jetty/start.ini
JETTY_START           =  /usr/share/jetty/start.jar
JETTY_CONF            =  /usr/share/jetty/etc/jetty.conf
JETTY_ARGS            =  jetty.state=/run/jetty/jetty.state jetty.pid=/run/jetty/jetty.pid --module=pid,state
JETTY_RUN             =  /run/jetty
JETTY_PID             =  /run/jetty/jetty.pid
JETTY_START_LOG       =  /run/jetty/jetty-start.log
JETTY_STATE           =  /run/jetty/jetty.state
JETTY_START_TIMEOUT   =  60
JETTY_SYS_PROPS       =
RUN_ARGS              =  -Djava.io.tmpdir=/tmp -Djetty.home=/usr/share/jetty -Djetty.base=/usr/share/jetty --class-path /etc/jetty/resources:/usr/share/jetty/lib/logging/slf4j-api-2.0.16.jar:/usr/share/jetty/lib/logging/jetty-slf4j-impl-12.0.16.jar:/usr/share/jetty/lib/jetty-http-12.0.16.jar:/usr/share/jetty/lib/jetty-server-12.0.16.jar:/usr/share/jetty/lib/jetty-xml-12.0.16.jar:/usr/share/jetty/lib/jetty-util-12.0.16.jar:/usr/share/jetty/lib/jetty-io-12.0.16.jar org.eclipse.jetty.xml.XmlConfiguration java.version=23.0.1 jetty.base=/usr/share/jetty jetty.base.uri=file:///usr/share/jetty jetty.home=/usr/share/jetty jetty.home.uri=file:///usr/share/jetty jetty.pid=/run/jetty/jetty.pid jetty.state=/run/jetty/jetty.state jetty.webapp.addHiddenClasses=org.eclipse.jetty.logging.,file:///usr/share/jetty/lib/logging/,org.slf4j. runtime.feature.alpn=true slf4j.version=2.0.16 /etc/jetty/jetty-bytebufferpool.xml /etc/jetty/jetty-pid.xml /etc/jetty/jetty-threadpool.xml /etc/jetty/jetty.xml /etc/jetty/jetty-state.xml
ID                    =  uid=0(root) gid=0(root) groups=0(root)
JETTY_USER            =  jetty
USE_START_STOP_DAEMON =  0
START_STOP_DAEMON     =  0

Then, check that it’s running with jps as root, and note that the nrpe user still can’t see the process using jps:

$ sudo /usr/share/jetty/bin/jetty.sh status 2>/dev/null | grep "running pid"
Jetty running pid=89969

$ sudo jps -l
89969 org.eclipse.jetty.xml.XmlConfiguration
91635 jdk.jcmd/sun.tools.jps.Jps

$ sudo -s -u nrpe jps -l
91711 jdk.jcmd/sun.tools.jps.Jps

We can decide whether to use the “jetty” service script as the process check tool, or whether to use jps. For this post, we’ll focus on jps.

We can run jps as root in a Bash script to dump the Java process ID into a temporary file. We can then use this script as an NRPE script.

#!/bin/bash

# Check if program name was provided
if [ -z "$1" ]; then
    echo "UNKNOWN: Please specify a program name to check"
    exit 3
fi

TEMP_FILE="/tmp/java_pid.txt"

# Clear the temp file first
> $TEMP_FILE

# Run jps and save output to temp file
jps -l > $TEMP_FILE

# Check if program is in the temp file
if grep -q "$1" $TEMP_FILE; then
    PID=$(grep "$1" $TEMP_FILE | awk '{print $1}')
    echo "OK - $1 is running with PID $PID"
    exit 0
else
    echo "CRITICAL - $1 is down"
    exit 2
fi

Conclusion

Most of time, using ps or pgrep will be adequate to check for running processes. This post just shows how to get the output from jps, when you need to use jps. As for checking the status of the jetty service, /usr/share/jetty/bin/jetty.sh status might be fine as well, but the process of filtering, parsing, and feeding the output to the NRPE plugin should be pretty much the same.

There could be another way to solve this issue, which I am not be aware of. The method I shared above is just one way to get jps’s report working with NRPE. Please let me know if you have experience with jps and Icinga/Nagios, and how you handle their reporting.

Fetching Outputs From Java Process Monitoring Tool with Icinga/Nagios

2021-06-07T00:00:00+00:00

Photo by Mihai Lupascu on Unsplash

Recently, I encountered an issue when executing NRPE, a Nagios agent which runs on servers that are being monitored from Icinga’s head server. Usually NRPE-related calls should run without issues on the target server, since it is declared in the sudoers file (commonly /etc/sudoers). In this post, I will cover an issue I encountered getting the output from jps (Java Virtual Machine Process Status Tool), which needed to be executed with root privileges.

Method

I wanted to use Icinga to get a Java process’s state (in this case, the process is named “Lucene”) from Icinga’s head server, remotely. jps works for this, functioning similarly to the ps command on Unix-like systems.

Usually, NRPE should be able to execute the remote process (on the target server) from Icinga’s head. In this case we are going to create a workaround through the following steps:

Dump the Java process ID into a text file.
Dump the running threads into another text file.
Put item 1 and item 2 above into a single bash script.
Create a cronjob to automatically run the bash script.
Create an NRPE plugin to evaluate the output of item 1 and item 2.

Test

To illustrate this, I ran the intended command locally on the target server as the nagios user. Theoretically, this should emulate the NRPE call as if it was executed from Icinga’s server remotely. The file check_lucene_indexing_deprecated was meant to demonstrate the NRPE execution failure, whereas check_lucene_indexing is the file which is expected to run the NRPE plugin successfully. The paths to both check_lucene_indexing_deprecated and check_lucene_indexing were already declared in /etc/sudoers file on the target machine.

To show the differences, I ran two different scripts from the Icinga’s head server.

Here is the output from both local script executions: first as the nagios user, then as the root user:

# sudo -s -u nagios ./check_lucene_indexing_deprecated
CRITICAL -- Lucene indexing down

# sudo -s -u root  ./check_lucene_indexing_deprecated
OK -- 2 Lucene threads running

As you can see, the script worked fine running as root, but not as the nagios user.

Let’s run the scripts from Icinga’s head server:

# /usr/lib64/nagios/plugins/check_nrpe -t 5 -H <the target server’s FQDN> -c  check_lucene_indexing_dep
CRITICAL -- Lucene indexing down (0 found)

# /usr/lib64/nagios/plugins/check_nrpe -t 5 -H  <the target server’s FQDN> -c  check_lucene_indexing
OK -- 2 Lucene threads running

In the background, we can see different output from running jps on the target server using the root user compared to the nagios user. Let’s say I want to check the jps process ID (PID):

# sudo -s -u nagios jps -l
29112 sun.tools.jps.Jps

And as root:

# jps -l
7541 /usr/share/jetty9/start.jar
29131 sun.tools.jps.Jps

The point of running the jps -l command is to get the process ID of /usr/share/jetty9/start.jar, which is 7541. However, as indicated above, the nagios user’s execution did not display the intended result, but the root user’s did.

The workaround

We can check the existence of the process ID by dumping it into a text file and letting the NRPE plugin read it instead.

In order to get NRPE to fetch the current state of the process, we will create a cronjob; in our case it will be executed every 10 minutes. This script will dump the PID of the Java process into a text file and later NRPE will run another script which will analyze the contents of the text file.

Cronjob, creating dump files

*/10 * * * * /root/bin/fetch_lucene_pid.sh

The cron script contains the following details:

PID_TARGET=/var/run/nrpe-lucene.pid
THREADS_TARGET=/var/run/nrpe-lucene-thread.txt

/usr/bin/jps -l | grep "start.jar" | cut -d' ' -f1 1>$PID_TARGET 2>/dev/null

PID=$(cat $PID_TARGET)

re='^[0-9]+$'

if  [[ -z $PID ]]  || ! [[ $PID =~ $re ]]  ; then
exit 0
fi

THREADS=$(/usr/bin/jstack $PID | grep -A 2 "ProjectIndexer\|ConsultantIndexer" | grep -c "java.lang.Thread.State: WAITING (parking)")

echo $THREADS > $THREADS_TARGET

So instead of running the jps command directly as nagios, we let the system run jps (as root) and dump the result into a file. Our NRPE-based script will read the output later and feed the result to the dashboard.

NRPE plugin file, reading values generated from the cronjob

So we will take a look at what was written in the successfully executed Bash script (that is, check_lucene_indexing).

The NRPE plugin file, check_lucene_indexing, contains the following script:

#!/bin/bash

PID_TARGET=/var/run/nrpe-lucene.pid
THREADS_TARGET=/var/run/nrpe-lucene-thread.txt

PID=$(cat $PID_TARGET)
THREADS=$(cat $THREADS_TARGET)

re='^[0-9]+$'

if  [[ -z $PID ]]  || ! [[ $PID =~ $re ]]  ; then
  echo "CRITICAL -- Lucene indexing down (a)"
  exit 2
fi


if [ $THREADS -eq 2 ]
then
  echo "OK -- $THREADS Lucene threads running"
  exit 0
else
  echo "CRITICAL -- Lucene indexing down (b)"
  exit 2
fi

From the NRPE plugin script you can see the following text files being used:

PID_TARGET=/var/run/nrpe-lucene.pid
THREADS_TARGET=/var/run/nrpe-lucene-thread.txt

PID_TARGET contains the process’s PID, which I used to determine whether the intended process is running or not.

THREADS_TARGET contains the number of the Java threads which are currently running.

The following is the content of the check_lucene_indexing_deprecated script:

#!/bin/bash

PID=$(/usr/bin/jps -l | grep "start.jar" | cut -d' ' -f1)

if [[ -z $PID ]]; then
  echo "CRITICAL -- Lucene indexing down"
  exit 2
fi

THREADS=$(/usr/bin/jstack $PID | grep -A 2 "ProjectIndexer\|ConsultantIndexer" | grep -c "java.lang.Thread.State: WAITING (parking)")

if [ $THREADS -eq 2 ]
then
  echo "OK -- $THREADS Lucene threads running"
  exit 0
else
  echo "CRITICAL -- Lucene indexing down"
  exit 2
fi

As you can see, check_lucene_deprecated was able to get the result if it is being executed locally on the target machine - but not from the remote (Icinga’s head server). This is because jps will provide limited results when executed as the nagios compared to the local root user. Note that I have defined the path of the script in the sudoers file prior to the script execution.

Defaults: nagios !requiretty
nagios  ALL = NOPASSWD: /usr/local/lib/nagios/plugins/check_lucene_indexing
nagios  ALL = NOPASSWD: /usr/local/lib/nagios/plugins/check_lucene_indexing_deprecated

Conclusion

The method which I shared above is just one of the ways to use jps reports with Icinga/Nagios plugins. As of now this solution works as expected. If you want to reuse the scripts, please customize them according to your environment to get the results you want. Also, as written in the documentation, getting the output by parsing the output from jps means we need to update the script any time jps changes its output format.

Please comment below if you have experience with jps and Icinga/Nagios, and tell us how you handle the reporting.

Streaming Replication time lag monitoring added to check_postgres

2015-08-10T00:00:00+00:00

I almost let this one sneak past! Guess I need to do some lag monitoring on myself. About a month or so ago, a new version of check_postgres was released, and that includes a bit of code I wrote. While the patch has been available in the git checkout for a while, now that it’s in the official release and will start appearing in repos (if it hasn’t already) it’s probably worth writing up a quick note describing its reasoning and usage.

What’s the feature? Time-based replication monitoring in the hot_standby_delay action. This was something that had been a long-standing item on my personal TODO list, and happened to scratch the itch of a couple of clients at the time.

Previously it would only take an integer representing how many bytes of WAL data the master could be ahead of a replica before the threshold is crossed:

check_hot_standby_delay --dbhost=master,replica1 --critical=16777594

This is certainly useful for, say, keeping an eye on whether you’re getting close to running over your wal_keep_segments value. Of course it can also be used to indicate whether the replica is still processing WAL, or has become stuck for some reason. But for the (arguably more common) problem of determining whether a replica is falling too far behind determining what byte thresholds to use, beyond simply guessing, isn’t easy to figure out.

Postgres 9.1 introduced a handy function to help solve this problem: pg_last_xact_replay_timestamp(). It measures a slightly different thing than the pg_last_xlog_* functions the action previously used. And it’s for that reason that the action now has a more complex format for its thresholds:

check_hot_standby_delay --dbhost=master,replica1 --critical="16777594 and 5 min"

For backward compatibility, of course, it’ll still take an integer and work the same as it did before. Or alternatively if you only want to watch the chronological lag, you could even give it just a time interval, “5 min”, and the threshold only takes the transaction replay timestamp into account. But if you specify both, as above, then both conditions must be met before the threshold activates.

Why? Well, that gets in to bit about the measurement of slightly different things. As its name implies, pg_last_xact_replay_timestamp() returns the timestamp of the last transaction it received and replayed. That’s fine if you have a database cluster that’s constantly active 24 hours a day. But not all of them are. Some have fluctuating periods of activity, perhaps busy during the business day and nearly idle during the night. In other words, if the master isn’t processing any transactions, that last transaction timestamp doesn’t change.

Then there’s the other end of the scale. With the SSD’s/high speed disk arrays a master server may in a short interval process more transaction data than it can send over a network wire. For example, we have a system that runs an ETL process between two local databases on a master server, and generates a ton of transaction log data in a short amount of time. However even if it has many megabytes of WAL data to transmit, the replicas never get any more than a handful of seconds behind and soon catch up.

Both conditions on their own are fine. It’s when both conditions are simultaneously met, when the replica is behind in both transaction log and it hasn’t seen a chronologically recent transaction, that’s when you know something is going wrong with your replication connection.

Naturally, this updated check also includes the chronological lag metric, so you can feed that into Graphite, or some other system of choice. Just make sure you’re system handles the new metric; our Nagios system seemed to ignore it until the RRD data for the check was cleared and recreated.

Oh, and make sure your clocks are in sync. The timestamp check executes only on the replica, so any difference between its clock and the master’s can show up as skew here. ntpd is an easy way to keep everything mostly synchronized, but if you really want to be sure, check_postgres also has a timesync action.

Monitorama, Berlin, EU - Day 2 and final considerations

2013-09-20T00:00:00+00:00

Moving toward the end of IT conferences your expectations sometime gets lower cause the speakers are tired and so are the attendees. You kind of expect things to get quieter but this wasn’t the case with Monitorama EU 2013.

On this second day I found that all of the talks were as interesting, entertaining and inspiring as the ones on the first day.

I enjoyed all the talks proposed today but I got very inspired by the ones from Jeff Weinstein which talked about how you can use data collected for metrics and monitoring to improve the whole company. I also appreciated the speech from Gareth Rushgrove which highlighted how security is actually still underrated in IT companies and how/why you should try to integrate monitoring with security auditing tools.

I’ve been asked a few times, during the day and the evening, which was my opinion about the conference and the answer was always “absolutely positive!”. I always add that though I don’t expect to see any rocket science in these conferences I kind of suppose that I’ll get a lot of hints, ideas and tips which are a wonderful trampoline for new personal or work-related projects. That is exactly what I got.

The other aspect you unexpectedly get in this kind of technical conferences is an incredibly positive social experience with plenty of information exchanged both on a personal and technical perspective. I always over-stress this aspect because sometimes people go to conferences, listen to the talks, learn something and go home unsatisfied. That’s mostly because they missed the point.

If you want to learn new stuff you have a marvellous thing called Internet with GitHub, SlideShare and many useful books out there. What you’ll always miss from all this sources is the “human factor” which you only get when you hang out with other people that share interests with you. Who are willing to share their knowledge with you (hint: open source communities) and in general are there to enjoy staying together, just for a few days, with other weird humans which share their same passion: their IT work.

So in the end my impressions are very positive, and I’d like to thank all the guys behind the conference organization to have it run as good as it did. I’m sure I’ll join you again, hopefully soon.

Monitorama, Berlin, EU - Day 1

2013-09-19T00:00:00+00:00

If you care about the quality of your IT infrastructure and work, there are times where you really need to focus on a valuable and important aspect: community.

The thing is that most people don’t realize how valuable the human factor is when working in the IT field, until they happen to be in such a marvellous conference as Monitorama has been so far.

I was lucky enough to be there, in Berlin from 2013.09.19 to 2013.09.20, to enjoy all the awesome talks and attendees which was present there. And what I’m really saying that besides most of the speeches were quite technically interesting and definitely good quality ones, they definitely didn’t only revolves about monitoring per se.

I won’t mention each and every talk, though they all would have deserved it, but I’ll say that while I was very inspired by Danese Copper’s talk about Open Source value and importance, I was also very entertained Ryan Dotsmith’s one about how you could/should learn from failures, either yours or others ones, or the very specific “on the field” one from Katherine Daniels.

On top of that while I generally don’t appreciate sponsors having “talks” during this kind of conferences, I actually appreciated how they managed to handle the sponsor advertising part of the conference where you have this little, brief demos from the sponsors which are fairly related and never boring or out of context as I have experienced at other conferences.

A few things I actually got from all talks is that future of monitoring will definitely be all about machine learning and having computers mimic the way human interprets data. We just have to teach computers how to do this kind interpretive tasks as good as we do. It all revolves around this and only then we could all stop saying “You know computers are dumb, you just have to cope with it”.

One last thing which needs to be told about this awesome conference and needs to be mentioned is the awesome location named “Golgatha Biergarten” park which hosted our dinner, based on local grilled meat and food and an awesome community binding moment which was the perfect enriching closing for such an awesome day.

So it’s hard to hide how thrilled I am to see what’s coming next tomorrow.

Automating checking for new versions of PostgreSQL

2013-07-03T00:00:00+00:00

Hubble image by castielstar

It is important to run the latest revision of the major branch of Postgres you are using. While the pgsql-announce mailing list is often touted as a canonical way to become aware of new releases, a better method is to use the check_postgres program, which will let you know if the version you are running needs to be upgraded or not. An example usage:

$ check_postgres.pl --action=new_version_pg --dbhost=iroh
POSTGRES_NEW_VERSION_PG CRITICAL:  Please upgrade to version 9.2.4 of Postgres. You are running 9.2.2

Postgres version numbers come in three sections. The first two indicate the “major version” of Postgres that you are running. The third number indicates the revision. You always want to be using the highest revision available for your version. In the example above, Postgres version 9.2 is being used, and it should be upgraded from revision 2 to revision 4. A change in the revision number is known as a minor release; these are only done for important reasons, such as security or important bug fixes. Read the versioning policy page for more information.

When a new version of PostgreSQL is made, there are two general ways of communicating this fact: the pgsql-announce mailing list, and the versions.rss file on the postgresql.org web site. While the mailing list is low volume, it is not ideal as it contains posts about conferences, and about other Postgres-related software. A better solution is to track the versions.rss file. You could simply subscribe to it, but this will only tell you when the file has been changed. The check_postgres program parses this file and compares the latest revision to the version of Postgres that you are using.

To use it, simply call check_postgres and pass new_version_pg as the action argument, as well as telling check_postgres which PostgreSQL instance you are checking. For example, to check that the Postgres running on your internal host “zuko” is up to date, just run:

check_postgres.pl --action=new_version_pg --dbhost=zuko

Here is what the default output looks like, for both a matching and a non-matching version:

$ check_postgres.pl --action=new_version_pg --dbhost=appa
POSTGRES_NEW_VERSION_PG CRITICAL:  Please upgrade to version 9.2.4 of Postgres. You are running 9.2.2

$ check_postgres.pl --action=new_version_pg --dbhost=toph
POSTGRES_NEW_VERSION_PG OK:  Version 9.2.4 is the latest for Postgres

Those examples are very Nagios specific, of course, as evidenced by those uppercase strings at the beginning of the output. If you are using Nagios, it’s a good idea to run this, perhaps once a day or more often. If you are not using Nagios, you can make the output a little cleaner with the –simple argument:

$ check_postgres.pl --action=new_version_pg --dbhost=azula
Please upgrade to version 9.2.4 of Postgres. You are running 9.2.2

$ check_postgres.pl --action=new_version_pg --dbhost=sokka
Version 9.2.4 is the latest for Postgres

One quick and simple trick is to make this into a cron job and add the –quiet argument, which prevents all output if the check was “OK”. In this way, cron will only send outout (e.g. mail someone) when a new revision has been released. A cron example:

## Twice a day, check that we are running the latest Postgres:
0 7,18 * * * check_postgres.pl --action=new_version_pg --dbhost=cabbage --quiet

Once this alarm goes off, you should strive to update your clusters as soon as possible. If you are using a packaging system, then it may be as simple as relying on it to do the right thing, e.g. “yum update postgresql”. If you are installing from source, you will need the new tarball and can simply “make install” onto the existing Postgres, and then restart it. Always check the release notes for the new revision—once in a blue moon the update requires some other action, such as a reindex of certain types of indexes.

The check_postgres program can verify versions of some other programs as well, including Bucardo, tail_n_mail, and check_postgres itself.

check_postgres meets pgbouncer

2010-10-22T00:00:00+00:00

Recently the already well-known PostgreSQL monitoring tool check_postgres gained an ability to monitor pgbouncer, the PostgreSQL connection pooling daemon more closely. Previously check_postgres could verify pgbouncer was correctly proxying connections, and make sure its settings hadn’t been modified. The pgbouncer administrative console, reports many useful pgbouncer statistics and metrics; now check_postgres can monitor some of those as well.

pgbouncer’s description of its pools consists of “client” elements and “server” elements. “Client” refers to connections coming from clients, and “server” to connections to the PostgreSQL server. The new check_postgres actions pay attention only to the pgbouncer “SHOW POOLS” command, which provides the following metrics:

cl_active: Connections from clients which are associated with a PostgreSQL connection. Use the pgb_pool_cl_active action.
cl_waiting: Connections from clients that are waiting for a PostgreSQL connection to service them. Use the pgb_pool_cl_waiting action.
sv_active: Connections to PostgreSQL that are in use by a client connection. Use the pgb_pool_sv_active action.
sv_idle: Connections to PostgreSQL that are idle, ready to service a new client connection. Use the pgb_pool_sv_idle action.
sv_used: PostgreSQL connections recently released from a client session. Use the pgb_pool_sv_used action.
sv_tested: PostgreSQL connections in process of being tested. Use the pgb_pool_sv_tested action.
sv_login: PostgreSQL connections currently logging in. Use the pgb_pool_sv_login action.
maxwait: The length of time the oldest waiting client has been waiting for a connection. Use the pgb_pool_maxwait action.

Most installations probably don’t want any client connections stuck waiting for PostgreSQL connections to service them, meaning the cl_waiting and maxwait metrics ought to be zero. This example will check those two metrics and complain when they’re nonzero, for a pgbouncer installation on port 5433 with pools “pgbouncer” and “maindb”:

postgres@db:~$ ./check_postgres.pl --action=pgb_pool_cl_waiting -p 5433 -w 3 -c 8
POSTGRES_PGB_POOL_CL_WAITING OK: (port=5433) pgbouncer=0 * maindb=0 | time=0.01 time=0.01

postgres@db:~$ ./check_postgres.pl --action=pgb_pool_maxwait -p 5433 -w 5 -c 15 
POSTGRES_PGB_POOL_MAXWAIT OK: (port=5433) pgbouncer=0 * maindb=0 | time=0.01 time=0.01

The typical check_postgres filtering rules will work; to filter out a pool called “ignore_this_pool”, for instance, add –exclude ignore_this_pool to the command line. Other connection options mean exactly what they would when connection to PostgreSQL directly.

These new actions are available in the latest version from git.

Getting Output from jps with NRPE

Getting process information with jps

Running jps as root, with output to a file

Conclusion

Related reading

Fetching Outputs From Java Process Monitoring Tool with Icinga/​Nagios

Method

Test

The workaround

Cronjob, creating dump files

NRPE plugin file, reading values generated from the cronjob

Conclusion

Streaming Replication time lag monitoring added to check_postgres

Monitorama, Berlin, EU - Day 2 and final considerations

Monitorama, Berlin, EU - Day 1

Automating checking for new versions of PostgreSQL

check_postgres meets pgbouncer

Fetching Outputs From Java Process Monitoring Tool with Icinga/Nagios