Saturday, 4 July 2015

Tomcat 8 on Debian Jessie with PSI Probe and OpenNMS to monitor performance (Part 3)

We looked at Tomcat 8 in Part 1 of this series and at PSI Probe as a replacement for the default Tomcat Manager application in Part 2.

Now PSI Probe is great for managing Tomcat and its applications, but it does not actively generate alerts if Tomcat is in difficulty, or any other part of the application stack for that matter, so this is where OpenNMS plugs the gap.

Network Management with OpenNMS

If you cut your teeth on the mainframes of the recent past, you will be familiar with network management tools like HP OpenView and IBM Tivoli. These tools are famous for their complexity and cost, so alternatives like Nagios (subsequently forked as Icinga) and OpenNMS emerged as open source alternatives.

OpenNMS is a Java-based, agentless monitoring tool that uses SNMP to monitor devices on the network. It was designed to manage service level agreements, so in addition to fault reporting (with trouble ticketing), it has a large number of service monitors to report against service thresholds.

If your device is SNMP-enabled, OpenNMS will auto-discover the device, add it as another node to your network topology and start monitoring it immediately, including vending machines.

OpenNMS Installation

The instructions for installing on Debian are here, but I will deviate slightly because I want to install OpenNMS on a pre-existing installation of Postgresql 9.4 on another server.

Create a database for OpenNMS

Login to the database server (called 'my-database-ip' here) and create an empty database called 'opennms', owned by 'opennms':
# su - postgres
# psql -c "CREATE USER opennms WITH LOGIN ENCRYPTED PASSWORD 'my-opennms-password';"

Install OpenNMS

On the server that you intend to run OpenNMS (called 'my-server-ip' here), add the OpenNMS repository to the APT sources list, add GPG key to verify the integrity of the package, update APT and install OpenNMS:
# vi /etc/apt/sources.list.d/opennms.list
   deb stable main
# wget -O - | apt-key add -
# apt-get update
# apt-get install opennms
Once the installation is complete, the database must be initialised, but before you run the scripts, you must give OpenNMS access to the database that you created by configuring the datasources:
# vi /etc/opennms/opennms-datasources.xml
  <jdbc-data-source class-name="org.postgresql.Driver" database-name="opennms" name="opennms" password="my-opennms-password" url="jdbc:postgresql://my-database-ip:5432/opennms" user-name="opennms">
  <jdbc-data-source class-name="org.postgresql.Driver" database-name="template1" name="opennms-admin" password="my-postgres-password" url="jdbc:postgresql://my-database-ip:5432/template1" user-name="postgres">
Now configure the JRE for OpenNMS and then run the installation:
# /usr/share/opennms/bin/runjava -s
# /usr/share/opennms/bin/install -dis
Install the iplike package to optimize lookups based on IP addresses, and then start the service:
# /usr/sbin/
# systemctl start opennms
If all is well, you should be able to browse to your instance of OpenNMS at http://my-server-ip:8980 and log in with admin/admin.

You can check that all the daemons are running with:
# opennms -v status
Note that OpenNMS uses two ports that you should be aware of: an httpAdaptor at 8181 and the application itself running on Jetty at 8980.

Once you have completed the install, you may want to comment out the OpenNMS repository that you added to APT, to prevent unsolicited updates of OpenNMS, particularly in a Production environment.

Discover nodes

By default OpenNMS monitors no network nodes at all, so the very first thing to do is to register the nodes to be monitored.

You can do a range scan but that will register every device in the range, whether you want to manage it or not, so rather register the nodes explicitly.

There are a number of ways to do this, but since you already have the IP address, the easiest is to register it directly:

Admin > Configure OpenNMS > Configure Discovery > Specifics > Add new

This will open a pop-up where you enter the IP address of the node, in this case it would be my-server-ip.

Click on 'Add", then when the pop-up closes, click on 'Save and Restart Discovery'.

Wait a moment while the discovery daemon runs, then list the node that you registered:

Info > Nodes

Now that the node has been discovered, I will show you how to monitor the JVM using JMX and SNMP in a later post. In the meantime, play around with the Service Level Management functionality in OpenNMS that comes out of the box.

Wednesday, 1 July 2015

Tomcat 8 on Debian Jessie with PSI Probe and OpenNMS to monitor performance (Part 2)

In Part 1 of this series of three, I looked at the installation of Tomcat 8 on Debian Jessie. The standard Tomcat Manager web application is limited in functionality, and the look-and-feel is distinctly lacking, so here is PSI Probe instead.

PSI Probe features

PSI Probe supplements the basic functionality of Tomcat Manager with a number of very useful features, including:
  • real-time monitoring of request traffic on the Connectors, from any Remote IP, or per application;
  • viewing of Data Source pool usage, and query execution;
  • monitoring of Logs, with the ability to dynamically change log levels at runtime;
  • viewing the Thread execution stack, with the option of killing threads;
  • dashboards to monitor the JVM, showing memory usage, swap file usage and garbage collection control;
  • and detailed System properties data.
PSI Probe is a fork of an older application called Lambda Probe that was last updated in 2006. This is a Lambda Probe:

Installing PSI Probe

You have to build PSI Probe from the source code because it is not available as a Debian package. It is a Java application, so it is a simple matter of cloning the project from GitHub and running Maven:
# git clone
Check that you are using Maven 3, and upgrade if necessary, because the build will fail if you use Maven 2. Don't forget to purge maven2 if you have to upgrade.
# mvn -version
Apache Maven 3.0.5
Maven home: /usr/share/maven
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
Default locale: en_ZA, platform encoding: UTF-8
OS name: "linux", version: "3.16.0-4-amd64", arch: "amd64", family: "unix"
If all is well, change to the PSI Probe directory and build the WAR file:
# cd psi-probe
# mvn package
The build will take some time the first time that you run it, because of all the JARs that must be downloaded from Maven Central to your local repository.

When it is done, use the Tomcat manager to deploy the war. Browse to http://my-server-IP:8080/manager, login with the same manager-gui role that you created in Part 1, scroll down to the section titled "Deploy" and use the option "Select WAR file to upload" to deploy the WAR file which you will find at psi-probe/web/target/probe.war.

Once it is deployed, browse to http://my-server-IP:8080/probe, login with the same manager-gui role, and dig around your Tomcat 8 installation. Much better looking, isn't it?

In Part 3, the last of this series, I will explain how OpenNMS can be used with SNMP to monitor Tomcat 8 and generate exceptions if any thresholds are exceeded.