Tomcat 8 on Debian Jessie with PSI Probe and OpenNMS to monitor performance (Part 3)

We looked at Tomcat 8 in Part 1 of this series and at PSI Probe as a replacement for the default Tomcat Manager application in Part 2.

Now PSI Probe is great for managing Tomcat and its applications, but it does not actively generate alerts if Tomcat is in difficulty, or any other part of the application stack for that matter, so this is where OpenNMS plugs the gap.

Network Management with OpenNMS

If you cut your teeth on the mainframes of the recent past, you will be familiar with network management tools like HP OpenView and IBM Tivoli. These tools are famous for their complexity and cost, so alternatives like Nagios (subsequently forked as Icinga) and OpenNMS emerged as open source alternatives.

OpenNMS is a Java-based, agentless monitoring tool that uses SNMP to monitor devices on the network. It was designed to manage service level agreements, so in addition to fault reporting (with trouble ticketing), it has a large number of service monitors to report against service thresholds.

If your device is SNMP-enabled, OpenNMS will auto-discover the device, add it as another node to your network topology and start monitoring it immediately, including vending machines.

OpenNMS Installation

The instructions for installing on Debian are here, but I will deviate slightly because I want to install OpenNMS on a pre-existing installation of Postgresql 9.4 on another server.

Create a database for OpenNMS

Login to the database server (called 'my-database-ip' here) and create an empty database called 'opennms', owned by 'opennms':

# su - postgres
# psql -c "CREATE USER opennms WITH LOGIN ENCRYPTED PASSWORD 'my-opennms-password';"
# psql -c "CREATE DATABASE opennms WITH OWNER=opennms ENCODING 'UNICODE';"

Install OpenNMS

On the server that you intend to run OpenNMS (called 'my-server-ip' here), add the OpenNMS repository to the APT sources list, add GPG key to verify the integrity of the package, update APT and install OpenNMS:

# vi /etc/apt/sources.list.d/opennms.list
   deb http://debian.opennms.org stable main
# wget -O - http://debian.opennms.org/OPENNMS-GPG-KEY | apt-key add -
# apt-get update
# apt-get install opennms

Once the installation is complete, the database must be initialised, but before you run the scripts, you must give OpenNMS access to the database that you created by configuring the datasources:

# vi /etc/opennms/opennms-datasources.xml
  <jdbc-data-source class-name="org.postgresql.Driver" database-name="opennms" name="opennms" password="my-opennms-password" url="jdbc:postgresql://my-database-ip:5432/opennms" user-name="opennms">
  <jdbc-data-source class-name="org.postgresql.Driver" database-name="template1" name="opennms-admin" password="my-postgres-password" url="jdbc:postgresql://my-database-ip:5432/template1" user-name="postgres">

Now configure the JRE for OpenNMS and then run the installation:

# /usr/share/opennms/bin/runjava -s
# /usr/share/opennms/bin/install -dis

Install the iplike package to optimize lookups based on IP addresses, and then start the service:

# /usr/sbin/install_iplike.sh
# systemctl start opennms

If all is well, you should be able to browse to your instance of OpenNMS at http://my-server-ip:8980 and log in with admin/admin.

You can check that all the daemons are running with:

# opennms -v status

Note that OpenNMS uses two ports that you should be aware of: an httpAdaptor at 8181 and the application itself running on Jetty at 8980.

Once you have completed the install, you may want to comment out the OpenNMS repository that you added to APT, to prevent unsolicited updates of OpenNMS, particularly in a Production environment.

Discover nodes

By default OpenNMS monitors no network nodes at all, so the very first thing to do is to register the nodes to be monitored.

You can do a range scan but that will register every device in the range, whether you want to manage it or not, so rather register the nodes explicitly.

There are a number of ways to do this, but since you already have the IP address, the easiest is to register it directly:

Admin > Configure OpenNMS > Configure Discovery > Specifics > Add new

This will open a pop-up where you enter the IP address of the node, in this case it would be my-server-ip.

Click on 'Add", then when the pop-up closes, click on 'Save and Restart Discovery'.

Wait a moment while the discovery daemon runs, then list the node that you registered:

Info > Nodes

Now that the node has been discovered, I will show you how to monitor the JVM using JMX and SNMP in a later post. In the meantime, play around with the Service Level Management functionality in OpenNMS that comes out of the box.

Tomcat 8 on Debian Jessie with PSI Probe and OpenNMS to monitor performance (Part 2)

In Part 1 of this series of three, I looked at the installation of Tomcat 8 on Debian Jessie. The standard Tomcat Manager web application is limited in functionality, and the look-and-feel is distinctly lacking, so here is PSI Probe instead.

PSI Probe features

PSI Probe supplements the basic functionality of Tomcat Manager with a number of very useful features, including:

• real-time monitoring of request traffic on the Connectors, from any Remote IP, or per application;
• viewing of Data Source pool usage, and query execution;
• monitoring of Logs, with the ability to dynamically change log levels at runtime;
• viewing the Thread execution stack, with the option of killing threads;
• dashboards to monitor the JVM, showing memory usage, swap file usage and garbage collection control;
• and detailed System properties data.

PSI Probe is a fork of an older application called Lambda Probe that was last updated in 2006. This is a Lambda Probe:

Installing PSI Probe

You have to build PSI Probe from the source code because it is not available as a Debian package. It is a Java application, so it is a simple matter of cloning the project from GitHub and running Maven:

# git clone https://github.com/psi-probe/psi-probe

Check that you are using Maven 3, and upgrade if necessary, because the build will fail if you use Maven 2. Don't forget to purge maven2 if you have to upgrade.

# mvn -version
Apache Maven 3.0.5
Maven home: /usr/share/maven
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
Default locale: en_ZA, platform encoding: UTF-8
OS name: "linux", version: "3.16.0-4-amd64", arch: "amd64", family: "unix"

If all is well, change to the PSI Probe directory and build the WAR file:

# cd psi-probe
# mvn package

The build will take some time the first time that you run it, because of all the JARs that must be downloaded from Maven Central to your local repository.

When it is done, use the Tomcat manager to deploy the war. Browse to http://my-server-IP:8080/manager, login with the same manager-gui role that you created in Part 1, scroll down to the section titled "Deploy" and use the option "Select WAR file to upload" to deploy the WAR file which you will find at psi-probe/web/target/probe.war.

Once it is deployed, browse to http://my-server-IP:8080/probe, login with the same manager-gui role, and dig around your Tomcat 8 installation. Much better looking, isn't it?

In Part 3, the last of this series, I will explain how OpenNMS can be used with SNMP to monitor Tomcat 8 and generate exceptions if any thresholds are exceeded.

Tomcat 8 on Debian Jessie with PSI Probe and OpenNMS to monitor performance (Part 1)

This is the first of three posts about the latest version of Apache Tomcat, still the most widely used application server, on Debian Jessie, with PSI Probe to manage it and OpenNMS to monitor its up-time.

What's new in Tomcat 8?

Debian Jessie was released with Apache Tomcat 8, which provides the following:

• Java Servlet 3.1, an incremental release of the servlet specification that includes non-blocking I/O to improve scaleability and an HTTP protocol upgrade mechanism that allows the client and the server to negotiate a transition from HTTP 1.1 to some other new chosen protocol;
• JavaServer Pages 2.3, a maintenance release to provide support for Expression Language (EL) 3.0 and to use the functionality of the Servlet 3.1 API;
• Java Unified Expression Language (EL) 3.0, which allows EL to run in a standalone mode outside of servlets or JSPs, plus lambda expressions and other Java 8 goodies;
• WebSocket 1.1, which allows full duplex communications over TCP so that bidirectional data can flow at the same time;
• A single, common resources implementation that merges Aliases, VirtualLoader, VirtualDirContext, JAR resources and external repositories into a single framework rather than a separate one for each feature;
• and Java EE 7 as a minimum pre-requisite, preferably Java EE 8, which has better support for HTML5, WebSockets, JSON and RESTful services.

Tomcat 8 is more flexible and better equipped to support web services than its predecessor, so let's proceed to the upgrade.

Installing Tomcat 8

Installing Tomcat 8 on Debian Jessie is dead simple. First, SSH to your web server (called 'my-server-IP' here) as root, and  execute 'java -version' to check that OpenJDK 7 is installed as a minimum:

# java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-1~deb8u1)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

If you don't see OpenJDK 7, you will have to install it and set it to be the default JVM:

# apt-get install openjdk-7-jdk openjdk-7-demo openjdk-7-source
# update-alternatives --config java

If you prefer, you can install OpenJDK 8 from the jessie-backports repository instead of OpenJDK 7, but it makes no difference to Tomcat 8.

Now install Tomcat 8:

# apt-get install tomcat8 tomcat8-admin tomcat8-docs tomcat8-examples tomcat8-user 

If all goes well, the service should start automatically and you should be able to browse to http://my-server-IP:8080 to see the default page:

Configuring Tomcat 8

Once you have installed Tomcat 8, you must enable the manager webapp, and you may optionally tune the JVM memory settings.

Edit /etc/tomcat8/tomcat-users.xml to include a manager (called 'my-user' here), with the appropriate roles to use the web console manager:

# vi /etc/tomcat8/tomcat-users.xml
<tomcat-users>
   <role rolename="manager-gui"/>
   <role rolename="admin-gui"/>
   <user username="my-user" password="my-user-password" roles="manager-gui,admin-gui"/>
</tomcat-users>

The manager-gui role has the highest privileges, allowing you to use the web console manager to deploy and undeploy apps, view stats, generate leak detection diagnostics, expire sessions, and so on. The admin-gui role is needed to access the virtual host manager.

Optionally edit the default Tomcat 8 configuration to make sure that JAVA_HOME is correctly set and to tune the performance of the JVM:

# vi /etc/default/tomcat8
JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
JAVA_OPTS="-Djava.awt.headless=true -Xms2048m -Xmx2048m -XX:+UseConcMarkSweepGC"

Now restart Tomcat 8, using the systemd way of managing services:

# systemctl restart tomcat8

and browse to http://my-server-IP:8080/manager.

Login with the username you created in tomcat-users.xml to access the Web Application Manager, an eye-watering horror of mustard and green that we will quickly replace with PSI Probe in Part 2 of this series, a much better-looking web console manager with a raft of extra functionality.

Location of Tomcat 8 directories on Debian

Note that the Debian package maintainers have a standard Debian way of doing things so the Tomcat 8 directories may not be where you expect them to be. The relevant locations are:

/etc/default: the initial default settings such as the Tomcat user ID, JAVA_HOME, JVM settings, and so on, are in the file 'tomcat8'.

/etc/tomcat8: the server configuration files are here, including context.xml, server.xml and tomcat-users.xml.

/etc/logrotate.d: the log settings for catalina.out are in the file 'tomcat8'.

/usr/share/tomcat8: this is $CATALINA_HOME, the root of the Tomcat installation, also known as $TOMCAT_HOME. It has startup, shutdown, and other scripts, as well as the Tomcat JARs in $CATALINA_HOME/lib (the original project to develop the servlet engine was called Catalina).

/var/lib/tomcat8: this is $CATALINA_BASE, also known as $TOMCAT_BASE, which holds instance-specific directories for web applications, with the code in $CATALINA_BASE/webapps.

/var/log/tomcat8: all the log files are stored here.

Now stay tuned for the installation of PSI Probe in Part 2.

Lightweight web-based transactional systems

Reanimating a legacy system

I have a set of data from an old legacy application that must be resurrected and web-enabled. The data was extracted from a relational database so it is structured in a conventional third-normal form, but the original application was pensioned off a long time ago, and probably a good thing too.

The user interface and its business logic must be rebuilt as a web-based transactional application so that the data can be accessible from a variety of devices, so I had to decide on a technology stack, but how? There are dozens of database systems, hundreds of software languages, scores of web-development frameworks, and more opinions about the matter than there are trolls on 9GAG.

The formal approach to making a decision would be to consider all the various technologies at each layer of the application stack and then weigh up the pros and cons of each.

A less formal approach would be to ask someone who has done this sort of thing before, like a grizzled Unix veteran or an 31337 h4x0r. 

In the end I applied a divide and conquer algorithm to eliminate dozens of decision points in a few strokes.

MVC or not?

The first thing to decide is the core structure of the application. The application must be web-based, and most dynamic web-sites are based on a Model-View-Controller architecture, but is this pattern still relevant today? Is it applicable to my use case?

I have a data set that must be viewed and maintained because of its historical value, so the application must have a Model. It is not a utility program that has no persistent data. 

It must be a web-based system, accessible from various user interfaces like a browser or smartphone app, so it must have a View. This is not a command-line application. 

Lastly, the data set will be the subject of various views, some of which will be used to maintain and update the data, so the system must have a Controller to manipulate the Model.

So the MVC pattern is most the logical structure for a web-based transactional application.

There are alternatives, like Facebook's Flux, but they are usually just refinements of the MVC pattern, although Joëlle Coutaz introduces a hierarchy of MVC-style layers in her Presentation–Abstraction–Control pattern which is useful for applications that need complex client tiers.

One thing to bear in mind is that web applications use HTTP which is a stateless protocol, so the system must have a Controller that is able to maintain state.

Open or closed source software?

Your average punter may not care whether they have access to the source code of the software or not, but if you are a software developer, you certainly should care. A mechanic would not buy a car with an engine that was locked up, so why would an I.T. professional use proprietary software? I want to see inside, dammit. I might not know what I am looking at, but I want to poke around anyway (and yeah, I know that Blogger is not open source but I will leave that fight to Richard Stallman).

So goodbye to products from Microsoft, Oracle, Adobe, and IBM, amongst others.

Okay, this binary chop between open and closed software means that in terms of the operating system, I am down to Linux, a BSD variant, OpenSolaris, or something less well known like Minix, Darwin, Plan 9 and the rest.

Let's be sensible and stick to Linux. I am currently using Debian Wheezy but we can argue about the distro later. 

Relational or NoSQL?

The next binary chop involves the data layer. My raw dataset is already in a highly normalised relational format, so it makes no sense to convert it to one of the NoSQL databases. Just import it into a relational database. But which one? 

The major open source relational databases are MySQL, MariaDB (which was forked from MySQL), and PostgreSQL. I don't trust Oracle with the custodianship of open source software, so that eliminates MySQL, and I don't see compelling technical arguments for MariaDB over PostgreSQL, so PostgreSQL it is. 

Static or dynamic type-checking? Functional or procedural? Compiled or interpreted?

I need a programming language to manipulate the model. The application needs a state engine, mechanisms to provide transactional integrity and role-based access control, so a server-side template engine is not going to cut it. I have to carve code, but what sort of code?

Entering into a discussion about the merits and demerits of a programming language is like competing in a bog snorkelling championship. There are no winners, just a bunch of cold, wet, exhausted people, covered in mud and weeds.

We can nail down some basic requirements though, like readability. Nobody wants to maintain code that looks like it was written by someone with bits of toast stuck in the keyboard, so that excludes Turing tar-pits like Brainfuck or functional languages like Lisp.

Variable typing is another. I like a variable to be clear about its role in a piece of code, and so do compilers. A language with static types will have fewer surprises at runtime and a compiler can get cracking with its optimisations straightaway, so I decided to go for Java because I am creating a web site, not writing a device driver, and C++ makes me feel seasick.

Java might not be as elegant as Haskell and it has its detractors, but it has a just-in-time compiler, garbage collection, a raft of libraries for reuse, and Java 8 introduces lambda expressions which reduces bulky anonymous inner classes to single expressions. Not too shabby then.

Bare bones or the full Monty? 

Okay, so now I need a Java application server that runs on Linux. The choices are Jetty, Geronimo, TomEE, GlassFish, Enhydra, Resin, JOnAS, JBoss EAP (now Redhat) or WildFly.

But do I really need a full blown application server? I want a bare-bones solution, simple, but not too simple, so I decided to stick with Apache Tomcat because it is one of the most widely used application servers and, although it is just a servlet container, it can be beefed up by adding other components of the Java EE stack as required. Well, within reason, unless you want to recreate TomEE on your own.

As a bonus the Tomcat API documentation is full of terms like Valve, Filter, Container, Pipeline and Engine which is what you want to hear in a software workshop.

Web frameworks for the JVM

Unfortunately I could not find a cleavage plane to divide up the hardest decision: which Java web development framework to use.

Community-driven or standards-driven? Component-based or request-based? Rich responsive user interface or server-side rendering? Cutting-edge or tried-and-tested? 

Matt Raible has done excellent work sketching out the landscape, as have the guys at ZeroTurnaround, so based on their spade work, I see that if this was a Reality TV contest called Survivor JVM 2014, SpringMVC would be leading the pack, with JavaServer Faces hard on its heels. Also in the race are Grails, Vaadin, Google Web Toolkit, Play, and Struts.

There are no right or wrong decisions at this point, they will all do the job, but I have to choose one.

SpringMVC is a request-based MVC framework so it provides a lot of control of the client-side HTML, CSS and JavaScript, but it does not follow the Java EE standard. It did give J2EE a good kick in the pants though, which was ultimately a good thing for web development in general.

Grails makes good reuse of Spring and Hibernate but I really don't want to wrap my head around Groovy, even though it is a superset of Java and runs on the JVM.

Play looks interesting, and I can see the attraction of convention over configuration, but if I am not using Scala, what are the advantages of abandoning the servlet specification?

Struts paved the way for other Java frameworks and is still widely used as a consequence, but it has long since been overtaken by the others.

Vaadin uses GWT widgets and they both produce very rich user interfaces. Vaadin is particularly impressive and is from Finland, like gravlax and the reindeer it is named after, but, like gravlax, it is too rich to eat every day.

So that leaves JavaServer Faces.

Like all the other frameworks, JSF has its detractors, including James Gosling himself, but it provides a Controller with a well documented request-response life-cycle that can manage state, and it has had a new lease on life thanks to component libraries like PrimeFaces.

Changing landscape

Having settled on JSF, it is worth noting that the View layer of Java web applications is currently in a state of flux.

The Java EE 8 spec includes an action-based MVC, but server-side web application frameworks are coming under pressure from HTML5 and client-side JavaScript MVC frameworks like AngularJS and Backbone that communicate with the server using JSON over REST or WebSocket. In fact, there is a project called AngularFaces that tries to combine AngularJS with JSF. 

The key is to make sure that the web application has a very clean separation of concerns so that if the decision to use JSF gets overtaken by events it can be stripped out without leaving too much damage to the remaining parts of the stack.

So the technology stack looks like this: a View consisting of HTML, CSS and JavaScript running in a browser, a Controller consisting of JavaServer Faces in a Tomcat servlet container, and a Model provided by PostgreSQL. The server will run on Linux.