Friday, 5 December 2014

Lightweight web-based transactional systems

Reanimating a legacy system

I have a set of data from an old legacy application that must be resurrected and web-enabled. The data was extracted from a relational database so it is structured in a conventional third-normal form, but the original application was pensioned off a long time ago, and probably a good thing too.

The user interface and its business logic must be rebuilt as a web-based transactional application so that the data can be accessible from a variety of devices, so I had to decide on a technology stack, but how? There are dozens of database systems, hundreds of software languages, scores of web-development frameworks, and more opinions about the matter than there are trolls on 9GAG.

The formal approach to making a decision would be to consider all the various technologies at each layer of the application stack and then weigh up the pros and cons of each.

A less formal approach would be to ask someone who has done this sort of thing before, like a grizzled Unix veteran or an 31337 h4x0r. 

In the end I applied a divide and conquer algorithm to eliminate dozens of decision points in a few strokes.

MVC or not?

The first thing to decide is the core structure of the application. The application must be web-based, and most dynamic web-sites are based on a Model-View-Controller architecture, but is this pattern still relevant today? Is it applicable to my use case?

I have a data set that must be viewed and maintained because of its historical value, so the application must have a Model. It is not a utility program that has no persistent data. 

It must be a web-based system, accessible from various user interfaces like a browser or smartphone app, so it must have a View. This is not a command-line application. 

Lastly, the data set will be the subject of various views, some of which will be used to maintain and update the data, so the system must have a Controller to manipulate the Model.

So the MVC pattern is most the logical structure for a web-based transactional application.

There are alternatives, like Facebook's Flux, but they are usually just refinements of the MVC pattern, although JoĆ«lle Coutaz introduces a hierarchy of MVC-style layers in her Presentation–Abstraction–Control pattern which is useful for applications that need complex client tiers.

One thing to bear in mind is that web applications use HTTP which is a stateless protocol, so the system must have a Controller that is able to maintain state.

Open or closed source software?

Your average punter may not care whether they have access to the source code of the software or not, but if you are a software developer, you certainly should care. A mechanic would not buy a car with an engine that was locked up, so why would an I.T. professional use proprietary software? I want to see inside, dammit. I might not know what I am looking at, but I want to poke around anyway (and yeah, I know that Blogger is not open source but I will leave that fight to Richard Stallman).

So goodbye to products from Microsoft, Oracle, Adobe, and IBM, amongst others.

Okay, this binary chop between open and closed software means that in terms of the operating system, I am down to Linux, a BSD variant, OpenSolaris, or something less well known like Minix, Darwin, Plan 9 and the rest.

Let's be sensible and stick to Linux. I am currently using Debian Wheezy but we can argue about the distro later. 

Relational or NoSQL?

The next binary chop involves the data layer. My raw dataset is already in a highly normalised relational format, so it makes no sense to convert it to one of the NoSQL databases. Just import it into a relational database. But which one? 

The major open source relational databases are MySQL, MariaDB (which was forked from MySQL), and PostgreSQL. I don't trust Oracle with the custodianship of open source software, so that eliminates MySQL, and I don't see compelling technical arguments for MariaDB over PostgreSQL, so PostgreSQL it is. 

Static or dynamic type-checking? Functional or procedural? Compiled or interpreted?

I need a programming language to manipulate the model. The application needs a state engine, mechanisms to provide transactional integrity and role-based access control, so a server-side template engine is not going to cut it. I have to carve code, but what sort of code?

Entering into a discussion about the merits and demerits of a programming language is like competing in a bog snorkelling championship. There are no winners, just a bunch of cold, wet, exhausted people, covered in mud and weeds.

We can nail down some basic requirements though, like readability. Nobody wants to maintain code that looks like it was written by someone with bits of toast stuck in the keyboard, so that excludes Turing tar-pits like Brainfuck or functional languages like Lisp.

Variable typing is another. I like a variable to be clear about its role in a piece of code, and so do compilers. A language with static types will have fewer surprises at runtime and a compiler can get cracking with its optimisations straightaway, so I decided to go for Java because I am creating a web site, not writing a device driver, and C++ makes me feel seasick.

Java might not be as elegant as Haskell and it has its detractors, but it has a just-in-time compiler, garbage collection, a raft of libraries for reuse, and Java 8 introduces lambda expressions which reduces bulky anonymous inner classes to single expressions. Not too shabby then.

Bare bones or the full Monty? 

Okay, so now I need a Java application server that runs on Linux. The choices are Jetty, Geronimo, TomEE, GlassFish, Enhydra, Resin, JOnAS, JBoss EAP (now Redhat) or WildFly.

But do I really need a full blown application server? I want a bare-bones solution, simple, but not too simple, so I decided to stick with Apache Tomcat because it is one of the most widely used application servers and, although it is just a servlet container, it can be beefed up by adding other components of the Java EE stack as required. Well, within reason, unless you want to recreate TomEE on your own.

As a bonus the Tomcat API documentation is full of terms like Valve, Filter, Container, Pipeline and Engine which is what you want to hear in a software workshop.

Web frameworks for the JVM

Unfortunately I could not find a cleavage plane to divide up the hardest decision: which Java web development framework to use.

Community-driven or standards-driven? Component-based or request-based? Rich responsive user interface or server-side rendering? Cutting-edge or tried-and-tested? 

Matt Raible has done excellent work sketching out the landscape, as have the guys at ZeroTurnaround, so based on their spade work, I see that if this was a Reality TV contest called Survivor JVM 2014, SpringMVC would be leading the pack, with JavaServer Faces hard on its heels. Also in the race are Grails, Vaadin, Google Web Toolkit, Play, and Struts.

There are no right or wrong decisions at this point, they will all do the job, but I have to choose one.

SpringMVC is a request-based MVC framework so it provides a lot of control of the client-side HTML, CSS and JavaScript, but it does not follow the Java EE standard. It did give J2EE a good kick in the pants though, which was ultimately a good thing for web development in general.

Grails makes good reuse of Spring and Hibernate but I really don't want to wrap my head around Groovy, even though it is a superset of Java and runs on the JVM.

Play looks interesting, and I can see the attraction of convention over configuration, but if I am not using Scala, what are the advantages of abandoning the servlet specification?

Struts paved the way for other Java frameworks and is still widely used as a consequence, but it has long since been overtaken by the others.

Vaadin uses GWT widgets and they both produce very rich user interfaces. Vaadin is particularly impressive and is from Finland, like gravlax and the reindeer it is named after, but, like gravlax, it is too rich to eat every day.

So that leaves JavaServer Faces.

Like all the other frameworks, JSF has its detractors, including James Gosling himself, but it provides a Controller with a well documented request-response life-cycle that can manage state, and it has had a new lease on life thanks to component libraries like PrimeFaces.

Changing landscape

Having settled on JSF, it is worth noting that the View layer of Java web applications is currently in a state of flux.

The Java EE 8 spec includes an action-based MVC, but server-side web application frameworks are coming under pressure from HTML5 and client-side JavaScript MVC frameworks like AngularJS and Backbone that communicate with the server using JSON over REST or WebSocket. In fact, there is a project called AngularFaces that tries to combine AngularJS with JSF. 

The key is to make sure that the web application has a very clean separation of concerns so that if the decision to use JSF gets overtaken by events it can be stripped out without leaving too much damage to the remaining parts of the stack.

So the technology stack looks like this: a View consisting of HTML, CSS and JavaScript running in a browser, a Controller consisting of JavaServer Faces in a Tomcat servlet container, and a Model provided by PostgreSQL. The server will run on Linux.

Sunday, 21 September 2014

Installing CyanogenMod 10.2 on a Samsung Galaxy Note 10.1 tablet (N8000) for Software Freedom Day

From broken screen to crapware

A couple of weeks ago I had an unpleasant on-line interaction with some troll that ended with my Samsung Galaxy Note 10.1 in a similar state to this one:

This is how much damage can be inflicted on an electronic device by jabbing it repeatedly with a rigid index finger. Learn from my experience.

After parting with a heart-stopping amount of money, I got the tablet back from the service centre with a new screen and a bunch of crapware because they "upgraded the software" as an additional service. 

I tried to uninstall some of these mysterious apps but they promptly reappeared of their own volition. Ironically, this happened on Software Freedom Day, which prompted me to think about the four essential freedoms in the Free Software Definition:

Freedom 0: the freedom to run the program as you wish, for any purpose.

I do not wish to run these programs at all. I have no purpose for them.

Freedom 1: the freedom to study how the program works, and change it so it does your computing as you wish.

I do not want to study these programs and I would like to change them to do my computing as I wish by removing them altogether.

Freedom 2: the freedom to redistribute copies so you can help your neighbor.

These programs are redistributing themselves freely but they are not helping me, my neighbours, or the dog, because he is barking at the fence.

Freedom 3: the freedom to distribute copies of your modified versions to others.

I can't modify these programs and I would never foist unwanted crapware on others.

So, today, the day after Software Freedom Day, I decided to replace the stock Samsung ROM and regain my lost software freedom.

CyanogenMod shall set you free

Fortunately, the dedicated crew at have produced an open source firmware distribution that runs on the Galaxy Note, so here are detailed instructions to install CyanogenMod 10.2, which is based on Android 4.3.1 (Jelly Bean), on a Samsung Galaxy Note 10.1 (GT-N8000) using a 64-bit desktop running Debian 7.6 (codename "wheezy").

Flashing non-stock firmware is not for the faint-hearted, but it is not too difficult if you follow the installation instructions exactly. There are four steps:
  • Download and install the software needed to do the flashing;
  • Flash the tablet with ClockworkMod Recovery to replace the stock recovery software;
  • Install the new CyanogenMod firmware; and
  • Re-install the Google apps.

Getting set up

Install ADB

The first step is to install Android Debug Bridge (ADB) which is a command line tool for interacting with the tablet over the USB cable. 

Edit the apt package manager's sources.list file to include the wheezy-backports repository so that you can install adb:

vi /etc/apt/sources.list

Then add the following entry, where is a nearby Debian mirror:

deb wheezy-backports main contrib non-free

Update the list of available packages and install the adb tools:

apt-get update
apt-get install android-tools-adb
apt-get install android-tools-fastboot

Install Heimdall

In order to flash the recovery firmware, you must download and install Heimdall (the software, I mean, not the music). Download it from here and install it:

dpkg -i debian7-heimdall_1.4.0-0_amd64.deb

Firmware downloads

Now download the software that you will flash to the tablet. Make sure that you are downloading the correct versions of each, otherwise it will all end in tears:

Okay, now flash CWM Recovery

Power off the tablet and wait a few seconds until it has died down completely. Don't plug in the USB cable yet.

Press Volume Down then the Power button and hold them down together until a Warning screen appears, then let them go.

The Warning tells you that a custom OS can cause critical problems and invites you to press Power Up if you are determined to proceed, which you will because you want your software freedom.

The tablet will go into stand-by mode:

In fact it is downloading nothing, it is just waiting for you to take the next step, so plug in the USB cable and use Heimdall to flash the Recovery image from the desktop to the tablet:

heimdall flash --RECOVERY clockworkmodrecovery.6046.n8000.img --no-reboot

A blue progress bar will appear on the device showing the recovery being transferred to the device. When the image has been flashed, unplug the USB cable, press Volume Up then Power, wait until the Samsung title bar appears, and release them both.

Make sure you reboot into recovery immediately otherwise the custom recovery will be overwritten and the device will reboot as though your custom recovery failed to install. All being well you should boot into the new custom Recovery.

Install CyanogenMod

Now you are in the Recovery mode, you can install the new firmware image, but first backup the stock ROM.

In ClockworkMod Recovery, you use the physical volume buttons to move up and down and the power button to confirm a menu selection, so use Volume Up to navigate to "backup and restore", confirm with the Power button, and then select "backup to SD card".

When the backup is done go back to the main menu and select "wipe data/factory reset".

Once that has completed, go back and select "install zip", followed by "install zip from sideload". Sideload will wait until you have plugged in the USB cable and entered on the desktop:

adb sideload

When the sideload is finished, the tablet will find the update package and install it. Note that it does not display an "Install complete" message. You can tell the install is complete if there were no fatal error messages and you have regained control over the menu.

Go back to the main menu and reboot into CyanogenMod. After an agonising wait, you should see:

Finally, Google Apps

Boot the tablet into recovery mode (Volume up, then Power, wait for Samsung title bar, then release them both) and select "install zip", followed by "install zip from sideload". Then on your desktop enter:

adb sideload

Now reboot to your stripped down, open source Galaxy Note and enjoy being free to control your own software.

Thursday, 12 June 2014

Setting up a Dashboard using Bootstrap

Bootstrap dashboard

I have a number of web-based tools to monitor and maintain the various devices and applications on my network. It would be very convenient to integrate the tools with an administration console hosted on a secured web server that will let me in from the outside but keep the script kiddies out. 

I decided to use Bootstrap to build a dashboard to do that. It is a front-end web development framework that was written by Mark Otto and Jacob Thornton at Twitter to standardize their development environment and share common design patterns and code. 

Why Bootstrap and not some other web framework?
  • It was released under the MIT open source licence in August 2011 on GitHub, so you are free to download and use the code, modify it, and distribute it, as long as you respect the license terms
  • It uses HTML 5 which makes the site design very responsive irrespective of the form factor of the device, whether desktop or mobile;
  • There are many free templates to get you going, and there is a market of premium templates if you want to pay for upscale designs;
  • It is simple, elegant, and easy to get up and running.

Install Bootstrap 3.1.1

Download Bootstrap

Use your favourite browser to go to the very minimalist Bootstrap home page and click on Download Bootstrap:

You will be given a choice of download options. Choose Download Bootstrap to download the precompiled and minified versions of Bootstrap's CSS, JavaScript, and fonts:

This download will not include documentation or the original source code files, so you don't have to bother compiling other files. Later, you can use the other download options to get the source code and play around with the Less compiler.

Install Bootstrap

Installation is simple: copy the download to your working directory, unzip it and rename it.

Create the index page

This Bootstrap download does not include a sample default index page, so using a text editor create a text file called index.html under the parent directory, "Bootstrap" in this example, and include this HTML code from the Bootstrap basic template:

Open index.html with your browser and you should see "Hello, World!".

Now try these examples on the Bootstrap site to get you fired up while I play around with templates for Admin themes.

Sunday, 8 June 2014

Yes, Another Blog About Technical Things

Another blog?

"A blog? Aren't there too many blogs?"

Absolutely. As of today, there are 188.7 million blogs on Tumblr, more than 70 million on, and nobody knows how many are on Blogger because Google keeps that a secret.

"Then why are you blogging?  Who is going to read yet another blog?"

Well, I never update my FaceBook page because I am not very sociable. I do have an account on Google Plus even though my son thinks it consists of tumbleweed and the lost souls of Linux administrators, and I occasionally post there, but the format does not lend itself to long technical posts.

I decided to blog for two reasons. Firstly, the notes I make while I wait for the kettle to boil get archived in the universal bit bucket of digital entropy on a regular basis, so a blog on a server owned by Google is a fine off-site disaster recovery site.

Secondly, the hundreds of millions of blogs out there host a rich seam of thought nuggets that help enormously when faced with some seemingly intractable technical problem. By posting here I may save someone else from an afternoon of banging their head on a keyboard or rocking to and fro in despair.

Thirdly, it is important to know that off-by-one errors are a common source of software defects and neckbeard humour.

Who am I?

"But who are you? Why do you think you have something interesting to say?"

For many years I worked in a senior position in government and am now consulting in the more general field of Open Government. Working in the public service demanded a certain level of decorum and restraint so I was not able to swear freely or make pointed observations that some may feel to be controversial, but as a consultant I am at now at liberty to blog as I see fit.

But I will not go so far as to hurl invective at vendors like Linus Torvalds did, although his rant did spur NVIDIA to contribute more support to the Nouveau open source driver for their cards.

What is this blog about?

"Okay, so you are determined to blog, even though there are probably more than 300 million other blogs out there. What are you going to blog about?"

Getting your teeth into something new in the I.T. world involves extensive googling for prior art, but that search will return all sorts of flotsam and jetsam that you have to assemble into something that works.

Since I keep detailed notes as I work through a problem because the chances are good that I will face the same problem again sooner or later, I decided to write up my notes and post them here as a record of various stages of a journey to nowhere in particular.