Mark Turansky

Business, software, and gardening. That's all.

Archive for category Engineering

Printable Design Patterns Quick Reference Cards

Jan 22

Posted by Mark Turansky in Design, Engineering, Technology | 9 Comments

The Gang of Four design patterns have been elegantly distilled into a quick reference guide suitable for printing on 8.5 x 11.

You can get a larger version for your office wall, too. Check out the poster size. It’s perfect for any software organization.

I’ve posted low-resolution versions of the two cards here with the author’s permission. Links to the high-res printable versions are below on the author’s website.


designpatterns2_sm.jpg designpatterns1_sm.jpg

Jason McDonald created these high-quality reference cards. Click here to view the printable high-resolution images.

Canaries in the coal mine

Jan 21

Posted by Mark Turansky in Architecture, Engineering | 4 Comments

Application logging is only as useful as your plan to actually use the logs. Without a plan to mine the data, collect metrics, and plot graphs, your logs are useless. It’s snowcrash in a console window. It’s gigs of spam in a file.

This reminds me of the Philosophy and Zen of Unix:

Rule of Silence: When a program has nothing surprising to say, it should say nothing.

But how do you know your program is running? You’ve got several options available to you, all good, and you should probably implement them all.

Canaries in the coal mine

In the good ol’ days, miners had a crude but effective way to test a mine shaft for adequate levels of oxygen: if the bird died, miners got out of the shaft.

Your program needs a canary in the coal mine. You need a way to smoke test your application when it first boots up and while it’s running. It either works or it doesn’t. The bird is dead, dying, or singing.

What kind of canary? One that tests some discrete bit of functionality of your application. You can use a simple site monitoring program with basic tests baked into a server page. You can run a load testing tool like JMeter to script mimicking what an end user would do. Just run a test of 1. In the messaging application I’m currently building, we send periodic test messages to the queues. The messages aren’t fancy, just tiny XML messages posted from a Python client requesting 2+2.

But 2+2 is important. It’s like your first Hello, World! program in a new language. Getting 2+2 running means you’ve successfully setup your environment, you understand the basics of compilation, packaging, deployment, and configuration management. You’ve also got your first benchmark of how quickly a simple message can pass through your system.

You want to log the data from the canaries in the coal mine. Ping your canaries every ten minutes. Keep those results and metrics. Create a plan to report on them, which puts you on the path of Statistical Process Control.

Statistical Process Control

You can spend a decade trying to attain CMM Level 5 accreditation or gaining your black belt in Six Sigma and probably still never completely grok the enormity of Statistical Process Control. You can, however, start improving your technical operations by using meaningful statistics to smooth out your Configuration Management practices.

So what is Statistical Process Control? Quoting Wikipedia:

Statistical Process Control (SPC) is an effective method of monitoring a process through the use of control charts. Much of its power lies in the ability to monitor both process centre and its variation about that centre. By collecting data from samples at various points within the process, variations in the process that may affect the quality of the end product or service can be detected and corrected, thus reducing waste and as well as the likelihood that problems will be passed on to the customer. With its emphasis on early detection and prevention of problems, SPC has a distinct advantage over quality methods, such as inspection, that apply resources to detecting and correcting problems in the end product or service.

In addition to reducing waste, SPC can lead to a reduction in the time required to produce the product or service from end to end. This is partially due to a diminished likelihood that the final product will have to be reworked, but it may also result from using SPC data to identify bottlenecks, wait times, and other sources of delays within the process. Process cycle time reductions coupled with improvements in yield have made SPC a valuable tool from both a cost reduction and a customer satisfaction standpoint.

In layman’s terms, you aggregate the data from your canaries into a graph. You watch the graph every day to eventually find your “center”, the normal singing voice of your canary. Your data tells you he sings at X decibels when healthy, and your graphs show you when there’s not enough oxygen in the mineshaft.

This entire process should be automated! If it’s not, you won’t do it. Your Ops center and CM folks should have at least one box set aside for automation and monitoring. Maybe it’s your build box. Put all your scripts there. Create cron jobs or Windows scheduled tasks to constantly parse your log files for data. Use Log4J’s JMS or JDBC Appender if you don’t want to parse text files. Get all your data in one place, mine it, and graph it.

Test Driven Deployment

Everyone knows about Test Driven Development, where you write your test code before your write your business logic. It forces you to actually design your code by making you interact with the class/object early in the process. Many preach TDD, some actually practice it.

I’m not personally aware of many architects or organizations that practice what I call Test Driven Deployment. This is the habit of understanding what your canaries are before you write your application. You will change how you architect, design, and deploy your software if you understand up front what data you want to capture and how you’ll access it. It forces you to design your solution before you try to implement it, just like Test Driven Development.

Pipe [debug] and [info] level logging to /dev/null

Divide your logging output into discrete files with meaningful names. Canaries and metrics can go to one set of files. Application errors and contextual information to help diagnose bugs should go to another file. Debugging output goes right to the black hole.

Make your log files useful. Practice Test Driven Deployment. Bring a canary with you down into the coal mine, and listen when he stops singing.

HOW TO: Bootstrap Java programs in isolated classloaders

Jan 11

Posted by Mark Turansky in Code Hints, Engineering, HOW TO | 16 Comments

Bootstrapping is the process by which you load a very small and very simple pure java program with no dependencies that, in turn, loads, configures, and runs more complex programs with varying dependencies. Bootstrapping lets you run your container without polluting the system classpath. This allows you to run your deployed applications with the unpolluted system classpath as its parent. You’ve achieved classloader isolation.

When would you want to bootstrap? Any time you want an unpolluted system classpath, which I’m finding is often convenient.

Let’s say you want to write some kind of middleware product, a container of some sort that deploys other applications within it. You will run into classloading issues. The dependencies that your container has (say, Spring 2.0.6) may not be what your deployed application requires (maybe, Spring 1.2.6). You will find that you cannot have commons-logging in both applications (container and child). There are many ways to encounter java.lang.LinkageErrors. It’s very easy to cross the streams when running in a mutli-app environment.

What you want to do is load your container and deployed apps in splendid isolation from each other. How do you do that? Bootstrapping!

Here’s how you bootstrap… 


import java.io.File;
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.ArrayList;
import java.util.List;

public class Bootstrap {

    public static void main(String[] args) throws Exception {

        /*
            Assume your application has a "home" directory
            with /classes and /lib beneath it, where you can put
            loose files and jars.

            Thus,

            /usr/local/src/APP
            /usr/local/src/APP/classes
            /usr/local/src/APP/lib
         */

        String HOME = "/usr/local/src/YOURAPP";
        String CLASSES = HOME + "/classes";
        String LIB = HOME + "/lib";

        // add the classes dir and each jar in lib to a List of URLs.
        List urls = new ArrayList();
        urls.add(new File(CLASSES).toURL());
        for (File f : new File(LIB).listFiles()) {
            urls.add(f.toURL());
        }

        // feed your URLs to a URLClassLoader!
        ClassLoader classloader =
                new URLClassLoader(
                        urls.toArray(new URL[0]),
                        ClassLoader.getSystemClassLoader().getParent());

        // relative to that classloader, find the main class
        // you want to bootstrap, which is the first cmd line arg
        Class mainClass = classloader.loadClass(args[0]);
        Method main = mainClass.getMethod("main",
                new Class[]{args.getClass()});

        // well-behaved Java packages work relative to the
        // context classloader.  Others don't (like commons-logging)
        Thread.currentThread().setContextClassLoader(classloader);

        // you want to prune the first arg because its your main class.
        // you want to pass the remaining args as the "real" args to your main
        String[] nextArgs = new String[args.length - 1];
        System.arraycopy(args, 1, nextArgs, 0, nextArgs.length);
        main.invoke(null, new Object[] { nextArgs });
    }

}

You can try this code out for yourself. Cut & paste the bootstrap code above into your favorite IDE, put that single Bootstrap.class onto your classpath, and run it like so:


java -cp . Bootstrap sample.HelloWorldMain Hello!
.

Click here to download the sample /usr/local/src/YOURAPP application.
Tip for Windows users, you can make the path c:\usr\local\src\YOURAPP it’ll work.

Tags: , ,

Horses for courses

Dec 9

Posted by Mark Turansky in Code Hints, Design, Engineering, Python | 4 Comments

Pick the right tool for the job.  It can save you a lot of time and effort.  This article explains how I replaced dozens of Java class files, an XSD, myriad library dependencies, a build script and a whole lot of complexity with 40 lines of Python.

A friend of mine recently bemoaned my use of the term “pick the right tool for the job.” Apparently, he thinks it’s a tired metaphor and a boring cliche. And so in deference to my friend, I will instead talk about choosing the right horse for the course.

Some horses are mudders, some are good at short-distance sprints, still others are the ones you want for the long race. If you’re eying the race’s purse, you choose the right horse for the course.

So, how do you choose the right horse? I’m guessing it has something to do with experience, breadth of knowledge, and understanding when a functional program makes more sense than an imperative one or knowing when a script is better than a fully engineered OO implementation. Here’s an example to fan the Java vs. Python flames.

Not too long ago, in my previous life working for a consulting company, I wrote a small Java application to monitor our many web applications. The requirements were simple: the server page should return “OK” (text/plain) else the contents of the entire page would be mailed to a list of interested recipients. This technique allows a developer to put whatever test they want in their server page (database connectivity, unit tests, whatever) knowing that any exception they write to the page would be mailed to them.

Not hard, right? Easy implementation? It was both times I wrote it, but one of them was a much better horse for the course.

The configuration file in Java was XML, natch, which required an XSD. The XSD required JAXB(or Castor, XMLBeans, etc.) to generate bindings for the XML. The Main program was well factored in that each class did one thing well. As a result, I had a class to poll a site, one of load the configuration, a class to send email, etc. Between Main and classes from Castor, I was up to a dozen .java files. Main, of course, required libraries (Castor, mail.jar, activation.jar, etc.), and those libraries required a script (.cmd in our case) to load them all onto the classpath. Oh, let’s not forget about building with Ant. Add a build file to the heap.

It worked, but damn, that’s a lot of files and jars and complexity for a simple monitoring program!

I rewrote it in Python in less than 40 lines of code. Two files. There are more comment lines than executable code.

Choose the right horse for the course. Occam’s Razor says the 40 line solution is the right one, but you can decide for yourself.

Here’s the python program and configuration file.

Many developers learn one or two mainstream languages and always run their favorite horse, irrespective of course. The best developers will be those that love learning new languages and techniques. I’m a better Java programmer today because of what I’ve learned from Python. I’m a better web developer today because of what I’ve learned from Ruby on Rails.

Learn more. Broaden your horizons. Try new things and new styles of development. Learn to pick the best horse for the course. Always use the right tool for the job.

Tags: , , ,

Switch to our mobile site