Mark Gregory Turansky

In this blog article, Michael Nygard discusses a talk he attended where a technical architect discussed an SOA framework at FIDUCIA IT AG, a company in the financial services industry. Nygard describes an architecture that echoes many of the features I implicitly spoke of in my first blog article about my big integration project / message bus.

You may be asking yourself right now, why does he keep talking about this particular project? Briefly: it’s been a very fun project, it’s ongoing, it consumes most of my daily brain cycles, we’re still growing it (it’s a brand new infrastructure for us), and it encompasses a whole lot of ideas that I thought were good and that are now being validated by other projects I read about online.

So, what other unsung features did we build in that I’ll now sing about?

Asynchronous Messaging

You’ll notice the Spooler component in the original broad sketch of our architecture. The high-level description I gave the Spooler touched on callbacks. Asynchronous messaging was left unsaid, but it is implied by having a mechanism for callbacks.

The description also labeled my Spooler an endpoint, which may be a web service endpoint. We use web services only because the Enterprise Service Bus (ESB) orchestrating work on our bus is .NET-based while our project is all Java. That said, we post Plain Ol’ XML (POX) over HTTP, which is deserialized quickly to a Java POJO. Our entire messaging system works on POJOs, not XML.

The outside world may use SOAP (or XML-RPC or flat files or whatever) when communicating with my company, but internally our ESB talks POX with the bus. Mediation and transformation (from SOAP –> POX) is part of the functionality of an ESB. Consumers, internally to our bus, would directly access queues instead of using web services.

Pure POJOs, but distributed

It’s extremely productive and useful to work with a pure POJO model, and it’s even more productive and useful when the state of those POJOs is automagically kept in sync across the cluster regardless of what node is working on it. This is where Terracotta Server shines.

We pass POJOs around through all the queues. Consumers — which can exist anywhere on the network — process the Service/Job/Message (all interchangeable terms, as far as I am concerned — they are all units of work). Our messages are stateful, meaning they enter our bus empty except for parameters in instance variables, get routed around to various and sundry consumers across the network, and get posted back (the callback) full of data to the ESB.

Why do we need distributed POJOs? Well, we found it to be highly useful. For example, we offer a REST API to abort a pending message (such as http://ourendpoint/message/abort/abcdefg-the-guid-wxyz). The easiest way we found to tell the entire bus to disregard this message was to flip the bit on the message itself. The endpoint is running under Terracotta Server, all of the queues live in TC, and our consumers are likewise plugged in. If you stick all your messages in a Map (or series of maps if you’re worried about hashing, locking, and high volumes) where the GUID is the key and the value is the message, then the endpoint or any consumer can quickly obtain the reference to the message itself and alter its state. We can also write programs that hook into TC temporarily to inspect or modify the state of the system. Persistent memory is cool like that. It exists outside the runtime duration of the ephemeral program.

The endpoint likewise has REST APIs for returning the state of the bus, queues sizes, current activity, and other metrics. All of this data is collected from the POJOs themselves, because the endpoint has access to the very object instances that are running all over the network. It just so happens this architecture works wonderfully inside a single JVM, too, without TC, for easier development and debugging.

Load balancing and routers

Straight from Michael Nygard’s article:

Third, they’ve build a multi-layered middle tier. Incoming requests first hit a pair of “Central Process Servers” which inspect the request. Requests are dispatched to individual “portals” based on their customer ID.

In other words, they have endpoints behind load balancers (we use Pound) and “dispatched” is another word for “routed.” We have content based routers (a common and useful Enterprise integration Pattern for messaging systems) that route messages/services/jobs of specific types to certain queues. Our consumers are not homogenous. We’ve configured different applications (the integration aspects of our project) to listen on different queues. This saved us from having to port applications off the servers where they were previously deployed. These apps are several years old. Porting would have taken time and money. Allowing messages to flow to them where they already exist was a big win for us.

More to come

I’ve got the outline for my white paper complete, where I bulleted the features above as well as those in my previous blog article. There are other features I haven’t covered yet. Overall, I think it will be an interesting paper to read.

Still, I’m a little jealous, though, that FIDUCIA IT AG has scaled out to 1,000 nodes in their system. I can’t say how many nodes we’re up to, but I can say I’m looking forward to the massive scalability that our new architecture will give us.

I’ve been fascinated about the software they run on the space shuttle ever since I read this article years ago:  They Write the Right Stuff

Today, I ran across this article about Self-Modifying Code written by someone that used to work at Lockheed on the shuttle. He describes using it for fault tolerance down near the hardware.

I imagine the computers running the Federal Reserve have similarly robust features baked in.  Interesting stuff.

I vividly remember when my neighbor two doors down got an Atari 2600 in 1978. I was probably 4 1/2, maybe 5, but I remember the first time I saw Space Invaders. It was the only game they had. My brothers and I pooled our allowances for a while and bought our own. Combat quickly became my favorite game. You could ricochet bullets from the tanks off the walls! My brothers didn’t stand a chance.

Fast forward to 2008 and I decided to go find an Atari emulator. A few minutes later, I was playing Space Invaders again. Combat in the emulator wasn’t fun because I had no one to play with.

I really cut my computer teeth on the C64. I remember when I’d walked into Electronics Boutique in the mall and see a wall full of C64 games. A few years later, there was a small “IBC PC” games section. The C64 was great. I spent a lot of time playing games on my C64, but I’d also try to write programs. I remember I was able to make a few basic sprites move across the page, but I had no idea what sprites really were. I remember trying to type in programs I’d find in a book or computer magazine. They never worked. All that mattered to an 8 or 10 year old is playing games.

There was one game in particular that I really fell in love with on the C64: Ultima IV, Quest of the Avatar. It exposed me to my first game with any depth and to a persistent world that I’d live in while playing the game. I was mesmerized. I spent countless hours exploring every nook and cranny in Britannia, figuring things out (I copied the game and didn’t have the manual), immersing myself in the story, and I loved every minute of it. I was probably 11 or 12 years old.

A close friend of mine always had a PC. I remember playing games with him on his 286 and 386. Police Quest and Chopper Commando were our favorites. He had to start the games from the command line.

All the time we had a computer, we used it for word processing. I’d type book reports or other papers on the computer and print them on my dot matrix printer. During my early college days, I worked for a small financial planning firm where I maintained their software (by keeping versions up to date with frequent update disks from insurance carriers and other financial services firms) and created spreadsheets for the agents in Lotus 1-2-3. I’d never even heard of Excel, but I’d gotten my first exposure to early Windows. I think it was Windows 3.1.

I left school to enlist in the Navy where I qualified to train as a Navy Intelligence Specialist. After training, I deployed to the USS Independence in Japan, where I pulled intelligence reports from a computer in the SCIF for the ship’s intel officer. Little did I know it, but it was the internet. Well, it was the government’s classified version of the internet, but looking back now, I can clearly remember clicking on links, printing the pages, and preparing a report for the department Commander.

After the Navy, I went back to finish school. A few of my friends had just gotten email and they told me to look into it. I remember using Pine in the school library to read my email. I didn’t have much email then.

I don’t remember how I learned of it, but my life changed when I learned that Ultima Online existed. Here it was, the game I loved at a kid, the world I spent years exploring (literally, through Ultima 4, 5, & 6), in a new online game!

I bought my first computer explicitly for the purpose of playing Ultima Online. It had a 300Mhz processor, 64mb of RAM, and I forget the size of the drive. It ran Windows 98. And that was the biggest timesink I had heretofore discovered in my life.

Soon, I ran across Ultima Offline eXperiment (UOX), which was (still is) an open source version of UO Server. It was created by a group of hackers to run the UO client. It allowed someone to have their own private game server with a world devoid of people except for those you invite. I remember I organized a tournament with 8 people with me both playing and hosting the server. I didn’t know anything about performance then, but I can laugh at myself in retrospect for thinking I can host 8 active players in a networked game on a 300Mhz machine. It crashed all the time, but it didn’t matter. I was completely amazed that people could do this. I browsed the source code. I knew it was something called “C++”, but I had no idea what I was looking at, yet I thought it looked beautiful. It may have been a kludgefest of cruft for I know, but I fell in love with code, with how it looked, and with what it could do.

So I decided to learn what this code stuff was all about. I bought Sam’s Teach Yourself Java in 21 Days. I installed Java 1.1 and learned Hello, World. Soon thereafter I was writing a program to feed a Jabbywocky. I still don’t know what a Jabbywocky is. I didn’t finish the book. Some of the concepts were over my head, and I could tell that it was all trite and contrived. I wasn’t going to be able to run a UO Server emulator after reading that book.

Still, something stuck and I kept learning new things. I learned HTML, JavaScript, and then ASP (using JavaScript). My first job out of college required me to make reports, so I learned SQL to pull data. Then I applied my new ASP skills to automate the reports. I’m lazy and grew bored with report-making. A few years later, I learned Java for real.

Here we are, a decade later, and I’m busy integrating legacy applications into our shiny new message bus. It’s highly concurrent, runs all our integrated applications in a single JVM but in isolated classloaders, and my company is porting all our automation and data processing to my message bus for integration. It’s got massive horizontal scalability capabilities. Our Linux servers have multiple processors with multiple cores that are 100x faster than my first PC and have 500x as much memory.

This current project of mine is a long way from Space Invaders. I guess 30 years will do that for you. It’s been fun thinking about how I’ve been involved with computers and software in some way (even as a consumer) for my entire life. I’m looking forward to another 30 in high technology and I’m excited to play a part. I might even learn what a Jabberwocky is.