Paul:
Good point about development-time assertions. They are a typically overlooked strategy in most Java-land projects I’ve seen (my own included!). I think I’m going to make it a point when we get back to do a sweep of the codebase and strengthen our pre/postcondition assertions into actual Assert assertions. I believe we got pretty focused on the “try to work for as long as possible” aspect prematurely. Thanks for the sanity check.

Ram:
Another good point. For some reason, Mark didn’t mention in his post that we relied heavily upon remote debugging and thread dumps to track down these problems. I don’t know what we’d do without the JMX console and jdb… I think it’d be nearly impossible to build a large, stable concurrent system like without them.

Problems often cascade, so an earlier ignored problem might actually the part of the cause of a latter one. It is always best to deal with the problems in the order they occurred.

That strategy is great for development, but lousy for support. Once in ‘wild’ the code should try to work for as long as possible (producing lots of nasty log messages if needed), and there should be plenty of room for work-arounds. There should be a way to get a huge amount of information ‘easily’ from the users once the system has crashed. In that type of case, too much information is never a problem, too little is. Once a problem has been diagnosed, a work-around is the first (and cheapest) approach, while a patch is a fall back.

I’ve always found that mixing between the two strategies based on running environment is the best way to handle the two completely opposite problems.

Paul.
http://theprogrammersparadox.blogspot.com

I had a deja vue straight ahead reading your article. We had exactly the same problem with toArray() in a server, it was a list that stored notification e-mails to be sent in case a fatal error occured. All program parts and Threads can access this interface. A customer called stating that no emails are sent any more so we analysed the problem, found the bug pretty fast and fixed it within minutes. It was a one in a million chance: Two parts of the program “went fatal” and wanted to send an email simultaneously.

Finding the erroneous part that quick only was possible because we have a kind of logging-guidelines and do improve the software from time to time. This way, we do not have that much “error” – logging entries (and merely never a “fatal” one) and can follow them up pretty fast. Will say: We try to avoid to log into the error-channel where possible. (Part)Recoverable actions that occur (i.e. connection errors where you can retry etc.) are logged into the info-channel more and more as the program evolves.

Thus I think it is not an option to exit a program (or more worse: To System.exit() in case you write a Framework or module for your program). Just imagine you go to productive level and forget a System.exit() – statement … .

It will help much more to improve where/what/how to log. If you want to, have a look at http://goit-postal.blogspot.com/2007/01/how-to-successful-log-in-programming.html .

Just my 0,02$, Georgi