Thoughts on good software

Classical reports of software engineering (2)

2009-06-30T00:19:00.001-07:00

One of the most important distinctions when developing software is the one between accidental and essential complexity. These terms were coined in the landmark article "No silver bullet" (Brooks, 1986)

While essential complexity is the intrinsic complexity of the problem to be solved, accidental complexity is the complexity that has to be accepted because of the technology used to solve the problem (not the actual algorithm itself). This is especially important when evaluating different tools that solve a particular problem. Should you e.g. solve a certain problem in C or in Perl or in Java? Look at the complexities! What accidental complexity would be introduced by either solution? For example string processing is a breeze in perl thanks to regular expressions and easy string handling, while in C it's a real chore.

In a 2004 blog posting, Joel Spolsky considered memory management one of the best improvements of the last years (I agree):

"A lot of us thought in the 1990s that the big battle would be between procedural and object oriented programming, and we thought that object oriented programming would provide a big boost in programmer productivity. I thought that, too. Some people still think that. It turns out we were wrong. Object oriented programming is handy dandy, but it's not really the productivity booster that was promised. The real significant productivity advance we've had in programming has been from languages which manage memory for you automatically. It can be with reference counting or garbage collection; it can be Java, Lisp, Visual Basic (even 1.0), Smalltalk, or any of a number of scripting languages. If your programming language allows you to grab a chunk of memory without thinking about how it's going to be released when you're done with it, you're using a managed-memory language, and you are going to be much more efficient than someone using a language in which you have to explicitly manage memory. Whenever you hear someone bragging about how productive their language is, they're probably getting most of that productivity from the automated memory management, even if they misattribute it."

Before you plan which technology or programming language you use, always think about how much accidental complexity you introduce by either choice.

"No Silver Bullet - Essence and Accident in Software Engineering", Brooks, F. P., Proceedings of the IFIP Tenth World Computing Conference, pp. 1069-1076, 1986.

Why is good UI design so hard for some Developers?

2009-05-05T09:42:00.000-07:00

A user asked on stackoverflow:

"Some of us just have a hard time with the softer aspects of UI design (myself especially). Are "back-end coders" doomed to only design business logic and data layers? Is there something we can do to retrain our brain to be more effective at designing pleasing and useful presentation layers?"

My reply:

"Let me say it directly:

Improving on this does not begin with guidelines. It begins with reframing how you think about software.

Most hardcore developers have practically zero empathy with users of their software. They have no clue how users think, how users build models of software they use and how they use a computer in general.

It is a typical problem when an expert collides with a laymen: How on earth could a normal person be so dumb not to understand what the expert understood 10 years ago?

One of the first facts to acknowledge that is unbelievably difficult to grasp for almost all experienced developers is this:

Normal people have a vastly different concept of software than you have. They have no clue whatsoever of programming. None. Zero. And they don't even care. They don't even think they have to care. If you force them to, they will delete your program.

Now that's unbelievably harsh for a developer. He is proud of the software he produces. He loves every single feature. He can tell you exactly how the code behind it works. Maybe he even invented an unbelievable clever algorithm that made it work 50% faster than before.

And the user doesn't care.

What an idiot.

Many developers can't stand working with normal users. They get depressed by their non-existing knowledge of technology. And that's why most developers shy away and think users must be idiots.

They are not.

If a software developer buys a car, he expects it to run smoothly. He usually does not care about tire pressures, the mechanical fine-tuning that was important to make it run that way. Here he is not the expert. And if he buys a car that does not have the fine-tuning, he gives it back and buys one that does what he wants.

Many software developers like movies. Well-done movies that spark their imagination. But they are not experts in producing movies, in producing visual effects or in writing good movie scripts. Most nerds are very, very, very bad at acting because it is all about displaying complex emotions and little about analytics. If a developer watches a bad film, he just notices that it is bad as a whole. Nerds have even built up IMDB to collect information about good and bad movies so they know which ones to watch and which to avoid. But they are not experts in creating movies. If a movie is bad, they'll not go to the movies (or not download it from BitTorrent ;)

So it boils down to: Shunning normal users as an expert is ignorance. Because in those areas (and there are so many) where they are not experts, they expect the experts of other areas to have already thought about normal people who use their products or services.

What can you do to remedy it? The more hardcore you are as a programmer, the less open you will be to normal user thinking. It will be alien and clueless to you. You will think: I can't imagine how people could ever use a computer with this lack of knowledge. But they can. For every UI element, think about: Is it necessary? Does it fit to the concept a user has of my tool? How can I make him understand? Please read up on usability for this, there are many good books. It's a whole area of science, too.

Ah and before you say it, yes, I'm an Apple fan ;)"

Don't forget to have a look at the wonderful other replies like this one that has a lot of further pointers into UI matters.

Kill productivity now!

2009-01-30T23:01:00.000-08:00

Kill productivity now by browsing through this thread: What's your favorite programmer cartoon?

I'm happy to see the great Exploits of a mom (a.k.a. Little Bobby Tables) cartoon at the top. Of course xkcd is your place to go for more of that brand of humour.

Oh and if you understand German don't forget this classic IBM ad with the wonderful line "Where are the web designers?".

Deconstructing Steve Jobs

2009-01-30T22:39:00.001-08:00

Tom Junod has written a very unusual article for Esquire called Steve Jobs and the Portal to the Invisible.

It's not average tech writing but a deep-field view of the way Jobs works and thinks.

One really gripping and essentially heartbreaking element is Jobs bashing the Amazon Kindle with the argument that people don't read books anymore anyway when his biological sister Mona Simpson is a writer. Kudos to Junod for writing such a holistic and insightful piece.

Toughest bug

2009-01-25T01:26:00.000-08:00

Some time ago a StackOverflow thread got started with the interesting question about the toughest bug one had fixed. I added this incident:

"This was on Linux but could have happened on virtually any OS. Now most of you are probably familiar with the BSD socket API. We happily use it year after year, and it works.

We were working on a massively parallel application that would have many sockets open. To test its operation we had a testing team that would open hundreds and sometimes over a thousand connections for data transfer. With the highest channel numbers our application would begin to show weird behavior. Sometimes it just crashed. The other time we got errors that simply could not be true (e.g. accept() returning the same file descriptor on subsequent calls which of course resulted in chaos.)

We could see in the log files that something went wrong, but it was insanely hard to pinpoint. Tests with Rational Purify said nothing was wrong. But something WAS wrong. We worked on this for days and got increasingly frustrated. It was a showblocker because the already negotiated test would cause havoc in the app.

As the error only occured in high load situations, I double-checked everything we did with sockets. We had never tested high load cases in Purify because it was not feasible in such a memory-intensive situation.

Finally (and luckily) I remembered that the massive number of sockets might be a problem with select() which waits for state changes on sockets (may read / may write / error). Sure enough our application began to wreak havoc exactly the moment it reached the socket with descriptor 1024. The problem is that select() works with bit field parameters. The bit fields are filled by macros FD_SET() and friends which DON'T CHECK THEIR PARAMETERS FOR VALIDITY.

So everytime we got over 1024 descriptors (each OS has its own limit, Linux vanilla kernels have 1024, the actual value is defined as FD_SETSIZE), the FD_SET macro would happily overwrite its bit field and write garbage into the next structure in memory.

I replaced all select() calls with poll() which is a well-designed alternative to the arcane select() call, and high load situations have never been a problem everafter. We were lucky because all socket handling was in one framework class where 15 minutes of work could solve the problem. It would have been a lot worse if select() calls had been sprinkled all over of the code.

Lessons learned:

even if an API function is 25 years old and everybody uses it, it can have dark corners you don't know yet
unchecked memory writes in API macros are EVIL
a debugging tool like Purify can't help with all situations, especially when a lot of memory is used
Always have a framework for your application if possible. Using it not only increases portability but also helps in case of API bugs
many applications use select() without thinking about the socket limit. So I'm pretty sure you can cause bugs in a LOT of popular software by simply using many many sockets. Thankfully, most applications will never have more than 1024 sockets.
Instead of having a secure API, OS developers like to put the blame on the developer. The Linux select() man page says
"The behavior of these macros is undefined if a descriptor value is less than zero or greater than or equal to FD_SETSIZE, which is normally at least equal to the maximum number of descriptors supported by the system."
That's misleading. Linux can open more than 1024 sockets. And the behavior is absolutely well defined: Using unexpected values will ruin the application running. Instead of making the macros resilient to illegal values, the developers simply overwrite other structures. FD_SET is implemented as inline assembly(!) in the linux headers and will evaluate to a single assembler write instruction. Not the slightest bounds checking happening anywhere.

To test your own application, you can artificially inflate the number of descriptors used by programmatically opening FD_SETSIZE files or sockets directly after main() and then running your application."

Classical reports of software engineering (1)

2008-11-30T02:23:00.000-08:00

In 1996, Charles Fishman wrote an influential article for FastCompany about the NASA shuttle software group: They write the right stuff

The article is enlightening in many ways. If you're serious about quality software, you will love it.

Fishman wrote a more engineering-centered follow-up in 2007 called To The Moon! (In a Minivan).

It's interesting to read how NASA adapts Google's approach to use off-the-shelf components.

StackOverflow

2008-09-25T11:30:00.000-07:00

I've found a great tool for programming questions: StackOverflow

It's a very web 2.0 way to find solutions for software development questions. If you're serious about your job, you should check it out to know how it works. Used in the right situation, it can save you large amounts of time.

0

2008-09-20T07:58:00.001-07:00

A short but important rule:

Whenever you write code with an integer divison operation, write code around it that avoids or handles a division by zero.

This should happen completely automatic when you use the division operator. It should immediately cross your mind that this call could be dangerous.

If you don't obey this rule, sooner or later your code will blow up. In C/C++ your whole process will die (a legacy of the days when software was on punched cards and the mainframe would stop processing on errors). In more modern languages, you will get thrown out of your method with a division-by-zero exception if you don't have an exception handler around it.

Typical error situations:

Calculating averages (value DIV number-of-samples) with a zero number of samples.
Variation: Dividing by time duration (x DIV (time_after MINUS time_before), but the duration was below one second and in the same second of system time, so your duration value became zero.
Using some floating point value smaller than 1 as divisor when your compiler creates an integer division from what you thought was a floating point division (this is especially dangerous in languages that convert easily between numerical types and don't have a separate integer division operator) Also occurs when code is bulk-converted to non-floating-point embedded hardware.
Hidden division in modulo calculations. a MOD b with b=0 will typically cause the same trouble as a division by zero.

Recommended reading to understand the importance and danger of zero:

Charles Seife, Zero: The biography of a dangerous idea.

(German Title: Zwilling der Unendlichkeit)

Software and people

2008-07-30T12:27:00.000-07:00

This is just a thought that I've been thinking about for the last few days... We like reliability in persons. When somebody does as we expect him or her to do, we find this reassuring. From adults we expect that they have found their identity and do not react explosively without a good reason. The same is true for software. Poor quality in software makes us think it is unreliable, not worthy to deal with. Each crash scares users away. Each function that does not work as expected gives us the impression of immature software, in usability terms that's called expectation conformity. Don't we often expect that conformity from people, too?

Just an idea I think is worthwhile to reflect on.

Why CPU usage has become almost irrelevant

2008-07-20T00:27:00.000-07:00

As a kid in the 1980s, there was the C64. Many may remember that little machine with its 1 MHz 6502 CPU and notorious 38911 BASIC BYTES FREE. Boy what small resources! And then we all programmed on interpreted BASIC which made the apparent speed of execution about a tenth of what it would have been in native machine code. An effective 0,1 MHz. You could literally watch the computer execute each statement.

Where are we now? Dual core, quadcore, 3 GHz, 4 GHz. Applications do not run in shared RAM anymore but in a world of seperate kingdoms called virtual address spaces, eagerly guarded by a memory management unit. 32bit operating systems offer 2-4 gigabyte, and 64bit operating systems gazillions more. Processors are so powerful that they can be shared to run multiple operating systems simultaneously with a hypervisor like VMWare.

So why do I bring up this comparison? To illustrate an important point: CPU usage in normal applications is so negligible that it doesn't play much of a role anymore. There are notable exceptions to this: any multi-dimensional information processing (pictures, movies, huge nested loops, huge databases, huge recursions), or in computer science lingo: O(N^2) and above. But for the rest of algorithms, I think programmers should reconsider what they learnt during their professional upbringing.

In the dark ages before lightning fast processors, optimizing CPU usage was of real importance. You could reach speed increases of seconds, minutes or even hours by optimizing code, throwing unneccessary function calls away here, using a funky assembler instruction there. These days are gone for application software. Sure, applications still fight for CPU time in the kernel scheduler. But compared to the massive bottleneck "disk access", the CPU pales in comparison. That's why the Megahertz race in processor advertising finally subsided, because nobody could understand anymore why they needed all this for typing letters in Microsoft Word.

Think of a typical application program that does disk I/O (either from a local drive or, worse, from a network share). An access may take 100ms to deliver data. One hundred milliseconds! Do you know what your CPU did in all that time, all these nanoseconds? It waited literally millions of cycles if the scheduler had nothing else to do for it.

Hard drives and network interfaces have taken over the role of "slow parts" in computers. That is why optimizing file and network access is key to optimizing software these days. Sure, you can still suffer from CPU shortage if you program recklessly, but many more delays in an application happen because there is only one thread of execution, and that thread of execution is waiting for an external event. The worst delays happen when the operating system itself is competing for hard disk access (read: access swap space). Swap space is RAM of absolutely last resort, not something you would want in normal operation of a PC.

If your application runs slowly on a customer system, be on the lookout for RAM shortage, badly fragmented hard drives and slow networks.

This brings me to one of the things I learned in many years working with computers: To make a computer faster, increase RAM, not necessarily processor speed. You can have a fast 500 MHz CPU when it has enough RAM and a good hard drive interface, while many gigahertz class machines are crippled by far too little RAM filled with all sorts of pre-installed software the user never uninstalls, and slow hard disk access.

Writing your own XML parser is wrong

2008-06-22T10:26:00.000-07:00

A lot of things in programming are left to personal taste. How you indent your code lines, how you manage your data structures. Most things have advantages and disadvantages, and everybody should come to their own decision.

Some things though violate the most basic rule of software design:

Do not reinvent the wheel.

Most developers agree to this rule, consider it a truism and do not try to re-invent task schedulers, file systems or window managers (except if that is exactly what they want to do, like e.g. in Linux).

For some reason there are certain sins left that developer seem attracted to. One of the cardinal sins is writing your own XML parser. I have seen this multiple times, and all attempts had failed. In theory, writing the parser should be straight forward. In practice, all sorts of things go wrong.

It is my impression that the initial motivation is like this: People look at an XML file. They figure out that it is an easy text format with a tree structure. Then they say to themselves: This is an easy format to store my application information. Why use a parser written by someone else? I can do that! And then the disaster begins.

What's the problem?

Using a protocol or seeing its output does not give you an impression of how complex its underlying technology really is. It's like school kids thinking that being a teacher must be an easy job, because you're just standing there for a few hours a week, and you have months of holidays every year. The difficulty of actually being a teacher is lost on them because they have never experienced the job from the other side.

People reinventing XML parsing (and ignoring the multitude of proven industry-strength parsers) always end up with the same result: Their parser is fixed to a certain, dumbed down version of XML of a certain structure. Most pseudo-XML parsers get the basic tree part about right, but only for their specific purpose. Many parsers show unpredictable behavior when parts of the XML are missing or duplicated. Character encodings are not taken account of ("We all use Windows encoding here in our files, so no need for anything else!") Then comes a typical flaw: None of the self-made XML parsers are able to verify their input except in very basic situations. Once I had to use a homebrew XML parser that would run into an endless loop when it got unexpected input. No one had tested that. Implementing support for XML schema files would have shown to the developer just how much he did not take into account while writing the parser.

Quick and dirty XML parsers violate the very idea behind XML: to offer a program-independent and exchangable way to distribute hierarchical data. They are charlatans. The data they process appears to be XML. But whenever one of the features above "it is a tree" of XML is used, they fail. When the line endings consist of different characters, they fail. When a tag spans two lines, they fail. And so on, and so on.

The excuse here will immediately be: I'm writing the XML too, so I have control over how I use it. Great. But then you are using a custom data format, not XML. You do not care about interoperability. You just make it look like you do. Is that really what you want?

Implementing a complete XML parser is hard. It takes months if not years of a concentrated development effort to build a good one, including painstakingly testing each feature. Just reading the Wikipedia page on XML should cure people of implementing all this themselves when stable parsers are ready to download around the corner. Pick the one you like (I like libxml2 for C applications, for example, although its documentation is temperamental). Choose an event-driven or tree-driven parser. Use it. Enjoy the bliss of not having to fight with internal XML features and use the time for your application.

This article should also have cured you of writing your own shallow XML parser. If you think it must be easy, because after all it's just text dammit, please read Robert Cameron's article on shallow-parsing XML and use his parser grammar instead.

By the way: Exactly the same goes for any homebrew implementation of complex standards that looks easy at first, but then makes you cringe at the details. A common other case is "I can implement an SMTP email send function in a day". Believe me, you will need more than a day. There are well-tested mail sender plugins and libraries out there. Use them instead.

Effective developers know how to search the net

2008-06-22T04:40:00.001-07:00

Back in the old days, handbooks were the most important source of information for software developers. With your operating system you would get a massive amount of documentation that was meant to be comprehensive for every imaginable application to be.

Things have changed slowly but surely. Today, an application developer not able to find online information about problems or errors encountered is in the way of productivity. He or she will be looking desperately for the explanation of an obscure API call or error message in old manuals when the solution could be found in seconds when using search engines. This is because error messages may be obscure to the single developer, but the net is so huge that somebody probably has stumbled upon it (and possibly found a good solution). Also, the paradoxon of manuals applies: The most difficult problems never appear in them, because they were simply not foreseeable to the API or system creator to be documented.

With the ever evolving APIs of today's libraries and operating systems, it is simply not economical to depend on printed documentation when things go bad. There are some examples of very good official documentation. The Java documentation is usually excellent, for example. Many parts of the MSDN are clear and precise, and tackle real world problems. But especially in open source land, documentation is usually done "in code" (cough) or very scarce and outdated. The quality of Linux man pages varies wildly.

When an error message "libXYZ: error 59, dropped packets" pops up, it is the best way to simply dump the error message verbatim into a search engine. Usually, a lot of hits will show up. Many may just be desperate postings in online forums, but one out of ten may have received the appropriate answer. Problem solved. Good manuals may assist, yes (although don't forget the saying: you can't grep a dead tree). Reading bad manuals as well as communicating with support hotlines would be much more time-consuming and should just be a way of last resort.

I think it would be useful to not only ask for development qualifications in job interviews for software developers, but also let them find out the problem for a particularly weird error message. Just ask what their approach would be. If you have developers who can't solve problems with search engines, your competitor might have them, and thus have a much better productivity.

Logging: Structuring debug output

2008-06-22T03:51:00.000-07:00

When we begin learning to program, we all write debugging output like

printf("found string\n");

or, in the parlance of our times,

System.out.println("found string");

For a certain time, this sort of debugging output is enough. We remember where we put that output statement.

Things get harder when the code grows beyond a few hundred lines, or when more than one developer is working on it. Someone else just does not understand what train of thought the message comes from. Especially hard to understand are low-level control flow debugging outputs like AAAABACCDDD where each letter means that one specific branch of an if statement or other control flow statement has been taken (don't laugh, I've seen this sort of output in a release version of Acrobat Reader on Linux). The more your application grows, the more you are in need of a fundamental mechanism: Logging.

Logging should be the first thing you lay out before writing the first debug message of a code. A good log message consists of at least 5 elements per line of output:

WHEN was the message caused?
WHAT happened?
WHO caused the message (which object/method/function)?
WHICH THREAD ran the code?
HOW SEVERE was the message in the context of the task to accomplish?

(This only sounds like the rules when calling 911.)

Our little example from above may now look like this:

21 Jun 08 11:24:03.434 INFO [search_thread] TextSearch::searchTopic(): Found topic 'Jungle'

Let me explain why each element of this line is important.

The WHEN

As irrelevant as it may seem at first, the point of time a log message is caused is extremely important. Especially in multithreaded or distributed applications with the danger of race conditions (two or more threads or processes concurrently racing to use a data field / ressource) you have to know the exact point of time a thread tried to access something. Depending on your application, millisecond precision may be important. You can also detect time wasting in your code that way ("Gee, that call really takes 100ms? Why?")

Time runneth fast. Always time the log message at the very beginning of your log method. If you write all sort of code before (like string formatting, output preparation etc.), a context switch may already have cost you an unknown amount of time, and the "real" point of time of the log message is lost.

A very hard challenge is synchronizing log messages between different computers. Although NTP and time synching has been around for quite some time, synching of system clocks is still not precise to the millisecond. If you are on a fast local network, you can use a broadcast message over a network to cause a message at one point of time. Even if you have diverging clocks, you can then find out the leap between all clocks because the message was received at almost the same millisecond.

The WHAT

All log mechanisms allow one or another sort of output formatting where the language does not offer on-the-fly string building like in Java. In C/C++ the usual way is offering printf-style formatting with varargs. A call may then look similar to a printf call:

log("Now looking for text %s",searchstring);

Good log messages are like good algorithms: They are clear to read and describe what is being done. Obscure log messages, like obscure algorithms, may look ingenious, but are usually a pain to read and to maintain. Be clear when formulating log messages. Neither be too short ("Value: "+acceleration) nor too chatty ("The algorithm is just now processing the value acceleration which is of value "+acceleration). One rule of thumb is: Values alone are confusing. Always attach the name of the variable to it, and a unit if possible ("acceleration=3.5 m/s")

The WHO

You definitively want to know which function or method is sending you a log message. In C/C++, you can automatically insert the function/method name by using compile-time directives like __PRETTY_FUNCTION__ (gcc) or __FUNCTION__ (Visual Studio compiler). These directives contain the current function (try printf(__PRETTY_FUNCTION__);) With another preprocessor wrapper like

#if defined(WIN32)

#define __PRETTY_FUNCTION__ __FUNCTION__

#endif

#define LL(l) Logging::l,__PRETTY_FUNCTION__

you can avoid to have to write the long directive in each log message. LL(loglevel) will evaluate to 'Logging::loglevel,__PRETTY_FUNCTION__' and thus fill two of the arguments of a call to the log function:

log(LL(SEVERE),'test');

would satisfy a log(loglevel,method,string) function with three arguments , and the current method or function name would be inserted automatically. (The Logging class has to have constant values for the log levels as static variables to make the example work of course.)

Java offers the java.util.Logging class where you have to enter the method name yourself. And there's LOG4J of course.

If you have to keep track of which object executes what method, use either its name (with uniquely named objects) or its unique pointer if you can get hold of it (this is easy in C/C++ but may be difficult in other languages).

The THREAD

Ten years ago most software was written as one long singular control flow. If the code did something, it was done in a strictly linear way. Only sometimes a worker thread was started so that the GUI could still be handled while calculating something.

Although there are some applications that still work in this peaceful way, most modern applications have an abundance of threads. In such an environment it is of key importance to know which thread caused a log message, especially when multiple threads of the same kind are running. If possible, use named threads and make sure that all thread names are unique. In Java and C# naming threads is a standard feature, while in pthreads, which many portable C/C++ applications use nowadays, you have to use TLS (thread local storage) to attach a name to a thread.

The SEVERITY

Log messages vary a lot in terms of importance. All serious log mechanisms take this into account and require an importance measure be attached to each message: the log level. Names of log levels vary, but the general layout is usually like this:

ERROR: a message that MUST be shown to the user, and which is fatal
WARNING: a message that something has gone wrong which the user should know. Non-fatal.
INFO: a message of an informative character that the user should know about.
DEBUG: messages only important to the developer. Filtered in production code.
TRACE: verbose debugging output that is usually annoying even to the developer except in special cases. Filtered in production code.

Log mechanisms vary in their number of loglevels. Some add a FATAL level above ERROR, others use more fine-grained log levels at the DEBUG or TRACE level. This is a designer's choice.

Most GUI applications also include logging, but they strictly hide it from the end-user. Then of course appropriate GUI action has to be taken to inform the user.

Design details of a logging mechanism

One beginner's mistake when building a log function is using methods in it that issue log messages themselves. Depending on the use of mutexes (none/recursive vs. non-recursive) this either causes a stack explosion by infinite recursion or a deadlock (can't lock the log mutex ever because the log method is locking it already) Be very careful when writing the log output/transmission function when you have no way of determining whether a logging caused it itself. A simple (yet not thread-safe) way to avoid recursion is using a static variable logging_in_progress that is set to true whenever a logging is just going on. If the log method is called while logging_in_progress is true, it immediately returns. In a multithreaded application, this behavior is of course not enough. We'll come to that in a minute.

The log queue

Multiple threads may cause a lot of messages at almost the same time. As a log mechanism usually has to use a mutex to protect the integrity of its output, it can become a needle eye in your code. One log message may be output on a terminal, delivered over network connections and entered into a database. Much too long for one thread to lock a mutex.

The technique to reduce the cost of sending log messages is using a special log thread for the actual output/transmission of log messages and a stub log() method that just appends log messages to a log queue. The log thread basically does two things in an endless loop:

Look for log messages
If log messages in the queue, output/transmit them

and so on. Both methods, the log sender and the log receiver, need to be extremely carefully designed. The time spent locking the log queue mutex must be kept as low as possible at any cost. It should go without saying that the timing of the log message must be done in the log sender method, not in the log thread.

Extra care must be taken that the log thread does not starve because other threads take over all the CPU time. In massively multithreaded applications with hundreds of log messages every second, care has to be taken that the log queue does not grow infinitely and eats up all of the address space available to the application. A way to avoid trouble is dropping older log messages (possibly retaining ERROR and WARNING messages) in an overflow situation.

Instead of checking a logging_in_progress static variable, you have to make sure the log thread never queues in log messages in its own queue. A good way to ensure this is checking for the logging thread's unique name when using named threads.

The log queue is so important in multithreaded applications that, in my opinion, it should be declared a design pattern. Conversely, the non-queued logging should be declared an anti-pattern for multithreaded applications because it is a real time waster in all but the most basic situations (everything more complicated than just issuing a printf() statement).

Separating content from layout

For web pages, content management systems separate the content to be displayed and the actual layout. It is a good rule to do the same to log messages: Keep them in a structured way as long as possible, i.e. transform them to a flat string as late as you can. If your log method just builds a flat string, structured entry of log messages e.g. in a database becomes a real pain. If you use a log queue, make sure it uses structured data fields so you can easily process them in the log thread.

Conclusion

Logging is a key element in successful software debugging. It helps all developers and should be designed and used carefully. A good log mechanism implements log levels, thread debugging (where needed) and log message formatting. Log messages should be kept short and informative. Good logging output gives any developer a good understanding of what the code is trying to achieve and how it has succeeded achieving its goals.

Hello

2008-06-22T03:36:00.000-07:00

Hi to everybody!

This will be my place to publish articles about topics I find interesting in software development. To me, software development is both a craft and an art form. We still do not produce software, but we write software. Every piece of code we create shows our thinking and our personal style.

In its best moments, the writing process of software flows uninhibited, and out comes beautifully styled code that is made to last. Experience shows that these flow moments are not so common. The reality of business makes compromises inevitable. How can we still maintain clarity and quality given the demands of today's fast-paced development schedules and ship-it-by-yesterday demands?

It is my wish that this blog may show ways how to stay on top of the development process. Many topics are of course controversional. We software developers love to argue what is the best editor/compiler/library/indenting style. Anyways, constructive criticism and comments are very welcome!