September 22, 2005

Unit testing is teh suck, Urr.

Recently an innocent reader inquired the following of me:

I'm curious to know how you approach product testing in such a small company. Do you have an established process? Do you use unit testing? Other structured testing? [...] Do you have suggestions specific to solo testing and debugging?
My answer, perhaps more politely phrased, was, like another famous Wil(l), "AW HELL NAW!"

I've certainly known companies that do "unit testing" and other crap they've read in books. Now, you can argue this point if you'd like, because I don't have hard data; all I have is my general intuition built up over my paltry 21 years of being a professional programmer.

But, seriously, unit testing is teh suck. System testing is teh suck. Structured testing in general is, let's sing it together, TEH SUCK.

"What?!!" you may ask, incredulously, even though you're reading this on an LCD screen and it can't possibly respond to you? "How can I possibly ship a bug-free program and thus make enough money to feed my tribe if I don't test my shiznit?"

The answer is, you can't. You should test. Test and test and test. But I've NEVER, EVER seen a structured test program that (a) didn't take like 100 man-hours of setup time, (b) didn't suck down a ton of engineering resources, and (c) actually found any particularly relevant bugs. Unit testing is a great way to pay a bunch of engineers to be bored out of their minds and find not much of anything. [I know -- one of my first jobs was writing unit test code for Lighthouse Design, for the now-president of Sun Microsystems.] You'd be MUCH, MUCH better offer hiring beta testers (or, better yet, offering bug bounties to the general public).

Let me be blunt: YOU NEED TO TEST YOUR DAMN PROGRAM. Run it. Use it. Try odd things. Whack keys. Add too many items. Paste in a 2MB text file. FIND OUT HOW IT FAILS. I'M YELLING BECAUSE THIS SHIT IS IMPORTANT.

Most programmers don't know how to test their own stuff, and so when they approach testing they approach it using their programming minds: "Oh, if I just write a program to do the testing for me, it'll save me tons of time and effort."

There's only three major flaws with this: (1) Essentially, to write a program that fully tests your program, you need to encapsulate all of your functionality in the test program, which means you're writing ALL THE CODE you wrote for the original program plus some more test stuff, (2) YOUR PROGRAM IS NOT GOING TO BE USED BY OTHER PROGRAMS, it's going to be used by people, and (3) It's actually provably impossible to test your program with every conceivable type of input programmatically, but if you test by hand you can change the input in ways that you, the programmer, know might be prone to error.

So, listen closely, because this is the method I've used since the dawn of time, and it works great:

1) When you modify your program, test it yourself. Your goal should be to break it, NOT to verify your code. That is, you should turn your huge intellect to "if I hated this code, how could I break it" as SOON as you get a working build, and you should document all the ways you break it. [Sure, maybe you don't want to bother fixing the bug where if you enter 20,000 lines of text into the "item description" your program gets slow. But you should test it, document that there is a problem, and then move on.] You KNOW that if you hated someone and it was your job to break their program, you could find some way to do it. Do it to every change you make.

When I test Delicious Library, I'm like, "Hey, to make sure my new queue works, I'm going to take a Library of 2,000 items and tell them all to reload themselves simultaneously." Now, honestly, in version 1.x, this is really slow. But, dang it, it works. And since our goal for version 1.x was to have it work with between 1K-2K items, I'm OK with this.

Too often I see engineers make a change, run the program quickly, try a single test case, see it not crash, and decide they are done. THAT IS TEH SUCK! You've got to TRY to break that shit! What happens when you add 10 more points to that line?

2) When you get the program working to the point where it does something and loads and saves data, find some people who love it and DO A BETA TEST. The beta test is often maligned, but the most stable programs I've ever written are stable because of beta testing. Essentially, beta testing is Nature's Way (TM) of making systems stable. You think nature creates unit tests and system tests every time it mutates a gene? Aw hell nah. It puts it out in the wild, and if it seems better it sticks around. If not, it's dead.

Now, if you're a pure-CS kind of person, you're thinking, "But, with beta testing, I might miss some really deep, conceptual bugs that unit testing would find, because there's only certain pathways people will test..." Well, there's really only one rational response to this:

WHO CARES!

Like in nature, any system is going to have bugs. And, again like nature, most of the time, in most situations, you want your system to work. But occasionally, under some circumstances, it could fail. This isn't great, but it is acceptable. The nice thing about beta testing is it lets you focus YOUR precious time as a programmer on the pathways that fail most often under normal use patterns, instead of just finding bugs at random that may or may not ever be triggered, and sucking up your time fixing them. (Hey, if you want to try to fix every bug in your program before you ship, be my guest, but at least fix them in order of how often they are reported, OK?) Unit testing doesn't give you information on how common a bug is, which is the single-most important piece of information about a bug, followed very closely by its severity.

Look, I know it's heretical to say "software is going to have flaws." But everything is going to have flaws. Pilots screw up. Surgeons screw up. People die. Denying this doesn't make it go away, it makes it worse. We need to say not "never screw up," but "make sure when you screw up, you recover nicely."

Make sure that the flaws your program ships with are the least-common ones possible, given the time you have to fix bugs. Honestly, crashers are more important than, say, drawing bugs, but if 1,000 users report a drawing glitch, and one reports a crasher, which one should you fix? THE DRAWING BUG. DO THE MOST GOOD WITH THE TIME YOU HAVE. It's your duty on this earth, and it's also the fast route to success.

Now, all this is NOT an excuse to write crappy subroutines and say, "Well, it'll got caught in the beta test." You, as a programmer, should be programming EVERY LINE as defensively as possible. You should assume that every other method in the program is out to kill you, and they are intentionally passing you corrupted buffers and pointers, and it's your job to handle all that crap and make the best of it. (And, if you can repair it, do so, and if not, raise an exception so you can catch it in beta testing, instead of silently failing and/or corrupting data.)

Let's sum up: Write good code to start with. Write every routine assuming that it'll be called by chuckle-heads. When you modify your program, test it yourself, and TRY to get it to fail, don't try to get it to pass. Then set up a real beta program, with a good pathway for sending in feedback and categorizing it, and let the beta-testers do the rest.

Testing is hugely important. Much too important to trust to machines. Test your program with actual users who have actual data, and you'll get actual results.

--

November 20, 2005 followup: My comments here have been widely reported, and sometimes misconstrued because of my own failure to disclose the boundaries of when and why I don't use Unit Tests. I was thinking of writing a follow-up, but bbum has done it for me, and I agree completely with what he says. So, please consider his post as the continuation of mine, only, you know, written by him.

Labels:

73 Comments:

Blogger Colin Barrett said...

A-freaking-men! On a similar note, the best way to write a secure webapp is to assume that every time you read data from the user, they are trying to fuck you up and inject SQL or Mail headers or something crazy like that.

September 23, 2005 12:49 AM

 
Blogger Abhi Beckert said...

I've been wondering for ages if I was making a big mistake by thinking exactly what you just posted...

September 23, 2005 1:12 AM

 
Anonymous Chris Hanson said...

Wow, Wil. I didn't know you could be so utterly and dangerously wrong.

Unit testing is just about as freaking far from teh suck as you can get. It has had real, virtually immediate, and very high ROI on every single project I have ever used it on. Especially when I've done test-driven development.

You know what's even better than testing your software when you change it? Testing your software when you change it -- or before you even write it -- in a way that ensures the tests will be carried out every time you hit command-B in Xcode. You can test your interface this way. You can test your logic this way. You can test the connection between the two this way. And you can do it in a repeatable fashion.

Does writing unit tests take time? Yeah. This is time spent not staring at stack traces and inspecting variables in the debugger. In fact, you might even spend less time overall to get to a higher level of quality using unit testing and test-driven development.

Honestly, what are you going to argue next? That you shouldn't use a revision control system? Because writing software in 2005 without unit testing is like writing software in 1995 without revision control.

September 23, 2005 1:48 AM

 
Anonymous Duncan Wilcox said...

Writing unit tests is a pain and a time drain, and while according to XP-think it should make you feel more free of making changes even when you're close to releasing, you're only as free as much as you're confident that your tests are covering you (easier for some big corporate backoffice server app than for a spiffy UI-centric app). And assuming you're covered, every significant change to your code requires double effort, as you have to change the tests too.

So let's assume that you don't do unit tests. You add new code defensively, try to break it as hard as you can, and then it's up to testers.

The real issue is, what about regressions? When you change a line of code someplace you wrote six months ago, do you remember all the dependencies, all the places it might possibly break, so that you can try and break them all? Do you just abuse your testers who re-report the same bugs? Remembering you already fixed the bugs, do you double-check them for regression before flagging them as duplicate?

For example I don't think WebKit would be able to progress significantly without the over 2000 layout tests that routinely flag a regression in some area.

I'd love to learn about defensive coding techniques that help you know and avoid the butterfly effect of regressions in unexpected parts of the code. Insult me!

September 23, 2005 3:50 AM

 
Anonymous Anonymous said...

Personally I think that both you and the commentors miss the point ever so slightly. Unit tests XP-style is indeed teh suck, but so is not having automated tests for old bugs. As I see it unit tests are a very good way to make sure you don't repeat mistakes. What you're saying about judging relevance of bugs is very true, but using a unit test to make sure you don't reintroduce bugs is a great way to confidently move forward in development.

September 23, 2005 3:55 AM

 
Anonymous Anonymous said...

I have to agree with Chris Hanson that unit testing is exremely important. It is essential for my work.

However, I think you have to differentiate a bit between library/API design and GUI applications. Unit testing is essential in my opinion for library design or what people like to call "business objects/logic", i.e. backend code that has a "code" interface. This "code" interface is ideal to test with unit tests. Unit tests make sure your code works correctly, keeps working correctly and, most importantly, allows you to refactor much more freely.

I won't ever write library code without unit tests again. It's not wasted time, it's saved time.

I'm also the author of a OSX GUI application and there I see more limited use for unit tests, because there's no good way to do automatic testing. Nor can there be I believe, because the combinations of actions users can pull off via a GUI are just so big that you'd literally spend ages recording all the possibilities in test intrumentation.

Instead, in this case I agree it is much more efficient to let beta users hammer the frequent use cases. When you don't have a chance to test everything, you should concentrate on the important bits and I agree with your classification more or less.

Cheers,
Sven A. Schmidt

September 23, 2005 3:57 AM

 
Anonymous L'g. said...

I fully agree on Wil's statement. The ugliest faukt you can make is to not test the stuff you developed. And there is perhaps a difference in how you test depending on the kind of software you write. If you have to code a software e.g. for medical control of a persons heart-disease you will for sure select a diffferent way compared to if you write an app for image manipulation. It's simply depending on the risk of failure and the consequenses / costs of such a failure. If the consequence would be a dead human (medical control) you WILL spend all this time and effort to test through by brute force ALL COMBINATIONS. If an image-manipulation-program crashes no one will die. It will cost perhaps 20 minutes of worktime. On the other side, if you app is used by a million people and 1000 of them report crashes this means 1000 x 20 = approx. 333 hours or say roughly 2 weeks of total lost time. This is where you will do it Wil's way and fix this bug as soon as possible. Before the seldomly occurring bug which only affected one person so far.

establishing testing procedures cannnot be done for free, so you will have to calculate the risk of not establishing it. If this is acceptable to you, fine.

I think one should also take into account, that using this time to improve the usability or extend the funcionality instead of doing academic unit testing will bring nearly always the better ROI (to answer in business speak). ;-)

September 23, 2005 4:55 AM

 
Anonymous ken anderson said...

I think Chris and Wil are both right. I find that a number of programmers who think following the unit test mantra is the "Right Thing" are mislead and write freakin dumb unit tests. It is not worthwhile to write a unit test for something that will be absolutely obvious when running the application (does this button actually call this method when I click it - kind of like, if I stick this pencil in my ear, will it hurt?). Unit tests should be used for more complex tests that are difficult to verify easily, like, does the result of this complex calculation match the known good quantities I've calculated outside the application environment.

Unit testing cannot encompass the level of testing that is necessary to do with a human. As Wil so eloquently put it, you really can't write a program to test usability and resilience in a user environment.

It's also important to realize that your goals often dictate how much of each methodolgy you should use. For Wil, where he's building a complete application system, integrated system testing is far more rewarding than writing a significant number of unit tests. For a developer who's working on reusable components, frameworks, and libraries, unit tests are by far the more necessary of the two, since you really can't sit in front of it and try to break it without another level of indirection, which often muddies the waters.

September 23, 2005 5:32 AM

 
Anonymous Anonymous said...

This discussion is a bit poorly focussed largely, I believe, because of the rather sloppy use of terminology in the original post.

The titles suggests that 'unit testing' is a waste of time yet most of the discussion, even those bits that actually use the phrase 'unit testing', talks either about other types of testing or at least very odd unit testing by most standards e.g. being hired to write unit tests after the fact for code you didn't even write is hardly XP best practice or Test-Driven Design. Also, I personally wouldn't mention unit testing if someone asked about how to test apps (like the initial prompt for this post), as I see it as part of development, i.e. something to prevent bugs rather than find them: part documentation, part design, part sanity check, part safety net.

Unit testing is also blurred in this discussion with any kind of automated testing e.g. the Webkit layout regression tests mentioned above. Will manages to write-off all automated testing ("Oh, if I just write a program to do the testing for me, it'll save me tons of time and effort.") as worthless but I don't know if that's intentional or if he just got carried away.

What's weird is that if you take out all the stuff dissing automated/unit testing you are left with a really strong argument *for* automated/unit testing (as well as some general good advice for how to approach testing your own code as a programmer).

* "You should test. Test and test and test." (that sure sounds like something that needs automated)

* "you can change the input in ways that you, the programmer, know might be prone to error." (Will claims you can't capture this info in a unit test, only in hand testing, but that's just silly. It does however suggest why unit tests written by the original programmer are better than hiring an intern to do it, or waiting for the test team or beta testers to catch these edge cases based on black-box testing)

* "When you modify your program, test it yourself. Your goal should be to break it, NOT to verify your code. That is, you should turn your huge intellect to "if I hated this code, how could I break it" as SOON as you get a working build, and you should document all the ways you break it. ... You KNOW that if you hated someone and it was your job to break their program, you could find some way to do it. Do it to every change you make. (All good advice, though how it argues against unit testing I'm not sure. However I'm going to repeat this one bit because it's pure gold, and could have come straight from any XP text: Do it to every change you make., I'd also like to pause and consider exactly how much time Will spends testing *everything* after *every* change. By hand. I'm getting bored just thinking about it.)

It's only when you set up the argument as choosing only one of automated/unit testing or beta testing that things get complicated, but again that's just silly. Here's my crazy (but it might just work) suggestion: do both (and other types of testing too!). Understand how and why to test a software product in different ways, and use different techniques (unit/automated/test teams/crash reporters/beta tests) as appropriate to the situation.

I'd recommend the Pragmatic Programmers' book as an introduction to the subject of Unit Testing. It comes in Java and C# flavours, but the core concepts are transferable and it talks as much about the Why as the How. And it's as well written as the rest of the PragProg range.

Pragmatic Unit Testing (in Java with JUnit)
http://www.pragmaticprogrammer.com/starter_kit/utj/index.html

September 23, 2005 5:56 AM

 
Anonymous Marcel Weiher said...

Wil,

I've disagreed with you on a couple of other points, but felt no need to respond, as the mistakes were mostly harmless. However, your stance on unit testing couldn't be more utterly, completely and dangerously (if anyone is foolish enough to believe you) wrong.

Unit testing in general, and TDD in particular, is not something some corporate weenies got out of a book, it is a technique adopted by experienced programmers BECAUSE IT WORKS.

While your experience at Lighthouse as a post-facto test-writer may have been traumatic, all it shows is that "testing after coding" is mind-numbingly boring as hell. Gosh, that was news. Not. However, taking that one specific experience and generalizing from that to all of unit testing is, well, mind-boggling.

Test-driven development is a completely different beast, and I don't know a single developer who has tried it who would ever go back. And I have introduced a number of developers to it, many of them initially quite skeptical.

TDD is *FUN*. And not only is it fun, it generally makes you go faster, write less but better code and dramatically cuts your defect rate, to somewhere near the detection threshhold. And then there's the refactoring goodness, sort of thrown in at no extra charge. (Well, those aspects are part of what make it fun...)

[I say 'generally' above because there are times when it slows you down, but this is more often than not for a good reason, because having to come up with a unit test often shows up that you were less lucid about the actual spec. than you thought you were]

You are right, by the way, that programmers don't know how to test their own stuff, but this applies only to after-the-fact testing, and is of course worse for manual tests than for automated tests. If you write the test BEFORE the code, and never write new code without a breaking test to fix, you have a pretty good guarantee of coverage.

(It is not a perfect coverage, mind you, because your code typically handles more than just the single test case. This is why you should at least try very hard to write the minimal/most specific code that gets the test to run, and only generalize the code when forced to do so by the tests.)

Your 3 'major flaws' don't make sense, at least not from a TDD perspective. First, there is no separate "test program", the tests are part of the program (and if you use MPWTest, the tests are part of each individual class, so every class knows how to test itself). Second, your *model* code *does* get used by other 'programs' (= other code), mainly your view and controller layers.

You have heard of MVC, haven't you? Essentially all your actual functionality should be in the model, and therefore acccessible by API, and hopefully, a well-defined API. In Cocoa terms, it should also be sitting in a separate framework, with any application(s) being a thin wrapper . If you have actual functionality in your application, YOU ARE DOING SOMETHING WRONG. (pardon the shouting, but this is important).

I am not going to comment on the entire rest, except to say that there is some good testing advice hidden between large blocks of logical inconsistencies.

September 23, 2005 6:05 AM

 
Anonymous Anonymous said...

I think Wil's comments are probably relevant for simple consumer apps. However, in real world Enterprise class apps, there is no way this model would work.

My company writes software that when deployed runs on hundreds of machines for the server and MILLIONS of endpoints for the client.

In such a scenario you can't just put it out there and see, you have to test the hell out of it. This type of software is so big you have to use unit tests and some sort of automation.

I've done side software projects that were great to use the model Wil outlined, and it worked great. Beta testers found all sorts of things that we had missed.

So, pick your model and stick with it.

September 23, 2005 6:23 AM

 
Anonymous Daniel Jalkut said...

Well before I make my comment let me come clean that I am "pro Unit-Test" even though I've written relatively few of them. I'm convinced, but I haven't gotten good enough at them to follow through with using them 100% of the time.

I would like to point out that in Wil's repeated comparisons between software and "the natural world," he refers to premature death as nature's way of stomping out bugs.

In the software release world, every "death in the natural world" runs the risk of being quite embarrassing or even legally risky for the software author. If a unit test could have prevented the bug that just wiped a beta tester's drive clean, then I'm all for unit testing.

I think the important message to take from Wil's post is that unit testing can't be the sole testing apparatus for your product. This has always been clear to me, and is professed even by the most ardent sellers of unit testing philosophy.

I haven't used Delicious Library extensively, but I have the impression that the various projects Wil has worked on are, in fact, relatively functional and ship with few bugs. So he must be doing something right. The big question is, say a major refactoring of the entire project is in order - how long does it take Wil Shipley vs. Chris Hanson to throw everything in the air and catch it again with no bugs?

September 23, 2005 6:42 AM

 
Blogger thomas Aylott said...

Obviously there are differences between writing a user-centric Mac play app and coding a framework like webkit or an ATM operating system.

Unit tests are the devil to a small team of developers working on apps where user experience is the most important thing.

Doi.

September 23, 2005 7:22 AM

 
Anonymous BrianC said...

The OmniAppKit and friends all seem to have a suite of accompanying unit tests. I just assumed Mr. Shipley was behind that, but from this post it sounds like that's not the case. Good to have that clarification (assumes my assumptions are correct).

September 23, 2005 7:44 AM

 
Anonymous Justin Walgran said...

I was the "innocent reader" that prompted this discussion. Wil's response is very appropriate given the specific question being asked, namely "When you are a one-man development house, should structured (automated programatic) testing be a part of your workflow?" Even many of the critical comments above end up supporting Wil's assertion that GUI-heavy apps written by 1 or 2 people will not benefit from automated testing. I was not asking about suit-and-tie-M$-backoffice apps, like the ones I develop at my day job, which never cease to suck the life out of me, but rather about the creative and beautiful apps that are coming out of the small Mac development houses these days (Voodoo Pad, Sub Etha Edit, Transmit, and of course Delicious Library). I am curious about the comment made by marcel weiher that it's practical, in any app, for your model code to have automated tests. Wil, would you agree with that point? And thank you, Wil, for the article-length answer to my original question. I appreciate it.

September 23, 2005 9:05 AM

 
Anonymous Anonymous said...

Heh heh eh heh eh heh.

"Unit testing"

September 23, 2005 10:06 AM

 
Anonymous Ryan Bates said...

Saying unit testing is bad or good is like saying the C language is bad or good. It depends completely on the environment it is used in, and how it is used, before considered if it is bad or good for the job. Looking at the comments, I see many different programming environments unit testing has been used in. Here's a few environmental variables that may affect your decision to unit test or not.

Size of the Programming Team
Unit testing is more essential in larger programming teams. Why? A one-mean-team has a complete understanding of the entire program and knows how everything relates. As a result, he knows if a given change will break a piece of code in another class. Larger teams are not familiar with the entire program, and therefore do not know how one change will effect the rest of the code.

Size of the Project
Unit testing is more essential for larger projects. Why? Because larger projects often have more complicated relationships deep within the code which are difficult to test. In a smaller project it is easier to know how a given change will effect the rest of the program.

GUI Driven or Logic Driven
Unit testing is more essential for logic-driven applications. Why? Because, for one, the GUI is near impossible to fully test with unit testing, and GUI driven applications usually contain a lot of view-related logic which again, can be difficult to unit test because it is closely related to the view. However, there are usually areas in any given GUI app that are pure logic which should be encapsulated from the view and possibly unit tested.

There are so many other factors involved in deciding wether unit testing is right or wrong for a given project. There are many different ways a unit test can be performed: Are the tests written before (test-driven development) or after the rest of the code? Will a test be written for every piece of logic? or just the tricky parts/bugs?

I have read a lot about Test-Driven Development, and one thing that seems to be stressed over and over in my readings is that it is not a replacement for other forms of testing, it is used as a design tool. It is really a completely different beast from traditional unit testing (test after the code is written). I am assuming Wil was targeting the traditional approach to unit testing in this article.

September 23, 2005 10:11 AM

 
Anonymous Nicolas Cianca said...

I agree that unit testing is largely a waste of time and money.

However, I also don't think that my programmers should really be doing a whole lot of testing either as programmers are notoriously the worst testers of their own code, partly because they tend to repeat the same tests (e.g. using the same customer id and then I'll try out a different one and it will break).

The absolute best tester (and test plan) I have in my company is a retired school teacher. She is awesome, pays close attention to detail, will try something 20 different ways and find all the ways it doesn't work, and then sit down with the programmer responsible and NOT LEAVE until all the bugs on her list and screen shots have been dealt with. Much better then allocating a whole bunch of engineers with complex unit testing plans....

September 23, 2005 10:58 AM

 
Anonymous keith ray said...

There are many ways to think about the design of a class or function before or during its development or modification. One of those ways uses unit tests (thus "test-driven development"). These tests also known as "executable specifications", "checked examples", "programmer tests", etc.

I figure out what I results I want from the code, and instead of noting that on paper, or in a CASE tool, or in my head, or as non-executable comments, I write a few lines of test code. A few minutes later, I make that test pass. Repeat test-code-refactor until the class or function is done.

I could be doing exactly the same code-refactor sequence without tests, but I find this takes about the same amount of time as designing and developing code WITH automated tests. There is an advantage for TDD: I can run those tests again later, perhaps months later, whenever anyone on the project makes a change to that code or some related code.

Yesterday that found a crashing bug from a code modification that would not have been revealed until certain features were manually re-tested possibly weeks from now. This is where the time-savings comes in. Not in the original coding, but later in the project (possibly days later, or months later).

September 23, 2005 11:01 AM

 
Blogger Wil Shipley said...

Wow. I honestly wasn't just writing crazy crap to generate responses (the "talk radio" maneuver), but I've got to say I'm really pleased to see so many interesting opinions, and I really appreciate them.

Yes, if I were working for Adobe on Photoshop, and there were very specific compliance tests that could be written for each filter and operation, and every change absolutely REQUIRED that no pixel changed even the slightest bit from the test, then, yes, I would do unit testing. (And, in fact they do.)

But, flipping that around, I don't think Photoshop is a really great app. I think there's a lot of cool stuff buried in about a thousand different UI metaphors, and the app needs a real re-think in terms of flow.

So, while I understand the need for unit testing in this case, I still feel it's fair to argue that the time they've spent on it has taken away from time they could have spent making the app better in other ways. (In their case, no, I don't think this was a mistake.)

When you're writing a new app, you don't have tons of users who are going to live or die by your changes during the beta test. If you're writing medical software or back-office or something where, if the user corrupts her files, she's dead, then, yes, you have to be a lot more careful.

So, for those that think I'm irresponsible, I'll certainly apologize for not making my starting criteria more clear: "Assuming you're a small shop and you're working on a new, commercial application for the home...."

---

Regression bugs: when beta-testers report bugs that I "thought I fixed" I pay MORE attention to them, not less. In this case, it's crucial to make sure their bug reports are tagged with the release number.

At Delicious, we built a little Xcode script that places the version number from subversion directly into every build's info panel and into the app's plist, so there's never ANY confusion about builds. No two builds from different code can possibly have the same version number.

If someone reports a bug "I thought I fixed" then I do an 'svn log' and make sure the fix came after their build, not before.

To use this, make sure your Info.strings has a version string with "v0000" in it, and it'll get replaced with the current build version.

BUILDNUMBER=`/usr/local/bin/svn info | egrep '^Revision: ' | cut -f2 -d' '`
INFOSTRINGS_FULL_PATH=${TARGET_BUILD_DIR}/${UNLOCALIZED_RESOURCES_FOLDER_PATH}/${DEVELOPMENT_LANGUAGE}.lproj${INFOSTRINGS_PATH}
sed "s/ (v[[:digit:]][[:digit:]]*)/ (v${BUILDNUMBER})/" < ${INFOSTRINGS_FULL_PATH} > ${INFOSTRINGS_FULL_PATH}.versioned
mv ${INFOSTRINGS_FULL_PATH}.versioned ${INFOSTRINGS_FULL_PATH}

--

My programming philosophy is "less code is bette code." Unit tests take a lot of code, and in my projects I don't find that they find very many bugs. Part of this is because I tend not to modify my individual methods much once they've been written; if my "append string to string" method works, it's really not going to get revisited anytime soon unless it has a bug.

Part of it is because, in fact, I do do integrated testing of a form, and I probably should have talked about that more. Yes, I use "NSAssert" all over the place, and I perform sanity checks and raise errors if there's a problem.

Debugging most errors, once found, should take very little time in a properly-written program, because the error should always be from one of your own sanity checks, and you know where those are in the code, so you just go there and figure out what went wrong. This may sound glib but it has been my experience.

There are deeper errors, but usually I've found them to be dealing with bugs in other parts of the system. I recently found that just linking an app to the Message.framework can cause that app to suck up 90% of the CPU on some machines, for some users, because the status panel (that tells you what your accounts are doing) was going crazy. That took a LONG time to figure out, but it also wasn't something unit testing was going to find for me. (I ended up posing as the status panel and always returning nil in my init method, and the bug's gone.)

In the early days we got all kinds of crash reports deep inside QuickTime's video capture code; those have dried up now. Again, unit testing wasn't going to be a help in finding Apple's bug.

--

Thank you all for your wise comments. I hope that readers will consider my viewpoint and yours to be of a piece -- that is, different thoughts in a discussion about testing.

Up at the start of this Chris said: "I didn't know you could be so utterly and dangerously wrong." That scares me. Of course I can be wrong. I said it in this very post, everyone makes mistakes. That includes me.

In this case, actually, I don't think I'm wrong, but that doesn't mean I'm not. Read my strong opinions, think about my background, and my agenda, and decide for yourself.

-W

September 23, 2005 1:39 PM

 
Anonymous Daniel Jalkut said...

I also use a subversion script to automatically set the build version to reflect the current state of the sources.

Note that with your script, you *can* end up with two builds that have the same version number but are different. You are going to get the same revision number even if something has been modified in the source tree but not checked in.

I decided for this reason to use "svnversion" to get the version number, and retain the "M" that gets tacked on. Now I know instantly from the build number in the about box if, for whatever reason, the build I'm looking at is in fact a one-off that came from sources that haven't been checked in yet.

More info in an entry about this on my blog:

http://www.red-sweater.com/blog/?p=23

September 23, 2005 2:56 PM

 
Blogger Wil Shipley said...

Excellent point, Daniel. I don't worry about this because the guy who does beta-tests (Drew) doesn't usually modify code and is trained very well to update right before any build that's going out to the real world. But this is a real concern, and I like your solution.

I would probably go so far as to force the build to fail if it's an 'install' build and there are uncommitted changes.

September 23, 2005 3:08 PM

 
Anonymous Anonymous said...

Wil, I was just about to, um.. "frankly express my views", but I see that Chris and Marcel have already done so much more throughly than I would have.

Test-driven development and unit testing are what made it possible for CoreData to ship in the Tiger timeframe. CoreData is a very complex framework, and it was developed in a very short amount of time by a very small team (I can't say how small, but trust me: it was *small*, even for a Cocoa Objective-C project), and it is amazingly stable, even as a first release. I probably only ran into three or four bona fide CoreData bugs when I was using the internal pre-release versions.

-jcr

September 23, 2005 5:06 PM

 
Anonymous Anonymous said...

Well. Um. okay. Whatever works for you I guess.

Now for me - as a 6 member team working on implementing a serialization scheme in a multi-plant company that HAS ALREADY shipped duplicate numbers to a government supplier.... long story that began long before my team was brought on....

We're talking about nearly 5 million numbers applied across some 5 thousand parts. Complex, customer-driven specs.... spanning some three systems over 25 years.

Newest ERP system was implemented 3 years ago. In haste. Way too much haste.

Okay. Screw the unit testing? That would kill any chance of us undestanding the SIX different groups who use SIX different production processes.... 85% of the time.l

Integrated testing? Anyone?

But hey, whatever works for you! I mean, it obviously DOES work.... and therefore should apply to anyone/annywhere.

Dave

September 23, 2005 8:41 PM

 
Anonymous Larry said...

This is so wrong it's not even funny. I posted a lengthy response to this on my blog:Wil Shipley on Unit Testing (and why he's wrong)

September 23, 2005 8:55 PM

 
Blogger Wil Shipley said...

Dave: You should really read all the way to the end of the comments before you put words in my mouth, since I SPECIFICALLY said what conditions I was talking about just, like, four posts ago.

Larry: If you find tests help you, go ahead and use them. I find it slows me down, and I know what to test when I make a change.

JCR: I'm glad CoreData was stable, as I'm using it in Delicious Library 2. But I don't write kits, I write apps. Nor do I write software for the medical profession or to run nuclear reactors. If I did, I would take a different approach.

Everyone: If you like my software and are curious how I wrote it, I occasionally write about my approach to software design and life in general in my blog. If you'd like to go your own path, please do.

September 24, 2005 1:29 AM

 
Anonymous Larry said...

Wil,
I certainly don't mean that you must conform to my way of doing things. My fundamental point (which I may not have made clearly), was that the post basically said "Unit Tests are always a useless waste of time", which may be true for you, but certainly not to most others. In most situations, writing them is better than not writing them, particularly if multiple people will be working on the codebase.

September 24, 2005 7:49 AM

 
Blogger Marcus S. Zarra said...

Wow, this entry could have easily been written on Slashdot! Amazing the number of people that don't read the entire post before writing a reply.

My first reaction to Wil's entry was "What the hell!?!". But then I read the entire thing, the comments and thought about it. Looking at it from a GUI point of view, unit tests are a waste of time. Writing a back end server processes or an API is a lot different then writing a GUI. In every situation I have run up against, both freelance and corporate, unit tests on a GUI are a waste.

On a Library or Back-end server process I am a firm believer in TDD. For GUIs I would not waste my time.

September 24, 2005 9:58 AM

 
Anonymous Anonymous said...

Will, I need to calibrate my intake valve. To wit, have you made any major changes in your workflow over the last 10 or 15 years?

September 24, 2005 3:37 PM

 
Anonymous Anonymous said...

Thinko. I meant "Wil".

September 24, 2005 3:38 PM

 
Anonymous leeg said...

Well, everyone else has chipped in on this thread, I thought I might do too :-) I'm a sysadmin who does some development (Cocoa, WO, NeXTSTEP, perl, python, {c,ba}sh, you name it) and my approach to testing is, well, variable. I disagree with Wil's complete dismissal of unit testing - it's a great timesaver when you don't know WTF you're doing ;-). OK, maybe you don't suffer from that so much, but I do...if you can't think holistically about a project, I find the act of writing unit tests a lot more objective than that of writing flow charts. It's also easier to chart your success when implementing code around a test framework - you can see how many tests now succeed. In these situations testing clearly isn't a blanket waste of time; although it's never as bad as you make out because you don't have to rewrite all your program logic if you [i]are writing reusable objects [ii]haven't written it yet...

When it comes to complete programs like scripts, I try to write them as UNIX pipelines then just test by hossing a load of data in one end and looking at what comes out the other, and that sometimes is the limit of my test. Sometimes that data is production data. I never test GUI code though and my reasons are fairly similar to yours - better just to use it and see who breaks what.q

September 25, 2005 1:27 AM

 
Anonymous Dominik Wagner said...

Thank you for publishing your strong opinions about programming and running your business! I enjoy your opinion and the resulting discussions very much, please do continue, and keep your opinions strong ;-)

And, of course, my own 2 cents:
- Unit testing is great for non-changing api's or frameworks.
- In the GUI world I don't know of any structured testing that makes sense whatsoever. Especially when in development change is the only constant. And if you change the workflow and ui, you get quite a useless test overhead to change too, or worse, rewrite.
- In SubEthaEdit our text transformation code has systematic tests, and they are important, because one fault could cause whole documents to get out of sync. And there doing the systematic test, that tests all cases can be done, and has to be done. In the UI of SubEthaEdit? no way. no one can test all the ways through the UI, and we consider SubEthaEdit to have fairly simple and UI..
- In my ruby on rails projects i do unit tests, they are easy to create and test driven developmen makes sense there. However I hate changing them when I do major changes to the project. It's like documentation: It gets out of sync too easily if you lack discipline. That's why I like objective-C so much: the long syntax makes adding comments for understanding quite obsolete if you choose good names.

That's it, please keep up the informative blogging! (and add some network features like sharing the library in the local subnet and let others register their wishes in my library, so i finally be a delicious library user ;-) )

September 25, 2005 2:03 PM

 
Blogger Mike said...

If you are a professional, writing software that real people will pay real money for, then you should have a real QA person working for you -- someone who is not doing the actual coding of the product in question. There is no substitute for thorough, structred third-party QA testing -- unit tests and beta testing can supplement, but not replace a real QA process.

It comes down to the fact that you can't "proofread" your own work. Stephen King doesn't edit his own copy, nor does he say to a friend "hey can you read this novel over for me and check it for typos" -- he has a professional copyeditor...i.e a QA person.

September 27, 2005 7:38 AM

 
Anonymous Jay Laney said...

For application development, I agree.

I do love me some cronned automated tests in web development though.

September 28, 2005 9:59 AM

 
Blogger Doug Simons said...

I have to disagree with Sven Schmidt's comment that "there's no good way to do automatic testing" of GUI apps. Many people aren't aware that tools exist that can do a good job of automated testing at the GUI level.

As co-author of one such tool (Eggplant, from Redstone Software), I can say that our customers have found it extremely valuable both for regression testing and for finding and tracking down new bugs. It's also universal enough in its approach to allow testing interactions across applications and automating any processes "just like a user", which lets you test, for example, the installation and integration of your app within a real user environment.

Of course, creating automated tests always takes a certain amount of effort, so it's important to take an intelligent approach to testing, balancing your efforts between defensive programming, unit tests at the framework level, automated GUI tests for user interactions that you want to test after every build, and of course, manual testing for how the application "feels" and bullet-proofing from crazy-user actions.

For someone with Wil's skills and self-discipline, unit testing may be more trouble than it's worth. Other folks have found it to be very beneficial.

Automating at the GUI level may not be appropriate for every project, either, but the benefits of automated testing (repeatable results, regression testing that you're never going to do manually, iterating over large data sets, etc.) are compelling in many situations.

Finally, Wil, I agree with most of your advice about prioritizing your efforts to include testing that helps you deliver a really usable product, but I've got to respond to the remark that you've "NEVER, EVER seen a structured test program that (a) didn't take like 100 man-hours of setup time, (b) didn't suck down a ton of engineering resources, and (c) actually found any particularly relevant bugs."

That's a challenge that I'd like to engage you on! Eggplant (in my obviously biased opinion(!) but based on actual experience):
(a) is so easy to learn that you can be writing tests within an hour of downloading it;
(b) is so straightforward that you can actually create regression tests in about three or four times as long as it would take to do the same test manually (so you get a net time savings for any test that you run at least 5 times; more complex data-driven, performance, or load tests will take more time up front, but allow you to do things you'll never do manually); and
(c) has enabled us to track down tough bugs like a pesky, intermittent failure that was caused by a threading issue that only showed up about every 50th time that we executed a particular action -- by running a script repeatedly all night we were able to capture the information we needed to nail it (turned out to be a bug in Apple's frameworks that we could work around). Clearly this wasn't something that unit tests would have caught, nor would we have wanted to devote the mind-numbing time needed to find it manually, or to prove to our satisfaction that the workaround actually solved the problem.

So, I'd like to invite you (or anyone else) to give it a try and tell me what you think. I'm not saying Eggplant is the panacea for all testing woes, but I think you'll find it can do things that other automation tools can't. Which may open your eyes to new possibilities.

I apologize if I sound like a marketing guy here, but since we are talking about how to make the best Cocoa apps possible, I think people should at least be aware that unit testing tools aren't the only approach to test automation, and for heavily GUI apps in particular a GUI testing approach may make sense as well.

October 04, 2005 11:50 AM

 
Blogger Wil Shipley said...

Doug: Actually, the way I learned things (and I do *NOT* claim it was the correct way, it was just at some huge cellular phone company that'll remain nameless, which is fine because it's been bought out twice already) I think what you're suggesting is what I call "System Testing".

I'm all for automated system testing, assuming you can come up with good tests without spending a lot of resources on them. It's "unit testing" I hate.

To define my usage, "unit testing" is when you write code that tests each individual method (or a small group of methods) to make sure that, given a certain set of inputs, they produce an expected output.

The reason I'm so against it IN THE CASE OF NEW, GUI APPLICATIONS is because, basically, you just don't write methods that have clean input->output type models, and most errors I've seen aren't the kind that would be caught by unit testing anyways -- if you're smart enough to write your unit tests to catch the errors, you're smart enough to have written your original code defensively. Plus, you have to maintain your unit tests.

But your challenge seems interesting -- if you contact me directly we can talk about it.

October 04, 2005 4:22 PM

 
Blogger Doug Simons said...

Wil, I basically agree about unit testing. I'm intrigued by the "write the test first, then write the code to make the test pass" idea, but haven't found too many cases where I've been able to make it work for me. I thought I just hadn't put in enough time and effort to get in the right mindset, but I think a lot of my code just doesn't lend itself to unit tests -- the effort required to write the tests exceeds the benefit (your point, basically). I think bbum's comments that unit tests work well for "infrastructure" code make a lot of sense. But most of my infrastructure code is (thankfully) written by Apple, not by me!

System testing, as you call it, validates your code from the point of view of a user -- can they perform certain tasks with your software, which likely involve the interaction of dozens of different "units" within your code. The individual units may all work fine, but still produce unexpected or undesired results when combined. Automating regression tests at this level can help ensure that you don't break user functionality as your software evolves.

And yes, I'll be in touch.

October 05, 2005 7:09 AM

 
Blogger Alan said...

If a test is good becuase it is done once, by a beta tester, then why is it not good to re-do, and re-do that test ?

Why is it good to fix, but not to make sure things stay fixed ?

October 07, 2005 3:25 AM

 
Blogger Wil Shipley said...

If your beta testers are sufficiently skilled to create unit (or system) tests and check in that code and maintain the code as APIs change, then by all means, go to town.

If not, then you're going to spend a bunch of programming time and effort writing and debugging the tests. Humans are MUCH better at doing things the first several times than computers -- it's only when you are doing something thousands of times that it's worthwhile to write a program to do it.

My point is not "automated tests are bad," it's "automated tests of most UI-based apps are simply not worth the effort for the return you get, compared to a beta program."

October 07, 2005 4:31 AM

 
Blogger Chris said...

I'll keep it short: whatever test a human being would run, wouldn't it be better to have it in a format where a computer can run it consistently and repeatedly? How is having a human being only do it once EVER, EVER better?

Furthermore, your conceptual point about people using software rather than software using software is not true, at least not in a tiered architecture. Human beings should be accessing your UI layer, but not the data layer, for example. This means that the inputs and outputs should be standardized and validated at the layer boundary. If they're not, well, there's your problem.

Yes, web UI unit testing currently sucks terribly, so you've got some karma there.

The problem here is not unit testing, but that the software community is still figuring out exactly how to write unit tests. I just recently wrote a database testing system that does nothing less than attach a temp copy of the database and throw it away before and after every test. It's quite a lot of effort to get what was needed.

Unit testing is hard to do at all, and very hard to do well, but your comments just don't square with my experience.

October 18, 2005 11:21 AM

 
Blogger Shawn Oster said...

Whoa, TEH SUCK is freaking white text on a freaking black background, are you trying to fry our eyeballs?

Ok, a few things...

- When was the last time you setup a xUnit style test framework? It takes about 5 minutes, I timed it in both Delphi and .NET.

- Unit testing is *all* about writing code to break your code. Unit tests should be written to break things... pass invalid indices, use things that aren't init'd, load massive amounts of data, etc. Your example of loading up 2k worth of items... if you do that only via the UI then what are you testing? Your framework or the UI? Maybe someone just wrote bad button handling code vs. your framework choking.

- Automated testing != Beta/Meat testing != unit testing. You need all of them.

- 21 years of experience matters for little unless it's in that field. "Programming" is so vast that saying someone has 50+ years experience doing it isn't saying anything. I have 20 years "programming experience" but I only really have about 10 in for OOD and another 10 is structured. I've interviewed people with 30+ years writing straight procedural C code and that was almost a detriment for them since I needed someone that could wrap their brain around OOD.

- It's very odd that you've pited these things together when they should be going hand-in-hand. UI and interaction testing is different than framework/API/library testing and they should be different.

- white on black still sucks :)

- Unit-testing/TDD becomes even more important when you are working in a group, sometimes a core-group of 2 or 3 and then contractors coming and going so you'll have 10 to 15 different people touching code at different stages. That newbie coder thought he just optimized a line of code? Nope, he just introduced a bug that had been working for ages. It worked for him, even when he tried to break it, but gee, he didn't realize that routine was called about a billion times from another part of the framework.

- My 21 years experience sees your 21-years and I raise you 8 years of very successful, very useful, very money saving benifits from using true TDD.

- Programmers are all egotistical soapbox raving unwashed semi-deities in their blogs, until they get into large social groups and all become suprisingly mellow. :)

October 18, 2005 1:51 PM

 
Anonymous Jeff Atwood said...

Yeah, you may have 21 years of programming experience, but Wil drives a TOTALLY HOT CAR.

Seriously, have you seen his car? Because it's awesome.

October 19, 2005 1:45 AM

 
Blogger Wil Shipley said...

Dude, if you've been programming for 21 years this really can't be the first time you've read white text on a black background. Unless you am from Bizarro-world? You not eat hot dogs?

October 19, 2005 2:10 AM

 
Anonymous Anonymous said...

Beta testers should never be writing unit tests. Do you send beta testers source code? It is impossible for them to write a unit test otherwise.

Also, unit testing is not designed for UI testing, and while I've seen people do it, that is certainly not a worthwhile investment of anyone's time, IMO.

Now, if you've been insane, and written all your business logic in UI code (eg event handlers), unit testing is not going to help you very much. Of course, then your code is not particularly maintainable either.

Of course, you (or better, someone else) should also do beta-testing, and perhaps other things, depending on the application.

October 19, 2005 4:12 AM

 
Anonymous Anonymous said...

i totally agree.. unit testing has its place but testing of the app banging on the keys.. inputting from left field , has its place too.. we test for not only what its suppose to do BUT what its not ... some only test for postive feed back and call it done.. and you need those beta testers to ATTACK it.. if you dont you will never really know.. no (2) humans think alike !!!

October 19, 2005 8:15 AM

 
Anonymous Anonymous said...

The term "teh suck" is extremely annoying; like your atitude towards a successful and proven way to test software. Get a clue ass-hat.

October 19, 2005 9:01 AM

 
Blogger Wil Shipley said...

Successful and proven are relative terms, friend. If you're not convinced already that my achievements are worth emulating then you are probably reading the wrong blog.

October 19, 2005 10:23 AM

 
Anonymous Anonymous said...

Automated unit tests are crutches we hand out to mediocre programmers. Wil, you're obviously what I'd call a "superstar" developer. You understand the architecture and design of the code you're working on so well that you don't need a low-level test to tell you the side effects of your changes.

Of course, superstars are hard to find. Most developers can't see the forest for the trees, and these low-level tests are a way to protect the product from their ignorance.

I hate them. But then, I'm a superstar too.

October 19, 2005 11:43 PM

 
Anonymous Jeffrey Palermo said...

My comment will be short. You make good points about the importance of testing in general. Too few people adequately test their programs. You main point, however, and the title of the post, "Unit testing is teh suck, Urr", is wrong, and you don't support it within your post. Your post should be, based on the content, "The importance of ruthless testing".

The statement, "Most programmers don't know how to test their own stuff. . ." is used as a reason why unit testing is inherently without worth. This is a logical fallacy, and no matter how many people do something badly, the something cannot be evaluated solely based on that. If we think that there are many lawyers who abuse our justice system and laws, does that alone pass judgement on the profession or our justice system?

Unit testing is hard. Decoupling software so that it is testable at the unit level is hard. I personally find that the term "unit testing" is used when the automated tests are actually integration tests. I find unit testing to be rare, actually, but the term is abused.

Nothing in this post has supported the argument that "Unit testing is teh suck, Urr."

October 20, 2005 8:06 AM

 
Blogger BlackTigerX said...

"if I hated this code, how could I break it"

amen

most programmers who write unit tests will write weak (and buggy, and will take 3 times as long) tests anyway, so what difference does that make

October 20, 2005 8:16 AM

 
Anonymous Jeffrey Palermo said...

BlackTigerX,
"What difference does it make?" The folks who do unit testing well will never encounter the same bug twice. Just because some people fail doesn't make it a bad idea.

Most new businesses fail, would you conclude that it's not worthwhile to start a new business?

Those who do it well reap huge benefits.

October 20, 2005 9:26 AM

 
Blogger Ideasculptor said...

Something that was entirely ignored before I got bored reading comments and skipped to the end:

Some programmers write code that is much more bug free than others. There's no getting around that fact. Some are just naturally better programmers, others just include a meticulous and effective inspection of any code that they've completed, others have their own processes, including TDD, which works for them. I've led an awful lot of software engineering teams over the years, and inevitably, some guys generate code that is much buggier than others.

If you happen to be one of those meticulous or talented guys whose code is less buggy by the time it compiles, then you can maybe get away with less unit testing, especially on a 1 or 2 man project, and especially if you are a top notch engineer who is able to maintain knowledge of the entire complexity of a project in your head. Judging by his body of work, Wil is one of those guys. I know that I am. I am consistently the fastest coder in any engineering group I work in, not because I write code faster (I probably spend much more time just sitting and thinking about it before writing a sigle line), but because I spend a lot less time getting it to work after I'm done writing the initial pass of the code. When taking advice/suggestions from Wil, it is wise to remember that he is a top tier engineer, both in skills and, apparently, in experience. You may not be.

When I do 1-man projects for myself at home, I do next to no unit testing, and then only if forced by a recurring problem of some kind. My time is better spent actually fixing problems or improving the product. But as soon as I am in a multiple-engineer environment, that all goes out the window. You have to use the minimum process necessary to keep the lowest common denominator of talent on the team productive, and that inevitably means unit testing, especially as the complexity of the project grows and every engineer doesn't have a clear picture of how every line of code in the project functions. If the lowest common denominator is too low for the rest of the team, perhaps it is time to reveiw your interview/hiring practices.

I don't know Wil and I don't read the blog regularly (but I do have a licensed copy of Delicious Library - something I've never actually used because I don't loan books - I give them, but I liked the interface.), but it seems to me that to advocate that unit testing is 'teh suck' in a public forum without being explicit about the context is, to some extent, showing off. I could tell people not to unit test because can develop successfully without it, but I don't, because I know it to be bad advice. I suspect that Wil does, as well. At the very least, most of his readers apparently do. On the other hand, I don't really fault someone for showing off a bit when they can. Those of us who make our living writing closed source software (especially those of us who don't write end user desktop apps) rarely get an opportunity to show off our chops or get any recognition at all. Heck, a significant percentage of all the bytes that traverse the world wide web (especially commercial bytes) get touched by code I wrote at some point in their journey, but no one would know it, since none of it is open source, used directly by an end user, or is published with a list of contributing developers (there, that was my bit of showing off).

In the end, you should test wherever testing is a benefit and is likely to save more time than it costs you. Only the developers on a project can accurately judge how much testing, and what kind, is necessary for that project. To suggest otherwise implies that you think you know more about how someone else works than they do, which isn't terribly likely. Being able to accurately judge what kind of testing is necessary is something that you only gain with experience. It is why software architects and lead engineers with lots of experience get paid more and can take on more responsibility than more junior engineers. No single development methodology works well in all environments, and their proponents would do well to keep that in mind. A good development team will put together the most effective hybrid methodology for their project (and skillset).

October 22, 2005 12:40 AM

 
Blogger Wil Shipley said...

I wasn't actually trying to show off; I was trying to dis "unit testing" as opposed to "system testing", but apparently not everyone defines those terms the way I do.

I haven't actually seen anyone defend "unit testing" here -- they've all defended what I tend to call "system testing" which is a different animal. Unit testing as I use it is when you write test code for each method to make sure its input and output match its definition. I find this useless.

I try not to give advice that assumes the reader is a God Among Coders, whether or not I consider myself one. :)

October 22, 2005 10:47 PM

 
Anonymous Jim Cooper said...

Wil, if you "haven't actually seen anyone defend "unit testing" here" you haven't been reading very carefully.

I find it astounding that you think unit testing is useless. Of course, you very much sound like you haven't ever done it properly. Writing unit tests speeds development, and enables refactoring to be used with confidence, resulting in more maintainable code.

It is most definitely an error to have writing unit tests for someone else's code as the norm. I'm still confused as to how you ever get beta testers to write them, as you claim to have done.

"System testing", as you call it, cannot possibly test every method you write. It is by definition far too slow. Therefore there will be untested code in your application.

However, "system testing" is required for UIs. Unit testing was never intended for that purpose. I've written projects where the UI was fairly complex, as was the underlying business logic/data access framework. The latter had unit tests and the UI did not. Guess which bit was the flakiest? If you are fortunate enough to have a good testing person, then many of the UI bugs can be found. But it is orders of magnitude slower than unit testing (because repeating a unit test takes no time at all, whereas repeating a test by having a human hammer a keyboard takes ages). It's an expense you have to bear, of course.

Now, if you were talking about whether automated "system testing" was worthwhile, that would be a more interesting discussion.

Finally, the amount of UI testing has a limited bearing on the usability of that UI, IME.

October 25, 2005 10:03 AM

 
Anonymous Anonymous said...

more like eunuch testing ... amirite?

November 07, 2005 3:28 AM

 
Anonymous Anonymous said...

What if you haven't tested your tests, though? Doesn't the code that you are using to test other code need to be tested too?

At some point you need to ignore testing to get any work done - because testing your test code, and testing the tested test code which was tested, will send you into an infinite loop.

Draw the line at some point.

One way to test code for correctness, is to have one programmer take a look at several functions, procedures or classes you have written. Have the second programmer come in and take a look at the code you wrote. Most of the time the second programmer will be very critical of someone else's code.

But don't go as far as doing this for every single procedure or function you write. Why? Well do it in bulk batches. Write 10-15 procedures or functions and then have the second programmer take a look. If you have the second programmer analyze each line of code you write in real time, it's just not convenient. That is the problem with pair programming - some of us don't have the convenience to get together on saturday, because one guy is a busy dad and the other guy is a single person with free time.

The hard part is finding a second person to analyze your code who agrees with you about general programming concepts. i.e. finding a partner is hard.

Then again, some times two programmers on each other's back is worse than just one guy cranking out code. It depends if the two programmers get along and agree with each other on programming practices. It can cost you more time if two programmers don't agree and end up arguing about procedural versus object oriented programming, instead of getting work done.

The bottom line is that there is no sure and proven way of doing things. When two programmers can agree with each other and test each other's code out in a positive but critical way, this is when excellent software can be built.

Having another programmer test your code, and having beta testers - is a good way to get testing done without going into the infinite loop of testing test code with test code which tests test code.

For correct software like medical software and rocket ship software, I could see the benefits of all sorts of testing (a combination of testing, including writing programs to test your programs). Even with compilers (which must be correct), more testing must be done. Compilers must be correct, so writing a compiler means doing lots more testing than writing a GUI application.

With most applications and libraries (i.e. not rocket ship software, just desktop software or web libraries) I'd argue that maybe handing your code to another programmer to analyze, would be faster than writing tests yourself.

Other programmers are more critical than you are about your own code. And I'm not talking about real time pair programming here - I'm talking about handing your code to another programmer after you have done a bit of work - 10-15 procedures or functions, or classes.

Now understand that the programmer who is checking your code, must be critical - he can't be just some easy going programmer. He has to look into your code and see things like "this isn't cross platform! you missed the carriage return for UNIX operating systems here!"

This one line feed error catch above, could save you lots of time, without writing all these complicated "unit tests" . Imagine if you had to test your application on 3 different platforms? Are you going to redo your tests on each platform, each time you change code? i.e. fire up a Mac, Linux, and Windows box to test your code every time you make a new class or procedure? I'd do batch testing instead. It's more productive.

Regards,
L505

January 14, 2006 2:39 AM

 
Anonymous paris hilton video said...

more like eunuch testing ... amirite?

March 20, 2006 6:24 AM

 
Anonymous Alex said...

Look at SWExplorerAutomation
(http://WebUnitTesting.com).

It is very simple to write Web UI tests - no training needed.

March 23, 2006 3:33 PM

 
Anonymous Joel said...

I agree with you completely. I hate unit tests, they completely remove all the fun from programming.

June 14, 2006 5:21 PM

 
Anonymous Dan said...

k i sing with u ... "TEH SUCK" :-)

January 27, 2007 5:45 AM

 
Anonymous Knuddel said...

damned u r good man. really excellent article. thanx 4 sharing ur ideas!

February 04, 2007 10:00 AM

 
Anonymous biggi said...

Interesting stuff with some good intentions and useful informations, thank you for sharing your thoughts

March 30, 2007 5:26 AM

 
Anonymous Teksty said...

Unit tests make sure your code works correctly, keeps working correctly and, most importantly, allows you to refactor much more freely.

April 16, 2007 3:13 AM

 
Blogger Sergio said...

"if 1,000 users report a drawing glitch, and one reports a crasher, which one should you fix? THE DRAWING BUG."
This is OK if you write software for entertainment. This is SO DAMN WRONG, if you are writing embedded software for... say, a car or an airplane! "People die", you say - well, you don't want their blood on your hands, do ya?

June 09, 2007 1:21 PM

 
Blogger Shalev said...

I feel as if a great light has finally dawned upon me. The dark hypocrisy's of people yelling "Don't Repeat Yourself!", and then turning around and writing their application twice have finally given way to the light of a new way.

The fact is that code that fully tests an application will always be bigger than the application code itself. This has always grated against the grain of my least-code-possible programmer soul. I thank you for finally validating what I others claimed a defect in my programming nature.*

* (And yes, I understand the need for extensive HUMAN testing vs. machine testing.)

July 16, 2007 8:50 AM

 
Blogger Jason said...

I'd love that list of test scenarios you use to to maliciously break your code and the process you then use to pass off to another developer who may be asked to maintain your code to replicate those test scenarios and ensure that they haven't broken something elsewhere in your system.

October 15, 2007 1:05 PM

 
Blogger Wil Shipley said...

process you then use to pass off to another developer

I don't pass my code on to other developers. I should be specific -- my testing practices work for a certain environment and a certain type of software. If the inputs change, the response should change as well.

October 15, 2007 1:09 PM

 
Anonymous Anonymous said...

Wil is soooooo right. "if you're smart enough to write your unit tests to catch the errors, you're smart enough to have written your original code defensively" and that's it. Thanks for the post, Wil.

October 23, 2007 12:07 AM

 
Anonymous Anonymous said...

Could we stop pretending people who develop for OS X are actually programmers?

You're not developers, you're traitors.

November 05, 2007 3:12 PM

 
Blogger Wil Shipley said...

You're not developers, you're traitors.

I thought I was a traitor because I object to Americans killing people for no reason.

I AM SO CONFUSED!

November 05, 2007 5:02 PM

 
Blogger Wil Shipley said...

Alternate response:

Well, then I guess you mom is going to jail for providing succor to the enemy, then.

November 05, 2007 5:03 PM

 
Anonymous Anonymous said...

I used to work for a large property company, and they hired in a team of uber developers (supposedly) who fully relied on unit tests.

Every time they changed code, they'd run unit tests through it. The unit tests came back clean, and they'd release the code.

Problem was they had written code around the logic in their brains, which in the real world doesn't work...e.g. a field in the database to say if a property was rented or for sale, the field set as null initially, they assumed that null also meant false. They never set this field, the unit tests were constructed around this logic, but when the information was set via webservices it broke...webservices don't like null values being coverted to strings and back again!

But they denied their code was at fault as it had passed unit tests. We struggled with this for weeks (well i didn't) and when i looked at their database, asked them to make sure the field was set to true or false, the unit tests still passed, and the webservices work.

My point is really, yes unit tests can be helpful but time consuming and will never cover every situation...so beta testing is still the best way to make sure code works.

January 07, 2008 1:44 AM

 
Anonymous Anonymous said...

I don't get it. If you follow Wil's advice you will be doing lots of ad hoc testing. Why not capture it? *looks confused*

December 29, 2008 3:16 PM

 

Post a Comment

<< Home