My answer, perhaps more politely phrased, was, like another famous Wil(l), "AW HELL NAW!"
I'm curious to know how you approach product testing in such a small company. Do you have an established process? Do you use unit testing? Other structured testing? [...] Do you have suggestions specific to solo testing and debugging?
I've certainly known companies that do "unit testing" and other crap they've read in books. Now, you can argue this point if you'd like, because I don't have hard data; all I have is my general intuition built up over my paltry 21 years of being a professional programmer.
But, seriously, unit testing is teh suck. System testing is teh suck. Structured testing in general is, let's sing it together, TEH SUCK.
"What?!!" you may ask, incredulously, even though you're reading this on an LCD screen and it can't possibly respond to you? "How can I possibly ship a bug-free program and thus make enough money to feed my tribe if I don't test my shiznit?"
The answer is, you can't. You should test. Test and test and test. But I've NEVER, EVER seen a structured test program that (a) didn't take like 100 man-hours of setup time, (b) didn't suck down a ton of engineering resources, and (c) actually found any particularly relevant bugs. Unit testing is a great way to pay a bunch of engineers to be bored out of their minds and find not much of anything. [I know -- one of my first jobs was writing unit test code for Lighthouse Design, for the now-president of Sun Microsystems.] You'd be MUCH, MUCH better offer hiring beta testers (or, better yet, offering bug bounties to the general public).
Let me be blunt: YOU NEED TO TEST YOUR DAMN PROGRAM. Run it. Use it. Try odd things. Whack keys. Add too many items. Paste in a 2MB text file. FIND OUT HOW IT FAILS. I'M YELLING BECAUSE THIS SHIT IS IMPORTANT.
Most programmers don't know how to test their own stuff, and so when they approach testing they approach it using their programming minds: "Oh, if I just write a program to do the testing for me, it'll save me tons of time and effort."
There's only three major flaws with this: (1) Essentially, to write a program that fully tests your program, you need to encapsulate all of your functionality in the test program, which means you're writing ALL THE CODE you wrote for the original program plus some more test stuff, (2) YOUR PROGRAM IS NOT GOING TO BE USED BY OTHER PROGRAMS, it's going to be used by people, and (3) It's actually provably impossible to test your program with every conceivable type of input programmatically, but if you test by hand you can change the input in ways that you, the programmer, know might be prone to error.
So, listen closely, because this is the method I've used since the dawn of time, and it works great:
1) When you modify your program, test it yourself. Your goal should be to break it, NOT to verify your code. That is, you should turn your huge intellect to "if I hated this code, how could I break it" as SOON as you get a working build, and you should document all the ways you break it. [Sure, maybe you don't want to bother fixing the bug where if you enter 20,000 lines of text into the "item description" your program gets slow. But you should test it, document that there is a problem, and then move on.] You KNOW that if you hated someone and it was your job to break their program, you could find some way to do it. Do it to every change you make.
When I test Delicious Library, I'm like, "Hey, to make sure my new queue works, I'm going to take a Library of 2,000 items and tell them all to reload themselves simultaneously." Now, honestly, in version 1.x, this is really slow. But, dang it, it works. And since our goal for version 1.x was to have it work with between 1K-2K items, I'm OK with this.
Too often I see engineers make a change, run the program quickly, try a single test case, see it not crash, and decide they are done. THAT IS TEH SUCK! You've got to TRY to break that shit! What happens when you add 10 more points to that line?
2) When you get the program working to the point where it does something and loads and saves data, find some people who love it and DO A BETA TEST. The beta test is often maligned, but the most stable programs I've ever written are stable because of beta testing. Essentially, beta testing is Nature's Way (TM) of making systems stable. You think nature creates unit tests and system tests every time it mutates a gene? Aw hell nah. It puts it out in the wild, and if it seems better it sticks around. If not, it's dead.
Now, if you're a pure-CS kind of person, you're thinking, "But, with beta testing, I might miss some really deep, conceptual bugs that unit testing would find, because there's only certain pathways people will test..." Well, there's really only one rational response to this:
Like in nature, any system is going to have bugs. And, again like nature, most of the time, in most situations, you want your system to work. But occasionally, under some circumstances, it could fail. This isn't great, but it is acceptable. The nice thing about beta testing is it lets you focus YOUR precious time as a programmer on the pathways that fail most often under normal use patterns, instead of just finding bugs at random that may or may not ever be triggered, and sucking up your time fixing them. (Hey, if you want to try to fix every bug in your program before you ship, be my guest, but at least fix them in order of how often they are reported, OK?) Unit testing doesn't give you information on how common a bug is, which is the single-most important piece of information about a bug, followed very closely by its severity.
Look, I know it's heretical to say "software is going to have flaws." But everything is going to have flaws. Pilots screw up. Surgeons screw up. People die. Denying this doesn't make it go away, it makes it worse. We need to say not "never screw up," but "make sure when you screw up, you recover nicely."
Make sure that the flaws your program ships with are the least-common ones possible, given the time you have to fix bugs. Honestly, crashers are more important than, say, drawing bugs, but if 1,000 users report a drawing glitch, and one reports a crasher, which one should you fix? THE DRAWING BUG. DO THE MOST GOOD WITH THE TIME YOU HAVE. It's your duty on this earth, and it's also the fast route to success.
Now, all this is NOT an excuse to write crappy subroutines and say, "Well, it'll got caught in the beta test." You, as a programmer, should be programming EVERY LINE as defensively as possible. You should assume that every other method in the program is out to kill you, and they are intentionally passing you corrupted buffers and pointers, and it's your job to handle all that crap and make the best of it. (And, if you can repair it, do so, and if not, raise an exception so you can catch it in beta testing, instead of silently failing and/or corrupting data.)
Let's sum up: Write good code to start with. Write every routine assuming that it'll be called by chuckle-heads. When you modify your program, test it yourself, and TRY to get it to fail, don't try to get it to pass. Then set up a real beta program, with a good pathway for sending in feedback and categorizing it, and let the beta-testers do the rest.
Testing is hugely important. Much too important to trust to machines. Test your program with actual users who have actual data, and you'll get actual results.
November 20, 2005 followup: My comments here have been widely reported, and sometimes misconstrued because of my own failure to disclose the boundaries of when and why I don't use Unit Tests. I was thinking of writing a follow-up, but bbum has done it for me, and I agree completely with what he says. So, please consider his post as the continuation of mine, only, you know, written by him.