I wrote a few times in the past about testing, and I think that I’ve finally got to a good point with how I think software should be tested.
First off, I think that most unit testing is stupid. Unit testing is awesome for testing algorithms and I’m not going to knock that one bit. The problem is that after running most unit tests you’re given a report about how much code coverage you’ve gotten from your testing. Well, that’s not really the problem — the problem is that now that you’d measured something, you’ll be measured by that metric as well.
The better testing in almost all cases is integration testing. It’s testing the software that you’ve written in the same way it’ll be run in a production environment. Add on some fuzz testing of calls to generate errors and you can exercise almost all of your code. This has a few problems of course. Often you won’t have a reliable test environment in cases where you have a heavily SOA system; you should fix that even without automated integration tests. Second, you generally don’t get code coverage reports from the integration tests.
The fact that you don’t get a report from your integration testing means that it’s really easy to not get measured on it.
I’ll offer this comparison: Would you rather have a system with 95% unit test coverage and no, or few, integration tests, or one with 30% unit test coverage and 60% integration test coverage? Personally, I would pick the latter option every time — the fact that I have a running system with all of the happy case scenarios working along with common errors gives me confidence in the system actually working in real life. The issue is that if you don’t have a report that includes coverage amounts for the integration tests you’re left with comparing 95% and 30%. Try explaining to your boss why the 30% is better.
Just because you can measure something doesn’t imply that that particular metric is better or more important than other attributes that are harder to measure. Sure, you could argue that you should work harder on measuring the other one too, but that’s not important to the outcome, really.
Testing is important. Testing well is even more important — because that’s how you get confidence that your system works.