Adventures in Qualitude

Current project is in a crunch and needs to be submitted to Apple today. This means that my iDevBlogADay for this week will be short. It also means that I’ve been testing a lot of code, and working through bug reports for the last week or so. And there’s a lot to be said about good and bad QA practices.

In the iOS world, we tend to see small developers, and a lot of work-for-hire directly for clients, as opposed to some of the more traditional models of large engineering staffs. As a result, you may not have a real QA team. This can be a mixed blessing. Right now, I’m getting bug reports from the client, who have an acute awareness of how the application will be used in real life. This is a huge advantage in that all too often, professional testers are hired late in the development cycle and come in when things are nearly done and ready to test. The problem is that if the nature of the application is non-intuitive, the testers will need time to develop a grasp of the business problem the app solves.

What they do in the meantime, inevitably, is to pound on the user interface. Along with a lot of arguments about aesthetics, you tend to get lots of bug reports that start with the phrase “If I click the buttons really fast…” These bugs are rarely 100% reproducible, but they are 100% annoying. Here’s the thing: I’m not happy if, say, you can pound really fast on my app and get it in some visually inconsistent state. That’s not right. But if you’re not using the application in a gratuitously non-useful way, does ending up in a non-useful state really matter?

The real sin of this kind of testing approach is that the important bugs — whether the business problem is really solved, and how well — don’t get discovered until much later. I had one project where the testers were happy to repeatedly re-open a bug involving a progress bar that incremented in stages — because it was waiting on indeterminately long responses from a web service — rather than moving at a consistent rate. In the meantime, they missed catastrophic bugs like a metadata update that, if sent to clients, would delete all their files in our application.

It sounds like I’m ripping on QA, and there’s often an adversarial relationship between testers and developers. But that’s not necessarily the way things should be. Good QA is one of the best things a developer can hope for. People who understand the purpose of your code, and can communicate where the code does and doesn’t achieve its goals, are actually pretty rare. In one of the best QA relationships I’ve ever been in, we actually had QA enforcing the development process and doing the build engineering (make files, source control, etc.). This is actually a huge relief, because it lets the engineers concentrate on code.

One thing I’ve learned from startups is that they tend to hire salespeople too early, and QA too late. It’s easy to understand why you’d do that — you want to get revenue as early as possible, whereas you wouldn’t seem to need QA until the product is almost done. But it turns out this is backwards — you don’t have anything to sell until later (and for some kinds of startups, salespeople can’t get their foot in the door and the management team does all the selling anyways). And a well-informed QA staff, working from the beginning alongside the programmers, gives you a better chance of having something worth shipping.

Of course, like I said in the beginning, a lot of us in iOS land don’t even have a QA staff. Still, some of the same rules apply: your clients are likely your de facto testers, and provided they understand what a work-in-progress is like, they can get you quality feedback early.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.