Alpha, Beta, Gamma Testing at Apple, Toyota, and Boeing

Saturday the floodgates opened and 120,000 people pre-ordered their Apple iPads, which is possibly the worst-named product in recent memory. (One woman journalist immediately exclaimed, "OK, so no women were on the naming panel, I see."

Exactly what they're pre-ordering isn't very well known. The specs that are available were updated just last week. Industry watchers are pretty sure that iPad 1.2 will probably have a USB port, and that iPad 2.0 will be able to multitask (that is, run apps simultaneously). PC Magazine's headline read, "iPad PreOrders for Idiots Only".

This leads to a discussion of the software release cycle, which has been adapted into a product release cycle. The software release cycle evolved from the 1960's IBM product test cycle.

Alpha Testing is testing done within the company, by people other than the engineers, programmers, and designers who built the product. It usually involves white box techniques, but can include black box and even grey box techniques.

Beta Testing (following Alpha Testing) is user testing within a controlled situation. The product is not released to the market. A "beta version" is the first version released outside the organization or community that develops the product, for the purpose of evaluation or real-world black/grey-box testing.

As the Internet has allowed for rapid and inexpensive distribution of software, and as competitive pressure has decreased time-to-market, companies have begun to take a looser approach to use of the word "beta". Netscape Communications was infamous for releasing alpha versions to the public and calling them "beta" releases. Gmail and Google News have been in beta for years. This technique may also allow a developer to delay offering full support and/or responsibility for remaining issues.

With the ubiquity of the web, a lot of people know about alpha and beta products. There's more than just those two. Gamma testing is the third level of testing, generally for safety. Delta testing is the fourth round of testing, and Omega is the last round of testing. (This is the sequence of letters in the Greek alphabet). Unfortunately, Gamma testing is becoming a thing of the past, killed off by decreased time cycles, competitive pressure, and the myopic focus on quarterly profits.

Saturday Apple started taking advance orders for the iPad (wifi not 3G), and 120,000 were ordered, sight-unseen, in the first 24 hours. People are willing to pay a premium to be an early-adopter and what is essentially a beta-tester. This in spite of the fact that the people who bought the first iPhone would shortly see an improved, updated version being sold for less.

For $600 you get a WiFi tablet with no camera, no Java, no Flash, no stylus, limited multitasking, and an Apple logo. For $600 you can also get a fully functional netbook. To a degree, such is the cachet of Apple and the major alpha-geek status derived from being the first person at the coffee bar with an iPad.

What's interesting is that Apple is selling a beta-test product to early adopters who are eager to participate in the process. Apple's not the only outfit selling a beta-test product to the public.

In the NY Times, Robert Wright blogs about Toyota and the increased tendency to conduct the "subsequent, de facto beta testing that is also known as 'selling the product and then reading the user forums'." I think the original Big Blue product managers would shudder at using Granny as a Gamma Tester.

In my time, the most egregious public-as-gamma-tester episode was the Boeing 737 rudder charade. There was a clear problem with the product that was not discovered during alpha and beta testing. 204 people died in these accidents between '91 and '94: United 585 (25 lost), Copa 737 (47 lost), USAir 427 (132 lost).

When the end users discovered the problem, the response was to keep the fleet flying (too big to ground) while the rudder hydraulics were reworked and updated. During the time between the discovery and the resolution, every passenger on a 737 was participating in an undeclared test flight. The government's bet in not grounding the fleet worked (in that we didn't have another disaster before the rudders were reworked) but IMO it was a terrible, cynical decision.

If the people pre-ordering iPads are happy to pay to be beta-testers, that's all good. But if they think they're buying a ready-for-market product, they're suckers.