SIGNALS & NOISE (Letters to Editor) (Sept. 1990)

Congrats on Stats

Dear Editor:

I am writing in response to Herman Burstein's article, "Statistics in A/B Testing: By the Numbers," which appeared in the February 1990 issue.

I am a college student majoring in business and am currently enrolled in a business statistics course. This article opened my eyes to the actual usefulness of statistics, which I began to doubt after a month's worth of boring lectures.

At age 20, I am quite the stereo enthusiast, considering I have already upgraded my entire system. I found your article extremely helpful in showing me that numbers don't necessarily mean anything, unless you can be assured that they are reliable and valid. Thanks for the lesson.

William S. Stranberg; Milwaukee, Wisc.

On Trials

Dear Editor:

I enjoyed Herman Burstein's treatise on statistics immensely ("Statistics in A/B Testing: By the Numbers," February 1990). Thanks to you and to him for the article. His discussion of the differences between a scientifically valid and a statistically reliable experiment has been missing from nearly every other discussion on audible differences and test analysis. However, in discussing sample size, Mr. Burstein implies that the classic A/B/X test is limited to 16 trials and states the procedure was not well studied from an analysis standpoint. That observation does not actually describe the A/B/X method very well.

While 16 trials is suggested as the minimum needed to reduce the probability of Type 1 error to reasonable levels, in actual practice, the magic 16 has been a suggested "session length" and not a "sample size." I compiled a summary of every published double-blind amplifier test, including 16 which employed the A/B/X method (eight from the SMWTMS Newsletter, six in Audio, compliments of David Clark, and two from High Fidelity conducted by Dan Shanefield). Only one contained as few as 16 trials (Audio, April 1985). The SMWTMS tests ranged from 49 to 253 trials, with an average of 109. The Audio A/B/X tests ranged from 16 to 160, with an average of 77. The tests that Dan Shanefield ran for High Fidelity each had 40 trials.

One could come to the conclusion from reading Mr. Burstein's article that A/B/X tests are or have been limited to 16 trials. Certainly not so. Most contain multiple listeners or multiple sessions and have both excellent scientific validity and statistical reliability. As in any other experiment, sample size is at the discretion of the experimenter. There is no evidence to support the notion that A/B/X tests have had limited sample sizes and high probability of Type 2 error.

-Tom Nousaine; Chicago, Ill.

Author's Reply: It was not my intention to imply that A/B/X tests are or should be limited to 16 trials. It should have been made clearer that larger sample sizes than 16 are desirable and generally used. I had in mind that A/B/X tests are sometimes multiples of 16, such as the same subject tested 16 times on each of two occasions, or several subjects each tested 16 times. Also, it seemed best to deal with a small sample for purposes of illustration. On the other hand, in the section on sample size, I did point out the desirability of larger samples, such as 50 or 32, in examples I gave.-Herman Burstein Gifts with Ribbons

Dear Editor:

I purchased a pair of Bob Carver's Amazing Loudspeakers, a rather difficult achievement considering that top of-the-line audio equipment does not come cheap. After approximately six months of enjoying these excellent speakers, the ribbons began malfunctioning in the extreme high frequencies. I am sure the loudspeakers could have been repaired, but Bob Carver offered to replace them with brand new loudspeakers. Not only did he replace them, he upgraded them to Carver Platinum Edition Amazing Loudspeakers, which have many improvements and carry a much higher price.

"Proudly made in the U.S.A." really means something to Bob Carver and his company.

-R. Van Etten Topeka, Kans.

(Source: Audio magazine, Sept. 1990)

= = = =