by VILLY HANSEN and S. K. PRAMANIK
[Villy Hansen is Manager of Bang & Olufsen's Acoustic; Department, which started the listening panel discussed in this article and S. K. Pramanik is Manager of Product Technology. Both are based in Struer, Denmark.]
In our pursuit of higher fidelity, objective tests have so far been unable to replace subjective testing as the ultimate judge of re produced sound quality. As the purpose of recording and reproducing sound is for it to be heard by human ears, the need for subjective testing may seem so obvious as to be trivial.
What is less obvious is that the human ear is not always a true and consistent evaluator. This fact was brought home to researchers at Bang & Olufsen some years ago, in 1978 specifically, while we were doing a comprehensive set of listening tests that we were conducting in as scientifically rigorous a way as possible. The listeners in the tests were generally regarded as qualified, being the engineers and technicians who design our audio products. The purpose of the testing was to determine whether moving-coil phono cartridges are inherently better, as is sometimes claimed, than other types of cartridges. We used several samples of different types of cartridges.
One of the simple out comprehensive tests spanned three whole days. We played a short excerpt from a musical selection three times, randomly selecting one cartridge for one play and using another cartridge for the other two plays. The listeners were asked to identify the two excerpts that had been played with the same cartridge, and to give their preference regarding overall sound quality. The tests were repeated for all listeners with all possible cartridge pairs and with different types of music. All the cartridge pairs and music types were repeated as a double check.
While one can always draw a conclusion based on an average in such tests, results are only meaningful when listeners are able to negotiate the checks and controls that should be built into any properly designed test procedure. In the test just de scribed, for example, it would be pointless to draw any conclusions about overall quality if the user was unable to identify the excerpt played by the cartridge used only once. Similarly, there must be a clear display of consistency--from one test to the next and from one type of music to another.
Otherwise, it must be assumed that the responses depend more on chance, whim, or at best-some imagined difference in sound quality rather than an actual perceived difference. Statistically, the differences in responses are measured as standard deviation, which is a test of the "goodness" of the results (but a little too mathematically complicated to go into here).
Unhappily, in the tests using some of our engineers as judges, the deviation between individual tests was so large that the result was judged to be meaningless. After thinking about this for a while, we repeated the same tests but this time used professional listeners, a group of reviewers for British audio magazines. The results were not perfect but showed substantially better consistency, that is, a significantly smaller deviation between repeated tests. This was the trigger we needed to start a professional listening panel at Bang & Olufsen, and the group started functioning in 1982.
For practical reasons, such as finding time to listen during working hours, the panel was chosen from among the staff of Bang & Olufsen who responded to an advertisement in our company magazine. From that group, we selected from those who had an interest in listening to music or playing an instrument and who were willing to spend some time listening outside of work in addition to the time spent during work.
Finally, we chose so as to have a reasonable distribution of age and sex. Seven regular members and one re serve were selected for the panel. They then started a program of training to achieve the level of professionalism we required for our purposes. It was not our intention that the panel members become qualified to make comparative judgments between the different technical qualities of two or more products. Similarly, we did not intend that they make judgments regarding the value of products, that is, the perceived quality versus price. The primary aim was to give the panel the ability to make judgments and to point out reproduction errors, both on an absolute scale.
Absolute judgments are, of course, also relative judgments but where the reference is a sound in nature rather than some reproduced sound, however highly regarded. To be able to make absolute judgments, it is necessary to have a firmly established reference to the sounds of various kinds of music and speech as they are heard in real life. There fore, one of the requirements for being a member of our listening panel is attending live concerts regularly, and the cost of attendance is considered a company expense. The panel members are also urged to play an instrument, and Bang & Olufsen sponsors a brass band, though this is a free-time activity.
Selection of the material used for the tests is a primary factor in any procedure that requires absolute judgments, and this is no less true for the music for listening tests. To make judgments on the fidelity of recordings is not easy, but it is part of the responsibility of the panel members. Recorded music is chosen as a group decision, following debate and criticism of potential material. This is one of the facets of compulsory general training. Occurring once a week, training sessions are intended to increase the level of knowledge and awareness of detail in the chain of re produced sound. For example, one session might focus on a specific instrument in an ensemble, on accessing microphones, or on evaluating room acoustics.
Members of the listening panel are not asked to do A/B testing, that is, they do not attempt to decide which of two or more products is the best. On the contrary, each member attempts to point out specific faults or deficiencies in the sound of each product they audition. The goal or focus of the test is generally not known by the listener, be it a loudspeaker, a cassette recorder, or a pair of signal cables. This is immaterial to the test procedure. What is of importance is that the listeners are able to convey a precise impression of what they hear to the engineers in the research and development departments. Training in the ability to find these faults or deficiencies, and to convey them precisely to the engineers, is comprised of the listening and discussion sessions. We aim for the development of an ex act descriptive jargon for communication be tween listener and technician.
It is then the job of the engineers to translate these impressions to improvements in the product. Provided engineers know the exact cause, any error can be removed, though not al ways without giving rise to side effects. In addition, not all subjective impressions can be translated into engineering models, but successive trials can lead to a better under standing of cause, error, and remedy. Continued trials thus help to establish relationships between subjective impressions and objective measurements. If at any time this correlation becomes perfect, the listening panel will have made itself redundant.
One of the pitfalls of a permanent listening panel is that it can become a vehicle for a particular kind of sound. This situation could very well be one of the contributors to "regional" preferences or ideal models in sound quality. It can be partially avoided by constantly refreshing the members' memory by exposure to live sound. In addition, each panel member is provided with a full set of Bang & Olufsen equipment, which is updated with the latest products. Any equipment that is purchased for review is circulated to the members, to make them aware of strong or weak points of competing equipment.
Joint listening exercises and panel discussions by members also help. Listening tests are done blind or double blind, with only an individual listener in the listening room in any one session, so as to avoid results by consensus. In all cases, the operator and the person conducting the test are out side the listening room. Considerable care is taken to ensure that no inadvertent clues are available to the listeners about the purpose of the test.
Finally, the listeners in such carefully conducted tests need the support of an equally carefully thought-out listening room. Such a room was specially constructed for our listening tests, following the principles laid down in the I.E.0 's 1985 publication, No. 268-13, "Sound System Equipment, Part 13: Listening Tests on Loudspeakers." Built on a flexibly mounted concrete floor, the room is completely isolated from the rest of the building in order to avoid transmitted sound. A double wall is used. The outer wall is of plaster board, while the inner is of brick, and they are separated by 250 mm (9.8 inches) of insulating material. With the double doors closed, the room is soundproof. A glass window and a microphone in the room allow the listener and the operator to communicate with each other.
The room itself has a floor area of slightly more than 6 x 5 meters and a height of 2.85 meters (19.7 x 16.4 x 9.4 feet). Note that these dimensions were chosen so as to avoid identical resonances in any two dimensions. A carpeted floor, a wood-paneled ceiling, and living room furniture help to simulate the environment of a home listening room. Thin, acoustically transparent curtains can be drawn at about a third of the distance from the end wall.
When lit from the listener's side by small ceiling spots, the curtains are optically opaque. All source equipment is placed in the operating room, rather than inside the listening room, with signals transmitted at line level from a low-impedance source to sockets in the end wall. From here the signals are amplified as required and fed to the loudspeakers via short cables.
The reverberation time of the room is about 0.4 S at mid-frequencies, falling slightly at high frequencies. However, care has been taken to ensure that there are no peaks or abrupt changes in reverberation at any frequency.
Small absorbent panels are placed specifically to damp flutter echo, which is often a problem in rooms with parallel hard surfaces. Rigid walls are necessary to avoid selective absorption due to surface vibration at specific frequencies.
The combination of a well-defined, consistent listening environment and a permanent, well-trained listening panel allows the facility to be used as a true measuring instrument for product evaluation and development.
(adapted from Audio magazine, Aug. 1992)
Also see:
Perception and Geometry [by Richard C. Heyser; June 1977]
SRS: Surround Sound With Only Two Speakers (Aug. 1992)
= = = =