Comparing methods of testing – subjective methods
There is a variety of approaches to carrying out sound quality testing. These can be divided into two main categories: Subjective testing and objective testing using metrics.
Subjective Methods
There are many ways to carry out subjective tests ranging from informal designer listening, to formal sound quality testing. Listeners can be selected to perform a variety of functions and can be prepared for the test in different ways, from naïve listeners through to listeners who are trained to be impartial experts. All tests have their own set of problems or limitations and these effects should be kept in mind and minimized where possible. Informal designer listening tests, for example, suffer from the fact that there are discrepancies between the preferences of designers and customers [1]. The lack of a ‘standard’ environment for listening to take place may also affect reproducibility or the ability of the listening test to predict user satisfaction in the home. On the other hand this type of evaluation is quick, easy and inexpensive to perform.
Subjects are usually selected to take part in a test by undergoing an initial screening procedure. This may consist of completion of a simple questionnaire, a practice run of the test procedure, or even full hearing threshold measurement. The depth of the screening procedure depends on the complexity of the test and the depth of the results required. Some work on sound quality also suggests that listeners could be classified into two groups, those who rate the acceptability of a sound primarily according to loudness and others who rate the sound according to other factors [2]. Prior to the commencement of a test we also need to decide if naïve listeners or trained experts are required. A rule of thumb is that the less trained the listener, the more simplified the test should be. So naïve listeners are mainly used in simplified tests such as paired comparison tests, and trained listeners are required for more complicated test circumstances. If trained listeners are required, an effective training procedure needs to be developed.
The experimental method could incorporate one or more of a wide range of established techniques. An initial procedure may require the creation of adjectives for sound sources (e.g. soft, powerful, and harsh) to establish a lexicon. Alternatively, or after this procedure the subject may be asked to compare different sounds with each other, or against a reference. Comparison of sound pairs requires short bursts of sound, of the order of a few seconds, which places limitations on the test. The results of a test may be marked on a graduated scale, an unmarked scale or may be bipolar (e.g. present/absent, yes/no, better/worse). Details for designing test procedures have arisen from several fields which incorporate sensory testing techniques, two main approaches are from the fields of psychoacoustics (which seeks to provide an understanding of the human hearing system) and sensory evaluation (which has its origin in the food and beverage industries).
Psychoacoustical Evaluation Methods (PEM) [3] frequently involve the use of one or more of the following techniques to provide quantitative measures of perceptions: The subject may be presented with control over the stimulus presented, for example to adjust it so it sounds like a reference tone, or to adjust the direction in which the tone varies (this is known as Békésy tracking if excursions and levels are plotted as a function of frequency). Alternatively the subject may be asked to make a judgment on a signal, assigning numbers according to the magnitude of a given perceptual dimension e.g. loudness, or simply by stating the presence or absence of a signal in a yes/no procedure. In another method a comparison of stimulus pairs could be made allowing the experimenter to test the effects of variations in many perceptual dimensions. Sensory Evaluation Methods (SEM) are widely used in the food and beverage industries [4] but can easily be adapted to give results for audio stimuli. There are two parts to this test; an initial ‘sensory measurement’ phase which is carried out by a panel of trained subjects, and a secondary ‘preference studies’ phase which is carried out by untrained listeners, preferably drawn from the consumers. There are ISO standards for training subjects and conducting the tests which could form a basis for developing acoustical test techniques [5, 6, 7]. Finally other approaches to testing may include measuring physiological responses e.g. finger pulse amplitude [8].
The environment in which the tests take place and the test procedure can have an effect on the final results, so it is important to define these adequately prior to testing. The test procedure and surroundings should be the same for each subject and should be carefully documented in case it is necessary to review the procedure. The sound of an object depends on both the source, and the surroundings. For example, in the simplest case, if the original noise source is presented in the test room the benefit gained from listening to the original source may be outweighed by the acoustics of the test surroundings if they are very different from the usual circumstances under which the source is listened to. There also may be the problem of impractical logistics if the source is large and/or requires special fittings as in the case of a washing machine.
When recording a sound there are many options to be considered each with their own benefits and problems. The source could be recorded and then presented monaurally to a subject (by headphones) which is a relatively cheap and simple option. In this case however no directional information from the source is retained which may alter subject’s perception of the sound. To retain directional information the noise source could be recorded using a stereo microphone, a head and torso simulator, or even a multiple microphone array, to list increasingly expensive options. The position of the microphone during recording is also important, for example should it reflect the position of a user or a bystander? The sound recording could be presented by loudspeaker, which would require a specialized listening room or by headphones which increases the options for the listener’s surroundings, but the frequency response of the headphones may need to be considered. The surroundings in which the recording takes place also need to be carefully planned. An expensive option may be to use an anechoic chamber for recording which has the benefit of reproducibility, though the sound itself may be ‘unnatural’ when compared with the source placed in a typical user environment, like a simulated kitchen for example.
As previously explained, the surroundings of the subject during a test are equally as important as the method of presentation of the source. There is interdependency between the options selected for recording and presenting the sound source and the surroundings of the subject during the test procedure. An absence of visual cues is also recommended in the test room, so the subject should not be able to see the source, though some test procedures recommend that an object representing the source could be provided to increase the realistic nature of the test. In the automobile industry a test rig to simulate sitting in the seat of a car combined with a sophisticated loudspeaker array may be used to simulate realistic driving conditions [9]. Other important factors to note are: The test should be kept simple and easy to follow, the sounds themselves should be faded in/out to reduce irritating clicks and the overall test length should not be too long (approx 20 minutes maximum) to reduce the likelihood of fatigue. The presentation of samples should be randomized from subject to subject, and the number of subjects should be selected to provide an adequate statistical sample.
There are many statistical techniques and tools which are useful in sound quality analysis. Techniques range from relatively straightforward and simple tests like the t-test, the chi squaredtest, or the ANOVA to more complex analysis e.g. principle component analysis (PCA). The use of these tests depends on the structure of the sound quality test, the hypothesis to be tested and the knowledge to be gained from carrying out the test. The design of any sound quality test should include careful planning of the statistical tests to be used. A simple and straight-forward statistical analysis would be preferable but is not always meaningful unless careful planning has taken place beforehand.
Guide to jury testing
References
[1] ‘Practical application of sound quality technique in pump development’ K. B.Ginn, H. Haslev, J.B. Nielsen, Internoise (2004)
[2] ‘Sensory Evaluation of Air-Conditioning Noise: Sound Design and Psychoacoustic Evaluation’, P. Susini et al. (Internoise 2001)
[3] ‘Psychoacoustics: Facts and Models’ Zwicker, Fastl, (Springer 1990)
[4] ‘Sensory Evaluation Techniques’, M. Meilgaard, G.V. Civille, B.T. Carr, Third Ed, CRC Press, 1999
[5] ISO 8586 ‘Sensory Analysis – General guidance for the selection, training and monitoring of assessors’ (1993)
[6] ISO 11035 ‘Sensory Analysis – Identification and selection of descriptors for establishing a sensory profile by a multidimensional approach’ (1994)
[7] ISO 11056 ‘Sensory Analysis – Methodology – Magnitude estimation method’ (1991)
[8] ‘The use of psychoacoustic parameters combined with A-weighted SPL in noise description’, K Genuit, Internoise (1999)
[9] ‘Recent Developments in Powertrain Noise Simulation’, AV Phillips, Noise and the Automobile, Selected Papers from Autotech ’93, Mechanical Engineering Publications Ltd (1993)