Development and content validation of coordinate response measure (CRM) corpus in Kannada for informational masking measurement

Abstract

Background

Assessing informational masking (IM) involves various methods, broadly categorized as electrophysiological and behavioral approaches. In behavioral methods, researchers often use a range of stimulus types such as tone-in-noise masking, speech masking, music masking, gap detection, modulation masking, and spatial masking. The evaluation of speech perception in noisy conditions often utilizes the coordinate response measure (CRM), a publicly available test developed by Moore (AGARD conference proceedings 311, 1981). The objective of the study is to develop and content validate the Kannada coordinate response measure (CRM) sentences corpus for the information masking measurement.

Results

Stage 1: A total of thirty-eight call signs, eight colors, and eight numbers in the Kannada language were compiled and were given to 15 experts to select the appropriate stimuli. Those stimuli that satisfied the criteria of CVI ≥ 0.90 and CVR ≥ .90 were selected, which consisted of eight call signs, seven numbers, and five colors. Stage 2: By incorporating the chosen call signs, numbers, and colors, seven sentence structures were formulated and were subjected to content validation (criteria of CVI ≥ 0.90 and CVR ≥ .90). The sentence structure “(Call sign) iga (Color) bannada (Number) torsu” was finalized.

Stage 3: Eight speakers (four males and four females) recorded Kannada sentences of about 3.5 s each, which were evaluated by five audiologists for voice quality, rate, and speech clarity using a 4-point scale. The evaluation identified two male and two female speakers with the best recordings.

Stage 4: The final stage involved recording 235 sentences by the selected speakers, which were then assessed by two Kannada-speaking audiologists for face validity on various parameters, ensuring linguistic appropriateness and cultural relevance for inclusion in the final CRM corpus.

Conclusion

The resulting corpus, meticulously curated and content-validated, presents significant potential for advancing research and clinical assessment of informational masking within the Kannada-speaking community. Overall, the CRM’s exceptional performance in challenging listening environments, inherent adaptability across languages, versatility for different age and literacy levels, and effectiveness in speech-intelligibility testing with multiple simultaneous talkers due to its use of call signs underscores its value as a superior measurement tool, which can be used in IM measurement in Kannada-speaking population.

Background

Speech perception involves a fundamental recognition process, typically serving as the essential foundation for recognizing words and non-words, such as phonology and morphology, as well as more advanced cognitive functions like memory recall and understanding context, as described by Pisoni et al. [1]. The ease with which a listener can distinguish between competing sounds is partly influenced by the perceptual similarities between the target sound and the competing sounds. When the competing sounds are more dissimilar, it becomes simpler for the listener to differentiate between them [2].

When the characteristics of sound sources cannot be properly discerned, it can hinder the formation of auditory objects, a phenomenon known as “masking.” Various forms of masking exist, including simultaneous masking, frequency masking, spectral masking, and temporal masking (including forward and backward masking). Another approach to categorize masking is by distinguishing between energetic and informational masking. Energetic masking (EM) occurs when a competing sound shares the same frequency and temporal characteristics as the signal of interest. This type of masking is primarily a result of the presence of relatively constant or steady-state noise [3], which is generated when excitation patterns overlap on the basilar membrane. Consequently, these competing excitation patterns contend for the same groups of auditory nerves.

In contrast, auditory perception can be hindered by sounds with relatively sparse distribution in terms of their spectro-temporal characteristics. As pointed out by [4], even the presence of just two to three simultaneous talkers in the background can make it difficult for listeners to distinguish the target signal from the background noise, even if the signal-to-noise ratio (SNR) is not particularly challenging. This form of masking, which is influenced by central factors such as attention and linguistic elements, is referred to as “perceptual masking” [5] or “informational masking” (IM) [4].

IM, or informational masking, has been assessed using different methods, which can be classified into electrophysiological and behavioral approaches. In electrophysiological methods, various cortical auditory evoked potential (CAEP) such as acoustic change complex (ACC), P1-N1-P2, P300, and mismatch negativity (MMN) have been used to measure the effect of IM [6,7,8,9,10,11,12]. Different parameters such as latency, amplitude, and morphology are used to measure informational masking. For example, studies utilizing cortical auditory evoked potentials (CAEPs) have shown that in the presence of interrupted noise, there are shorter latencies and larger amplitudes compared to continuous noise at equivalent signal-to-noise ratios [13, 14]. Additionally, experiments focusing on the effect of masking on cortical auditory evoked potentials with speech stimuli have revealed that modulated noise leads to lower masking effects and improved amplitude measurements of cortical components, indicating a release from masking [15]. Furthermore, event-related potential studies have highlighted that spatial separation between competing speech streams results in increased N1 and P2 amplitudes, indicative of cortical auditory processing benefiting from the release of informational masking [16]. Behavioral methods for assessing informational masking (IM) include evaluating speech perception in noisy conditions or measuring speech perception scores under various environmental circumstances. One of the categorical methods used to study speech perception in complex settings is the speech-in-noise test [17]. One approach to assess speech perception is to focus on how people perceive speech features rather than individual words or phonemes. Speech characteristics like voicing, place of articulation, and manner can be examined on acoustic continua representing voice-onset time, second formant change, and onset duration, respectively. Consonant speech sounds, differentiated by their place of articulation, exhibit variations along a continuum of change in the second formant trajectory. This means that sounds like /ba/, /da/, and /ga/ are perceived as distinct once they cross a definite boundary. Categorical perception boundaries make these distinctions more pronounced [18].

A publicly available experiment for studying auditory perception is the coordinate response measure (CRM), developed by Moore TJ, [19]. The CRM corpus consists of standardized sentences in a specific format, such as “ready (call sign) go to (color, number) now.” These sentences include eight different call signs like “Ringo,” “Laker,” and “Baron,” along with four colors (blue, red, green, and white) and eight numbers ranging from 1 to 8. The call signs are used to identify the target sentences that the listener should focus on, while combining a number and a color is the target element within the sentences. In the study conducted by Bolia et al. [20], a total of 256 sentences were recorded by four female and four male speakers. These sentences were structured to include eight call signs, four colors, and eight numbers, creating a comprehensive set of sentences. In the experiment, listeners or participants were presented with sentences and asked to identify the specific color and number within a sentence that was associated with a given call sign. For example, a typical sentence from the experiment would be “Ready Baron, go the blue five now,” where “Baron” serves as the call sign, and “blue five” is the color and number combination to be identified. Competitive random masker sentences were used, containing various call signs, colors, and numbers different from those in the target sentence to add a masking effect. For instance, a masking sentence might sound like “Hear Ringo, go the red six now.”

The CRM corpus is considered context-free, meaning it does not provide any cues or context that would allow the listener to predict the sentence color or number beforehand. This lack of context is crucial when calculating informational masking (IM) [4]. Apart from the American CRM version by [20], two additional versions were created in a British accent. The first British version utilized sentences from the American version. In contrast, the second version, developed by Semeraro et al. [21], was specifically designed as a speech recognition test for assessing the speech perception of the UK Armed Forces. Furthermore, the CRM corpus was expanded to include other languages. Behrens et al. [22] developed a Danish-language version, while Amiri et al. [23] introduced a Persian-language version.

Hence, the CRM corpus presents a remarkable opportunity for distinguishing between Energetic Masking (EM) and Informational Masking (IM), as noted by Amiri et al. [23]. Its context-free nature and substantial semantic and syntactic similarity, when employed as both target and competing signals, make it an essential tool for studying IM. Additionally, the CRM corpus has a limited vocabulary, primarily consisting of colors and numbers, making it suitable for younger populations. This underscores the utility of the CRM in measuring speech perception in the presence of informational masking. While existing CRM corpora cater to several languages, there remains a resource gap for Kannada (which belongs to the Dravidian language family, spoken in Karnataka state, India), a language with unique phonological characteristics. The present study aimed to develop CRM corpus in the Kannada language using the Delphi technique.

Methods

The development of the Kannada CRM corpus unfolded through a systematic four-stage process guided by the Delphi technique, facilitating consensus among experts. These are the following stages: Stage 1, Selection and content validation of Kannada words for CRM corpus; Stage 2, Development and content validation of sentence structure for CRM corpus; Stage 3, Audio recording and content validation of developed corpus; and Stage 4, Compilation of Kannada CRM corpus. CVI ≥ 0.90 and CVR ≥ .90 were considered to achieve consensus at each stage. The details of each stage have been discussed below:

Stage 1: selection and content validation of Kannada words for CRM corpus

This stage involved selecting Kannada words encompassing call signs, numbers, and colors. For this first, bisyllabic words (color, number) were collected from 3 adult native Kannada speakers, whereas monosyllabic and bisyllabic call signs were collected from ten students of class 3 to 6 having a mean age range of 10.33 ± 1.5 years. While collecting the call sign, the instruction for the children was to write down the cartoon character names they are aware of. The reason behind considering the children for choosing the call sign was that it was made sure that children are aware of call signs and will help them concentrate more during the actual testing. The most commonly found monosyllabic or/and bisyllabic words for call sign and bisyllabic words for color and number in Kannada were finalized for content validation. These selections were subjected to content validation by 15 Kannada-speaking audiologists, ensuring linguistic appropriateness.

Stage 2: formulation and content validation of sentence structure for CRM corpus

Based on the content-validated Kannada words from Stage 1, seven sentence structures were formulated, incorporating the chosen call signs, numbers, and colors, which gives a similar sentence structure of the original English CRM. A panel of 5 expert audiologists evaluated these sentence structures for content validation, assessing their relevance and accuracy in the Kannada language as original English CRM.

Stage 3: audio recording and content validation of developed corpus

The sentence structure that emerged as most accurate was selected for audio recording. Initially, few sentences were crafted and subsequently recorded by 4 male and 4 female speakers, utilizing Adobe Audition at a 44,000 Hz sampling rate in a controlled audiometric environment. The sentences were recorded using a Behringer C-1 Studio XLR Condenser Unidirectional Microphone placed 5 cm away from the speaker’s mouth. The microphone was connected to the Behringer U-PHORIA UM2 Audio interface for recording purposes. The sentences had an average duration of 3.5 s; silent gaps were meticulously removed without disturbing the target linguistic features. The recorded sentences underwent evaluation by five experienced audiologists, who ranked the recorded sentences for voice quality, rate, and speech clarity.

Stage 4: recording and compilation of Kannada CRM corpus

The amalgamation of selections from 2 male and 2 female speakers resulted in creating the final CRM corpus, representing a diverse and authentic representation of the Kannada language. The final sentences were de-noised and normalized to get the best audio quality.

Results

The development of the CRM corpus followed a methodology similar to that used by Bolia et al. [20] and Amiri et al. [23]. The CRM phrases were structured in a rigid format, akin to the CRM English format presented by Bolia et al. [16], which consists of phrases like “Ready [call-sign] go to [color] [number] now.” In the English version of the CRM, these phrases were used to create eight different call signs, four colors, and eight numbers ranging from one to eight. For the Kannada version of CRM, the sentences were constructed in accordance with the grammatical rules of the Kannada language.

Stage 1: In the initial stage, a total of thirty-eight call signs, eight colors, and eight numbers in the Kannada language were compiled. These entire stimuli were given to the 15 experts using Google form and asked to rate the words on 4 point rating scale 1 being not appropriate, 2 stand for somewhat appropriate, 3 for appropriate, and 4 for totally appropriate. Those stimuli that satisfied the criteria of CVI ≥ 0.90 and CVR ≥ .90 were selected [24, 25]. These selected stimuli consisted of eight call signs, seven numbers, and five colors in the Kannada language are mentioned in Table 1. This selection process ensured that only the most linguistically appropriate and relevant bisyllabic words (color and number) were retained for further use in the study, guaranteeing the high quality and suitability of the stimuli for their intended purpose.

Table 1 Developed Kannada CRM corpus

Stage 2: In this particular stage, the focus was on developing sentence structures in the Kannada language. These structures incorporated the previously chosen elements, including call signs, numbers, and colors. To achieve this, seven different sentence structures were formulated. These structures likely varied in composition, word order, and overall linguistic complexity. After formulating these sentence structures, they were presented to a panel of 7 audiologists with expertise in the Kannada language using a rating scale of 1–5 (strongly disapprove, disapprove, neither approve nor disapprove, approve, strongly approve) in Google form. The sentence structure satisfied the criteria of CVI ≥ 0.90, and CVR ≥ .90 was selected. In the end, out of the seven initially formulated sentence structures, only one was chosen based on its strong performance in content validation (CVI ≥ 0.90 and CVR ≥ .90). This careful selection ensured that the project would proceed with a single sentence structure that met the highest linguistic and contextual standards, guaranteeing the quality and effectiveness of the linguistic stimuli to be used.

Finalized sentence structure is as follows: “(Call sign) iga (Color) Bannada (Number) torsu,” e.g., Tom īga biḷi baṇṇada ondu tōrsu.”

Stage 3: In this stage of the process, the focus was on finding the best voice for recording sentences in the Kannada language. Eight speakers, four males and four females, were chosen to carry out this task. Few recorded sentences were meticulously crafted and designed to be of duration lasting approximately 3.5 s each. After the recording, the next step involved evaluating the recorded sentences. A panel of five audiologists assessed the voice quality, rate, and speech clarity of the recorded samples using the 4-point rating scale [26]. These audiologists carefully listened to and analyzed the recordings produced by the eight speakers. Following the evaluation, it was determined that two male speakers and two female speakers had recordings that demonstrated the best voice quality and speech clarity.

Stage 4: In the final stage of the process, a total of 235 sentences were recorded by the four selected speakers, which included two male and two female speakers. After the recording of these sentences was completed, the next step involved conducting a face validity assessment. In this assessment, two Kannada-speaking audiologists were asked to rate sentences based on several parameters. They were asked to evaluate whether the sentences were easy to perceive, free from distortion, of adequate duration, displayed good naturalness, and were spoken at an appropriate rate. Ratings were given on a scale of 1 to 4, where 1 indicated strong disagreement, and 4 indicated strong agreement. The purpose of face validity in this context was to ensure that the recorded sentences met basic criteria for linguistic appropriateness and cultural relevance within the Kannada-speaking context. Once the recorded sentences passed the face validity assessment by the two Kannada-speaking audiologists, they were considered suitable for inclusion in the final CRM corpus.

Discussion

The current work used the Delphi technique to create a CRM corpus in Kannada. The investigation was conducted in four stages. Stage 1 involved choosing and validating the content of bisyllabic Kannada words for the CRM corpus; Stage 2 involved creating and validating the sentence structure for the CRM corpus; Stage 3 involved recording audio and validating the content of the developed corpus; and Stage 4 involved compiling the Kannada CRM corpus.

In Stage 1, thirty-eight Kannada words, comprising call signs, colors, and numbers, were compiled.

Selection criteria of CVI ≥ 0.90 and CVR ≥ 0.90 were applied, resulting in the choice of eight call signs, seven numbers, and five colors for the study. This selection process ensured the retention of linguistically appropriate stimuli, guaranteeing high quality for their intended purpose. Across various studies, maintained the same content word categories, but with certain differences. In American English, Bolia et al. [20] and Brungart [4] employed eight bisyllabic call signs, four color terms, and eight numbers. In British English, Semeraro et al. [21] utilized 18 bisyllabic call signs from the NATO phonetic alphabet, nine monosyllabic color terms, and nine monosyllabic numbers. The Persian study by Amiri et al. [23] utilized bisyllabic call signs (10), monosyllabic numbers (1–9), and color terms (4).

Stage 2 of the study aimed to develop sentence structures in Kannada, incorporating chosen elements like call signs, numbers, and colors. Seven diverse sentence structures were formulated, likely differing in composition and linguistic complexity. Following content validation criteria (CVI ≥ 0.90 and CVR ≥ 0.90), only one structure stood out, ensuring the project proceeded with the highest linguistic and contextual standards. This careful selection guaranteed the quality and effectiveness of the chosen linguistic stimuli for the study.

Across these studies, a common approach was employed to create sentence structures by combining content words such as call signs, colors, and numbers. However, the specific sentence structures varied to suit the linguistic requirements of each language. In Dantale, the structure followed the pattern of “name verb numeral adjective noun,” for example, “Michael had seven yellow boxes.” In American and British English, the structure typically involved sentences like “Ready ‘call sign,’ go to ‘color’ number ‘one’.” Similarly, in Persian, the structure paralleled that of English, maintaining the same order. This linguistic adaptation demonstrates how the same content words can be incorporated into different sentence structures to accommodate the linguistic characteristics of the respective languages.

The third section aimed to identify the optimal voices for recording Kannada sentences, selecting eight speakers evenly distributed between genders. Audiologists carefully assessed recordings from all eight speakers, ultimately determining that two males and two females exhibited the best voice quality and speech clarity. These selected speakers were deemed most suitable for the project based on the evaluation. In earlier research conducted in American English, a relatively extensive dataset was used, involving recordings from a total of eight talkers. This group of talkers comprised an equal distribution of both genders, with four men and four women. In the Persian study, the dataset was comparatively more limited, consisting of recordings from only four talkers, included two men and two women, while in the context of the Danish study, a specific focus was placed on female speakers, with a dataset comprising recordings from five female talkers.

Finally, a total of 235 sentences were recorded by two male and two female speakers. Subsequently, a face validity assessment was conducted by two Kannada-speaking audiologists, evaluating aspects such as sentence clarity, naturalness, duration, and cultural relevance. Sentences meeting these criteria were deemed suitable for inclusion in the final CRM corpus, ensuring linguistic appropriateness within the Kannada-speaking context. The total number of sentences in the CRM corpus varies slightly among different languages. The original American English sentences were recorded as 250 sentences. In Persian, there were 240 sentences recorded, and for Dantale, 150 sentences were included in the corpus. These variations in sentence count may reflect the variation total number of content words across different studies. The finalization of sentences in these studies involved different approaches. In the American and British English studies, a psychometric function was employed, and sentences with the same SNR-50 were selected. Sentences with poor SNR-50 scores were likely excluded from the final set. This approach ensures that the selected sentences are acoustically comparable and meet specific quality criteria in terms of noise levels. In the Persian study, the selection process involved evaluating content validity and considering error rates in noisy conditions. This approach likely aimed to ensure that the chosen sentences were linguistically and semantically appropriate for the study while also accounting for their performance in noisy environments. In the Danish study, the specific method for finalizing the sentences is not mentioned, indicating that the approach used to select sentences for inclusion in the corpus is unclear. It is possible that the Danish study employed a different or unreported methodology for sentence selection.

Researchers commonly separate energetic and informational masking factors in experiments by either varying the uncertainty of the target and/or masker or by controlling the spectro-temporal overlap. This is achieved by manipulating the variability or presentation of the stimulus to adjust uncertainty while keeping energetic masking constant and varying non-energetic factors like language to attribute performance changes to informational masking influences [27].

Studies investigating IM have used various speech materials, including consonant–vowel combinations, monosyllabic and disyllabic words, and sentences. Among these, coordinate response measure (CRM) sentences are particularly useful for distinguishing between energetic masking (EM) and IM [4, 28]. CRM sentences are context-free, and when used as target and competing signals, they show high semantic and syntactic similarity, which is essential for introducing IM [28].

Although the limited vocabulary of the CRM prevents it from replacing comprehensive, phonetically balanced intelligibility measures like the modified rhyme test (MRT), it offers several advantages that justify its use in specific testing situations. One significant advantage, as demonstrated by Brugart [28], is the CRM’s sensitivity to small intelligibility changes in extremely challenging listening environments with Articulation Index (AI) values less than 0.25. This sensitivity makes the CRM particularly useful for testing intelligibility in very noisy conditions.

Additionally, the CRM’s intrinsic portability across different languages is a notable benefit. Since all languages have words for colors and numbers, the CRM can provide a rough functional measure of intelligibility without needing a phonetically balanced word list for each language. Furthermore, because the CRM uses only colors and numbers, it can be used to assess a wide age range of individuals, including children and illiterate adults who can identify colors and numbers, thus aiding in the development of IM.

Another advantage of the CRM is its applicability to speech-intelligibility testing with multiple simultaneous talkers. The call signs in the corpus allow experimenters to designate the target phrase without relying on differences in location, onset time, or talker characteristics to distinguish it from masking phrases. Listeners simply respond with the color and number coordinates mentioned in the phrase addressed to their assigned call signs.

The CRM-Kannada like other versions of CRM will offer relative ease in setting up and running speech-intelligibility experiments with its corpus. Since the listener is limited to the same 35 possible responses, each trial can be evaluated using the same response list, simplifying the collection and processing of intelligibility data compared to sentence-based or phonetically balanced tests like the MRT.

Lastly, given the critical role of informational masking (IM) in noisy environments, developing new tests to measure IM would be highly valuable for Kannada-speaking population. These tests could effectively evaluate the impact of rehabilitation programs aimed at improving speech perception before and after treatment. They can also provide clinicians with better insights into speech pathologies (stuttering, speech sound disorder) in different age groups. CRM-Kannada can be used to evaluate the functional development of Cochlear implants, after establishing its test–retest reliability as done for the English corpus by Saleh [29].

Conclusion

The utilization of the Delphi Technique in the development and content validation of the CRM corpus in Kannada for informational masking measurement exemplifies a robust and inclusive approach. The accessibility of an online and anonymous platform facilitated the involvement of diverse professionals, pooling their heterogeneous expertise for consensus-building. The outcome is a curated and content-validated corpus that holds immense promise for advancing research and clinical assessment pertaining to informational masking within the Kannada-speaking community. By fostering accurate measurement and understanding of informational masking phenomena, the corpus contributes to the formulation of effective interventions and strategies to address communication challenges in noise-laden environments. Overall, the Kannada CRM corpus’s superiority in challenging listening environments, intrinsic portability across different languages, versatility across various age and literacy levels, and suitability for speech-intelligibility testing with multiple simultaneous talkers due to its use of call signs make it an excellent and valuable test in measurement. This study underscores the significance of tailored resources in linguistically diverse contexts, serving as a valuable template for similar initiatives in other Indian languages.