The effect of explanation on MMN presence in Mcgurk effect

all-graphs-for-thing
The red line indicates the averaged brain waves in the deviant condition (audio playing “BA” while mouth pronouncing “FA”). The blue line indicates the averaged brain waves in the standard conditions ( audio playing “BA” and mouth pronouncing “BA”).

        Our experiment focused on the auditory illusions known as the McGurk Effect. The McGurk effect shows that visual cues can affect what a person hears. This is shown when  mouth pronunciations in a video can affect what a person hears. Even when the audio only plays “ba, ba, ba”, if the mouth pronounces “fa”, the person will hear “fa, fa, fa”. In cognitive science we can use certain brain waves as markers for certain activities in the brain. We wanted to see if explaining the McGurk effect to participants would affect the brain wave component known as Mismatch Negativity (MMN). The MMN is found in the 100-400 millisecond range after stimuli is shown.  When it appears, this component shows that the brain registered the mouth was pronouncing a different word than the sound playing- in other words, it was not fooled by the illusion.

We averaged the brainwaves found in two specific electrodes- FZ and OZ. The FZ electrode is found in the frontal lobe of the brain, which is where we would expect the MMN to show. The OZ electrode is in the occipital (back) of the head, which is where we should not see an MMN. No MMN was found in the OZ electrode area, as expected. We found that individuals who were not informed about the illusion did not have an MMN wavelength (top left graph), while individuals that did get it explained did show the MMN (top right graph).  This is interesting because although all participants reported experiencing the illusion, those who were explained the illusion had, on an unconscious level, not been fooled by the illusion. Their brain noticed the difference.

Research Justification

Here is an excerpt from the IRB we submitted for review before beginning the experiment. The original document contains 20 pages of content. This section describes our justification for our experiment:

        The McGurk effect, discovered by Harry McGurk and John MacDonald, is an audio illusion where visual information is powerful enough to cause a change in the perception of audio information. In the original study done by McGurk, different lip movements caused the perception of auditory stimuli to change. For example, if a person watched a video of a person mouthing the syllable “va”, but they played the syllable “ba”, they would  hear “va” instead of “ba”. It was also recorded that sometimes the perceived sound was a fusion of both the auditory and the visual information given:  playing the lip movements made for the syllable “ga” while playing the sound “ba” lead to the perception of the sound “da”.  The McGurk effect is very curious because even after participants are informed about what the McGurk effect is, they still experience the same perceptual illusion.

        Originally, the McGurk effect was evaluated using behavioral questions asking participants what syllable they heard. Another way to test for the McGurk effect is to use the ERP marker known as a Mismatch Negativity, or MMN. The MMN is an ERP component associated with higher order processing in the auditory cortex. The MMN has been successfully elicited in individuals from all ages, from newborns, to infants, to adults, and even in the womb. It is a well studied indicator of the auditory detection of changes in acoustic stimuli; in other words, it shows when the brain is detecting unusual changes in reliable, unchanging patterns. This type of stimuli uses the principle of the oddball paradigm, where in a sequence of the same stimuli there is is one deviant stimulus.  For example, an MMN would be elicited to “va” in the following auditory stream: “ba, ba, ba, va, ba, ba,…” This is used to the benefit of studying the McGurk effect, since a deviant sound perceived to sound the same as the repeating syllable will not elicit the MMN even though the actual acoustic stimulus is different.

        In the study done by Saint-Amour et al. (2006), the McGurk effect was used in conjunction with the MMN in the method outlined above. Participants listened to audio stimuli while watching lip movements while hooked up to an EEG cap recording the ERP information associated with each trial. In addition to looking at the ERP information the researchers used source analysis to determine from where in the brain these ERP components were being generated in order to give insight into the specific cortical areas that were associated with the tasks. This analysis allows a look into the systems needed to simultaneously process the visual and auditory information not only to stimulate the McGurk illusion, but also similar tasks like listening to music or participating in a conversation.

        The goal of our study is to replicate the study done by Saint-Amour et al. using source analysis as well as their stimuli. We are interested in seeing if informing participants about the McGurk illusion will alter the strength of the MMN markers elicited during the experiment. We know that the effect is powerful enough to continue being perceived even after explanation, but we are curious as to how this knowledge might alter the MMN we record. Half of the participants will read an explanation on the McGurk effect (experimental group) while the other half will read a description of molecular gastronomy (control group). Both the MMN and the source analysis of each group will be compared to see if there is a significant difference between the two. We predict that individuals that read the McGurk illusion explanation will have a weaker MMN response than individuals who do not read the explanation of the illusion they are being presented with.

References

Kislyuk, D.S., Möttönen, R., and Sams, M. (2008). Visual processing affects the neural basis of auditory discrimination.  Journal of Cognitive Neuroscience, 20, 2175-2184

McGurk, H., and MacDonald, J.(1976).  Hearing lips and seeing voices. Nature, 264, 746-748.

Näätänen, R., and Kreegipuu, K.  (2011).  The Mismatch Negativity (MMN). In S. Luck & E. Kappenman (Eds.), The Oxford Handbook of Event-Related Potential Components.  Oxford University Press.

Rosenblum, L., Schmuckler, M.A., and Johnson, J.A. (1997). The McGurk Effect in infants. Perception and Psychophysics, 59, 347-357.

Saint-Amour, D., De Sanctis, P., Molholm, S., Ritter, W., and Foxe, J. (2006). Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia, 45, 587-597.

Skipper, J.I., van Wassenhove, V., Nusbaum, H.C., and Small, S.L. (2007). Hearing lips and seeing voices: How cortical areas supporting speech production mediate audiovisual speech perception. Cerebral Cortex, 17, 2387-2399.

Yu, X., Lui, T., and Gao, D. (2015). The mismatch negativity: An indicator of perception of regularities in music. Behavioral Neurology, 2015, 1-12.