Abstract
Captioning is the process of transcribing speech and acoustical information into text to help deaf and hard of hearing people accessing to the auditory track of audiovisual media. In addition to the verbal transcription, it includes information such as sound effects, speaker identification, or music tagging. However, it just takes into account a limited spectrum of the whole acoustic information available in the soundtrack, and hence, an important amount of emotional information is lost when attending just to the normative compliant captions. In this article, it is shown, by means of behavioral and EEG measurements, how emotional information related to sounds and music used by the creator in the audiovisual work is perceived differently by normal hearing group and hearing disabled group when applying standard captioning. Audio and captions activate similar processing areas, respectively, in each group, although not with the same intensity. Moreover, captions require higher activation of voluntary attentional circuits, as well as language-related areas. Captions transcribing musical information increase attentional activity, instead of emotional processing.
References
Jul 1, 1986·Brain and Cognition·E K Silberman, H Weingartner
Aug 1, 1980·Electroencephalography and Clinical Neurophysiology·H ShibasakiA M Halliday
May 1, 1993·Journal of Neurophysiology·T A OrtizM J Aminoff
Sep 19, 1998·NeuroImage·J R PedersenA Gjedde
Nov 18, 2000·Journal of Personality and Social Psychology·M C Green, T C Brock
Dec 18, 2001·International Journal of Psychophysiology : Official Journal of the International Organization of Psychophysiology·C H Brunia, G J van Boxtel
Dec 18, 2001·International Journal of Psychophysiology : Official Journal of the International Organization of Psychophysiology·K B BöckerM N Verbaten
Aug 5, 2006·Science·Benedetto De MartinoRaymond J Dolan
Mar 30, 2007·Brain : a Journal of Neurology·Ingrid R OlsonYoussef Ezzyat
Jan 1, 2008·Biological Psychology·Jonas K OlofssonJohn Polich
Oct 3, 2009·Clinical Neurophysiology : Official Journal of the International Federation of Clinical Neurophysiology·Connie C DuncanCyma Van Petten
Feb 11, 2011·Brain and Cognition·Marta Bianchin, Alessandro Angrilli
Sep 6, 2011·Neuropsychologia·Laura M SkipperIngrid R Olson
Jun 20, 2012·Journal of Experimental Psychology. Human Perception and Performance·Frank A RussoDeborah I Fels
Jul 4, 2012·Frontiers in Human Neuroscience·Ulrike AltmannArthur M Jacobs
Nov 20, 2012·Cerebral Cortex·Carolyn Parkinson, Thalia Wheatley
Apr 26, 2013·Social Cognitive and Affective Neuroscience·Gal RazTalma Hendler
May 21, 2013·Social Cognitive and Affective Neuroscience·Sabine AustMalek Bajbouj
Aug 7, 2013·Journal of Cognitive Neuroscience·Thomas A CarlsonL Robert Slevc
Aug 22, 2013·Frontiers in Psychology·Sébastien PaquettePascal Belin
Dec 4, 2013·Social Cognitive and Affective Neuroscience·Corinna PehrsLars Kuchinke
May 9, 2015·PloS One·Kamil Konrad ImbirJarosław Żygierewicz
May 27, 2015·PloS One·Matjaž DebevcInes Kožuh
Nov 26, 2015·Cerebral Cortex·Corinna PehrsStefan Koelsch
Feb 9, 2016·PeerJ·Javier Sanchez-LopezThalia Fernandez
Mar 24, 2016·Cognitive, Affective & Behavioral Neuroscience·Eva M MorenoManuel Martín-Loeches
Mar 30, 2016·Neuropsychologia·Carlos Romero-RivasAlbert Costa
May 19, 2016·PloS One·Carl HopkinsJane Ginsborg