Citation: Ceciu, Ramona L. 2020. Neurocinema/tics, the (Brain)Child of Film and Neuroscience. Journal of Communication and Behavioural Sciences/JCBS, vol. 1 (2): 46-62.
Abstract
This paper examines the evolution of cinema since its initial paradigms to the present-day experiments where film and neuroscience have joined to give birth to a new form of art – the ‘scientific art’ of neurocinematics. It looks at the inter- and multidisciplinary exchange between psychology, cognitive sciences, film, neuroscience and neuromarketing, their instruments of research and their input in the creation of neurocinema. The main argument of this article is that neurocinema dismisses classical notions of realism, authorship, film language and signification, to become an instrument of probing and modelling the human mind, emotional response, and behavior according to specific objectives. In this context, the paper surveys studies highlighting the main cognitive processes involved in film viewing, the benefits and disadvantages of employing neuroimaging techniques in filmmaking, as well as the effects of neurocinematics on spectator’s perception, cognition and emotion triggered in the complex dialogical interaction between films and audiences.
Key words: neurocinema, film[ology], neuroscience, psychology, cognition, emotion.
Introduction
Film has been since its inception an experimental art form where multiple arts and sciences have joined to force and widen the limits of perspective and knowledge, to challenge so-called “established truths” and realities. This paper analyses such challenges, relevant theoretical approaches and practical applications of film theories since the early phases of cinema to the new experiments in the fields of film, psychology, neuroscience and neuromarketing which give birth to the new genre of neurocinema, while comparatively engaging with the specialized literature dealing with the subject. It surveys the panoply of studies on (neuro)cinema with the aim to put available data in context, to examine various cognitive processes involved in film viewing, the instruments and methods of research employed in neurocinema, the influence of neurocinematics on the viewers’ perspective, perception, cognition and emotion, as well as the advantages and drawbacks that such experiments have. This is important because neurocinema is a reality which will redefine the next decades in terms of filmmaking, inter- and multidisciplinary research in visual arts and not only, in terms of understanding the new paradigms, the movie–audience interaction, the “horizon of expectations” (Jauss, 1982), representation, as well as the deconstruction and rewriting of the filmic language, its grammar and meanings.
The first argument put forward is that neurocinema dismisses classical notions of authorship, realism, film language and signification, to become an instrument of probing and modelling the human emotional response, thinking and behavior according to specific objectives, predictions and expectations of producers, scientists and even spectators. Secondly, while paradoxically adding up new technical effects and devices, such neurocinematic modelling of movies, of the film-viewing experience and response has the potential to decrease the flexibility (and fluidity) of the film language and meaning, the hermeneutical exercise, as well as the creativity and curiosity of both filmmaker and spectator. This article distinguishes between the two terms, namely neurocinema – as a genre (and as a branch of cinema generating an industry of its own), and neurocinematics – which I view as the entire apparatus of neurocinema (and cinema), including devices, techniques, research and filming methods, the relationship between the neurofilm (as the product of neurocinema) and the audiences, the film–viewer interactions, power relations, issues of control, cultural dynamics, processes and connections which modulate the viewer’s reality, cognition, emotional reactions (and ‘answerability’, Ceciu, 2013) and behaviors; spelled as neurocinema/tics, it refers to both terms taken together, the genre and the apparatus with all other elements, network of relations etc. To clarify these aspects, it is important to start with the early years of cinema and film theory so that one can have a richer image of the forerunners of neurocinema.
Film/ Cinema from Classical to Cognitive Film Theories
The concept of neurocinematics was proposed by a research group in the Psychology Department of New York University to synthesize cognitive neuroscience and film studies, and refers to “the study of films through the use of functional magnetic resonance imaging during film watching under experimental conditions” (Poulaki, 2014, p. 1). The term was used by Uri Hasson (2008) with reference to the study of the brain reactions when watching a film. However, the concept has evolved to incorporate several other dimensions, although research is in progress for new findings. Most studies from the fields of psychology and neuroscience use the two terms neurocinematics and neurocinema interchangeably; but from a film studies perspective a terminological distinction may be useful due to their panoply of conceptual and epistemological connotations.
In his theory on cinema André Bazin (1967), one of the most influential film theorists of the early era, supported “realism as the most important function of cinema”, the need for less or no montage and more “objective reality”, the use of mise-en-scène (setting and everything it implies) to render the flow (which he called “true continuity”) rather than editing techniques and special effects; he emphasized the importance of the filmmaker’s vision which allows at the same time the spectator to derive his own interpretation and meanings from the film. Therefore, Bazin distinguished between “those directors who put their faith in the image and those who put their faith in reality” (Bazin, 1967, p. 24). While constituting a premise for the auteur theoryand the French New Wave, his perspective stresses in fact the distinction between different degrees of image manipulation through film conceptualization and editing techniques, which lie beneath the approaches to film as “real life documentation” versus film as “manipulated reality” rendered through intricate editing, continuity conventions and montage.
The montage experiments had started since the 1910-20 decade, an example being the Kuleshov effect–resulting from Lev Kuleshov’s formalist experiments where the juxtaposition of certain images determined the spectator to extract specific meanings which would be different from the viewing of the same images as single isolated pictures (for instance, a neutral facial expression juxtaposed with a bowl of soup would suggest hunger, while the same neutral expression juxtaposed with a baby girl in a coffin would suggest sorrow, grief). He also experimented with the creative geography or the artificial landscape which he created by editing and joining images of different locations and moments into a new sequence determining the viewers to perceive it as a continuous time frame and a single location (Kuleshov, 1974).
Kuleshov’s experiments have been researched and validated in recent times with neuroscientific instruments. Dean Mobbs, Weiskopf and their research team (Mobbs et al., 2006) prove that coupling identical faces with either neutral or emotionally intense contextual movies, the ‘Kuleshov Effect’, “results in both altered attributions of facial expression and mental-state”; using functional neuroimaging (fMRI), these researchers demonstrate that “faces paired with emotional movies enhance BOLD responses in the bilateral temporal pole, anterior cingulate cortices, amygdala and bilateral superior temporal sulcus relative to identical faces juxtaposed with neutral movies. An interaction was observed in the right amygdala when subtle happy and fear faces were juxtaposed with positive and negative movies, respectively. An interaction between happy faces and negative context was also observed in bilateral amygdala suggesting that the amygdala may act to prime or tag affective value to faces. A parametric modulation of BOLD signal by attribution ratings indicated a dissociation between ventrolateral and the ventromedial prefrontal cortex for negative and positive contextually evoked attributions, respectively. These prefrontal regions may act to guide appropriate choices across altering contexts. Together, these findings offer a neurobiological basis for contextual framing effects on social attributions” (Mobbs et al., 2006, p. 95).
Kuleshov’s student, the Latvian/Russian filmmaker, draughtsman and set designer Sergei Eisenstein experimented with montage of attractions and vertical montage to “synchronize the senses” (Eisenstein, 1957). Inspired by Meyerhold’s principles of bio-mechanic theatre (Meyerhold, 1969) and also by Asian cultures and linguistic principles (Chinese and Japanese ideograms), Eisenstein developed a syntax of cinematic language and methods of “dialectic montage” (covering metric, rhythmic, tonal, overtonal – that is, the mood arises from the synthesis of images during projection – and intellectual levels), “collision of shots” (see his films, Strike, 1924; Battleship Potemkin, 1925, with its memorable Odessa steps sequence), and juxtaposition (Kupść, 2003, p. 36). In Eisenstein’s view, “montage is essentially micro-dramaturgy, [it is] above all an active method of narrative” and “the exposition of an event through montage enables the viewer’s attention to be captured and led along the necessary sequence of vision” (Eisenstein, 1985, p. 10-11). Elsewhere, he would state that “at the moment of shooting the scenery, sets and props are often wiser than the director. One must be really gifted and greatly skilled to be able to hear and understand what the scenes suggest, to be able to listen as one edits the film to the whispering of the shots which, on the screen, live a life of their own, frequently extending beyond the limits of the imagination that has conceived them” (Eisenstein, 1959, p. 26). What he means by this is that the spectator would integrate the moving images into his/her own knowledge database and reality and decode from the visual text more messages and more varied meanings than the filmmaker himself had intended. Consequently, Eisenstein wanted to recreate reality and generate in spectators intense emotional responses, through images and visual counterpoints that could construct themselves in the human mind: “a work of art, understood dynamically, is just this process of arranging images in the feelings and mind of the spectator”; “in the actual method of creating images, a work of art must reproduce–that process whereby, in life itself, new images are built up in the human consciousness and feelings” (Eisenstein, 1957, p. 18, my emphasis). Eisenstein put forward that “the desired image is not fixed or ready-made but arises – is born. The image planned by author, director and actor is concretized by them in separate representational elements and is assembled-again and finally-in the spectator’s perception” (Eisenstein, 1957, p. 31).
There is no dearth of film theories for each paradigm, from the formalist, structuralist, to the poststructuralist, cognitivist, psychoanalytical, postcolonial film paradigms and countless others, but the limited space of this paper allows only for the survey of few approaches relevant to the present topic. While the psychoanalytic film theories focus more on the ‘unconscious of the film’ and the spectator (Christian Metz, 1982; Laura Mulvey, 1975; R. Allen, 1999), the cognitive theories of film focus on all psychological phenomena dealing with the human mind and embodiment (‘the thinking body’), from attention, to perception, mental representation, cognitive schemata, object recognition, to superior cognitive processes (thinking, memory, imagination), emotion and spectators’ interaction with the films, so on and so forth.
Representative for the cognitivist theories in film studies are David Bordwell (1989), Gregory Currie (1995), Joseph D. Anderson (1996), Raymond Gibbs (2006), Torben Grodal (2007), Noël Carroll (2008), and several others. One specific cognitive perspective is that the spectator is involved in the act of film viewing “intentionally” with one’s entire being: mind, body, thinking, feeling, behavior and spirit; the experiments on selective attention show that there is a degree of message processing residing beyond the center of attention, cognitive psychology distinguishing between ‘automatic processes’ (which do not require intention, effort or control) and ‘cognitive processes’ which demand conscious control (Rusu, 2008). On the other hand, for Jerome Bruner (1986) the cognitive activity goes beyond the given information, so the sensorial input is filtered and transformed, the blanks are filled-in, and then this input is compared with other inputs with the aim of “constructing a consistent and stable world” which is premised on learning processes (in Rusu, 2008, p. 161).
Thomas H. Peake notes that seeing in films “the self-defeating or redeeming things people do helps us discover out blind spots and find solutions”; movies “beat the idea of only learning about humanity through dry scholarly writing” and “can give us a new perspective and new appreciation of emotions and relations”, while stimulating the entire brain and creating connections with our memories through all our senses; narrative film is like a folktale and a myth, a “way of making sense in a senseless world” through creativity, relationships and growth which “involve a balance of form and passion” (Peake, 2004, p. 2-4). And indeed, as we know, the brain loves to tell stories and search for personal significance in the surrounding phenomena, as the Default network of the brain processes semantic structures, stories, autobiographical content. Among other functions it makes constant simulations and predictions about future, always pursuing one’s wellbeing, preferences, and attitudes, while the Salience network processes input relating to self-relevance, to disgust or criticism etc. (Cîrneci, 2011).
In David Bordwell’s view “stories are designed by human minds for human minds. Stories bear the traces of not only local conventions of sense making but also the constraints and biases of human perception and cognition”; thus, a film “must manage several channels of information (image, speech, noise, and music)” (Bordwell, 2007, p. 187). The information decoding is done in different degrees according to self-relevance and myriad other personal factors. According to Bordwell and N. Carroll (1996) the cognitive paradigm “seeks to understand human thought, emotion, and action by appeal to processes of mental representation, naturalistic processes, and (some sense of) rational agency” (Bordwell and Carroll, 1996, in Hasson et al., 2008, p. 21).
The narrative film (which constitutes the focus of neurocinema) is based on “image schemas” (term coined by Mark Johnson and George Lakoff in 1987) which include spatial primitives, image schemas and schematic integration: “spatial primitives are the first conceptual building blocks formed in infancy, image schemas are simple spatial stories built from them, and schematic integrations use the first two types to build concepts that include non-spatial elements, such as force and emotion”; at the same time, “preverbal conceptualization needs to be taken into account for a complete understanding of image schemas and their uses” (Mandler and Cánovas, 2014, p. 510). Such schemas intermingle and according to several individual factors they contribute to the interactions and set of expectations one has from a certain narrative.
Raymond Gibbs suggests that image schemas are “attractions within human self-organizing systems. Attractors such as Balance, Source-Path-Goal, Resistance, Verticality, and Path reflect emerging points of stability in a system as it engages in real-world interaction”; essentially, “attractors are not localized representations, but emerging patterns of entire systems in action (i.e., interplay of brain, body and world)” (Gibbs, 2006, p. 115). Research studies on schemas prove that people do not remember sequences that have no connection with the story, they try constantly to place disparate episodes into one piece, to rebuild the disconnected narrative so that it fits the canonic story; hence Mandler (1984) notes that the canonical story is a mental representation, a structure that is essential for the comprehension of narrative texts. Further, Roger Schank (1977) proposed the concepts of “mental scenario” and “conceptual dependency” while scripting models of artificial intelligence to understand stories and elaborating on how human knowledge is structured; Schank and Abelson (1977) argue that “a central attributional problem is to explain extended sequences of behavior. To do this people must relate actions in a sequence to one another and construct a coherent scenario from them. Because the relation among actions is not given, people must use detailed social and physical knowledge to make connecting inferences” (Schank and Abelson, 1977 in Read, 1987 in Read, 1987, p. 288). In several such studies the problem of meaning and how one derives meaning seems pertinent. In relation to artificial intelligence some consider that meaning can be derived by breaking down sentences into words and words into primitives, “when you need to exploit ‘meaning’ itself” (Schank and Abelson, 1977, p. 16). Alternatively, when one needs to unravel the meaning of a visual text, a diegetic reality, then attention to and comprehension of the film language and signification become important.
Torben Grodal argues that “although the basic emotional mechanisms are made to avoid negative experiences and approach pleasant ones, a series of adaptations modify such mechanisms. Goal-setting in narratives implies that a certain amount of negative experiences are gratifying challenges, and comic mechanisms make it possible to deal with negative social emotions such as shame. Innate adaptations make negative events fascinating because of the clear survival value, as when children are fascinated by stories about loss of parental attachment. Furthermore, it seems that the interest in tragic stories ending in death is an innate adaptation to reaffirm social attachment by the shared ritual of sadness, often linked to acceptance of group living” (Grodal, 2007, p. 21). Thus, the relevance and meaning of a narrative get structured in the viewers’ mind depending on what aspects are more important to them, according to their needs and characteristics. The cognitive identification determines empathic connections with the story characters (be they heroes or villains) and is equally determined by motivation and existential interests, the viewer simulating the character’s emotions.
‘Through the Looking-Glass’: Metamorphosis of Filmology into Neurocinema
Adriano D’Aloia and Ruggero Eugeni (2014) examine the classical filmology and its changes up to the contemporary neurocinema. A manifesto of filmology appeared in 1946 (Essai sur les principes d’une philosophie du cinema by Gilbert Cohen-Séat), stating that “systematic study of cinema and a comprehensive analysis of the ‘cinematic fact’ and ‘filmic fact’ were essential to found an autonomous and specific discipline, accountable for the complexity of the film ‘enterprise’ as both a social and a psychological object” (in D’Aloia and Ruggero, 2014, p.11). Those studies aimed to prove and “measure viewers’ psychophysiological response to ‘experimental’ films – made specifically for these studies – or short sound films, by means of electroencephalograms (EEG)”; although filmology established exchanges between empirical sciences and humanities, there were some deficiencies concerning the epistemological foundation and methodology which led to “a theoretical and practical impasse” (D’Aloia and Ruggero, 2014, p. 12-13). The main critical remarks regarding these initial studies were directed to the fact that EEG was not enough to offer valid results about spectators’ responses to films, more instruments and methods were necessary, and that it was difficult to reproduce in the laboratory stimuli equivalent to those normally specific to real life (Mario Roques, 1954, p.3-5).
Vittorio Gallese and Michele Guerra (2014), mixed behavioral and high-density EEG experiments and demonstrated that the steadicam produces stronger viewers’ brain activation compared to other camera movements (e.g., dolly or zoom-in), which means that the camera has the capacity to simulate the virtual presence of the viewers inside the movie (Gallese and Guerra, 2014, p. 103). But I would say that many film auteurs were aware of such capacity of the movie camera and intuitively, through steadicam shots, they intended to bring the film viewers inside the diegesis and share the directorial look.
It is true that “art encompasses all representations of thought, using all forms of symbolic languages and mediums manifested by the human brain” (Siler, 2016, p. 184). Researchers Saurabh Prasad, Badrinath Roysam and Jose Contreras-Vidal at the University of Houston’s Integrative Strategies for Understanding Neural and Cognitive Systems have designed and built a mobile headset qEEG device that can “map the creative process and functional connectivity of freely behaving people experiencing art-related activities in free range environments” (while viewing art in a museum, painting etc.); it documents the interconnectivity of the motor cortex, sensory cortex, the somatosensory cortex processes and the sensory association area, temporal lobe, occipital lobe and several other areas of the brain (Prasad, Roysam and Contreras-Vidal, 2015, in Siler, 2016, p. 190-191). All these brain regions fulfill several functions in a person’s homeostasis and interactions with the world and its objects, including visual arts.
Neurocinema is premised on the multidisciplinary interaction and methodological input from neuroscience, neuroimaging, cognitive science, and film studies (which also relates to media, cultural studies, and social sciences). Several neuroscientists argue that “mirror neurons underlie the social behaviors of imitation and empathy and have found support for this view of theories of simulation and embodied cognition” (Lanzoni, 2016). It has been showed that there is an interdependence between the appetitive information processing, consecutive emotional response, and appetitive conditioning; a key component of this circuit is the mesolimbic dopaminergic system, the pathway from ventral tegmental area (VTA) to nucleus accumbens, which acts as a “rheostat” of reward, that is, the more rewarding an activity would be, the better the brain will recall it and repeat it (Morrison and Salzman, 2009). The subjective processing and emotions associated with reward and pleasure take place over large areas of the brain; apart from the amygdala, appetitive conditioning also involves structures such as the hypothalamus, corpus striatum, ventral pallidum, insular cortex, anterior and posterior cingulate cortex, ventromedial and ventrolateral prefrontal area (Morrison and Salzman, 2009).
Researchers also note that “anticipatory activation in reward-related regions largely reflects the motivational relevance of an upcoming event”; consequently, “for both self-directed and charity-directed trials, activation in the NAcc (nucleus accumbens) and VTA increased to anticipated gains, as predicted by prior work, but also increased to anticipated losses” (McKell et al., 2009, p. 1). This can be equally valid when it comes to pleasure and loss of pleasure (engagement or interest) in watching a film till the end, depending on our anticipation of what we can get out of it. Through neuroimaging, researchers can identify such brain structures, the neural networks of the brain which get activated during film viewing, when a viewer watches a product (or advertisement, in case of neuromarketing), as well as the specific context (i.e., the association of images) when such activation and reactions occur.
Neuromarketing (concept coined by Ale Smidts in 2002) is a new area of research focused on studying consumers’ responses to marketing stimuli, their behavior concerning various ad services and products, through the identification of brain mechanisms involved in distinct responses and reactions. MindSign Neuromarketing along with the film producer Peter Katz have developed a project based on the application of neuroscience techniques to the feature film production. The team calls this new hybrid of neuromarketing applied to films, neurocinema (Curtis, 2009). The techniques used in neuromarketing and neurocinema include eye-tracking, fMRI/ functional MRI, EEG/ Electroencephalography, TMS, Magnetoencephalography (MEG), Positron Emission Tomography (PET), single-photon emission computed tomography (SPECT), visual evoked potentials (VEPs) and brain-stem auditory evoked potentials (BAEPs) allow neuroscientists to observe the brain mechanisms and dynamics. Various studies use Eye-tracking, fMRI and EEG to show the brain processes which are stimulated by attention, eye movement, emotion and memory, being efficient in predicting consumers’ preferences and the success of an advertisement campaign even before the product is released on the market.
In his neurocinema experiment, Peter Katz uses fMRI to scan the brain of a young woman (Bridget) while she watches sequences of a horror film, Pop Skull (2007). Every sequence is screened multiple times, with a 20 second break between screenings to allow the subject to reach a neutral state so that the differences between the activated states of the brain would be better scanned during film watching. The computer displays the images that the fMRI captures, especially the amygdala which becomes active in fight or flight situations, in fear and fury, but other brain structures as well. With the help of neurocinema, the producer can identify the cinematic techniques which can generate in the spectator specific responses and feelings and the visual contexts of such responses (Katz, 2009). Some critical views related to neurocinema and neuromarketing would be that they involve conflicting standards, and that Hollywood would get to ‘use’ spectators’ minds to win awards (Randall, 2011).
Uri Hasson and colleagues introduced the paradigm of inter-subject correlation of brain activity, and applied functional magnetic resonance imaging (fMRI) during free viewing of movies, and used inter-subject correlation analysis (ISC) to evaluate similarities in the spatiotemporal responses across viewers’ brains during film watching. They observed that “the level of control over viewers’ brain activity differed as a function of movie content, editing, and directing style” and that “ISC may be useful to film studies by providing a quantitative neuroscientific assessment of the impact of different styles of filmmaking on viewers’ brains, and a valuable method for the film industry to better assess its products” (Hasson et al., 2008, p. 1). In different experimental phases, the subjects were required to watch the opening 30 minutes of a Sergio Leone film, The Good, The Bad and the Ugly (1966), a 10-minute, unedited, one-shot video clip in documenting a real-life scene, a Sunday morning concert in Washington Square Park (NY), a Hitchcock episode (Bang! You-re Dead, 1961) and Larry David’s Curb Your Enthusiasm (2000),so that the researchers could compare the brain activation for different screenings. Their study suggests that “a mere mechanical reproduction of reality, with no directorial intention or intervention, is not sufficient by itself for controlling viewer’s brain activity” and “achieving a tight control over viewers’ brains during a movie requires, in most cases, intentional construction of the film’s sequence through aesthetic means” (Hasson et al., 2008, p. 8-9). The results showed that Hitchcock’s film produced the highest and most extensive activation of the cortex (65%), followed by Leone’s film (45%), then Larry David’s film (18%), and the least brain activity was noticed while the subjects watched the short documentary of the Washington Park (5%). The empirical data provided by such studies come to support the classical distinction between those films that remain as faithful as possible to reality – following Bazin’s tenets, the “Bazinian” filmmakers who allow for multiple interpretations of images – and those filmmakers who seek to control or distort reality through continuity editing and other cinematic techniques, like the Hollywood directors (Hasson et al., 2008, p. 16). The researchers note that “part of the mesmerizing power of movies stems from their ability to take control of viewers’ minds, and that viewers often seek and enjoy such control because it allows them to become deeply absorbed (and mentally engaged) in the movie” (Hasson et al., 2008, p. 17). This aspect of control, and other problematic themes, have been widely criticized in relation to cinema and media studies.
In a critical stance Poulaki notes that Hasson and his team’s research has some drawbacks: they “criticize the notion of control that themselves prioritize as most important in film effectiveness and the way they define it. Even though they acknowledge that film effectiveness can be ideologically problematic, they still establish it on the same grounds of controlled aesthetics” (Poulaki, 2014, p. 7). However, Hasson and colleagues draw attention to the fact that on the one hand, “maximal control over viewers’ minds might simplify and trivialize the art work”; on the other hand, “taken to an extreme, the possibility to achieve a tight grip on the viewers’ minds can be used for creating an unethical form of propaganda or brainwash” (Hasson et al., 2008, p. 18). Of course, one can realize that the level of aesthetic control plays a role (among other factors involved) in such distinction between spectators’ reactions and the level of control films exert on audiences. Through conscious and informed engagement with films a viewer can be aware of such control and decide how to respond to it, while remembering that they see images filtered through the camera-eye.
There are lots of theories surrounding the types of ‘look’, gaze, and the ‘visual pleasure’ practiced in cinema (see Laura Mulvey, 1975, Peter Wollen, 2007, and others). For instance, Dziga Vertov (see his experimental film, Man with A Movie Camera) considered that looking was both mental (cognitive) and optical activity, while for Lev Kuleshov looking was purely subjective. The relations with the narrative and screen characters take place at various levels; “to the spectator the characters on the screen are real as long as the film lasts, but for them the spectator has no reality at all” (Vertov, 1928, in Wollen, 2007, pp. 103-106). In any case, for any type of film camera represents an intermediary agency, a mediation between what the director/ filmmaker sees and what the spectator sees; both sides see and relate to different realities even if the image is the same. The spectators engage in variable narratives, as they may create in their mind (through intertextual readings and association) multiple stories in one moving image.
The study realized by researchers from Tallinn University, presented by Piia Tikka (2018), included the film Memento (2000), whose sequences were screened for the subjects chronologically and in reverse order. The research identified certain fingerprint patterns in the brain for narrative reconstruction and comprehension which were activated when the repeated key images appeared the second time on screen. The study showed that the neural activation took place in similar brain regions simultaneously in all subjects who watched the same sequences with high emotional impact, suggesting that in future the technology can be adapted so that the films would become interactive and respond to the spectators’ reactions and expectations during film watching (similar activation appears in the neuro-study of “Avatar” at BrainMovie, 2009).
Elsewhere, Pia Tikka and Mauri Kaipainen (2014) start from Husserl’s concepts of “retention” and “protention” and Francisco Varela’s neurophenomenological exploration of time consciousness, and contend that “film narratives are intrinsically time-dependent designs”; they propose a “model of nowness relating this to the neural epiphenomena of narrative experience”, along with the concept of “enactive cinema” (studied by the interdisciplinary research group aivoAALTO from Finland)–“a model that assumes changes in the psychophysiological reactions of participants (enactors) to represent implicit and unconscious reactions of the mind and determine the changes made to the narrative presentation in real-time” (in D’Aloia and Ruggero, 2014, p. 25). But beyond time, film narratives are also reliant on other devices and methods such as camera movements, settings, syntagmatic and paradigmatic arrangements of shots, spectators’ attention, capacity of filling-in the missing connections in the narrative, and so forth.
Christoforou C. and colleagues (2015) proposed new parameters in the analysis of narrative films/ video clips combining eye-tracking – to examine eye-gaze dispersion (statistically measured by the eye-gaze Divergence Index/ iGDI) according to the video stimuli – along with the Heart Rate Variability/ HRV (measured by PPG/ photoplethysmography signal sensors attached to the left ear lobe), in order to monitor the biological processes associated with the shifts in attention; they suggest that “audience preferences on video are modulated by the level of viewers lack of attention allocation” (Christoforou et al., 2015, p. 1-4). They show that for the narrative video-genre “no single frame or element can be explicitly identified as being of interest, but rather the entire sequence of frames work together to drive the communication”; “the more the scenes in a video that allow the viewer attention to diverge from a common attention path, the more opportunities for the viewer to disconnect from the narrative and the message, and lose interest” (Christoforou et al., 2015, p. 9-10). Such studies allow for an understanding of the ways in which visual information can influence perception, the impact of trailers on audiences and how they trigger interest for certain film genres, as well as for the study of desensitization to media violence (Christoforou et al., 2015, p. 10). These can be listed as benefits of employing neuroimaging techniques in films and film reception.
Aya Ben-Yakov and Richard N. Henson in their study of “the hippocampal film editor” (2018) observed that the hippocampal activity during film viewing, measured and scanned by fMRI, is “both sensitive and specific to event boundaries, identifying a potential mechanism whereby event boundaries shape experience by modulation of hippocampal activity”; hippocampal events are seen as “moments in time with the strongest hippocampal activity” (2018, pp. 10057). In film watching, these event boundaries appear when the spectators view the transition from one event to another in the film, characterized by visual and auditory changes. These researchers found that the hippocampus’ activity increased “only at those cortical pattern shifts that coincided with an annotated boundary, suggesting that it is the boundaries, not the pattern shifts, that drive hippocampal activity”; moreover, the hippocampal activation “was not only triggered by a boundary, but was also graded according to the salience of a boundary measured by the level of agreement across observers” (Ben-Yakov and Henson, 2018, p. 10065). On the other hand, “the hippocampus responded at some moments in time not characterized by a large perceptual change, whereas some salient perceptual changes went “unnoticed” by the hippocampus if they were not experienced as a boundary”; thus, the hippocampus appears to be important for segmenting continuous experience, most likely to transform continuous experience into representations of discrete events for registration into memory (Ben-Yakov and Henson, p. 10066).
These findings are supported by the results of other studies and theories as well: the Event Segmentation Theory (Zacks et al., 2007) suggests that “we naturally segment continuous experience into events and this segmentation is driven by moments in time when prediction of the immediate future fails (event boundaries). Segmentation affects not only our perception of the experience, but its subsequent organization in long-term memory, such that elements within an event are bound together more cohesively than elements across events; the hippocampus exhibits a general sensitivity to prediction error, combined with its well established role in episodic memory formation (Zacks et al., 2007, in Ben-Yakov and Henson, 2018, p. 10058). Event boundaries are very well marked when accompanied by changes in aural environment, music or rhythm changes, conspicuous shifts in mood, image texture, lighting, narrative disruptions, and other elements, and all contribute to the viewers’ immersion into the films.
The engagement with film brings together the act of immersion (that is, full involvement in the diegetic matrix) and aesthetic distance, as well as identification, which every spectator negotiates in one’s own terms. With neuroscience, the act of immersion into the filmic narratives can become more controlled, guided towards particular purposes. Apart from the extremes to which this control can be forced (i.e., unethical propaganda and brainwash), such control can reduce the spectators’ interpretative capacity, the development of their aesthetic affinities, their curiosity and abilities to deconstruct the text and its metaphors in search for hidden meanings which, while they might not be very meaningful for one’s self, they might constitute valid principles and values for humanity and society at large.
When a film is created with the objective to modulate certain cognitive patterns and behaviors, or to stir up certain (consumerist) interests, the filmmaker’s approach is distinct. While the director may still enjoy some authorial position up to a point in neurocinema, this is different from the actual auteur perspective (though subjective) from which the filmmaker puts forward a recognizable style, great technical proficiency, a particular philosophy or thematic concern deeply rooted in his artistic personality. Film d’auteur usually falls under the category of art film/ cinema (see French filmmakers and auteur theories since the 1950s, the French New Wave, the New German cinema etc.); it is often characterized by specific aesthetics, value systems and signifying practices (some films being quite revealing, philosophical, or simply, ingenious), which challenge the intellect and encourage the spectators towards a rich hermeneutical exercise and decoding of messages at manifold levels based on axiological assumptions and previous knowledge. When one names film auteurs like Jean-Luc Godard, François Truffaut, Ritwik Ghatak, Andrei Tarkovsky, or others, one surely knows what aesthetics, quality and signature their films carry.
Neurocinema does not seem concerned with such values or aesthetics, with classical notions of realism (all the more so because most spectators seem to get less engaged in art and documentary films, realism, or moving images showing ‘life as it is’) and authorship (with its ‘signatures’, ‘languages’ and signifying practices), or with art qualities; it is more focused on applying instruments of mathematical exploration, on recording and predicting human behavior. In case of neurocinema, especially when it aims at becoming interactive and changing the narrative presentation in real-time according to viewer’s responses, the spectator turns out to be co-participant in the filmmaking process, acquiring a more authorial position. Of course, most studies up to present have focused on narrative films, so more research is necessary with emphasis on other film styles and genres, and possibly someday there will appear an auteur neurofilm/ neurocinema as well.
Through neurocinematics, the experience of film reception and responses to visuals (and virtual realities) change. The neurofilm focuses on identifying spectator’s reactions in order to provide them in turn with the same stimuli that trigger those reactions and the needs/ desires/ expectations they entail. By modelling its content for specific audience reactions (in interactive films), it has the potential to incite a repeating pattern and circuit between stimulus–need/desire–attention–emotion–cognition–reward and back (see the reward circuit, dopamine and serotonin implications), to offer the subject-spectator exactly what s/he wants and likes (i.e. a certain type of thriller, horror, erotism etc.). Thus neurofilm may turn into a ‘need’ which in time may lead to obsessive behaviors, alteration of social relations, disharmonic emotions, reduced neuroplasticity, and cognitive abilities, in different degrees similarly to addiction to virtual reality and computer games. Moreover, as the spectator constantly receives/has the same experiences (and possibly a ‘sense’ in an aimless life), with fewer intellectual challenges – though this can happen with other less thought-provoking films as well – the neuroplasticity may be affected, because the brain needs to ‘learn’ new things, be tested and kept active in order for neurogenesis and neuroplasticity to ensue. Neurocinema may limit the novelty of discovery in the film-viewer and director-film interactions, the curiosity of exploration and the creativity of both filmmaker and spectator.
Conclusions
In conclusion, neurocinema focuses on the scientific aspects of films, more specifically narrative films, with the aim of exploring and modelling the human mind, response, and behavior according to precise objectives. The paper has examined several studies highlighting the main cognitive processes involved in film viewing, as well as the effects of neurocinematics on spectator’s perception, cognition and emotion prompted by the complex dialogical interaction between films and audiences. Intrinsically, neurocinema research prompts questions about methods, ethics and ethical concerns and needs to consider the participant’s informed consent and the effects of such research on the subjects.
Research in neurocinema and neuroscience cannot offer accurate data on the films’ quality, their cultural and aesthetic value, but they can offer information on the effects that different films, genres and film styles can have on the human brain through varied instruments and methods. Indeed seeing live the emotional responses of spectators on the fMRI scans represents a wonderful achievement for researchers and the world, but they do not say much about the beauty of an artwork, about what makes it a valuable asset to human civilization, or about the creative genius who has released it to the world, the dialogical interactions, or ‘the metachronotope which keeps the artwork alive’ (concept developed elsewhere, Ceciu, 2013). I do believe that science is an art in itself but science without art and imagination would be devoid of the capacity of evolution and procreation (i.e., of insights, knowledge).
Apart from the visual information processing, the desensitization to media and film violence, and other uses already mentioned, the advantages of employing neuroscience and neuroimaging techniques in films include several other aspects: researchers can get to better comprehend how the human brain processes emotion, and the moving images versus static images, how these images are experienced in everyday life; how people interact with a film, characters and objects, how they make sense of certain visual settings, movements, actions, cinematic music etc. Neurocinema can become useful in psychotherapy, in treating trauma, phobia, depression, anxiety, and other such afflictions; such techniques have medical benefits as well. Further research is necessary to find more possible applications and ways in which neurocinema can contribute to emotional healing, to treating trauma and other psychosomatic conditions. One drawback in applying neuroscience and neurocinematics to psychotherapy and other research fields would be the high expenses, the expertise, the complex technology and equipment required for such applications. Thus, neurocinematics can be better studied comparatively in multiple contexts and employed towards the benefit of people and society, through the integration and dialogical interaction of several disciplines, to encourage and develop an inter- and multi-disciplinary research methodology beyond the existing disciplinary limitations.
References
Allen, Richard. (1999). “Psychoanalytic Film Theory.” In A Companion to Film Theory, ed. by Toby Miller and Robert Stam. 123-145. Blackwell Publishing.
Anderson, Joseph D. (1996). The Reality of Illusion: An Ecological Approach to Cognitive Film Theory. Carbondale, IL: Southern Illinois Univ. Press.
Bazin, André. (1967). What Is Cinema? Essays. Berkeley: University of California Press.
Ben-Yakov, Aya and Richard N. Henson. (2018). “The Hippocampal Film Editor: Sensitivity and Specificity to Event Boundaries in Continuous Experience.” The Journal of Neuroscience, 38 (47), 10057–10068, https://doi.org/10.1523/JNEUROSCI.0524-18.2018, accessed 29/11/2020.
Bordwell, David. (1989). “A Case for Cognitivism.” Iris 9, 11–40.
Bordwell, David. (2007). Poetics of Cinema. New York: Routledge.
Bordwell, David and Noël Carroll, eds. (1996). Post-Theory: Reconstructing Film Studies. Madison: University of Wisconsin Press.
Carroll, Noël. (2008). The Philosophy of Motion Pictures. Oxford: Blackwell.
Ceciu, Ramona L. (2013).“The ‘Architectonics’ of Corporeal and Textual Selves: From Durga via Charulata and Banalata Sen to the ‘Virtual’ Indian Woman.” Asian Studies, I (XVII), 1, 65–90. Ljubljana: University of Ljubljana, http://revije.ff.uni-lj.si/as/article/view/247/305.
Christoforou C., Christou-Champi S., Constantinidou F. and Theodorou M. (2015). “From the eyes and the heart: a novel eye-gaze metric that predicts video preferences of a large audience.” Frontiers in Psychology, vol. 6, 579-89, doi: 10.3389/fpsyg.2015.00579, accessed 23/11/2020.
Cîrneci, Dragoș. (2011). “Three anti-correlated neural networks managing brain activity.” Romanian Journal of Neurology, vol. X, nr.1, 19-26.
Curtis, Silver. (2009). “Neurocinema aims to change the way films are made.” Wired, September, https://www.wired.com/2009/09/neurocinema-aims-to-change-the-way-movies-are-made/, accessed 24/11/2020.
D’Aloia, Adriano and Ruggero Eugeni. (2014). “Neurofilmology: An Introduction.”Cinéma & Cie International Film Studies Journal, vol. XIV, no. 22/23, Spring/Fall, 9-26, https://www.researchgate.net/publication/297045397_Neurofilmology_An_Introduction, accessed 30/11/2020.
Eisenstein, Sergei. (1957). The Film Sense. Trans. Jay Leyda. New York: Meridian Books.
Eisenstein, Sergei. (1959). Notes of a film director. Trans. X. Danko, Moscow: Foreign Languages Pub. House, https://archive.org/details/notesoffilmdirec00eise, accessed 23/11/2020.
Eisenstein, Sergei. (1985). The Short Fiction Scenario. Trans. Alan Y. Upchurch. Calcutta: Seagull Books.
Gallese, Vittorio. (2001). “The ‘Shared Manifold’ Hypothesis from Mirror Neurons to Empathy,” Journal of Consciousness Studies, 8, No. 5–7, 33–50.
Gallese, Vittorio and Michele Guerra. (2014). “The Feeling of Motion: Camera Movements and Motor Cognition.” Cinéma & Cie International Film Studies Journal, vol. XIV, no. 22/23, Spring/Fall, https://www.academia.edu/14959149/The_Feeling_of_Motion_Camera_Movements_and_Motor_Cognition, accessed 3/12/2020.
Gibbs, Raymond W. (2006). Embodiment and Cognitive Science. New York: Cambridge University Press.
Grodal Torben. (2007). “Pain, Sadness, Aggression, and Joy: An Evolutionary Approach to Film Emotions.” Projections, vol. 1, issue 1, Summer, 91-107, doi: 10:3167/proj.2007.010107, accessed 26/11/2020.
Hasson, Uri; Ohad Landesman, Barbara Knappmeyer, Ignacio Vallines, Nava Rubin & David J. Heeger. (2008). “Neurocinematics: The Neuroscience of Film.” Projections: The Journal for Movies and Mind, Vol. 2, Issue 1, Summer, 1–26, doi: 10:3167/proj.2008.020102, accessed 26/11/2020.
Jauss, Hans Robert. (1982). Toward an aesthetic of reception. “Theory and history of literature” series, vol. 2. Trans. from German by Timothy Bahti. Minneapolis: Univ. of Minnesota Press.
Kuleshov, L.V. (1974). Kuleshov on Film: Writings. Berkeley, CA: University of California Press.
Kupść, Jarek. (2003). The History of Cinema (illustrated). Chennai: Orient Longman. First edition, 1989.
Lakoff, George. (1988). “Cognitive semantics.” In Meaning and Mental Representations, ed. by Umberto Eco, 119–154. Bloomington: Indiana University Press, https://escholarship.org/uc/item/04086580, accessed 21/11/2020.
Lanzoni S. (2016). “Imagining and Imaging the Social Brain: The Case of Mirror Neurons.” Canadian Bulletin of Medical History, Fall, 33 (2), 447-64, https://doi.org/10.3138/cbmh.33.2.151-27012015, accessed 30/11/2020.
Mandler, Jean M. and Cristóbal Pagán Cánovas. (2014). “On defining image schemas.” Language and Cognition, 6, 510-532, https://doi.org/10.1017/langcog.2014.14, accessed 27/11/2020.
Metz, Christian. (1982). Psychoanalysis and Cinema: The Imaginary Signifier. Bloomington: Indiana University Press.
Morrison, S.E. and Daniel Salzman. (2009). “The Convergence of Information about Rewarding and Aversive Stimuli in Single Neurons.” Journal of Neuroscience 16 September, 29 (37) 11471-11483, https://doi.org/10.1523/JNEUROSCI.1815-09.2009, accessed 3/12/2020.
Mulvey, Laura. (1975). “Visual pleasure and narrative cinema.” Screen. 16 (3): 6–18, doi:10.1093/screen/16.3.6, accessed 03/12/2020.
McKell, Carter R., MacInnes Jeff, Huettel Scott & Adcock R. Alison. (2009). “Activation in the VTA and nucleus accumbens increases in anticipation of both gains and losses.” Frontiers in Behavioral Neuroscience, vol. 3, 1-15, https://doi.org/10.3389/neuro.08.021.2009, accessed 30/11/2020.
Mobbs, Dean, Nikolaus Weiskopf, Hakwan C. Lau, Eric Featherstone, Ray J. Dolan & Chris D. Frith. (2006). “The Kuleshov Effect: the influence of contextual framing on emotional attributions.” Social Cognitive and Affective Neuroscience, Vol. 1, Issue 2, September, 95–106, https://doi.org/10.1093/scan/nsl014, accessed 22/11.2020.
Peake, Thomas H. (2004). Cinema and Life Development: Healing Lives and Training Therapists. Westport: Praeger Publishers.
Poulaki, Maria. (2014). “Neurocinematics and the Discourse of Control: Towards a Critical Neurofilmology.” Cinéma & Cie International Film Studies Journal, no. 22-23, Spring/Fall, 1-15, https://www.academia.edu/11836830/Neurocinematics_and_the_discourse_of_control_Towards_a_critical_neurofilmology, accessed 24/11/2020.
Randall, Kevin. (2011). “Rise of Neurocinema: How Hollywood Studios Harness Your Brainwaves to Win Oscars.” Fast Company, 25 February, https://www.fastcompany.com/1731055/rise-neurocinema-how-hollywood-studios-harness-your-brainwaves-win-oscars, accessed 3/12/2020.
Read, Stephen J. (1987). “Constructing causal scenarios: A knowledge structure approach to causal reasoning.”Journal of Personality and Social Psychology, Volume 52(2), February, 288-302, https://psycnet.apa.org/fulltext/1987-15555-001, accessed 24/11/2020.
Rusu, Claudia E. (2008). Psihologie și Cinematografie: funcțiile psihosociale ale filmelor. București: Cartea Daath.
Schank, R. and Abelson. (1977). R. Scripts, Plans, Goals and Understanding: An Inquiry into Human Knowledge Structures. Hillsdale, NJ: Erlbaum Associates.
Siler, Todd Lael. (2016). “Neurotranslations: Interpreting the Human Brain’s Attention System.” AIMS Medical Science, 3(2), 179-202, http://www.aimspress.com/article/10.3934/medsci.2016.2.179, accessed 26/11/2020.
Vertov, Dziga. (1928). “The Man with the Movie Camera.” In Kino-Eye: The Writings of Dziga Vertov, Annette Michelson ed. Kevin O’Brien tr. University of California Press, 1995.
Wollen, Peter. (2007). “On Gaze Theory.” New Left Review, no. 44, 91-106.
BrainMovie-Avatar Trailer. (2009).https://www.youtube.com/watch?v=KSKIkXvqruI&feature=youtu.be, accessed 29/11/2020.
Katz, Peter. (2009). Neurocinema, https://www.youtube.com/watch?v=8XYAQ69x918#action=share, accessed 24/11/2020.
Nolan, Christopher. (2014). “18-Minute Analysis by Christopher Nolan On Story & Construction Of Memento.” The Lord Louis Show, November 20, 2014, https://www.youtube.com/watch?v=tYScJZWhaHA&feature=youtu.be, accessed 27/11/2020.
Tikka, Piia. (2018). „TLÜ 13: Neurocinematics–a peek into the narrating mind.” Tallinn University, 13 March 2018, https://www.youtube.com/watch?v=jiV-Doflpjw, accessed 29/11/2020.
Your Brain on Film by Rachel M. (2017). https://www.youtube.com/watch?v=6Vcu_7lcHDM, accessed 30/11/2020.
