Over the past six months we – Mandy Rose and Tom Livingstone, the UWE research team from the “Experimental Productions” work package – have been scoping a research theme under the title: Multisensory Storytelling. The research intends to explore the synergies between several strands of active research and development within MyWorld and the creative industries more generally. Our aim was to explore the synergies between multi-modal Human-Computer Interaction and cultural experiences that enlist more than just the eyes and ears of the audience, engaging our full bodies, as well as our touch, smell and even taste.

We wished to consider whether a new field was emerging at this intersection of emerging technologies and innovative storytelling practices, that operated beyond the audio-visual paradigm that dominates media. As a culmination to the initial phase of this research, we organised a workshop targeted at stakeholders and practitioners within the MyWorld ecosystem, as a way of sharing our initial thoughts and gathering additional perspectives on the concept of Multisensory Storytelling.

We received a great deal of interest and on 5th June 2024, at Pervasive Media Studio, we brought together fifteen engaged participants (see below), all of whom shared their experience and expertise with generosity. Our first and perhaps most valuable takeaway is a simple one: there is innovative practice, expertise and a strong appetite in the region and beyond, to get to grips with what is going on in this space. If, as we envision, multisensory storytelling is an emerging category of cultural experience and technical R&D that will only expand in significance over the coming years, the South West has the know-how and imagination to shape the space.

 

Multisensory storytelling experiences

The workshop was split into three thematic sessions, with each session offering three simultaneous topics for conversation, debate and knowledge sharing. The first was focused on Multisensory Storytelling Experiences, broken down into three topics: fiction, non-fiction, and installation. This session successfully broke the ice, with many participants sharing recent shows, experiences and installations that exemplified our overall concept of multisensory storytelling as a loose, but vital category of cultural experience that stretches beyond the familiar envelope of screen-based linear media.

“9D” cinema was an obvious touchstone; two participants were fresh from PunchDrunk’s latest piece Viola’s Room, which is an audio driven navigation of a variety of tight spaces, which are traversed barefoot. Heritage and non-fiction experiences utilising the evocative potential of smell and/or temperature were mentioned. Historical precedents were discussed. John Cage’s 4:33’ – created in 1952 – became an interesting touchpoint: withdrawing the presence of a composed melodic element to pull focus onto the embodied experience of sitting in an auditorium listening to silence. Does this count as multisensory storytelling? This being the opening session, there was a lively discussion about definitions. Surely all storytelling is multi-sensory storytelling; isn’t multisensory just a synonym for immersive? If so, is it better or worse? Is it even a useful term?

 

Multisensory storytelling technologies

The second session focussed on multisensory storytelling technologies, and the eager sharing of firsthand experiences, transformed into a more generalised querying. Is technology a prerequisite of multisensory storytelling? What tools and technologies are there? Are they readily available to any given production, or the gate-kept crown jewel of a media conglomerate (Disney’s omnidirectional treadmill), or the far-fetched prototype of a hyper-specialised university lab (Norimaki Sythesizer – a “taste display using Ion Electrophoresis”)? When it came to the technologies more familiar to immersive media producers and consumers (i.e. sub-packs and other haptic devices), the question of availability remained prevalent, as well as concerns over whether, ultimately, these technologies were being well deployed by the creative teams who have access to them.

In the context of MyWorld’s available equipment, the question arose as to whether or not the availability of the devices needed a complementary skills and training offering. What might that training look like? Might a Haptics School go some way towards mainstreaming and/or evolving the use of haptics and the sense of touch within virtual experiences, beyond gamified cues like force-feedback? These conversations rested on the previous debate about the quality of multisensory storytelling as it currently is: are artists being too literal in their use of other senses? Is the current generation of technologies subtle enough to transcend the interface and become part of the palette available to storytellers?

 

Multisensory storytelling stakeholders

The final session capitalised on the momentum in the room and focussed on the Stakeholders within the Multisensory Storytelling landscape, specifically: Venues, Audiences, and Creatives. The overriding question was: what are the obstacles facing each of these groups, in the generation of a vibrant and sustainable culture of multisensory storytelling? As with the first session, there were numerous accounts of first-hand experiences, many arising from the fact that uncharted creative and technological areas can only be explored through trial and error. Analysis of these frustrations aside, the thinking about the sector more widely struck consensus in several areas.

With regards to audiences: the lack of standardised processes for onboarding and offboarding, as well as for testing the integration of sensory modalities within experiences, makes converting immersive media sceptics, or first-timers, into “try anything” media consumers challenging. That said, Fever’s immersive experience platform is proving successful in driving audiences to multisensory offerings, from Van Gogh: The Immersive Experience to Bubble Planet. These events sit at one end of a growing wave of immersive experiences, but nevertheless go some way towards demonstrating the affordances that technologically rich exhibition practices embracing the multisensory have for improving access to cultural spaces for audiences with different access requirements.

There was a strong consensus that Multisensory Storytelling has the potential to reconceive how inclusion is practiced from the ground up. A case in point is The Museum of Austerity directed by Sacha Wares, who participated in the workshop. Staged at the Bristol Old Vic in the week following the workshop, it is a promenade AR experience that was able to offer multiple accessibility options. Sacha is developing her thinking and practice in this area further with Inside, (in which MyWorld is a co-investor and Bristol Old Vic a production partner). Inside will tell the story of the life of artist Judith Scott, born with Down Syndrome and profoundly deaf, in a manner that would be fully accessible to Judith were she alive to experience it. These examples aside, it remains troubling that multisensory storytelling, as it is currently positioned within the exhibition sector, still comes with such a high price tag. This leaves audiences with different access needs in the position of needing to pay extra in order to be addressed inclusively.

The problems for audiences are problems for venues also, exacerbated by the lack of a critical vernacular comfortable with appraising and promoting multisensory storytelling. The burden falls on venues to “make newness safe” and manage the expectations of audiences unfamiliar with the new technologies of exhibition that accompany multisensory experiences. The opening of the Undershed, in Bristol in October will undoubtedly move these conversations forward. The venue will exhibit/stage/host pieces that run the full gamut of what is meant by ‘immersive’ media and no doubt attract initiated and uninitiated audiences, alike.

 

Challenges and potential

All in all, the workshop demonstrated a range of productive tensions that will steer and galvanise our research going forward. A key takeaway is that multisensory storytelling as a cultural and technical phenomenon is advancing rapidly but in an uneven fashion, concentrating around certain practices, demographics and technologies. That said, the potential impact of multisensory storytelling as it is understood by those most intimately involved in its production, exhibition and consumption, vastly exceeds the niches in which it finds itself. As such, the mapping that this workshop undertook offered some significant indication of how the region might contribute to the evolution of the field.

The conversations in the workshop underlined that multisensory storytelling and universal design in VR are inseparable concerns. As a field of technological R&D and experimental practice, multisensory storytelling has the potential to significantly augment some of the naturalised presumptions and relations of ocular-centric media that are unassailable within the audio-visual linear media paradigm. It is in this area that the workshop contributed significantly to the thinking that we have already undertaken with regards the epistemological ramifications of these new media formations, especially as regards non-fiction and documentary media. This recent paper, delivered at the Interactive Film & Media Conference in June 2024, offers a theoretical overview of how multisensory storytelling relates to documentary practice and what new ways of understanding media we need to activate, in order to fully appreciate the impact Multisensory Storytelling is set to have.

Looking forward, our research will address these questions, particularly as they play out in research and productions within the MyWorld ecosystem. In 2025 we’ll be hosting a larger event to which all MyWorld partners will be welcome: an industry/academic seminar that will bring MyWorld and the region’s Multisensory Storytelling innovation into dialogue with other field-leading national and international work.

 

Workshop participants

Rachel Pownall – MyWorld Innovation Manager, Bath Spa University, Agapanthus Productions

Giles Chiplin – Producer & Head Of Bristol Old Vic Films, Bristol Old Vic

Melissa Blackburn – Senior Producer, MyWorld

Dr Lisa May Thomas, researcher-practitioner in performance and technology, Senior Research Associate, Centre for Sociodigital Futures, University of Bristol / Artistic Director, Soma

Verity McIntosh – Associate Professor of Virtual and Extended Realities, UWE Bristol

Tim Powell – Creative R&D Producer, MyWorld / Director, Buried Giants Ltd

Michelle Rumney – Producer, Immersive Arts, Pervasive Media Studio

Marcin Gawin – Interdisciplinary Artist

Sacha Wares – Director, Trial and Error Productions

Naomi Smyth – MyWorld Creative Technologist, Bath Spa University

Rosie Poebright – Creative in Chief, Splash & Ripple

Ruth Mariner – Creative Director / Writer

Eirini Lampiri – Creative Director / Production Designer / Maker