Harry Willmott was one of six Watershed Fellows in Residence as part of the MyWorld Ideas programme and spent just over six months in residency with Zero Point Motion, who were awarded MyWorld CR&D funding for their project Digibeat. His Fellowship focused on devising and conducting user research for immersive experiences enriched with finger tracking and physiological monitoring, to understand how these features impact the user’s presence and immersion in the experience.
Over the course of the Fellowship, Harry spent a lot of time on user testing design and felt that playtesting was a useful and effective approach for the types of questions Zero Point Motion were interested in exploring. Including:
- How do we induce a better sense of immersion and of user presence in a VR experience, and what metrics would we measure to capture these?
- What types of stimuli induce specific emotions in the user that will best demonstrate the technology’s ability to measure physiological responses?
- How do we evaluate the improvement in user presence and immersion in the VR experience?
- How do we prepare the user of the VR experience to define a suitable emotional baseline? (e.g. using a dark room before they put on a headset)
- What should we record from the user after they have been through the VR experience?
Zero Point Motion have adopted Harry’s playtesting frameworks as part of their development cycles, which is a great outcome.
Harry has written a simple guide to playtesting for practitioners and SMEs starting out in the so called ‘immersive maker space’. Harry is keen to support with cultivating the West of England Immersive community and would love to hear feedback, create connections, or just have a chat. You can let him know via this form.
Why playtest
In whatever you are creating, whether it’s a VR game or interactive experience, if you intend the Engagent (Audience member acting as an ‘engaging agent’) within your experience to feel immersed, cultivate a sense of presence or predict their actions in any way, you will need to conduct a Playtest, ideally more than once. Even if you are beyond confident that an audience will flow and delight within your brilliant experience, a Playtest is at the very least a fun excuse to ‘Yes And’* your development. The value of regularly observing how people interact with your experience from its infancy to maturity is so important to the developmental success of your work it cannot be overstated.
There are significant distinctions between Playtesting and User testing. The latter implies the user is under scrutiny, though in reality it’s the opposite and just focuses on the isolated interaction between a ‘user’ and a specific technology. User tests often start with known questions and hypothetical answers, Playtests are explorative. During a playtest, it’s the experience as a whole that is being tested; seeing how Engagents, spaces and technologies interplay. Play is a rigorous and unforgiving form of evaluation and has an insightful impact on the development of any immersive experience, of anything. Play means to explore and discover, to ask and find answers, to pry and pull at the seams, to bend and break the rules. Through Play you can define the shape of your experience by learning how robust its edges are and the true depth of its core message.
The simple act of observing someone playing in an environment or experience of your design can answer and ask a hundred questions you didn’t know existed.
Conceptual Alchemy
Do not wait until you have something built that can be experienced, you should start Playtesting as soon as you start developing. As soon as you have your first concepts, playtest them. With your experience not yet being a real thing, perhaps the technology doesn’t even exist yet, Playtesting can be very daunting, like being overwhelmed by a blank canvas. All you need to do is perform some Conceptual Alchemy to distil your idea into your first few insightful Playtests.
Take the primordial DNA of your idea and add more conceptual elements like synonyms, allegories, analogies, substitutes and parallels, really basic stuff. Think of analogues to the interactions and inspirations behind your core message, mix it all together into a messy soup. It will be messy.
Treat this soup as a concoction of ingredients that you can scoop experiential snippets from. These small tastes of your experience should focus on specific emotions, interactions or States of Play**.
Next, find the antithesis experiences to these snippets and arrange them together like a conversation or a call and response. This provides your Playtest with some comparative data, a bit of pepper to balance the salt.
There you have it, you have conjured up your first Playtest from basically nothing!
Conceptual alchemy is about expanding your ideas, collecting all the ingredients you can and then distilling them into tiny tastes, vignettes of insight, ready to inform and support your experience’s development down the line, like little health potions.
The rest of your Playtests can follow a similar formula and will become easier to devise as your experience develops.
Conducting a Playtest
The act of Play is as unique to a person as their gait, the way they walk. Therefore, it is important to expose your experience to as diverse a spectrum of people as you can manage to gather the most valuable insight. This isn’t just about diverse demographics, you will need to Playtest with people who have no idea what your experience is about, who may not even be aware that this type of experience exists, who may not come across an experience like this in their normal day to day life. If appropriate, Playtest with Toddlers to Nonagenarians! It’s equally important to Playtest with your colleagues and friends for they will experiment with abandon and wrestle with the experience without prejudice or fear. They will also see the deeper truths and mechanics behind your work to honestly and constructively critique it.
Insight from a Playtest is gained principally from an observational modality. Leave your Playtesters alone as much as possible, interfering only when absolutely necessary. Do not ask them their opinions about any aspect of the Playtest while they are Playtesting, save any questions and discussion for afterwards. Enable space for reflection and time for open discussion. Directly talking to people about their experiences and asking them to express how they have experienced something is not only valuable to your experiences development but also a fascinating joy for yourself. Do not rush the conversation, make sure to provide ample time for you and your Playtester to get as much contemplation done as possible. Play is often instinctual and it can be hard for a person to describe their choices during a Playtest, you need to enable them to explore their choices thoroughly. Don’t be afraid of exploring deeper questions that border on leading questions, the important thing is to guide them and yourself into understanding the Playtest in more detail.
On the day
Brief everyone involved of what you intend to do during the Playtest. It can be difficult sometimes to provide adequate content warnings without giving too much away or accidentally introducing bias to the Playtesters. Just be careful with your wording.
If you can, employ someone on the day to take the notes, or run the playtest and you focus on taking notes. Do not try and run the Playtest and also take notes, you will miss things. Filming the entire Playtest, including discussions afterwards helps immeasurably when seeking deeper insights down the line during development. Trying to remember what this particular Playtester did at this particular time and only having your memory or sporadic notes from the day to aid you is folly. If you have the means, film everything. Use the streaming software OBS to directly record to a PC. Using OBS you can also include additional data like an onscreen time code, other angles or even a biometric data feed and record it all directly to the same video file. A wide angle, good quality video with clear audio and a timecode for reference is such a necessity for the longevity and effectiveness of your Playtests. Do not worry about capturing the best angles etc, just focus on capturing everything clearly.
You will need to get consent from your Playtesters to record them before the Playtest begins, this is a good opportunity to get them to sign an NDA as well if necessary. Find secure data storage that is in compliance with GDPR and store the recordings, data and surveys. Inform your Playtesters how and when their data will be used and let them know when it will be removed from storage. Store the data for as long as you expect the project to last then add another six months. You never know when you might need it.
Analytics
Finally, surveying your Playtesters can be insightful but only if you commit to a well prepared set of questions and are equipped to analyse them thoroughly. Conducting surveys for your Playtests can be very boring, confusing and time wasting if you don’t have the capacity to do it properly. Generally analytical surveys (as opposed to opinion surveys) provide nice looking graphs or charts and mostly only confirm insights you have previously inferred from observing and talking to Playtesters. Unless you require robust academic support for your experience, by supporting your observational data with analytical data, conducting super in depth and regular surveys isn’t necessary for Playtesting.
However, if you do want to include a survey, I recommend the following.
It is no good simply asking your Playtesters for their opinion about your Playtest (on top of your discussions with them) and expecting insight straight from their answers, you need to make sure that comparative data can exist to begin with. Make sure your playtest produces two or more sets of data. For example if you want to measure an Engagents sense of presence, you may need to set up a Playtest that enables the Playtester to demonstrate presence in a VR experience compared to a non-VR experience. Then you’ll be able to compare a sense of presence between the two and potentially prove via a lovely graph matrix that a VR experience enables more presence than non a VR experience. This is why many scientific tests have a control group, a set of basic data to compare the new data to. A control playtest doesn’t have to be the absence of a particular experience like in the example above, it can be a substitute or parallel experience. Devising a Playtest with two experiences that are like a call and response can enrich the insights you gather which can feed directly into making your project a far more nuanced experience.
Measuring presence in VR. Analogue(control) experience survey data vs VR experience survey data.
Once your Playtest can generate the data you need for your survey, you can begin writing the questions. Survey writing is a science of itself and fun to read about if you’re into that kind of thing, otherwise it’s very dry. Here’s some tips to get you started.
Even if your pool of people providing answers to your survey is small, use a ‘7 point Likert scale’ model of questions. Each question is a statement with multiple choice answers from negative to positive: Strongly Disagree, Disagree, Slightly Disagree, Neither Agree or Disagree, Slightly Agree, Agree, Strongly Agree.
Save yourself a headache down the line and make sure your questions are all balanced in a way that all the answers scale along the same negative to positive direction, left to right. That way your data won’t be tarnished with false positives/negatives and need some answers to be ‘flipped’. Basically make sure you don’t ask negative questions eg, “I didn’t experience a sense of flow” should be changed to “I experienced a sense of flow”.
A good starting point is to have 15 questions, each in three themed groups of five. That way you can deepen your analysis by comparing the groups. For example when measuring presence, you can split the questions into groups themed around ‘Embodiment’, ‘Flow’ and ‘Coherence’ and you can discover nuances between them. If you have a call and response Playtest or one with a control group etc, don’t forget you need to ask the same group of questions for both.
To aid in discussion after the playtest, if you have ample time, mix in a few subjective questions that encourage imagination and reflection, for example you could provide a space for Playtesters to draw a representation of each section of the playtest. You can compare them to the other Playtesters but because the drawings are exclusively subjective, they won’t provide as immediate insight as the survey analytics would.
Using the results?
Actually using the results from your Playtests? That’s down to you and your interpretation of everything you observe. Take everything you learn from the Playtest and feed it back into your development and conduct more Playtests! Let me know how it goes.
If you want to know more details about the playtests I conducted during the Fellowship, let me know. I intend to write about it all eventually and it would be a great motivation to know people are interested in reading more! As I said at the top, I’m keen to help cultivate the local ‘immersive makers’ community here in Bristol and the surrounding area so if you enjoyed this article, want to add your own advice or would like to get involved, please fill in this form.
*’Yes, and” is a GMing/Improv tool that invites creative collaboration (https://www.youtube.com/watch?v=etlxTPy4QcY)
** ‘States of Play’ is an idea that encourages immersive makers to consider their audience as engaging agents that can adapt how they interact with an experience on the fly. Instead of prescribing audiences as simply ‘Players’, ‘Users’ or ‘Participants’ etc, they are ‘Engagents’ who are observing at one point and exploring, hiding or playing at other times. With this perspective of the audience, immersive makers are able to better plan their audience’s experience and may predict what kind of ‘state of play’ they are in at a particular moment and adapt accordingly.