VR/360/AR/MR

It’s been a while since I’ve written about films and that is mainly because my research activity has been consumed by work on Virtual Reality for the last 12 months. As part of the Colliding Worlds: Virtual Reality and Beyond project at Victoria University of Wellington, I’ve been working with Professor Neil Dodgson and Matt Plumber to bring together academics in different disciplines to share their expertise on VR as well as think about the crossovers with 360-degree imaging, Augmented Reality and Mixed Reality. This has led to some of my own experiments with 360-degree filming and will hopefully lead to more practice-based research in the future.

Although I’m interested in the multifaceted cultural and industrial contexts in which VR content is being produced, my background in Film Studies means I’m most intrigued by the new wave of cinematic VR content. I include 360-degree filmmaking in this even though VR purists might argue that its lack of interactivity means that it does not truly present a virtual reality. What’s likely to come, then, is a series of blog posts where I take a close look at some VR ‘experiences’ and, similar to my posts on 3D films, try to find emerging trends and themes.

d2605e3d47e0c3d63e1f57a5c3dbaf15

All three of the experiences that I am looking at in this post are hosted on the Jaunt platform. One of the useful things about the Jaunt platform is that it can be viewed on desktop computers (I use Firefox and, as far as I know, it also works in Chrome). You don’t get the same immersive experience that occurs when you view the experiences in a VR headset (I’ve been using the HTC Vive headset for viewing) but you can use your mouse to navigate your way around the 360-degree spherical view. I would liken this viewing possibility to the difference between watching 2D and 3D versions of the same film: the desktop version lets you follow the narrative and grasp many of the aesthetic qualities but you don’t have the same multi-sensory, embodied, closeness that you get in the headset view. The following experiences are a somewhat random selection of content currently available on Jaunt but they do point towards some of the different and similar ways that filmmakers are approaching cinematic VR.

Always Sunny in Philadelphia: Project Badass

The Always Sunny in Philadelphia experience isn’t the first time Jaunt has teamed up with a well-known brand; the platform has already produced content in collaboration with Star Wars, ABC news, The North Face, ESPN and others. It is similar to many other spin-off/tie-in VR experiences whereby a mini narrative is produced that fits into the fictional world of the brand (the Always Sunny in Philadelphia TV series) but is not necessary-viewing for anyone following the rest of the entertainment series. In this case, biker Mac invites viewers to follow him as he creates yet another video to prove to his friends that he is ‘badass,’ this time by attempting to ride his motorbike off the end of a pier.

maxresdefault

Like many other cinematic VR experiences, the opening shot situate the viewer in the very centre of a visually rich scenario. Positioned on top of Mac’s motorbike, you can see a small crowd of people waiting eagerly at the bottom of the pier, a large crane behind you and the ocean populated with surfers stretching out on either side. The shot encourages you to crane your neck to look around but, in doing so, you are in no danger of missing any of the narrative as the characters remain in relatively fixed positions, talking rather than completing any complex action. This central, immovable, position is justified in the narrative when Mac’s friends Dee and Dennis discuss you as if you are there, a friend of Mac who is strapped to the motorbike and forced along for the ride. While this type of exposition, explaining why you are situated where you are, is starting to feel somewhat over justified in VR experiences, it does have the effect of allowing characters to talk to the viewer directly so that this fourth-wall-breaking lends a greater sense of presence. The dialogue is crafted to allow this to happen: you are asked rhetorical questions and none of the comments suggest gender, ethnicity, age or other identifiers that would suggest someone other than you is being talked to. Furthermore, this set up provides justification for why the viewer is given access to a privileged position amongst the inner circle characters; something that the diegetic spectators at the end of the pier are not allowed. Rather than operating as the omnipresent viewer common to 2D cinema who can witness all action, but from a distance, the viewer is grounded centrally and expected to feel events as they occur.

At the same time, Mac’s friend Frank playfully undermines the experiential concentration of the viewer by presenting a stripper ready to take her clothes off to the song Uptown Funk. Telling Mac, ‘if you want people to see your video, you’ve got to have some skin,’ Frank suggest what we have always been suspicious off: that viewers prefer to be the voyeur rather than active participants. Mac’s response ‘it’s going to distract from the performance’ also highlights a conundrum facing creators of 360-degree content. The lack of a frame means that it is impossible to focus viewer attention in sustained ways and so narrative has to be able to take into account the increased opportunity for visual distraction.

alwaysunny_360_video

When the motorbike finally takes off, there is an extremely visceral feeling of speed as it races along the pier, even though your view point is partially blocked by Mac driving the bike. After zooming up the ramp, the bike crashes into the water so that we are subsumed in the depths of the ocean. In the same way that many underwater scenes occurred in 3D films, it is likely that cinematic VR will capitalise on the greater sense of sensory immersion that these visual technologies can create when placed in liquid fields. In this short segment, the experience thus combines the visceral feeling of motion and immersive submersion that many recent VR ‘demos’ used to promote the new wave of VR headsets.

One last thing worth mentioning is that Always Sunny in Philadelphia is shot as if it were a one-take film. There are in fact a number of ‘hidden’ cuts: one of the characters momentarily places a hood on the viewer’s head (presumably allowing the change from actor Rob McElhenney to a stunt double); the motorbike travels through smoke leading to a white out (allowing the stunt man to be removed from shot before the bike hits the water). However, the sense of uninterrupted action adheres to the idea that VR experiences should immerse us in the world as if we were really there. No cuts in reality translate to no cuts in this artificial visual world. While I’m not in favour of sticking dogmatically to principles such as this, for a short cinematic experience like Always Sunny in Philadelphia the lack of edits give it a self-contained quality that emphasises the simple goal of experiencing something thrilling: a daredevil stunt with the potential to go horribly wrong.

My Brother’s Keeper

Unlike Always Sunny in Philadelphia, My Brother’s Keeper is a stand-alone 11-minute experience. Set in the US Civil War and depicting two brothers on either side of the conflict, the experience treats much more serious themes than the former. The slow pace of the experience suits the long shots which allow significant time to look around (towering crop fields interspersed with soldiers, forests providing cover for advancing troops, flashbacks to the places where the brothers played as children). In most cases, the edits are gentle ellipses across time that allow us to transition from one scene to the next. Even when they become more rapid, they move us through time or to a completely different space rather than allow us a different perspective on the same scene. In this way, My Brother’s Keeper follows a standard, that has been set up in the first wave of twenty-first century VR cinematic content, in order to minimise the disruptions that edits impose on the embodied presence of the viewer. Nonetheless, there are experiments with expressive shot construction that move the experiences away from an overtly realist visual framework. When a type of shot/reverse-shot occurs with each of the brothers (current age and their childhood version) shooting their guns in individual shots, we are given the feeling that they are interacting with each other even though they are not in the same space. This is emphasised by a type of blurring out of the 180-degrees field in front of them, an artistic diminishing of the plentiful 360-degree view. The effect feels quite jarring but I am inclined to believe it is because we are only at the initial stages of expressionistic experiments with 360-degree cinema language and it will take a while for viewers to become accustomed to them.

20170119_164407_64214420170118_032019_717956mbk_btsgallery_5-jpg-1280x720_q85

My Brother’s Keeper also plays around with camera positioning in a way that I haven’t seen in other 360-degree experiences. In a dramatic moment towards the end of the experience, one brother (then the other) is knocked to the ground and the camera lays next to him. Our balance is shifted as we are still upright in our external viewing position but laying on the ground in our internal view of the brother. Although we are used to seeing the camera tilt, move onto its side and even go upside down in 2D cinema, the lack of frame in the 360-degree spherical view makes this a much more disarming experience. It is a little like what happens in 3D films when this unbalancing of the cameras’ equilibrium asks our bodies to react more keenly.

Helping us through these experiments with the visual field is a continual voice over from one of the brothers. While not uncommon in 2D cinema, I have the feeling that voice over is becoming prevalent in cinematic VR for a number of reasons. In the first instance, it allows a sense of continuity as we are moved from one spherical field of view to the next: a sensation that can be jarring for first time users of VR headsets. Secondly, the disembodiment of the voice-over helps us remember that embodiment in virtual spaces is contingent rather than a given so that we are open to having our sense of embodiment in the virtual space played with. Finally, one of the difficulties of shooting within a 360-degree spherical field of view is that it is hard to direct viewer attention to where they should look when they can potentially look in any direction. Providing exposition and narrative information within a voice-over that is not dependent upon what the viewer is looking towards at any given time is thus useful for the storytelling process.

image_thumbnail_2_v1484907881

One of the ways in which the Jaunt platform has been supporting new work such as My Brother’s Keeper is through the provision of a 360-degree camera rig. Included in their rig are enough cameras for stereoscopic capture. Although the different images produced by stereoscopic visual sets compared to monoscopic visual sets in 360-degree filmmaking are less obvious than in traditional screen media where the frame makes it clear if objects are in front of or behind the screen plane and thus imbued with additional depth, there are some moments when stereoscopic 360-degree shooting makes a difference. In My Brother’s Keeper this most commonly occurs when we are placed very close to one of the brothers and are encouraged to concentrate on their faces in the same way when confronted with an extreme close-up. Stereoscopy in these shots means that our heightened depth perception is matched with a visual understanding of the intricate contours of their faces so that we are presented with a more intense engagement with the characters. In one particular shot, one of the brothers holds a shotgun and its long barrel extends from his shoulder and into the inner sphere of our personal space. The stereoscopic depth relations allow a greater sense of connection between the fictional object and the embodied space that we occupy. At the same time, one of the difficulties of working with a stereoscopic 360-degree camera rig is that it is much more difficult to stitch all the different images together into one spherical field of view. Certain artefacts of this imprecise process remain in My Brother’s Keeper when, for example, some objects bulge towards us when they should recede and vice versa.

Invisible, 1: Ripped from Reality

Of the three experiences, the first episode of Invisible, an original content six-part VR series produced by Jaunt, is one of the most experimental in terms of editing and visual effects. Its supernatural theme and science fiction feel means that it is distinct from the numerous documentaries and other realist works that have so far provided the bulk of VR cinematic content. Indeed, the episode’s title ‘Ripped from Reality’ indicates that this will be the case. Opening on a street in Haiti, a girl passes by our central position but specifically peers at us, thus breaking the fourth wall and giving us a sense of presence at the scene. In this way, we are given a type of POV shot that situates us in the scene but, a few seconds later when the experience cuts to the inside of a house where a young woman, Susan, is in labour, spatial edits jump us across space to the other side of the room. This type of edit, giving us different viewpoints on the same scene, situate us as the omnipresent viewer common to 2D cinema. However, the lack of frame to allow us to feel outside the scene means that we feel as if our whole body jumps with us, rather than just our viewpoint. As with My Brother’s Keeper, the use of a voice-over from Tatiana (Susan’s cousin) during this scene helps narrative flow as the viewer keeps up with the movement around visual space.

invisible-1

Shortly after, there is a change of scene to excerpts from news footage that explain the death of Tatiana’s grandfather. This information is displayed as a series of screens within the 360-degree sphere, stretching all around us and competing for attention. It is a simple set up but one that reminds us that we are within a new visual scenario that is different from the framed screens represented by the news footage.  We then enter a series of credits that begin with an aerial shot taking us over a suburban landscape. When looking ahead, in the direction the camera is travelling, the footage is clear but when looking behind, a type of wave-like visual effect blurs the landscape. Similar to the visual effects in My Brother’s Keeper, this shot moves away from a realist framework and towards a visual experience that is more expressive. This is particularly apt in the next shot of the credits where we seem to be placed with a foetus inside the womb: a quite creepy scenario, particularly as the foetus has an uncanny, digital quality.

In the next scene, in a hospital, we are positioned at the intersection of two corridors, meaning it is possible to view two related but separate pieces of action unfolding at the same time: an anxious pair of relatives and Susan’s boyfriend approaching another relative. There is also action in other directions as patients and hospital workers move around the space. Previously, this type of multiple action would only have been viewable using split screen effects. Now there is the opportunity for the viewer to decide which piece of action they will turn their head towards and it is this potential that is leading many filmmakers to call 360-degree cinema interactive even though it does not have the 6 degrees of freedom in movement that other interactive VR experiences allow.

inv_tlr3_susan_penultimateshot

Following a series of spatial cuts that take us around the hospital, we see a series of strange events occur as a semi invisible body attacks numerous people. In some ways, because we are so close the action, and feel physically amongst it, the over-dramatic acting and the there/not-there quality of the digitally created semi-invisible body means that the scenario becomes farcical. However, when the semi invisible body comes towards, and seems to cover us there is a feeling of encroachment on our bodies, an amplification of the looming effect in other media (think of the jumpy moments in horror films when a monster lunges at you and you feel the need to physically recoil). When we later see one of the relatives dead on the floor, the stereoscopic depth relations allow him to stretch towards us and increase the sense of co-presence in the space. This aspect, combined with the looming effect mean that there is a more heightened sense of sensory stimuli than can be found in other 360 experiences.

Taken together, these three Jaunt experiences offer a good snapshot of the current state of fictional VR cinematic 360-degree storytelling. Their experiments as well as their repetitions of established 360-degree tropes point to the exploration that is taking place in this area yet their short duration – no more than 11 minutes – also indicates that it might be some time before filmmakers work out how to combine these elements in effective feature-length storytelling.