We've all been staring at a lot of video calls this last year. Pressure is starting to build, with headlines that video calls trigger our fight or flight response. We want rest from zoom fatigue!
Well for those of us privileged to have access to Mixed Reality Spatial Computer headsets, we can already participate in the answer to video calls: being avatars together.
With Microsoft's latest announcement of Mesh it seems the perfect time to explain to anyone who will hear that the answer to video malaise we are stuck in, and the portal to a new world of possibility and innovation, is already here in the form of virtual avatar platforms and tools.
Microsoft Mesh is unique in that it is taking an SDK approach of being a platform that other developers can use to make advanced experiences. This post will focus on virtual avatars from Mesh, Magic Leap Social and Spatial.io, yet another frontier is 'holoportation' using 3D scanning cameras to bring a realtime volumetric video feed. That aspect is part of the Mesh vision, yet for now the focus is on virtual avatars. So what's this experience like?
Being a virtual avatar, and hanging out with others who are, is a bit complex to explain as a technology yet super easy to explain as an experience: It's like hanging out with cartoon people.
Natural little things that we do together in rooms come back. We meet each other at random places then configure a circle to talk around. One of us goes to get a notebook and sits down to write in it. Another counts on their fingers. And the eye contact is amazing, instead of looking askew at a video representation, I'm looking into the eyes of the other avatar and they are looking at mine. Looking up and left to recall, right to think. Side eye and subtle head nods during another's dialog. shifting weight between heels, scratching the head and of course high fives, fist bumps and other bodily interaction.
Yet this is an entirely new medium. So we conjure up drawings and models between us and have a new kind of rich media immersive conversation, or is it an experience? Wait is this an 'interface' anymore or could it be a language itself...
I have been using Spatial since it was released and its ability to provide a 3rd person view and photorealistic avatars brings a new frontier for virtual presence and even enables sharing out to 2D folks via video. It serves as not just a meeting tool, but a design tool and has already helped projects move from spending lots of time in theory and discussion and moving straight into tangible prototyping.
These avatars are not just for personal use, this ability changes the nature of education, particularly for complex topics like medicine, as innovators like Shafi Ahmed are exploring.
It is not a perfect simulation of a person by any means, but it does not have to be. Emotion is conveyed, a presence is given to focus on. After every avatar chat I've had on magic leap and hololens 2 I feel refreshed, energized, and usually inspired. Now, granted, it helps that the people working on this technology are amazing and inspiring to me, but still, it's a feeling I have never carried over from a video call. To stand in space with your friend for an hour or so, vs to sit in a chair and look at each other, or listen to each other while we each awkwardly look around.
Being together as avatars feels like how we want to experience communication. We are embodied creatures and the new medium of virtual avatar has long had the chance to give us a chance to feel present together. Now the technology is here, available now, to make it real.
My first 'mixed reality' experience of virtual avatars was with Magic Leap's built in Social application. I had a great session with their head of product for Social you can see here, and I've linked the videos below.
Spatiate remains my favorite cross platform Avatar and drawing app, as it can serve so many roles. All activity within a session can be saved as a 3D model to sketchfab (you can see some of mine on my art page). The mobile app can serve as a 'camera' for innovative video possibilities and also can serve as a canvas, as well as enabling more participants who don't yet have headsets. Note the second video in this playlist where upon meeting a new friend as an avatar we proceed to naturally draw a flower together in space between us.
See a lecture I gave to RMIT students that brings this all together over 2D video. My experiences giving talks using this avatar technology to 2d video audiences have shown me that there can still be an 'upgrade' of capability and experience even if the audience is not immersed. yet for best results, we will all want to be part of it together rather than just observing.
The next frontier for these platforms is in sharing the scanned room data so that we can merge contexts and digitize physical objects, but I'll save that for another post.
As a final thought on avatar design and the limits of technology, an excellent talk/demonstration from SIGGRAPH 2017 featuring one of the founders of computer graphics, Ken Perlin, is a must watch. His message: keep the emotions readable, even well before the realism. Accurate emotional conveyance is the key that will let us communicate effectively in this new medium.
The Magic Leap Social and Microsoft Mesh avatars do an excellent job along these lines. The Spatial avatars are impressive yet uncanny, and the Spatiate avatars are an exercise in minimalism. Each one still delivers on the promise of embodied virtual presence and I highly recommend giving them all a try, and then switching from video forever! ;)
So if you can get a headset and give it a try. Find me on Mesh (johnhanacek [at] gmail.com) and view the AvatarMEDIC Spatial room for an example.
PS I've included a talk by the legendary Ken Perlin about how to enable a new 'between us' language and shared imagination space. Virtual avatars are about more than just communication of what we know, they offer us a portal into entirely new possibilities. Human conversation is about to get radically upgraded. For now though, at least they solve video fatigue!