Feel free to check out the video first then read here for a more in depth look at the tech, process and next steps.
One of our team’s big goals is to create rich spatial collaboration tools for people working remotely. Evelyn lives in Boston, Vi in the mountains, and M in San Francisco, but we want to be able to be in overlapping spatial environments while we work. For years, we have done this exclusively in VR, working in a variety of software that let us co-locate for meetings, prototype walk-throughs and art explosions. But since the beginning of these experiments, we have been wanting to to incorporate our physical worlds. There was the scale model of M’s house they made in VR, Evelyn placed in AR, and Vi 3D printed. There was the recreation of Evelyn’s barn remixed recursively at multiple scales. There were virtual sculptures of our real dogs and 3D prints of our virtual code.
We can now move beyond simple recreations and bring our physical spaces closer together with our new prototype. Here’s how it works:
First we created an MRE (Mixed Reality
Environment) on the host computer. The MRE is then served to your headset of choice via a web socket. The HoloLens slurps up this data using MRTK (the Mixed Reality Tool Kit) embedded in a Unity app. Meanwhile, the VR headset (we used a Quest in our experiments) accesses the MRE via the MRE SDK Widget available in Altspace.
It’s like WebVR but it actually works cross-R.
Evelyn, Master Wizard of gluing bits together til they work, did all the expert finding out about these tools and baking them into this very exciting pie. Or maybe it’s more of a croissant. Magic croissant!
For our first big experiment with this set up, we created a large scale installation in Evelyn’s barn using paper, string and holograms. We started with the physical materials. Using string as an armature for the thin paper flags, we created light weight volumes in the space you could walk around and through as they fluttered in the breeze of the space heaters. Next we added the holograms.
When working with the physical materials, our problem solving mode was all about tension and knots. When we switched to working in the MRE, Evelyn went back to flatland to prod code into shape. M stood in the space with the HoloLens on, trying to figure out what a 1 unit measurement meant in the real world. There was a lot of trying to measure things with arm widths and marking out X and Z depths in tape on the floor. Because this is a very early prototype, all of our creative capacities are still restricted to the 2D code, so in order to place the holograms in the right spots in relation to the physical space we had to guess at scale and position numbers, reload the MRE, look at the results in the HoloLens and then guess again until we fine tuned each position.
With both the physical and virtual materials installed we could play with interaction. M and Evelyn were both in the physical space of the installation and Vi joined us remotely in VR. Walking around the installation wearing the HoloLens, the thin paper with its subtle floating motion was layered with candy-bright see-thru blocks and planes. The holograms, with their perfectly crisp corners and silky smooth faces, started off that particular kind of dead-still unique to simulated things. But once Vi touched them, from thousands of miles away, they spun to life, changing color and tripping over themselves ending up at odd angles.
The Big Bug
We did encounter one very frustrating but not project-killing bug. While the HoloLens could see the objects being served by the MRE, and even sync color and animation changes caused by the VR user’s interactions, it could not interact with the objects itself. Our original intention with this prototype was that both the local HoloLens user and remote Quest users could interact with the objects, but this bug meant that the HoloLens user was stuck in spectator mode for now.
Goals
Our short term goal for this project is to experiment with different interaction models that this combination of MRE and MRTK, HoloLens and VR makes possible. For this iteration we had the privilege of being in the same large working space together for a week, but our next iterations will go with out that luxury. Check back soon for a new progress report as we will continue to work cross-R as well as cross-continent, collaging together our studio spaces and practices.