[I was writing something up on the situation VR is in (spoiler: it’s fucked but Valve could turn things around, but they probably won’t) but I thought I should put this out first and let people know. I’m the person who maintains the “ACAB good games list,” a guide to getting into Vupgrading a PC, and I’m just made a "How to use SteamVR" guide getting you familiar with all the settings and using your desktop in VR.]
The Valve Index is an AR Headset
In June, Valve released 3D passthrough for the Index. They actually paid another company to do it and in all likelihood they could have done it way way sooner. Either way, it means that, besides some warping around your hands, you have a view of the room you’re in that actually looks real. And as it turns out, there isn’t an actual reason why you couldn’t run a game or other software over that passthrough.
Basically you can judge for yourself. Go on steam, download a program called “Metachromium” (turn it off in your startup settings so it doesn’t launch on its own with steamVR), turn on your passthrough, and then run Metachromium. It’s a desktop UI so just use your desktop view on your dashboard to input any WebXR URL. Choose one without a background environment and MoonRider’s controls don’t work unfortunately. Any stuttering is the WebXR sites, not the AR.
Here are two sites you can try, just load them in Metachromium with your passthrough running and then click the VR button on the page. Bump up your headset brightness if needed. https://zach-geek.gitlab.io/vartiste/# https://whiteboard-xr.herokuapp.com/vr.html
Here is facebook’s own depiction of the best the Quest 2 can do. https://preview.redd.it/lflqne3gnnq51.jpg?width=400&format=pjpg&auto=webp&s=266863e0808374c0d89cbd6365b195a0f924c8dd
And this is what took me five minutes to figure out on my index (It looks better in-headset, especially everything arranging itself on my walls and the lighting)
https://preview.redd.it/fk3rojahnnq51.jpg?width=400&format=pjpg&auto=webp&s=2bc40b49f93266ef3bb6eab3527fca65bc146c66 https://preview.redd.it/khf0xruinnq51.jpg?width=400&format=pjpg&auto=webp&s=ef0c0f771f233e023a92d5b06968916052c40460 To take screenshots, you need to turn on SteamVR mirror, and then capture your monitor view. SteamVR screenshots are terrible and don't include any overlays, passthrough, or anything else
This is the absolute bare minimum of what is possible. This is what is possible with nothing officially supported or made to use with AR, and with super basic WebXR programs. You might not be all that impressed with passthrough AR but it’s a real thing. Apple is rumored to be making a passthrough AR headset, some french startup is making one for $1,500, and Facebook is adding a low resolution, black and white, “passthrough+” mode for AR next year on the Quest 2 for their own apps and then third party apps including work software “Spatial.”
Valve just isn’t going to make this functionality official apparently, technically they could have done it when the Index came out, since they just paid some other company to add 3D passthrough anyway. Over a year later, this is a really big deal and it needs to be made usable before Infinite Office and Quest 2 Passthrough AR comes out. Valve could flip a few switches, make a guide for Devs to take advantage of it, and then the Index is the first consumer AR headset and they can add an “Augmented Reality” tab to Steam. History made, at no additional cost to you.
And then there is Aardvark.
Aardvark is a community project by a Valve programmer and some open XR people like the creator of Pluto, a program that lets people hang out in VR without being in the same program. The idea behind Aardvark is that you can create AR apps inside of VR, from floating UI to spatial tools, and since it runs on top of VR programs it can have a lot more functionality. The same way that AR glasses will one day let you answer a phone call by waving your hand, it's envisioned as a way of allowing Devs to make AR in the here and now.
The difference between Aardvark and normal SteamVR overlays is that these can communicate with each other, meaning if someone makes a wrist mounted UI, someone else could make a discord integrated UI add on that slots right into it. And Aardvark overlays can communicate with another person’s, so you could share your screen, play poker, pick up a line of text and hand it to another person so they can paste it, etc.
It definitely has huge potential and if you’re a coder or a dev, you should definitely look into it. It’s on github and already works over passthrough even if it’s a bit stuttery. There are actually not a lot of things you couldn’t do with it in theory. And since none of these things need to be the dedicated focus of your time or the only overlay you’re running, it basically brings the app structure to VR where you download some QoL, UI, or system tweak for a dollar and can use it whenever you want. One of the bigger draws of the Quest is the idea that it’s a super smooth experience and the closed garden allows less friction than open PCVR can achieve. That is something that tools like Aardvark could completely reverse because they would allow a level of integration, resources, and third party applications that just aren't possible on mobile, and a level of customization, control, and pluggability that Facebook will never allow. We could get new features all the time from the community either as open source projects or paid add ons.
I myself submitted a bunch of ideas for Aardvark, and was told that nearly all of them were possible, but obviously they’re just concepts. Some examples:
- A rear warning system that tells you the exact object you’re about to walk into and where it is relative to you, making irregular boundaries more viable or letting you ignore things like walking into your couch. Could even allow you tracking a moving object if you have a Vive trackespare Vive wand.
- A full HOTAS that renders over a game, basically adding VR motion controls to any game even if it doesn’t support them (Squadrons/Sturmovik/MS Flight Simulator)
- Macropads that let you do work or stream in VR and have a macro UI that can do anything you want. You could have all the funcationality of a half dozen $200 macropads for free.
- An in VR gameboy that lets you play emulators or steam games with your VR controllers or a gamepad in the middle of a social VR app. This could even extend to playing multiplayer (screenshare plus hooking into Remote Play Together). Imagine you each sitting on your couch, seeing your friends next to you in AR, and playing on a display rendered over your TV.
- An input output organizer that lets you set up your own chains, like RSS feed photos popping up in SteamVR home, letting you record a message and have it tweeted out, control your room fan to match up with the level in your game
- Card games you can play with other people in AR on a real life table.
- A body based UI system based on the HEV suit that easily plugs in any and all add ons you have onto your arms or chest or wherever you want and be shifted to avoid covering up any in game UI.
- Replacing the entire SteamVR game launching system with a VAR bookcase where every game you have is rendered as a case on the shelf with the name on the spine, art on the cover, and Steam description on the back. You can arrange them however you want and save the layout and shelves you make.
- Universal avatars you can use across any software you want or even outside software with dedicated avatar systems.
- Passthrough Portals that let you mark out the couch as a 3D passthrough zone, so when you’re playing with friends in the room, you can always turn back and see them, so VR is basically no longer isolating if you don’t want it to be. You could also mark out your keyboard, mouse, even a glass of water so you easily work in VR. A person holding a Vive tracker could be in the game with you if you want.
- A controller assistance system that would let you put a friend in VR and use your phone or desktop (even over parsec/remote play together) to highlight buttons when you’re showing them what to do.
- A translator that listens to what you say, translates it, and shows it as text in front of you as you talk, showing you a reverse translation of what other people are hearing to make sure it’s not too bad.
- Metamatchmaking that lets you mark multiplayer games you want to play and then matching you up with anyone else who wants to play them and alerting all of you to start up the game and play.
- Replacements for the steamVR keyboard with one that actually works and supports other languages, as well as a virtual mouse that locks into a flat plane instead of a laser pointer
- Hand tracking through the Vive SDK, the Leap Motion, or emulated with Index controllers.
The idea I was most interested in was the idea of tracing out your room in 3D vector shapes, then making them invisible occlusion zones. Do it once in 30 minutes and then all your apps can use it from now on. That would allow real world objects to occlude virtual ones, a really key part of AR (occlusion in AR is like the transition from 3doF to 6doF in VR) that would allow for completely new experiences. It would also let you mark out the things you traced with context, so a character in a VR game could sit in a chair, walk through a door, etc or your friends could appear in AR sitting in the chair next you and the game you’re playing could appear on your real table. And since it’s preprogrammed and done through SteamVR tracking it could be much smoother than anything done with machine vision currently.
AR is the real prize of everything in XR; if VR is a billion dollar industry that will change gaming then AR is expected to be a trillion dollar industry that will change the world. Facebook only does VR to build a hardware, software, production, and dev base for AR. They want to beat Apple, who have the most advanced AR SDK of anyone and make their own silicon, and are reportedly working on AR glasses. Valve basically invented consumer AR back in 2013 and just didn’t ship it; future Facebook exec Micheal Abrash fired Valve’s whole AR division and so their head AR engineer Jeri Elsworth took her research and made her own startup, TiltFive.
Facebook’s main showcase of their AR right now is Infinite Office. It’s an app where you can have a black and white low res background view of the world, use a special keyboard you have to buy, use the trackpad on the keyboard since mouses aren’t supported, and you can control your browser since it doesn’t let you control your PC. That’s it. Valve could absolutely stomp that by giving you full color higher res passthrough, let you control your PC and actual keyboard and mouse, add virtual macropads, dashboards, and other things to help you work, multimonitor support, and even letting you work alongside someone else in AR and share screens. Third parties are getting all in on facebook passthrough, like remote work software Spatial where again, PC headsets are the only ones comfortable enough for working and allow for a lot more functionality (including occlusion and context awareness) and full color so it’s a huge waste for AR to not be supported on PC. Facebook is clearly hoping this will be a huge draw for the Quest (and I'm sure the VR outlets will really lay on the praise), push it into workplaces, and make it a devkit for AR. Valve can and should kill that in the cradle, even if killing the Quest 2 itself will take a lot more.
If you’re interested in deving AR on the index I guess you should look into WebXMetaChromium and especially Aardvark. If you’re a developer of a game like Cubism or Steady, where the environments aren’t important to the gameplay, there is a way to render over passthrough in Unity even without Valve’s official support. I would reach out to the team making Pluto and ask how to do it. If you're just a member of the community who knows how to code and wants to try making something cool, look into Aardvark and see if you can make something.
I hope Valve makes this all official. I can’t think of anything better right now than giving a set of huge new features to everyone who bought an Index. I hope they open it up to devs and offer the kind of support Facebook is offering and more. I hope they seed WebXR and Aardvark with some money to get things rolling. I hope they offer Jeri Ellsworth a boatload of money to handle their AR stuff or something and add compatibility with her TiltFive. I hope Valve makes SteamVR a general XR platform and uses AR as a way to explore that. I hope things like Aardvark can make PCVR a lot more usable both to increase how much people use PCVR and to create a new market for AR software.
They probably won’t, and community efforts won’t pick up enough momentum unless it can create some utility apps that people will pay for to get things moving. Valve is in a bubble where they think that because they’re doing a lot, or because they’ve done a lot, that they don’t need to escalate to compete with Facebook. They also seem to unfairly expect a lot from the community without offering much help, like with SteamVR Home modding, Alyx modding, and generally expecting us to market VR and Alyx for them. It’s also a dysfunctional nightmare inside that company, like the (ethical) foil to Zuck's dangerous dictatorship, and who knows how many people actually work on or care about VR, it could be a hundred or just ten incredibly productive people. Either way, if they don’t see the bigger picture the next two years (and beyond) could be a disaster for everyone.