Mixed reality proof of concept and new Virtual Desktop with passthrough mask
-
What the, what the…?
What am I seeing here…!?
Can you provide context for the “senior” generation.What is this witchcraft…?!
I can see YOU and your desk and smartphone, etc. and the game “around” you.
But - WHERE is that 3D world…? Are you wearing a VR helmet…? But - you can still see out of it to use keyboard and see HOTAS buttons and physical MFD’s.
Then… sometimes I can see YOU. Like it is YOU in the cockpit. Right…?
This is NOT VR…? What is this…? Where is that 3D world…? NOT a monitor, right…?
What are the practical applications of this witchcraft…?Is it to put your actual SELF inside the 'Pit (instead of the 3D model pilot)
OR
Is it to be able to SEE “under the visor” style without cutting your VR headset
OR
Is it BOTH of these
OR
Is it NONE of these…?I feel old.
Please explain, because the Thread Title just grabbed me immediately.
Cheers for the pending education.
SeniorGorn.
EDIT: Oh, yeah. What is a “see through mask” and what is a “virtual desktop”…?
-
A few weeks back I started to do a proof of concept in C++ that captured the SteamVR DirectX rendered image, then applied a mask built from a basic wavefront 3d model and mixed it with passthrough.
I wrote about it a couple of days back in viperpits to share the results. My idea was to use some external “high quality” cams to have a better resolution for passthrough (for seeing MFDs, ICP, …). I also tested with the headset’s own cameras, but it had some synchronization problems between passthrough and rendered world. This could be fixable with a little bit of work but then…
Yesterday evening a new Beta version came out for Virtual Desktop, that supports passthrough mask, this means that you choose a mask color and everything that is rendered in the 3d world with that color will be shown directly thru headset RGB cameras. This is the way to go.
The only modification that we would have to do in BMS is adding some kind of model for the tub pit view with the front, left, right consoles in pink. Something like the concept I’ve shown above in pink. The hud would have to be modeled.
This way we would be able to see the virtual world outside of the cockpit, and a physical cockpit with our own hand and everything inside. Just like the video that you’ve probably already seen, but without the need to spend a few thousand dollars to get the Varjo XR-3
From my point of view this is the future of home pits! And we are close to achieving it!
-
Ah…!
So - the desk and the hands and the cellphone and stuff I was seeing was NOT being seen with the user’s OWN eyes.
I thought THAT was a “real life” view, and only the Falcon stuff was being seen in a helmet… er… somehow.
But that real-life stuff was actually being seen INSIDE the VR helmet by using a camera “built in” to the helmet…?
How would that work with - for example - my G2 Reverb that (AFAIK) doesn’t have a camera…?
Would it only work with headsets that have built-in external cameras…?(Or, am I still completely misunderstanding?)
-
@Aragorn in a nutshell, what you see in the video is a recording from inside his VR set.
Some VR headsets have cameras that allow you to film the real environment and show that in your headset.
Virtual Desktop is an app you run on your pc and your VR headset to stream your desktop or game to your VR set.
Since an update of VD, it allows to set a ‘mask’ for pass through. Compare it with a green screen for video effects. VD will stream the game/pc graphics by default (in this case, BMS in VR), and everything that has a certain defined color (green, pink, whatever you set), it will not show the game on that location but will pass through the video image the cameras on you headset capture.
That way you create an immersive mixed reality. You see everything in your VR headset but what you see is a mix of pc/game footage and surroundings camera footage. -
@Focaldesign said in Mixed reality proof of concept and new Virtual Desktop with passthrough mask:
@Aragorn in a nutshell, what you see in the video is a recording from inside his VR set.
Some VR headsets have cameras that allow you to film the real environment and show that in your headset.
Virtual Desktop is an app you run on your pc and your VR headset to stream your desktop or game to your VR set.
Since an update of VD, it allows to set a ‘mask’ for pass through. Compare it with a green screen for video effects. VD will stream the game/pc graphics by default (in this case, BMS in VR), and everything that has a certain defined color (green, pink, whatever you set), it will not show the game on that location but will pass through the video image the cameras on you headset capture.
That way you create an immersive mixed reality. You see everything in your VR headset but what you see is a mix of pc/game footage and surroundings camera footage.Thanks for such a clear explanation!!
I couldn’t have explained it better
-
@Focaldesign GOT IT…!!
Wow.
That is amazing.
That Varjo thing looks… amazing.
So - it knows to only show the camera view when you are looking DOWN or in the right position.
When he is looking up in the air, the external camera isn’t working.
Or - like the green screen is ONLY in the 'pit, yeah.
Wow. Just… wow.
-
The whole pitbuilder community is watching you for sure.
Good work and progress. Keep going! -
I think I get it now. So when using a VR headset, you would usually not be able to see your desk or keyboard or your full physical pit if you had one.
So if you wanted to press keys on the keyboard you would have to fumble around blindly, or briefly peek “under” the headset. Now you can let the VR headset show you your surroundings thru its cameras and that’s called “passthrough” - is that about right ?
-
@jayb Exactly, It’s super handy for those with physical pit build. When you can see both 3D world and your cockpit in headset.
For those still using keyboard reliable handtracking would be the thing. -
Exactly, the whole goal of this mixed reality is for pitbuilder guys!
With good RGB passthrough cameras coming in the next few months I think the inmersion will be total!!
IMHO: For using keyboard in VR and so on I’m not sure if it’s really worth it
-
WOW! This is a jaw dropping concept - way to go!! I don’t do BMS in VR (yet) but gonna definitely check this out when I get a VR with outward cams… and build more desktop cockpit controls… I’m still on Oculus Rift CV1 with sensor masts lol
Keep up the great work!!
-
This is why I’m building a pit! I thought I wouldn’t see this for a couple years. Great work!
-
Hi!
This is what I am waiting for more than 20 years. WOOOOOWWWW!!!
Definitely the right way for fly experience in home built cockpit.I will be watching this thread for any progress.
Br
Borut -
@oleuzop "From my point of view this is the future of home pits! And we are close to achieving it!
this is the most interesting
-
@oleuzop How are you creating the actual passthrough mask in DCS? Do you modify a .lua file? How do you assign the correct color? I understand how to use Virtual Desktop to implement the passthrough, but am uncertain how to create the mask itself.
-
@FlyDoc said in Mixed reality proof of concept and new Virtual Desktop with passthrough mask:
@oleuzop How are you creating the actual passthrough mask in DCS? Do you modify a .lua file? How do you assign the correct color? I understand how to use Virtual Desktop to implement the passthrough, but am uncertain how to create the mask itself.
This was a POC with an external app to have a 3d model mask to be on a layer above the rendering and showing into the headset.
When VD was out with Passthrough it was outdated, so I stopped developing the app.
I think it’s better to ask directly into DCS forums for a solution for this simulator to use VD passthrough.
-
See this guy’s Youtube Channel. He has built an entire cockpit from scratch.
https://www.youtube.com/@RainmanHST
He has already made the VR mask with pass-through using a Quest3 and he will be doing a how-to video soon.
Here is a Youtube shorts of his cockpit in VR and pass-through.
-
@zap737 Yeap, that’s a really well known member of the BMS community
AR is comming in 4.37.4 as stated in this post:
https://forum.falcon-bms.com/post/385316
Kind regards!