Mixed reality proof of concept and new Virtual Desktop with passthrough mask
-
Ah…!
So - the desk and the hands and the cellphone and stuff I was seeing was NOT being seen with the user’s OWN eyes.
I thought THAT was a “real life” view, and only the Falcon stuff was being seen in a helmet… er… somehow.
But that real-life stuff was actually being seen INSIDE the VR helmet by using a camera “built in” to the helmet…?
How would that work with - for example - my G2 Reverb that (AFAIK) doesn’t have a camera…?
Would it only work with headsets that have built-in external cameras…?(Or, am I still completely misunderstanding?)
-
@Aragorn in a nutshell, what you see in the video is a recording from inside his VR set.
Some VR headsets have cameras that allow you to film the real environment and show that in your headset.
Virtual Desktop is an app you run on your pc and your VR headset to stream your desktop or game to your VR set.
Since an update of VD, it allows to set a ‘mask’ for pass through. Compare it with a green screen for video effects. VD will stream the game/pc graphics by default (in this case, BMS in VR), and everything that has a certain defined color (green, pink, whatever you set), it will not show the game on that location but will pass through the video image the cameras on you headset capture.
That way you create an immersive mixed reality. You see everything in your VR headset but what you see is a mix of pc/game footage and surroundings camera footage. -
@Focaldesign said in Mixed reality proof of concept and new Virtual Desktop with passthrough mask:
@Aragorn in a nutshell, what you see in the video is a recording from inside his VR set.
Some VR headsets have cameras that allow you to film the real environment and show that in your headset.
Virtual Desktop is an app you run on your pc and your VR headset to stream your desktop or game to your VR set.
Since an update of VD, it allows to set a ‘mask’ for pass through. Compare it with a green screen for video effects. VD will stream the game/pc graphics by default (in this case, BMS in VR), and everything that has a certain defined color (green, pink, whatever you set), it will not show the game on that location but will pass through the video image the cameras on you headset capture.
That way you create an immersive mixed reality. You see everything in your VR headset but what you see is a mix of pc/game footage and surroundings camera footage.Thanks for such a clear explanation!!
I couldn’t have explained it better
-
@Focaldesign GOT IT…!!
Wow.
That is amazing.
That Varjo thing looks… amazing.
So - it knows to only show the camera view when you are looking DOWN or in the right position.
When he is looking up in the air, the external camera isn’t working.
Or - like the green screen is ONLY in the 'pit, yeah.
Wow. Just… wow.
-
The whole pitbuilder community is watching you for sure.
Good work and progress. Keep going! -
I think I get it now. So when using a VR headset, you would usually not be able to see your desk or keyboard or your full physical pit if you had one.
So if you wanted to press keys on the keyboard you would have to fumble around blindly, or briefly peek “under” the headset. Now you can let the VR headset show you your surroundings thru its cameras and that’s called “passthrough” - is that about right ?
-
@jayb Exactly, It’s super handy for those with physical pit build. When you can see both 3D world and your cockpit in headset.
For those still using keyboard reliable handtracking would be the thing. -
Exactly, the whole goal of this mixed reality is for pitbuilder guys!
With good RGB passthrough cameras coming in the next few months I think the inmersion will be total!!
IMHO: For using keyboard in VR and so on I’m not sure if it’s really worth it
-
WOW! This is a jaw dropping concept - way to go!! I don’t do BMS in VR (yet) but gonna definitely check this out when I get a VR with outward cams… and build more desktop cockpit controls… I’m still on Oculus Rift CV1 with sensor masts lol
Keep up the great work!!
-
This is why I’m building a pit! I thought I wouldn’t see this for a couple years. Great work!
-
Hi!
This is what I am waiting for more than 20 years. WOOOOOWWWW!!!
Definitely the right way for fly experience in home built cockpit.I will be watching this thread for any progress.
Br
Borut -
@oleuzop "From my point of view this is the future of home pits! And we are close to achieving it!
this is the most interesting
-
@oleuzop How are you creating the actual passthrough mask in DCS? Do you modify a .lua file? How do you assign the correct color? I understand how to use Virtual Desktop to implement the passthrough, but am uncertain how to create the mask itself.
-
@FlyDoc said in Mixed reality proof of concept and new Virtual Desktop with passthrough mask:
@oleuzop How are you creating the actual passthrough mask in DCS? Do you modify a .lua file? How do you assign the correct color? I understand how to use Virtual Desktop to implement the passthrough, but am uncertain how to create the mask itself.
This was a POC with an external app to have a 3d model mask to be on a layer above the rendering and showing into the headset.
When VD was out with Passthrough it was outdated, so I stopped developing the app.
I think it’s better to ask directly into DCS forums for a solution for this simulator to use VD passthrough.
-
See this guy’s Youtube Channel. He has built an entire cockpit from scratch.
https://www.youtube.com/@RainmanHST
He has already made the VR mask with pass-through using a Quest3 and he will be doing a how-to video soon.
Here is a Youtube shorts of his cockpit in VR and pass-through.
-
@zap737 Yeap, that’s a really well known member of the BMS community
AR is comming in 4.37.4 as stated in this post:
https://forum.falcon-bms.com/post/385316
Kind regards!