Mixed reality proof of concept and new Virtual Desktop with passthrough mask
-
Hi,
The last couple of weeks I’ve been working on a POC to test mixed reality without having to sell my body to buy XR-3.
The result is this one:
And this one in DCS =>
But today Virtual Desktop has released a Beta to have passthrough mask => https://github.com/guygodin/VirtualDesktop/releases/tag/v1.29.0
I was wondering if someone with experience in 3d moding could help me to achieve the following in the viper 3d cockpit model
So we can test how is the performance in Virtual Desktop using passthough mask and Falcon BMS.
Any help would be really appreciated
Kind regards!
-
I’ve done it myself in a fast and dirty way.
I’ve done a super fast test just painting 80XX DDS textures to pink, nothing fancy, but I confirm that it works QUITE well. Latency is minimal. Still there, but almost un-noticeable
We just need to modify cockpit model a little bit, to be all pink. Anybody could help?
I think this is the way to go for mixed reality, even more with upcoming headsets with dual RGB cameras (pico 5 is supposed to have dual 16 megapixel cameras)
What do you think about this?
-
Interesting!!
-
I’ve tested with Virtual Desktop + Passthrough mask and I think it’s the way to go.
All I’ve done is set the mask to black… For best results we would have to re-do the tub cockpit model and add a more different mask (like pink or similar).
I’ve also used an old tablet with Falcon BMS Control to check MFDs clarity and it’s quite ok, much better than I expected.
My conclusions are:
- We are really close to have something usable with mixed reality at an affordable price
- Tomorrow Quest 3 is being presented. I’ve done the tests with Pico 4, they have similar resolution of 4 megapixel in the RGB camera(s), but it’s a beginning.
- In February 2024 Pico 5 is supposed to be presented with two 16 megapixel RGB cameras for passthrough, I think it will be totally fine for seeing MFDs and all instruments through the cameras. https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F8oo2pshfcmpb1.jpg
From my point of view, overall the experience of this Virtual Desktop is fine. Latency is minimal. No motion sickness at all. I only wonder how to calibrate the virtual world with the real cockpit, but I imagine it will be just resetting the view a few times until they match.
This is a duplicate post from viperpits.org, but I’ve considered to leave this message here too to give you an update. Thanks go to jjbravo and Korbi for the idea of using the tub view
Kind regards everyone!
-
These are great experiments! Thank you for keeping us informed on your progress. I’ll be very interested to follow this.
-
What the, what the…?
What am I seeing here…!?
Can you provide context for the “senior” generation.What is this witchcraft…?!
I can see YOU and your desk and smartphone, etc. and the game “around” you.
But - WHERE is that 3D world…? Are you wearing a VR helmet…? But - you can still see out of it to use keyboard and see HOTAS buttons and physical MFD’s.
Then… sometimes I can see YOU. Like it is YOU in the cockpit. Right…?
This is NOT VR…? What is this…? Where is that 3D world…? NOT a monitor, right…?
What are the practical applications of this witchcraft…?Is it to put your actual SELF inside the 'Pit (instead of the 3D model pilot)
OR
Is it to be able to SEE “under the visor” style without cutting your VR headset
OR
Is it BOTH of these
OR
Is it NONE of these…?I feel old.
Please explain, because the Thread Title just grabbed me immediately.
Cheers for the pending education.
SeniorGorn.
EDIT: Oh, yeah. What is a “see through mask” and what is a “virtual desktop”…?
-
A few weeks back I started to do a proof of concept in C++ that captured the SteamVR DirectX rendered image, then applied a mask built from a basic wavefront 3d model and mixed it with passthrough.
I wrote about it a couple of days back in viperpits to share the results. My idea was to use some external “high quality” cams to have a better resolution for passthrough (for seeing MFDs, ICP, …). I also tested with the headset’s own cameras, but it had some synchronization problems between passthrough and rendered world. This could be fixable with a little bit of work but then…
Yesterday evening a new Beta version came out for Virtual Desktop, that supports passthrough mask, this means that you choose a mask color and everything that is rendered in the 3d world with that color will be shown directly thru headset RGB cameras. This is the way to go.
The only modification that we would have to do in BMS is adding some kind of model for the tub pit view with the front, left, right consoles in pink. Something like the concept I’ve shown above in pink. The hud would have to be modeled.
This way we would be able to see the virtual world outside of the cockpit, and a physical cockpit with our own hand and everything inside. Just like the video that you’ve probably already seen, but without the need to spend a few thousand dollars to get the Varjo XR-3
From my point of view this is the future of home pits! And we are close to achieving it!
-
Ah…!
So - the desk and the hands and the cellphone and stuff I was seeing was NOT being seen with the user’s OWN eyes.
I thought THAT was a “real life” view, and only the Falcon stuff was being seen in a helmet… er… somehow.
But that real-life stuff was actually being seen INSIDE the VR helmet by using a camera “built in” to the helmet…?
How would that work with - for example - my G2 Reverb that (AFAIK) doesn’t have a camera…?
Would it only work with headsets that have built-in external cameras…?(Or, am I still completely misunderstanding?)
-
@Aragorn in a nutshell, what you see in the video is a recording from inside his VR set.
Some VR headsets have cameras that allow you to film the real environment and show that in your headset.
Virtual Desktop is an app you run on your pc and your VR headset to stream your desktop or game to your VR set.
Since an update of VD, it allows to set a ‘mask’ for pass through. Compare it with a green screen for video effects. VD will stream the game/pc graphics by default (in this case, BMS in VR), and everything that has a certain defined color (green, pink, whatever you set), it will not show the game on that location but will pass through the video image the cameras on you headset capture.
That way you create an immersive mixed reality. You see everything in your VR headset but what you see is a mix of pc/game footage and surroundings camera footage. -
@Focaldesign said in Mixed reality proof of concept and new Virtual Desktop with passthrough mask:
@Aragorn in a nutshell, what you see in the video is a recording from inside his VR set.
Some VR headsets have cameras that allow you to film the real environment and show that in your headset.
Virtual Desktop is an app you run on your pc and your VR headset to stream your desktop or game to your VR set.
Since an update of VD, it allows to set a ‘mask’ for pass through. Compare it with a green screen for video effects. VD will stream the game/pc graphics by default (in this case, BMS in VR), and everything that has a certain defined color (green, pink, whatever you set), it will not show the game on that location but will pass through the video image the cameras on you headset capture.
That way you create an immersive mixed reality. You see everything in your VR headset but what you see is a mix of pc/game footage and surroundings camera footage.Thanks for such a clear explanation!!
I couldn’t have explained it better
-
@Focaldesign GOT IT…!!
Wow.
That is amazing.
That Varjo thing looks… amazing.
So - it knows to only show the camera view when you are looking DOWN or in the right position.
When he is looking up in the air, the external camera isn’t working.
Or - like the green screen is ONLY in the 'pit, yeah.
Wow. Just… wow.
-
The whole pitbuilder community is watching you for sure.
Good work and progress. Keep going! -
I think I get it now. So when using a VR headset, you would usually not be able to see your desk or keyboard or your full physical pit if you had one.
So if you wanted to press keys on the keyboard you would have to fumble around blindly, or briefly peek “under” the headset. Now you can let the VR headset show you your surroundings thru its cameras and that’s called “passthrough” - is that about right ?
-
@jayb Exactly, It’s super handy for those with physical pit build. When you can see both 3D world and your cockpit in headset.
For those still using keyboard reliable handtracking would be the thing. -
Exactly, the whole goal of this mixed reality is for pitbuilder guys!
With good RGB passthrough cameras coming in the next few months I think the inmersion will be total!!
IMHO: For using keyboard in VR and so on I’m not sure if it’s really worth it
-
WOW! This is a jaw dropping concept - way to go!! I don’t do BMS in VR (yet) but gonna definitely check this out when I get a VR with outward cams… and build more desktop cockpit controls… I’m still on Oculus Rift CV1 with sensor masts lol
Keep up the great work!!
-
This is why I’m building a pit! I thought I wouldn’t see this for a couple years. Great work!
-
Hi!
This is what I am waiting for more than 20 years. WOOOOOWWWW!!!
Definitely the right way for fly experience in home built cockpit.I will be watching this thread for any progress.
Br
Borut -
@oleuzop "From my point of view this is the future of home pits! And we are close to achieving it!
this is the most interesting
-
@oleuzop How are you creating the actual passthrough mask in DCS? Do you modify a .lua file? How do you assign the correct color? I understand how to use Virtual Desktop to implement the passthrough, but am uncertain how to create the mask itself.