okay, so i had some more time and actually tried to find out how to bring all the little hardware pieces together to form a working system. for the moment, it seems to me that the gaming engine unity3d is the best way to integrate most of the stuff i have although there are some nice things with point clouds i’d like to do with the asus xtion sensor (a derivation of primesense’s kinect that they developed together with microsoft).

but let’s start small, otherwise there’s just flat chaos. first of all, i’d like to show you what the current problem is and then describe how i got here. somehow i don’t find any information in the internet, so i think it’s worthwhile to write it up and get some people’s minds around it.

here’s what i built in about ten iterations with unity3d, leapmotion and the oculus rift:

 

here’s me manipulating the cubes with the leap motion on the table and the hmd on my head. i had to remove the leap motion from the hmd (see my last post taping it) during experimentation as i didn’t get any tracking first and later somehow the hands were mirrored, so i decided to simplify the setup and get it running on the table first.

here’s two things i learned in unity:

in the box (a prefab from the leap motion package), the rift camera has to be IN FRONT of the leap motion controller icon. the screenshot above shows the correct positioning from top view (leap motion icon is in the middle, right to the lamp icon). it may sound trivial but it took me two hours with many different settings (like the leap motion config menu also allows for changing orientations, etc).

second learning:

the z-axis must not be mirrored. secondly, i scaled the sandbox by 40, this allows you to move freely in the box and your hands seem to be roughly in proportion to the cubes (default size from unity).

one thing i still don’t know: how can i actually get a screen recording while i’m playing? this is unity’s output (ok, ok i did 11 iterations):

“direct to rift” works for me with my hmd, but on the pc screen there’s just a black window (as in my second video above), without “direct to rift” i see a live window, but the hmd isn’t showing anything. of course, the hmd is then used as a second screen on the pc, and i could move the window to that screen but then it won’t be full screen. and the second problem is that the software actually captures the mouse for controlling the navigation in the virtual space, so i can’t move the window as soon as i the animation actually starts… any ideas? haven’t found anything in the unity “build settings”.

here’s the software that i used:

• unity 4.5.5: http://unity3d.com/unity/download
• from unity’s asset store, you need the leapmotion skeletal tracking (v2): https://www.assetstore.unity3d.com/en/#!/search/leapmotion
• on oculus, you need the windows runtime and the unity integration pack: https://developer.oculusvr.com/?action=dl
• from leap motion, you need the latest SDK: https://developer.leapmotion.com/

i was inspired to do the cubes in the box by this video that is a brief tutorial into unity3d and leap motion integration (but has no oculus rift integration).

 

there’s a second very inspiring video that shows how to get unity running and install the oculus rift package for it (but no leap motion integration):

actually, a couple of things are easier now as you simply can register after downloading unity3d right at first start.