okay, so i had some more time and actually tried to find out how to bring all the little hardware pieces together to form a working system. for the moment, it seems to me that the gaming engine unity3d is the best way to integrate most of the stuff i have although there are some nice things with point clouds i’d like to do with the asus xtion sensor (a derivation of primesense’s kinect that they developed together with microsoft).

but let’s start small, otherwise there’s just flat chaos. first of all, i’d like to show you what the current problem is and then describe how i got here. somehow i don’t find any information in the internet, so i think it’s worthwhile to write it up and get some people’s minds around it.

here’s what i built in about ten iterations with unity3d, leapmotion and the oculus rift:

here’s me manipulating the cubes with the leap motion on the table and the hmd on my head. i had to remove the leap motion from the hmd (see my last post taping it) during experimentation as i didn’t get any tracking first and later somehow the hands were mirrored, so i decided to simplify the setup and get it running on the table first.

here’s two things i learned in unity:

in the box (a prefab from the leap motion package), the rift camera has to be IN FRONT of the leap motion controller icon. the screenshot above shows the correct positioning from top view (leap motion icon is in the middle, right to the lamp icon). it may sound trivial but it took me two hours with many different settings (like the leap motion config menu also allows for changing orientations, etc).

second learning:

the z-axis must not be mirrored. secondly, i scaled the sandbox by 40, this allows you to move freely in the box and your hands seem to be roughly in proportion to the cubes (default size from unity).

one thing i still don’t know: how can i actually get a screen recording while i’m playing? this is unity’s output (ok, ok i did 11 iterations):

“direct to rift” works for me with my hmd, but on the pc screen there’s just a black window (as in my second video above), without “direct to rift” i see a live window, but the hmd isn’t showing anything. of course, the hmd is then used as a second screen on the pc, and i could move the window to that screen but then it won’t be full screen. and the second problem is that the software actually captures the mouse for controlling the navigation in the virtual space, so i can’t move the window as soon as i the animation actually starts… any ideas? haven’t found anything in the unity “build settings”.

here’s the software that i used:

• unity 4.5.5: http://unity3d.com/unity/download
• from unity’s asset store, you need the leapmotion skeletal tracking (v2): https://www.assetstore.unity3d.com/en/#!/search/leapmotion
• on oculus, you need the windows runtime and the unity integration pack: https://developer.oculusvr.com/?action=dl
• from leap motion, you need the latest SDK: https://developer.leapmotion.com/

i was inspired to do the cubes in the box by this video that is a brief tutorial into unity3d and leap motion integration (but has no oculus rift integration).

there’s a second very inspiring video that shows how to get unity running and install the oculus rift package for it (but no leap motion integration):

actually, a couple of things are easier now as you simply can register after downloading unity3d right at first start.

okay, so i’ve tried the device on a couple of family members and since the virtual desktop example is fast enough for my current little computer, it was a great experience for everybody.

it seems though that the head tracking in the example is just for turning your head but somehow it’s not tracking your relative height, i.e. looking under the table somehow doesn’t work. Continue reading →

okay, this is a new era. let’s face it. ever since william gibson’s neuromancer (the guy who invented the word cyberspace), i’ve been waiting to dive in into data. in the mid nineties, i bought myself a cybermaxx helmet. and yes, i recently Continue reading →

YEAH! can’t wait to hook up the oculus rift dk2 up to my wii fit board. i think that navigation in the virtual space via finger, keyboard, mouse, wii nunchuck or razor hydra etc is sub optimal. you have to have your hands free to do things that hands do in the real space. walking is obviously not part of that…

so i would like to stand on the board and read out the pressure sensors of the wii fit board to accelerate ahead to steer left / right just by moving my body similar to using a skateboard.

and the leap motion. it is of outmost importance to use your hands in the space and leap motion did a great pivot recently when adapting their SDK to the oculus rift, i.e. adding support for finding your hands even if the leap motion controller is not on the table but attached to your helmet. can’t wait to see this. plus the leap motion controller / sdk has built-in support for gesture recognition so that you can easily implement a couple of basic interactions with objects in the virtual space.

and my two asus xtions, the guy below uses three kinects. it will be a great thing to fuse the real world and the virtual world. i’d love to track my body via openni and / or see myself in the virtual space. the rift’s head tracking and possibly the xtion’s spatial position tracking will allow me to walk in space but also give real out-of-body experiences in the helmet. soo cool, check the video out below:

BRING. IT. OOOOON!