Category Archives: virtual reality

All things related to VR, Oculus Rift, Unity 3d, etc.

locomotion in VR

I just came across this video on Facebook:

and found the original on youtube:

The original video is from 2009, we had the Computer Vision breakthrough in 2012 with Deep Learning. All the new technologies understanding human body posture and movements have been developed further since then. See for example this research from ETH Zurich predicting body movements:

Also, moving tiles are in production already at huge warehouses:


I was a supporter in the kickstarter campain of the Omni by Virtuix, but didn’t get one as the machine is so heavy that the developers decided to only ship it to professional companies, not private persons and/or also not internationally. The Omni is a treadmill:

Treadmills try to keep you in place physically, they are not really good at sensing where you are. In contrast, the tiles concept above is actually working around you as it has to understand where you are going. It is a real interface between the virtual environment (as it has to know what comes next) and your movements (and thus has to track you decently). It may be clumsy (what if the person is running?) and inefficient right now, but in my eyes, it has all the potential to make you feel running as you like and feeling the environment.


Improvemnts necessary:

  • It has to become smarter in seeing and predicting the body movements
  • it has to become faster, maybe even smaller then.
  • possibly the plates have to tilt in directions to emulate environments better (going up a hill).

I would love to see this concept advancing. That said, I’m still in love with the bionic chair by Govert Flint as it would work in constrained spaces, i.e. for consumer use.


Wireless streaming stereo video from an RC rover to the Oculus Pi

The next iteration was of course to simply try out the wireless transmission of video. And this doesn’t make sense if you still have the camera attached to yourself, so I “augmented” another project of mine, the rover:

Arduino / RC mixed autonomy Rover

Arduino / RC mixed autonomy Rover

Also I did a little self-sufficient stereo video transmission pack, including the Blackbird 2 camera:

Battery, video sender, stereo camera

Battery, video sender, stereo camera

And attached the two. I love modular designs where you can recombine your projects easily. So these two simply have two batteries etc. Here’s the full setup:

Oculus Pi wireless streaming video rover setup

Oculus Pi wireless streaming video rover setup

Here’s the video:

(Don’t know what the heck Pinnacle Studio was thinking to put the black frame around the video when exporting.)

Strange seems that we seem to get some kind of “interlace distortion” when the car is moving too quickly. I don’t quie know whether that is an artefact from the analog video transmission or from the digitisation process itself:

interlace artifactsWhen playing with this, you intuitively move your head to look around. So I could read out the head movements and send them to the Arduino on the rover to actually move the camera. I’d also need to add another servo to the camera to actually be able to move along at least two axis’. Let’s see…


2 3D cams & 1 2D cam on the DK2 @ Pi3

Cryptic title of a blog post, I know. But I urgently needed to try out my cameras with the Pi3 for the use on the Oculus Rift DK2 (video below). My last attempt to stream video locally into the Rift was successfull, so I wanted more. The initial single 2D camera was a logitech webcam C525 (for 60$)

logitech hd webcam c525

logitech hd webcam c525

I have two stereoscopic cameras. The Minoru which is “kind of” a cheap camera at about 70€, but then again: not, because it’s just 2x 640×480 resolution… I stripped the camera to reduce its size, weight an volume to actually use it on the Trashbot since I know that it is supported by the Raspbian as a camera.

Minoru Steroscopic Cam

Minoru Steroscopic Cam, around 70€

I also experimented already with the Blackbird 2, an analog camera for streaming video from drones to video goggles. However, that attempt was not really successful since the camera software on PC (!) was laggy and I was thinking that it may be due to the video capture card Easycap (10-20€).

BlackBird 2

BlackBird 2, around 180 USD

But let’s see, what happens when I run it on the Pi 3:

Learning: the 190€ combo beats the other two BY FAR in experience.

PS: In the video, you see that I attached the cameras to a different Pi than the one attached to the Oculus. This is because, the Oculus Pi has display settings tuned towards the goggles and I need to invest time to make the config changeable via software to actually switch between the googles and a real external display.

cam fits nicely the cables of the Rift and enough space on the straps to lead cables.

Oculus Pi untethered, pt. 2: a camera!

Last time, I was able to get the Oculus Rift DK2 to run on the Raspberry Pi 3, including the head tracking. However, the first interactions showed that it’s cumbersome to work with the desktop (since it’s not distributed on the two eyes but really is using the LCD as one screen) and also to use the keyboard.

Also, in the context of making the Oculus mobile and untethered, it is necessary to have a camera onboard, at least, until I get the XTtion to work. It is interesting that it is not too easy to find software that simply can display a video stream from a local webcam, most blogs just describe how to stream from a remote webcam or make a local USB cam accessible via some webservice.

My last attempt to get a stereo analog cam to work was not really cool since the latencies on a PC plus some weird display software as an .exe were not the ideal setting to really improve things.

So after some research, I found a git repository for streaming a local webcam to a dedicated  view, independent from the desktop. You need to install CMake and libbsd-dev to make it run via:

sudo apt-get install libbsd-dev
sudo apt-get install cmake

Also, I was able to install the camera in a nice position without additional mechanics:

cam fits nicely the cables of the Rift and enough space on the straps to lead cables.

cam fits nicely the cables of the Rift and enough space on the straps to lead cables.

Here’s the video walk through with some live feed to see the latency:

Next, I may try to either position the video on one eye, or even double the stream to both eyes. Or I may try to use the Minoru Stereo cam that I’ve been working on last year.



Oculus Rift DK2 on a Raspberry Pi3: towards untethered VR.

I’ve been long dreaming of connecting these to important technologies and run them in way that I actually understand. I’m not good at Android programming (for those cardbox VRs) and not really good at Unity (although that would be fun to dive deeper) and somehow my driver situation with the DK2 and windows 10 deteriorated.

There have been earlier attempts to run the Oculus on a Pi, but in my eyes the latest iteration I found is the most fitting to me (including Python bindings and 3d libraries).

So recently, I came across Wayne Keenan’s blog how actually did exactly what I’m dreaming of, namely to run the DK2 controlled by the Pi3. I’m not so much interested in maximising the complexity of geometry in VR but really more in the interaction design and what untethered VR (or even AR) can actually feel like.

His github recipe really works nicely, and so the installation was really a matter of 30min:

Pi3, DK2 and mini keyboard

Continue reading

Blackbird 2, 3D FPV camera and the Oculus Rift DK2, pt. 1

Slowly inching towards the Blackbird2 on the Oculus Rift DK2.

Right now, I have the ancient Logilink video capture card running on Windows 10 and found a software that is old enough to be compatible with it but young enough to run on Windows 10.

It installed the DirectPlay on Windows10, so that’s the tribute to old tech, I guess.

So I get the video on my desktop now, have updated the Nvidia driver 358.87 for my GTX980 and the lastest Oculus Rift driver (v8.0).

What we get is here:

blackbird video on PC

As you can see, it’s black and white somehow. The capture says it sees 525 (PAL60) or 625 (PAL_N), with the latter setting, I also get some sort of colors but they change rapidly and are mostly false…

I can also put the video on full screen, but it seems that with the latest drivers for the Oculus, I can’t use the DK2 as second output display, so I don’t know how to put the camera video into the DK2. Clearly, more research needed…

Blackbird 2 FPV 3D camera unboxing

I’ve been dreaming of building a remote controlled 3D camera rig for the Oculus Rift for over a year now. Recently, I came across the Blackbird 3D First Person View camera here in a forum. And then I learned that the producer of this nice thing actually just launched the version 2 which is natively capable of producing images suitable for the Oculus Rift by rotating the videos glueing them next to each other and distorting them adequately. And I had to buy it right away. That was about three weeks ago and finally, today I received it.

It thought I just share the unboxing of it with you, I hope I’ll have time to play with it (the rest of the hardware is on my desk already: an analog display, wireless transmission, USB video digitisation and a 2 DoF servo gimbal to be remote controlled). Yay!

You can see some similar projects here. Continue reading

make munich 14, pt 2: oculus rift und kamera

eins der projekte, die ich auch gerne mal basteln wollen würde ist eine kamera führung, die mit drei servos entlang drei achsen geschieht. die servos werden durch die kopf orientierungsdaten der oculus rift gesteuert, die aufnahme der kamera kommt in der rift an.

im vorliegenden projekt wurden dynamixel servos genommen, ca 30€ das stück, die seriell vernetzt werden, was die komplexität verringert. allerdings war nur eine 2d kamera drauf bzw nur ein kamera chip, nicht zwei, sodass wohl im rift kein echter 3d eindruck entstehen konnte.

darüberhinaus war dazwischen ein relativ langsamer laptop, sodass die latenzen nicht wirklich überzeugt haben dürften. schade eigentlich, denn latenzen sind meines erachtens wesentlich wichtiger für einen lebendigen eindruck in der virtuellen realität.

hier das video dazu:


der mensch hier hat das etwas ernsthafter betrieben, nur zum vergleich:


concepts for moving in virtual space yet safe in real world.

we all know the onmi by virtuix that allows for walking / running in the virtual space while remaining on one spot in the real world. i’ve backed this project (i hope they will deliver soon) as it seems that the user experience is conceptually well executed through special shoes that can be tracked by the floor:

but just today i came across a “chair” that actually frees your body from the floor, i.e. it allows for moving more freely in the virtual space while the physical device is actually home-compatible compared to “lawn mower man” kind of interfaces. govert flint did this as a graduation project at the design school in eindhoven:

it seems that it has different ways of collecting data, i’ve read about accelerometers, probably even potentiometers would probably do the job. then, they added some software to actually recognise gestures. i think we should go further than that and really use the whole body posture. in that way, we could use the kinect to record the posture and simplify the design of the chair. would be great to try that out (i do love the possibility to swing left and right, that’s going beyond a bar stool. actually, that guy should do a kickstarter project.

btw: the original lawnmower man interface is akin to a gyroskope:

but that’s clearly not consumer compatible. so i’m sold on that chair thingy.

then there’s the züricher hochschule für künste who allow the user to lie on “bed” and use wings attached to the arms to fly like a bird. also quite inspiring, but again, in terms of practicality, not for me at home:



(oculus rift + leap motion) * unity3d = fun^2

okay, so i had some more time and actually tried to find out how to bring all the little hardware pieces together to form a working system. for the moment, it seems to me that the gaming engine unity3d is the best way to integrate most of the stuff i have although there are some nice things with point clouds i’d like to do with the asus xtion sensor (a derivation of primesense’s kinect that they developed together with microsoft).

but let’s start small, otherwise there’s just flat chaos. first of all, i’d like to show you what the current problem is and then describe how i got here. somehow i don’t find any information in the internet, so i think it’s worthwhile to write it up and get some people’s minds around it.

here’s what i built in about ten iterations with unity3d, leapmotion and the oculus rift:


here’s me manipulating the cubes with the leap motion on the table and the hmd on my head. i had to remove the leap motion from the hmd (see my last post taping it) during experimentation as i didn’t get any tracking first and later somehow the hands were mirrored, so i decided to simplify the setup and get it running on the table first.

here’s two things i learned in unity:

oculus leap unity3d

in the box (a prefab from the leap motion package), the rift camera has to be IN FRONT of the leap motion controller icon. the screenshot above shows the correct positioning from top view (leap motion icon is in the middle, right to the lamp icon). it may sound trivial but it took me two hours with many different settings (like the leap motion config menu also allows for changing orientations, etc).

second learning:

z axis not mirrored

the z-axis must not be mirrored. secondly, i scaled the sandbox by 40, this allows you to move freely in the box and your hands seem to be roughly in proportion to the cubes (default size from unity).

one thing i still don’t know: how can i actually get a screen recording while i’m playing? this is unity’s output (ok, ok i did 11 iterations):

unity file outputs

“direct to rift” works for me with my hmd, but on the pc screen there’s just a black window (as in my second video above), without “direct to rift” i see a live window, but the hmd isn’t showing anything. of course, the hmd is then used as a second screen on the pc, and i could move the window to that screen but then it won’t be full screen. and the second problem is that the software actually captures the mouse for controlling the navigation in the virtual space, so i can’t move the window as soon as i the animation actually starts… any ideas? haven’t found anything in the unity “build settings”.

here’s the software that i used:

i was inspired to do the cubes in the box by this video that is a brief tutorial into unity3d and leap motion integration (but has no oculus rift integration).


there’s a second very inspiring video that shows how to get unity running and install the oculus rift package for it (but no leap motion integration):

actually, a couple of things are easier now as you simply can register after downloading unity3d right at first start.