Stereoscopic First Person View System for Drone Navigation

Summary

A system was developed for ground controllers to have an immersive experience flying the UAV by implementing stereo vision. Users in the ground can have a low latency FPV of the drone for navigation that pans the camera around based on head movement and employs stereo vision so as to help the controller estimate depth better.

Objective

One of the major challenges of UAVs is their teleoperation from the ground. Most consumer drone control setups only include low resolution screens with a limited field of view. The drone setups used for drone racing championships use FPV glasses and can achieve speeds of 30 m/s. However, they possess a very narrow field of view of about 30-40 degrees using monocular views.

Hence, the research revolves primarily around developing an FPV system for operating drones using stereo vision to provide a real time immersive experience for ground operators. It proposes using methods such as head tracking for panning the camera as well as making technical adjustments to reduce latency for minimizing motion sickness.

Method

The system consists of a wide-angle stereo camera with a 185 degree lenses attached to the UAV. The camera setup was custom-made and weighed roughly 1.5 kilograms. The left and right stereo frames are combined into one image and the video feed is streamed live onto an Oculus Rift headset that a ground pilot can use for navigation. The pilot can move their head around for observing different angles while controlling the UAV via a controller. The more technical aspects of the hardware used can be found in the research paper.

The computer on the UAV concatenates the two stereo images into one, encodes the fused image in H264, and streams it over WiFi. The CPU on the UAV is capable of depth calculation but cannot simultaneously calculate depth and stream the video.

For the experiment, 7 participants were asked test the UAV via the Oculus Rift headset and controller. They were asked to evaluate different viewing systems, i.e., mono versus stereo, on how it affected perception and they were asked to estimate height of flight. Furthermore, another experiment was conducted to study simulator sickness.

Result

Participants were able to take off, fly, and land the drone at speeds less than 7 m/s. Moreover, none of the participants faced moderate or severe simulator sickness. For the first experiment, as expected, participants using stereo cameras were able to more accurately predict flight height, although the experiment did result with participants suffering simulator sickness. The second experiment yielded much data that showed what factors simulator sickness depended upon.

In essence, the results provided evidence that an immersive stereoscopic first person view control for drones might enhance the ground control experience.

That isn’t it to say there aren’t shortcomings. The drone with the cameras and other hardware weighed in at 3.3 kilograms which reduces flight speed. Furthermore, it does induce simulator sickness based on various factors.

Inference

There are very interesting takeaways from this research. Firstly, the research focuses on video streaming to a ground operator; however, it does mention that it is possible to estimate depth as well. If implemented for that purpose, depth data can be sent to the ground operator. However, it is entirely possible to implement the system the way proposed in the research paper of having the ground operator have an FPV for navigating depending on the use case.

However, there are certain shortcomings to the paper. Firstly, only 7 participants were selected for the experiments and hence the results might be slightly unreliable. Furthermore, the hardware used for the research is both heavy and may not fit in well with our current architecture. There are certain trade-offs that may need to be made in case this needs to be implemented. However, one good note is that there are very specific and technical resources provided for the research.

Autonomy