System flowchart
Light Direction Estimation and Hand Touchable Interaction for Augmented Reality

Augmented reality is a technology that combines virtual world with the real world. How to improve the realism of augmented reality is an important topic. One focus of this paper is lighting consistency between virtual and real world, and the other is interaction with virtual object using hands.
Estimating lighting conditions through traditional methods often requires many prior knowledge of the scene. We propose a method that estimates the light direction based on shadows and foreground objects with only one scene image. We detect and calculate the relative direction of object and its shadow in the scene to estimate the azimuth of light, and use area size ratio of object and its shadow to estimate the elevation angle of light. We used some real scenes to test our method. However, the exact light direction of real world is difficult to acquire, so we further verified our method by establishing a number of virtual scenes with pre-set light direction.
Moreover, hand gesture based human-computer interaction provides a natural and easy way for interaction. Traditional augmented reality interactions use markers or touch screens. We apply gesture recognition and hand touchable interaction in augmented reality, allowing the user’s hand to occlude the virtual object in the picture, and recognize the gesture. By adding estimated light and hand touchable interactions, we enhance the realism of augmented reality.

SUMMARY (中文總結):


  • Result

  • Reference:

      [1] Frederic Cambon. Pikachu model., 2014.

      [2] Matterport. Internet images for virtual object insertion., 2017.

      [3] Shih-Hsiu Chang, Ching-Ya Chiu, Chia-Sheng Chang, Kuo-Wei Chen, Chih-Yuan Yao, Ruen-Rone Lee, and Hung-Kuo Chu. Generating 360 outdoor panorama dataset with reliable sun position estimation. In SIGGRAPH Asia 2018 Posters, page 22, 2018.

      [4] Yannick Hold-Geoffroy, Kalyan Sunkavalli, Sunil Hadap, Emiliano Gambaretto, and Jean-Francois Lalonde. Deep outdoor illumination estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7312–7321, 2017.


Virtual object insertion

Cumulative angular error distribution comparison with Hold-Geoffroy et al. [4]