IRCAD (2002 - 2003)

• Institut de Recherche contre les Cancers de l'Appareil Digestif
• Startup
  • Virtual-Surg team
• Augmented Reality Research Engineer

Dassault Systèmes (2003+)

• 3D Visualization Engineer
  • Scenegraph, Materials
  • Geometry, Tessellation
• Virtual and Augmented Reality (XR) Engineer
• XR Research Engineer
• XR Research Manager

Dassault Systèmes

From Shape to Life

Course prerequisites

• Math
  • 3D vectors and matrices
• Programming
  • JavaScript notions, or any similar language (HTML kept minimal)
• 3D Web API
  • THREE.js notions strongly recommended (see Web 3D course)
  • alternatives: Babylon.js, WebGL, WebGPU
• Desktop + Smartphone (or, even better, a XR HMD!)
  • VSCode

Planning

• AR Intro (20 min)

• Theory (30 min)

  • THREE.js intro (10 min)
  • WebXR intro (10 min)

• Practice (1.5 hours)

  • Simplest 3D webpage (Canvas, WebGL, THREE.js)
  • SceneGraph
  • Simple XR app (interactions, environment)

Definitions

• Milgram's Reality-Virtuality Continuum

Milgram, Paul; H. Takemura; A. Utsumi; F. Kishino (1994). "Augmented Reality: A class of displays on the reality-virtuality continuum".

Google's version

Properties of an AR system

(according to Azuma, 2001)

• combines real and virtual objects in a real environment
• runs interactively, and in real time
• registers (aligns) real and virtual objects with each other.

Not AR:

• special effects in movies
  • technology close to AR
  • not real time
  • not in a real environment
• Google Glass
  • combines real and virtual objects in a real environment
  • no registration
  • it's a HUD (Head-Up Display)
    • can still be useful! (maintenance, sports etc.)

Google Glass concept video (2012)

Definitions

• VR : Virtual Reality Jaron Lanier, 1987

• AR : Augmented Reality Thomas P. Caudell, 1990

• MR : Mixed Reality

  • marketing term used by Microsoft
  • no clear definition! Term must be defined before use!
    • cf. What is Mixed Reality?

• XR : X = { eXtended / Cross (+) / Any (*) / A+V } Reality

  • recent generic term which encompasses AR and VR

AR | VR

Effects of VR

• Isolates the user from the real world
• Teleports the user to another world, which is entirely virtual

Tiltbrush

ITI

Effects of AR

• The user stays in the real world
• AR enhances the real world with contextual information
• Augmented user: acquires new senses!
• Information becomes visible
  • spatialized information overlaid on top of the real world

Audi AR manual (Metaio, 2012)

Takeaways

• VR immerses the user in a virtual wolrd
• AR brings virtual objects into the real world

Choosing the right paradigm

• Immersion useful ?

  • Yes VR
  • No 3D

• Immersion and real environment useful ?

  • Yes AR
  • No VR

• Keep in mind continuum to pick the right paradigm to create the best possible experience

Choosing the right paradigm

Prehistory (1966)

• Ivan Sutherland invents the first AR headset
• Can display a cube
• Follows the movements of the head (6 DOF)
• Nickname : The Sword of Damocles
  
• paper: The Ultimate Display

Markers (1999)

• Monochrome markers
• ARToolkit created by Hirokazu Kato
  
  
• Alternatives: ARTag, ArUco
• PC + Webcam

NFT, GPS (2005)

• NFT: Natural Feature Tracking
  • Color photo tracking
• Wikitude, Layar (GPS)
  • no image processing needed with GPS!
• Vuforia
• Marketing use-cases
• PC, mobile phones, tablets

SLAM, 3D (2015)

• 3D environment tracking
• SLAM: Simultaneous Location And Mapping
• 3D object tracking
• Deep Learning
• Occlusion 3D
• ARKit, ARCore
• Smartphones, HoloLens, Azure Kinect

Azure Kinect + HoloLens 2

Near future

• Form-factor: glasses
• AI
  • contextual assistance
  • understands both environment and user
• 5G
  • application and information streaming (Edge Computing)
• Spatialized Web: AR Cloud

Far future

• AR will replace or complement smartphones
  • users will raise their heads again
    • but will they see better?
• Contact Lens (Mojo Vision)
  • RIP (2023)
• Ambient Computing
• Ubiquitous Computing
• Smart Cities

AR [is for] adding shared meaning in the interaction between people.
Johnny Lee, Google I/O 2017

• Dropped mobile VR (Cardboard , Daydream )
• Dropped Tango , to reach more devices: rely on RGB camera + AI
• API ARCore, competes with Apple's ARKit
• Google wants to provide cross-platform AR services
• Google + Qualcomm + Samsung XR Headset coming in 2025

The Web connects the world's information, and AR connects information with the physical world. So together they can be applied to solve real life problems.
Andrey Doronichev, Google I/O 2017

Gorillaz mobile AR app

Hardware

• Adds LiDAR for a robust SLAM (e.g. white walls scenario)

• Extends its 'wearables' category

  • AirPods
  • Apple Watch
  • Apple Vision Pro
    • unveiled in June 2023
    • released in February 2024

• Created Horizon Worlds social network for virtual encounters, and Horizon Workrooms for remote collaboration
• Believes that AR will replace smartphones and all other screens in 10 years and does not want to miss this revolution: Project Aria
• Interested in personal data, centers of interest of their users (eye-tracking, scanned environment), although they claim the opposite
• VR with Oculus, but they now focus on AR, cf. Infinite Office
• AR filters Facebook and Instagram via Spark AR RIP
• Great WebXR supporter

Michael Abrash in 2019

Project Aria

Tracking techniques

• Marker
  • accurate, fast
  • tangible, printable
  • need to display a marker to enable AR
  • non-aesthetic
  • can be hard to detect (low lighting, motion blur, occlusions)
• NFT: same but
  • more aesthetic, easier to embed in the real world (ads)
  • more robust to occlusions

Tracking techniques

• SLAM : NFT evolution + reconstruction
  • more natural markerless experience
  • partial scene reconstruction
    • allows advanced functionalities (occlusions, collisions etc.)
  • not very accurate
    • drift, loop closure
    • scene reconstructed and refined in real-time
    • difficult to define the origin of the scene
      • stable anchor points required

Tracking techniques

• 3D object detection in a real scene
  • using computer vision (lighting, edges, silhouette)
    • generic algorithm
    • but slow, especially during initial registration
  • using Deep Learning
    • faster initial detection
    • more robust regarding occlusions and lighting changes
    • not generic: requires per model training

Tracking techniques

Conclusion

• No tracking technique is ideal
• Keep them all in mind and choose the right one according to:
  • the scenario of the AR experience
    • industrial context, consumer, generic or specific
  • constraints
    • indoor, outdoor, mobile

Rendering

• Realistic or not
• Lighting
  • detect the direction and intensity of real lights
  • fast environment reconstruction to simulate reflections (SLAM + AI)
• Occlusions
  • people, objects

Interactions

• The missing part of the equation
• Often neglected (cf. NReal)
• Myth of the dying mouse (p. 17)
  • each form factor has an optimal interaction technique
  • most headsets handle hand tracking, but also offer controller, keyboard and mouse support!
• The XR equivalent of the mouse has not been invented yet!

Interaction techniques

• Screen, when using a smartphone
  • not very immersive but accurate, and provides tactile feedback
• Controllers with buttons
  • great haptic feedback but not immersive
• HoloLens GGV : Gaze, Gesture, Voice
  • natural interactions, with no external hardware
  • great but tiring, lacks privacy ("hey Cortana!"), and accuracy
• Tangible interactions
  • markers or accessories to add some tactile feedback

Interactions

Conclusion

• Immersive AR interactions have yet to be invented!
• No interaction paradigm has become a standard yet
• We must guide the users and try to understand their intent

3D Reconstruction

• Colmap
• AliceVision
• Capturing Reality
  • RealityScan for iOS (link)
• MicMac
• Scaniverse

Photo Credits

https://unsplash.com/photos/RgPVZvA4wBM

https://unsplash.com/photos/r2CAjGQ0gSI