Why use the Web for AR?

• mobile experiences
• open technologies
  • cross-platform
  • non-propietary (unlike Unity or Unreal)
  • free
• easy integration with many existing Web APIs
  • anchors the the web to the real world

Run: Demo

• Check that your phone supports QR code scanning
• iOS : Camera app by default
• Android :
  • use Google Chrome + scan button
  • or install a reliable QR code scanning app such as Trend Micro
• Web alternatives :
  • webqr.com
  • qrcodescan.in

Magic!

• 100% web AR application
• open-source
• cross-platform
• no installation needed
• easy to share
• printable!

Creating your own QR code

• https://duckduckgo.com
  • "qr " + url
  • you must modifiy the link in THREE.js examples to remove iframes (iframes are not supported by XR Viewer on iOS) :
    (https://threejs.org/examples/?q=paint#webxr_xr_paint)
    https://threejs.org/examples/webxr_xr_paint
• Alternatives:
  • use Google Chrome's QR code generator
  • https://www.the-qrcode-generator.com/
  • https://codepen.io/chriscoyier/pen/QyPbXz

Debug : how does it work?

• QR code URL
• corresponding URL
  • https://jeromeetienne.github.io/AR.js/three.js/examples/mobile-performance.html
  • Debug (F12)
  • source code: https://github.com/jeromeetienne/AR.js/blob/master/three.js/examples/mobile-performance.html
• loads the code (see next slide)

Code

<script src='vendor/three.js/build/three.min.js'></script>
<script src="vendor/three.js/examples/js/libs/stats.min.js"></script>

<script src="../build/ar.js"></script>
<script>THREEx.ArToolkitContext.baseURL = '../'</script>

• THREE.js
• AR.js includes:
  • jsartoolkit: asm.js version of ARToolkit using emscripten
  • GeoAR.js: geospatial AR (discussed later)
• THREEx: THREE.js extensions (ancient code!)

AR.js Features

• Monochrome marker detection (cf. ARToolkit 1999!)
• Markers can be customized here and here (QR code)
• Color image detection using NFT (Natural Feature Tracking) (cf. 2010!)
  • you must choose between marker and NFT
• Geospatial AR
• But no SLAM!
  • there is another library for that (AlvaAR)
  • we can still imagine many AR experiences using AR.js

App examples

• Many (old) examples using AR.js + THREE.js + THREEx

  • official
    • basic example
    • you can choose between live video, simple photo and a video a file!
  • Lee Stemkoski

Build: tools

• Web development

  • Web browser (Firefox, Chrome, Safari Mobile)
  • Git (optional but very useful)
  • Code Editor (VSCode), or Glitch (no installation needed!)

• Technologies: HTML, JS, CSS, WebGL, THREE.js, WebXR

• Optional:

  • OpenStreetMap, OpenLayers, CesiumJS, Géoservices IGN

Installing the browser

• Firefox installed by default
  • should be enough!
• Chrome
  • to test compatibility and some features
  • install the latest version (119+) on mobile
  • install Chromium on Linux
    • open-source version without proprietary services

sudo apt-get install chromium-browser

Installating Git

sudo apt-get install git

git config --global user.name "myusername"
git config --global user.email myname@mymailprovider.com

Installing VSCode

sudo apt update
sudo apt install software-properties-common apt-transport-https wget

wget -q https://packages.microsoft.com/keys/microsoft.asc -O- | sudo apt-key add -

sudo add-apt-repository "deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main"

sudo apt install code

Remove GPG warnings

sudo gpgconf --kill dirmngr
sudo chown -R $USER:$USER ~/.gnupg

Customize VS Code

• Avoid UI blinking by changing the settings:

set window.titleBarStyle to custom

• Recommended extensions
  • Live Server or Five Server
  • Git Graph and/or Git Lens
  • glTF Tools
  • WebGL GLSL Editor, glsl-canvas
  • Todo Tree, Color Highlight

Local development caveats

• Impossible to load a file with a user action
• CORS: Cross Origin Resource Sharing
  • one of the many secuirty measures used by web browsers

need to run a server, like Live Server, or using Python:

$ cd /home/somedir
$ python -m SimpleHTTPServer

$ python3 -m http.server

Then open http://localhost:8000 in your browser

Glitch: online interactive editor

• https://glitch.com/
  • Basic THREE.js example

Alternatives:

• Codepen.io
• Repl.it
• CodeSandbox etc.

Modification

• Modify the previous example
  • turn the cube red

A-Frame

• or how to create a web 3D app in just 10 lines of code

<html>
  <head>
    <script src="https://aframe.io/releases/1.0.3/aframe.min.js"></script>
  </head>
  <body>
    <a-scene>
      <a-box position="-1 0.5 -3" rotation="0 45 0" color="#4CC3D9"></a-box>
      <a-sphere position="0 1.25 -5" radius="1.25" color="#EF2D5E"></a-sphere>
      <a-cylinder position="1 0.75 -3" radius="0.5" height="1.5" color="#FFC65D"></a-cylinder>
      <a-plane position="0 0 -4" rotation="-90 0 0" width="4" height="4" color="#7BC8A4"></a-plane>
      <a-sky color="#ECECEC"></a-sky>
    </a-scene>
  </body>
</html>

A-Frame Pros

• built on top of THREE.js
• declarative API: introduces new HTML tags
• uses components
  • Entity-Component System (similar to ECSY)
• extensible: create your own components and tags
• abstracts VR and AR APIs
• no JS code needed for simple scenes!
• powerful debugger
• many interactive lessons

A-Frame AR

• a web AR app in just 10 lines of code (documentation)

<html>
  <head>
    <title>A-Frame + AR.js</title>
    <script src="https://aframe.io/releases/0.8.0/aframe.min.js"></script>
    <script src="https://jeromeetienne.github.io/AR.js/aframe/build/aframe-ar.js"></script>
  </head>
  <body>
    <a-scene embedded arjs="debugUIEnabled:false;">
      <a-marker preset="hiro">
      <a-box position="0 0.5 0" color="yellow"></a-box>
      </a-marker>
      <a-entity camera></a-entity>
    </a-scene>
  </body>
</html>

Modification

• Modify the previous example
  • turn the cube red

A-Frame + Geolocated AR

• interesting markerless technology for outdoor experiences
• A-Frame + GeoAR.js (documentation)

<html>
<head>
    <script src='https://cdn.jsdelivr.net/gh/aframevr/aframe@1c2407b26c61958baa93967b5412487cd94b290b/dist/aframe-master.min.js'></script>
    <script src='../../build/aframe-ar.min.js'></script>
</head>
<link rel="stylesheet" href="../../src/location-based/stylesheets/index.css">
<!-- Dynamically add places from Javascript -->
<script src="./places.js"></script>
<body style='margin: 0; overflow: hidden;'>
    <a-scene
        cursor='rayOrigin: mouse; fuse: true; fuseTimeout: 0;'
        raycaster="objects: [gps-entity-place];"
        vr-mode-ui="enabled: false"
		embedded
		arjs='sourceType: webcam; sourceWidth:1280; sourceHeight:960; displayWidth: 1280; displayHeight: 960; debugUIEnabled: false;'>
        <a-camera gps-camera rotation-reader>
		</a-camera>
	</a-scene>
</body>

Examples

• Official A-Frame examples

• Lee Stemkoski

  • A-Frame
  • A-Frame + AR.js

• FrameVR

  • course for beginners
  • inspiration for your projects

Show a portion of a map on top of the marker

• download map data
• create a texture
• create a tangible interface by manipulating the marker
  • e.g.: change the scale or the content of the map according to the position of the marker relative to the camera
• now try retriving map data live

Display points of interest on a smartphone using GeoAR.js

• define a fictional point of interest and display it using GeoAR.js
• retrieve nearby points of interest (https://www.data.gouv.fr/fr/datasets/r/17e5a7a9-d503-4457-a86d-3e8ff08fc8eb)
• retrieve current GPS coordinatesto create a point of interest

Animation

• Display a map on top of a marker
• Add animated objects (vehicles, Godzilla etc.)

Interactions

• Display a map on top of a marker
• Interact with a second marker
  • changethe displayed data when the second marker comes close to the first one

Fun

• Adapt existing code to leverage AR interactions
  • https://aframe.io/examples/showcase/snowglobe/
  • https://glitch.com/~happy-2020-webxr
  • https://aframe.io/docs/1.0.0/guides/building-a-minecraft-demo.html
• https://glitch.com/culture/master-webvr-with-a-week-of-experiments/
• AR Pong
• https://roland-dubois.github.io/aframe-meetup-nyc/

Native AR

• Download Just A Line, by Google
  • for iOS
  • and Android no longer available: try SketchAR instead
    • install ARCore if needed ("Google Play Services for AR")
    • allows to check that markerless AR works on your device
    • recent smartphone required
    • OS up to date

Demo

Web AR using WebXR

• Android
  • install Chrome 81+
  • if needed, enable WebXR AR module using chrome://flags
• iOS
  • iPhone 6S et +
  • iOS 12+
  • install Mozilla WebXR Viewer (open-source)
• NOTE: experiences are similar but not always compatible (Draft API, non standard yet)

"caniuse"

WebXR
Report

Mozilla WebXR Emulator

• uses WebXR Polyfill
• fake mobile AR device
• very convenient when you don't have an AR device or for debugging
• hand tracking (WIP)

WebXR in Safari inside Apple Vision Pro Emulator

Demo

Android

Source code

https://aframe.io/blog/webxr-ar-module/
https://glitch.com/edit/#!/xr-spinosaurus

<a-scene>
  <a-assets timeout="30000">
    <a-asset-item id="spinosaurus" src="dino-model.glb"></a-asset-item>
  </a-assets>
  <a-camera position="0 1.2 0"></a-camera>
  <a-entity id="dino" position="-1 0 -3" scale="0.5 0.5 0.5">
    <a-entity position="0 2.15 0" rotation="0 55 0"
              gltf-model="#spinosaurus"
              animation-mixer
              shadow="cast: true; receive: false"></a-entity>
  </a-entity>
</a-scene>

Demo 1

iOS

Demo 2

VR only
on iOS?

Source code

https://webxr-ios.webxrexperiments.com/
https://blog.mozvr.com/responsive-webxr-a-painter-xr/

• Code might be obsolete
• uses a non-standard WebXR API
  • cf. WebXR Polyfill
• THREE.js code a bit more complex
  • more than 10 lignes (but still pretty straightforward!)
• A-Frame WebXR AR Module: recent version
  • needs testing, code might need to be adapted

WebXR Basic Concepts XR + AR

• Tracking (spaces) and geometry of the real scene:
  • detect planes and geometry (point cloud or mesh) using SLAM, or similar technologies
• XR Frame: RGB image + camera info (pose, focal, tracking, light)
• Hit test intersection between a virtual ray and the real scene
  • frequent constraint: RGB camera + depth estimation
• Anchors and worldmap :
  • points of interest placed by the used
  • updated continuously as the real world gets reconstructed

WebXR Advanced Concepts

• Occlusion handling
  • human occlusion (ARKit)
  • real world occlusion (ARKit + LiDAR, ARCore)
• Perception
  • of the environment (Vision, IA, LiDAR)
    • reconstruction + classification floor, wall, table
  • of the user
    • hand gestures, gaze, intentions

Hybrid Apps

• Some feature are not available via JS in the standard browser
  • e.g.: no LiDAR access
• Web Plugins are forbidden
  • security, installation and update issues
• Idea: reuse existing web browser components
  • native low-level code (C++, Java, Kotlin, Objective-C, Swift)
  • create new APIs in JS, which call the native code
• issues with non-standard APIs: if we create our own APIs the code might not be portable

Architecture

• Hybrid App example: PhoneGap Cordova

Why?

• Pros
  • expose missing APIs in JS: often the only solution!
  • speed: call native code, faster than pure JS
  • portable code: app logic written in JS
• Cons
  • still slower than a 100% native app
  • wasted performance when converting between JS and native
  • lowest common denominator to satisfy most platforms
  • non-native look and UI (not very important for immersive apps)

Conclusion

• ideally:
  • 100% web applications
  • fast, thanks to standards well-supported by the browser and the hardware (cf. WebGL)
• however standardization will take time
  • XR hardware and software stacks are very fragmented
  • OpenXR adoption should help
• in the mean time, hybrid apps are a good compromise
  • and allow us to start coding AR apps in JS right now!

Standard APIs

• WebVR avoid, obsolete
  • but sometimes the only API available
• WebXR == "WebVR 2.0 + AR"
  • W3C Draft API, not stable yet, evolving fast
  • Chrome 81 , AR still experimental, cf. chrome://flags
• Polyfills
  • allow converting between APIs
  • let you use the latest WebXR API or the old WebVR
  • allow a limited WebXR API emulation if needed

Code examples

• WebXR + WebGL
  • https://github.com/immersive-web/webxr-samples/blob/master/immersive-ar-session.html
  • https://github.com/immersive-web/webxr-samples/blob/master/hit-test.html
• WebXR + THREE
  • https://threejs.org/examples/webxr_ar_hittest
• WebXR + A-Frame
  • Basketball: https://ada.is/blog/2021/01/14/making-an-ar-game/

Security constraints

• permissions
  • camera, location, movement
• https mandatory
  • use localhost + SSL, or glitch, or github.io, vercel etc.
  • better: install ngrok
    • allows to create a https tunnelvery easily
• requires user action to start
  • AR / VR / XR Button to switch to AR

AR Initialization

isSessionSupported(‘immersive-ar’);

// RequestSession on Button press
navigator.xr.requestSession

// Add listener for ARButton Press

// Request reference spaces
localReferenceSpace = await session.requestReferenceSpace(‘local');
viewerReferenceSpace = await session.requestReferenceSpace(‘viewer');

// Request hitTest

session.requestHitTestSource

// RequestAnimationFrame
// NOTE: THREE.js must use
 renderer.setAnimationLoop 
 // instead of window.requestAnimationFrame
// Or else use session.requestAnimationFrame(render)

Draw

// On each Draw
// Callback on every draw, with an XRFrame


const render = (t, frame) => {

    const pose = frame.getViewerPose(localReferenceSpace);

    frame.getPose(localReferenceSpace, viewerReferenceSpace).transform.matrix

    const hitTestResults = frame.getHitTestResults( hitTestSource );
    const hit = hitTestResults[ 0 ];
    reticle.matrix.fromArray( hit.getPose(viewerReferenceSpace ).transform.matrix );

}

Selection (onTouch)

Example: https://github.com/mrdoob/three.js/blob/master/examples/webxr_ar_cones.html

// Get hand, controller, or phone
controller = renderer.xr.getController( 0 ); 

// See also selectstart, selectend, squeeze etc.
controller.addEventListener( 'select', onSelect );

scene.add( controller );

// Before rendering, update the controller, and apply position to mesh (in meters)
mesh.position.set( 0, 0, - 0.3 ).applyMatrix4( controller.matrixWorld );

Experimental

• HoloLens
  • WebVR: obsolete and slow, but works (using Edge!)
  • HoloJS: obsolete
  • Spin Viewer: HoloJS version available on MS Store
    • examples
  • remplaced by BabylonNative?
• HoloLens 2
  • WebXR: works with the new Edge (Chromium based)
  • Firefox Reality Wolvic see this article

Vision

• OpenCV.js : official JS port (wasm via emscripten)
  • terrible documentation but nice tutorials, see this Mozilla article
• JSFeat: lightweight 100% js library (old code)
• Tracking.js: computer vision lib (old code)
• headtrackr, PicoJS: face tracking
• ARuCo Marker tracking: ARToolkit alternative
• Tensorflow.js: recent Vision + IA demos
  • PoseNet
  • HandPose 3D

WebXR + THREE.js

• download https://github.com/fdoganis/three_vite
  • see Web3D course
• choose a webxr example
  • preferably with 'ar / ' in its description
  • ths simplest is AR Cones : https://threejs.org/examples/webxr_ar_cones
• make it run on your phone (or on an emulator)
  • remember taht you can generate a QR code with the URL!

Build: reminders

• npm install
• npm run build
• if you have any issues, check the security constraints

Change the code to display another model

• SketchFab
  • A-Frame component
  • Downloaded model
• Google Poly poly.pizza
  • example
  • A-Frame component
  • Downloaded

Create your own project!

Port your THREE.js project to use the real world!

• real environment using a mobile + WebXR
  • retrieve phone 3D pose
  • compute the intersection with real planes (floor, walls)
  • WebXR is the ideal API for AR apps!
• marker + desktop + webcam or mobile + AR.js
  • possibility to create multi markers (cube)
  • tanglible interactions
• GPS coordinated with mobile + AR.js (outdoors only!)

Code links : Browser-based AR and VR

http://webglworkshop.com/presentations/Workshop31-ar-vr.html#/21

https://webxr.io/webar-playground/

http://learningthreejs.com/blog/2015/07/16/hatsune-miku-dancing-in-augmented-reality/

http://studioknol.com/phase-two-building-with-virtuality/

https://github.com/rodrigocam/ar-gif
https://github.com/XingMeansOK/slamjs_samples (RGBD)

Courses

https://medium.com/arjs/webar-playground-ar-in-a-few-clicks-67a08cfb1534

https://blog.halolabs.io/building-ar-vr-with-javascript-and-html-97af4434bcf6

Guides

https://developer.apple.com/design/human-interface-guidelines/ios/system-capabilities/augmented-reality/