Augmented Reality Writing

A 3D pen that writes on any surface and in mid-air, with the ink rendered in augmented reality. Built in 2011, patented in 2018.

Company: HP / Palm
Year: 2011
Role: Concept, system design and implementation · inventor on the granted patent
Team: Future Concepts and Prototyping team, HP/Palm R&D Center
Location: Sunnyvale, CA

Technology

OptiTrack motion capture
Tracked pen and tablet
Video see-through mobile AR
Persistent 3D drawing
Environment color picking
C

01The opportunity

In 2011 tablets and styluses were arriving together, and every pen was trapped on the glass. HP/Palm's Future Concepts team was looking at unconventional uses of tablets and pens. I wanted to know what happens when the pen leaves the surface.

02The idea

Write or draw anywhere: on the table, on the wall, or in the air above them. Nothing physical is marked. The strokes exist as persistent 3D geometry in the room and become visible through a tablet or phone acting as a mobile AR window. Pick a color from the environment first, then draw with it.

03My role

I developed the prototype at HP/Palm: the concept, the tracking integration, the AR rendering and the interaction, with the proof-of-concept code written in C. The work was completed in December 2011 and resulted in a granted patent after six and a half years of review.

04How it worked

An OptiTrack motion capture system tracks the pen and the tablet in six degrees of freedom. Pen strokes are stored as 3D geometry anchored to the room. The tablet's camera feed is composited with the strokes from the tablet's tracked viewpoint, so the writing appears exactly where it was made, on the surface or above it, and stays there as the viewer moves. The production intent was on-device tracking; the external sensors were a means to prove the experience.

05Prototype / product

Fig. 01PrototypeThe fully functional prototype. Writing on surfaces and, from around 1:30, in mid-air, viewed through a video see-through tablet with the tracking infrastructure visible.

Mock-up showing the word Hello rendered on the tablet where the user wrote it on the table — Fig. 02Mock-upMock-up made before the first prototype to define the intended experience: the display renders “Hello” precisely where the user wrote it, though no ink was used.

Mock-up of augmented reality writing extending above the table surface — Fig. 03Mock-upStrokes are not limited to the surface: they extend above and around it in 3D space.

Side view of the augmented reality writing setup showing that nothing is visible on the table — Fig. 04Mock-upSeen from the side rather than through the display, the table is blank.

Figure from the granted augmented reality writing patent — Fig. 05ResearchFigures from the granted patent, taken directly from the prototype video and images.

Wide mock-up image of the augmented reality writing experience — Fig. 06Mock-upThe augmented reality writing mock-up.

06Outcome

A working proof of concept completed in December 2011 and a granted U.S. patent, Augmented reality writing system and method thereof, issued in 2018 after the IP moved to Qualcomm with HP/Palm's portfolio.

Working prototype, December 2011
Granted U.S. patent (2018)

07Patents, press, links

Patents

US 10067568 B2Augmented reality writing system and method thereof2018