A major gaze-tracking patent published last week was written by one of Apple’s mixed reality, computer vision superstars

Last Thursday the US Patent & Trademark Office published a patent application from Apple that relates to assessing characteristics of pupils of eyes, and in particular, to systems, methods, and devices for assessing pupil characteristics using light reflected off the eyes to estimate the user’s gaze direction using multiple glints to identify the user’s eye shape, position, and orientation. In various implementations, pupil characteristic assessment is used to facilitate gaze tracking, which may be used to enable user interaction.

Last week Apple analyst Ming-Chi Kuo stated that “Apple’s future headset will include innovative human-machine UI technologies such as gesture control, object detection, eye/gaze tracking, iris recognition, voice control, skin detection, expression detection, and spatial detection.“

Patently Apple has covered a number of eye and gaze tracking patents (01, 02, 03, 04 & 05) and it’s clear that this technology will play an important role in Apple’s future mixed reality headsets. Apple fans will appreciate this technology once they’re able to see it in action. For now, Apple’s patent on this technology is very technical and likely to be more appreciated by developers and engineers in this field.

Apple notes in their patent filing that existing pupil detection techniques may use an image sensor and integrate light intensity level over an exposure period to produce greyscale images and then attempt to detect the pupil using the greyscale images. The pupil is detected based on greyscale contrast between the pupil region and the surrounding iris region and thus rely on there being significant contrast between the pupil and the iris. Such pupil detection techniques may not be as accurate or efficient especially in circumstances in which the contrast between pupil and iris is less significant. This is what Apple’s invention sets out to remedy.

Apple invention covers a pupil characteristic assessing system that includes a light source, an image sensor, and a processor that performs pupil characteristic assessments on data received from the light sensor regarding light from the light source reflected off an eye of a user. In various implementations, a pupil characteristic is determined using on-axis illumination from the light source so that light from the light source is reflected off the retina of the eye to produce a bright pupil-type light pattern in data obtained by the image sensor.

The light may be modulated or otherwise pulsed at a frequency and frequency segmentation may be used to distinguish reflections through the pupil off the retina from reflections of light from other light sources. In some implementations, the image sensor is a frame-based camera and the method subtracts one image from the next image along a sequence of images to identify light pulses that occur at the frequency in the images.

In some implementation, the image sensor is an event camera and the amount of time between events corresponding to light reflected off the retina and through the pupil is used to determine events that occur at the frequency.

The event camera has light sensors at multiple respective locations. In response to a particular light sensor detecting a change in intensity of light, the light sensor generates an event message indicating a particular location of the particular light sensor.

An event camera may include or be referred to as a dynamic vision sensor (DVS), a silicon retina, an event-based camera, or a frame-less camera. Thus, the event camera generates (and transmits) data regarding changes in light intensity as opposed to a larger amount of data regarding absolute intensity at each light sensor.

For deeper details behind Apple’s patent application number 20210378509, click here. 

The patent lists Walter Nistico as the sole inventor of this patent. Nistico is the Engineering Manager – Deep Learning and Computer Vision at Apple. Nistico came to Apple 4.5 years ago with the acquisition of SMI SensoMotoric Instruments. Nistico’s LinkedIn profile states that Designed and shipped 7 products redefining state-of-the-art in Eye Tracking, used by all major university, research institutions. Prior to joining Apple, Nistico also worked on VR and AR projects with Google, Qualcomm, Sony, Nvidia and Intel.

Considering that this is a patent application, the timing of such a product to market is unknown at this time.