Facebook researchers explore mechanical display shifting to reduce screen door effect

Researchers from Facebook Reality Labs and the University of Arizona published new work exploring the use of high-speed mechanical display shifting to reduce the so-called screen-door-effect (SDE) of immersive displays. SDE is caused by unlit spaces between pixels to the immersion-reducing appearance of a ‘screen door’ between the viewer and the virtual world. The researchers experiment with rapidly and minutely shifting the entire display to cause the display’s pixels to fill in the gaps.

SDE has been one the visual artifacts in modern VR headsets since the introduction of the Rift DK1 development kit in 2013. While SDE can be defeated with brute-force by employing extremely high density displays—in which the unlit spaces between pixels are too small to be seen by the naked eye—most consumer VR headsets still exhibit SDE (with the near exception of Reverb G2), hurting immersion and visual clarity.

Beyond ultra high pixel density, other methods have been employed to reduce SDE. For instance, some headsets choose a smaller field of view which reduces the apparent visibility of SDE. Other headsets use a diffuser film on the display to help blend the light from the pixels into the unlit spaces between them.

Another proposal is to rapidly and minutely shift the display such that nearby pixels fill in the unlit gaps. While this might seem like it would create the appearance of a dizzying jiggling display, it’s been demonstrated with other display technologies that moving a point of light (ie: a pixel) quickly enough can create the appearance of a stable image.

Researchers Jilian Nguyen, Clinton Smith, Ziv Magoz, and Jasmine Sears from the University of Arizona and Facebook Reality Labs Research explored and experimented with the idea in a paper titled Screen door effect reduction using mechanical shifting for virtual reality displays.

Rather than building a VR headset with mechanical display shifting right out of the gate, the paper’s goal was to demonstrate and quantify the efficacy of the method.

Display Actuation and Modes

The researchers designed a static platform with two piezoelectric actuators which, together, shift the display in a circular motion at 120Hz—in effect, causing each pixel to trace a 10µm circle 120 times per second. The size of the circle was picked based on the distance between the display’s pixels in order to optimally fill in the unlit spaces between pixels. The researchers call this circular path ‘Non-redundancy’ mode.

They also smartly utilized a 480Hz display which allowed them to experiment with a more complex pixel shifting path which they called ‘Redundancy’ mode. This approach aimed to not only fill in the gaps between the pixels with some additional overlap, but split the displayed frame into four sub-frames which are each uniquely shifted and displayed to account for the pixel movement. This means that when a pixel shifts to a location where it would fill in the SDE gap, it uses the correct color which would be used if a pixel was located in that position in the first place.

While the paper is limited to exploring these two pixel paths, the researchers say that others could be employed based on display characteristics.

“Pixel shifting is not limited to a circular shape. Indeed, an elliptical path or even a figure-eight path could be used by controlling the amplitude of each axis’ movement. Paths can be traced in many ways to explore screen door reduction,” the researchers wrote. “For the micro OLED display, a circular path was well-suited to the square pixel and sub-pixel layouts. This path is used to balance the length of the path with the fill factor, minimizing the speed the actuators must operate at.”

With the platform built and capable of shifting the display rapidly in the desired paths, the next step was to objectively quantify the amount of SDE reduction, which proved to be difficult.

Quantitative Measurement of Mechanical SDE Reduction

The authors first sought to objectively measure where each subpixel began and ended, but found that the resolution of the camera they employed for the task was not fine enough to clearly delineate the start and end of each subpixel, let alone the spaces between them.

Another approach to quantify SDE reduction was to measure the contrast ratio of a section of the display and compare it to when the screen actuation was on vs. off. Lower contrast would imply less SDE due to moving pixels filling in the unlit spaces and creating a more solid image. While the authors maintained that this measurement isn’t necessarily a reflection of the SDE reduction as the naked eye would see it, they believe it’s a meaningful quantitative measurement.

Qualitative Assessments of Mechanical SDE Reduction

Beyond their efforts to quantitatively measure the SDE reduction, the researchers also wanted to look qualitatively at the change. The clearest demonstration of the benefits came from looking at a natural photo with complex scenery.

Here, the ‘Non-redundancy’ mode clearly reduced the SDE while apparently retaining equal sharpness. Impressively, the ‘Redundancy’ mode not only reduced SDE, but even appears to noticeably sharpen the image (note the zoomed-in sections showing details in the rear of the car).

The image sharpening of the ‘Non-redundancy’ mode is an interesting additional benefit because it actually increases the resolving power of the display without increasing the number of pixels.

Based on their experimentation the researchers also suggest a user-study approach for future investigations which could be used to quantify any SDE reduction method, whether that be mechanical shifting, diffusers, or different sub-pixel layouts and optics.