PlayStation’s Dominic Mallinson on next-gen VR, wireless, gaze tracking, foveated rendering, and AR

PlayStation is winning the high-end VR battle. Sony has sold 4.2 million PlayStation VR headsets as of March, as Dominic Mallinson, senior vice president of R&D at Sony, made sure to point out at Collision 2019 in Toronto this week.

“We’re very happy with those numbers and very happy with the position,” Mallinson noted. “But we know we can do better. There are over 96 million PlayStation 4s in the market. And every single one of those is capable of delivering a great VR experience. So we’d like to convert many, many more of those people to be PSVR users. And we won’t just stop with PS4.”

Mallinson is of course referring to Sony’s promise last month that PlayStation 5 will support PSVR. Because he’s part of Sony’s R&D labs, Mallinson gets to see some of the experiments, direction, and new content “that’s coming through the pipeline, as he put it. That’s what helped him name his session “The future is bright for virtual reality.”

During his presentation, Mallinson left the crowd with three big takeaways:

We followed up with Mallinson after he got off stage to dig in a little more.

The first takeaway is straightforward: TV shows, regular video games, movies, books, and radio all give you a certain level of immersion. VR goes the furthest. “VR engages all of your senses in a more heightened way,” Mallinson stated. “It engages your eyes in a more heightened way. It engages optics in a more heightened way. You just don’t get that in any other medium.”

Mallinson briefly talked about how people who suck at golf feel empowered playing the game in VR. He described — but emphasized that it’s impossible to do it justice — the incredibly unique feeling of flying around like Iron Man.

Speaking of feelings, we asked him how he felt seeing his classic PlayStation hit Wipeout come to VR.

“It was wonderful,” Mallinson said with a big smile. “It got me a little bit emotional because it’s what we always dreamed of doing when we created Wipeout — having it in VR. So to see them actually deliver that, and do a really good job as well, was just fantastic. Unfortunately for me, personally, I get a little bit sick in VR. So much as I enjoy playing it, I couldn’t play it for too long because it is a little bit more extreme.”

This is the takeaway Mallinson talked about the most, both onstage and in our follow-up. He gave three examples of evolutionary hardware improvements first, and offered his predictions.

“The first is resolution,” Mallinson explained. “This is more pixels per degree. It’s about the sharpness and the clarity of the display. And you have to be able to match what people expect to see with high definition. I would expect the resolution to roughly double in the next set of VR products.”

“Along with that, we also need a greater field of view,” he continued. The human visual system is out to about 180 degrees. Most VR headsets are about 100 degrees. There are diminishing returns to get wider. But I would expect the next set of products to be roughly 120 degrees in terms of field of view.

“And finally, HDR. In the TV industry, HDR is already incredibly important to creating the best experiences. The human eye sees an enormous range of light from bright sunlight to deep shadow. VR panels only capture a tiny fraction of that. So in order to increase the sense of presence, I do expect to see HDR adopted in the near future.”

And with that, Mallinson quickly moved to revolutionary changes. Mallinson cares a lot about wireless high-end VR because of the adoption implications.

“User comfort is incredibly important in order to widen the adoption of virtual reality,” Mallinson emphasized. “There are many aspects to user comfort. I’m just going to touch on one here. And that is the cable. Being tethered to this cable is inconvenient. And it’s not just about getting tangled up in the cable. It’s not just about the restriction in your motion. It’s also about how you set things up, how you configure the system, where you store it. Let’s face it, having a mess of cables in your living space is just not attractive. So this is something that we have to solve in order to get wider adoption.”

He offered two solutions: an all-in-one headset, where the compute is part of the headset, and using wireless transmission technology to replace the cable.

“In both cases, these require a battery, either on your head or close to your head,” Mallinson noted. “Having a battery on your head is a little bit inconvenient in terms of ergonomics and industrial design. But I think that the all-in-one headsets that you’re beginning to see now are actually getting pretty good. But honestly speaking, they cannot possibly compete with a wired headset because of the enormous amount of compute and rendering performance you can get on a high-end PC or a games console. You just can’t put that on your head.”

Thankfully, progress is moving quickly here too.

“But fortunately, wireless transmission technology is getting better,” Mallinson said. “New technologies such as 60 gigahertz are allowing for these options to become possible for VR products. But it might well remain an option, because it will be more costly than with the cable.”

The second takeaway led us to an obvious follow-up question: Are there plans to modularize PSVR2? Not exactly, but there could be multiple versions.

“It’s certainly an option,” Mallinson agreed. “I talked about wireless, for example. That’s one easy way to do it. Here’s a wired headset. You can take the wire and replace it with wireless. And then you can have a range. So you can have an introductory model and a high-end model. That’s something we’ve done with PlayStation 4. We could do that with PSVR.”

Mallinson didn’t commit to anything, but the options he raised will send the rumor mill into overdrive.

Saving the best for last, Mallinson talked about the form of user input that he believes will succeed touch.

“Gaze tracking — this is the technology that excites me the most,” Mallinson declared. “We’re already beginning to see this in some products on display at industry events. I think it has the greatest potential to change the VR user experience at a pretty fundamental level. I think it was Shakespeare who coined this phrase that ‘the eyes are a window to our souls’. I’ve been a little more prosaic by saying that ‘the eyes are a window to our thoughts’. I think everyone can intuitively understand just how rich human communication becomes when you have that eye contact.”

“So what do I mean by gaze tracking? I mean the technology to understand where you’re looking in this virtual world. What is your attention point? And then on top of that, we can then layer extra things. We can understand perhaps your attention by measuring pupil dilation. We can do biometrics to understand who you are looking at. … And we can measure your IPD (interpupillary distance) — the distance between your pupils. This is very important to VR because it allows us to accurately set up the optics and the rendering to give you maximum comfort, and to really get the correct sense of distance and scale in VR. So fundamentally, with this technology, we know what you’re looking at in VR. And this allows for countless user interface and user experience possibilities.”

Mallinson offered a few potential applications of gaze tracking. There’s the obvious use case of being able to steer a cursor and to select things on a menu. Controlling a cursor with your eyes (see Windows 10’s eye tracking feature) is the most basic, but “even that is incredibly useful.”

Then there’s the broader idea of addressability. Imagine multiple virtual characters and being able to look at each one individually and give them a command, or ask each a question. “And very naturally, without having to use a mouse or a keyboard or a joystick, those comments are addressed correctly.” That also applies to social VR — virtual eye contact can go a long way in driving adoption when multiple people inside VR are talking to each other and expressing themselves.

“I think that the gaze tracking is the most exciting change that we’ll see in next-gen VR,” Mallinson declared. “So really, if you look at the history of user input, starting off with keyboards, and then the mouse, and recently touchscreen interfaces, I seriously think that having gaze as a user input is going to be as fundamental as each of those changes we’ve had in the past. That’s my number one point about next-generation VR: Gaze will allow much, much richer user interaction.”

He did admit, however, that there is no consensus about this in the VR industry.

“Everyone has a different opinion about this, but I believe that is the biggest differentiator. Some people only care about visuals, and therefore highest resolution might be more important than anything else. And certainly if you have an application where having 20/20 vision is super important, like if you have to be able to read very small text, then yeah resolution is of top importance. But for me, across the whole spectrum experiences, is gaze.”

The other reason Mallinson is so bullish about gaze tracking is because it enables foveated rendering.

“More pixels needs more rendering performance,” Mallinson explained. “If you just brute force it, it requires a lot of extra rendering performance. The human eye has a part in the retina called the fovea, which is responsible for our super-sharp vision. We don’t see very much in the peripheral vision. So if we can match our rendering performance to the fovea, we can deliver higher effective resolutions, and also better quality images. So gaze tracking is a win-win in this respect.”

Gaze tracking thus “pays for itself.” It brings a new user experience, and enables the hardware optimization of foveated rendering.

All of the above together consists of the rapid improvements in technology that Mallinson expects will drive wider adoption of VR.

The other reason Mallinson is so bullish about gaze tracking is because it enables foveated rendering.

“More pixels needs more rendering performance,” Mallinson explained. “If you just brute force it, it requires a lot of extra rendering performance. The human eye has a part in the retina called the fovea, which is responsible for our super-sharp vision. We don’t see very much in the peripheral vision. So if we can match our rendering performance to the fovea, we can deliver higher effective resolutions, and also better quality images. So gaze tracking is a win-win in this respect.”

Gaze tracking thus “pays for itself.” It brings a new user experience, and enables the hardware optimization of foveated rendering.

All of the above together consists of the rapid improvements in technology that Mallinson expects will drive wider adoption of VR.

“This has been talked about so many times before, but it bears repeating,” Mallinson reiterated. “Because it is the most important thing. Content is everything. Without content, VR is just a piece of boring tech. So we really need that great content. And fortunately, we already do have a lot of great games for VR. But there’s also many, many excellent non-game experiences: sports, movies, training, therapy, and so much more. Social VR is going to be a big driver in terms of the adoption of VR.”

Mallinson really wanted to drive home the point that VR extends beyond video games. “There’s so much more to VR than games. It’s great for storytelling. It’s wonderful for training and simulation. And even pain management is something that you can do with VR.”

Sony is investing in non-game VR applications, but it’s sticking to the entertainment realm. That means social VR, narrative storytelling, and so on, but not medical use cases.

“We need more people to come in and make compelling experiences for VR. We have a pretty good handle on games. So I’m not too worried about that. And we’re still investing in our own first-party games as well. But I think everyone else out there — they could step up their game a little bit. So we could see more education, see more training, see more medical applications.”

While the majority of our conversation was about VR, we did ask about PlayStation’s AR plans.

“We are not doing anything with AR. But obviously from a research and development perspective, we’re keeping a very close eye on things.”

Why not?

“AR is very exciting. I think it has great potential in the future. But it’s an order of magnitude more difficult to do well. So if we can crack VR, we can move to AR. But to leap directly into AR, I think, is a stretch too far. Getting the visuals to match what’s happening in the real world is very hard.”

Mallinson gave three examples. He talked about the vergence-accommodation conflict, which is a focus issue. He explained that that it’s a very hard problem to solve, and is currently sidestepped in VR. You can’t sidestep it in AR.

He also talked about registration of the real world — making virtual objects appear solid. If you put a virtual object on a real object and move the latter quickly, the former has to “catch up.” It should move accordingly — that’s also very hard to address.

And finally, Mallinson said capturing real-world lighting is very difficult. It’s not easy to draw accurate shadows for virtual objects based on the real light that is hitting them.

“And this list goes on,” Mallinson complained. “And specifically in gaming, it’s not even clear that all types of games are suitable for AR. If you’re playing a Lord of the Rings game, or a fantasy game of any kind, do you really want to play a fantasy game in your kitchen? It’s not obvious you do. So there’s just a big list of problems in AR, which we need to overcome before AR becomes huge in gaming. And I think it will, but we need to focus first on getting VR to be as good as it can be. And then we can naturally progress to AR.”

How integral will VR be to the next PlayStation?

“It’s important. I think you will still have a great experience if you don’t have VR. If you have VR, you’ll get something that you otherwise wouldn’t be able to get. But you don’t have to have VR.” (Sony likely doesn’t want to repeat Microsoft’s Kinect screwup.)

What does Sony need to do to keep up momentum for PSVR?

“Many, many things. The most important one is to keep good content coming through the pipeline. To really encourage all the developers out to there to make more great VR content. That is number one. From a hardware point of view, it is the comfort and the encumberance of the device. So the easier to take it off, the easier it is to set it up, and have a very comfortable, ergonomic experience, that’s our biggest goal.”