By Dr. Raj Talluri, President & CEO of Enovix
In 1838, Charles Wheatstone was recognized for explaining that our binocular vision is the result of our brains combining and processing two 2D images into a single 3D object. His work led to the earliest stereoscope. By the 1930s, the idea of movie goggles was making its way into fiction. And in 1962, the first VR machine was patented–the Sensorama — that looked like a seated video game with a large hood that included everything from scent spray, a vibrating chair and stereo speakers. The demand for immersive experiences continued with more innovation in the 60s and 70s, from flight simulators to the Aspen Movie Map out of MIT. The 80s and 90s ushered in games by well-known brands such as Mattel and Sega and, in 1991, NASA invested in a VR system to control the Mars robot rovers from earth.
I started my career at the Indian Space Research Center working on remote sensing satellites in the late 80s. I was working on the problem of spatial localization of satellites as they orbit the earth using stars as guidance. This problem fascinated me and I enrolled in a Ph.D program in computer vision at the University of Texas at Austin. After completing my Ph.D, I joined Texas Instruments research labs, continuing my work in computer vision. This is where I was first exposed to VR, which at the time, was little more than a lab research project because the technology for displays, cameras, processors, memory, and sensors was not ready to make a realistic VR product.
Much later in my career, when I was leading the Snapdragon Application Processors team at Qualcomm Inc., we worked with the Samsung smartphone team to create the Gear VR headset. It was a nifty idea to place a high-end smartphone inside a plastic case and use software to create a VR experience. Google cardboard, launched in 2014, was also of the same ilk and a lower cost alternative. In 2017, Qualcomm and ODG announced the first AR smartglasses powered by the Snapdragon 835 processor, enabling lighter, smaller and sleeker smartglasses. And by 2020, the Snapdragon-powered Oculus Quest 2 was launched, receiving mostly positive reviews.
Nowadays, there are quite a few standalone VR devices powered by various companies–Microsoft’s HoloLens, Apple’s Vision Pro, Magic Leap (which one can argue is not a standalone device as the compute engine is housed outside the headset) and many others; however, VR still has not reached its full potential.
The Challenges of Creating Immersive Experiences
There are quite a few technical challenges to overcome in order to create a truly immersive VR experience.
Accurate tracking and motion detection: VR systems must capture user movements precisely. This requires the development of advanced tracking technologies, such as inside-out tracking, capable of translating real-world actions into the virtual environment seamlessly. My Ph.D dissertation was in this area, matching images to compute the user eye position, which gave me a real appreciation of the mathematical and computing complexity involved in solving this issue.
Motion to Photon Latency: Another key problem in generating a realistic VR experience is “Motion to Photon Latency.” In real life, when someone moves their gaze, the new scene appears seemingly instantaneously. However, in a VR headset, the compute engine needs to detect the user’s new gaze, compute the difference from the previous location, and render the new scene in less than 18 milliseconds to avoid ill effects like nausea and dizziness. This requires a very low latency sensor system and compute engine, along with a powerful graphics processor and clever rendering technology.
Screen Door Effect: To create a realistic VR experience, the displays used in VR headsets need to have very high resolution (> 4K pixels per eye). If the resolution is not sufficient the eye will notice the spaces between the pixels in the display causing the “Screen Door Effect” — where it looks like we’re viewing a projected image through a screen door. However, these high-resolution displays are power hungry, requiring very high-performance compute and graphics processors with larger memories to drive them.
Input devices: Natural user interfaces are critical to interact with various applications. Many of the VR devices come with accessories such as wireless controllers to enable this. The headset also needs motion sensors to precisely locate and track the users head movements. Some recent devices use visual and infrared cameras to track the user’s hands and eyes to create a more natural interface mechanism. Mixed reality headsets also have video cameras to enable the user to see the environment, in addition to the VR display. The inputs from these sensors must be processed quickly and with low latency for the VR experience to feel natural.
Processing Power: So far, truly immersive experiences have been held back by processing power. It costs hundreds of millions of dollars to develop a dedicated processor that addresses all of these computational requirements adequately. It doesn’t make good business sense to develop a processor for a niche market like VR. Hence, headset manufacturers have been mostly re-purposing existing smartphone processors to enable VR headsets, as these have traditionally been the lowest power processors available. This is one of the reasons the existing VR headsets have not provided a truly immersive experience.
Apple’s Vision Pro
Apple has taken VR to the next level, creating what I believe is the first true mixed reality experience. In the Vision Pro, they have seamlessly blended VR and AR. The cameras are so good, it’s like seeing the real world. The Vision Pro has 12 cameras, five sensors and six microphones. On the outside there are two cameras pointing forwards and two pointing down to track hand movements. There’s also a 3D LiDAR scanner, infrared eye tracking and a TrueDepth camera to take 3D pictures.
Each of the high-resolution micro-OLED (organic light-emitting diode) twin displays contains more pixels than a 4K TV, which minimizes the screen door effect. A recent article in CNET said, “think 4K TV resolutions on the size of postage stamps.”
All Apple apps are integrated with games and video in realistic, high resolution, all of which are powered by monstrous processors. Apple developed this headset with the extremely powerful M2 chip, used for the MacBook. Apple also developed the dedicated R1 chip to process all of the sensor inputs at exceptionally low latency. This two-processor combination with high-performance memories can address the challenges I described above.
Apple has solved all of the problems beautifully — but imagine if it had multi-day battery life.
Why the World Needs a Better Battery
What we need in order to fully realize the potential of mixed reality is a light, slim and safe battery with multi-day capacity. As I mentioned in my article in February, our technology-related experiences are being held back due to a lack of advancements in Li-ion batteries. While we’ve seen tremendous advances in other areas of technology, from processing power to cameras and memory, increases in battery energy density have been comparatively meager.
We’ve come a long way since the first stereoscope. Our team at Enovix is pushing the boundaries and creating a next-gen battery to power the technologies of the future.
In my next article, I’ll share how some of my favorite AI use cases can come to fruition with a better battery.
