Overcoming Difficulties of VR App Development
Overcoming Difficulties of VR App Development
Virtual reality is a gateway to possibilities previously thought impossible. Whether it's gaming or training, see how VR can guide the path to the future, problems included.
Join the DZone community and get the full member experience.Join For Free
“The Grid. A Digital Frontier. I tried to picture clusters of information as they moved through the computer. What did they look like? Ships? Motorcycles? Were the circuits like freeways? I kept dreaming of a world, I thought I'd never see. And then, one day, I got in.” — Kevin Flynn, TRON: Legacy
A fashion trend, a cutting-edge technology, an absolutely new medium, a new art direction, a radical philosophical and lifestyle shift. First of all, virtual reality (VR) is a rapidly growing market. Many researchers state that the VR market is on its way to its own Big Bang with anticipation of a 400% increase in number of active VR users by 2018. These are just some of the reasons that there's such high interest in VR app development recently. So let's look through its traits, features and novelty, its perspectives, and its difficulties and some ways of dealing with them.
Difference from Older Tech
Keep in mind that virtual reality application development is far more complicated than just adding the Oculus Rift plugin to an existing application. There is a big difference between conventional (i.e. display-oriented) software and the software destined for the VR helmets and exo-gloves. The whole concept has drastically changed. Whereas in conventional software development there are years and years of steady progress and expertise, goals are quite known, and, since dealing with simple 2D screens and straight-forward controllers, the nice looking, comfortable-to-use and fast working software is achievable with much fewer resources.
Even modern 3D screens are still standing on their tables, and while providing great, entertaining capabilities, they still do not aim for full-presence territory. Immersion is not possible with conventional displays because of the lack of user movement while watching a movie — user has to sit still and watch the movie and can’t participate in it. The whole idea of an immersive movie allowing users to take part is VR-exclusive.
Clearly seen spatial environments are an additional problem for immersion into active games. Modern wireless motion sensing controllers, Wiimote and Kinect for instance, have enriched playing experience, but nevertheless, while dancing or playing tennis on Xbox or Wii, users have to pretend they are somewhere else while seeing their living rooms around. The VR concept is different, though. The idea is to put the user into an artificial world intentionally isolated from natural reality, giving an illusion of presence, and very importantly, that presence must be full-time.
So how hard could it be? Just a little step from 3D movies, couple new features here, some movement freedom there. Well, it isn’t like that. You see, when creating a VR application of any kind, you fight the strongest power in biological world — evolution. Millions of years shaped our brain and sensory system to become that good in perceiving natural reality, to tell real from fake or even real from realistic. It is not easy, not for a second, or more likely a fraction of one, to trick a human brain with a series of images in order to achieve the holy grail of VR — true immersion.
The thing is, the user does not interact with virtual reality directly, only by means of technical devices: input controllers and HMDs (most common nowadays, with lots of options in the future: domes, rooms, etc.), neither of which the user observes exists in a simple natural way. Virtuality — the observed virtual world — is only there if it was programmed to be, and only when looked at. Since the whole idea is to provide the user a subconscious presence, a detailed and meticulous approach is a must.
The creators of any virtual landscape must take care of each and every detail in order to create a legible, coherent world. It also obliges them to vary their own thinking because their new goal is not realism, but an understandable and crisp virtual environment. Sure there are projects with surreal, phantasmagorical environments, sometimes even grotesque, but still, some rules mimicking natural ones must be proposed to the user, ensuring smooth and comprehendible motion for the great virtual experience.
The Power And the Looks
When discussing VR development opportunities, developers seem seriously concerned about system resources: VR is extremely intensive in terms of computing power, so lots of skill and agility should be put on the altar of effective utilization of computer resources.
And then there is the design. It's a hard job balancing between fantasy and users’ perception. It's a new branch of art, sometimes a way to express company philosophy in a whole new way and the cornerstone of usability design needs a special team and a serious amount of time. This is the price to be paid in the visual perception territory.
And another serious issue: VR now is in the same place television was at the very beginning. The TV was a step forward from radio — it added lots of sensorial information to its predecessor, expanded perception, and all of that needed changes in directing, editing, sound, and cinematography in order to deliver a complete picture. The same is happening now: Transitioning from TV and 3D movies to VR requires both new techniques in all aforementioned areas and a very different set of equipment. It’s rather expensive, it must be said, though VR is obviously a much more interesting and advanced experience compared to television, even despite new Apple's TV apps potential, for example.
The Difficulties and The Solutions
So, as seen above, development of virtual or augmented reality apps raises a number of challenges specific to this relatively young area (we will later see why it is still “young,” despite being around for some time). Let’s name the main technical challenges of VR app development:
Latency, if high, totally kills the presence effect. The human brain would never believe the reality of the pictures shown if the lag between a movement made by user and the actual picture change was higher than certain threshold. Famouslu, John Carmack stated that the world appears stable with fewer than 20ms of lag. As short as 4ms latency is advised by some researchers for a smooth experience and a strong subconscious presence.
In a decent normal display of 60 fps, that translates into almost 17ms of frame-to-frame lag. Meanwhile, both the Oculus Rift and HTC Vive operate with a 90Hz refresh rate, ending up with a bit more than 11ms. It still isn’t the sought 4ms latency, but the difference is clear, and the progress is moving toward higher frame rates.
The latency of the controls, their ergonomics, and abilities are also of extreme importance. They are the only means for the user to interact with virtuality, and their own latency must not hinder the smoothness of delivery. Mind that Kinect has a latency of up to 100ms — this gives the perspective of how advanced those controllers must be.
The solution: New wireless interfaces, improved communication protocols, and faster SoC solutions with faster embedded processors are reducing latency to a keen 40ms, and further reduction is not too far off.
As for higher computing power, both Oculus Rift and HTC Vive operate dual (one per eye) 1080x1200 screens with a 90Hz refresh rate. All the VR systems also run a number of sensors, including accelerometers, photosensors, laser position sensors, etc., and very sophisticated sound systems, as well. This translates to their almost identical minimum system requirements:
Video Card: NVIDIA GTX 970 / AMD R9 290 equivalent or greater
CPU: Intel i5-4590/ AMD FX-8350 equivalent or greater
Memory: 8GB+ RAM
Memory: 4GB+ RAM
Video Output: Compatible HDMI 1.3 video output
Video Output: HDMI 1.4 or DisplayPort 1.2 or newer
USB Ports: 2x USB 3.0 ports plus 1x USB 2.0 port
USB Port: 1x USB 2.0 or greater port
OS: Windows 7 SP1 64 bit or newer
And these are minimum requirements for the current models. The more serious your application gets and the cooler its graphics are, the more resources you'll need to make it work. High FPS graphics in a high resolution and rendering a highly detailed, fully simulated world are both CPU and GPU thirsty. And the successors to the current systems are on the go with even greater resolutions and refresh rates.
As you can see, all the hardware required for the really smooth and deep immersion experience is pretty modern — that is why we consider the VR a relatively young technology. There is an abyss between modern samples and the outdated, cheesy world of 1992's The Lawnmower Man. High demand for computing power spurs the development of more and more potent hardware.
There are more than a couple of VR platforms these days, from the ultra-cheap cardboard headsets for Android devices to the big bunch of multiple third-party headsets, also supporting iPhones, not to leave iOs users out of the fun. And then there are the very famous Oculus Rift, HTC Vive, and a variety of HMDs (including big players like Microsoft HoloLens, Samsung Gear VR, and Sony Computer Entertainment PS VR), and stereoscopic walls. And the number of platforms (the platform fragmentation) is going to grow. That vast variety could be rather painful to adapt to — it all takes time and effort. In this condition, high expertise of developers in cross-platform programming and highly scalable, versatile solutions are the key to success.
All that means that VR apps must be fast. Very fast. With the ever-increasing demand for computing power, good optimization is a must for full-time immersion’s sake. Modern achievements in algorithmics and new rendering techniques are here to provide great progress in this field. Optimization also helps with overcoming computing bottlenecks, providing smoother delivery, and is absolutely necessary in the multi-platform market.
All these hardware and programming issues need some further attention. The next gen of HMDs are going to bear even mightier screens. There is already some demand for 4K VR. It isn't widespread yet, but it will be soon. The hardware vendors, meanwhile, have recognized the VR as a promising trend and have begun developing and releasing hardware with the motto “VR to the masses.”
As such, AMD have recently released a new series of graphic cards, the RX family, with the highest-performing RX 480 claiming a “Premium VR experience” with a starting price of $199.99 (compare to the R9 290 — $299.95 and the GTX 970 — $384.95). Not forgetting the “fragmentation problem,” Intel has just added to that motley crew their newest brainchild — Intel Alloy, an all-in-one virtual reality solution. Alloy differs from competition a lot being a standalone solution. It is a fresh release, so an SDK is anticipated soon. Also Intel promises open APIs in 2017, allowing their partners to create their own branded products from the Alloy design. Industry movers have additionally announced a quite predictable collaboration with Microsoft for the sake of various software optimizations both for Alloy and Windows-based systems.
So far, 2016 has been a very good year for VR. In addition to everything above, there are three more important bits of news marking the future of VR development. They all address latency, computation power, and optimization problems, effectively solving lots of issues in the near future. A couple of them also bring unparalleled help to overcome those aforementioned platform fragmentation troubles, at least on the computer side.
First, Microsoft. The DirectX 12 API collection, launched with Windows 10 and soon becoming the main Microsoft standard, shows serious shifts in gaming performance. DirectX 12 implements support of asynchronous computation, benefitting AMD graphic cards. This isn't surprising, seeing as DirectX 12 absorbed a lot from AMD’s Mantle API. The main advantage it brings to VR is that it eliminates driver overhead, offering developers a previously unavailable low-level hardware access similar to Mantle. It can seriously improve performance and cut latency while allowing simultaneous (!) use of even different brands of graphic cards. Yes, a Radeon and a GeForce working together on one system. DirectX 12 support is not wide yet, but it's spreading. The only obvious limitation here is the platform. DX12 is a Windows 10 exclusive, like the Apple Metal for OS X.
This brings us to another and more recent piece of news. AMD Vulkan. Vulkan has been developed by the famous Khronos group, responsible for the long industry standard OpenGL, with contributions by AMD and Valve. It was released early in 2016. AMD Vulkan is directly based on Mantle, so it utilizes all that asynchronous compute and low-overhead goodness. It also largely improves performance of multicore CPUs, slightly underutilized in the DirectX 11 times.
Vulkan is an open standard and, being a complement to OpenGL (it was even previously addressed by Khronos as "next generation OpenGL initiative"), it offers an unmatched advantage unique in the industry and helping a lot with the computer side of the fragmentation problem. Vulkan is cross-platform and the only high-performance graphic API featuring multi-platform support. Where DirectX 12 and Metal are exclusive to their newest respective systems, Vulkan even supports a bunch of older ones.
Surely, such flexible tech comes at a price — this close to a metal approach requires much higher programming skills and effort due to reduced ability to just rely on the drivers. In exchange, though, there's much more precise control over the hardware.
Finally, there one more bit of news from AMD — a platform specific one, but very promising and directly addressed to VR development. It's AMD LiquidVR.
Quoting its developers: “LiquidVR is an AMD technology initiative aimed at enabling a fully immersive and comfortable virtual reality experience by creating a new programming model that fully exploits the capabilities of modern GPUs, like AMD Radeon products. One of the key technology goals of LiquidVR is to reduce unwanted processing latency (reduce motion-to-photon latency) and deliver a consistent frame rate. AMD’s LiquidVR is ushering in a new era of visual computing, designed to empower developers and fuel fully immersive VR experiences.”
This shows that they are aware of VR development issues and want to help developers around the globe in overcoming them. Especially on their platforms.
Anyway, the LIquidVR SDK is highly potent and implements full support of Radeon’s progressive features, capable of great progress in VR development. Radical performance improvements are promised via use of asynchronous shaders, which are capable of performing compute operations in parallel with rendering tasks. That means reducing lag to previously unreachable levels, all for high immersion’s sake.
Valve (HTC/Valve Vive, mind you) has shown high interest in AMD’s new initiatives, directly stating their preferences to Vulkan over DirectX 12. That is quite understandable. Steam Machines are going to run Debian Linux, derived SteamOS, and won’t have any use for DirectX. Strong support of HTC Vive is obviously included in LiquidVR, and Oculus Rift support is also there (there is a partnership among AMD, Oculus, and Dell). All this gives developers very powerful new tools that would power new, brilliant Virtual Reality applications. Vast improvements of new APIs and SDKs also contribute to the wider spread of VR, improving the experience on slightly weaker hardware.
Now, let's look at some conceptual points.
First comes the incarnation tool. Avatars are the visible representation of a user wearing a head-mounted display. It brings developers to yet another challenge, especially if the avatar’s dimensions differ from those of its user, or if there is a limited number of trackers. What is needed here are carefully programmed and mathematically adjusted relations between avatars and virtual worlds. It's body language translation instead of strict game mechanics, and new interfaces add a lot to user-avatar association.
Interaction with the virtual world is diverse and complex, starting with object manipulation and navigating while ending with custom menus and typing. All these activities must be smooth and natural not to spoil the user experience. A studious approach to ergonomics with close cooperation with the design team is necessary here.
Just compare the design of the original 1982 TRON to the more modern TRON: Legacy of 2010. Or let’s recall Turok, which still looks nice and almost modern (in an HD remastering) due to its thought-through design. Design matters. A lot. There is a solution, though: a skilled and artful dedicated design team.
There are several pieces of advice based on our practice for the above.
Interactions in VR involve both virtual and real controllers. And whereas software developers are not influencing real controllers, virtual ones are their playground. Some VR applications may use rendered copies of real life switches and dashboards, say, for example, training simulators for vehicles, construction or loading equipment, flight simulators, etc.
Simulations of natural behavior are also blessed with easier avatar implementations. For example, a VR game simulating attending a Victorian era gala dinner has a brilliant and uncomplicated way for naturally placing gaming tools (forks and plates this time) in their normal places. Operating them is also quite easy in this example. We strongly recommend using this method over virtual, unseen inventory if possible.
Remember inventory and interacting with it in the distinguished Fallout series? You had to left-click the corresponding button and a window showing your possessions opened. Then you had to manipulate items with your mouse cursor.
In modern VR apps, while simulating natural behavior, we do our best to avoid this model. For instance, in medical simulators for surgeons, it is much better to put a table or a stand with surgical instruments and an NPC nurse. That way, to start an incision, the user would take a scalpel (cyberglove involved) or ask the nurse (voice recognition and natural language interpretation systems required), without addressing a Photoshop-style instrument panel, totally ruining both immersion and the student’s attention.
Still, there are a vast number of applications using virtual reality that do require instrument panels. Those aren’t usually aimed at deep immersion, but rather at the unmatched visual capabilities VR provides for design purposes. They are not games, but rather the likes of architectural, engineering, or interior designing applications. As such, they feature an enormous variety of tools and materials, and representing them on rendered palettes and easels is both effective and good looking. Samples of textures representing wallpapers, for example, are well-depicted as a grid with tiles, much like it would be represented in a design application for a conventional display.
But there are some particular VR-specific properties: That palette would occupy a significantly bigger area in user’s view than on a screen, and cursors are usually of no good in VR apps, so a reticle and little light glow around the current selection will do much better. It is a good idea for that reticle not to be present constantly, but to appear for aiming purposes only corresponding to a user’s actions.
There is also the feedback that is adding a lot to users’ VR experience; many controllers feature haptic feedback that can do well for confirming actions in professional apps and bringing tactile experiences to games. Audio feedback is also a powerful and flexible tool, especially for professional apps. Common decency requires audio feedback to be unintrusive, brief, and polite. Its goal is comfortable confirmation, not ruining an architect’s pleasure of listening to his favorite jazz band while designing new building. You might also visit that center in the future.
VR is a great tech, but, as always, there are some concerns due to its novelty and specifics. First of all, there are certain health and safety worries. Most VR systems show consumer warnings because of possible motion sickness hazards, for instance. Also, philosophical, social, and conceptual questions are brought up by the new tech and its capabilities. Some researchers assume serious changes in human behavior, economics, communication, and cognition. Some even predict “virtual space migration.” The answers are yet to come, but a serious influence on humanity is quite obvious.
Despite all the difficulties, and sometimes due to them, virtual reality is an exciting new tech with nearly endless possibilities. VR is a quantum leap to the future. A leap to be taken.
Opinions expressed by DZone contributors are their own.