Over a million developers have joined DZone.

Running an MR App on ‘Ordinary’ PCs (With an Xbox One Controller)

DZone's Guide to

Running an MR App on ‘Ordinary’ PCs (With an Xbox One Controller)

Want to run a mixed reality app on a plain PC? Great for demonstrations, see how you can control MR apps with an Xbox controller.

· IoT Zone ·
Free Resource

Let’s face it – although Windows' Mixed Reality has seen a steady uptick (at least I think I can draw that conclusion from the increasing download numbers of my two mixed reality apps in the Windows Store) – not everyone has a mixed reality headset, or even has a PC capable of supporting that. Time will take care of that soon enough. In the meantime, as a mixed reality developer, you might want to show all 700 million Windows 10 users a glimpse of your app, instead of ‘only’ the HoloLens and mixed reality headset owners out there. Even in a reduced state, it gives you eyeballs, and it might entice them to get themselves a headset after all. It’s not like they are expensive these days.

Does This Sound Familiar?

Well, it should. This is far from original. I have been down this road before, describing how to run a HoloLens app on a Raspberry PI2. That’s the U in UWP for you. Only now, we are going to run on a full PC – in my case, a Surface Pro 4. That’s a sufficiently high-end device for a nice experience, but it predates the Windows Mixed Reality era by almost two years and does not support it. But you can’t walk around without a headset, so we will need another means to change our point of view.

Parts List

  • One reasonably nice performing PC not capable of supporting mixed reality – or at least with the Mixed Reality /portal not installed
  • Unity 2017.2.1p2
  • The Mixed Reality Toolkit 
  • One XBox One controller

The first point is important – for if you have the portal installed, your PC will launch it like a good boy trying to do the logical thing, and you won’t see the effect I am trying to show you.

Setting Up the Project

I created a new project in Unity, copied in the latest Mixed Reality Toolkit, then clicked the three menu options under Mixed Reality Toolkit/Configure.

Then I added my standard empty game objects — “Managers” (with nothing in it)  and “HologramCollection” with a cube and a sphere — to have something to see:


There is more to those two objects than meets the eye, but we will get to that later.

Control the Point of View Using an XBox Controller

There’s a simple class for that in my ever-growing HolotoolkitExtensions that starts like this:

using HoloToolkit.Unity.InputModule;
using UnityEngine;

namespace HoloToolkitExtensions.Utilities
    public class XBoxControllerAppControl : MonoBehaviour, IXboxControllerHandler
        public float Rotatespeed = 0.6f;
        public float MoveSpeed = 0.05f;
        public float TriggerAccerationFactor = 2f;

        private Quaternion _initialRotation;
        private Vector3 _initialPosition;

        private readonly DoubleClickPreventer _doubleClickPreventer = 
                                                new DoubleClickPreventer();
        void Start()
            _initialRotation = gameObject.transform.rotation;
            _initialPosition = gameObject.transform.position;

I tend to offer settings to the Unity editor as much as possible to make it easy to reuse this class and adapt its behavior without code changes. Here, I offer some speed settings. You can set the max rotation speed, the max speed the camera moves, and the ‘speed up factor’ that is applied to all values when the right trigger is pressed. Be advised these are all analog values between 0 and 1, so you can control the speed any way by varying the amount of pressure you apply to the sticks or the D-pad. But sometimes, you just wanna go fast, hence the trigger. Also, notice how initial rotation and position are retained.

The main routine is, of course, OnXboxInputUpdate, as the IXboxControllerHandler mandates its presence.

public void OnXboxInputUpdate(XboxControllerEventData eventData)
    if (!UnityEngine.XR.XRDevice.isPresent)
        var speed = 1.0f + TriggerAccerationFactor * eventData.XboxRightTriggerAxis;

        gameObject.transform.position += eventData.XboxLeftStickHorizontalAxis * 
                                         gameObject.transform.right * MoveSpeed * speed;
        gameObject.transform.position += eventData.XboxLeftStickVerticalAxis * 
                                         gameObject.transform.forward * MoveSpeed * speed;

            eventData.XboxRightStickHorizontalAxis * Rotatespeed * speed);
            -eventData.XboxRightStickVerticalAxis * Rotatespeed * speed);

            eventData.XboxDpadHorizontalAxis * Rotatespeed * speed);

        var delta = Mathf.Sign(eventData.XboxDpadVerticalAxis) * 
                    gameObject.transform.up * MoveSpeed * speed;
        if (Mathf.Abs(eventData.XboxDpadVerticalAxis) > 0.0001f)
            gameObject.transform.position += delta;

        if (eventData.XboxB_Pressed)
            if (!_doubleClickPreventer.CanClick()) return;
            gameObject.transform.position = _initialPosition;
            gameObject.transform.rotation = _initialRotation;


Let’s unpack that a little.

The if (!UnityEngine.XR.XRDevice.isPresent) is important. We only want this behavior to do its work when there is no headset present whatsoever – no Mixed Reality headset, no HoloLens.

  • First, we calculate a possible ‘speed up factor’ to be applied when the trigger is used. If it is not, it’s simply 1 and has no effect on the actual movement or rotation.
  • The left stick is used for movement in the ‘horizontal’ plane – forward, backward, left, right. Be aware the axes are relative. So if you are rotated 45 degrees left and you move left, you will move 45 degrees left. It’s actually logical – your frame of reference is always yourself, not some random rotation that happened to be in place when you get somewhere.
  • The right stick is used for rotation around your top and horizontal axes (left to right). Moving it to the right will make you spin to the right (I negate the actual value coming from the stick as you can only rotate a game object around its left axis), pushing it forward will make you look at the floor.
  • That leaves moving up and down and rotating left and right. The D-pad fills the voids: Pushing it left or right will make you rotate sideways (like you are falling to the left or right), and pushing it up or down will make your point of view move up or down.

This is exactly the way it works when you use an Xbox Controller to steer the Unity editor in play mode. The D-pad feels a bit counter-intuitive to me, but when you try to move in three dimensions using sticks that move both in only two dimensions, you will need something extra, and the D-pad is the only thing left. It feels odd to me, but it works.

Then, finally, the B button – when you press that, you get back to your initial position. This is very useful if you have messed around a bit too much and completely lost track of where you are. And that is mostly all of it.

A Tiny Bit of SOLID

protected virtual void HandleCustomAction(XboxControllerEventData eventData)

Hardly worth mentioning, but should you want to add your own logic handling controller buttons or triggers, you can make a child class of this XBoxControllerAppControl and override this method. It’s a hook that makes it open for extension, but keeps its own logic intact. That’s better than making OnXboxInputUpdate virtual because that enables you to interfere with the existing logic by not calling the base OnXboxInputUpdate. It’s the O of SOLID.image

How to Use It

Simply drag it to the MixedRealityCameraParent, change the settings to your liking, and you are done. I think I took some reasonable default settings.

But Wait, There’s more!

I have found that the Xbox controller buttons tend to stutter – that is, they sometimes fire repeatedly, and rapid-fire events can make a bit of a mess.

So I created this little helper, DoubleClickPreventer, that is not exactly rocket science, but very useful:

using UnityEngine;

namespace HoloToolkitExtensions.Utilities
    public class DoubleClickPreventer
        private readonly float _clickTimeOut;

        private float _lastClick;

        public DoubleClickPreventer(float clickTimeOut = 0.1f)
            _clickTimeOut = clickTimeOut;

        public bool CanClick()
            if (!(Time.time - _lastClick > _clickTimeOut))
                return false;
            _lastClick = Time.time;
            return true;

It’s rather simple: whenever the method CanClick is called, a time is set. If the method is called twice within 0.1 seconds, it returns false. Otherwise, it returns true. It’s actually used twice within this sample: It’s also in the little helper class “SelectorDemo” that makes the sphere and the cube go “plonk” and flash blue when you click them using the Xbox's “A” button. I won’t go into that – you can find it in the demo project, and it’s inner workings are left as an exercise to the reader.

And it Looks Like…

There are a few things you might notice. First of all, I apparently am able to select something, but I never coded for it. That’s courtesy of the Mixed Reality Toolkit – your Xbox Controller’s “A” button is acting the same as saying “Select” in a Mixed Reality app while you are gazing at something, air tapping while using a HoloLens, or pointing your Mixed Reality controller to an object and pressing the trigger.

Also, you might notice this at the end of the video:


A clear sign Windows is not really happy with this. It figures – because if nothing prevents you from downloading an app that simply does not work on your machine — it might disgruntle users. But still, the app launches and seems to work.

Some Other Things to Consider

  • Use the right Unity version: 2017.2.1p2. That’s the one that goes with this release of the Mixed Reality Toolkit. when using newer versions of Unity or the toolkit (like the development branch), I got results varying from the app not wanting to compile, crashing, or simply not starting. I also got just this “Can’t open app” dialog and nothing else.
  • You can also see (very small) “Development build” in the lower right corner. There’s a checkbox in Unity that everyone tells you to use, and then that text will go away. The trouble is, that does not work. What will make it go away is building the app with the Master configuration. That and only that. For mixed reality apps, this checkbox apparently only is there for show. At least as far as this post is concerned, and as far as I can see!


  • And finally, when making these apps run on an ordinary PC, you might want to rethink the UI a bit at places. Floating menus, which are very cool in real mixed reality environments, can be really hard to use on a flat screen, for instance. Also, placing things on top of the ‘floor’ might be a bit of a challenge without a floor – even a virtual one.

Concluding words

I am not sure if this will continue working going forward with the MRTK and Unity, how useful this will be in the real world, or if that the MR team even appreciates this approach. I am simply showing you what’s possible and a possible way to tackle this. Your mileage may vary, very much in fact. Have fun!

Once again – demo project here.

iot ,mixed reality app development ,xbox controller ,unity ,tutorial

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}