Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Dragging and Placing Holograms With Unity

DZone's Guide to

Dragging and Placing Holograms With Unity

If you're into VR development, particularly using Unity, it's helpful to know how to interact with, drag, and place holograms you want without a lot of nervous twitching.

· IoT Zone
Free Resource

Discover why Bluetooth mesh is the next evolution of IoT solutions. Download the mesh overview.

I've been trying to create the effect of what you get in the – in the meantime good old – Holograms app: When you pull a hologram out of a menu, it ‘sticks to your gaze’ and follows it. You can air tap, and then it stays hanging in the air where you left it, but you can also put it on a floor, on a table, or next to a wall. You can’t push it through a surface. That is, most of the time. So, like this:

In the video, you can see that it follows the gaze cursor floating through the air until it hits a wall to the left and then stops, then goes down until it hits the bed, and then stops, then up again until I finally place it on the floor.

A New Year, a New Toolkit

As happens often in the bleeding edge of technology, things tend to change pretty fast. This is also the case in HoloLens country. I have taken the plunge to Unity 5.5 and the new HoloToolkit, which has a few big changes. Things have gotten way simpler since the previous iteration. Also, I would like to point out that for this tutorial, I used the latest patch release.

Setting Up the Initial Project

This is best illustrated by a picture. If you have set up the project, we basically only need this. Both Managers and HologramCollection are simply empty game objects meant to group stuff together. They don’t have any other specific function here. Drag and drop the four blue prefabs in the indicated places, then set some properties for the cube

imageimage

The Cube is the thing that will be moved. Now it’s time for ‘some’ code.

The Main Actors

There are two scripts that play the leading role, with a few supporting roles.

  • MoveByGaze
  • IntialPlaceByTap

The first one makes an object move, the second one actually ends it. Apropos, the actual moving is done by our old friend iTween, whose usefulness and application was already described in part 5 of the AMS HoloATC series. So, you will need to include this in the project to prevent all kinds of nasty errors. Anyway, let’s get to the star of the show, MoveByGaze.

Moving With Gaze

It starts like this:

using UnityEngine;
using HoloToolkit.Unity.InputModule;
using HoloToolkit.Unity.SpatialMapping;

namespace LocalJoost.HoloToolkitExtensions
{
    public class MoveByGaze : MonoBehaviour
    {
        public float MaxDistance = 2f;
        public bool IsActive = true;
        public float DistanceTrigger = 0.2f;
        public BaseRayStabilizer Stabilizer = null;
        public BaseSpatialMappingCollisionDetector CollisonDetector;

        private float _startTime;
        private float _delay = 0.5f;
        private bool _isJustEnabled;
        private Vector3 _lastMoveToLocation;
        private bool _isBusy;

        private SpatialMappingManager MappingManager
        {
            get { return SpatialMappingManager.Instance; }
        }

        void OnEnable()
        {
            _isJustEnabled = true;
        }

        void Start()
        {
            _startTime = Time.time + _delay;
            _isJustEnabled = true;
            if (CollisonDetector == null)
            {
                CollisonDetector = 
                  gameObject.AddComponent<DefaultMappingCollisionDetector>();
            }
        }
    }
}

Up above are the settings:

  • MaxDistance is the maximum distance from your head the behavior will try to place the object on a surface. Further than that, and it will just float in the air.
  • IsActive determines whether the behavior is active (duh).
  • DistanceTrigger is the distance your gaze has to be from the object you are moving before it actual starts to move. It kind of trails your gaze. This prevents the object from moving in a very nervous way.
  • Stabilizer is the stabilizer made, used, and maintained by the InputManager. You will have to drag the InputManager from your scene on this field to use the stabilizer. It’s not mandatory, but it's highly recommended
  • CollisionDetector is a class we will see later – it basically makes sure the object that you are dragging is not pushed through any surfaces. You will need to add a collision detector to the game object that you are dragging along – or maybe a game object that is part of the game object that you are dragging. That collision detector then needs to be dragged on this field by the MoveByGaze. This is not mandatory. If you don’t add one, the object you attach the MoveByGaze to will just simply follow your gaze and move right through any object. That’s the work of the DefaultMappingCollisionDetector, which is essentially a null pattern implementation. 

Anyway, in the Update method all the work is done:

void Update()
{
    if (!IsActive || _isBusy || _startTime > Time.time)
        return;
    _isBusy = true;

    var newPos = GetPostionInLookingDirection();
    if ((newPos - _lastMoveToLocation).magnitude > DistanceTrigger || _isJustEnabled)
    {
        _isJustEnabled = false;
        var maxDelta = CollisonDetector.GetMaxDelta(newPos - transform.position);
        if (maxDelta != Vector3.zero)
        {
            newPos = transform.position + maxDelta;
            iTween.MoveTo(gameObject,
                iTween.Hash("position", newPos, "time", 2.0f * maxDelta.magnitude,
                    "easetype", iTween.EaseType.easeInOutSine, "islocal", false,
                    "oncomplete", "MovingDone", "oncompletetarget", gameObject));
            _lastMoveToLocation = newPos;
        }
        else
        {
            _isBusy = false;
        }
    }
    else
    {
        _isBusy = false;
    }
}

private void MovingDone()
{
    _isBusy = false;
}

Only if the behavior is active, not busy, and the first half second is over we are doing anything at all. And the first thing is – telling the world we are busy indeed. This method, like all Updates, is called 60 times a second and we want to keep things a bit controlled here. Race conditions are annoying.

Then we get a position in the direction the user is looking, and if that exceeds the distance trigger – or this is the first time we are getting here – we start off finding how far ahead along this gaze we can place the actual object by using CollisionDetector. If that’s possible – that is, if the CollisionDetector does not find any obstacles — we can actually move the object using iTween. It's important to note that whenever the move is not possible, _isBusy immediately gets set to false. Also, note the fact that the smaller the distance, the faster the move. This is to make sure the final tweaks of setting the object in the right place don’t take a long time. Otherwise, _isBusy is only reset after a successful move.

Then the final pieces of this behavior:

private Vector3 GetPostionInLookingDirection()
{
    RaycastHit hitInfo;

    var headReady = Stabilizer != null
        ? Stabilizer.StableRay
        : new Ray(Camera.main.transform.position, Camera.main.transform.forward);

    if (MappingManager != null &&
        Physics.Raycast(headReady, out hitInfo, MaxDistance, MappingManager.LayerMask))
    {
        return hitInfo.point;
    }

    return CalculatePositionDeadAhead(MaxDistance);
}

private Vector3 CalculatePositionDeadAhead(float distance)
{
    return Stabilizer != null
        ? Stabilizer.StableRay.origin + 
             Stabilizer.StableRay.direction.normalized * distance
        : Camera.main.transform.position + 
            Camera.main.transform.forward.normalized * distance;
}

GetPostionInLookingDirection first tries to get the direction in which you are looking. It tries to use the Stabilizer’s StableRay for that. The Stabilizer is a component of the InputManager that stabilizes your view – and the cursor uses it as well. This prevents the cursor from wobbling too much when you don’t keep your head perfectly still (which most people don’t – this includes me). The stabilizer takes an average movement of 60 samples, and that makes for a much less nervous-looking experience. If you don’t have a stabilizer defined, it just takes your actual looking direction – the camera’s position and your looking direction.

Then it tries to see if the resulting ray hits a wall or a floor – but no further than MaxDistance away. If it sees a hit, it returns this point, if it does not, if gives a point in the air MaxDistance away along an invisible ray coming out of your eyes. That’s what CalculatePositionDeadAhead does – but also trying to use the Stabilizer first to find the direction.

Detect Collisions

Okay, so what is this famous collision detector that prevents stuff from being pushed through walls and floors, using the spatial perception that makes the HoloLens such a unique device? It’s actually very simple, although it took me a while to actually get it this simple.

using UnityEngine;

namespace LocalJoost.HoloToolkitExtensions
{
    public class SpatialMappingCollisionDetector : BaseSpatialMappingCollisionDetector
    {
        public float MinDistance = 0.0f;

        private Rigidbody _rigidbody;

        void Start()
        {
            _rigidbody = GetComponent<Rigidbody>() ?? gameObject.AddComponent<Rigidbody>();
            _rigidbody.isKinematic = true;
            _rigidbody.useGravity = false;
        }

        public override bool CheckIfCanMoveBy(Vector3 delta)
        {
            RaycastHit hitInfo;
            // Sweeptest wisdom from 
            //http://answers.unity3d.com/questions/499013/cubecasting.html
            return !_rigidbody.SweepTest(delta, out hitInfo, delta.magnitude);
        }

        public override Vector3 GetMaxDelta(Vector3 delta)
        {
            RaycastHit hitInfo;
            if(!_rigidbody.SweepTest(delta, out hitInfo, delta.magnitude))
            {
                return KeepDistance(delta, hitInfo.point); ;
            }

            delta *= (hitInfo.distance / delta.magnitude);
            for (var i = 0; i <= 9; i += 3)
            {
                var dTest = delta / (i + 1);
                if (!_rigidbody.SweepTest(dTest, out hitInfo, dTest.magnitude))
                {
                    return KeepDistance(dTest, hitInfo.point);
                }
            }
            return Vector3.zero;
        }

        private  Vector3 KeepDistance(Vector3 delta, Vector3 hitPoint)
        {
            var distanceVector = hitPoint - transform.position;
            return delta - (distanceVector.normalized * MinDistance);
        }
    }
}

This behavior first tries to find a RigidBody and, failing that, adds it. We will need this to check the presence of anything ‘in the way’. But – this is important – we will set ‘isKinematic’ to true and ‘useGravity’ to false, or else our object will come under the control of the Unity physics engine and drop on the floor. In this case, we want to control the movement of the object.

So, this class has two public methods (its abstract base class demands that). One, CheckIfCanMoveBy (that we don’t use now), just says if you can move your object in the intended direction over the intended distance without hitting anything. The other essentially does the same, but if it finds something in the way, it also tries to find a distance over which you can move in the desired direction. For this, we use the SweepTest method of RigidBody. Essentially, you give it a vector, a distance along that vector, and it has an out variable that gives you info about a hit, should any occur. If a hit does occur, it tries at again at 1/4th, 1/7th, and 1/10th of that initially found distance. Failing everything, it returns a zero vector. By using this rough approach, an object moves quickly in a few steps until it can't any longer.

And then it also moves the object back over a distance you can set from the editor. This keeps the object just a little above the floor or from the wall, show that be desired. That’s what KeepDistance is for.

The whole point of having a base class BaseSpatialMappingCollisionDetector, by the way, is a) enabling null pattern implementation which as implemented by DefaultMappingCollisionDetector and b) making different collision detectors based upon different needs. It's a bit of architectural consideration within the sometimes-bewildering universe of Unity development.

Making It Stop: InitialPlaceByTap

Making the MoveByGaze stop is very simple – set the IsActive field to false. Now we only need something to actually make that happen. With the new HoloToolkit, this is actually very, very simple:

using UnityEngine;
using HoloToolkit.Unity.InputModule;

namespace LocalJoost.HoloToolkitExtensions
{
    public class InitialPlaceByTap : MonoBehaviour, IInputClickHandler
    {
        protected AudioSource Sound;
        protected MoveByGaze GazeMover;

        void Start()
        {
            Sound = GetComponent<AudioSource>();
            GazeMover = GetComponent<MoveByGaze>();

            InputManager.Instance.PushFallbackInputHandler(gameObject);
        }

        public void OnInputClicked(InputEventData eventData)
        {
            if (!GazeMover.IsActive)
            {
                return;
            }

            if (Sound != null)
            {
                Sound.Play();
            }

            GazeMover.IsActive = false;
        }
    }
}

By implementing IInputClickHandler, the InputManager will send an event to this object when you air tap and it is selected by gaze. But by pushing it as the fallback handler, it will get this event also when it’s not selected. The event processing is pretty simple – if the GazeMover in this object is active, it’s de-activated. Also, if there’s an AudioSource detected, its sound is played. I very much recommend this kind of audio feedback.

Wiring It All Together

On your cube, you drag the MoveByGaze, SpatialMappingCollisionDetector, and InitialPlaceByTap scripts. Then you drag the cube itself again on the CollisionDetector field of MoveByGaze, and the InputManager on the Stabilizer field. Unity itself will select the right component.

image

So, in this case, I could also have used GetComponent<SpatialMappingCollisionDetector> instead of a field where you need to drag something on. But this way is more flexible – in-app, I did not want to use the whole object’s collider, but only that of a child object. Note that I have set the MinDistance for the SpatialMappingCollisionDetector for 1 cm – it will keep an extra centimeter distance from the wall or the floor.

Concluding Remarks

So this is how you can more or less replicate part of the behavior of the Holograms App, by moving objects around with your gaze and placing them on surfaces using air tap. The unique capabilities of the HoloLens allow us to place objects next to or on top of physical objects, and the new HoloToolkit makes using those capabilities pretty easy.

Full code, as per my MVP ‘trademark’, can be found here.

Take a deep dive into Bluetooth mesh. Read the tech overview and discover new IoT innovations.

Topics:
iot ,unity ,hololens ,game development

Published at DZone with permission of Joost van Schaik, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}