Migrating to MRKT2: Interaction With Irregular or Complex Objects

Interaction with objects that are not simple shapes like cubes, spheres, capsules, etc. poses some challenges. The Mixed Reality Toolkit 2 offers some great components, but they all require a top-level collider. Now, consider the helicopter above.

It consists of a lot of small objects. Default: It does not even have a collider. You cannot add a "Near Interaction Touchable" on top of the object because it simply cannot find a collider. Now, you can generate those on import, but that makes the object kind of heavy with regards to required processing power, and hooking all those colliders up to their own "Interaction Touchable" is a lot of work.

There is a simpler way of doing that, fortunately. Actually, there are two variants of this, but I am going to show the one I think works the best (and is the most beautiful).

Adding a Colliding 'Catcher' to Rule Them All

First of all, we are going to add a surrounding object inside the model itself. I took a capsule, as this gives IMHO the most beautiful result

The result now is the helicopter is now almost completely covered in what looks like a giant suppository, which is definitely not what you want.

So by fiddling around, I created this material. Note: Its actual color is fully transparent black — basically 0,0,0,0. so that we can see the helicopter again.

But more importantly, it has a hover light override color of green with a light intensity of 0.4.

And now, if a cursor strikes the object, you get this what I think is rather pretty ghostly glow indicating this is your focused object.

Making it Interactable

Making it actually interact with events is now pretty simple. Assuming your actual 'controller' behavior needs to control the whole game object, it needs to sit on top — so it can do more things than just interaction (like moving the helicopter, for instance, which it does not do now). It looks like this:

using System;
using TMPro;
using UnityEngine;

public class InteractionResponder : MonoBehaviour
{
    [SerializeField]
    private TextMeshPro _text;

    private int _timesClicked;

    private int _timesTouched;

    private int _timesFocus;

    public void Click()
    {
        _timesClicked++;
        UpdateText();
    }

    public void TouchStart()
    {
        _timesTouched++;
        UpdateText();
    }

    public void OnFocus()
    {
        _timesFocus++;
        UpdateText();
    }

    private void UpdateText()
    {
        _text.text = string.Format("Clicked: {0}{1}Touched: {2}{1}Focused: {3}", 
            _timesClicked, Environment.NewLine, _timesTouched, _timesFocus);
    }
}

Super simple, basically, there are only three event response methods that can be called, and it tries to display the text in a TextMeshPro object that also sits in the HologramCollection, just like the Helicopter itself. As I said, the InteractionResponder behavior will be sitting on the helicopter object itself:

Now, we need to go back to the SurroundingCapsule and add an "Interactable" and a "Near Interaction Touchable Volume" script to that. The latter is apparently new or something I missed: where an ordinary "Near Interaction Touchable" only takes a rectangular collider, the Volume script also takes a capsule:

Then, you will need to select "Select" for Input Actions, then drag the helicopter in the little box under OnClick, like you usually do in this kind of event hookup, and select ÏnteractionResponder.Click.

Then, you click "Add Event", select "InteractableOnTouchReceiver" and hook the On Touch event up to the InteractionResponder.TouchStart  method (we will ignore the "On Touch End" event in this sample).

In a similar fashion, you will add another event, select "InteractableOnFocusReceiver" and hook that to the InteractionResponder.OnFocus  event.

And if you have done everything right, and hit the play button in the editor, you will see this happening when you click, touch or focus:

Conclusion

With very little code and one extra component, you can make an irregular and complex object like a helicopter have an easy to touch/focus/click target that also gives some subtle but very clear visual feedback about what is happening. And this is only a start; we might as well have it respond to touch by showing some direct feedback, or show that it knows it's focused, even without a cursor hitting it (eye gaze!). Life is going to get interesting and much more immersive once HoloLens 2 comes around!

The demo project can be found here.

Further Reading

Migrating to MRTK2: Integrating With the Spacial Map

Migrating to MRTK2: Missing Singleton and 3DTextPrefab