Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

How to Place Objects on Top or in Front of Obstructions in MR-based Apps

DZone's Guide to

How to Place Objects on Top or in Front of Obstructions in MR-based Apps

Want to learn more about developing with obstructions in your Mixed Reality apps? Check out this tutorial to learn how to place objects with obstructions.

· IoT Zone ·
Free Resource

Introduction

In my previous post, I described some complicated calculations using a BoxCastAll method to determine how to place a random object on top of or in front of some obstruction in the looking direction of the user, be it another object or the spatial mesh. Because the post was long enough as it was, I described the calculations separately. They are in an extra method called GetObjectBeforeObstruction in my HoloToolkitExtension, LookingDirectionHelpers. And, I wrote a very simple Unity behavior to show how it could be used. But, that behavior simply polls every so many seconds (two seconds is the default setting) where the user looks, then it calls GetObjectBeforeObstruction and moves the object there. This gives a kind of nervous result. I promised a more full-fledged behavior, and here it is: the AdvancedKeepInViewController. It’s basically sitting in a project that looks remarkably like the demo project in the previous post, with the same scenery, only there’s a fourth element that you can toggle using the T button or by saying “Toggle.”

image

Features

  • Only moves the object if the head is rotated more than a certain number of degrees per second or the user moves a certain number of meters per second. It uses the CameraMovementTracker from Walk the World that I described in an earlier post.
  • Optionally fine-tunes the location where the object is placed after doing an initial placement (effectively doing a BoxCastAll twice per movement).
  • Optionally scales the object to have a more or less constant viewing size. This is indented for billboards like objects, for example, floating screens.
  • Optionally makes an object appear right in front of the user if it's enabled, instead of moving it in view the first time from the last place where it was before it got disabled.
  • Optionally makes the object disappear when the user is moving a certain number of meters per second to prevent objects from blocking the view or providing distractions. This is especially useful when running an app in a HoloLens, while you are on a factory floor where you really want to see things like handrails, electricity cables, or holes in the floor (possibly with a smelter in it).

The code is not that complicated, but I thought it best to explain it step-by-step. I skip the part where all the editor-settable properties are listed. You can find them in the AdvancedKeepInViewController's source in the demo project. I have added an explanatory tooltip description to almost all of them.

Starting Up

The start is pretty simple:

void Start()
{
    _objectMaterial = GetComponentInChildren<Renderer>().material;
    _initialTransparency = _objectMaterial.color.a;
}

void OnEnable()
{
    _startTime = Time.time + _delay;
    DoInitialAppearance();
    _isJustEnabled = true;
}

private void DoInitialAppearance()
{
    if (!AppearInView)
    {
        return;
    }

    _lastMoveToLocation = GetNewPosition();
    transform.position = _lastMoveToLocation;
}


We get the material of the first Render's material. We can find it's initial transparency, as we need be able to revert to that later. Then, we need to check if the user has selected to object to initially appear in front and, if so, perform the initial appearance. At the end, you see GetNewPosition being called, that's a simple wrapper around  LookingDirectionHelpers.GetObjectBeforeObstruction. It tries to project the object to hit an obstruction at a certain max direction; if there is no obstruction in that range, just give a point at the maximum distance. Since it's called multiple times and I am lazy, I made a little method of it:

private Vector3 GetNewPosition()
{
    var newPos = LookingDirectionHelpers.GetObjectBeforeObstruction(gameObject, MaxDistance,
        DistanceBeforeObstruction, LayerMask, _stabilizer);
    if (Vector3.Distance(newPos, CameraCache.Main.transform.position) < MinDistance)
    {
        newPos = LookingDirectionHelpers.CalculatePositionDeadAhead(MinDistance);
    }
    return newPos;
}


Moving Around

The main thing is, of course, driven by the Update loop. The Update method, therefore, is the heart of the matter:

void Update()
{
    if (_startTime > Time.time)
        return;
    if (_originalScale == null)
    {
        _originalScale = transform.localScale;
    }

    if (!CheckHideWhenMoving())
    {
        return;
    }

    if (CameraMovementTracker.Instance.Distance > DistanceMoveTrigger ||
        CameraMovementTracker.Instance.RotationDelta > DeltaRotationTrigger || 
        _isJustEnabled)
    {
        _isJustEnabled = false;
        MoveIntoView();
    }
#if UNITY_EDITOR
    if (_showDebugBoxcastLines)
    {
        LookingDirectionHelpers.GetObjectBeforeObstruction(gameObject, MaxDistance,
            DistanceBeforeObstruction, LayerMask, _stabilizer, true);
    }
#endif
}


After the startup timeout (0.1 seconds) has expired, we gather the original scale of the object (needed if we actually scale). If the user is moving fast enough, you can hide the object and stop it from doing anything. Otherwise, use the CameraMovementTracker that I wrote about two posts ago to determine if the user has moved or rotated enough to warrant a new location for the object (and the first time the code gets here, repositioning should happen anyway). Then, it simply shows the Box Cast debug lines that I already extensively showed off in this previous post.

So, the actual moving around is done by these two methods (using — once again — good ol' LeanTween), and the second one is pretty funky indeed:

private void MoveIntoView()
{
    if (_isMoving)
    {
        return;
    }

    _isMoving = true;
    var newPos = GetNewPosition();
    MoveAndScale(newPos);
}

private void MoveAndScale(Vector3 newPos, bool isFinalAdjustment = false)
{
    LeanTween.move(gameObject, newPos, MoveTime).setEaseInOutSine().setOnComplete(() =>
    {
        if (!isFinalAdjustment && EnableFineTuning)
        {
            newPos = GetNewPosition();
            MoveAndScale(newPos, true);
        }
        else
        {
            _isMoving = false;
            DoScaleByDistance();
        }
    });
    _lastMoveToLocation = newPos;
}


So, the move MoveIntoView method first checks to see if a move action is not already initiated. Then, it gets a new position using GetNewPositionand calls  MoveAndScaleMoveAndScale  moves the object to its new position, then it calls itself an extra time. The idea behind this is as follows: the actual bounding box of the object might have changed between the original cast in MoveIntoView and the eventual positioning, if the object you move is locked to be looking at the Camera while it moved, using something like the Mixed Reality Toolkit's BillBoard or (as in my sample) my very simple LookAtCamera behavir . A second 'fine-tuning' call is done, using the isFinalAdjustment parameter. And, if we are done moving, we do some scaling. And, this looks like this:


You might also notice that the cubes appear from the camera’s origin, this means that the floating screen initially appears in the right place. This is another option that you can select.

Scale It Up — Or Down

For an object like a floating screen with text, you might want to ensure readability. So, if your text is projected too far away, it might become unreadable. If it is projected too close, the text might become huge. Either way, the user can only see a small portion of it — and, effectively, it's unreadable, too. Hence, this little helper method:

private void DoScaleByDistance()
{
    if (!ScaleByDistance || _originalScale == null || _isScaling)
    {
        return;
    }
    _isScaling = true;
    var distance = Vector3.Distance(_stabilizer ? _stabilizer.StablePosition : 
        CameraCache.Main.transform.position,
        _lastMoveToLocation);
    var newScale = _originalScale.Value * distance / MaxDistance;
    LeanTween.scale(gameObject, newScale, MoveTime).setOnComplete(() => _isScaling = false);
}


I think this only makes sense for 'text screens,' not for 'natural' objects. Therefore, it's an option you can turn off in the editor. But, if you do turn it on, it determines the scale by multiplying the original scale by the distance divided by the MaxDistance, assuming that is the distance that you want to see your object on with its original scale as defined in the editor. Be aware that the autoscaling can make the screen appear inside other objects again, so use wisely and with caution.

Fading Away When Necessary

This method should return false whenever the object is faded out, or fading in or out — that way, MoveIntoView does not get called by Update.

private bool CheckHideWhenMoving()
{
    if (!HideWhenMoving || _isFading)
    {
        return true;
    }
    if (CameraMovementTracker.Instance.Speed > HideSpeed &&
        !_isHidden)
    {
        _isHidden = true;
        StartCoroutine(SetFading());
        LeanTween.alpha(gameObject, 0, FadeTime);
    }
    else if (CameraMovementTracker.Instance.Speed <= HideSpeed && _isHidden)
    {
        _isHidden = false;
        StartCoroutine(SetFading());
        LeanTween.alpha(gameObject, _initialTransparency, FadeTime);
        MoveIntoView();
    }

    return !_isHidden;
}

private IEnumerator SetFading()
{
    _isFading = true;
    yield return new WaitForSeconds(FadeTime + 0.1f);
    _isFading = false;
}


Basically, this method says explains how to use this if it should be hidden at high speed and is not already fading in or out:

  • If the user’s speed is higher than the threshold value and the object is visible, hide it.
  • If the user’s speed is lower than the threshold value and the object is invisible, show it.

The way of hiding and showing is once again done with  LeanTween, but I found that using the .setOnComplete was a bit unreliable for detecting when the fading in or out came to an end. So, I simply use a coroutine that sets the  blocking _isFading, and wait a wee bit longer than the  FadeTime. Then, you will throw the clears _isFading again. That way, no multiple fades can start or stop.

The Tiny Matter of Transparency

The HideWhenMoving feature has a dependency. For it to work, the material needs to support transparency. That is to say, it’s rendering mode needs to be set to transparent (or at least not opaque). As you move around quickly, the semi-transparent box and the double boxes will fade out nicely:


But, if you move around and the floating screen wants to fade, you will see only the text fade out – the window outline stays visible. This has a simple explanation — the material’s rendering mode is set as opaque, not transparent

image

The background of the screen with the button fades out nicely, though, because it uses a different material. Actually, it uses a copy, but with only the rendering mode set to transparent:

image

If you look really carefully, you will see that the entire screen does not fade out. Part of the button screen seems to remain visible. The culprit is the button’s backplate:

image

Now, it’s up to you. You can change the opacity of this material, and, then, it will be fixed for all buttons. The problem is that this material is part of the Mixed Reality Toolkit. So, if you update that, it will most likely be overwritten. And, then, you will have to keep track of changes like this. Or, you can manually change every backplate of every button or do that once and make your own prefab button. There are multiple ways to roam in this case.

Nice — All Those Unity Demos…

But, how does it look in real life? Well, like in this video. First, it shows you that the large semi-transparent cube actually disappears, if you move quickly enough. Then, it shows the moving and scaling of the "Hello World" screen, but it also shows that, when you move quickly enough, it will try to fade, but only the text will fade. The two cubes show nothing special other than that they appear more or less on the spatial mesh, and the "Screen with button" shows shrinking and growing, as well. It will fade completely, except for the backplate, but I have told you how to fix that.

Some Tidbits

If you try to run the project in an HoloLens or Immersive Headset and wonder where the cubes, capsule, and other 'scenery' is located, that is clearly visible in the Unity editor. They are explicitly turned off by something called the HideInRuntime behavior that sits in the "Scenery" game object, where all the scenery resides it. This is because, in a HoloLens, you already have real obstructions. If you want to try this in an Immersive headset, please remove or disable this feature. Otherwise, you will be in a void with almost nothing to test the behavior at all.

Conclusion

Unlike the previous one, this behavior makes full use of the possibilities that GetObjectBeforeObstruction offers. I think that there’s still room for improvements here. For instance, if you want to use this behavior to move and place stuff, simply disable the behavior when it’s done. But, this behavior, as it is, is very usable, and, in fact, I now use it myself in various apps.

Topics:
iot ,mr ,holotoolkitextension ,hololens ,hololens apps ,mixed reality ,tutorial

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}