Developing for MS HoloLens
Now that you've decided to develop an augmented reality app for HoloLens, here is how to start and configure your project for 3D visuals and interaction.
Join the DZone community and get the full member experience.Join For Free
This is the second article in a multi-part series about setting up and developing your first HoloLens application. In the interest of transparency, this material and technology are so new that there is very little reference material, and most of my experience has been via trial and error, many cups of coffee, and the sometimes helpful advice of the various communities (Unity and Microsoft).
Furthermore, due to the ever-changing technology, I am still in the process of developing/redeveloping my applications, and you should take this tutorial as a reference to hopefully save you some of the time and energy that it has cost me.
Some assumptions that we can start with: The first assumption is that you have your development environment set up and you have done some of the Microsoft Academy exercises. If not, I may suggest that you do and reference my first article on the subject on DZone.
Also, I may suggest that when reading the following, you take into account that I have made some basic assumptions on setting up elements such as the camera, etc. An example of such an assumption is that you have made the background camera color the way you want or used the suggestions of the Academy exercises.
Step 1: What Do You Want This Application to Do?
This is actually an important question to ask yourself. Augmented reality applications can range from simple programs to very complicated integrated programs. But the problem with these types of applications is how to make it useful to a user and not just some really cool tech.
For the purposes of this article, I will not dwell on the clearly obvious applications that many have read about such as simulations for educational, entertainment (games), and healthcare. To be frank, many of these types of applications require tremendous resources and time to do them correctly. So where does that leave you if you are not part of a large organization with lots of resources?
This is a question that I pondered for some time and have had many discussions with many different people. The biggest point I saw raised is, "Yeah, so I can do that with existing technology and don't have to wear or buy anything else?" This is a conversation you don't want to get into with anybody and often leads to, "That sounds cool, now show me." So let's show them.
Let's create a House layout tool for a general contractor to use with his clients. So now you have an application and an opportunity. Let's dig into that a bit more and ask another question: Is this for the contractor or the contractor's clients? That's important because that those would be two completely different applications. For the sake of this article, we will focus on the client of the contractor and start to develop an application that allows a client to visualize what the contractor is proposing by overlaying objects in a room and, ultimately, saving them to a file on a local machine.
Step 2: How Is It Going to Work?
If this application is being made for the client, then we need to consider how the client would actually use it. This is different than a normal 2D application. You will need to visualize and imagine all the elements a user would need to successfully use the application. Also, it has to be graphically intuitive. If it isn't, then people can get lost in the application and not feel they are getting the full benefit from it. The power behind a Hololens or AR device is its ease of use and the ability of an end user to easily interact with objects.
Another question is, "What do we want the user to be able to do?" Do we want the user to just look at a picture of the work in the room? That would place the burden of the development work on the contractor to construct the image. This, probably, isn't going to work for most independent contractors. Again, the business cases can be developed later — this article is to let you know some of the things you need to think about when developing an app.
My answer to this fictional case is to allow the end user to "install" features in the room. This would allow both the contractor and the home end user to mutually see what the final product should look like and thereby eliminate the typical confusion that often happens.
Step 3: Do You Want to Create a 2D or 3D Application?
One of the very interesting and unique aspects of the HoloLens is that it can do 2D or a regular application as well as fully immersive 3D applications. Of course, one may ask why you need to do a 2D application with a HoloLens, and an answer is that it allows you to manage your IT resources in a different way. But that is material for a different article.
Instead of being stuck looking at a monitor, you can have your applications in logical places throughout your space. For the sake of this article and application, a 2D program simply would not work or make much sense.
A 3D application would allow for the customer to immerse themselves into the experience and be able to visually see how things would work. It would give them the power to experience changes overlaying the existing space in real time and see what things would truly look like. No longer would a client and contractor have to waste time and money and energy discussing something and, unfortunately, miscommunicating. Having known people on both sides of the equation, this is certainly a major concern and issue.
Step 4: Identifying Your Required Team and Resources
Sometimes, the hardest aspect of designing and coding an AR project using the HoloLens, or any other type of device, is figuring out what you need based on your project. As a general note, if you are making a 3D, fully immersed HoloLens or AR device application, then you will need a very good 3D artist (if you are not one yourself) and, depending on the project, you will need either a sound person and/or Unity animator to bring objects to life.
For our project, we need to have a bunch of graphics that represent objects such as cabinets parts (surfaces, sides, hardware for the cabinets, appliances, doors, windows, light fixtures, etc.). The big question is why?
As you will recall from earlier in this article, we decided that this application was going to be used by the consumer to envision a completed room. In order for that to be done, we need to create a tool chest of graphical objects that a user can place in the room. This will enable the end user to fully visualize the new room with the new features.
Step 5: Starting the Project in Unity
When you start a project in Unity, you need to make sure that you have the latest MS Hololens Unity toolkit. (Now called the MixedRealityToolkit-Unity). If you don't have that, and the most recent versions of Unity and MS Visual Studio (in this case 2017), I would suggest that you download those applications and set them up, then install the MixedRealityToolkit-Unity. The purpose of the Mixed Reality Toolkit for Unity is that it allows for the Unity Editor to be set up and ready for the development of mixed reality applications.
To find the Mixed Reality Toolkit for Unity on GitHub, click here.
Download it and extract it into a folder of your making. Once you have it extracted, you should see a file with the .unity extension. You are now ready to set up your development environment and start your project
Now you will need to import it into Unity and do a few other things to complete the setup. All you need to do is go to Assets/Import Asset/Custom and select the .unity extension from the local directory. The system will whirl around for a few minutes, then you should have a new menu option called HoloToolkit on the menu bar (if not, restart Unity).
At this point, you should select the HoloToolkit menu choice and select configure/ApplyHololens Scene Settings and configure/ApplyHololens Project Settings. These options will set up Unity to be ready for HoloLens development and deployment to Visual Studio without you having to muck around with various settings.
Configure your project. Please note that the first few steps are actually available on the GitHub page under Getting Started. I am just putting them in this section as a starting point to save you the effort of looking for them online.
You will need to remove the default camera and lighting from the project menu. Scroll down to the assets and click on Utilities/Input/Prefabs, then add the Hololens pre-fab to the main project, or scene (if you like Unity terminology better). I would also add the basic cursor prefab as well.
Now we need something to handle all the inputs (the gestures, hands, etc.). With the Mixed Reality Toolkit, all of these are actually in something called the InputManager — I would suggest that you create an Empty Game object and rename it Manager. This is where we will put the InputManager prefab.
Optionally, but usefully (which we will see in a future post), add spacialmapping and spatialunderstanding to our application.
Step 6: Testing the Basic Elements of the Project
Now that you have your workspace sorted out and you are using the Mixed Reality Toolkit for Unity, you should be ready to start your first project.
Now for the sake of this specific article, we will not import a 3D object. Instead, we will do a very simple test so that you can appreciate the way things work in a 3D space and move an object around.
Click on Game Object, scroll down to 3D Object, and select Square. Unity will create a 3D square and drop it into the scene.
At this point, we need to make sure that your object is visible and located in the right place before we can start to interact with it. It is possible to drop an object anywhere and then try and find it. It may be fun for you, but it will not be for the user.
You should click on the HololensCamera in the project Hierarchy. This will highlight the camera and show you a bunch of lines coming from the camera. Move the box into the middle of those lines. You should then be able to see it in the camera view.
Now, if you were to build and run the application, it should show a box hanging in the space that you can't do much with. So, let's add some interactivity to our box. The toolkit comes with a tap-to-place script. So let's click on the cube, as this is the object we want to move, and select Add Component. Type in "tap to place" and click on the script. Done.
At this point, if you build your object, you should be able to see your cube and the room around you. Simply click and move the object as you wish.
So for this example, we simply created a 3D object that can be moved around your space and that integrate the Mixed Reality Toolkit for Unity.
You have created your very first live application and now you can dig deeper into your workspace to manipulate objects, add new ones, set up events etc. At this point, we will want our objects to interact with the surrounding space and really bring your experience to life. That would include spatialmapping, spatialunderstanding, and importing the important graphics elements to your tool chest.
You must always remember to think of the end user and their experience.
Opinions expressed by DZone contributors are their own.