Over a million developers have joined DZone.

Using an iPhone as a 3D Mouse with Multipeer Connectivity in Swift

How to create the most impressive mouse system in the world!

· Mobile Zone


My recent experiment with CoreMotionCoreMotion Controlled 3D Sketching on an iPhone with Swift, got me wondering if it would be possible to use an iPhone as a 3D mouse to control another application on a separate device. It turns out that with Apple's Multipeer Connectivity framework, it's not only possible, it's pretty awesome too!

The Multipeer Connectivity framework provides peer to peer communication between iOS devices over Wi-Fi and Bluetooth. As well as allowing devices to send discrete bundles of information, it also supports streaming which is what I need to allow my iPhone to transmit a continuous stream of data describing its attitude (roll, pitch and yaw) in 3D space.

I won't go in the the finer details of the framework, these are explained beautifully in the three main articles I used to get me up to speed:

My single codebase does the job of both the iPad "Rotating Cube" app which displays a cube floating in space and the iPhone "3D Mouse" app which controls the 3D rotation of the cube. As this is more of a proof-of-concept project rather than a piece of production code, everything is in a single view controller, this isn't good architecture, but when rapidly moving between the two "modes", it was super quick to work in.

The iPad "Rotating Cube App"

Apps using Multipeer Connectivity can either advertise a service or browse for a service. In my project, the Rotating Cube App takes the role of the advertiser so my view controller implements the MCNearbyServiceAdvertiserDelegate protocol. After I start advertising:


    func initialiseAdvertising()

    {

        serviceAdvertiser = MCNearbyServiceAdvertiser(peer: peerID, discoveryInfo: nil, serviceType: serviceType)



        serviceAdvertiser.delegate = self

        serviceAdvertiser.startAdvertisingPeer()



    }


...the protocol's advertiser() method is invoked when it receives an invitation from a peer. I want to automatically accept it:


    func advertiser(advertiser: MCNearbyServiceAdvertiser, didReceiveInvitationFromPeer peerID: MCPeerID, withContext context: NSData?, invitationHandler: (Bool, MCSession) -> Void)

    {

        invitationHandler(true, self.session)



    }


The iPhone "3D Mouse App"

Since the Rotating Cube App is the advertiser, my 3D Mouse App is the browser. So my monolithic view controller also implements MCNearbyServiceBrowserDelegate and, much like the advertiser, it starts browsing:


    func initialiseBrowsing()

    {

        serviceBrowser = MCNearbyServiceBrowser(peer: peerID, serviceType: serviceType)



        serviceBrowser.delegate = self

        serviceBrowser.startBrowsingForPeers()



    }


...and once it's found a peer, it sends that invitation we saw above to join the session:


    func browser(browser: MCNearbyServiceBrowser, foundPeer peerID: MCPeerID, withDiscoveryInfo info: [String :String]?)

    {

        streamTargetPeer = peerID



        browser.invitePeer(peerID, toSession: session, withContext: nil, timeout: 120)



        displayLink = CADisplayLink(target: self, selector: Selector("step"))

        displayLink?.addToRunLoop(NSRunLoop.mainRunLoop(), forMode: NSDefaultRunLoopMode)



    }


Here's where I also instantiate a CADisplayLink to invoke a step() method with each frame. step() does two things: it uses the streamTargetPeer I defined above to attempt to start a streaming session...


        outputStream =  try session.startStreamWithName("MotionControlStream", toPeer: streamTargetPeer)



        outputStream?.scheduleInRunLoop(NSRunLoop.mainRunLoop(), forMode: NSDefaultRunLoopMode)



        outputStream?.open()


...and, if that streaming session is available, sends the iPhone's attitude in 3D space (acquired using CoreMotion) over the stream:


    if let attitude = attitude where outputStream.hasSpaceAvailable

    {

        self.label.text = "stream: \(attitude.roll.radiansToDegrees()) | \(attitude.pitch.radiansToDegrees()) | \(attitude.yaw.radiansToDegrees())"



        outputStream.write(attitude.toBytes(), maxLength: 12)



    }


Serialising and Deserialising Float Values

The attitude (of type MotionControllerAttitude) struct contains three float values for roll, pitch and yaw, but the stream only supports UInt8 bytes. To serialise and deserialise that data, I found these two functions by Rintaro onStackOverflow that take any type and convert to and from arrays of UInt8:


    func fromByteArray(value: [UInt8], _: T.Type) -> T {

        return value.withUnsafeBufferPointer {

            return UnsafePointer<T>($0.baseAddress).memory

        }

    }





    func toByteArray(var value: T) -> [UInt8] {

        return withUnsafePointer(&value) {

            Array(UnsafeBufferPointer(start: UnsafePointer<UInt8>($0), count: sizeof(T)))

        }



    }


My MotionControllerAttitude struct has a toBytes() method that uses toByteArray() with flatMap() to create an array of UInt8 that the outputStream.write can use:


    func toBytes() -> [UInt8]

    {

        let composite = [roll, pitch, yaw]



        return composite.flatMap(){toByteArray($0)}



    }


...and, conversely, also has an init() to instantiate an instance of itself from an array of UInt8 using fromByteArray():


    init(fromBytes: [UInt8])

    {

        roll = fromByteArray(Array(fromBytes[0...3]), Float.self)

        pitch = fromByteArray(Array(fromBytes[4...7]), Float.self)

        yaw = fromByteArray(Array(fromBytes[8...11]), Float.self)



    }


This is pretty brittle code - again, this is just a proof of concept!

Rotating the Cube

Back in the Rotating Cube App, because the view controller is also acting as the NSStreamDelegate for the steam (you can see now things are yearning to be refactored!), the stream() method is invoked when the iPad receives a packet of data.

I need to check the incoming stream is actually a NSInputStream and it has bytes available. If it is and it does, I use the code above to create a MotionControllerAttitude instance from the incoming data and simply set the Euler angles on my cube:


    func stream(stream: NSStream, handleEvent eventCode: NSStreamEvent)
    {
        if let inputStream = stream as? NSInputStream where eventCode == NSStreamEvent.HasBytesAvailable
        {
            var bytes = [UInt8](count:12, repeatedValue: 0)
            inputStream.read(&bytes, maxLength: 12)

            let streamedAttitude = MotionControllerAttitude(fromBytes: bytes)

            dispatch_async(dispatch_get_main_queue())
            {
                self.label.text = "stream in: \(streamedAttitude.roll.radiansToDegrees()) | \(streamedAttitude.pitch.radiansToDegrees()) | \(streamedAttitude.yaw.radiansToDegrees())"

                self.geometryNode?.eulerAngles = SCNVector3(x: -streamedAttitude.pitch, y: streamedAttitude.yaw, z: streamedAttitude.roll)

            }
        }
    }


In Conclusion

This project demonstrates the power of Multipeer Connectivity: whether you're creating games or content creation apps, multiple iOS devices can work together and stream any type of data quickly and reliably. Conceivably, a roomful of iPads could be all hooked up as peers and act as a render farm or a huge single multi-device display.

As always, the source code for this project is available at my GitHub repository here.

I haven't covered the CoreMotion code, this is all discussed in CoreMotion Controlled  3D sketching on an iPhone with Swift.

Topics:
swift ,motion

Published at DZone with permission of Simon Gladman, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}