Over a million developers have joined DZone.

Physics Based Computer Generated Music With AudioKit and SpriteKit

· Java Zone

Navigate the Maze of the End-User Experience and pick up this APM Essential guide, brought to you in partnership with CA Technologies


This post looks at a little experimental app I've created using AudioKit and SpriteKit to generate music (of a sort) based on a simple physics simulation. The app, being a sprightly sounding thing that uses SpriteKit is aptly named Spritely.

The user interface allows users to create static boxes, that are assigned an audio frequency depending on their length, and bouncing balls. When a ball collides with a box, AudioKit plays a “Vibes” sound at the box’s frequency with an amplitude based on the ball's velocity.

Boxes are created with a pan gesture and can be moved and rotated. Balls are created with a long press gesture. Once the balls fall off the bottom of the screen, they reappear at the top at their original x-position. Although balls interact with boxes, they don’t interact with each other - this allows for a regular, repeatable pattern.

I’ve extended two of the SpriteKit classes for use in Spritely:
  • TouchEnabledShapeNode extends SKShapeNode and is used for the boxes. It has a frequency property that populates a label and also accepts a delegate of type TouchEnabledShapeNodeDelegate which allows it to report back when it’s been touched.
  • ShapeNodeWithOriginalso extends SKShapeNode and is used for the balls. It has an optional startingPositionproperty. This allows me to reset the balls to their original x-position after they wrap around the screen.
Loosely speaking, there are three main parts to Spritely, let’s look at each:


Gesture Handling


The view controller handles the user's gestures. Its view has three different gesture recognisers, a long press, a pan and a rotate.

The long press handler, longPressHandler(), is pretty simple: it invokes my createBall() function which adds a new circular ShapeNodeWithOrigin to the SpriteKit scene.

The pan handler, panHandler(), has two responsibilities: if a box is currently selected, a pan gesture moves that box, and if there’s no selected box, a pan gesture creates one.

Remembering that gestures have three main states of interest in this context. The move action starts with the gesture’sBegan, where it sets a value for panGestureOrigin based on the gesture’s location. Since a pan gesture is continuous, each invocation of panHandler() with a Changed state moves the selected box by the difference betweenpanGestureOrigin and the current gesture position:

            [...]
            else if recogniser.state == UIGestureRecognizerState.Changed
            {
                let currentGestureLocation = recogniser.locationInView(view)
                
                selectedBox!.position.x += currentGestureLocation.x - panGestureOrigin!.x
                selectedBox!.position.y -= currentGestureLocation.y - panGestureOrigin!.y
                
                panGestureOrigin = recogniser.locationInView(view)
            }
            [...]
Finally, when the gesture ends or is cancelled, I null the panGestureOrigin variable. 

If a box isn’t selected, the pan gesture begin adds a temporary shape node named creatingBox to the scene. This acts as a placeholder during the creation process. With each gesture Changed call, I have to recreate that box because the geometry of SKShapeNodes is immutable - I use not only the distance between the current gesture location and thepanGestureOrigin but the angle too to set the new box’s rotation:

            [...]
            else if recogniser.state == UIGestureRecognizerState.Changed
            {
                creatingBox!.removeFromParent()
                
                let invertedLocationInView = CGPoint(x: recogniser.locationInView(view).x,
                    y: view.frame.height - recogniser.locationInView(view).y)
                
                let boxWidth = CGFloat(panGestureOrigin!.distance(invertedLocationInView)) * 2
                
                creatingBox = SKShapeNode(rectOfSize: CGSize(width: boxWidth, height: boxHeight))
                creatingBox!.position = panGestureOrigin!
                
                creatingBox!.zRotation = atan2(panGestureOrigin!.x - invertedLocationInView.x, invertedLocationInView.y - panGestureOrigin!.y) + CGFloat(M_PI / 2)
                
                scene.addChild(creatingBox!)
            }
            [...]

Finally, one the pan gesture is finished, I actually create the box at the final location and angle. Spritely contains an arrays of frequencies that map to music notes and the createBox() method uses this array to assign a frequency to a box based on its length:

        [...]
        let frequencyIndex = Int(round((actualWidth - minBoxLength) / (maxBoxLength - minBoxLength) * CGFloat(frequencies.count - 1)))

        box.frequency = frequencies[frequencyIndex]
        [...]
The rotate handler, rotateHandler(), is only relevant when a box is selected. It uses a similar technique as panHandler(), in that when the gesture begins, it sets the value of rotateGestureAngleOrigin to the gesture’s angle and with eachChanged step, rotates the selected box by the difference between the last and current angles.


SpriteKit Physics


The view controller acts as the contact delegate for the SpriteKit scene. All the logic for playing tones based on collisions and wrapping the balls around is implemented in didBeginContact(). This function is passed a instance ofSKPhysicsContact which has properties, bodyA and bodyB, for the two SKPhysicsBodys involved in the collision. By looking at the categoryBitMasks of those bodies, I’m able to determine what collision actors are balls, bodies or static objects such as the floor.

If one of those bodies is a box, I can be certain a ball-box collision has taken place. In that case, I can cast the body which is a box to a TouchEnabledShapeNode and calculate the velocity of the other body. With those two numbers, I can play a tone:
        [...]
        if contact.bodyA.categoryBitMask == boxCategoryBitMask
        {
            let amplitude = Float(sqrt((contact.bodyB.velocity.dx * contact.bodyB.velocity.dx) + (contact.bodyB.velocity.dy * contact.bodyB.velocity.dy)) / 1500)

            let frequency = (contact.bodyA.node as? TouchEnabledShapeNode)?.frequency
                
            conductor.play(frequency!, amplitude: amplitude)
        }
        else if contact.bodyB.categoryBitMask == boxCategoryBitMask
        {
            // do the opposite
        [...]

If one of the bodies is the floor, the other will be a ball (which is a ShapeNodeWithOrigin instance). Simply setting the position of a SKShapeNode instance with an associated physics body doesn’t work, so I need to temporarily remove the physics body, set the position using the startingPosition and reapply the physics body.

        [...]
        if contact.bodyA.categoryBitMask & ballCategoryBitMask == ballCategoryBitMask && contact.bodyB.categoryBitMask == floorCategoryBitMask
        {
            physicsBodyToReposition = contact.bodyA
        }
        else if contact.bodyB.categoryBitMask & ballCategoryBitMask == ballCategoryBitMask && contact.bodyA.categoryBitMask == floorCategoryBitMask
        {
            physicsBodyToReposition = contact.bodyB
        }
        
        if let physicsBodyToReposition = physicsBodyToReposition
        {
            let nodeToReposition = physicsBodyToReposition.node
            let nodeX: CGFloat = (nodeToReposition as? ShapeNodeWithOrigin)?.startingPostion?.x ?? 0
            
            nodeToReposition?.physicsBody = nil
            nodeToReposition?.position = CGPoint(x: nodeX, y: view.frame.height)
            nodeToReposition?.physicsBody = physicsBodyToReposition
        }
        [...]

AudioKit Sound


The final part of the project was to add AudioKit support. My initial approach was to assign each box its own instrument which was played individually. However, stopping the orchestra, adding a new instrument and restarting the orchestra each time a new box was created was very choppy. I then created an instruments provider that created a load of instruments at startup and from which each box would request an instrument from at instantiation: this approach worked but was totally unnecessary.

After collaborating with Aurelius Prochazka, he implemented a much better solution. He created a single Conductorclass with an single instrument. It’s that Conductor class’s play() method that’s invoked in the didBeginContact()method above. 

When a Conductor is instantiated, it creates an instance of BarInstrument (which determines the type of sound, e.g. a mandolin, struck metal bar or, in this case a “vibes” sound), which in turn has a BarNote (which determines the frequency and amplitude). When the conductor’s play() method is invoked, it creates a new BarNote of the required frequency and amplitude and passes this to its instrument via playNote():
    func play(frequency: Float, amplitude: Float) {
        let barNote = BarNote(frequency: frequency, amplitude: amplitude)
        barNote.duration.value = 3.0
        barInstrument.playNote(barNote)
    }
The end result is, in my opinion, quite effective. Short boxes create higher pitched notes and long boxes create lower pitched notes. Since each ball follows an identical path after its first loop, regular patterns of notes can be created and by mixing boxes of different lengths, interesting, ambient soundscapes can be created.

Once again, massive thanks to Aurelius, his input proved invaluable in optimising the code. Of course, big thanks to the entire AudioKit team, not only do their libraries make tinkering with sound super easy, their documentation makes getting up and running an absolute breeze. The source code for this project is available at my GitHub repository here.

Addendum: A Note About The Video: bizarrely, on some machines the sound on the video embedded above is slightly broken. The longer 130Hz box should play a lower note the the shorter one, but on some machines it plays a higher note. This seems to be a YouTube issue, the source soundtrack is correct and the sound issue is only on some machines. 

Thrive in the application economy with an APM model that is strategic. Be E.P.I.C. with CA APM.  Brought to you in partnership with CA Technologies.

Topics:

Published at DZone with permission of Simon Gladman, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}