Over a million developers have joined DZone.

Physics Based Computer Generated Music With AudioKit and SpriteKit

DZone's Guide to

Physics Based Computer Generated Music With AudioKit and SpriteKit

· Java Zone ·
Free Resource

Secure your Java app or API service quickly and easily with Okta's user authentication and authorization libraries. Developer accounts are free forever. Try Okta Instead.

This post looks at a little experimental app I've created using  AudioKit and  SpriteKit to generate music (of a sort) based on a simple physics simulation. The app, being a sprightly sounding thing that uses SpriteKit is aptly named  Spritely.

The user interface allows users to create static boxes, that are assigned an audio frequency depending on their length, and bouncing balls. When a ball collides with a box, AudioKit plays a “Vibes” sound at the box’s frequency with an amplitude based on the ball's velocity.

Boxes are created with a pan gesture and can be moved and rotated. Balls are created with a long press gesture. Once the balls fall off the bottom of the screen, they reappear at the top at their original x-position. Although balls interact with boxes, they don’t interact with each other - this allows for a regular, repeatable pattern.

I’ve extended two of the SpriteKit classes for use in Spritely:
  • TouchEnabledShapeNode extends SKShapeNode and is used for the boxes. It has a frequency property that populates a label and also accepts a delegate of type TouchEnabledShapeNodeDelegate which allows it to report back when it’s been touched.
  • ShapeNodeWithOriginalso extends SKShapeNode and is used for the balls. It has an optional startingPositionproperty. This allows me to reset the balls to their original x-position after they wrap around the screen.
Loosely speaking, there are three main parts to Spritely, let’s look at each:

Gesture Handling

The  view controller  handles the user's gestures. Its view has three different gesture recognisers, a long press, a pan and a rotate.

The long press handler, longPressHandler(), is pretty simple: it invokes my createBall() function which adds a new circular ShapeNodeWithOrigin to the SpriteKit scene.

The pan handler, panHandler(), has two responsibilities: if a box is currently selected, a pan gesture moves that box, and if there’s no selected box, a pan gesture creates one.

Remembering that gestures have three main states of interest in this context. The move action starts with the gesture’sBegan, where it sets a value for panGestureOrigin based on the gesture’s location. Since a pan gesture is continuous, each invocation of panHandler() with a Changed state moves the selected box by the difference betweenpanGestureOrigin and the current gesture position:

            else if recogniser.state == UIGestureRecognizerState.Changed
                let currentGestureLocation = recogniser.locationInView(view)
                selectedBox!.position.x += currentGestureLocation.x - panGestureOrigin!.x
                selectedBox!.position.y -= currentGestureLocation.y - panGestureOrigin!.y
                panGestureOrigin = recogniser.locationInView(view)
Finally, when the gesture ends or is cancelled, I null the  panGestureOrigin variable. 

If a box isn’t selected, the pan gesture begin adds a temporary shape node named creatingBox to the scene. This acts as a placeholder during the creation process. With each gesture Changed call, I have to recreate that box because the geometry of SKShapeNodes is immutable - I use not only the distance between the current gesture location and thepanGestureOrigin but the angle too to set the new box’s rotation:

            else if recogniser.state == UIGestureRecognizerState.Changed
                let invertedLocationInView = CGPoint(x: recogniser.locationInView(view).x,
                    y: view.frame.height - recogniser.locationInView(view).y)
                let boxWidth = CGFloat(panGestureOrigin!.distance(invertedLocationInView)) * 2
                creatingBox = SKShapeNode(rectOfSize: CGSize(width: boxWidth, height: boxHeight))
                creatingBox!.position = panGestureOrigin!
                creatingBox!.zRotation = atan2(panGestureOrigin!.x - invertedLocationInView.x, invertedLocationInView.y - panGestureOrigin!.y) + CGFloat(M_PI / 2)

Finally, one the pan gesture is finished, I actually create the box at the final location and angle. Spritely contains an arrays of frequencies that map to music notes and the createBox() method uses this array to assign a frequency to a box based on its length:

        let frequencyIndex = Int(round((actualWidth - minBoxLength) / (maxBoxLength - minBoxLength) * CGFloat(frequencies.count - 1)))

        box.frequency = frequencies[frequencyIndex]
The rotate handler,  rotateHandler(), is only relevant when a box is selected. It uses a similar technique as  panHandler(), in that when the gesture begins, it sets the value of  rotateGestureAngleOrigin to the gesture’s angle and with each Changed step, rotates the selected box by the difference between the last and current angles.

SpriteKit Physics

The view controller acts as the contact delegate for the SpriteKit scene. All the logic for playing tones based on collisions and wrapping the balls around is implemented in  didBeginContact() . This function is passed a instance of SKPhysicsContact  which has properties,  bodyA  and  bodyB , for the two  SKPhysicsBody s involved in the collision. By looking at the  categoryBitMask s of those bodies, I’m able to determine what collision actors are balls, bodies or static objects such as the floor.

If one of those bodies is a box, I can be certain a ball-box collision has taken place. In that case, I can cast the body which is a box to a  TouchEnabledShapeNode  and calculate the velocity of the other body. With those two numbers, I can play a tone:
        if contact.bodyA.categoryBitMask == boxCategoryBitMask
            let amplitude = Float(sqrt((contact.bodyB.velocity.dx * contact.bodyB.velocity.dx) + (contact.bodyB.velocity.dy * contact.bodyB.velocity.dy)) / 1500)

            let frequency = (contact.bodyA.node as? TouchEnabledShapeNode)?.frequency
            conductor.play(frequency!, amplitude: amplitude)
        else if contact.bodyB.categoryBitMask == boxCategoryBitMask
            // do the opposite

If one of the bodies is the floor, the other will be a ball (which is a ShapeNodeWithOrigin instance). Simply setting the position of a SKShapeNode instance with an associated physics body doesn’t work, so I need to temporarily remove the physics body, set the position using the startingPosition and reapply the physics body.

        if contact.bodyA.categoryBitMask & ballCategoryBitMask == ballCategoryBitMask && contact.bodyB.categoryBitMask == floorCategoryBitMask
            physicsBodyToReposition = contact.bodyA
        else if contact.bodyB.categoryBitMask & ballCategoryBitMask == ballCategoryBitMask && contact.bodyA.categoryBitMask == floorCategoryBitMask
            physicsBodyToReposition = contact.bodyB
        if let physicsBodyToReposition = physicsBodyToReposition
            let nodeToReposition = physicsBodyToReposition.node
            let nodeX: CGFloat = (nodeToReposition as? ShapeNodeWithOrigin)?.startingPostion?.x ?? 0
            nodeToReposition?.physicsBody = nil
            nodeToReposition?.position = CGPoint(x: nodeX, y: view.frame.height)
            nodeToReposition?.physicsBody = physicsBodyToReposition

AudioKit Sound

The final part of the project was to add  AudioKit  support. My initial approach was to assign each box its own instrument which was played individually. However, stopping the orchestra, adding a new instrument and restarting the orchestra each time a new box was created was very choppy. I then created an instruments provider that created a load of instruments at startup and from which each box would request an instrument from at instantiation: this approach worked but was totally unnecessary.

After collaborating with  Aurelius Prochazka , he implemented a much better solution. He created a single  Conductor class with an single instrument. It’s that  Conductor  class’s  play()  method that’s invoked in the  didBeginContact() method above. 

When a  Conductor is instantiated, it creates an instance of  BarInstrument (which determines the type of sound, e.g. a mandolin, struck metal bar or, in this case a “vibes” sound), which in turn has a  BarNote (which determines the frequency and amplitude). When the conductor’s  play() method is invoked, it creates a new  BarNote of the required frequency and amplitude and passes this to its instrument via  playNote():
    func play(frequency: Float, amplitude: Float) {
        let barNote = BarNote(frequency: frequency, amplitude: amplitude)
        barNote.duration.value = 3.0
The end result is, in my opinion,  quite effective. Short boxes create higher pitched notes and long boxes create lower pitched notes. Since each ball follows an identical path after its first loop, regular patterns of notes can be created and by mixing boxes of different lengths, interesting, ambient soundscapes can be created.

Once again, massive thanks to Aurelius, his input proved invaluable in optimising the code. Of course, big thanks to the entire AudioKit team, not only do their libraries make tinkering with sound super easy, their documentation makes getting up and running an absolute breeze. The source code for this project is available at my GitHub repository here.

Addendum: A Note About The Video: bizarrely, on some machines the sound on the video embedded above is slightly broken. The longer 130Hz box should play a lower note the the shorter one, but on some machines it plays a higher note. This seems to be a YouTube issue, the source soundtrack is correct and the sound issue is only on some machines. 

Secure your Java app or API service quickly and easily with Okta's user authentication and authorization libraries. Developer accounts are free forever. Try Okta Instead.


Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}