Simon Gladman [Deactivated] [Suspended]

Combining fluids can be a frustrating experience for Houdini developers. From the point cloud to the Blast nodes, here's how to get your fluids swirling around.

How to use Apple's vImage API in Swift to manipulate the histogram of images programmatically.

How to create images maps in SceneKit on iOS.

We all cut and paste images for all kinds of reasons. Here's one way to make those images look good in a 3D world.

More image magic in Swift from Simon Gladman.

How to create really smooth lines for your drawing apps on iOS.

Here's a fun little experiment showing the power of Swift's Protocol Extensions to apply a CIGaussianBlur Core Image filter to any UIView with no developer overhead. The code could be extended to apply any Core Image filter such as a half tone screen or colour adjustment. Blurable is a simple protocol that borrows some of the methods and variables from a UIView: var layer: CALayer { get } var subviews: [UIView] { get } var frame: CGRect { get } func addSubview(view: UIView) func bringSubviewToFront(view: UIView) ...and adds a few of its own: Obviously, just being a protocol, it doesn't do much on its own. However, by adding an extension, I can introduce default functionality. Furthermore, by extending UIView to implement Blurable, every component from a segmented control to a horizontal slider can be blurred: extension UIView: Blurable { } The Mechanics of Blurable Getting a blurred representation of a UIView is pretty simple: I need to begin an image context, use the view's layer's renderInContext method to render into the context and then get a UIImage from the context: UIGraphicsBeginImageContextWithOptions(CGSize(width: frame.width, height: frame.height), false, 1) layer.renderInContext(UIGraphicsGetCurrentContext()!) let image = UIGraphicsGetImageFromCurrentImageContext() UIGraphicsEndImageContext(); Once I have the image populated, it's a fairly standard workflow to apply a Gaussian blur to it: guard let blur = CIFilter(name: "CIGaussianBlur") else { return } blur.setValue(CIImage(image: image), forKey: kCIInputImageKey) blur.setValue(blurRadius, forKey: kCIInputRadiusKey) let ciContext = CIContext(options: nil) let result = blur.valueForKey(kCIOutputImageKey) as! CIImage! let boundingRect = CGRect(x: -blurRadius * 4, y: -blurRadius * 4, width: frame.width + (blurRadius * 8), height: frame.height + (blurRadius * 8)) let cgImage = ciContext.createCGImage(result, fromRect: boundingRect) let filteredImage = UIImage(CGImage: cgImage) A blurred image will be larger than its input image, so I need to be explicit about the size I require in createCGImage. The next step is to add a UIImageView to my view and hide all the other views. I've subclassed UIImageView to BlurOverlay so that when it comes to removing it, I can be sure I'm not removing an existing UIImageView: let blurOverlay = BlurOverlay() blurOverlay.frame = boundingRect blurOverlay.image = filteredImage subviews.forEach{ $0.hidden = true } addSubview(blurOverlay) When it comes to de-blurring, I want to ensure the last subview is one of my BlurOverlay remove it and unhide the existing views: func unBlur() { if let blurOverlay = subviews.last as? BlurOverlay { blurOverlay.removeFromSuperview() subviews.forEach{ $0.hidden = false } } } Finally, to see if a UIView is currently blurred, I just need to see if its last subview is a BlurOverlay: var isBlurred: Bool { return subviews.last is BlurOverlay } Blurring a UIView To blur and de-blur, just invoke blur() and unBlur() on an UIView: segmentedControl.unBlur() segmentedControl.blur(blurRadius: 2) Source Code As always, the source code for this project is available at my GitHub repository here. Enjoy!

This post looks at how you could implement ShinpuruNodeUI in your own app.

I'm a big fan of Apple's Core Image technology: my Nodality application is based entirely around Core Image filters. However, for new users, the code for adding a simple filter to an image is a little oblique and the implementation is very"stringly" typed This post looks at an alternative, GPUImage from Brad Larson. GPUImage is a framework containing a rich set of image filters, many of which aren't in Core Image. It has a far simpler and more strongly typed API and, in some cases, is faster than Core Image. To kick off, let's look at the code required to apply a Gaussian blue to an image (inputImage) using Core Filter: let inputImage = UIImage() let ciContext = CIContext(options: nil) let blurFilter = CIFilter(name: "CIGaussianBlur") blurFilter.setValue(CIImage(image: inputImage), forKey: "inputImage") blurFilter.setValue(10, forKey: "inputRadius") let outputImageData = blurFilter.valueForKey("outputImage") as CIImage! let outputImageRef: CGImage = ciContext.createCGImage(outputImageData, fromRect: outputImageData.extent()) let outputImage = UIImage(CGImage: outputImageRef)! ...not only do we need to explicitly define the context, both the filter name and parameter are strings and we need a few steps to convert the filter's output into a UIImage. Here's the same functionality using GPUImage: let inputImage = UIImage() let blurFilter = GPUImageGaussianBlurFilter() blurFilter.blurRadiusInPixels = 10 let outputImage = blurFilter.imageByFilteringImage(inputImage) Here, both the filter and its blur radius parameter are properly typed and the filter returns a UIImage instance. On the flip-side, there is some setting up to do. Once you've got a local copy of GPUImage, drag the framework project into your application's project. Then under the application target's build phases, add a target dependency, a reference to GPUImage.framework under link binaries and a copy files stage. Your build phases screen should look like this: Then, by simply importing GPUImage, you're ready to roll. To show off some of the funkier filters contained in GPUImage, I've created a little demonstration app,GPUImageDemo. The app demonstrates Polar Pixellate, Polka Dot, Sketch, Threshold Sketch, Toon, Smooth Toon, Emboss, Sphere Refraction and Glass Sphere - none of which are available in Core Image. The filtering work is all done in my GPUImageDelegate class where a switch statement declares aGPUImageOutput variable (the class that includes the imageByFiltering() method) and sets it to the appropriate concrete class depending on the user interface. For example, if the picker is set the threshold sketch, the following case statement is executed: case ImageFilter.ThresholdSketch: gpuImageFilter = GPUImageThresholdSketchFilter() if let gpuImageFilter = gpuImageFilter as? GPUImageThresholdSketchFilter { if values.count > 1 { gpuImageFilter.edgeStrength = values[0] gpuImageFilter.threshold = values[1] } } If you build this project, you may encounter a build error on the documentation target. I've simply deleted this target on affected machines. GPUImage is fast enough to filter video. I've taken my recent two million particles experiment and added a post processing step that consists of a cartoon filter and an emboss filter. These are packaged together in aGPUImageFilterGroup: let toonFilter = GPUImageSmoothToonFilter() let embossFilter = GPUImageEmbossFilter() let filterGroup = GPUImageFilterGroup() toonFilter.threshold = 1 embossFilter.intensity = 2 filterGroup.addFilter(toonFilter) filterGroup.addFilter(embossFilter) toonFilter.addTarget(embossFilter) filterGroup.initialFilters = [ toonFilter ] filterGroup.terminalFilter = embossFilter Since GPUImageFilterGroupextends GPUImageFilterOutput, I can take the output from the Metal texture, create aUIImage instance of it and pass it to the composite filter: self.imageView.image = self.filterGroup.imageByFilteringImage(UIImage(CGImage: imageRef)!) On my iPad Air 2, the final result of 2,000,000 particles with a two filter post process on a 1,024 x 1,024 image still runs at around 20 frames per second. Here's a real time screen capture: The source code for my GPUImageDemo is available at my GitHub repository here and GPUImagelives here.

As part of a project to create a GPU based reaction diffusion simulation, I stated to look at using Metal in Swift this weekend. I've done similar work in the past targeting the Flash Player and using AGAL. Metal is a far higher level language than AGAL: it's based on C++ with a richer syntax and includes compute functions. Whereas in AGAL, to run cellular automata, I'd create a rectangle out of two triangles with a vertex shader and execute the reaction diffusion functions in a separate fragment shader, a compute shader is more direct: I can get and set textures and it can operate of individual pixels of that texture without the need for a vertex shader. The Swift code I discuss in this article is based heavily on two articles at Metal By Example: Introduction to Compute Programming in Metal and Fundamentals of Image Processing in Metal. Both of which include Objective-C source code, so hopefully my Swift implementation will help some. My application has four main steps: initialise Metal, create a Metal texture from a UIImage, apply a kernel function to that texture, convert the newly generated texture back into a UIImage and display it. I'm using a simple example shader that changes the saturation of the input image. so I've also added a slider that changes the saturation value. Let's look at each step one by one: Initialising Metal Initialising Metal is pretty simple: inside my view controller's overridden viewDidLoad(), I create a pointer to the default Metal device: var device: MTLDevice! = nil [...] device = MTLCreateSystemDefaultDevice() I also need to create a library and command queue: defaultLibrary = device.newDefaultLibrary() commandQueue = device.newCommandQueue() Finally, I add a reference to my Metal function to the library and synchronously create and compile a compute pipeline state: et kernelFunction = defaultLibrary.newFunctionWithName("kernelShader") pipelineState = device.newComputePipelineStateWithFunction(kernelFunction!, error: nil) The kernelShader points to the saturation image processing function, written in Metal, that lives in my Shaders.metal file: kernel void kernelShader(texture2d inTexture [[texture(0)]], texture2d outTexture [[texture(1)]], constant AdjustSaturationUniforms &uniforms [[buffer(0)]], uint2 gid [[thread_position_in_grid]]) { float4 inColor = inTexture.read(gid); float value = dot(inColor.rgb, float3(0.299, 0.587, 0.114)); float4 grayColor(value, value, value, 1.0); float4 outColor = mix(grayColor, inColor, uniforms.saturationFactor); outTexture.write(outColor, gid); } Creating a Metal Texture from a UIIMage There are a few steps in converting a UIImage into a MTLTexture instance. I create an array of UInt8 to hold an emptyCGBitmapInfo, then use CGContextDrawImage() to copy the image into a bitmap context let image = UIImage(named: "grand_canyon.jpg") let imageRef = image.CGImage let imageWidth = CGImageGetWidth(imageRef) let imageHeight = CGImageGetHeight(imageRef) let bytesPerRow = bytesPerPixel * imageWidth var rawData = [UInt8](count: Int(imageWidth * imageHeight * 4), repeatedValue: 0) let bitmapInfo = CGBitmapInfo(CGBitmapInfo.ByteOrder32Big.toRaw() | CGImageAlphaInfo.PremultipliedLast.toRaw()) let context = CGBitmapContextCreate(&rawData, imageWidth, imageHeight, bitsPerComponent, bytesPerRow, rgbColorSpace, bitmapInfo) CGContextDrawImage(context, CGRectMake(0, 0, CGFloat(imageWidth), CGFloat(imageHeight)), imageRef) Once all of those steps have executed, I can create a new texture use its replaceRegion() method to write the image into it: let textureDescriptor = MTLTextureDescriptor.texture2DDescriptorWithPixelFormat(MTLPixelFormat.RGBA8Unorm, width: Int(imageWidth), height: Int(imageHeight), mipmapped: true) texture = device.newTextureWithDescriptor(textureDescriptor) let region = MTLRegionMake2D(0, 0, Int(imageWidth), Int(imageHeight)) texture.replaceRegion(region, mipmapLevel: 0, withBytes: &rawData, bytesPerRow: Int(bytesPerRow)) I also create an empty texture which the kernel function will write into: let outTextureDescriptor = MTLTextureDescriptor.texture2DDescriptorWithPixelFormat(texture.pixelFormat, width: texture.width, height: texture.height, mipmapped: false) outTexture = device.newTextureWithDescriptor(outTextureDescriptor) Invoking the Kernel Function The next block of work is to set the textures and another variable on the kerne function and execute the shader. The first step is to instantiate a command buffer and command encoder: let commandBuffer = commandQueue.commandBuffer() let commandEncoder = commandBuffer.computeCommandEncoder() ...then set the pipeline state (we got from device.newComputePipelineStateWithFunction() earlier) and textures on the command encoder: commandEncoder.setComputePipelineState(pipelineState) commandEncoder.setTexture(texture, atIndex: 0) commandEncoder.setTexture(outTexture, atIndex: 1) The filter requires an addition parameter that defines the saturation amount. This is passed into the shader via anMTLBuffer. To populate the buffer, I've created a small struct: struct AdjustSaturationUniforms { var saturationFactor: Float } Then newBufferWithBytes() to pass in my saturationFactor float value: var saturationFactor = AdjustSaturationUniforms(saturationFactor: self.saturationFactor) var buffer: MTLBuffer = device.newBufferWithBytes(&saturationFactor, length: sizeof(AdjustSaturationUniforms), options: nil) commandEncoder.setBuffer(buffer, offset: 0, atIndex: 0) This is now accessible inside the shader as an argument to its kernel function: constant AdjustSaturationUniforms &uniforms [[buffer(0)]] Now I'm ready invoke the function itself. Metal kernel functions use thread groups to break up their workload into chunks. In my example, I create 64 thread groups, then send them off to the GPU: let threadGroupCount = MTLSizeMake(8, 8, 1) let threadGroups = MTLSizeMake(texture.width / threadGroupCount.width, texture.height / threadGroupCount.height, 1) commandQueue = device.newCommandQueue() commandEncoder.dispatchThreadgroups(threadGroups, threadsPerThreadgroup: threadGroupCount) commandEncoder.endEncoding() commandBuffer.commit() commandBuffer.waitUntilCompleted() Converting the Texture to a UIImage Finally, now that the kernel function has executed, we need to do the reverse of above and get the image held inoutTexture into a UIImage so it can be displayed. Again, I use a region to define the size and the texture's getBytes() to populate an array on UInt8: let imageSize = CGSize(width: texture.width, height: texture.height) let imageByteCount = Int(imageSize.width * imageSize.height * 4) let bytesPerRow = bytesPerPixel * UInt(imageSize.width) var imageBytes = [UInt8](count: imageByteCount, repeatedValue: 0) let region = MTLRegionMake2D(0, 0, Int(imageSize.width), Int(imageSize.height)) outTexture.getBytes(&imageBytes, bytesPerRow: Int(bytesPerRow), fromRegion: region, mipmapLevel: 0) Now that imageBytes holds the raw data, it's a few lines to create a CGImage: let providerRef = CGDataProviderCreateWithCFData( NSData(bytes: &imageBytes, length: imageBytes.count * sizeof(UInt8)) ) let bitmapInfo = CGBitmapInfo(CGBitmapInfo.ByteOrder32Big.toRaw() | CGImageAlphaInfo.PremultipliedLast.toRaw()) let renderingIntent = kCGRenderingIntentDefault let imageRef = CGImageCreate(UInt(imageSize.width), UInt(imageSize.height), bitsPerComponent, bitsPerPixel, bytesPerRow, rgbColorSpace, bitmapInfo, providerRef, nil, false, renderingIntent) imageView.image = UIImage(CGImage: imageRef) ...and we're done! Metal requires an A7 or A8 processor and this code has been built and tested under Xcode 6. All the source code is available at my GitHub repository here.

The other day, Morgan at Swift London mentioned it may be interesting to use my Swift node based user interface as the basis for an audio synthesiser. Always ready for a challenge, I've spent a little time looking into Core Audio to see how this could be done and created a little demonstration app: a multi channel tone generator. I've done something not too dissimilar targeting Flash in the past. In that project, I had to manually create 8192 samples to synthesise a tone and ended up using ActionScript Workers to do that work in the background. After some poking around, it seems a similar approach can be taken in Core Audio, but to create pure sine waves there's a far simpler way: using Audio Toolbox. Luckily, I found this excellent article by Gene De Lisa. His SwiftSimpleGraph project has all the boilerplate code to create sine waves, so all I had to do was add a user interface. My project contains four ToneWidget instances, each of which contain numeric dials for frequency and velocity and asine wave renderer. There's also an additional sine wave renderer that displays the composite frequency of all four channels. The sine wave renderer creates a bitmap image of the wave based on some code Joseph Lord tweaked in my Swift reaction diffusion code in the summer. It exposes a setFrequencyVelocityPairs() method which accepts an array ofFrequencyVelocityPair instances. When that's invoked, the drawSineWave() method loops over the array creating and summing vertical values for each column in the bitmap: for i in 1 ..< Int(frame.width) { let scale = M_PI * 5 let curveX = Double(i) var curveY = Double(frame.height / 2) for pair in frequencyVelocityPairs { let frequency = Double(pair.frequency) / 127.0 let velocity = Double(pair.velocity) / 127.0 curveY += ((sin(curveX / scale * frequency * 5)) * (velocity * 10)) } [...] Then, rather than simply plotting the generated position for each column, it draws a line between the newly calculated vertical position for the current column and the previous vertical position for the previous column: for yy in Int(min(previousCurveY, curveY)) ... Int(max(previousCurveY, curveY)) { let pixelIndex : Int = (yy * Int(frame.width) + i); pixelArray[pixelIndex].r = UInt8(255 * colorRef[0]); pixelArray[pixelIndex].g = UInt8(255 * colorRef[1]); pixelArray[pixelIndex].b = UInt8(255 * colorRef[2]); } The result is a continuous wave: Because the sine wave renderer accepts an array of values, I'm able to use the same component for both the individual channels (where the array length is one) and for the composite of all four (where the array length is four). I was ready to use GCD to do the sine wave drawing in a background thread, but both the simulator and my old iPad happily do this in the main thread keeping the user interface totally responsive. Up in the view controller, I have an array containing each widget and another array containing the currently playing notes which is populated inside viewDidLoad(): override func viewDidLoad() { super.viewDidLoad() view.addSubview(sineWaveRenderer) for i in 0 ... 3 { let toneWidget = ToneWidget(index: i, frame: CGRectZero) toneWidget.addTarget(self, action: "toneWidgetChangeHandler:", forControlEvents: UIControlEvents.ValueChanged) toneWidgets.append(toneWidget) view.addSubview(toneWidget) currentNotes.append(toneWidget.getFrequencyVelocityPair()) soundGenerator.playNoteOn(UInt32(toneWidget.getFrequencyVelocityPair().frequency), velocity: UInt32(toneWidget.getFrequencyVelocityPair().velocity), channelNumber: UInt32(toneWidget.getIndex())) } } When the frequency or the velocity is changed by the user in any of those widgets, I use the value in thecurrentNotes array to switch off the currently playing note, update the master sine wave renderer and play the new note: func toneWidgetChangeHandler(toneWidget : ToneWidget) { soundGenerator.playNoteOff(UInt32(currentNotes[toneWidget.getIndex()].frequency), channelNumber: UInt32(toneWidget.getIndex())) updateSineWave() soundGenerator.playNoteOn(UInt32(toneWidget.getFrequencyVelocityPair().frequency), velocity: UInt32(toneWidget.getFrequencyVelocityPair().velocity), channelNumber: UInt32(toneWidget.getIndex())) currentNotes[toneWidget.getIndex()] = toneWidget.getFrequencyVelocityPair() } The updateSineWave() method simply loops over the widgets, gets each value and passes those into the sine wave renderer: func updateSineWave() { var values = [FrequencyVelocityPair]() for widget in toneWidgets { values.append(widget.getFrequencyVelocityPair()) } sineWaveRenderer.setFrequencyVelocityPairs(values) } Once again, big thanks to Gene De Lisa who has really done all the hard work navigating the Core Audio and Audio Toolbox API research on this. All of the source code for this project is available at my GitHub repository here.