Getting Started With Apple's Vision Framework

The Vision framework was introduced by Apple in 2017 at WWDC as part of iOS 11. Its launch marked a turning point in the evolution of machine vision and image analysis, providing developers with native tools to analyse visual content and perform subsequent processing as needed.

In 2017, Vision introduced the following:

• Text recognition
• Face recognition
• Detection of rectangular shapes
• Barcode and QR code recognition

Since its debut, Apple has continuously enhanced the Vision framework, ensuring it evolves to meet modern requirements. By the end of 2024, with the release of iOS 18, Vision now offers:

• Improved text recognition accuracy with support for a large number of languages
• Detection of faces and their features
• The ability to analyse movements
• The ability to recognise poses, including the position of hands and key points of the human body
• Support for tracking objects in video
• Improved integration with CoreML for working with custom machine learning models
• Deep integration with related frameworks, such as AVKit, ARKit

With the advent of the Vision framework, developers gained the ability to perform advanced image and video analysis tasks natively, without relying on third-party solutions. These capabilities include scanning documents, recognising text, identifying faces and poses, detecting duplicate images, and automating various processes that streamline business operations.

In this article, we will look at the main scenarios of using Vision with code examples that will help you understand how to work with it, understand that it is not difficult, and start applying it in practice in your applications.

1. VNRequest

Vision has an abstract class, VNRequest, that defines data request structures in Vision, and descendant classes implement specific requests to perform specific tasks with an image.

All subclasses inherit the initializer from the VNRequest class:

public init(completionHandler: VNRequestCompletionHandler? = nil)

This returns the result of processing the request. It is important to clarify that the result of the request will be returned in the same queue in which the request was sent.

VNRequestCompletionHandler is a typealias:

public typealias VNRequestCompletionHandler = (VNRequest, (any Error)?) -> Void

The VNRecognizeTextRequest class from the abstract VNRequest class is designed to handle text recognition requests in images.

Example of implementing a request for text recognition:

import Vision
import UIKit

func recognizeText(from image: UIImage) {
    guard let cgImage = image.cgImage else { return }
    
    let request = VNRecognizeTextRequest { request, error in // 1
        guard let observations = request.results as? [VNRecognizedTextObservation] else { return } // 2
        
        for observation in observations {
            if let topCandidate = observation.topCandidates(1).first {
                print("Recognized text: \(topCandidate.string)")
                print("Text boundingBox: \(observation.boundingBox)")
                print("Accuracy: \(topCandidate.confidence)")
            }
        }
    }
    
    request.recognitionLevel = .accurate
    request.usesLanguageCorrection = true
    
    let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3
    try? handler.perform([request]) // 3
}

1. Create a VNRecognizeTextRequest for text recognition.
2. Receive the results of the text recognition request as arrays of VNRecognizedTextObservation objects. The VNRecognizedTextObservation object contains:
   • Array of recognised texts (VNRecognizedText().string)
   • Recognition accuracy (VNRecognizedText().confidence)
   • Coordinates of the recognized text on the image (VNRecognizedText().boundingBox)
3. Create a request for image processing and send a request for text recognition.
   • Example: Recognition of tax identification number and passport number when developing your own SDK for document recognition

2. VNDetectFaceRectanglesRequest

This class finds faces in an image and returns their coordinates.

Example of implementing a face recognition request:

import Vision
import UIKit

func detectFaces(from image: UIImage) {
    guard let cgImage = image.cgImage else { return }
    
    let request = VNDetectFaceRectanglesRequest { request, error in // 1
        guard let results = request.results as? [VNFaceObservation] else { return } // 2
        
        for face in results {
            print("Face detected: \(face.boundingBox)")
        }
    }
    
    let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3
    try? handler.perform([request]) // 3
}

• Receive the results of the text recognition request as arrays of VNFaceObservation objects.

The VNFaceObservation object contains:

1. Coordinates of the recognized face – VNFaceObservation().boundingBox
2. Create a request for image processing and send a request for face recognition.
3. Example: In banks, there is a KYC onboarding where you have to take a photo with your passport. This way, you can confirm that this is the face of a real person.

3. VNDetectBarcodesRequest

This class recognizes and reads barcodes and QR codes from an image.

Example of implementing a request for recognizing and reading a barcode and QR code:

import Vision
import UIKit

func detectBarcodes(from image: UIImage) {
    guard let cgImage = image.cgImage else { return }
    
    let request = VNDetectBarcodesRequest { request, error in // 1
        guard let results = request.results as? [VNBarcodeObservation] else { return } // 2
        
        for qrcode in results {
            print("qr code was found: \(qrcode.payloadStringValue ?? "not data")")
        }
    }
    
    let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3
    try? handler.perform([request]) // 3
}

1. Create a VNDetectBarcodesRequest for recognition.
2. Get the results of the VNBarcodeObservation object array request.

The VNBarcodeObservation object contains many properties, including:

1. VNFaceObservation().payloadStringValue – the string value of the barcode or QR code
2. Create a request for image processing and send a request for face recognition.
3. Example: QR scanner for reading QR codes

We've covered the three main types of queries in Vision to help you get started with this powerful tool.