How to Make a Picture-in-Picture Feature in iOS App Using AVFoundation

AVFoundation works on Apple’s entire product line. With this framework, you can add recording, editing, and playback features to your apps. In this article, I will show how to place one video over another and thus enable a picture-in-picture (“duet”) mode – one of the most popular TikTok-like features. We are going to use only the native tools for this job.

Building Blocks

This is what we need for this task:

• AVPlayerViewController lets you play the videos and has the UI and standard controls.
• AVAssetExportSession lets you set the format for video export. With this, your app will save the final clip to the device’s storage.
• AVMutableComposition will be used to manage and modify video/audio tracks.
• AVMutableVideoComposition will define the way the tracks mix for each frame of the resulting video.
• AVMutableVideoCompositionInstruction lets you set the instructions for the changes of the layers in the output video.
• AVMutableVideoCompositionLayerInstruction allows changing the opacity of a certain video track. It also allows cropping and resizing the video – just what we need to make the overlay clip small and place it at the location we want.

In this example, we will use two videos. The main video asset is the one that will be placed behind the other one and played in full size. The overlay video asset will be placed in the foreground, over the main video asset.

Step By Step

Start by loading the videos. Let’s assume that they are both located inside the application bundle as resources:

let mainVideoUrl = Bundle.main.url(forResource: "main", withExtension:

"mov")!

let overlayVideoUrl = Bundle.main.url(forResource: "overlay",

withExtension: "mov")!

let mainAsset = AVURLAsset(url: mainVideoUrl)

let overlayAsset = AVURLAsset(url: overlayVideoUrl)

There are two videos, so make two blank tracks in AVMutableComposition and the appropriate composition:

let mutableComposition = AVMutableComposition()

guard let mainVideoTrack =

mutableComposition.addMutableTrack(withMediaType: AVMediaType.video,

preferredTrackID: kCMPersistentTrackID_Invalid),

let overlayVideoTrack =

mutableComposition.addMutableTrack(withMediaType: AVMediaType.video,

preferredTrackID: kCMPersistentTrackID_Invalid) else {

// Handle error

return

}

The next step is to place the tracks from AVURLAsset into the composition tracks:

let mainAssetTimeRange = CMTimeRange(start: .zero, duration:

mainAsset.duration)

let mainAssetVideoTrack = mainAsset.tracks(withMediaType:

AVMediaType.video)[0]

try mainVideoTrack.insertTimeRange(mainAssetTimeRange, of:

mainAssetVideoTrack, at: .zero)

let overlayedAssetTimeRange = CMTimeRange(start: .zero, duration:

overlayAsset.duration)

let overlayAssetVideoTrack = overlayAsset.tracks(withMediaType:

AVMediaType.video)[0]

try overlayVideoTrack.insertTimeRange(overlayedAssetTimeRange, of:

overlayAssetVideoTrack, at: .zero)

Then make two instances of AVMutableVideoCompositionLayerInstruction:

let mainAssetLayerInstruction =

AVMutableVideoCompositionLayerInstruction(assetTrack: mainVideoTrack)

let overlayedAssetLayerInstruction =

AVMutableVideoCompositionLayerInstruction(assetTrack: overlayVideoTrack)

Now use the setTransform method to shrink the overlay video and place it where you need it to be. In this case, we will make it 50% smaller than the main video in both dimensions, so we will use identity transform scaled by 50%. Using the transform, we will also adjust the position of the overlay.

let naturalSize = mainAssetVideoTrack.naturalSize

let halfWidth = naturalSize.width / 2

let halfHeight = naturalSize.height / 2

let topLeftTransform: CGAffineTransform = .identity.translatedBy(x: 0, y: 0).scaledBy(x: 0.5, y: 0.5)

let topRightTransform: CGAffineTransform = .identity.translatedBy(x: halfWidth, y: 0).scaledBy(x: 0.5, y: 0.5)

let bottomLeftTransform: CGAffineTransform = .identity.translatedBy(x: 0, y: halfHeight).scaledBy(x: 0.5, y: 0.5)

let bottomRightTransform: CGAffineTransform = .identity.translatedBy(x: halfWidth, y: halfHeight).scaledBy(x: 0.5, y: 0.5)

overlayedAssetLayerInstruction.setTransform(topRightTransform, at: .zero)

Note that in this example, both videos are in the 1920x1080. If the resolutions differ, the calculations need to be adjusted.

Now create AVMutableComposition and AVMutableVideoComposition. You will need them to play and export the video.

let instruction = AVMutableVideoCompositionInstruction()

instruction.timeRange = mainAssetTimeRange

instruction.layerInstructions = [overlayedAssetLayerInstruction,

mainAssetLayerInstruction]

mutableComposition.naturalSize = naturalSize

mutableVideoComposition = AVMutableVideoComposition()

mutableVideoComposition.frameDuration = CMTimeMake(value: 1, timescale:

30)

mutableVideoComposition.renderSize = naturalSize

mutableVideoComposition.instructions = [instruction]

The output video with the picture-in-picture mode enabled will be played through AVPlayerViewController. First, create a new instance of AVPlayer and pass it as a reference to AVPlayerItem that will use AVMutableVideoComposition and AVMutableComposition that you made earlier. 

If you get the "Cannot find 'AVPlayerViewController' in scope" error, you forgot to import the AVKit framework in your source file.

let playerItem = AVPlayerItem(asset: mutableComposition)

playerItem.videoComposition = mutableVideoComposition

let player = AVPlayer(playerItem: playerItem)

let playerViewController = AVPlayerViewController()

playerViewController.player = player

Now that you have the player, you have two options:

• Fullscreen mode: Use UIViewController methods like present().
• In line with other views: Use APIs like addChild() and didMove(toParent:). 

And finally, we need to export the video with AVAssetExportSession:

guard let session = AVAssetExportSession(asset: mutableComposition,

presetName: AVAssetExportPresetHighestQuality) else {

// Handle error

return

}

let tempUrl =

FileManager.default.temporaryDirectory.appendingPathComponent("\

(UUID().uuidString).mp4")

session.outputURL = tempUrl

session.outputFileType = AVFileType.mp4

session.videoComposition = mutableVideoComposition

session.exportAsynchronously {

// Handle export status

}

This is what you should get in the end:

You can also modify the final video, e.g. round out the edges, add shadow, or a colored border. To do that, create a custom compositor class for the AVVideoCompositing protocol and a corresponding instruction for AVVideoCompositionInstructionProtocol.