Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Full-Stack Test Automation Frameworks — Video Recording on Test Failure

DZone 's Guide to

Full-Stack Test Automation Frameworks — Video Recording on Test Failure

Learn how to create a cross-platform video recording engine to record your automated tests when they fail for ease of troubleshooting.

· DevOps Zone ·
Free Resource

Some of the must-have features for 5th generation frameworks are related to ease of troubleshooting. With increasing test count and complexity, it is even more critical that the tests be maintainable. A significant part of this effort is easier troubleshooting and better support for locating errors.

A big part of maintainability is troubleshooting existing tests. Most in-house or open-source solutions don't provide lots of features to make your life easier. This can be one of the most time-consuming tasks, having 100 failing tests and finding out whether there is a problem with the test or a bug in the application. If you use plugins or complicated design patterns, debugging the tests will be much harder, requiring lots of resources and expertise.

Two of the ways Bellatrix as full-stack test automation framework handles these problems are through full-page screenshots and video recording on test failure.

In this article, I am going to show you how to create a similar cross-platform video recording engine. Moreover, you will be able to configure the engine entirely via attributes. The tests’ integration is implemented through the Observer Design Pattern (previously discussed in the series).

Cross-Platform Video Recording Engine

IVideoRecorder Interface

The first thing that we need to do is to create a new interface for our video recorder. It should be as simple as possible. Our interface derives from the IDisposable interface because the recording engines need to free some resources before and after the saving of the recorded video.

public interface IVideoRecorder : IDisposable
{
    string Record(string filePath, string fileName);
    void Stop();
}


FFmpegVideoRecorder

public class FFmpegVideoRecorder : IVideoRecorder
{
    private Process _recorderProcess;
    private bool _isRunning;
    public void Dispose()
    {
        if (_isRunning)
        {
            // Wait for 500 milliseconds before finishing video
            Thread.Sleep(500);
            if (!_recorderProcess.HasExited)
            {
                _recorderProcess?.Kill();
                _recorderProcess?.WaitForExit();
            }
            _isRunning = false;
        }
    }
    public string Record(string filePath, string fileName)
    {
        string videoPath = $"{Path.Combine(filePath, fileName)}";
        string videoFilePathWithExtension = GetFilePathWithExtensionByOS(videoPath);
        try
        {
            if (!Directory.Exists(filePath))
            {
                Directory.CreateDirectory(filePath);
            }
        }
        catch (Exception ex)
        {
            throw new ArgumentException($"A problem occurred trying to initialize the create the directory you have specified. - {filePath}", ex);
        }
        if (!_isRunning)
        {
            var startInfo = GetProcessStartInfoByOS(videoFilePathWithExtension);
            _recorderProcess = Process.Start(startInfo);
            _isRunning = true;
        }
        return videoFilePathWithExtension;
    }
    public void Stop() => Dispose();
    private string GetFFmpegPath() => Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location) ?? throw new InvalidOperationException(), "ffmpeg.exe");
    private ProcessStartInfo GetProcessStartInfoByOS(string videoFilePathWithExtension)
    {
        var startInfo = new ProcessStartInfo
        {
            FileName = GetFFmpegPath(),
            RedirectStandardInput = true,
            RedirectStandardOutput = true,
            RedirectStandardError = true,
            UseShellExecute = false,
            CreateNoWindow = false,
        };
        if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
        {
            startInfo.Arguments = $"-f gdigrab -framerate 30 -i desktop {videoFilePathWithExtension}";
        }
        else if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
        {
            startInfo.Arguments = $"-f avfoundation -framerate 30 -i default {videoFilePathWithExtension}";
        }
        else if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
        {
            startInfo.Arguments = $"-f x11grab -framerate 30 -i :0.0+100,200 {videoFilePathWithExtension}";
        }
        else
        {
            throw new NotSupportedException("The OS is not supported by FFmpeg video recorder. Currently supported OS are Windows, MacOS, Linux.");
        }
        return startInfo;
    }
    private string GetFilePathWithExtensionByOS(string videoPathNoExtension)
    {
        string videoPathWithExtension;
        if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
        {
            videoPathWithExtension = $"{videoPathNoExtension}.mpg";
        }
        else if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
        {
            videoPathWithExtension = $"{videoPathNoExtension}.mov";
        }
        else if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
        {
            videoPathWithExtension = $"{videoPathNoExtension}.mp4";
        }
        else
        {
            throw new NotSupportedException("The OS is not supported by FFmpeg video recorder. Currently supported OS are Windows, MacOS, Linux.");
        }
        return videoPathWithExtension;
    }
}


There are a few primary parts of the code. When you call the Record method, a new C# process is created. Based on the current OS, different arguments are passed to the ffmpeg.exe since we use different encoders for a different OS. Also, there is a separate method for choosing the right file extension, based again on the OS. Once the process is started, the recorder captures everything in the background. To stop it, we call the Dispose method, which kills the recorder's process. When this happens, the results file is saved.

Video Recording Engine — Integration in Tests

VideoRecordingAttribute

We are going to configure the recording engine in the same manner as the execution engine discussed in the previous article from the series (Dynamically Configure Execution Engine). The attribute can be set again on the class or method level. This time, it contains a single property of the type VideoRecordingMode, which specifies when the video recording should be performed.

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method,
Inherited = true,
AllowMultiple = false)]
public sealed class VideoRecordingAttribute : Attribute
{
    public VideoRecordingAttribute(VideoRecordingMode videoRecordingMode)
    {
        this.VideoRecording = videoRecordingMode;
    }
    public VideoRecordingMode VideoRecording { get; set; }
}


VideoRecordingMode

You can always record the tests' execution, or only for failed/passed tests. Also, you can turn off the recording entirely.

public enum VideoRecordingMode
{
    Always,
    DoNotRecord,
    Ignore,
    OnlyPass,
    OnlyFail
}


VideoBehaviorObserver

I am not going to explain again in much detail how the Observer solution works. If you haven't read my articles about it, I suggest you do so: Advanced Observer Design Pattern via Events and Delegates and Dynamically Configure Execution Engine. In the PreTestInit method, we get the specified VideoRecordingMode. The global app.config configuration ShouldTakesVideosOnExecution, if set, overrides all attributes. As in the previous examples, the attributes on the method level override those on the class level. We get the values of these attributes through reflection. If the mode is not equal to DoNotRecord, we start the video recording.

public class VideoWorkflowPlugin : BaseTestBehaviorObserver
{
    private readonly IVideoRecorder _videoRecorder;
    private readonly IVideoRecorderOutputProvider _videoRecorderOutputProvider;
    private VideoRecordingMode _recordingMode;
    private string _videoRecordingPath;
    public VideoWorkflowPlugin(IVideoRecorder videoRecorder, IVideoRecorderOutputProvider videoRecorderOutputProvider)
    {
        _videoRecorder = videoRecorder;
        _videoRecorderOutputProvider = videoRecorderOutputProvider;
    }
    protected override void PostTestInit(object sender, TestExecutionEventArgs e)
    {
        _recordingMode = ConfigureTestVideoRecordingMode(e.MemberInfo);
        if (_recordingMode != VideoRecordingMode.DoNotRecord)
        {
            var fullTestName = $"{e.MemberInfo.DeclaringType.Name}.{e.TestName}";
            var videoRecordingDir = _videoRecorderOutputProvider.GetOutputFolder();
            var videoRecordingFileName = _videoRecorderOutputProvider.GetUniqueFileName(fullTestName);
            _videoRecordingPath = _videoRecorder.Record(videoRecordingDir, videoRecordingFileName);
        }
    }
    protected override void PostTestCleanup(object sender, TestExecutionEventArgs e)
    {
        if (_recordingMode != VideoRecordingMode.DoNotRecord)
        {
            try
            {
                bool hasTestPassed = e.TestOutcome.Equals(TestOutcome.Passed);
                DeleteVideoDependingOnTestOutcome(hasTestPassed);
            }
            finally
            {
                _videoRecorder.Dispose();
            }
        }
    }
    private void DeleteVideoDependingOnTestOutcome(bool haveTestPassed)
    {
        if (_recordingMode != VideoRecordingMode.DoNotRecord)
        {
            bool shouldRecordAlways = _recordingMode == VideoRecordingMode.Always;
            bool shouldRecordAllPassedTests = haveTestPassed && _recordingMode.Equals(VideoRecordingMode.OnlyPass);
            bool shouldRecordAllFailedTests = !haveTestPassed && _recordingMode.Equals(VideoRecordingMode.OnlyFail);
            if (!(shouldRecordAlways || shouldRecordAllPassedTests || shouldRecordAllFailedTests))
            {
                // Release the video file then delete it.
                _videoRecorder.Stop();
                if (File.Exists(_videoRecordingPath))
                {
                    File.Delete(_videoRecordingPath);
                }
            }
        }
    }
    private VideoRecordingMode ConfigureTestVideoRecordingMode(MemberInfo memberInfo)
    {
        VideoRecordingMode methodRecordingMode = GetVideoRecordingModeByMethodInfo(memberInfo);
        VideoRecordingMode classRecordingMode = GetVideoRecordingModeType(memberInfo.DeclaringType);
        VideoRecordingMode videoRecordingMode = VideoRecordingMode.DoNotRecord;
        var shouldTakeVideos = bool.Parse(ConfigurationManager.AppSettings["shouldTakeVideosOnExecution"]);
        if (methodRecordingMode != VideoRecordingMode.Ignore && shouldTakeVideos)
        {
            videoRecordingMode = methodRecordingMode;
        }
        else if (classRecordingMode != VideoRecordingMode.Ignore && shouldTakeVideos)
        {
            videoRecordingMode = classRecordingMode;
        }
        return videoRecordingMode;
    }
    private VideoRecordingMode GetVideoRecordingModeByMethodInfo(MemberInfo memberInfo)
    {
        if (memberInfo == null)
        {
            throw new ArgumentNullException();
        }
        var recordingModeMethodAttribute = memberInfo.GetCustomAttribute<VideoRecordingAttribute>(true);
        if (recordingModeMethodAttribute != null)
        {
            return recordingModeMethodAttribute.VideoRecording;
        }
        return VideoRecordingMode.Ignore;
    }
    private VideoRecordingMode GetVideoRecordingModeType(Type currentType)
    {
        if (currentType == null)
        {
            throw new ArgumentNullException();
        }
        var recordingModeClassAttribute = currentType.GetCustomAttribute<VideoRecordingAttribute>(true);
        if (recordingModeClassAttribute != null)
        {
            return recordingModeClassAttribute.VideoRecording;
        }
        return VideoRecordingMode.Ignore;
    }
}

Most of the work is done in the PostTestCleanup method of the observer. Based on the test's outcome and the specified video recording mode, we decide whether to save the file or not. All of the code here is surrounded by a try-catch-finally. In the "finally" block, we call the Dispose method of the video recording engine. To make the code more testable, we use IVideoRecorderOutputProvider to get the output folder and the video file name.

VideoRecorderOutputProvider

public class VideoRecorderOutputProvider : IVideoRecorderOutputProvider
{
    public string GetOutputFolder()
    {
        var outputDir = ConfigurationManager.AppSettings["videosFolderPath"];
        if (!Directory.Exists(outputDir))
        {
            Directory.CreateDirectory(outputDir);
        }
        return outputDir;
    }
    public string GetUniqueFileName(string testName) => string.Concat(testName, Guid.NewGuid().ToString());
}


Test Examples

SearchEngineTests

Additional to the previously created ExecutionEngineAttribute, we add the new VideoRecordingAttribute. It is configured to save the videos only for the failed tests. Because of that, we fail the tests through the Assert.Fail() method. The videos are saved in the folder specified in the app.config. Also, ffmpeg.exe is copied always to the bin folder.

[TestClass,
ExecutionEngineAttribute(ExecutionEngineType.WebDriver, Browsers.Chrome),
VideoRecordingAttribute(VideoRecordingMode.OnlyFail)]
public class BingTests : BaseTest
{
    [TestMethod]
    public void SearchForAutomateThePlanet()
    {
        var bingMainPage = Container.Resolve<BingMainPage>();
        bingMainPage.Navigate();
        bingMainPage.Search("Automate The Planet");
        bingMainPage.AssertResultsCountIsAsExpected(264);
        Assert.Fail();
    }
    [TestMethod]
    public void SearchForAutomateThePlanet1()
    {
        var bingMainPage = Container.Resolve<BingMainPage>();
        bingMainPage.Navigate();
        bingMainPage.Search("Automate The Planet");
        bingMainPage.AssertResultsCountIsAsExpected(264);
        Assert.Fail();
    }
}


Topics:
devops ,test automation ,automated testing ,tutorial

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}