Headless Visual Testing for Web Applications
Headless Visual Testing for Web Applications
Learn why visual testing is so important to make sure your app works on different devices and operating systems and how to get started.
Join the DZone community and get the full member experience.Join For Free
xMatters delivers integration-driven collaboration that relays data between systems, while engaging the right people to proactively resolve issues. Read the Monitoring in a Connected Enterprise whitepaper and learn about 3 tools for resolving incidents quickly.
Some time back, a team had come across an interesting challenge. The team, let’s call them "Rangers," had embarked on a journey creating a cross-platform enterprise application which would allow them to write business logic once and deploy it as an iOS app, an Android app, and a web application.
The Rangers were doing great. They had precise test suites honoring the test pyramid, a well-thought-out CI system to execute them, and a clear path to production. They also implemented a UI testing framework to run on top of their current infrastructure and capture UI issues as part of the feedback. It worked flawlessly on the mobile apps...
...But not the web app. The tests were flaky and would fail for no apparent reason in some runs while they would pass on others. The Rangers spent hours debugging the root cause: the Chrome browser version, the Chrome driver, ImageMagick comparisons, and the fuzz factor. They even tried to have exact identical agents, hardware, and software images but still couldn’t pinpoint the root cause.
On verge of giving up, one Ranger asked a very important question: "Is it the physical displays on the agents?"
Is It Physical Displays?
Images are made up of pixels. The physical displays (desktop monitors, laptop screens) arrange and render pixels onto the illuminated screen. The number of pixels in an image is always constant. However, it’s representation varies across displays depending on the capability of a display to pack pixels on the screen. this capability is measured as Pixel Per Inch (PPI) and can be deduced using the formula below where height and width is the resolution set for the display (1080p, 720p, etc..) and diagonal is the diagonal length of display:
This means that for the same resolution, displays of different sizes will render the same image with a fuzz or slight variance.
The Rangers realized that the key to getting repeatable robust results is to control this PPI. However, the PPIs are the property of physical display and cannot be altered or mocked. What required was a virtual display which can be configured, looked at and taken the screenshot off. and Xvfb does just that. It’s a virtual frame buffer for X server, a full-fledged implementation which mocks the physical display with detailed configuration and even allows one to VNC!
The Rangers decided that the best robust, repeatable, and portable solution would be a containerized setup with test suites mounted and run against.
The plan was to have a Docker image with Xvfb, Chromium, and Chrome-driver setup to start at launch, along with x11vnc to provide a VNC just in case it was required to debug. The required modules for test automation viz. Selenium, ImageMagick was also baked into the image.
Consider below sample Dockerfile which creates a lightweight Docker image and starts the Xvfb and VNC:
FROM alpine RUN echo "http://dl-4.alpinelinux.org/alpine/latest-stable/main" >> /etc/apk/repositories && \ echo "http://dl-4.alpinelinux.org/alpine/latest-stable/community" >> /etc/apk/repositories RUN apk update && \ apk add xvfb x11vnc chromium chromium-chromedriver xwininfo imagemagick ARG UID=1000 ENV DISPLAY :99 EXPOSE 5900 VOLUME /images WORKDIR /app RUN echo 'echo "starting Xvfb and x11vnc"' >> /app/init.sh && \ echo "`which Xvfb` $DISPLAY -dpi 100 -screen 0 1024x768x24 &" >> /app/init.sh && \ echo "`which x11vnc` -display $DISPLAY -forever >&2 &" >> /app/init.sh && \ echo 'echo "done."' >> /app/init.sh RUN adduser -D -u $UID testrunner USER testrunner CMD ["/bin/sh","-c","sh init.sh && /bin/sh"]
The above Docker image, once built, can then be executed as
$ docker run -ti -p 5900:5900 -v ~/test_dir/images:/images <image_name>
The above will start up the Docker image, mount the directories of interest, and start Xvfb and x11vnc that can be VNC'ed at 127.0.0.1:5900.
The flakiness was gone and it opened up new possibilities of testing in isolation various different device configurations like iPhone, Samsung S8, tablets for responsive web tests using Chromium flags. But that’s a story to be told some other day.
Note: There are tools and frameworks in markets which can be used instead, but have costs associated to them. The idea here is to utilize best of open source and existing investments to get fast feedback.
Published at DZone with permission of Manish Katoch . See the original article here.
Opinions expressed by DZone contributors are their own.