Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Katalon Studio vs Selenium-Based Open Source Frameworks

DZone's Guide to

Katalon Studio vs Selenium-Based Open Source Frameworks

In this article, we compare a few testing frameworks that developers and testers might find helpful when dealing with performance issues.

· Performance Zone
Free Resource

1. Target Users

Software test automation usually needs technical skills and some knowledge about the application under test (AUT) for its successful application. Knowledge about an AUT’s related technologies is not a must but is recommended. For example, having knowledge of HTML and JavaScript would be a great help if you are testing on the web.

Katalon Studio’s users could be testers with limited technical knowledge. Katalon Studio hides all technical complexities behind the scenes and provides a friendly UI with the manual mode (users can drag-drop, select keywords, and test objects to form test steps), but still keeps the necessary tools for more technically inclined users who are able to dig deeper into coding with the scripting mode that fully supports development conveniences like syntax highlighting, code suggestion, and debugging.

Selenium and Appium mainly have automated testing libraries which are more suitable for testers who possess good programming skills. They will typically need to use an IDE like Eclipse or Visual Studio, to import Selenium and Appium libraries (and certainly their dependencies) into a development project and to install necessary unit testing frameworks like JUnit, NUnit and TestNG before starting to write a test case.

Let’s imagine an analogy in the automotive industry where many cars share the same engine while manufacturers design their cars’ interior and exterior differently, depending on their cars’ models.

Similarly, Katalon Studio is basically an all-in-one solution. But if you still prefer (and have the necessary skills) to build everything from scratch on your own, you can pick up an engine like Selenium, Appium, etc. to build another automated testing solution like Katalon Studio. Besides Katalon Studio, there are a few test frameworks built on top of Selenium and Appium such as Robot Framework (http://robotframework.org).

2. Deployment and Getting Started

Most open source projects are hard to setup and deploy because many different developers and providers who submit their work into a central place, i.e. GitHub, contribute to them. You may have to build everything using the submitted source code. But even if frameworks are packaged into a ready-to-use package like Selenium and Appium, you still have to go through technical guidelines for their API and integrate them with an IDE and a preferred programming language. Below are high-level steps required by Selenium, Appium, and Katalon Studio to setup and run.

  • Selenium:
    • Download and install Java SDK.
    • Download and set up test libraries: Selenium client API of your preferred language, Selenium Server, or a standalone package (a bundle of client and server in only one library).
    • Download IEDriver and ChromeDriver, set up the environment and PATH variables.
    • Download and set up the development IDE and your preferred testing frameworks like JUnit, NUnit, and TestNG.
    • Go through the Selenium API document in order to start to create a project and writing some code.
  • Appium:
    • Download and install the SDK for your programming language; either Java, Python, Ruby or C#.
    • Download and set up test libraries (Appium, Node.js, Android SDK, XCode if you're developing for Mac and iOS), set up the environment and PATH variables.
    • Download and set up the development IDE and your preferred testing frameworks like JUnit, NUnit, and TestNG.
    • Go through the Selenium API document in order to start to create a project and writing some code.
  • Katalon Studio is built as a unified bundle which includes almost all necessary things like Java, Android SDK, Web drivers to drive browsers, and required dependencies. All you have to do is to download and install it on your computer. Minimum installation is required:
    • As Appium is frequently updated, Katalon Studio does not bundle Appium. If you are testing mobile apps, you have to install Appium, Node.js (on Windows) and XCode (on Mac).
    • You can quickly get familiar with Katalon Studio as it has friendly UI and sample projects. The detailed user documentation (docs.katalon.com) definitely helps you go through quickly from its simple to advanced features.

3. Test Management and Scripting Language Support:

  • Management of test artifacts:
    • Katalon Studio provides a rich graphical UI with menus, tree views, tables, etc. to manage test cases, objects, and data files. Its composer for scripting with full syntax highlighting, code suggestion, and debugging is ideal for testers who just want to drag-drop or have limited programming skills.

Katalon 1

Katalon Studio manual mode.

Katalon 2

Katalon Studio scripting mode.

Katalon 3

Katalon Studio debug mode.

  • Selenium and Appium, when integrating with a dedicated development IDE like Eclipse and Visual Studio, offer intuitive environments for developers but unfamiliar and unnecessarily complex features for manual or non-technical testers. Managing tests is challenging for non-technical testers when using Selenium and Appium.

Katalon 4

The development mode of Selenium + TestNG + Eclipse.

Katalon 5

Test cases, Test suites, test data are mainly organized in XML file and programming code.

Scripting language support:

  • Katalon Studio: Java/Groovy is the only option for now.
  • Selenium, Appium + an IDE: Many choices for scripting languages.

4. Performance

Katalon Studio uses Groovy, a language built on top of Java, and has to load many libraries for parsing test data, test objects, and logging. So, it may be a bit slower as compared to Java for long test cases with hundreds of steps.

5. Usability and Functionality

5.1 Object Spy:

Object spy is the most important and time-consuming activity in test automation. Katalon Studio supports object spy for the web (by using browser plugins for IE, Firefox, Chrome) and for mobile apps (by using the screenshot simulator). Katalon Studio also supports high-lighting captured objects in AUT:

Katalon 6

Katalon Object Spy with object highlighting support.

Selenium does not provide this feature, and testers have to manually do object spy by using some browser add-ons, such as Firebug for Firefox.

Appium provides a companion mobile object spy tool which is useful, but it has no highlighting feature.

Katalon 7

5.2 Test Data Management and Data-Driven Testing

  • Katalon Studio provides a Data File object which can query data from external sources: CSV files, Excel files, Relational DBs. Katalon Studio also supports data driven testing on a data file object. For example, when testing the login feature, you can specify 100 pre-defined accounts in an Excel spreadsheet or a database.
  • Selenium and Appium: testers have to hard-code test data directly in the code. Any support for external data sources requires considerable programming effort.

5.3 Recording and Test Generation

Test recording is the best way to help users learn and quickly adopt to test automation. It helps an automation tester to do two primary jobs: capturing objects (during recording) and generating test steps with appropriate actions or keywords, saving at least 50-60% of the time needed to compose test cases.

  • Katalon Studio provides browser plugins for recording tests on the web:

Katalon 8

  • Selenium has Selenium IDE to record user actions and can export into JUnit/TestNG test cases, but object identification information is hard-coded directly into generated code without object management support:

Katalon 9

Test Recording with Selenium IDE.

Katalon 10

Test case generated by Selenium IDE.

  • Both Katalon Studio and Appium do not support recording on mobile apps.

5.4 Built-in Keywords

  • Katalon Studio is released with pre-defined sets of commonly used keywords or actions, users will find them enough to start implementing most test cases, but if they have more advanced or specific needs they still have a way to create custom keywords for their projects. Custom keywords are an extension of built-in keywords. They can be used the same way as built-in keywords and can be shared between users.
  • Selenium and Appium: Users have to build common and reusable actions on their own in forms of programming functions.

5.5 Reporting

  • Katalon Studio provides intuitive reports with friendly and readable messages and captured screenshots at the time of failure which can be exported into CSV, HTML, and PDF files.

Katalon 11

Katalon Report with screenshot attachment associated with failure.

  • Selenium and Appium do not have reporting tools themselves, but based on the testing framework being used will have some simple report templates.

Katalon 12

TestNG Report with programming stack-trace, no screenshot of failure.

6. Integration

  • Katalon Studio can integrate with several test management systems and bug tracking systems including qTest (https://www.qasymphony.com) and JIRA to submit bugs, defects, test logs, and test results. Katalon Studio can also integrate with Kobiton, a cloud-based mobile lab for cloud-based mobile testing (https://kobiton.com).
  • Selenium and Appium: no integration, users have to build it themselves.

7. CI Integration

  • Katalon Studio provides a command line interface, allowing CI Systems to call and trigger test execution.
  • Selenium and Appium: users have to create and configure execution jobs (Ant tasks, maven goals…) to trigger test execution.

8. Customer Support

  • The Katalon team actively supports Katalon Studio’s users more regular than the Selenium and Appium communities, via its website and forum.
  • Selenium and Appium are open source, and users may find online help from other users or the Selenium group.
Topics:
selenium ,open source framework ,performance ,performance testing ,performance tests

Published at DZone with permission of Oliver Howard. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}