Top 5 Reporting Tools for your Selenium Automation Framework

A good test reporting tool could do wonders for your test automation framework and, in turn, the quality of the application-in-test. These tools would depend on the framework that you have set up for the application. Selenium appears to be the most popular automation testing technology nowadays. Multiple reporting tools can provide a range of benefits to testing frameworks that are based on Selenium. Here is a curated list of the top 5 reporting tools that can help your testing reports provide all the necessary information and look great too.

We will start with a handful of essential tools that you can introduce to your Selenium testing framework. The tools are listed in increasing order of benefits that they have to offer.

Dive deeper into test automation framework and best test reporting tools by contacting Kobiton today!

Get in Touch with Kobiton

1. TestNG Reporting Tool

If you are working with the TestNG framework, you could do a great deal with just that. With the TestNG library, you could create a simple report or a customized one.

Simple TestNG Report

TestNG library has a default feature that generates reports in the form of an HTML file. Once the test run is complete, the test report file named ‘index.html’ will be placed in the ‘test-output’ folder under your project’s root directory. The HTML file provides valuable information such as:

• The directory path to the XML file
• Number of test cases
• Number of grouped tests
• The time duration of each test
• The name and number of passed, failed, and skipped test cases.

You also find an ‘emailable-report.html’ file under the ‘test-output folder, representing the same information in the form of a table, highlighting the failed test cases in red. This version of the report makes it easier to share it with your stakeholders.

Custom TestNG Reports using ITestListener Interface

Although a simple report gives just enough information about the test cases’ status, it may not be sufficient to debug the error, help you troubleshoot or customize your test execution report. You can add conditions to your report at certain points during execution using methods such as onTestStart(), onTestFailure(), and onFinish().

Note: It is important to remember that when you use the ITestListener interface, you will also need to include it in your XML configuration in order to invoke the method.

Now onto the disadvantages of using a TestNG reporting tool:

• It can only be accessed if you are using the TestNG framework.
• It generates reports in HTML format only.
• It does not support any language other than Java.

This tool is excellent if you are working on a TestNG framework in Java. Otherwise, investigate the tools mentioned below.

2. JUnit Reporting Tool

On a similar note, there are two classes within the JUnit library that give you the option to generate HTML reports. These exclusive classes are called the ‘JUnitTestReporter’ class and the ‘JUnitHTMLReporter’ class. These are easily accessible and require minimal coding. These are available only if you are using the JUnit framework for testing, particularly in Java, which is the most significant drawback of this tool.

JUnit Test Reporter Class:

Using this class will reap only primary benefits, for example, displaying the description of the passed or failed test result using the methods succeeded() and failed() respectively. The passed and failed test cases’ names are displayed on the console along with their test status.

Note: To utilize this tool, ensure that your test class extends the ‘JUnitTestReporter’ class. Be sure to import the TestWatcher package (import org.junit.rules.TestWatcher) from the JUnit libraries to access this class.

JUnit HTML Reporter Class:

We can use another class called ‘JUnitHTMLReporter’ inherent in the JUnit library to generate HTML reports. To use the ‘JUnitHTMLReporter’ class, you must import the Java – File (import java.io.File) as well as the BufferWriter packages (import java.io.BufferedWriter). These would provide an overview of your test report.

Along with the packages, you would need to use a couple of the JUnit annotations, such as @BeforeClass and @AfterClass. They are used to tell the test suite when to create the HTML file and the accurate information to be filled. Speaking of information, the report will display the test cases’ status, names, and error information. This tool can be quite handy when you wish to raise defects or even attach to your test cases in the application management system you use in your organization.

3. Extent Reports

Extent Reports can be adapted to JUnit, NUnit, and TestNG frameworks seamlessly. One distinct advantage that this tool has over those discussed prior is that you can capture screenshots and display them for each step of the test scenario, which is also why it is one of the most popular reporting tools in the test automation market. Using Extent Reports would be a great way of showing evidence along with the status of the test cases to the developers, managers, and other major stakeholders. Another fabulous feature of this tool is that it is open-source and easy to integrate with some widely used testing frameworks.

Extent Reports is highly customizable in that it can generate an HTML report with pie charts to represent the test results graphically. It can also display the time taken to execute the test case. Extent Reports can be best utilized when the test cases are run in batches. This way, one can track each test case’s result along with an overview of the test suite.

Extent Reports uses two classes: the ‘ExtentReports’ class and the ‘ExtentTest’ class. While the former helps create an HTML report file in the designated path, the latter aids in logging each test step into the report. These classes are used in conjunction with a few built-in methods as listed below:

• startTest()
• endTest()
• flush()
• Log()

The first two methods are used to set certain pre-and post-conditions to the test cases. The flush method erases the previous report’s data if you have chosen to override the same HTML file, creating a new report. The Log method helps log the status of the test steps to show whether it has passed, failed, or skipped. If it has failed, it displays the error that caused it to fail along with its corresponding screenshot.

Another feature that makes this tool beneficial is that it can be customized to generate interactive reports with graphical representations and dashboards, making them presentable and, thereby, emailable to stakeholders. Extent Reports is undoubtedly worth its praise and a must-try for your testing framework with these advanced reporting features.

4. Allure

Allure is yet another open-source library used to generate test reports. Let’s go straight into its advantages. Allure works with a range of testing languages such as Java, C#, Python, Ruby, PHP, and Scala. It is mainly instrumental in BDD (Behavior Driven Development) frameworks.  You can customize the visual representation to display the parameters, fixtures, steps, logs, timing, and attachments. You can integrate it with CI/CD pipeline with tools like GitLab CI and Jenkins. Just like Extent Reports, Allure also captures screenshots of each step, including failures.

The report structure is relatively comprehensive. It consists of a few pages within the report that contain features. Here are the pages and the types of information that they represent:

• Overview: Overall report statistics, report per test launch, and overall test launch statistics, user story based behavioral information, history trend represented as a graph, and information on the test environment itself. You can customize these widgets by dragging them to the desired position on the page.
• Categories: This space allows you to classify the defects that apply to your test results.
• Suites: Displays the test suites executed, grouped suites, and the classes.
• Graphs: This is where you can see visually appealing results that convey a lot of information about the test execution in a concise graphical manner. There are three graphs.
  • A pie chart shows the percentage of test cases passed, failed, skipped, or broken.
  • A bar graph represents the severity of the defects found.
  • Another bar graph represents the time taken to execute the test cases in seconds.
• Timeline: Displays a visual representation of the exact time taken to execute the test cases.
• Behaviors: This page is beneficial to the testing frameworks that use a behavior-driven approach to testing. You can see the test results grouped by the user story, feature, and epic tags on this page.
• Packages: Finally, the packages page displays a tree-like structure of the test results as the different packages in the framework group them.

In short, Allure is a flexible reporting tool that encourages major stakeholders’ involvement in the quality of the application in test.

5. Calliope.pro

Calliope.pro is an independent, continuously monitoring, DevOps test results reporting tool. This tool monitors the test results as well as share them with the agile and business teams. This tool monitors test reports over a period of time, encapsulating multiple sprints/releases.

The test reports generated as JSON and XML files in your existing testing framework must be fed to the Calliope.pro tool to create a reporting dashboard that can be analyzed in detail. You can use this tool with multiple frameworks such as Cucumber, JUnit, Rest Assured, etc. It can be used by any tester, developer, or product owner to get critical information at a glance. It helps cross-functional teams collaborate seamlessly.

If you are a manual tester, you can manually upload the test results and perform the test result analysis. In an automation testing environment, integrate the test report files (JSON or XML) into Calliope.pro, and the result is an intuitive, interactive test report/dashboard. Over time, you can view a detailed analysis of the test results, the dynamic way the test data changes, and the historical trend of the test results. These fragments of information help one analyze the health, quality, and progress of the application-in-test.

One boon of using this platform is that you can use it to schedule test execution too. Therefore, it is easy to integrate with your CI/CD pipeline. This tool may well replace management tools like Atlassian Jira due to its insightful features that will help managers make informed decisions.

Each of these tools has unique benefits. Choosing one tool or a combination of them would depend on what you seek to gather from your test results. Since all of them are open-source tools, you may want to try them out to see which one best suits your needs.