Listeners

TestNG Listeners listen to the events in your test scripts and modify the behavior. Listeners are used mostly for logging or reporting. TestNG provides several listeners in the form of interfaces that allow you to modify TestNG's behavior. These interfaces are broadly called TestNG Listeners. Some of the important listeners are as follows:

IAnnotationTransformer
IHookable
IInvokedMethodListener
IMethodInterceptor
IReporter
ISuiteListener
ITestListener

In TestNG listeners can be implemented in following ways:

By adding listeners element in testng.xml file

XML

Copy

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Listeners Example">
    <listeners>
        <listener class-name="org.example.Listener1"></listener>
    </listeners>
    <test name="Listeners Example">
        <classes>
            <class name="org.example.ExampleTest1"/>
        </classes>
    </test>
</suite>

By adding @Listener annotation in java file.

Java

Copy

package org.example;

import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
@Listeners({Listener1.class})
public class ListenerExample {
    @Test
    public void ExampleTest1Method1()
    {
        System.out.println("Example Method 1");
    }
}

The @Listener annotation can contain any class that implements ITestNGListener. Since listener annotations will apply to your entire suite, just as if you had specified it in a testng.xml file.

Note:

IAnnotationTransformer is used to rewrite the annotations. So this listener need to known very early in the process and hence this is the one you need to write in testng.xml file.

Now let's learn about each listeners.

IAnnotationTransformer

Annotation Transformers are used to updated the annotation attribute in the run time. This method will be invoked by TestNG to modify any TestNG annotation read from your test classes. You can change the values you need by calling any of the setters on the ITest interface.

Let's understand this by an example.

Create a java file Listener1.java

Java

Copy

package org.example;

import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
public class Listener1 implements IAnnotationTransformer {
    @Override
    public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod)
    {
        int newInvocationCount = 5;
        if(testMethod.getName().equals("ExampleTest1Method1"))
        {
            System.out.println("The invocation count of method " + testMethod.getName() + " is changed to " + newInvocationCount);
            annotation.setInvocationCount(newInvocationCount);
        }
    }
}

Create a java file ListenerExample.java

Java

Copy

package org.example;

import org.testng.annotations.Test;
public class ListenerExample {
    @Test(invocationCount = 2)
    public void ExampleTest1Method1()
    {
        System.out.println("Example Method 1");
    }
}

Now to run this test we need to create a xml file listener1.xml

XML

Copy

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Listeners Example">
    <listeners>
        <listener class-name="org.example.Listener1"></listener>
    </listeners>
    <test name="Listeners Example">
        <classes>
            <class name="org.example.ExampleTest1"/>
        </classes>
    </test>
</suite>

In the above example, we have created a listener file Listener1.java for IAnnotationTransformer. This listener file will modify the annotation attribute newInvocationCount of test method ExampleTest1Method1 and set it to 5 from 2.

Then we have created a test file ListenerExample.java. Since IAnnotationTransformer needs to be loaded early in the process, we have created an xml suite file listener1.xml to run the suite. Now run the listener1.xml file and you should see that the ExampleTest1Method1 run 5 time.

IHookable

Using IHookable listener TestNG allows you to override and possibly skip the invocation of test methods. If a test class implements this interface, its run() method will be invoked instead of each @Test method found within. The invocation of the test method will then be performed upon invocation of the callBack() method of the IHookCallBack parameter.

In the following example, we will implement a IHookable listener interface and run a test with data provider. We will override the test invocation and skip based on the parameter value.

Create a test DataProviderExample.java

Java

Copy

package org.example;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners({IHookableListener.class})
public class DataProviderExample {
    @Test(dataProvider = "dataPrviderExample")
    public void ExampleTestMethod(String firstName, String lastName, int age)
    {
        System.out.println("First Name is " + firstName);
        System.out.println("Last Name is " + lastName);
        System.out.println("Age is " + age);
    }
    @DataProvider(name = "dataPrviderExample")
    public static Object[][] UserData() {
        return new Object[][] {{"James", "Smith", 26}, {"John", "Ioannidis", 76}, {"Marie", "Curie", 66}};
    }
}

Create a IHookable listener IHookableListener.java

Java

Copy

package org.example;

import org.testng.IHookCallBack;
import org.testng.IHookable;
import org.testng.ITestResult;

public class IHookableListener implements IHookable {
    @Override
    public void run(IHookCallBack iHookCallBack, ITestResult iTestResult)
    {
        if(iHookCallBack.getParameters()[0].equals("John"))
        {
            iTestResult.setStatus(ITestResult.SKIP);
        }
        else
        {
            iHookCallBack.runTestMethod(iTestResult);
        }
    }
}

Run the test DataProviderExample.java and you will see the TestNG skipped the test iteration with data having First Name John. The log should look as below:


First Name is James

Last Name is Smith

Age is 26


Test Method : ExampleTestMethod is skipped.


Test ignored.

First Name is Marie

Last Name is Curie

Age is 66


===============================================

Default Suite

Total tests run: 3, Passes: 2, Failures: 0, Skips: 1

===============================================

IInvokedMethodListener

IInvokedMethodListener gets invoked before and after a method is invoked by TestNG. This listener will be invoked for configuration and test methods irrespective of whether they pass/fail or get skipped. It implements two methods beforeInvocation and afterInvocation. Let's understand this by an example.

In the following example we we have created a file IInvokedMethodListenerClass.java which implements IInvokedMethodListener and implemented its two methods beforeInvocation and afterInvocation. Second file a IInvocationMethodListenerTest.java in which we have a test method and all the @before and @after methods.

IInvokedMethodListenerClass.java

Java

Copy

package org.example;

import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;

public class IInvokedMethodListenerClass implements IInvokedMethodListener {
    @Override
    public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
        System.out.println("Inside InvocationMethod Listener. After " + method.getTestMethod().getMethodName());
    }
    @Override
        public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
        System.out.println("Inside InvocationMethod Listener. Before " + method.getTestMethod().getMethodName());
    }
}

IInvocationMethodListenerTest.java

Java

Copy

package org.example;

import org.testng.annotations.*;
@Listeners({IInvokedMethodListenerClass.class})
public class IInvocationMethodListenerTest {
    @BeforeSuite
    public void beforeSuite()
    {
        System.out.println("Before Suite");
    }
    @BeforeTest
    public void beforeTest()
    {
        System.out.println("Before Test");
    }
    @BeforeClass
    public void beforeClass()
    {
        System.out.println("Before Class");
    }
    @BeforeMethod
    public void beforeMethod()
    {
        System.out.println("Before Method");
    }
    @Test
    public void IInvocationMethodListenerTestMethod1()
    {
        System.out.println("IInvocationMethodListenerTest Method 1");
    }
    @AfterMethod
    public void afterMethod()
    {
        System.out.println("After Method");
    }
    @AfterClass
    public void afterClass()
    {
        System.out.println("After Class");
    }
    @AfterTest
    public void afterTest()
    {
        System.out.println("After Test");
    }
    @AfterSuite
    public void afterSuite()
    {
        System.out.println("After Suite");
    }
}

Run the IInvocationMethodListenerTest.java and you will see before all the @Before methods and @Test beforeInvocation method inside the IInvokedMethodListenerClass is invoked. Similarly after every @After method and @Test methd IInvokedMethodListenerClass afterInvocation inside IInvokedMethodListenerClass is invoked.

IMethodInterceptor

IMethodInterceptor is used to alter the list of test methods that TestNG is about to run. An instance of this class will be invoked right before TestNG starts invoking test methods. Only methods that have no dependents and that don't depend on any other test methods will be passed in parameter. When you implement this interface, you need to return a list of IMethodInstance that represents the list of test methods they want run. TestNG will run these methods in the same order found in the returned value. Typically, the returned list will be just the methods passed in parameter but sorted differently, but it can actually have any size which mean it can be empty or it can be of the same size as the original list or it can contain more methods).
The ITestContext is passed in the intercept method as second parameter so that when you implement, you can set user values using ITestContext.setAttribute(String, Object), which you can then look up later while generating the reports.

Let's understand this by following example.
First we will create listener IMethodInterceptorClass.java and implement it's intercept method. This method will have two parameter List (list of method instances) and ITestContext. This method will expect a List as return value. Within this method we ca have all the filtering and re-sorting of the method order based on our criteria.
Second file we will create a IMethodInterceptorTest.java where we will have 3 test methods in that two test methods will be part of group fast. The expectation from the listener is the re-order the test methods based on the group name. Without listener the methods will run in the default order of the test method.

IMethodInterceptorClass.java

Java

Copy

package org.example;

import org.testng.IMethodInstance;
import org.testng.IMethodInterceptor;
import org.testng.ITestContext;
import org.testng.annotations.Test;

import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

public class IMethodInterceptorClass implements IMethodInterceptor {
    @Override
    public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
        List<IMethodInstance> result = new ArrayList<IMethodInstance>();
        for (IMethodInstance method : methods)
        {
            Test test = method.getMethod().getConstructorOrMethod().getMethod().getAnnotation(Test.class);
            Set<String> groups = new HashSet<String>();
            for (String group : test.groups())
            {
                groups.add(group);
            }
            if (groups.contains("fast"))
            {
                result.add(0, method);
            }
            else
            {
                result.add(method);
            }
        }
        return result;
    }
}

IMethodInterceptorTest.java

Java

Copy

package org.example;

import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners({IMethodInterceptorClass.class})
public class IMethodInterceptorTest {
    @Test
    public void IMethodInterceptorTestMethod1()
    {
        System.out.println("IMethodInterceptor Test Method 1");
    }
    @Test(groups = "fast")
    public void IMethodInterceptorTestMethod2()
    {
        System.out.println("IMethodInterceptor Test Method 2");
    }
    @Test(groups = "fast")
    public void IMethodInterceptorTestMethod3()
    {
        System.out.println("IMethodInterceptor Test Method 3");
    }
}

If you run the IMethodInterceptorTest.java file, you will see below result as we have filtered the method instances based on the group fast and prioritized these method to run in the first position in the list. As you can see in the below log, the Test Method 3 is executed first.


IMethodInterceptor Test Method 3

IMethodInterceptor Test Method 2

IMethodInterceptor Test Method 1


===============================================

Default Suite

Total tests run: 3, Passes: 3, Failures: 0, Skips: 0

===============================================

IReporter

If you want a customized report you can implement IReporter listener. It implements a method called generateReport() which is invoked when all the suites of TestNG are executed. It accepts three parameter.

xmlSuite: It is a list of suites for execution extracted from the XML file.
suites: It is list of information about the suite and execution results.
outputDirectory: It is the path of directory where the report will be saved.

This method will be notified when all the suites have been run by TestNG. It means this will not be a realtime but final execution report and you can write your custom report using the information available through XmlSuite and ISuite. Let's see an example where we will print the report of execution both passed and failed test methods.

IReporterClass.java

Java

Copy

package org.example;

import org.testng.IReporter;
import org.testng.ISuite;
import org.testng.ISuiteResult;
import org.testng.ITestNGMethod;
import org.testng.xml.XmlSuite;

import java.util.Arrays;
import java.util.List;
import java.util.Map;

public class IReporterClass implements IReporter {
    @Override
    public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
        for (ISuite suite :suites )
        {
            System.out.println("Suite : " + suite.getName());
            Map<String, ISuiteResult> result = suite.getResults();
            for (Map.Entry<String,ISuiteResult> res: result.entrySet())
            {
                System.out.println("Passed Tests : ");
                Arrays.stream(res.getValue().getTestContext().getPassedTests().getAllMethods().toArray(new ITestNGMethod[0])).forEach(x-> System.out.println(x.getMethodName()));
                System.out.println("Failed Tests : ");
                Arrays.stream(res.getValue().getTestContext().getFailedTests().getAllMethods().toArray(new ITestNGMethod[0])).forEach(x-> System.out.println(x.getMethodName()));
            }
        }
    }
}

IReporterTest.java

Java

Copy

package org.example;

import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(IReporterClass.class)
public class IReporterTest {
    @Test
    public void IReporterTestTestMethod1()
    {
        Assert.assertTrue(true);
    }
    @Test
    public void IReporterTestTestMethod2()
    {
        Assert.assertTrue(false);
    }
    @Test
    public void IReporterTestTestMethod3()
    {
        Assert.assertTrue(true);
    }
}

In the above example we have 3 test methods in the IReporterTest.java file. Tow are getting passed and 1 is getting failed. In IReporterClass.java file we have implemented the generateReport method where we have created two for loops to extract the information from the ISuite parameter. If you run the IReporterTest.java file you will see the below output.


===============================================

Default Suite

Total tests run: 3, Passes: 2, Failures: 1, Skips: 0

===============================================


Suite : Default Suite

Passed Tests : 

IMethodInterceptorTestMethod1

IMethodInterceptorTestMethod3

Failed Tests : 

IMethodInterceptorTestMethod2

ISuiteListener

This will run at a suite level. It has got two methods, onStart invoked before the suite execution starts and onFinish invoked after the suite execution finished. This listener will let you add a listener to a suite using addListener method at run time.
Let's understand this by following example.

ISuiteListenerClass.java

Java

Copy

package org.example;

import org.testng.ISuite;
import org.testng.ISuiteListener;

public class ISuiteListenerClass implements ISuiteListener
{
    @Override
    public void onStart(ISuite suite) {
        System.out.println("Before Suite");
    }
    @Override
    public void onFinish(ISuite suite) {
        System.out.println("After Suite");
    }
}

ISuiteListenerTest.java

Java

Copy

package org.example;

import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(ISuiteListenerClass.class)
public class ISuiteListenerTest {
    @Test
    public void ISuiteListenerTestTestMethod1()
    {
        System.out.println("ISuiteListenerTest Test Method 1");
    }
}

If you run the ISuiteListenerTest.java you will see below result.


Before Suite

ISuiteListenerTest Test Method 1

After Suite


===============================================

Default Suite

Total tests run: 1, Passes: 1, Failures: 0, Skips: 0

===============================================

ITestListener

This is one of the commonly and frequently used listener in TestNG. ITestListener listens to the events and the class implements this listener overrides it's methods and execute it accordingly.
Following are the methods from ITestListener :-

onTestStart : Invoked each time before a test will be invoked.
onTestSuccess : Invoked each time a test succeeds.
onTestFailure : Invoked each time a test fails.
onTestSkipped : Invoked each time a test is skipped.
onTestFailedButWithinSuccessPercentage : Invoked each time a method fails but has been annotated with successPercentage and this failure still keeps it within the success percentage requested.
onTestFailedWithTimeout : Invoked each time a test fails due to a timeout.
onStart : Invoked before running all the test methods belonging to the classes inside the <test> tag and calling all their Configuration methods.
onFinish : Invoked after all the test methods belonging to the classes inside the tag have run and all their Configuration methods have been called.

Unlike other listeners ITestListener captures the events in real time, hence it is used for custom reporting. Let's understand with a simple example.

ITestListenerClass.java

Java

Copy

package org.example;

import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;

import java.util.HashMap;
import java.util.Map;

public class ITestListenerClass implements ITestListener {
    Map<String, String> customReport = new HashMap<>();
    @Override
    public void onTestSuccess(ITestResult result) {
        System.out.println(result.getMethod().getMethodName() + " , " + result.isSuccess());
        customReport.put(result.getMethod().getMethodName(),"Passed");
    }
    @Override
    public void onTestFailure(ITestResult result) {
        System.out.println(result.getMethod().getMethodName() + " , " + result.isSuccess());
        customReport.put(result.getMethod().getMethodName(),"Failed");
    }
    @Override
    public void onTestSkipped(ITestResult result) {
        System.out.println(result.getMethod().getMethodName() + " , " + result.isSuccess());
        customReport.put(result.getMethod().getMethodName(),"Skipped");
    }
    @Override
    public void onFinish(ITestContext context) {
        for(Map.Entry<String, String> entry : customReport.entrySet()) {
            System.out.println(entry.getKey() + " : " + entry.getValue());
        }
    }
}

ITestListenerTest.java

Java

Copy

package org.example;

import org.testng.Assert;
import org.testng.SkipException;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(ITestListenerClass.class)
public class ITestListenerTest {
    @Test
    public void ITestListenerTestTestMethod1()
    {
        Assert.assertTrue(true);
    }
    @Test
    public void ITestListenerTestTestMethod2()
    {
        Assert.assertTrue(false);
    }
    @Test
    public void ITestListenerTestTestMethod3()
    {
        throw new SkipException("Skipped");
    }
}

If you run the ITestListenerTest.java file, you will see following results in the logs.
ITestListenerTestTestMethod2 : Failed ITestListenerTestTestMethod3 : Skipped ITestListenerTestTestMethod1 : Passed =============================================== Default Suite Total tests run: 3, Passes: 1, Failures: 1, Skips: 1 ===============================================