Parallel Test Execution on Simulators and Emulators

Parallel test execution on simulators and emulators are ideal for testing early in the dev cycle for basic functional testing. Depending on your use case, this can be a useful asset to understand.

One thing to keep in mind is that in today’s fast moving world, teams who are testing for usability and performance will find that real device testing is going to be the game-changer. It aids in the limitations emulators and simulators have, exposing unknown bugs earlier and applying real user conditions for more realistic testing.

In this article, we are going to discuss how you can leverage parallel test execution in TestING on simulators and emulators. We will be using Java + TestNG, a great combination in the world of Automation Testing allowing us to run tests on parallel.

Before going to deep into parallelization details, we will go through basics of TestNG.

• TestNG: TestNG is a open source testing framework written in Java, featuring:
  • Annotations support, which reduces complexity at the class level.
  • It is written in Java and has clean Object Oriented features.
  • All tests can be run from one place via the testng.xml file that specifies the tests to be executed.
  • It has a grouping feature so you can group the test cases and can run them group wise.
  • Supports multi threading and parallel testing in TestING at the 1) method(test), 2) class and 3) test suite level.
  • Supports multi threading and parallel testing in TestING at the 1) method(test), 2) class and 3) test suite level.

Let’s take a refresher on how you can execute test cases from a Test Class file in IntelliJ Idea. You will recall that you just need to select the Test Method Name > Right click on it > Run.

Now before jumping into parallel testing, let’s revisit the following simple code for iOS we used in an earlier article. This is the code we will be using as an example for parallel execution:

import io.appium.java_client.ios.IOSDriver;
import io.appium.java_client.ios.IOSElement;
import org.openqa.selenium.By;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.testng.Assert;
import org.testng.annotations.BeforeTest;
import org.testng.annotations.Test;
import java.net.MalformedURLException;
import java.net.URL;
public class IOSTestCases {
    public IOSDriver driver;
    @BeforeTest
    public void setUp() throws MalformedURLException {
        String appiumServerURL = "https://127.0.0.1:4723/wd/hub";
        DesiredCapabilities dc = new DesiredCapabilities();
        dc.setCapability("platformName", "iOS");
        dc.setCapability("platformVersion", "11.4");
        dc.setCapability("app", "/Users/pratik/Downloads/FirstAutomationTest/src/test/resources/DemoApp-iPhoneSimulator.app");
        dc.setCapability("deviceName", "iPhone X");
        driver = new IOSDriver(new URL(appiumServerURL), dc);
    }
    @Test
    public void sampleTestCase() throws InterruptedException {
        int a = 5;
        int b = 10;
        driver.findElement(By.id("IntegerA")).sendKeys(a + "");
        driver.findElement(By.id("IntegerB")).sendKeys(b + "");
        driver.findElement(By.id("ComputeSumButton")).click();
        String answer = driver.findElement(By.id("Answer")).getText();
        Assert.assertEquals(answer, a + b + "", "Expected and Actual Result didn't match!");
    }
}

After putting the above test script into a IntelliJ java project, we can execute the test cases. But now we will use the testng.xml method for execution.

Creation of testng.xml:

Unfortunately, by default testng.xml is not available to the project so we need to create it. But you can do it in 2 ways:

1. Manually create testng.xml:
   1. Specifying the Package name, to execute tests from whole package.

• look liike
  2. Specifying the Test Class name, to execute tests for particular Test Class.
  3. Specifying the Test Method name, to execute a particular test case.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
			<suite name="All Test Suite">
			    <test verbose="2" preserve-order="true" name="/Volumes/Disk2/AppiumBook/Chapter9-Test Execution on Parallel simulators and emulators">
			     <classes>
			         <class name="testcases.IOSTestCases">
			             <methods>
			                 <include name="sampleTestCase"/>
			              </methods>
			          </class>
			      </classes>
			    </test>
			</suite>

4. Specifying the specific groups to be included or excluded. Here you need to set the group to Test case first like:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="All Test Suite">
    <test verbose="2" preserve-order="true" name="/Volumes/Disk2/AppiumBook/Chapter9-Test Execution on Parallel simulators and emulators">
        <groups>
            <run>
                <include name="sample"/>
            </run>
        </groups>
        <classes>
            <class name="testcases.IOSTestCases"/>
        </classes>
    </test>
</suite>

2. Automatically create testng.xml:

• That may seem a little confusing right now since you don’t yet know the details of what all of that does. Hang in there, it’ll start making sense in a bit.
• Creating it via IntelliJ involves the following steps:
  • Move to IntelliJ Idea project and open Preferences.

• Now move to plugin section and click on Browse repositories… which will takes you to the Plugins dialog.

• Search for “Create testng” type string and you will find the plugin named “Create TestNG XML”

• After installing the plugin, you need to restart IntelliJ IDEA.

• Now after restarting the IntelliJ IDEA, you are able to generate a TestNG xml file.

• In a fraction of a second testng.xml will be created and you will get the below confirmation modal dialog.

• By default testng.xml will be created under the project root directory, so it won’t be identified by the java compiler at run/compile time. So you need to move it to test/resources.

• So as you can see in above screenshot, there are some pre-defined XML tags present. Here by default class tag has name=”testcases.IOSTestCases” attribute, which means TestNG will execute all test cases under this test class only. And as we have only one test case (sampleTestCase), it is defined under testcases.IOSTestCases it will run/execute only one test case. You can also change the testng.xml and make it run at the package, class, method and group level.

Now How can you run the testng.xml?

Well, the answer is just a 2 step process:

1. Right click on testng.xml
2. Select Run

And that’s it, your tests under the IOSTestCases will start the execution sequentially.

1) Parallel execution of Tests on iOS Simulator:

Our Goal is to execute a single test case(sampleTestCase) written in IOSTestCase.java on 3 iOS Simulators (iPhone 7, iPhone 8 and iPhone X) in parallel.

In order to achieve that we need to understand 2 important concepts:

1. We need to manage Appium Server from Code:
   • Up until now we were using the Appium Desktop Application to start the Appium server. Normally, one Appium server is bound to one Appium Session (or you can say to one device/simulator), but now as we need to run the test case on 3 iOS simulators at the same time, we need 3 Appium servers running on three different ports. So now you can’t rely on the Appium Desktop Application as it will be able to run only one Appium Server. So the best option left to us is to create and run 3 Appium servers at runtime, and using Java you can easily create the Runtime Appium Server by mentioning the particular port. In our case, we will need 3 different ports to start 3 different Appium Servers(or Sessions). Below is the code which will start and stop the Appium server for port 4725. // Create AppiumDriverLocalService object with specifying the port. AppiumDriverLocalService service = new AppiumServiceBuilder().usingPort(4725).build(); // To start Appium server. service.start(); // To End Appium server. service.start();
2. We need to use parameters in Test Class and testng.xml
   • We need to start 3 Appium Servers on 3 different ports, and each device should be assigned to a single Appium Server with a unique WDA port. What is a WDA Port? It’s nothing but used to forward traffic from the Mac host to the iOS Simulator to real ios devices over USB This table may make it more clear: No. Appium Server Port Device Name (Simulator) WDA Local Port 1 4725 iPhone 7 8100 2 4726 iPhone 8 8200 3 4727 iPhone X 8300 Now we need to pass these values from testng.xml to our Test Class before all initialization of the WebDriver takes place. And we will use the TestNG @Parameters annotation for that. IOSTestCases.java @Parameters({“wda”, “deviceName”, “port”}) @BeforeTest public void setUp(long wda, String deviceName, String port){ AppiumDriverLocalService service = new AppiumServiceBuilder().usingPort(Integer.valueOf(port)).build(); service.start(); DesiredCapabilities dc = new DesiredCapabilities(); dc.setCapability(IOSMobileCapabilityType.WDA_LOCAL_PORT, wda); dc.setCapability(MobileCapabilityType.DEVICE_NAME, deviceName); …. …. } testng.xml <?xml version=”1.0″ encoding=”UTF-8″?> <!DOCTYPE suite SYSTEM “https://testng.org/testng-1.0.dtd”> <suite name=”All Test Suite”> <test name=”test1″> <parameter name=”wda” value=”8100″/> <parameter name=”deviceName” value=”iPhone 7″/> <parameter name=”port” value=”4725″/> <classes> <class name=”testcases.IOSTestCases”/> </classes> </test> </suite> In the above example we are passing 3 values from testng.xml to the IOSTestClass.java:
     1. wda: Which will be passed to Desired capabilities IOSMobileCapabilityType.WDA_LOCAL_PORT
     2. deviceName: Which also would be passed to Desired capabilities of MobileCapabilityType.DEVICE_NAME
     3. port: It will be used to create the Appium server.

Now let’s come to the parallelization part. If you want to run test cases in parallel then you need to use attributes along with the tag.

• parallel: It has a number of possible values such as tests, classes, methodandinstances.
• If you want to run parallelization at then you can use parallel=”tests”
• thread-count: No of threads to execute in parallel. If you want to execute 5 test cases in parallel then thread-count=”5″
• <suite name=”All Test Suite” parallel=”tests” thread-count=”5″>

To reach our goal, we need to execute test cases from IOSTestCases.java on 3 iOS simulators in parallel. Below is the testng.xml file we can use:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="All Test Suite" parallel="tests" thread-count="3">
    <test name="test1">
        <parameter name="wda" value="8100"/>
        <parameter name="deviceName" value="iPhone 7"/>
        <parameter name="port" value="4725"/>
        <classes>
            <class name="testcases.IOSTestCases"/>
        </classes>
    </test>
    <test name="test2">
        <parameter name="wda" value="8200"/>
        <parameter name="deviceName" value="iPhone 8"/>
        <parameter name="port" value="4726"/>
        <classes>
            <class name="testcases.IOSTestCases"/>
        </classes>
    </test>
    <test name="test3">
        <parameter name="wda" value="8300"/>
        <parameter name="deviceName" value="iPhone X"/>
        <parameter name="port" value="4727"/>
        <classes>
            <class name="testcases.IOSTestCases"/>
        </classes>
    </test>
</suite>

After adding this to testng.xml, you just need to Right click on it and select Run option, and you will see the 3 iOS Simulators will open and each of them will execute test cases in parallel. Great!

We have discussed only one possible way to achieve parallelization. There are many other ways out there and you can also create your own.

2) Parallel execution of Tests on real iOS devices:

In the previous section, we looked at test execution on iOS Simulators, but what if you want to execute tests on Real Devices? In the next article, we will look at using a cloud testing service like Kobiton, but for now, let’s look at how you may need to run parallel tests using the real-devices on-hand.

The answer is pretty simple – we just need to pass UDID as a 4th parameter.

Let’s understand this by way of an example. Let’s say we have 2 real iOS Devices connected to our Mac host and we want to run the test cases on that in parallel. Please look at the table below for Device capability and Port information.

No.	Appium Server Port	Real Device Name	UDID	WDA Local Port
1	4725	John’s iPhone	2b6f0cc904d137be2e1730235f5664094b831186	8100
2	4726	iPhone	d137be2e12b6f0cc90473031186235f5664094b8	8200
3	4727	iPhone X	137b30235f5664094b831186e22b6f0cc904de17	8300

IOSTestCases.java and testng.xml will look like below.

IOSTestCases.java

		@Parameters({"wda", "udid", "deviceName", "port"})
		@BeforeTest
		public void setUp(long wda, String udid, String deviceName, String port){
		AppiumDriverLocalService service = new AppiumServiceBuilder().usingPort(Integer.valueOf(port)).build();
		service.start();
		DesiredCapabilities dc = new DesiredCapabilities();
		dc.setCapability(IOSMobileCapabilityType.WDA_LOCAL_PORT, wda);
		dc.setCapability(MobileCapabilityType.UDID, udid);
		dc.setCapability(MobileCapabilityType.DEVICE_NAME, deviceName);
		….
		….
		}

testng.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="All Test Suite" parallel="tests" thread-count="2">		    
    <test name="test1">
        <parameter name="wda" value="8100"/>
        <parameter name="udid" value="2b6f0cc904d137be2e1730235f5664094b831186"/>
        <parameter name="deviceName" value="John’s iPhone"/>
        <parameter name="port" value="4725"/>
        <classes>
            <class name="testcases.IOSTestCases"/>
        </classes>
    </test>
    <test name="test2">
        <parameter name="wda" value="8200"/>
        <parameter name="udid" value="d137be2e12b6f0cc90473031186235f5664094b8"/>
        <parameter name="deviceName" value="iPhone"/>
        <parameter name="port" value="4726"/>
        <classes>
            <class name="testcases.IOSTestCases"/>
        </classes>
    </test>		 
    <test name="test3">
        <parameter name="wda" value="8300"/>
        <parameter name="udid" value="137b30235f5664094b831186e22b6f0cc904de17"/>
        <parameter name="deviceName" value="iPhone X"/>
        <parameter name="port" value="4727"/>
        <classes>
            <class name="testcases.IOSTestCases"/>
        </classes>
    </test>
</suite>

Everything else remains consistent.

3) Parallel execution of Tests on real Android Devices:

You just need to change the device name in testng.xml. Using $ adb devices you can get the connected real device names.

testng.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Android Test Suite" parallel="tests" thread-count="2">
    <test name="test1">
        <parameter name="platformVersion" value="7.0"/>
        <parameter name="deviceName" value="B6AUTSZDYPSOZD7S"/>
        <parameter name="port" value="4739"/>
        <classes>
            <class name="testcases.AndroidTestCases"/>
        </classes>
    </test>
    <test name="test2">
        <parameter name="platformVersion" value="8.0"/>
        <parameter name="deviceName" value="c4e3f3cd"/>
        <parameter name="port" value="4740"/>
        <classes>
            <class name="testcases.AndroidTestCases"/>
        </classes>
    </test>
</suite>

When you execute the above testng.xml you can get output result similar to the below image.

You can also find this example on our Github page as well: https://github.com/appiumbook/appiumbook/tree/master/Chapter9-Test%20Execution%20on%20Parallel%20simulators%20and%20emulators

So, that was all about parallel execution on Appium using TestNG. As we discussed, there are many other ways to achieve parallelization using TestNG, so using your custom logic, you can even execute tests for both iOS and Android.