Image Comparison Using Appium

The mass advancement in machine learning and artificial intelligence is affecting every sector of every industry, and test automation is no exception

AI is being used in multiple areas of software testing, including:

1. Visually automated testing: Drive testing through the UI by using Image Comparison
2. Automated API Testing: Using machine learning algorithms we can analyze the API calls in more effective ways
3. Test Coverage: Knowing what to test is a science unto itsel – sometimes a small change has a large impact or vice versa. Using AI tools you can know what areas of the app changed based on source code analysis and which test cases should be executed (or updated)
4. Self-Healing Test Scripts: The most common scenario in automation test cases is test case failure due to locator changes. In order to fix that we need to find the valid locator again which is time-consuming and happens frequently. Machine learning/AI algorithms can learn and observe the changes of an application’s domain object modeling structure and can automatically suggest the new locators to use.
5. Automatically write the test cases: As AI keeps improving, we will see an increased ability for test cases to be automatically created, based either on self exploring or by user observation.

Most of the AI solutions are being developed by 3rd party vendors and not within the Appium framework directly. However, the Appium community recently introduced the Image comparison feature which is great for testing at the UI level and comparing images, which makes our test scripts less brittle. Essentially, we get the benefit of a new Image locator strategy we can use in our test scripts.

The feature was developed by incorporating OpenCV, one of the leading image comparison libraries.

This chapter is split into 2 sections:

1. Setup and Linking OpenCV with Appium
2. Using the Image comparison feature in automation

• 1) Setup and Linking OpenCV with Appium

• In order to link the OpenCV library to Appium, we must install the Appium CLI. This is 3 step process:

  1. Install the Appium CLI.
  2. Install the OpenCV library.
  3. Link the OpenCV library with Appium.

     • 1) Install Appium CLI:

       1. Make sure you have installed node and npm. If you have brew installed on your mac then you can just execute: $ brew install node to install node.js along with npm.
       2. Install appium: $ npm install appium
       3. Verify that appium is installed correctly using the command: $ appium -v
       4. Also check the path where Appium is installed: $ which appium and move to that path.

          

          Figure-1: Appium location.

       5. Now find the actual path of the appium binary and move to it.

          

          Figure-2: Appium binary location.

          

          Figure-3: Appium Node module.

          As you can see in the above screenshot all globally defined node modules are present under /usr/local/lib/node_modules location, so you have to install the OpenCV library at this global location

     • 2) Install OpenCV library.

       1. As we discussed previously we need to install the OpenCV module globally so we will use the -g flag. Use this command to install the OpenCV library for node: $ npm i -g opencv4nodejs After successful installation you can find the opencv4nodejs library under the same global location( /usr/local/lib/node_modules).

          

          Figure-4: opencv4nodejs Node module.

     • 3) Link the OpenCV library with Appium (or install the opencv4nodejs package inside the appium directory).

       1. Now we need to link this module with Appium, using:

          $ npm link opencv4nodejs

       2. Now move to the appium node module and move to the node_modules directory, there you will find that opencv4nodejs is linked.

          

          Figure-5: opencv4nodejs module is linked with appium.

          That’s it, now you are ready to use Image comparison feature in Appium.

          NOTE: You can also directly install the opencv4nodejs node module under Appium node_modules, the steps would be:

          1. Move to appium directory: $ cd /usr/local/lib/node_modules/appium
          2. Install opencv4nodejs: $ npm install opencv4nodejs

• 2) Using the Image comparison feature in automation.

  • Why do we need to image comparison feature in appium?

    There are cases when in the application there is no unique locator present for a particular UI element, and in that case, you can’t do anything except tapping at a particular location which is a very fragile operation (that is, prone to failure). It is workable only if the location of that UI element is static every time – if the UI element is changing its location every time you open your application or based on the device it’s executing on, then tapping on a particular location isn’t the preferred option. With the new image comparison feature inside Appium, we have an image locator strategy. Using this locator we locate elements based on their image.

    With the image locator strategy, instead of typical unique locators, you need to pass the string which is the Base64 encoded format of the image.

    There are many ways to get the Base64 encoded version of the Image, but we use the simplest.

    Use this method to convert an image(.png/.jpg file) to Base64 String format.

    public String getReferenceImageB64(String imgPath) throws URISyntaxException, IOException {
    URL refImgUrl = getClass().getClassLoader().getResource(imgPath);
    File refImgFile = Paths.get(refImgUrl.toURI()).toFile();
    return 
    Base64.getEncoder().encodeToString(Files.readAllBytes(refImgFile.toPath()));
    				}
    Now that we have the image encoded as Base64, we can locate elements by this image.

    Image Locator Strategy:

    					MobileElement elementByImage = (MobileElement) driver.findElementByImage(refImageBase64);
    				

    NOTE: Just as with the other locator strategies we can perform actions like click(), getText(), sendKeys() etc…

    Now to understand this better let’s take one practical automation test case which uses the image locator strategy.

  • Image Comparison Automation Test Case:

    We have taken one simple image application into consideration in order to understand the image comparison feature of Appium. That application will display the image from the given image URL.

    Now the main problem is that the DOM structure contains no details for the image view, so we won’t be able to verify whether image view is visible or not after clicking on submit. Moreover even if we verify that image view is visible it won’t give us confidence that the displayed images are being fetched from given image URL. So in order to check this use case, we need to use Appium’s image comparison functionality.

    

    Figure-6: ImageView is not available for appeared image.

    Here we will download the image from the given URL and will check that the downloaded image is present on the application(after clicking the submit button).

    So it is a 2 step process:

    1. Download the image from the image URL.
    2. Verify that the downloaded image is present on the app after entering the image URL and clicking on the Submit button.

    But before moving to the business logic part we need to integrate the latest version of the Appium java-client(7.0.0) which supports the new image locator strategy.

    build.gradle

    ...
    dependencies {
    ...
    testCompile group: 'io.appium', name: 'java-client', version: '7.0.0'
    }
    ...
    

    After defining the dependency we can proceed further and continue on famous page object model in order to automate the scenario.

    • 1) Download the image from the image URL and convert it to Base64-encoded version:

      • In the image locator strategy we are dealing with the base64-encoded format of the image so downloading the image is not sufficient, we need to convert it to base64-encoded format. Below method will convert any image from it’s URL to base64-encoded format.

      • public String getBase64FormatOfImageFromURL(String imageURL) throws IOException, URISyntaxException { URL url = new URL(imageURL); try { InputStream is = url.openStream(); byte[] bytes = org.apache.commons.io.IOUtils.toByteArray(is); return org.apache.commons.codec.binary.Base64.encodeBase64String(bytes); } catch (Exception e) { throw new RuntimeException(“Please check the network on your server! It seems disconnected.”); } }

    • 2) Verify downloaded image is present on the app (after entering the image URL and clicking on the Submit button).

      • After getting the base64 format of the downloaded image, you need to check that base64-encoded string of that image is present on the application (after entering the image URL and clicking on the submit button on the app).

        Please find the below code for reference:

        									public void isImageAppearOnApplication(String base64FormatImage) throws IOException, URISyntaxException {									    waitUtils.staticWait(5000);									    try {									        Assert.assertTrue(AppiumUtils.isElementDisplayed((AndroidElement) driver.findElementByImage(base64FormatImage)), "Expected Image did not appear on dashboard Screen.");								    } catch (NoSuchElementException e) {								        throw new RuntimeException("Expected image didn't display on Application!");	 									 }										}								

        In the above code, notice how we have put a static wait of 5 seconds – this is because as soon as the SUBMIT button is tapped, it will take some time to fetch and display the image on the app from the image URL.

        NOTE: Here we could have put the dynamic wait instead of static if Image view was present in DOM for the displayed image.

        This feature opens the door to new possibilities because there were cases when the traditional locator strategy could not help us. But now, by using the Image locator strategy you can literally find any UI element and apply an action upon it.

        You can find this complete example on our GitHub page: Github page