Understanding Automation Testing Frameworks

What is an Automation Testing Framework

01

What is a Framework

A framework is a reusable design of the whole or part of a system, manifested as a set of abstract components and the methods of interaction between component instances. It specifies the architecture of the application, clarifies the dependencies between the entire design and collaborating components, responsibility allocation, and control processes, represented as a set of abstract classes and the methods of collaboration between their instances. Frameworks are generally mature and robust, capable of handling many details of the system, such as transaction handling, security, data flow control, etc. Additionally, frameworks are generally used by many people, so their structure is solid, leading to good scalability, and they are continuously upgraded, allowing direct benefits from others’ upgraded code.

02

What is an Automation Testing Framework

An automation testing framework is the infrastructure built to provide an execution environment for automated test cases or scripts. Automation testing frameworks help effectively develop, execute, and report automated test cases.

1. Advantages:

• Code Reusability

• Improved Testing Efficiency

• Higher Test Coverage

• Low Maintenance Costs

• Earlier Detection and Recording of Bugs

2. Basic Components of a Framework

• Configuration File Management Generally, a configuration file is needed to control some environment information and switches. Configuration files can be txt/xml/yaml/properties/ini, with .properties commonly used in JAVA, while Python usually opts for ini files.

• Separation of Business Logic Code and Test Scripts If the code and scripts are in one class file, then code refactoring and reusability are not utilized. After separating the code and case files, it becomes clearer, allowing more people to develop scripts and facilitating debugging.

• Report and Log File Output How many cases were executed, what were the case results, all need reports to display; generally, third-party plugins are used to achieve this functionality. Many report formats are in html, simple and clear style. Log output is also very important; if an error occurs and the script execution fails, logs help quickly locate where the problem occurred.

• Encapsulation of Custom Libraries Many functions need to be called repeatedly; they can be written as a common method and placed in a toolkit for easy access, such as the encapsulation of browser engine classes and basepage.py.

• Management and Execution of Scripts For example, the unittest framework in Python has a very high usage rate.

• Third-party Plugin Integration Sometimes, some functions need to leverage third-party plugins to better achieve, such as AutoIT (for file uploads and downloads). Also, utilizing third-party reporting plugins to generate HTML format test reports.

• Continuous Integration git, svn, ant, maven, jenkins, we will integrate this into jenkins to achieve continuous integration and execute test scripts with one click.

Conclusion:

A simple automation testing framework may consist of the following components:

03

Why Do We Need an Automation Testing Framework

• Make the implementation of automation testing easier: A good automation testing framework allows even those who are not very technical to write automation testing scripts.

• Address issues inherent to automation testing scripts: such as exception handling and scenario recovery.

• Testing is easy to maintain.

• Reusability: Can implement some common functions to simplify the script development process.

• Test Reports are aesthetically pleasing and easy to read.

04

Determining Whether Automation Testing is Needed

Automation testing has high cost investment and risks, and not all projects are suitable for automation. The advantage of automation testing lies in repeated iterations, and its value output is in long-term regression testing to ensure the tested product’s stable version updates over time.

Applicable Projects:

• Project cycles are long and demands do not change frequently.

• Most objects in the system need to be identifiable, and there are no large numbers of third-party plugins.

02

Types of Automation Testing Frameworks

01

Classification of Automation Testing Frameworks

Below are the currently popular types of automation testing frameworks. These frameworks may differ from each other based on support for different key factors (such as driving type, reusability, ease of maintenance, etc.).

1. Module-based Testing Framework:

This framework divides the entire “application under test” into many logical and independent modules. For each module, an independent test script is created; when these scripts are combined, they build a larger test script, representing multiple modules that are isolated by abstraction layers, so changes made in various parts of the application do not affect that module.

• Disadvantages: While implementing test scripts for each module, test data is already embedded into the test scripts, which requires modifications to the test scripts when testing with different test data.

2. Library Architecture Testing Framework

The library architecture testing framework is built on top of the module-based testing framework, but it has some additional advantages over the latter. It does not divide the tested application into test scripts but into functions. Therefore, a public library consisting of common functions is created for the application under test, which can be called from the test scripts as needed. Its basic principle is to identify common steps and group these steps into functions under the public library, which can be called in the test scripts when needed.

For example: A login step can be combined into a function and saved to a library, so when logging in, this function can be called directly from the library without rewriting the code.

• Disadvantages: Like the module-based framework, test data is embedded in the test scripts, requiring script modifications to change data; as more libraries are introduced, the framework may become increasingly complex.

3. Data-Driven Testing Framework

The data-driven testing framework separates the logic of the test scripts from the test data. Test data can be stored separately, and the storage formats can include xml files, excel files, text files, csv files, ODBC databases, etc. Generally, data is stored in key-value format for easy access.

• Disadvantages:

• ① The process is relatively complex; it may require customization of the test data source format and reading mechanism.

• ② Only similar test cases with the same execution steps can be combined into multiple data groups. Different execution step sets require different test cases.

4. Keyword-Driven Testing Framework

The keyword-driven testing framework is an extension of the data-driven testing framework; in a sense, it not only separates the test data from the scripts but also stores the specific code sets of the data test scripts in external data files. These code sets are called keywords, and each keyword represents an operation. Both keywords and test data are independent of the framework.

• Disadvantages:

• ① Requires understanding of the keyword creation mechanism to develop keywords.

• ② As more keywords are introduced, the framework may become increasingly complex.

• ③ Test cases become longer and more complex, affecting the maintainability of the test cases.

5. Hybrid Testing Framework

The hybrid testing framework is a combination of the aforementioned (modular, data-driven, and keyword-driven) types of frameworks. It utilizes the advantages of various types of frameworks to create a hybrid testing framework.

In this framework, test cases are integrated into the modular testing framework, developing test cases from modular scripts. Each test case uses a driver script, which utilizes data files from the data-driven framework and operation files from the keyword-driven framework.

• Disadvantages: It may be more complex than other types of testing frameworks, such as in terms of reading and maintenance.

6. Behavior-Driven Development Framework The behavior-driven testing framework allows developers, testers, and others to implement functional verification automation in an easily readable and understandable format. Testing functionalities, test scenarios, test steps, test results, etc., can be described using natural language.

02

Popular Automation Testing Frameworks

1. Robot Framework If you want to use a Python automation testing framework for test automation, Robot Framework is the best choice. Robot Framework is based on Python, but scripts can also be written using Jython (Java) or IronPython (.NET).

• Robot Framework uses a keyword-driven approach, making test case creation easy.

• Robot Framework can also test MongoDB, FTP, Android, Appium, etc.

• It has many testing libraries, including the Selenium WebDriver library and other commonly used tools.

• Robot Framework has many APIs to help make it as extensible as possible.

2. WebdriverIO WebdriverIO is a Node.js-based automation testing framework. It has an integrated test runner, capable of running automated test cases for web applications as well as native mobile apps. Furthermore, WebdriverIO can run on both the WebDriver protocol and Chrome Devtools protocol, making it very effective for cross-browser testing based on Selenium Webdriver or Chromium-based automation. As WebDriverIO is open-source, testing engineers can access a range of plugins that meet automation needs.

3. Citrus Citrus is an open-source testing framework that allows testers to automate integration tests for any messaging protocol or data format. Citrus testing framework is suitable for testing messaging integrations for any messaging transport like REST, HTTP, SOAP, or JMS. If interaction with the user interface is needed, followed by backend verification, Citrus can be integrated with Selenium.

4. Cypress Cypress is a developer-centric test automation framework that makes TDD (Test Driven Development) a reality for developers. Its design principle is to package and bundle everything together for complete end-to-end testing. Cypress’s architecture differs from Selenium; Selenium WebDriver runs remotely outside the browser, while Cypress runs inside it. This approach helps understand everything happening inside and outside the browser, providing more consistent test results.

5. Selenium One of the most popular open-source test automation frameworks for web applications. Selenium has cross-platform and cross-browser capabilities, and it can also serve as the foundation for many other testing tools. Selenium supports multiple programming languages such as Java, C#, PHP, Python, Ruby, etc. It has one of the largest online communities, making it easier to maintain.

Selenium can be highly extensible through a wide range of libraries and APIs to meet each team’s needs. Selenium is the preferred testing framework for testers because it allows for writing more advanced test scripts to meet various complexities. It provides playback tools for test creation without needing to learn a specific scripting language.

6. Carina Carina is built using popular open-source solutions (like Appium, TestNG, and Selenium), reducing dependency on specific technology stacks. Testers can test apps (native apps, hybrid apps, web apps), web applications, REST services, and databases. The Carina framework supports different types of databases, such as MySQL, SQL Server, Oracle, PostgreSQL, providing a perfect experience for implementing the DAO layer using the MyBatis ORM framework.

Carina supports all popular browsers and mobile devices, and reuses up to 80% of the test automation code between iOS/Android. API testing is based on the Freemarker template engine, providing great flexibility in generating REST requests. Carina is cross-platform and can easily run test cases on Unix or Windows OS.

03

Selenium

Selenium is a tool for testing web applications, comprising a complete web application testing system that includes recording tests (Selenium IDE), writing and running (Selenium Remote Control), and parallel processing of tests (Selenium Grid). Selenium’s core, Selenium Core, is based on JsUnit and is entirely written in JavaScript, allowing it to be used on any browser that supports JavaScript. Selenium can simulate real browsers, automate testing tools, and supports various browsers, primarily solving JavaScript rendering issues in web crawlers. The main functions of this tool include testing browser compatibility and checking if your application works well across different browsers and operating systems.

Advantages:

Open Source: No cost and convenient for secondary development, such as extracting processing for JSON and XML for data-driven implementation.

Flexibility: With the addition of XPath (though in large projects, XPath should be used cautiously; stable attributes or IDs should be preferred), combined with IDE for locating elements, the results are quite good.

More comprehensive resources, with an increasing number of companies using this framework.

Disadvantages:

Both RC and WebDriver have certain requirements regarding the coding level of testers.

Ant, TestNG, Hudson usage is also niche; most people need a long time to learn and adapt before executing this framework.

Time is relatively short, and it is not as complete as QTP.

01

Selenium 1.0

Selenium 1.0 includes two parts:

• Selenium Server

• Client Libraries

Supported browsers include IE (7, 8, 9, 10, 11), Mozilla Firefox, Safari, Google Chrome, Opera, Edge, etc.

1. Selenium Server The Selenium Server is responsible for controlling the behavior of the browser. It mainly includes launcher, HTTP Proxy, and Selenium Core. Selenium Core is embedded into the browser page using the Selenium Server. Essentially, Selenium Core consists of JS functions, allowing us to perform operations on the browser programmatically.

2. Client Libraries Libraries that control the Selenium Server when writing test cases.

3. Execution Process of Test Cases

• Test cases (Testcase) send HTTP requests to the Selenium Server (an independent intermediary server, specifically a proxy server) through the Client Lib interface, requesting to establish a connection with the Selenium Server.

• The Selenium Server’s Launcher starts the browser, loads the Selenium Core into the browser page, and sets the browser’s proxy to the Selenium Server’s HTTP Proxy.

• Test cases send HTTP requests to the Selenium Server through the Client Lib interface; the Selenium Server parses the request and sends JS commands via the HTTP Proxy to notify the Selenium Core to perform actions on the browser.

• The Selenium Core executes the operations upon receiving instructions.

• The browser receives new page request information (because in 4, the operations of Selenium Core may trigger new page requests) and sends an HTTP request for a new page. Since the Selenium Server has manipulated the browser at startup, it will receive all requests sent from the browser it initiated.

• Upon receiving the HTTP requests sent by the browser, the Selenium Server reconstructs the HTTP request and retrieves the corresponding web page.

• The HTTP Proxy of the Selenium Server returns the received web page to the browser.

02

Selenium 2.0 (WebDriver)

Selenium 2.0 encapsulates the native API of the browser into the WebDriver API, allowing direct manipulation of elements on the browser page, and even the browser itself (such as taking screenshots, resizing windows, starting, closing, installing plugins, configuring certificates, etc.), mimicking real user operations.

1. Architecture of Selenium 2:

WebDriver is designed according to the classic server-client design pattern:

• The server side is the remote server, which can be any browser: our script starts the browser, and that browser becomes the remote server, whose responsibility is to wait for client requests and respond accordingly.

• The client side is simply our test code: Some actions in the test code, like opening the browser and navigating to a specific URL, are sent to the server (the browser under test) as HTTP requests, which accepts the requests, executes the corresponding operations, and returns execution status and return values in response.

2. WebDriver Wire The WebDriver Wire Protocol is a protocol designed and defined by Selenium, which is very powerful and can perform almost any operation on the browser, including opening, closing, maximizing, minimizing, locating elements, clicking elements, uploading files, etc. The WebDriver Wire protocol is universal; regardless of whether it is FirefoxDriver or ChromeDriver, it starts a Web Service based on this protocol on a specific port after initialization. For example, upon successful initialization of FirefoxDriver, it defaults to http://localhost:7055, while IE defaults to http://localhost:52432.

3. Working Principle of WebDriver:

• After starting the browser, selenium-webdriver binds the target browser to a specific port; the started browser acts as the WebDriver’s remote server.

• The client (the test script) sends HTTP requests to the server side (communication protocol: The WebDriver Wire Protocol; in the body of the HTTP request, a string in the JSON format specified by the WebDriver Wire protocol will indicate to Selenium what actions we want the browser to perform next).

• The server side relies on the native browser components to convert the Web Service commands into native calls to the browser to complete the operations.

04

Appium

Appium is an open-source, cross-platform automation testing tool that supports iOS, Android, and FirefoxOS platforms. With Appium, developers can test mobile applications without recompiling the app or making any adjustments, allowing test code to access backend APIs and databases. It achieves dual-platform support by driving Apple’s UIAutomation and Android’s UiAutomator frameworks, while also binding Selenium WebDriver for testing older Android platforms. Developers can write test scripts in any language compatible with WebDriver, such as Java, OC, JS, PHP, Python, Ruby, C#, Clojure, and Perl.

01

UIAutomation

UIAutomation is Apple’s UI automation testing framework, written in JavaScript. Based on UIAutomation, there are extension-type tool frameworks and driver-type frameworks:

• Extension-type frameworks provide many useful JS tools via JavaScript extension library methods.

• Injection-type frameworks usually provide some libs or frameworks, requiring testers to import these contents into the code project of the application under test, enabling the framework to drive the app through them.

• Driver-type UI Automation uses the UI Automation library at the automation testing base level, driving UI Automation to complete automated testing via TCP communication; thus, the scripting language is no longer limited to JavaScript.

Automation is the official UI automation testing solution provided by Apple, but the interfaces are not rich enough.

02

UiAutomator

UIAutomator is a testing framework provided by Google; it offers high-level UI testing for native Android apps and games. This is a Java library containing APIs for creating functional UI tests, along with an execution engine for running tests. The library comes with the Android SDK.

• Advantages: It grants JUnit test cases privileges when running across different processes. The library is supported and maintained by the Google community.

• Disadvantages: Only supports Android 4.1 (API level 16) and above. Does not support script recording. The focus is on Java.

If cross-app operations are involved in automation testing, UiAutomator can be integrated to achieve this.