Source: Python DevelopersThis article is 3700 words long, and it is recommended to read in 10 minutes. This article presents five types of Python automation testing frameworks for comparison and discussion.
Since being ranked as a programming language in 2018, Python has consistently been at the top of various rankings. Currently, it ranks third in the Tiobe index, only behind Java and C. With the widespread use of this programming language, Python-based automation testing frameworks have emerged and continue to evolve and enrich.Therefore, developers and testers need to consider many factors when choosing a testing framework for their projects, including the quality of the framework’s scripts, the simplicity of test cases, and the potential technical weaknesses of the running modules. To avoid “choice paralysis,” I have prepared five types of Python automation testing frameworks for comparison and discussion.
1. Robot Framework
As one of the most important Python testing frameworks, Robot Framework is mainly used in test-driven development and acceptance testing. Although developed in Python, it can also run on .Net’s IronPython and Java’s Jython. Additionally, as a Python framework, Robot is compatible with platforms such as Windows, MacOS, and Linux.
Prerequisites
Before using Robot Framework (RF), you need to install Python version 2.7.14 or higher. I recommend using Python 3.6.4 to ensure that proper annotations can be added to code snippets and to track changes in the program. You also need to install the Python package manager, pip.Of course, you must also download the corresponding development framework, such as the popular PyCharm Community Edition among developers. Moreover, since the code snippets themselves do not depend on any IDE, you can completely continue using your existing IDE tools.
Advantages
- By using a keyword-driven testing approach, it helps testers easily create readable test cases, simplifying the entire automation process.
- Users can easily test the syntax in the data.
- Because it consists of various common tools and testing libraries, it has a large ecosystem and can use its various elements in standalone projects.
- With many types of APIs, the framework is highly extensible.
-
Although not a built-in feature, Robot can help users run various parallel tests through Selenium Grid (see https://www.lambdatest.com/blog/lambdatest-now-live-with-an-online-selenium-grid-for-automated-cross-browser-testing/).
Disadvantages
Despite the above conveniences, Robot Framework can be cumbersome in creating custom HTML reports. At most, you can use it to generate brief reports in xUnit format.Additionally, Robot Framework is not particularly strong in parallel testing.
Robot’s Features Compared to Competitors
Due to Robot’s rich built-in libraries and the use of a simpler test-oriented DSL (Domain Specific Language), it is easier for newcomers to the field of automation testing, especially those lacking development experience, to use it as a Python testing framework compared to the Pytest or Pyunit mentioned below. Of course, if you need to develop complex automation frameworks, please use Pytest or other types of Python frameworks.
2. Pytest
Suitable for various software testing, Pytest is another Python automation testing framework. With its open-source and easy-to-learn characteristics, this tool is frequently used by QA (Quality Analysis) teams, development teams, individual teams, and various open-source projects. Given that Pytest has practical features like “assert rewriting,” many large internet applications, such as Dropbox and Mozilla, have transitioned from the unittest (Pyunit) mentioned below to Pytest.
Prerequisites
Besides basic Python knowledge, users do not need more technical reserves. Additionally, users only need a testing device with a command line interface and have installed the Python package manager and an IDE tool for development.
Advantages
- In the past, developers had to include their tests in various large classes. Now, Pytest allows users to write more compact test suites.
- Other testing tools require developers or testers to use debuggers or log checks to detect the source of a specific value. However, when users write test cases with Pytest, they can store all values within the test cases until they learn which values failed and which passed assertions.
- Due to the minimal boilerplate code involved, users can easily write and understand various tests.
- Fixture functions are often used to add parameters to test functions and return different values. In Pytest, you can modularize another using a fixture. Additionally, users can use multiple fixtures to cover all combinations of parameters without rewriting test cases.
- Pytest developers maintain the framework’s extensibility by releasing various practical plugins. For example, pytest-xdist can be used to execute parallel tests without using other testers. At the same time, unit tests can achieve parameterization without copying any code.
-
By providing various special routines for developers, it makes writing test cases simpler, less error-prone, while also shortening and making the code easier to understand.
Disadvantages
The aforementioned special routines also mean that users must give up some compatibility. While they facilitate writing test cases, these cases cannot be used with any other testing frameworks.
Pytest’s Features Compared to Competitors
In terms of writing functional test cases and developing complex frameworks, Pytest surpasses the UnitTest mentioned below. However, like Robot Framework, it is only suitable for developing simple testing frameworks.If you are considering using Pytest, please refer to “Automating Testing with Pytest and Selenium WebDriver” for further information.
3. UnitTest/PyUnit
Inspired by JUnit, UnitTest/PyUnit is a standardized Python class automation testing framework for unit testing. Its base class TestCase provides various assertion methods and all cleanup and setup routines. Therefore, each method in the TestCase subclass is prefixed with “test” to indicate that they can be run as test cases. Users can use the load method and TestSuite class to group and load various tests.Of course, you can also build custom test runners by using them together. Just as we use Junit to test Selenium, UnitTest will also use UnitTest-sml-reporting and can generate various XML-type reports.
Prerequisites
Since UnitTest uses Python by default, we do not need any prerequisites. In addition to having basic knowledge of the Python framework, you can additionally install pip and an IDE tool for development.
Advantages
- Developers do not need to install any other modules.
- UnitTest is a derivative product of xUnit, and its operation is very similar to other xUnit frameworks. Therefore, those without a strong Python background can quickly get started.
- Users can run individual test cases in a simpler way. You only need to set the name on the terminal, and the framework can flexibly execute various test cases and produce concise output.
- It can generate various test reports in just a few milliseconds.
Disadvantages
Although this framework commonly uses snake_case to name various Python codes, it still retains some traditional camelCase naming methods due to its origin from Junit. This often leads to confusion.Because it overly supports abstract methods, the purpose of the test code can sometimes be unclear, requiring a lot of boilerplate code.
PyUnit’s Features Compared to Competitors
Similar to the views of some Python developers, I believe that Pytest can cultivate testers to write better automation code in a very compact manner, which is a good programming habit. Although UnitTest is the default automation testing framework in Python, its working principles and naming conventions are still slightly different from standard Python code. Simultaneously, it requires too much boilerplate code. Therefore, this framework is not very popular.
4. Behave
We all know that Behavior-Driven Development (BDD) is an agile software development method that encourages collaboration among developers, business participants, and QA personnel. As another Python testing framework, Behave allows teams to execute BDD tests while avoiding various complexities. Essentially, this framework is very similar to SpecFlow and Cucumber and is often used for automated testing. Users can write test cases in simple, readable language and can paste them into the code during execution. Moreover, the established behavioral specifications and steps can also be reused in other testing scenarios.
Prerequisites
Anyone with basic Python knowledge can use Behave. Other prerequisites include:
-
You must install Python version 2.7.14 or higher.
-
You need to cooperate with Behave through the Python package manager or pip.
-
Most developers choose PyCharm as their development environment, but you can also opt for other IDE tools.
Advantages
-
Since system behavior uses semi-formal language and domain vocabulary, Behave helps maintain consistent behavior within organizations.
-
It allows development teams working on different modules to have similar characteristics through coordinated handling.
-
The various functional blocks it builds can execute various test cases.
-
Due to the more thought-provoking details it provides, the target product will have better specifications.
-
Based on similar specification formats, it allows project managers and stakeholders to have a clearer understanding of the outputs from development and QA teams.
Disadvantages
It is only suitable for black-box testing.
Behave’s Features Compared to Competitors
Behave is best suited for black-box testing, such as simple web tests. However, it is not a good choice for integration and unit tests with complex scenarios. As an alternative to Behave, some developers and testers recommend using pytest-bdd. This tool combines all the advantages of Pytest and implements various behavior-driven testing scenarios.If you are considering using Behave, you can borrow the automation scripts mentioned in “Behave with Selenium” (see https://www.lambdatest.com/support/docs/display/TD/Behave+with+Selenium+-+Running+Behave+Automation+Scripts+on+LambdaTest+Selenium+Grid).
5. Lettuce
Lettuce is another behavior-driven automation tool based on Cucumber and Python. Lettuce mainly focuses on ordinary tasks with behavior-driven development characteristics. It is not only simple to use but also makes the entire testing process smoother and even more enjoyable.
Prerequisites
You need to install Python version 2.7.14 or higher with an IDE. Of course, you can also use PyCharm or any other IDE tool. Additionally, you need to install the Python package manager.
Advantages
- Like other BDD testing frameworks, Lettuce allows developers to create multiple scenarios and describe different functionalities in simple natural language.
- Based on similar specification formats, development and QA teams can coordinate and work together.
- Lettuce is very suitable for behavior-driven test cases in black-box testing.
Disadvantages
To successfully implement behavior-driven testing, continuous communication between development teams, QA personnel, and stakeholders is required. If such communication is lacking, it can lead to unclear processes and cause problems for the team. Competitively, Cucumber is considered more practical for executing BDD tests by ordinary developers and automation testers. In contrast, Pytest-bdd is viewed as better by Python developers and QA personnel. The strength of Pytest lies in its framework, which achieves compact and easy-to-understand code that closely integrates with behavior-driven testing.
Conclusion
In summary, among the five Python automation testing frameworks mentioned above, Pytest, Robot Framework, and UnitTest can mainly be used for functional and unit testing, while Lettuce and Behave are only suitable for behavior-driven testing. Through further comparison, we believe that for functional testing, Pytest is the best. If you are a newcomer to Python automation testing, Robot Framework is the best entry tool. Although its functionality is somewhat limited, it is very easy to get started. For Python-based BDD testing, both Lettuce and Behave are excellent. However, if you already have some experience with Pytest, please use pytest-bdd. I hope this article can help you select a suitable Python testing framework and successfully carry out your testing work.Editor: Wang JingProofreader: Lin Yilin