Introduction:
CTest is the CMake testing tool. This article explores the functionalities of CTest by writing and testing code that can sum integers. To illustrate that CMake imposes no restrictions on the language used for actual testing, we will use C++ executables for testing, as well as Python scripts and shell scripts. For simplicity, we will not use any testing libraries, but we will familiarize ourselves with the C++ testing framework in subsequent notes.
✦
Project Structure
✦
.
├── CMakeLists.txt
├── main.cpp
├── sum_integers.cpp
├── sum_integers.h
├── test.cpp
├── test.py
└── test.sh
Project URL:
https://gitee.com/jiangli01/tutorials/tree/master/cmake-tutorial/chapter4/01
✦
CMakeLists.txt
✦
cmake_minimum_required(VERSION 3.10 FATAL_ERROR)
project(test_ctest LANGUAGES CXX)
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_EXTENSIONS OFF)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
find_package(PythonInterp REQUIRED)
find_program(BASH_EXECUTABLE NAMES bash REQUIRED)
# example library
add_library(sum_integers sum_integers.cpp)
# main code
add_executable(sum_up main.cpp)
target_link_libraries(sum_up sum_integers)
# testing binary
add_executable(cpp_test test.cpp)
target_link_libraries(cpp_test sum_integers)
enable_testing()
add_test(
NAME bash_test
COMMAND ${BASH_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test.sh $<TARGET_FILE:sum_up>
)
add_test(
NAME cpp_test
COMMAND $<TARGET_FILE:cpp_test>
)
add_test(
NAME python_test_long
COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test.py --executable $<TARGET_FILE:sum_up>
)
add_test(
NAME python_test_short
COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test.py --short --executable $<TARGET_FILE:sum_up>
)
find_program(BASH_EXECUTABLE NAMES bash REQUIRED)
This is used to find the bash executable file in the system and store the path of the executable file in the variable BASH_EXECUTABLE. This can be used to execute bash scripts or commands during the CMake build process.
enable_testing()
This enables testing for this directory and all subfolders (since we placed it in the main CMakeLists.txt).
add_test(
NAME cpp_test
COMMAND $<TARGET_FILE:cpp_test>
)
This defines a new test and sets the test name and the command to run.
In the above code, a generator expression is used: $<TARGET_FILE:cpp_test>. A generator expression is an expression that is evaluated during the generation of the build system. We will introduce and learn about generator expressions in subsequent learning materials. Then, the variable $<TARGET_FILE:cpp_test> will be replaced with the full path of the cpp_test executable target.
NOTE Generator expressions are very convenient during testing because there is no need to hard-code the location and name of the executable program into the test. Implementing this in a portable way is very troublesome because the location of executables and their extensions (e.g., .exe on Windows) may vary across different operating systems, build types, and generators. By using generator expressions, we do not have to explicitly know the locations and names.
Parameters can also be passed to the test command to be executed:
add_test(
NAME python_test_short
COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test.py --short --executable $<TARGET_FILE:sum_up>
)
NOTE CTest can run test suites in any programming language. CTest cares about whether the test case passes based on the return code of the command. The standard convention that CTest follows is that returning zero means success, while a non-zero return means failure. Scripts that can return either zero or non-zero can be used as test cases.
✦
Related Source Code
✦
sum_integers.h
#ifndef SUM_INTEGERS_H
#define SUM_INTEGERS_H
#include <vector>
int sum_integers(const std::vector<int> &integers);
#endif // ! SUM_INTEGERS_H
sum_integers.cpp
#include "sum_integers.h"
int sum_integers(const std::vector<int>& integers) {
auto sum = 0;
for (auto i : integers) {
sum += i;
}
return sum;
}
main.cpp
#include <iostream>
#include <string>
#include "sum_integers.h"
int main(int argc, char *argv[]) {
std::vector<int> integers;
for (auto i = 1; i < argc; i++) {
integers.push_back(std::stoi(argv[i]));
}
auto sum = sum_integers(integers);
std::cout << sum << std::endl;
return 0;
}
test.cpp
#include "sum_integers.h"
int main() {
auto integers = {1, 2, 3, 4, 5};
if (sum_integers(integers) == 15) {
return 0;
} else {
return 1;
}
}
test.py
import subprocess
import argparse
# test script expects the executable as argument
parser = argparse.ArgumentParser()
parser.add_argument('--executable', help='full path to executable')
parser.add_argument('--short',
default=False,
action='store_true',
help='run a shorter test')
args = parser.parse_args()
def execute_cpp_code(integers):
result = subprocess.check_output([args.executable] + integers)
return int(result)
if args.short:
# we collect [1, 2, ..., 100] as a list of strings
result = execute_cpp_code([str(i) for i in range(1, 101)])
assert result == 5050, 'summing up to 100 failed'
else:
# we collect [1, 2, ..., 1000] as a list of strings
result = execute_cpp_code([str(i) for i in range(1, 1001)])
assert result == 500500, 'summing up to 1000 failed'
test.sh
#!/usr/bin/env bash
EXECUTABLE=$1
OUTPUT=$($EXECUTABLE 1 2 3 4)
if [ "$OUTPUT" = "10" ]
then
exit 0
else
exit 1
fi
✦
Appendix
✦
1. Consider the following definition:
add_test(
NAME python_test_long
COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test.py --executable $<TARGET_FILE:sum_up>
)
The previous definition can be rephrased by explicitly specifying the WORKING_DIRECTORY for the script to run, as follows:
add_test(
NAME python_test_long
COMMAND ${PYTHON_EXECUTABLE} test.py --executable $<TARGET_FILE:sum_up>
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
)
Test names can contain the / character, which is useful for organizing related tests by name, for example:
add_test(
NAME python/long
COMMAND ${PYTHON_EXECUTABLE} test.py --executable $<TARGET_FILE:sum_up>
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
)
Sometimes, we need to set environment variables for the test script. This can be achieved through set_tests_properties:
set_tests_properties(python_test
PROPERTIES
ENVIRONMENT
ACCOUNT_MODULE_PATH=${CMAKE_CURRENT_SOURCE_DIR}
ACCOUNT_HEADER_FILE=${CMAKE_CURRENT_SOURCE_DIR}/account/account.h
ACCOUNT_LIBRARY_FILE=$<TARGET_FILE:account>
)
This approach may not always work on different platforms, and CMake provides a way to solve this problem. The following code snippet is equivalent to the above code snippet, where the environment variables are pre-set by invoking CMake through CMAKE_COMMAND before executing the actual Python test script:
add_test(
NAME
python_test
COMMAND
${CMAKE_COMMAND} -E env
ACCOUNT_MODULE_PATH=${CMAKE_CURRENT_SOURCE_DIR}
ACCOUNT_HEADER_FILE=${CMAKE_CURRENT_SOURCE_DIR}/account/account.h
ACCOUNT_LIBRARY_FILE=$<TARGET_FILE:account>
${PYTHON_EXECUTABLE}
${CMAKE_CURRENT_SOURCE_DIR}/account/test.py
)
Also, note the use of the generator expression $<TARGET_FILE:account> to pass the location of the library file.
2. Different Platform Test Commands We have already executed tests using the ctest command. CMake will also create targets for the generator (Unix Makefile generator uses make test, Ninja tool uses ninja test, or Visual Studio uses RUN_TESTS). This means there is another (almost) portable way to run tests:
$ cmake --build . --target test
When using the Visual Studio generator, we need to use RUN_TESTS instead:
$ cmake --build . --target RUN_TESTS
