Recently, I worked on an IoT gateway project where the gateway device needed to call the cloud’s REST API. The cloud implemented certain business logic and encapsulated resources into various URI interfaces for external systems to call.
As a programmer, you should be quite familiar with REST APIs. Many of the basic services around us provide REST API interfaces. For example, to check the weather in a city, we call the meteorological service’s REST API. To query navigation map information, we call the Gaode map’s REST API. To inquire about logistics information, we can call SF Express’s backend REST API. For smart devices, we call iFlytek’s intelligent voice REST API, and for facial recognition, we request Face Plus Plus’s REST API, and so on.
The cloud encapsulates algorithms, data, and facility resources into web services, achieving decoupling between endpoints. The endpoints do not need to care about how the other side is implemented or what programming language is used, allowing independent development, testing, operation, and deployment. Currently, among the mainstream web service implementation solutions, REST has become increasingly popular as it is simpler compared to complex SOAP and XML-RPC.
REST (Representational State Transfer) is an architectural style proposed by Dr. Roy Thomas Fielding in his doctoral thesis in 2000, aimed at facilitating the exchange of information between different software/programs over the network (for example, the internet). Note that REST is an architectural style, not a technical specification; it differs from protocol standards defined by RCF, such as HTTP, TCP, and IP. REST is not a normative standard but a design style, typically based on using HTTP, URI, and widely popular protocols and standards like XML and JSON.
We know that for data transmission over public networks, packets can be easily eavesdropped and tampered with. This means that for plaintext transmitted data, it is impossible to ensure that the data is secure and trustworthy, which is crucial for services that emphasize data security and trustworthiness (such as online banking, user login, IoT device control, etc.). To ensure the security and trustworthiness of data during transmission, it is necessary to establish an encrypted data link and use encrypted packets for ciphertext transmission. While we cannot guarantee that public data links cannot be monitored, the encrypted packets ensure that even if they are intercepted, the plaintext meaning cannot be obtained, thus protecting the original data from being stolen or tampered with.
Transport Layer Security (TLS), and its predecessor Secure Sockets Layer (SSL), is a security protocol designed to provide security and data integrity for internet communications. To provide authentication for web servers and protect the privacy and integrity of exchanged data, Netscape introduced the HTTPS protocol in 1994, using SSL for encryption, marking the origin of SSL. The IETF standardized SSL, releasing the first version of the TLS standard document in 1999 and subsequently releasing RFC 5246 (August 2008) and RFC 6176 (March 2011). The TLS protocol is widely adopted in applications such as browsers, email, instant messaging, VoIP, and web fax. Major websites like Google and Facebook also use this protocol to create secure connections for data transmission, and it has become the industrial standard for confidential communications on the internet.
SSL includes a record layer and a transport layer, with the record layer protocol determining the encapsulation format for transport layer data. The transport layer security protocol uses X.509 authentication, followed by asymmetric encryption algorithms for identity authentication of the communicating parties, then exchanges symmetric keys as session keys. These session keys are used to encrypt the data exchanged between the two applications, ensuring confidentiality and reliability in communication, preventing attackers from eavesdropping on the communication between the client and server applications.
Hypertext Transfer Protocol Secure (HTTPS) is a network security transmission protocol. In computer networks, HTTPS communicates via the hypertext transfer protocol, but uses SSL/TLS to encrypt data packets. The main idea of HTTPS is to create a secure channel over an insecure network, providing reasonable protection against eavesdropping and man-in-the-middle attacks when appropriate encryption packets and trusted server certificates are used.
HTTPS utilizes SSL/TLS for encryption of data packets, and the TLS protocol is independent of application layer protocols. High-level application layer protocols (such as HTTP, FTP, Telnet, etc.) can transparently be built on top of the TLS protocol. The TLS protocol completes the encryption algorithms, communication key negotiations, and server authentication before the application layer protocol communication begins. After this, all data transmitted by the application layer protocol will be encrypted, ensuring the privacy of communication.
SSL/TLS protocols use public key encryption algorithms. The client first requests the public key from the server, then encrypts the information with the public key. After the server receives the ciphertext, it uses its private key to decrypt it. Before the client and server begin exchanging encrypted information protected by TLS, they must securely exchange or agree on the encryption keys and the passwords to be used for encryption. In other words, the client first requests the public key from the server, encrypts the information with the public key, and the server decrypts the ciphertext with its private key. However, before establishing the encrypted ciphertext, the data is transmitted in plaintext, meaning that the public key is transmitted in plaintext. How do we ensure that the public key is not tampered with? The solution is to place the public key in a digital certificate. As long as the certificate is trusted, the public key is trusted. This requires a “handshake phase”.
TLS negotiates a stateful connection to transmit data through a handshake process. During the handshake, the client and server negotiate various parameters to create a secure connection:
-
When the client connects to a server that supports the TLS protocol, it requests to create a secure connection and lists the supported cipher combinations (encryption cipher algorithms and encryption hash functions), the handshake begins.
-
The server selects encryption and hashing functions from the list and notifies the client.
-
The server sends back its digital certificate, which usually contains the server’s name, the trusted certificate authority (CA), and the server’s public key.
-
The client verifies the validity of the issued certificate.
-
To generate a session key for the secure connection, the client encrypts a randomly generated key using the server’s public key and sends it to the server, which can only decrypt it using its private key.
-
Using random numbers, both parties generate symmetric keys for encryption and decryption. This is the TLS protocol handshake, and the connection remains secure until it is closed. If any step fails, the TLS handshake process fails, and all connections are terminated.
Thus, the server first sends back its digital certificate, which usually contains the server’s name, trusted certificate authority (CA), and the server’s public key. After receiving the server’s response, the client first verifies the server’s certificate. If the certificate is not issued by a trusted authority, or if the domain name in the certificate does not match the actual domain name, or if the certificate has expired, a warning will be displayed to the visitor, who can choose whether to continue the communication. If the certificate is valid, the client extracts the server’s public key from the certificate.
For high-security applications, the server may also need to confirm the client’s identity, which will include a request for the client to provide a “client certificate.” For example, financial institutions often only allow authenticated clients to connect to their networks and provide USB keys containing a client certificate.
To determine whether the server is trustworthy, we will verify the server’s digital certificate, checking whether the certificate’s server name matches the HTTPS service domain we want to access, whether the certificate authority is trusted, and whether the certificate is within its validity period.
In the field of cryptography and computer security, a root certificate is an unsigned or self-signed public key certificate identifying a root certificate authority (CA). The root certificate is part of the public key infrastructure scheme. Most commonly used commercial schemes are based on the ITU-T X.509 standard, which usually includes a digital signature from the certificate authority. The certificate authority can issue multiple certificates in a tree structure, with the root certificate being the top-level certificate in this structure, and its private key is used to “sign” other certificates. All certificates issued after the root certificate inherit the trustworthiness of the root certificate — the signature of the root certificate is somewhat similar to a “notary” in the real world. Further down the chain, certificates rely on intermediate authorities’ trust, commonly referred to as “subordinate certificate authorities.”
Our operating systems usually come with digital signature certificates from trusted root certificate authorities. If your server’s SSL certificate is issued by a globally trusted certificate authority (CA) that verifies the server’s identity, it can be easily verified for legitimacy, but this requires additional fees, which are paid to these certificate authorities based on the certificate’s validity period.
Another method is to use a self-signed digital certificate and manually or programmatically add the digital certificate to the trusted certificate authorities to achieve digital certificate trust confirmation.
The REST API service I am using will utilize a self-signed digital certificate based on the TLS 1.2 protocol, employing both server-side and client-side certificate mutual authentication to achieve reliable encrypted data transmission.
Next, I will share some technical details and pitfalls I encountered during implementation.
Using OpenSSL to Generate Server and Client Certificates
We use the open-source tool OpenSSL to generate self-signed certificates for server and client SSL connections.
If you are using MAC OS or Linux, OpenSSL is already installed by default. If you are using Windows, you can install Git and use the shell that comes with Git or the Windows version of OpenSSL, which can be downloaded from this link:
http://slproweb.com/products/Win32OpenSSL.html
Generating the Server Certificate Private Key
Type the command in the shell
openssl genrsa -out server.key 2048
This will generate a server.key file in the current directory.
Next, use the server.key private key to generate a Certificate Signing Request (CSR) file
openssl req -new -key server.key -out server.csr
After executing the command, you will be prompted to enter your relevant information. Fill in as follows:
1. Country Name:
Fill in your country’s ISO standard code, e.g., CN for China, US for the United States.
2. State or Province Name:
Fill in the province/autonomous region/municipality where your unit is located, e.g., Guangdong Province or Guangdong.
3. Locality Name:
Fill in the city/county/district where your unit is located, e.g., Guangzhou or Guangzhou.
4. Organization Name:
Fill in the legal name of your unit/institution/company, e.g., Wu Chuanbin’s Blog or www.mr-wu.cn.
5. Organizational Unit Name:
Fill in the department name, e.g., Technical Support Department.
6. Common Name:
Fill in the domain name, e.g., api.mr-wu.cn. When there are multiple domain names, fill in the primary domain name. This is very important because when we verify the server certificate, we will check whether the Common Name matches the URL domain of the HTTP request; if it does not match, SSL will return an error.
7. Email Address:
Fill in your email address; this is optional, press enter to skip.
8. ‘extra’ attributes
All can be skipped, press enter until the command completes.
Here, it is important to emphasize that except for items 1, 6, 7, and 8, please fill in items 2-5 uniformly in either Chinese or English, meaning do not mix Chinese and English.
If the command is successfully executed, a certreq.csr file will be generated in the current directory.
Next, generate the server-side certificate
openssl x509 -req -days 2000 -in server.csr -signkey server.key -out server.crt
The certificate adopts the x509 format, and -days indicates the validity period of the certificate, which I set to 2000 days. If the command is successfully executed, a server.crt certificate file will be generated in the current directory.
Next, generate the client certificate
The method and steps for generating the client certificate are the same as those described for generating the server certificate.
Execute the command
openssl genrsa -out client.key 2048
openssl req -new -key client.key -out client.csr
openssl x509 -req -days 2000 -in client.csr -signkey client.key -out client.crt
Now, we have successfully generated the server’s private key and digital certificate, as well as the client’s private key and digital certificate. Next, we need to upload the server’s private key, digital certificate, and client’s digital certificate to the server’s HTTP Server and configure it to enable SSL connections and listen on port 443.
I am using Nginx here, and the specific configuration method will depend on the version of the HTTP Server you are using.
| 1234567891011 | listen 443 ssl http2; Enable SSL encrypted transmission and listen on port 443 ssl on; Enable SSL encryption ssl_certificate /etc/nginx/ssl/api.mr-wu.cn.key/server.crt; Server certificate path ssl_certificate_key /etc/nginx/ssl/api.mr-wu.cn.key/server.key; Server private key path ssl_client_certificate /etc/nginx/ssl/api.mr-wu.cn.key/client.crt; Client certificate path ssl_verify_client on; Enable client connection certificate authentication |
Then, you can use the curl command to test the server-side SSL connection.
| 1 | curl -v -s -k --key client.key --cert client.crt https://api.mr-wu.com |
If you find that access is not possible, for example, if you are using a cloud system like Aliyun, it will create a security group for the server, only allowing open ports, so you need to configure port 443 in the security group.
The HTTPS Web Server is set up. As for how to implement the REST API, I won’t go into detail here; you can use PHP frameworks, NodeJS frameworks, or JAVA frameworks; just configure Nginx’s reverse proxy.
Next, the QT5 program will connect to the cloud’s REST API via SSL. In the QT5 program implementation, we use the QNetworkAccessManager class to manage network connection requests, the QNetworkRequest class to perform HTTP operations such as post, get, delete, etc., and the QSslConfiguration class to configure SSL connections, including CA certificate configuration, client certificate, and client private key configuration, as well as setting the SSL protocol version and authentication mode.
There is a pitfall in QT5, which is that QT5 supports OpenSSL programming implementation but does not come with the corresponding binary runtime libraries. This may cause your program code to compile correctly, but when running and debugging in the IDE, SSL errors will be reported:
| 123456789 | QSslSocket: cannot call unresolved function SSLv23_client_method QSslSocket: cannot call unresolved function SSL_CTX_new QSslSocket: cannot call unresolved function SSL_library_init QSslSocket: cannot call unresolved function ERR_get_error QSslSocket: cannot call unresolved function ERR_get_error |
The solution is to copy the OpenSSL runtime libraries to QT. For Windows versions, these two DLL link libraries are:
ssleay32.dll
libeay32.dll
You can download a pre-compiled installation package for OpenSSL and copy these two files to QT, or as I did, use the MinGW QT installation, which includes the OpenSSL runtime libraries.
In the QT 5.9.0 MinGW version I installed, QT was installed in the D drive under the Qt directory, so the ssleay32.dll and libeay32.dll files are located in the “D:\Qt\Qt5.9.0\Tools\mingw530_32\opt\bin” directory. Copy them to QT’s bin directory: “D:\Qt\Qt5.9.0\5.9\mingw53_32\bin”.
After running the QT SSL connection program, it no longer reports errors. Of course, when you compile and release your program, remember to include the OpenSSL link library files.
For QT running on ARM, we need to cross-compile based on the QT source files, and during the compilation, OpenSSL must also be included.
Below is an example code snippet for QT5 SSL requests to REST APIs:
First, add network support in the QT project file (.pro)
QT += network
Then the C++ source code:
| 1234567891011121314 | // Check if the system supports OpenSSLqDebug() << "Supports OpenSSL: " << QSslSocket::supportsSsl(); // Create client certificateQFile fileCrt("client.crt");fileCrt.open(QIODevice::ReadOnly);const QSslCertificate certificate(&fileCrt, QSsl::Pem);fileCrt.close(); // Create client private keyQFile fileKey("client.key");fileKey.open(QIODevice::ReadOnly);const QSslKey privateKey(&fileKey, QSsl::Rsa);fileKey.close(); |
Here, when constructing the QSslCertificate and QSslKey classes, the QIODevice is directly reading the key and CRT files saved on disk. This means that when your program is released, it must also include the key and CRT files in the disk directory. This practice is not ideal; we can include the key and CRT files in QT’s resource files or convert the key and CRT directly into QByteArray constants, passing them directly into the class constructor. This way, we can avoid the key and CRT files being accidentally lost or modified.
| 123456789101112131415161718192021222324252627282930313233343536 | // Configure SSL connection using QSslConfiguration classQSslConfiguration config ;// Set SSL verification mode, here I set it to QSslSocket::VerifyPeer to verify the server's certificate during SSL handshakeconfig.setPeerVerifyMode(QSslSocket::VerifyPeer);// Use TLS 1.2 protocol version, depending on your server's supportconfig.setProtocol(QSsl::TlsV1_2);// Add the previously created client private key privateKey and client certificate certificate to the configconfig.setPrivateKey(privateKey);config.setLocalCertificate(certificate); // Since this is a self-signed server certificate, we also need to add the server certificate to the CA certificate database; otherwise, the server certificate will fail verification.QList<QSslCertificate> caCerList;// Create server certificateQFile fileServCrt("server.crt");fileServCrt.open(QIODevice::ReadOnly); const QSslCertificate caCertificate(&fileServCrt, QSsl::Pem);// Add the service certificate to the CA listcaCerList << caCertificate;config.setCaCertificates(caCerList); // Network connection pipelineQNetworkAccessManager *manager = new QNetworkAccessManager(this);QUrl url("https://api.mr-wu.cn/v1/");QNetworkRequest request(url);// Add SSL configuration informationrequest.setSslConfiguration(config);// Set REST API data content type to JSON formatrequest.setHeader(QNetworkRequest::ContentTypeHeader, "application/json"); connect(manager, SIGNAL(finished(QNetworkReply*)), this, SLOT(syncRequestFinished(QNetworkReply*))); connect(manager, SIGNAL(sslErrors(QNetworkReply*,QList<QSslError> )), this, SLOT(sslErrors(QNetworkReply*,QList<QSslError> ))); manager->get(request); |
Thank you for patiently reading this article. If there are any errors in the text, feel free to leave a comment for correction, and I hope this article can be helpful to you. Let’s ignore the pitfalls of SSL, O(∩_∩)O~
