In embedded development, reliable communication between devices has always been a challenge.
Today, we will share an excellent open-source project—LwPKT, and see how it solves complex communication protocol issues with less than 1000 lines of code.
1. Pain Points of Embedded Communication
In embedded development, we often encounter the following issues:
- Memory Constraints: MCUs have only a few KB of RAM, and complex protocol stacks cannot run at all.
- High Real-Time Requirements: In industrial control scenarios, millisecond-level delays are intolerable.
- Reliability Requirements: Data transmission errors can lead to device failures or even safety incidents.
- Multi-Device Networking: Connecting dozens of devices on an RS-485 bus requires address management.
- Development Efficiency: Writing communication protocols from scratch leads to long debugging cycles and many bugs.
The LwPKT (Lightweight Packet Protocol) introduced today is designed to address these pain points.
- Industrial Automation: Communication between RS-485 network devices.
- MCU Communication: Interconnection of microcontrollers such as STM32 and ESP32.
- IoT Devices: Sensor data collection and control command issuance.
- Wireless Communication: Data encapsulation for LoRa, WiFi, and Bluetooth modules.
2. Introduction to LwPKT
LwPKT was developed by renowned embedded developer Tilen MAJERLE. It is a lightweight packet protocol library written in C, applied in the embedded field. The current version is v1.4.1.
https://github.com/MaJerle/lwpkt
Let’s first look at its core features:
2.1 Core Features
- Ultra Lightweight: Core code is less than 1000 lines, with a minimum memory footprint of <100 bytes.
- Platform Independent: Pure C implementation, supports all mainstream MCU platforms.
- Configurable: Functional modularization, enabling features as needed to avoid resource waste.
- Reliable Transmission: Supports CRC checks, timeout detection, and multi-layer error handling.
- Variable-Length Encoding: Theoretically supports unlimited-length packets, with high transmission efficiency.
- Event-Driven: Asynchronous processing mode, does not block the main program execution.
2.2 Application Scenarios
- Industrial Automation: Communication between RS-485 network devices.
- MCU Communication: Interconnection of microcontrollers such as STM32 and ESP32.
- IoT Devices: Sensor data collection and control command issuance.
- Wireless Communication: Data encapsulation for LoRa, WiFi, and Bluetooth modules.
3. Learning Core Principles
3.1 Core File Structure
lwpkt/
├── lwpkt/src/
│ ├── include/lwpkt/
│ │ ├── lwpkt.h # Core API interface
│ │ └── lwpkt_opt.h # Configuration options
│ └── lwpkt/
│ └── lwpkt.c # Core implementation
├── examples/
│ └── example_lwpkt.c # Usage example
├── tests/
│ └── test_main.c # Test cases
└── docs/ # Documentation
3.2 Packet Format
The packet format of LwPKT is carefully designed to ensure both functional integrity and resource consumption:
Field Description:
- START (0xAA): Start flag, helps the receiver synchronize.
- FROM/TO (optional): Sender and receiver addresses, supports 8-bit or 32-bit.
- FLAGS (optional): User-defined flags, can be used for priority, type identification, etc.
- CMD (optional): Command field, supports 8-bit or multi-byte commands.
- LEN (variable-length encoding): Data length, using variable-length encoding to save transmission bandwidth.
- DATA: Actual data payload.
- CRC (optional): Checksum, supports CRC-8 or CRC-32.
- STOP (0x55): End flag.
3.3 Core Data Structure
The core of LwPKT is the <span>lwpkt_t</span> structure:
Some members are configurable, with functional modularization, enabling features as needed to avoid resource waste.
3.4 Circular Buffer
LwPKT relies on the LwRB (Lightweight Ring Buffer) library to manage data buffering, which is a classic producer-consumer pattern application:
We shared LwRB in our previous article: A lightweight ring buffer management library suitable for embedded systems!
Data sending process: Application -> LwPKT -> TX buffer -> Hardware interface. 
Data receiving process: Hardware interface -> RX buffer -> LwPKT -> Application.
3.5 Protocol Parsing
The protocol data reception and parsing use a finite state machine (FSM) design.
State enumeration:
State transition function:
The state transition function utilizes the fallthrough feature of C language to automatically skip disabled fields based on configuration.
In C language,
<span>fallthrough</span><span> refers to the feature where, after a case branch in a </span><code><span>switch</span>statement is executed, the program flow automatically “falls through” to the next case branch without using a<span>break</span>statement. This is the default behavior of the<span>switch</span>structure in C, which is both a flexible feature and can lead to logical errors if misused.To distinguish between “intentional fallthrough” and “unintentional omission”, the C17 standard introduced the
<span>[[fallthrough]]</span>attribute to explicitly mark “this fallthrough is intentional”, eliminating compiler warnings.
The advantages of the state machine are:
- Clear Logic: Each state has a single responsibility, making it easy to understand and debug.
- Efficient Processing: O(1) time complexity, suitable for real-time systems.
- Error Recovery: Any exception in any state can quickly recover to the initial state.
3.6 Variable-Length Encoding and Decoding
LwPKT uses MSB (Most Significant Bit) encoding to implement data encoding. Its core idea is “allocate as needed”—small numbers use fewer bytes, while large numbers use more bytes, thus saving transmission bandwidth in most cases.
In the LwPKT protocol, key fields in the protocol are designed to support 32-bit (4-byte) data:
// Address field - supports extended 32-bit address
#if LWPKT_CFG_ADDR_EXTENDED
typedef uint32_t lwpkt_addr_t; // 32-bit address
#else
typedef uint8_t lwpkt_addr_t; // 8-bit address
#endif
// Command field
uint32_t cmd;
// Flag field
uint32_t flags;
// Data length - size_t type (usually 32-bit or 64-bit)
size_t len;
Using variable-length encoding can significantly reduce protocol overhead and improve transmission efficiency.
Fixed-Length Encoding vs Variable-Length Encoding:
For example, for a uint32_t type command field, if assigned the value 1, fixed-length encoding would require 4 bytes for transmission, while MSB (Most Significant Bit) variable-length encoding would only require 1 byte.
LwPKT Encoding and Decoding Process:
Key points of MSB variable-length encoding:
-
MSB Control: Highest bit = 1 indicates continue, = 0 indicates end.
-
7-bit Data: Each byte uses only 7 bits to store data, 1 bit for control.
-
Little-endian Assembly: Low-order byte first, high-order byte last.
-
On-Demand Expansion: Small numbers use fewer bytes, large numbers use more bytes.
Encoding and decoding related code:
3.7 Configuration Mechanism
LwPKT adopts a three-layer configuration system, achieving complete configurability from compile time to runtime:
First Layer: Global Configuration at Compile Time
// lwpkt_opt.h - Default configuration
#define LWPKT_OFF 0 // Function completely disabled
#define LWPKT_ON_STATIC 1 // Function statically enabled
#define LWPKT_ON_DYNAMIC 2 // Function dynamically controlled
// Users can override in lwpkt_opts.h
#ifndef LWPKT_CFG_USE_CRC
#define LWPKT_CFG_USE_CRC LWPKT_ON_STATIC
#endif
Second Layer: Conditional Compilation Control
// Decide whether to compile code based on configuration value
#if LWPKT_CFG_USE_CRC
// CRC related code
static void prv_crc_init(lwpkt_t* pkt, lwpkt_crc_t* crcobj);
static uint32_t prv_crc_in(lwpkt_t* pkt, lwpkt_crc_t* crcobj, const void* inp, const size_t len);
#endif
// Function parameters are also affected by conditional compilation
static uint8_t prv_write_bytes_var_encoded(
lwpkt_t* pkt,
uint32_t var_num
#if LWPKT_CFG_USE_CRC
, lwpkt_crc_t* crc // This parameter only exists when CRC is enabled
#endif
);
Third Layer: Runtime Dynamic Control
// Instance-level flag control
typedef struct lwpkt {
uint8_t flags; // Runtime control flags
// ...
} lwpkt_t;
#define LWPKT_FLAG_USE_CRC ((uint8_t)0x01)
#define LWPKT_FLAG_CRC32 ((uint8_t)0x02)
#define LWPKT_FLAG_USE_ADDR ((uint8_t)0x04)
#define LWPKT_FLAG_ADDR_EXTENDED ((uint8_t)0x08)
#define LWPKT_FLAG_USE_CMD ((uint8_t)0x10)
#define LWPKT_FLAG_CMD_EXTENDED ((uint8_t)0x20)
#define LWPKT_FLAG_USE_FLAGS ((uint8_t)0x40)
// Dynamic control function
void lwpkt_set_crc_enabled(lwpkt_t* pkt, uint8_t enable) {
if (enable) {
pkt->flags |= LWPKT_FLAG_USE_CRC;
} else {
pkt->flags &= ~LWPKT_FLAG_USE_CRC;
}
}
3.8 Event Mechanism
The event mechanism of LwPKT implements a lightweight observer pattern, allowing applications to listen to various state changes and operation events of the protocol stack.
Related Articles: Embedded Programming Models | Observer Pattern
3.8.1 Core Event Types
typedef enum {
LWPKT_EVT_PKT, // Packet ready event
LWPKT_EVT_TIMEOUT, // Timeout event
LWPKT_EVT_READ, // Read operation event
LWPKT_EVT_WRITE, // Write operation event
LWPKT_EVT_PRE_WRITE, // Pre-write event
LWPKT_EVT_POST_WRITE, // Post-write event
LWPKT_EVT_PRE_READ, // Pre-read event
LWPKT_EVT_POST_READ, // Post-read event
} lwpkt_evt_type_t;
3.8.2 Event Mechanism and Trigger Timing
// Callback function definition
typedef void (*lwpkt_evt_fn)(struct lwpkt* pkt, lwpkt_evt_type_t evt_type);
// Event registration interface
lwpkt_ret_t lwpkt_set_evt_fn(lwpkt_t* pkt, lwpkt_evt_fn evt_fn) {
pkt->evt_fn = evt_fn;
return lwpktOK;
}
// Event sending macro
#if LWPKT_CFG_USE_EVT
#define SEND_EVT(pkt, event) \
do { \
if ((pkt)->evt_fn != NULL) { \
(pkt)->evt_fn((pkt), (event)); \
} \
} while (0)
#else
#define SEND_EVT(pkt, event) // Empty implementation, zero overhead
#endif
Trigger timing:
// Triggered by lwpkt_read interface
lwpkt_ret_t lwpkt_read(lwpkt_t* pkt) {
lwpkt_ret_t res = lwpktOK;
uint8_t b, e = 0;
// 1. Pre-read event before read operation
SEND_EVT(pkt, LWPKT_EVT_PRE_READ);
// Main loop for processing received data
// ...
retpre:
// 2. Post-read event after read operation
SEND_EVT(pkt, LWPKT_EVT_POST_READ);
// 3. If data was processed, send read event
if (e) {
SEND_EVT(pkt, LWPKT_EVT_READ);
}
return res;
}
// Triggered by lwpkt_write interface
lwpkt_ret_t lwpkt_write(lwpkt_t* pkt, /* parameter list */) {
lwpkt_ret_t res = lwpktOK;
// 1. Pre-write event before write operation
SEND_EVT(pkt, LWPKT_EVT_PRE_WRITE);
// Packet construction process...
// Write START, address, command, length, data, CRC, STOP
fast_return:
// 2. Post-write event after write operation
SEND_EVT(pkt, LWPKT_EVT_POST_WRITE);
// 3. If write is successful, send write event
if (res == lwpktOK) {
SEND_EVT(pkt, LWPKT_EVT_WRITE);
}
return res;
}
// Triggered by lwpkt_process interface
lwpkt_ret_t lwpkt_process(lwpkt_t* pkt, uint32_t time) {
lwpkt_ret_t pktres = lwpkt_read(pkt);
if (pktres == lwpktVALID) {
pkt->last_rx_time = time;
// 1. Packet ready event
SEND_EVT(pkt, LWPKT_EVT_PKT);
} elseif (pktres == lwpktINPROG) {
if ((time - pkt->last_rx_time) >= LWPKT_CFG_PROCESS_INPROG_TIMEOUT) {
lwpkt_reset(pkt);
pkt->last_rx_time = time;
// 2. Timeout event
SEND_EVT(pkt, LWPKT_EVT_TIMEOUT);
}
} else {
pkt->last_rx_time = time;
}
return pktres;
}
3.9 Code Practice
Let’s implement a minimal example of LwPKT:
#include <stdio.h>
#include <string.h>
#include "lwpkt/lwpkt.h"
/* Define buffer size */
#define BUFFER_SIZE 64
/* LwPKT instance and ring buffer */
static lwpkt_t pkt;
static lwrb_t tx_rb, rx_rb;
static uint8_t tx_data[BUFFER_SIZE], rx_data[BUFFER_SIZE];
void simulate_transfer(void) {
uint8_t byte = 0;
while (lwrb_read(&tx_rb, &byte, 1) == 1) {
lwrb_write(&rx_rb, &byte, 1);
}
}
int main(void) {
lwpkt_ret_t result;
const char* message = "Hello LwPKT!";
printf("============ LwPKT Test ============\n");
/* Initialize ring buffer */
lwrb_init(&tx_rb, tx_data, sizeof(tx_data));
lwrb_init(&rx_rb, rx_data, sizeof(rx_data));
/* Initialize LwPKT instance */
if (lwpkt_init(&pkt, &tx_rb, &rx_rb) != lwpktOK) {
printf("LwPKT initialization failed!\n");
return -1;
}
/* Send data packet */
printf("\n====== Sending ======\n");
printf("Sending data packet: %s\n", message);
result = lwpkt_write(&pkt,
0x01, /* Target address */
0x12345678, /* Flags */
0x01, /* Command */
message, strlen(message)); /* Data */
if (result != lwpktOK) {
printf("Sending failed: %d\n", result);
return -1;
}
/* Simulate network transfer */
simulate_transfer();
/* Receive and parse data packet */
result = lwpkt_read(&pkt);
if (result == lwpktVALID) {
printf("\n====== Receiving ======\n");
/* Print packet information */
printf("Sender address: 0x%08X\n", (unsigned)lwpkt_get_from_addr(&pkt));
printf("Receiver address: 0x%08X\n", (unsigned)lwpkt_get_to_addr(&pkt));
printf("Flags: 0x%08X\n", (unsigned)lwpkt_get_flags(&pkt));
printf("Command: 0x%02X\n", (unsigned)lwpkt_get_cmd(&pkt));
size_t data_len = lwpkt_get_data_len(&pkt);
printf("Data length: %zu bytes\n", data_len);
if (data_len > 0) {
uint8_t* data = lwpkt_get_data(&pkt);
printf("Data content: ");
for (size_t i = 0; i < data_len; i++) {
printf("%c", data[i]);
}
printf("\n");
}
}
return 0;
}
3.10 Comparison with Other Protocols
| Feature | LwPKT | Modbus RTU | JSON over UART | Custom Protocol |
|---|---|---|---|---|
| Code Size | <1KB | ~10KB | ~50KB | Variable |
| RAM Usage | <100B | ~1KB | ~5KB | Variable |
| Parsing Speed | Very Fast | Fast | Slow | Variable |
| Configuration Flexibility | Very High | Low | Medium | High |
| Learning Curve | Low | Medium | Low | High |
| Ecological Support | Medium | High | High | Low |
Selection Recommendations:
- Severely Resource-Constrained: Choose the minimal configuration of LwPKT.
- Need Standardization: Prioritize Modbus RTU.
- Rapid Prototyping: JSON solutions have high development efficiency.
- Special Requirements: Custom development based on LwPKT.
4. Conclusion
By studying the LwPKT project, we can see the qualities that an excellent embedded communication protocol should possess:
- Minimal Design Philosophy: A complete protocol stack implemented in less than 1000 lines of code.
- High Configurability: Enable features as needed to avoid resource waste.
- Reliability Assurance: Multi-layer error detection, suitable for harsh environments.
- Performance Optimization: State machine driven, O(1) complexity processing.
- Easy Integration: Pure C implementation, no external dependencies.