-begin-
The define command can encapsulate quick commands for step-by-step debugging, but during regression testing or version iterations, multiple core dump files (more than 10) often arise. Analyzing them one by one not only takes time but may also overlook common issues.The gdb -x command can execute debugging operations in batches through scripts, traversing all core dumps like an “automated detective”, quickly tallying crash reasons and improving batch analysis efficiency by 10 times.
Scenario Restoration: “Version Regression” of Smart Home Firmware
Previously, when developing a smart home firmware, after a certain version iteration, 5 core dump files appeared during regression testing, each from different test cases (such as “device pairing”, “firmware upgrade”, “network reconnection”). Manually analyzing each file took at least 10 minutes, and it was difficult to determine if there were common issues.
After switching to batch analysis using the gdb -x script, conclusions were drawn in just 2 minutes: 4 out of the 5 core dumps crashed at the same line in the wifi_connect() function, and the call stack showed memcpy operations—clearly indicating that the same bug triggered different scenarios.
Command Practice: Using Scripts to Batch Extract Crash Information
First, write a batch_analyze.gdb script to define the debugging commands to be executed:
|
# batch_analyze.gdb: Script for batch analyzing core dumps set logging file core_analysis.log # Output logs to file set logging on # Enable logging echo ====== Analysis Start: $arg0 ====== bt # Print the crash thread call stack echo info threads # Record thread states echo frame 0 # Switch to the crash stack frame info locals # Print local variables echo set logging off # Disable logging quit # Exit gdb |
Then use gdb -x to call the script and batch process all core dump files:
|
# Traverse all core files in the current directory and analyze them one by one for core in core.*; do gdb -batch -x batch_analyze.gdb -ex “set args $core” ./smart_home_firmware $core done |
The generated core_analysis.log clearly records the key information of each core dump:
|
====== Analysis Start: core.12345 ====== #0 0x0000000000402345 in wifi_connect (ssid=0x7ffff0001234, len=32) at wifi.c:156 #1 0x0000000000401876 in pair_device () at device.c:210 … ===== All Thread States ===== 1 Thread 0x7ffff7fc8700 (LWP 12345) 0x0000000000402345 in wifi_connect () … ===== Local Variables at Crash Point ===== ssid = 0x7ffff0001234 “MyHomeWiFi1234567890123456789012” # Length 32 buf = “\x00\x00\x00\x00\x00\x00\x00\x00” # Target buffer, size 16 bytes len = 32 # Input length exceeds buffer size ===== Analysis End ====== ====== Analysis Start: core.12346 ====== #0 0x0000000000402345 in wifi_connect (ssid=0x7ffff0001235, len=28) at wifi.c:156 … # Also crashes at wifi.c:156, local variables show len=28 > buf size 16 |
From the logs, it is clear that: in the wifi_connect function, memcpy(buf, ssid, len) does not check len, and when the ssid length exceeds buf of 16 bytes, it triggers a buffer overflow, causing crashes in different test cases. After fixing by adding len = min(len, 16), all core dumps disappeared.
Command Breakdown: The “Batch Crime Solving Technique” of gdb -x
The core of gdb -x <script file> is “automated batch execution”, defining a fixed debugging process through scripts, allowing gdb to repeat the following operations for each core dump file:
1. Logging: Use set logging to save output to a file, avoiding manual copying;2. Key Information Extraction: Use commands like bt, info threads, info locals to extract core data such as crash location, thread states, variable values, etc.;3. Automatic Exit: Use the quit command to automatically exit after completing the analysis of one core dump, continuing to process the next.
In batch analysis, its key value lies in:
• Efficiency Improvement: Analyzing 10 core dump files manually takes 1 hour, while the script processes them in just 5 minutes;• Commonality Mining: Summarizing all analysis results into logs, quickly discovering recurring crash points through text searches (e.g., searching “wifi.c:156”);• Standardized Output: Ensuring consistent analysis dimensions for each core dump (e.g., including call stacks, local variables) for easy comparison.
Practical Tips: 3 Techniques to Make Batch Scripts More Useful
4. Add Parameter Recognition: Use $arg0 in the script to get the core dump file name (must be passed in the command line with -ex “set args $core”), making it easy to mark the source in the logs;5. Focus on Key Information: Avoid printing redundant content (like complete memory data), only extract core information such as crash points and variable values to reduce log size;6. Combine Text Tools for Analysis: Use grep, awk, and other tools to process logs, counting the functions with the most crashes (e.g., grep “#0” core_analysis.log | sort | uniq -c), quickly locating high-frequency issues.
Pitfall Alerts
• Avoid using interactive commands in scripts (like continue, next), as core dumps are static memory snapshots and cannot execute dynamic debugging;• Ensure the executable program matches the core dump (same compilation version, debug symbols), otherwise a “no symbol table” error will occur;• For complex scripts, it is recommended to test single files first (e.g., gdb -x script.gdb ./app core.12345), confirming correct output before executing in batches.
The core of gdb -x is “liberating manpower with scripts and discovering patterns with tools”. In embedded development, frequent version iterations often lead to commonalities in core dumps generated during regression testing, and batch analysis can quickly lock down “new bugs that appeared after changing a line of code”. The issue with that smart home firmware, if analyzed manually one by one, could take half a day to discover it was the same overflow, but after batch processing with scripts, the root cause was located in just 10 minutes.
-end-
If this technique helps you, please like and follow, as I will share more practical insights on embedded debugging in the future~