Hello, Web3 surfers and developers!
Today, we are going to discuss a topic that gives all project teams headaches—smart contract security.
In this world, code is law, and a single line of code negligence can mean millions of dollars in assets going up in smoke. According to the FailSafe Web3 Security Report, the losses from Ethereum smart contracts in 2024 reached as high as $2.6 billion, making the importance of security audits self-evident.
Traditional code audits either rely on automated tools, which have high false negative rates, or spend huge sums hiring human expert teams, often costing thousands or even tens of thousands of dollars, with long cycles.
Is there a faster, more accurate, and cheaper way?
Recently, a paper titled “LLM-SmartAudit: Advanced Smart Contract Vulnerability Detection” provided us with an astonishing answer. Researchers from institutions such as Beijing Institute of Technology, University of Auckland, and Nanyang Technological University developed a framework called LLM-SmartAudit.
In simple terms, they created an “AI Agent audit team” where several AI experts in different roles chat and debate with each other to conduct a comprehensive audit of smart contracts. How effective is it?
It boasts an accuracy rate of 98%, with an average audit cost of only about $1 per contract!
Today, we will delve into this disruptive framework.
—1—
📖 A Good Man Needs Three Helpers: The AI Agent Audit Team
In the past, we let a single large model (like GPT-4o, Claude) analyze the code directly. While it had some effect, it was like asking a general practitioner to diagnose all specialized diseases; there are always times when it falls short, even leading to “hallucinations” and false reports of vulnerabilities.
The core idea of LLM-SmartAudit is role specialization and collaborative combat. It is no longer a solo effort but has established a virtual audit team.
This team consists of four core AI roles:
-
🧑💼 Project Manager: Responsible for overseeing the entire process and ensuring the audit runs smoothly.
-
🕵️ Smart Contract Auditor: A security expert who looks for potential vulnerabilities from the attacker’s perspective.
-
👩🏫 Smart Contract Counselor: Responsible for verifying findings, refining conclusions, and ultimately forming the report.
-
💻 Solidity Programming Expert: A code master who deeply analyzes the structure, logic, and implementation details of the code.
These four AI agents hold meetings like a real team, discussing through multiple rounds of dialogue, examining the contract from different angles to ensure the depth and breadth of the audit.
—2—
🔍 Dual Sword Combination: Two Modes
To address different scenarios, LLM-SmartAudit has designed two complementary audit modes:
1. Broad Analysis (BA) Mode 🗺️
This is like a large net, conducting a carpet search. It is not limited to known vulnerability types but uses the extensive knowledge base of large models to discover various potential, even entirely new security risks. This mode is very suitable for preliminary screening of contracts.
2. Targeted Analysis (TA) Mode 🎯
This is like a cruise missile, striking precisely. The researchers preset 40 common and high-risk vulnerability scenarios (such as reentrancy attacks, integer overflows, etc.). The TA mode conducts in-depth, focused analysis for each scenario. This mode is very effective for detecting known critical vulnerabilities.
The paper also recommends a Hybrid Mode, which combines the results of both modes. Practice has shown that this mode can identify the most types of unique vulnerabilities, covering the widest range.
—3—
How Do AI Agents Collaborate? Witness the Power of “Role Swapping”
The most interesting aspect of this framework is the collaboration mechanism between AIs. They reach consensus through continuous dialogue and debate.
To prevent AIs from being stubborn or making misjudgments, the framework introduces a “Role Swapping” mechanism.
For example:
-
Initial Analysis: The Auditor AI 🕵️ checks a piece of code and believes it has both “integer overflow” and “reentrancy attack” vulnerabilities.
-
Role Swapping: At this point, the system allows the Auditor AI 🕵️ and the Programming Expert AI 💻 to swap roles. The Programming Expert AI re-examines from the perspective of code implementation and finds that the so-called “reentrancy” risk is actually unfounded, just an illusion..
-
Final Consensus: After a round of “debate”, the team reaches a consensus and ultimately reports only the confirmed vulnerability of “integer overflow”, avoiding false reporting.
This mechanism simulates the cross-validation process of human experts, greatly enhancing the accuracy and reliability of the final results.
—4—
Significant Results: What Do the Data Say?
Words alone are not enough; let’s take a look at the astonishing performance of this framework in experiments.
-
✅ Crushing Traditional Tools: On a dataset containing 10 common vulnerabilities, LLM-SmartAudit (GPT-4o configuration) achieved an accuracy rate of 98%, far surpassing well-known traditional tools like Mythril (54%) and Slither (46%).
-
🌍 Real-World Performance: On a real-world project dataset containing 6454 contracts, the TA modeidentified 47.6% of vulnerabilities, a result significantly better than all other solutions using large models alone.
-
🛡️ Easily Identifying Known “Big Risks”: In tests targeting 13 well-known CVEs (Common Vulnerabilities and Exposures), LLM-SmartAuditsuccessfully identified 12, surpassing other specialized LLM auditing tools like PropertyGPT and GPTScan.
—5—
Cost is the Real Driving Factor!
Having discussed so much, let’s talk about the most critical aspect—money.
How expensive are traditional manual audit services? Top companies like Trail of Bits start at $5000 to $15000.
And what about LLM-SmartAudit?
Researchers estimated based on the API price of GPT-4o that the most powerful TA mode has an
average cost of only $0.98 per contract; while the faster BA mode costs as low as $0.21
This means that what was once only affordable for top project teams can now be completed by ordinary developers for the price of a cup of coffee. This greatly lowers the barrier for blockchain security analysis.
—6—
Conclusion
LLM-SmartAudit, through its innovative multi-agent collaboration framework, proves that AI can not only understand complex code logic but also perform professional security audits with a high accuracy rate and extremely low cost, far exceeding traditional tools.
Although this framework still relies on static analysis and has limited capabilities for detecting certain runtime vulnerabilities, the potential it demonstrates is enormous. As large language models continue to evolve, we have reason to believe that the future world of smart contracts will become safer due to such “AI Agent audit teams”.
Read the original article
Title: LLM-SmartAudit: Advanced Smart Contract Vulnerability Detection
Publication Date: October 12, 2024
Link:
https://arxiv.org/pdf/2410.09381
arxiv
https://github.com/LLMAudit/LLMSmartAuditTool1
Github