Basic Information
Original Title: MultiFuzz: A Dense Retrieval-based Multi-Agent System for Network Protocol Fuzzing
Original Authors: Youssef Maklad, Fares Wael, Ali Hamdi, Wael Elsersy, Khaled Shaban
Affiliations:
-
MSA University, Giza, Egypt (first four authors)
-
Qatar University, Doha, Qatar (Khaled Shaban)
Keywords: Protocol Fuzzing, Cybersecurity, Finite State Machine, Reverse Engineering, Large Language Models, Multi-Agent Systems, Dense Retrieval, Retrieval-Augmented Generation, Chain-of-Thought
Original Link: https://arxiv.org/abs/2508.14300
Open Source Code: Not available
Key Points of the Paper
Paper Summary: This paper presents an innovative network protocol fuzzing framework called MultiFuzz. It addresses the limitations of traditional AFL-based protocol fuzzing tools in semantic understanding and mutation methods by integrating recent advancements driven by LLMs (Large Language Models) to construct a dense retrieval-based multi-agent system. This system implements Retrieval-Augmented Generation (RAG) and structured tool-assisted reasoning, enhancing both the exploration of protocol state spaces and the rationality and stability of protocol mutations.
MultiFuzz is centered around modular multi-agent collaboration, where each group of agents is responsible for specific sub-tasks. It utilizes dense vector retrieval to semantically index protocol documents, enhancing the contextual adaptation and reasoning capabilities of the fuzzing process. Experiments focused on the Real-Time Streaming Protocol (RTSP) showed that MultiFuzz significantly outperformed cutting-edge tools such as NSFuzz, AFLNet, and ChatAFL across multiple dimensions, including state space exploration and branch coverage. This work not only broadens the new paradigm of automated protocol fuzzing but also provides a scalable infrastructure for future research on intelligent agent fuzzing systems.
Research Objectives: This paper focuses on the field of network protocol fuzzing, particularly the challenges faced by existing fuzzing tools when applied to complex, state-rich network protocols. Traditional methods struggle to effectively understand and utilize the complex semantics and grammatical structures of protocols, lack the capability for deep state exploration, and find it difficult to maintain effective state management in session-level multi-packet interactions. In recent years, LLMs have been introduced to assist in grammar reasoning and input generation for fuzzing, but issues such as model hallucinations, unreliable outputs, and excessive assumptions about protocol knowledge still leave blind spots in practical applications. To address these challenges, the authors propose a novel framework centered on multi-agent collaboration, retrieval-augmented generation, and tool-assisted reasoning, aiming to significantly enhance fuzzing’s ability to explore protocol state spaces, increase the grammatical compliance of mutations, reduce system failure points, and remain compatible with mainstream fuzzing ecosystems. The ultimate goal is to advance protocol fuzzing towards intelligent and autonomous directions, addressing bottlenecks in protocol awareness, deep state exploration, and stability.
Research Contributions:
-
Proposed MultiFuzz, a multi-agent protocol fuzzing system based on the ChatAFL architecture. Each group of agents focuses on specific sub-tasks, enhancing tool integration, vector database contextual awareness, and CVE-based vulnerability knowledge.
-
Introduced a novel protocol RFC document processing method based on agent chunking and embedding, achieving semantic indexing and retrieval of protocol knowledge to support multi-agent reasoning processes.
-
Integrated a dense retrieval mechanism into the agent reasoning process, enabling protocol contextual awareness and efficient fuzzing mutation guidance.
-
Conducted systematic evaluations in state protocol scenarios, demonstrating that MultiFuzz outperforms mainstream cutting-edge tools like NSFuzz, AFLNet, and ChatAFL in terms of branch coverage, state exploration, and state transitions.
-
Achieved a deep integration of protocol awareness and generative AI intelligent coordination in MultiFuzz, laying the foundation for building scalable intelligent automated fuzzing systems.
Introduction
Network protocols are at the core of modern communication systems, and their security, if compromised, poses a serious threat to the reliable operation of the entire infrastructure. Protocol fuzzing, as an important method for vulnerability discovery, has been widely applied to uncover weaknesses in protocol implementations through automated test case generation, fault injection, and anomalous interactions. With the increasing complexity and scale of network services, exploring defects in protocol implementations, especially those involving finite state machines (FSMs), has become increasingly important. Protocol fuzzing aims to systematically test protocol implementations by generating malformed, unexpected, or semi-legal protocol messages to detect abnormal system behaviors. However, traditional fuzzing methods struggle to cope with structurally complex and state-rich protocols, failing to efficiently manage deep state transitions and session-level multi-message interactions, often leading to session failures due to a lack of semantic understanding.
In recent years, academia has made several innovations in the field of protocol fuzzing, including state-aware input generation and automated protocol analysis supported by reverse engineering. Nevertheless, achieving high coverage and deep state exploration in fuzzing remains challenging, especially under undisclosed or proprietary protocols. Meanwhile, the rise of LLMs has brought new opportunities to automated software engineering and cybersecurity. LLMs, with their powerful reasoning and generation capabilities, have been applied to tasks such as protocol grammar reasoning, effective input sequence generation, and state behavior simulation, demonstrating immense potential. LLM-driven protocol fuzzing tools like ChatAFL help enhance the depth of state space exploration, but their results depend on the inherent knowledge of LLMs, which can be plagued by hallucinations and inconsistent outputs, making it difficult to fully overcome practical demand bottlenecks.
To address these challenges, this paper proposes MultiFuzz: a new architecture for protocol fuzzing that integrates dense retrieval and multi-agent collaboration. Based on ChatAFL, it combines retrieval-augmented generation with ReAct-style chain-of-thought reasoning, modularly distributing the fuzzing process across multiple agents with specific responsibilities, utilizing tool assistance and vector database retrieval to maintain and infer protocol context efficiently. The authors experimentally validate the effectiveness of the MultiFuzz framework in complex state protocol scenarios such as RTSP (Real-Time Streaming Protocol), with data showing significant improvements in branch coverage, state exploration, and state transitions compared to the current state-of-the-art (SOTA) mainstream protocol fuzzing tools. The paper concludes with a structural arrangement and outlines future research directions.
Related Work and Background
Network protocol fuzzing is one of the key technologies in the field of information security, aimed at discovering security vulnerabilities in protocol implementations through anomalous packet injection, input mutation, and state traversal. Traditional network protocol fuzzing techniques mainly include mutation-based and generation-based methods. Mutation-based methods generate new test cases by performing bit flipping, arithmetic replacement, and block-level adjustments on existing valid inputs. While easy to deploy, they struggle to fully explore the complex grammar and states of protocols. Generation-based methods, on the other hand, automatically construct test cases based on protocol specifications or templates but rely on detailed protocol knowledge, making them difficult to adapt to closed-source or poorly documented implementations. In terms of knowledge about the target system, fuzzing can be categorized into black-box (driven solely by input-output observations), white-box (requiring code-level analysis, such as symbolic execution and taint analysis to guide testing), and gray-box (using lightweight instrumentation and coverage feedback to guide mutations). In practice, gray-box methods (like AFLNet) are widely adopted due to their balance of low deployment costs and good coverage capabilities, constrained by the availability of protocol code.
In recent years, the introduction of LLMs has greatly expanded the intelligent boundaries of protocol fuzzing. LLMs, through large-scale corpus and grammar pre-training, possess the ability to abstractly understand complex protocol semantics and input reasoning. Recent studies like ChatFuzz and ChatAFL leverage LLMs to assist in grammar reasoning, protocol input construction, and state prediction, significantly improving coverage and partially overcoming the limitations of traditional fuzzing in input mutation and state exploration. MSFuzz utilizes LLMs to extract protocol abstract syntax trees, aiding in semantic-aware mutations. Meanwhile, LLMs are increasingly integrated into other automated processes, such as RPA and security scanning, showcasing their vast potential. The combination of large models and security automation is gradually forming a new technological paradigm.
Multi-agent systems have also gained attention in the field of automated security testing in recent years. A typical example is the PentestAgent framework, which decomposes the penetration testing process into several specialized agents, each performing its role and collaborating to enhance efficiency and task adaptability. In protocol fuzzing, multi-agent architectures help decompose complex processes, improving task division and flexibility in upstream and downstream collaboration, but they also introduce new challenges such as task allocation optimization, context sharing, and memory management.
Overall, previous methods for protocol fuzzing have consistently faced bottlenecks in deep state exploration and protocol context understanding. While LLM-driven new methods enhance intelligence, they remain susceptible to hallucinations and insufficient coverage of protocol knowledge. The combination of multi-agent systems, LLMs, and dense retrieval is emerging as a new direction to integrate multiple advantages and break through existing limitations, with the MultiFuzz framework representing an innovative example at this technological intersection.
Detailed Explanation of the MultiFuzz Framework and Method
The MultiFuzz framework is a next-generation protocol fuzzing platform built on the foundations of AFLNet and ChatAFL, integrating dense semantic retrieval and multi-agent collaboration. The basic idea is to preprocess protocol documents (RFCs) through intelligent filtering, chunking, propositional reconstruction, and vector embedding, establishing a contextual knowledge base that can be instantaneously retrieved by multiple agents. Each agent collaborates in different stages of fuzzing, utilizing dense retrieval to obtain highly relevant protocol knowledge, assisting in input generation and mutation decision-making, achieving structured mutations, protocol awareness, and scenario adaptability.
The first step of the entire process is to preprocess the protocol RFC documents. This paper subdivides the protocol documents into paragraphs, automatically filtering out technical materials related to the protocol using a semantic classifier, including state machine descriptions, command formats, and response specifications, retaining only the content useful for fuzzing. Subsequently, LLM-driven propositional processing converts each paragraph into independently understandable atomic propositions, forming the basis of protocol knowledge.
After propositionalization, the Agentic Chunking mechanism further utilizes LLMs to merge semantically similar propositions into several semantically consistent chunks, each generating concise titles and summaries, achieving a “theme-entity-semantic trinity” of protocol knowledge indexing. All chunks are embedded in high-dimensional vectors and stored in a Chroma database, supporting subsequent dense retrieval.
The framework’s agent system is divided into three main “crews”: the Grammar Extraction Crew, Seed Enhancement Crew, and Coverage Breakthrough Crew. The Grammar Extraction Crew is responsible for utilizing retrieval knowledge to generate structured JSON templates and formatted outputs for protocol requests, enriching the subsequent fuzzing input mutation space. The Seed Enhancement Crew inserts new protocol requests into existing test sequences based on the protocol FSM and contextual retrieval, ensuring that input sequences are compatible at the protocol grammar and state machine levels and are thoroughly explored. The Coverage Breakthrough Crew focuses on generating intelligent inputs that break state limitations and trigger new state transitions when encountering “coverage bottlenecks” during the fuzzing process, effectively avoiding prolonged stagnation in state exploration.
Each crew is equipped with a dense retrieval agent responsible for contextual retrieval, supporting collaborative reasoning and integrated contextual awareness. To achieve high modularity and automation, MultiFuzz employs LangChain and CrewAI as the foundational framework for agent and tool orchestration, supporting structured division of labor and automated process advancement.
Experimental Design and Evaluation Metrics
The experimental design of MultiFuzz fully considers the depth of protocol state machines, grammatical complexity, and fair comparisons with mainstream tools. The authors selected the RTSP protocol of the Live555 media streaming server as the test subject, known for its rich state interactions and complex session management, making it an ideal evaluation scenario for state-aware fuzzing. The experiments utilized the Llama series of large language models, assigned to each agent queue, balancing reasoning capabilities with long text windows, and enhancing the collaborative efficiency of sub-tasks through various model combinations during actual operation.
The experiments were conducted in the same hardware and operating system environment (Ubuntu 24.04.02 LTS, Intel Core i5-11300H, 16 GB RAM) and compared against three SOTA fuzzing tools: AFLNet, NSFuzz, and ChatAFL. All tools were run with default parameters for 24 hours of independent testing, with three repeated experiments to ensure fairness and statistical significance of the results. Testing metrics included the number of unique crashes in protocol implementations, state coverage, branch coverage, and total paths, measuring the efficiency and depth of the fuzzing system in actual protocol vulnerability discovery across multiple dimensions.
To facilitate reproducibility and data management, the evaluation platform was built on the ProFuzzBench automated benchmarking environment, running in Docker containers and strictly controlling consistency in experimental conditions and testing periods. All raw data were statistically analyzed for mean values after multiple rounds of experiments to eliminate the influence of randomness and testing noise.
The evaluation focused on three core metrics:
-
Branch Coverage: The total number of unique conditional branches reached during testing, measuring testing depth and vulnerability discovery probability.
-
Number of States: The number of protocol finite state machine states explored, reflecting the breadth of coverage of the protocol behavior space.
-
Number of State Transitions: The number of effective protocol state transitions triggered, measuring the exploration capability of the dynamic changes in the protocol state machine.
This multi-dimensional design ensures that the horizontal comparison with existing optimal protocol fuzzing tools is scientific, reliable, and representative.
Experimental Results and Analysis
The experimental results fully validate the significant improvements of MultiFuzz across key metrics in protocol fuzzing. First, in terms of branch coverage, MultiFuzz achieved an average coverage of 2940 branches, compared to ChatAFL’s 2912.67, AFLNet’s 2860, and NSFuzz’s 2807, representing improvements of approximately 0.9%, 2.8%, and 4.7%, respectively. Although these percentage changes may seem limited, considering that each new branch in protocol fuzzing is a potential breakthrough point for vulnerability discovery, this cumulative improvement is significant. Additionally, MultiFuzz’s branch coverage performance is very stable, with a small coverage range and limited fluctuations between repeated experiments, outperforming the high volatility of AFLNet and NSFuzz, as well as the occasional extreme values of ChatAFL, further confirming the contribution of the structured framework and multi-agent collaboration to result stability.
In terms of deep exploration of the protocol state machine, MultiFuzz also performed excellently. The average number of state transitions across three rounds of experiments was the highest at 163.33, significantly leading ChatAFL (159.67), AFLNet (84.0), and NSFuzz (90.33). The advantages over AFLNet and NSFuzz are particularly pronounced, with improvements of 94.4% and 80.8%, respectively, highlighting the capability of dense retrieval and multi-agent reasoning to deeply probe protocol state transitions and break through local extrema.
Further analysis of the successfully covered protocol FSM states shows that MultiFuzz explored an average of 14.67 states, higher than ChatAFL (14.33), AFLNet (10.0), and NSFuzz (11.7). Compared to ChatAFL, MultiFuzz improved by 2.4%, while the increases against AFLNet and NSFuzz in complex state protocols were even more pronounced, reaching 46.7% and 25.4%, respectively. These data not only indicate that MultiFuzz more intelligently uncovers the boundaries of protocol behavior, enhancing the breadth and depth of fuzzing, but also reflect that the multi-agent and dense retrieval mechanism achieves efficient knowledge collaboration, state navigation, and systematic strategy deployment.
Comparative analysis further reveals that traditional gray-box fuzzing (like AFLNet), while stable in operation, lacks the ability to understand protocol semantics and construct contextually appropriate inputs, easily falling into local optima at the state level. LLM-driven single-agent methods (like ChatAFL), although improving testing efficiency in grammar and sequence reasoning, still face limitations in structured context maintenance, state transition localization, and breaking through “coverage bottlenecks” due to model hallucinations and unstable outputs. MultiFuzz, through multi-agent division of labor, contextually dense retrieval, and tool assistance, optimizes the protocol compliance and contextual relevance of input generation, making the testing process more automated and exploratory, while effectively avoiding blind mutations and redundant path explorations.
Through systematic testing of multi-dimensional performance in protocol state machines, branch spaces, and state transitions, MultiFuzz demonstrates cutting-edge technical strength with significant promotional value, representing an important achievement in the future intelligent automation breakthroughs in the field of protocol fuzzing.
Conclusion of the Paper
This paper addresses the deep intelligence challenges of protocol fuzzing by proposing MultiFuzz, an innovative architecture based on dense retrieval and multi-agent collaboration. Through intelligent RFC knowledge extraction, semantic chunking, contextual awareness, and structured reasoning, MultiFuzz effectively compensates for the shortcomings of traditional fuzzing and single-agent LLM-driven methods in protocol intelligent understanding, deep multi-state exploration, and session context maintenance. Experimental results demonstrate that MultiFuzz surpasses currently mainstream optimal tools in core metrics such as protocol branch coverage, state transitions, and deep state exploration, exhibiting higher stability and scalability. More importantly, the multi-agent and retrieval-augmented generation paradigm proposed in this paper promotes the transition of protocol fuzzing from feedback-driven to knowledge-driven, from fragmented reasoning to structured intelligent collaboration, providing a theoretical and engineering foundation for future protocol reverse engineering, automated vulnerability discovery, and AI security testing system construction.
The paper also notes that MultiFuzz can further integrate automated reverse analysis, symbolic resolution, and traffic fingerprint recognition to achieve more refined full-process automation. Additionally, guiding intelligent agents’ fine-tuning with domain data is expected to continuously enhance reasoning and adaptability in proprietary protocols and complex custom scenarios. In summary, MultiFuzz not only validates the breakthrough potential of multi-agent, dense retrieval, and generative AI in practical protocol fuzzing but also provides new ideas and foundational frameworks for the future development of intelligent security testing systems.
-End-
