Security Capabilities and Trustworthy Governance of AI Agent Systems: The Three Lines of Defense

In the current era of accelerated evolution and widespread implementation of AI technology, Agent systems, which possess autonomous perception, reasoning, and execution capabilities, are increasingly becoming the core driver of enterprise intelligent transformation. However, the accompanying system complexity, task autonomy, and cross-domain collaboration capabilities have also triggered unprecedented security challenges and governance issues. To build an Agent system that operates efficiently while being controllable and trustworthy, it is essential to introduce systematic security protection and governance mechanisms at the architectural level. This article, based on research, discusses the security capabilities and protection methods of AI Agent systems, proposing a three-dimensional protection system of “Prediction – Defense – Detection – Response – Audit” aimed at providing a practical path and framework support for the secure and trustworthy evolution of AI Agents.

1. Security Capabilities of AI Agent Application Systems and Representative Vendors

To meet the security needs of AI Agent systems, leading security vendors are accelerating the layout of related capabilities and product systems, actively exploring new security solutions that adapt to large models and intelligent architectures. However, as large models and Agent technology are still in a rapid development phase, there are currently few traditional security vendors with deep research and productization capabilities. The overall market is still in the early stages of transitioning from exploratory validation to systematic construction, and the relevant security capability system still needs further improvement and consolidation.

From the research trends, vendors are currently focusing on the AIGC (Generative Artificial Intelligence) security governance field, engaging in technological innovation around model behavior control and content compliance management. Typical security capabilities include: AI security assessment, large model security gateways, and security fence mechanisms.

AI Security Assessment

AI Security Assessment evaluates the safety, stability, and compliance of large models and AI Agent systems in terms of input and output, reasoning processes, and abnormal behaviors, assisting enterprises in achieving controllable risks before model deployment. The core goal is to verify model quality, risk levels, and business adaptability, providing an objective basis for development, deployment, regulation, and optimization.

Application Scenarios: These include large model selection, compliance assessment, pre-deployment capability and security assessment, industry AI application evaluation, third-party AI security audit services, and open-source model benchmark comparisons.

AI security assessment, especially the evaluation of safety and trustworthiness for generative AI models, has become one of the core issues of concern in the industry. Currently, the entities involved in this field include security vendors and specialized institutions focused on AI technology research.

Some institutions not only possess self-developed assessment tools but have also launched systematic security assessment services, with representative institutions including: NSFOCUS, JunTong Future, and Shanghai Artificial Intelligence Laboratory. These institutions have conducted substantial explorations in model attack surface analysis, security benchmark testing, prompt risk identification, and output content compliance assessment, promoting the formation of preliminary industry practice standards.

(1) NSFOCUS: Large Model Security Assessment System AI-SCAN

AI-SCAN is primarily used to assess the safety of AI-generated content, identifying and preventing potential risk content, including but not limited to false information, malicious speech, privacy breaches, and copyright infringement, ensuring the safety, compliance, and reliability of AI-generated content, and avoiding legal disputes or negative social impacts caused by content risks.

• Compliance: Meets regulatory and compliance assessment requirements for large model application products and services, such as TC260-003 technical standards, large model filing, algorithm filing, etc.

• Risk Identification: Tests the model throughout its AI lifecycle, including training, deployment, and testing phases, to prevent risks early and conduct targeted defense reinforcement work.

• Model Selection: Conducts horizontal capability comparison analysis of multiple local or online models, producing evaluation analysis reports to assist clients in model selection.

(2) JunTong Future: Evaluation Verification System “JunHe, JunJian”

JunTong Future is a domestic startup focused on AI ecological governance, established in June 2024, headquartered in Hangzhou. Through the dual systems of “JunHe, JunJian” (evaluation verification) and “JunKong” (protection control), it forms a full-stack AI trustworthy governance solution from evaluation, monitoring, to protection. “JunHe” is a generative AI evaluation verification system, while “JunJian” is a decision-making AI evaluation enhancement system, used to quantitatively assess the trustworthiness, performance, and risk control capabilities of large models in actual business scenarios. Delivery methods include both products and services.

(3) Shanghai AI Laboratory: Open-source Large Model Evaluation Platform OpenCompass

OpenCompass is an open-source large model evaluation platform developed by the Shanghai AI Laboratory, also known as “SiNan”. It aims to provide one-stop evaluation services for large language models, multimodal models, and various other models, offering objective large model capability references from a purely technical and neutral perspective for academia and industry.

The platform summarizes evaluation directions into five capability dimensions: knowledge, language, understanding, reasoning, and examination, integrating over 70 evaluation datasets, providing over 400,000 model evaluation questions, and assessing three types of large model technical capabilities: long text, safety, and code. The OpenCompass platform will also publish scoring rankings for large models, including large language models, multimodal models, and models in various vertical fields, providing users with comprehensive, objective, and neutral evaluation references.

Large Model Security Gateway

The large model security gateway serves as an intermediary layer for model invocation, responsible for reviewing input prompts and output content, policy control, and risk interception, preventing issues such as prompt injection, unauthorized access, and non-compliant generation. The core goal is to protect the model system from external malicious attacks and non-compliant content. For example, conducting content safety analysis, risk identification, and policy control on model input requests and output responses to prevent sensitive information leakage, unauthorized access, and prompt injection attacks.

Applicable Scenarios: Suitable for deployment at unified entry/model API access before and after, applicable to general security scenarios such as malicious traffic, compliant access, and data leakage protection.

The security gateway consists of a set of relatively static security policies, typically running independently of the model itself, ensuring the overall security and compliance of the model invocation chain. Currently, there are several representative vendors both domestically and internationally engaged in this field.

Representative solution providers for foreign large model security gateways include: Microsoft, Prompt Security.

(1) (USA) Microsoft: Azure AI Content Safety Gateway

The Azure AI Content Safety Gateway is an enterprise-level content safety protection component for large models launched by Microsoft, serving as an important supporting capability for Azure OpenAI Service, aimed at providing enterprises and developers accessing GPT series models with input/output content safety analysis, compliance control, and risk interception capabilities.

(2) (USA) Prompt Security: Prompt Security Gateway

Prompt Security is a US security startup focused on preventing risks associated with generative AI for enterprises, established in 2023. In September 2024, it launched the Prompt Security Gateway, a security and governance solution for Microsoft 365 Copilot, with core functions including prompt attack detection and model invocation behavior analysis. Delivery methods support SaaS or on-premises deployment.

Representative solution providers for domestic large model security gateways include: ByteDance, Zhidao Chuangyu.

(3) ByteDance: Large Model Application Firewall

The Volcano Engine large model application firewall provides security protection products for large language model inference services, ensuring bidirectional privacy, safety, usability, and trustworthiness of input and output content, protecting large language models from OWASP LLM Top 10 attacks, and providing security protection against computational consumption attacks, prompt attacks, etc.

(4) Zhidao Chuangyu: Chuangyu Large Model Gateway

The Chuangyu large model gateway is a security protection gateway product launched by Zhidao Chuangyu for large model access security. The product is deployed in a proxy manner, compatible with large model interfaces that comply with OpenAI API protocols or third-party AI gateway interfaces, supporting unified access management for large models, observability of large models, and multidimensional security protection capabilities such as sensitive data leakage and content safety.

Security Fence

The security fence sets “soft and hard boundaries” for AI Agent behavior, limiting its permissions and capabilities within specific business scopes, preventing Agent from unauthorized operations, executing malicious instructions, or causing business risks. This is typically achieved through preset permission policies, behavior whitelists, or execution path rules, dynamically making decisions during the execution process of intelligent systems to prevent unauthorized behavior, data leakage, and intent deviation, serving as an important line of defense for Agent system security.

Applicable Scenarios: Compared to large model security gateways, security fences represent a more refined and dynamic policy constraint mechanism. They are suitable for scenarios involving context analysis, behavior constraints, and data access. Typically deployed within internal Agent systems/intelligent frameworks, they limit the model’s usage scope, behavior boundaries, and permission constraints during model operation or Agent execution phases.

Representative providers of foreign security fences include Google.

(1) (USA) Google: Vertex AI Guardrails

Vertex AI Guardrails is a systematic security mechanism launched by Google for enterprise-level generative AI applications, covering multi-dimensional protection from content review, tool invocation control, identity isolation to configuration monitoring. For scenarios involving the construction of complex Agent systems or business-sensitive AI applications, Guardrails can directly impose policy boundaries within the execution path. Related research shows that on the G2 platform, the content review rules and compliance detection functions of Vertex AI Guardrails have achieved approximately 90% user satisfaction.

Representative providers of domestic security fences include: China Telecom Artificial Intelligence Technology, Shumei Technology, NSFOCUS.

(2) China Telecom Artificial Intelligence Technology: AIGC Security Governance Solution

China Telecom Artificial Intelligence Technology, officially known as China Telecom Artificial Intelligence Company Security Operations Company, was established in November 2023, evolving from a subsidiary of the China Telecom Group’s Big Data and AI Center. In early 2025, the company launched the AIGC security governance solution targeting foundational large models, large model applications, and intelligent agents. This solution focuses on data, model, and content security, providing six core capabilities: AIGC training corpus processing, security protection, content labeling capabilities, security assessment capabilities, and deep forgery detection. Through a SaaS model, it supports multi-tenant, multi-scenario customized strategies, with over 30 detection engines built-in, capable of real-time interception of input/output risks.

(3) Shumei Technology: AIGC Application Security Fence

Shumei Technology, established in June 2015, is a professional provider of online business and content risk control solutions. In response to the risk challenges faced by AIGC applications, Shumei Technology has launched the AIGC application security fence solution based on its experience in content compliance and account security. The product spans the entire chain from “data – model – operation,” providing pre-release compliance support, content review during operation, and account protection risk control capabilities.

(4) NSFOCUS: Large Model Application Security Protection WAF-SLLM

NSFOCUS’s large model application security protection WAF-SLLM starts from the web application security and API protection of large models, providing scenario-based security protection capabilities covering supply chain security, data security, and operational security scenarios under large model applications, ensuring that users’ large model applications can develop safely and efficiently.

• Content Compliance: Ensures the safety calibration of model input content and that model output content meets compliance requirements, as well as monitoring content accuracy.

• Data Security: Detects sensitive data returned by the model to avoid privacy theft scenarios.

• Model Security: Conducts security detection against prompt injection, jailbreak attacks, and other inputs to enhance the security and robustness of large models.

Differences Between Evaluation, Gateway, and Fence

From the perspective of the differences between evaluation, gateway, and fence, the three have significant distinctions in terms of target positioning, functional levels, deployment methods, operational phases, and technical means, as shown in the table.

Differences Between Evaluation, Gateway, and Fence

2. The Three Lines of Defense for Trustworthy Governance of AI Agent Systems

With the widespread application of large language models and multi-agent systems across various industries, how to achieve secure, controllable, and trustworthy operation of systems has become a priority consideration for enterprises and institutions deploying AI Agent systems.

This section, based on research and investigation of AIGC security capabilities, combines the distinctions and relationships between the three major security capabilities of evaluation, gateway, and fence, and proposes the “Three Lines of Defense” for trustworthy governance of AI Agent systems.

(1) First Line of Defense: Model Evaluation – “Identifying Risks, Establishing Baselines”

AI evaluation is the starting point for trustworthy governance of Agent systems, primarily conducted before the model goes live, systematically assessing the capability boundaries and potential risks of the model itself. By combining manual testing and automated evaluation, it can be conducted from the following dimensions:

• Function Evaluation: Includes question-answer accuracy, logical reasoning ability, coherence of multi-turn dialogues, etc.

• Security Evaluation: Covers prompt injection attacks, jailbreaks, sensitive information responses, etc.

• Compliance Evaluation: Detects whether there are violations of legal regulations or ethical boundaries, such as false information, discriminatory remarks, politically sensitive content, etc.

The evaluation results can be transformed into security policy baselines, providing data support for the subsequent design of gateway and fence rules. For example, if an evaluation reveals that a certain type of prompt injection attack is effective, corresponding detection strategies should be deployed on the gateway side.

(2) Second Line of Defense: Security Protection – “Intercepting Risks, Preventing Spread”

The security gateway and security fence are two core capabilities in the AI Agent system’s security protection system, with different focuses and levels of progression. Although there is some functional overlap, such as risk control of model input and output, each has its emphasis in system positioning, control mechanisms, and design goals. In engineering, they can be integrated into one product or exist as two independent modules, depending on the vendor’s architectural design and application scenario requirements.

The security gateway serves as the external protection layer, particularly suitable for scenarios with multiple access users, open APIs, or large-scale usage, and is a core component ensuring model controllability. The large model security gateway is primarily deployed at the input-output interfaces of the Agent system, acting as a runtime security gatekeeper, responsible for content review, risk identification, and access control.

• Input Interception: Identifies dangerous prompts, malicious user inputs, unauthorized access requests, etc., to preemptively block potential attacks.

• Output Filtering: Monitors the content generated by the model in real-time to avoid leaking PII (Personally Identifiable Information), sensitive outputs, or illegal content.

• Policy Control and Log Auditing: Supports custom policy rules and records all risk behaviors for traceability and compliance auditing.

The security fence mechanism focuses on the internal operational phase of the AI Agent system, concentrating on limiting and governing model invocation behaviors and functional boundaries. It is typically deeply integrated with Agent frameworks (such as LangChain, AutoGen, or self-developed enterprise platforms).

• Function Invocation Restrictions: Regulates the model’s access capabilities to external tools/plugins/databases through role-based access control (RBAC), whitelisting, and invocation frequency limits.

• Execution Path Setting: Predefines task execution processes, limiting unexpected jumps, nesting, or unauthorized operations.

• Behavior Policy Monitoring: Tracks model behavior paths internally, preemptively intercepting sensitive operations or confirming them during execution.

(3) Third Line of Defense: Security Auditing – Compliance Traceability Assurance

In the highly autonomous and complex operational environment of AI Agent systems, security auditing is not only a traditional logging tool but also a foundational capability supporting the entire system’s trustworthy governance, security response, and compliance assurance, playing an irreplaceable role. It is responsible for recording, tracing, collecting evidence, and holding accountable for the entire process of system operation, serving as the core guarantee mechanism for achieving explainability, regulatory compliance, and traceability.

• Behavior Traceability: Records the key behavioral trajectories of the AI Agent during task execution, including perception inputs, internal reasoning, decision paths, output content, and interaction objects; supports full-process tracking of “who initiated, what was called, how reasoning occurred, and what the final result was.”

• Prompt and Response Log Retention: Audits the prompt interactions between the AI Agent and large models, which can be used to detect prompt injection, unauthorized intentions, sensitive generation risks; providing data foundations for prompt safety control and content compliance audits.

• Policy Execution Verification: Audits whether various security policies (such as access control, output filtering, security fences) are effective and whether they have been bypassed, assisting in policy optimization.

• Abnormal Behavior Identification and Traceability: Combining log analysis and behavior profiling technology, it can detect and trace back events such as “Agent unauthorized behavior,” “model output anomalies,” and “system call violations” in real-time.

• Supporting Compliance and Accountability Requirements: Meets national data security, content compliance, algorithm filing, and other regulatory requirements, establishing a clear audit responsibility chain; providing evidence for subsequent liability determination and incident handling.

The higher the level of intelligence of the AI Agent system, the greater its potential uncertainty and risks. The three lines of defense form a complete closed loop of “Prediction – Defense – Detection – Response – Audit” within the overall architecture of the AI Agent, which is essential for ensuring the stable, secure, and compliant operation of AI Agent systems. In the future, as the capabilities of large models evolve and business complexities increase, the interlocking mechanisms between the three lines of defense will become even more critical, warranting deep attention and continuous optimization from all AI system builders.

Cooperation Phone: 18311333376Cooperation WeChat: aqniu001Submission Email: [email protected]