❝
Hello everyone, this is “Translate Data”, a public account focused on sharing cutting-edge data technologies and AI from around the world.
❞
Recently, Meta (formerly Facebook) shared on the scale channel in the field of data infrastructure: how to utilize AI Agents to address data access and security challenges in its massive data warehouse.
With the explosive growth of data volume and the proliferation of AI applications, ensuring data security while allowing engineers to efficiently access the required data has become a common challenge faced by all large technology companies. The Meta engineering team shared their solution, revealing the AI approach to next-generation data management.
The Increasingly Severe Challenge: The “Impossible Triangle” of Scale, Efficiency, and Security
Meta’s data warehouse supports numerous business scenarios ranging from business analytics to machine learning and cutting-edge AI research. At such a large scale, data access management becomes exceptionally complex.
Traditional management methods rely on hierarchical resource structures and role-based access control, as shown in the figure below. This system has deeply optimized the interaction between humans and services but has also brought a significant management burden.
With the rise of AI applications, data access patterns have become increasingly complex, and the demand for cross-domain access has grown. Engineers spend a significant amount of time obtaining data access permissions, while data owners are equally overwhelmed with approvals and management.
How to break the “impossible triangle” of scale, efficiency, and security has become a challenge that the Meta data infrastructure team must overcome. They believe that AI is not only the source of the challenge but also the key to solving the problem.
Meta’s Answer: A Collaborative Multi-Agent System
The core of Meta’s solution is an innovative multi-agent system, primarily composed of two types of agents:Data-User Agents and Data-Owner Agents. They form an efficient collaborative loop with data users, data owners, and the data warehouse.
-
Data-User Agents: These agents serve as valuable assistants to engineers, aiming to help them access data more smoothly. It consists of three specialized sub-agents:
- Alternative Suggestion Agent: When users access restricted data, it can intelligently recommend alternative datasets that are unrestricted or have fewer restrictions, and even help users rewrite queries to bypass sensitive data.
- Low-Risk Data Exploration Agent: During the data exploration phase, users typically only need a small sample of data. This agent can provide context-aware, task-relevant low-risk data access permissions.
- Permission Request and Negotiation Agent: This agent can help users draft more standardized permission requests and negotiate directly with data owner agents, greatly simplifying the application process.
Data-Owner Agents: These agents aim to free data owners, allowing them to manage data permissions more efficiently. It mainly includes two sub-agents:
- Security Operations Agent: Like a junior engineer, this agent can automatically handle most permission requests based on the standard operating procedures (SOP) preset by the data owner.
- Access Rules Configuration Agent: This agent can proactively configure access rules for the team, intelligently automating the traditional manual “role mining” process.
These two types of agents collaborate to form an efficient and intelligent data access management closed loop.
In-Depth Analysis: Implementation of “Partial Data Preview”
To give everyone a more concrete understanding of how this system operates, Meta shared a typical use case called “Partial Data Preview”. This feature collaborates four key capabilities through an agent-driven workflow:Context-Driven, Granular Access Control, Data Access Budget, and Rule-Based Risk Management.
-
Context Driven: By analyzing the user’s recent activities (such as code commits, tasks, documents, etc.), it accurately understands their business needs. The system can utilize automatic, static, and dynamic methods to load and narrow the context.
-
Granular Access Control: Analyzing the query itself at a very fine granularity, such as determining whether it is an aggregate query or random sampling.
-
Data Access Budget: Setting a daily refreshing data access “budget” for each employee as the first line of defense.
-
Rule-Based Risk Management: Serving as a “guardrail” for AI agents, it controls risks by analyzing rules to prevent misjudgments or attacks on AI agents.
The entire system architecture leverages the language and reasoning capabilities of large language models (LLMs) to understand complex business needs while ensuring the system’s security and stability through strict rules and guardrails. Below is the complete processing flow for a “Partial Data Preview” request:
All decisions and logs will be securely recorded for future audits and analysis.
Future Outlook: The Path to an Agent-Ready Data Warehouse
Meta admits that they still have a long way to go to achieve a fully “Agent-Ready” data warehouse. Future work will focus on:
- Collaboration Between Agents: How to efficiently and securely support use cases where agents directly access data on behalf of users.
- Evolution of Tools and Platforms: Existing data warehouses and tools are designed for humans; how to make them better suited for agent use.
- Evaluation and Benchmarking: Continuously developing evaluation and benchmarking systems to ensure the system remains on the right track.
“Translate Data” believes that Meta’s work not only demonstrates the immense potential of AI agents in data management but also provides valuable insights for other companies facing similar challenges. Transforming AI from a data user to a participant and enabler in data governance may be the future of data infrastructure.
We hope today’s sharing inspires you. Please follow “Translate Data”, and I will continue to bring you the latest interpretations of global cutting-edge data technologies.