Active and retired U.S. military officers have warned that adversaries may exploit vulnerabilities in AI chatbots through “prompt injection attacks” to steal files, distort public opinion, or manipulate decision-making. Large language models are unable to distinguish between malicious and trustworthy commands, making them susceptible to injections that can steal data or disrupt systems. For instance, attackers can hide commands within text, PDFs, or web pages, tricking chatbots into executing operations such as deleting records or making biased decisions. Microsoft’s annual Digital Defense Report has highlighted that AI systems have become high-value targets for attacks, with adversaries intensifying their use of techniques like prompt injection. OpenAI’s Chief Information Security Officer has also acknowledged that this issue remains an unresolved security challenge.
To mitigate risks, Microsoft’s security team is continuously attempting to breach Copilot to discover vulnerabilities and block users who exploit these vulnerabilities. The Army has deployed the “Ask Sage” tool to restrict AI model access to sensitive data, isolating user prompts from external data sources to prevent data leaks. Experts recommend that organizations limit AI assistants’ access to sensitive data and enhance the deployment of cybersecurity AI to counteract AI-driven rapid attacks. For example, the Army uses this tool to isolate data, akin to “blocking internal spies.” Additionally, the military collaborates with critical infrastructure sectors like power to simulate defenses against AI-driven cyberattacks, emphasizing the need for more accessible and affordable cybersecurity AI tools to counter attacks that outpace human response.