2025
Agentic AI Security
RAJAN IYER
hashtag#AISecurity Learning Alert
Join me and my friend David Linthicum for the next 10 business days as we
review the OWASP Top 10 critical vulnerabilities for LLMs and provide practical suggestions
for mitigation of these vulnerabilities
In today's episode, Dave will walk you through the Key Security Challenges in Agentic AI
Solutions
A new category of “security related” challenges and considerations has emerged, with the
adoption of GenAI and AI Agents these are:
Hallucinations
Prompt Injections and Jailbreaking
Privacy Protection
Agentic Access Control
Let’s review:
Hallucinations: “Hallucination” in the context of Large Language Models (LLMs) refers to
instances when the AI generates responses or content that appear coherent and plausible
but are factually incorrect, fabricated, or misaligned with the provided context or prompt.
In Retrieval-Augmented Generation (RAG) applications, hallucinations occur when the AI
generates false or misleading information that diverges from the retrieved knowledge base.
These hallucinations can lead to inaccurate or harmful outcomes, such as providing incorrect
answers, undermining user trust, or spreading misinformation. In the field of AI Agents,
hallucinations can manifest as erroneous interpretations, flawed plans, or incorrect
reasoning. These hallucinations can drive wrong or catastrophic decisions, especially when
the agent operates autonomously in high-stakes environments, such as healthcare, finance,
or critical infrastructure. Managing and minimizing hallucinations is essential to ensure
safety, reliability, and ethical AI deployment.
Prompt Injections and Jailbreaking: In Large Language Models (LLMs) and AI Agents, the
prompt is a critical input that guides the model’s behavior and output. Unlike internal
components of a solution, such as pre-trained parameters or system-level safeguards, the
prompt is externally influenced by the environment, including user input or interactions. This
external nature makes prompts inherently dynamic and susceptible to manipulation,
especially in open-ended or interactive systems where user inputs are integral to the
workflow. Example: An attacker injecting commands like “Ignore the above instructions”
could access restricted information or alter agent behavior.
02/03/2025
Privacy Protection: AI agents often have access and process sensitive personal and
organizational data. Without stringent privacy measures, this data could be exposed to
unauthorized access or misuse.
Agentic Access Control: As agents interact autonomously with multiple systems, ensuring
1
they only access authorized resources and actions becomes critical.
cc: Brandi Boatner
02/03/2025
2
Day 2--hashtag#AgenticAI Security Learning
Working with my colleague David Linthicum, we’re going to review the OWASP Top 10 critical
vulnerabilities for LLMs and provide practical suggestions for mitigation of these vulnerabilities
What has the Open Web Application Security Project identified as the ten critical vulnerabilities for
LLMs?
OWASP’s 10 Critical Risks for AI Agentic Solutions
The Open Web Application Security Project (OWASP) has identified ten critical vulnerabilities for
Large Language Model (LLM) solutions. These guidelines form the foundation for creating secure
agentic AI frameworks:
1. Prompt Injection:
What is the risk? Manipulated prompts lead to unauthorized actions.
What is the Mitigation? Input sanitization and monitoring for anomalous patterns.
2. Insecure Output Handling
What is the risk? AI outputs could contain malicious code or sensitive data.
What is the mitigation? Output sanitization and context validation.
3. Training Data Poisoning
What is the risk? Maliciously altered training data with embedded vulnerabilities
What is the mitigation? Data provenance tracking and anomaly detection.
[Link] of Service (DoS)
What is the risk? Overloaded systems through resource-intensive queries.
What is the mitigation? Rate limiting and computational complexity caps.
5. Supply Chain Vulnerabilities
What is the risk? Risks can come from compromised third-party plugins or dependencies.
What is the mitigation? The use of SBOM (Software Bill of Materials) tracking and dependency
vetting.
02/03/2025
6. Sensitive Information Disclosure
What is the risk? Exposing private or sensitive data in outputs.
3
What is the mitigation? Anonymization and access-level validation.
7. Insecure Plugin Design
What is the risk? Plugins with excessive privileges causing system vulnerabilities.
What is the mitigation? Granular permission enforcement and sandboxing.
8. Excessive Agency
What is the risk? Agents performing unauthorized or overly autonomous actions.
What is the mitigation? Guardrails, human oversight, and output tagging.
9. Overreliance on LLMs
What is the risk? Trusting unverified outputs, leading to potential misinformation.
What is the mitigation? External validation and hallucination mitigation systems.
10. Model Theft
What is the risk? Stolen model weights or architectures.
What is the mitigation? Encryption and watermarking for intellectual property protection.
02/03/2025
4
Day 3--Agentic AI Security Learning
Join David Linthicum and I for the next several business days to discuss securing hashtag#AgenticAI
We’re going to review the OWASP Top 10 critical vulnerabilities for LLMs and provide practical
suggestions for mitigation of these vulnerabilities
Today’s topic is Security around Input Validation
What is it?
In agentic AI, "input validation" refers to the process of checking and verifying the data received by
an AI agent before it is used to make decisions, ensuring that the input is accurate, relevant, and
within expected parameters, preventing potential errors or malicious actions by filtering out invalid
or harmful information.
Examples of input validation in agentic AI:
Checking data format: Verifying that a user input is in the correct format (e.g., date, number,
text).
Range validation: Ensuring that numerical values fall within an acceptable range.
Pattern matching: Using regular expressions to check if an input matches a predefined pattern.
Content filtering: Removing potentially harmful or inappropriate content from user inputs.
Pragmatic Steps to Take
Here are 4 ways to ensure strict input filtering:
Whitelisting, Blacklisting, Data type validation, length limitations
Whitelisting: Only allow specific, predefined formats and patterns of input to be processed by the
AI.
Blacklisting: Actively identify and block known malicious patterns or keywords.
Data type validation: Ensure inputs are of the correct data type (e.g., numbers, text, specific
formats).
Length limitations: Set maximum character or token limits to prevent excessive input.
02/03/2025
hashtag#cloud hashtag#cloudsecurity hashtag#cloudai hashtag#aisecurity
5
Day 4--hashtag#AgenticAI Security Learning--
Join me and David Linthicum as we take you on an hashtag#AgenticAI Security Learning journey
offering not only definitions but pragmatic advice
Today, for Day 4 of that journey, we're talking--Prompt Engineering Security for Agentic AI
What is it?
The practice of directing generative artificial intelligence (generative AI) systems to produce desired
results is known as prompt engineering. Though generative AI aims to emulate human behavior,
producing meaningful, high-quality output necessitates precise instructions. Prompt engineering
skills include selecting the most suitable formats, expressions, words, and symbols. You help the AI
engage with your users in a more meaningful way through prompt engineering services. To generate
a library of input texts such that an application’s generative AI functions as intended, prompt
developers combine creativity with trial & error.
What should you be thinking about in relation to prompt engineering and security? There are 3
things: Contextual Awareness, Safety prompts and Sanitization
Contextual awareness: Design prompts that clearly define the expected response and scope of
the AI's operation.
Safety prompts: Incorporate safety guidelines and restrictions into the prompts to prevent
harmful outputs.
Sanitization: Cleanse input prompts to remove potentially malicious elements before processing.
What are the Benefits of Prompt Engineering in Security?
There are 4 main benefits to prompt engineering in security
Investigating and comprehending emerging technology and risks--Security experts may stay up to
date on the most recent advancements and threats in the industry by using rapid engineering to
make use of AI technologies for information gathering and analysis and insight generation.
Prompt engineering is helpful in recognizing and identifying harmful code—Well done Prompt
engineering for AI allows for speed. The ability to scan, parse, and understand code, find flaws, and
highlight possible exploits or malware faster is key
Creating solutions and countermeasures--Prompt engineering techniques can be used by security
experts to give AI tools instructions on how to develop and test defensive tactics like firewalls,
02/03/2025
patches, encryption, and authentication.
Conveying and summarizing the conclusions and findings--To provide brief and understandable
6
reports, presentations, or suggestions based on their data and research, security professionals might
employ quick engineering to request AI tools.
02/03/2025
7
Day 5--hashtag#AgenticAI Security Learning
Join David Linthicum and me as we take you on an AgenticAI Security learning journey. We're going
to review the OWASP Top 10 critical vulnerabilities for LLMs and provide practical suggestions for
mitigation of these vulnerabilities
Today Dave is covering Insecure Output Handling
What is it?
Security and Output Handling for Agentic AI refers to the practices and mechanisms implemented to
ensure the safety and reliability of actions taken by autonomous AI agents, including rigorous
validation of their outputs, strict access controls, monitoring their activities, and implementing
safeguards to prevent unintended consequences, all while maintaining transparency in their
decision-making process; essentially, it's about protecting sensitive data, preventing malicious
actions, and ensuring the outputs produced by an Agentic AI system are accurate, ethical, and
aligned with intended use cases.
What are 4 ways to check output?
Fact-checking: Comparing generated outputs against known facts and established knowledge
bases to verify accuracy.
Logic checks: Implementing rules to identify illogical or inconsistent outputs based on the
context.
Bias detection: Using algorithms to identify potential biases in generated outputs and take
corrective actions.
Content filtering: Removing inappropriate or offensive content from generated outputs
What are 3 Guardrails and Safety Mechanisms that can be put in place?
Alerting systems: Setting up triggers to notify operators when outputs deviate significantly from
expected norms.
Human-in-the-loop: Integrating human review and approval processes for critical decisions
made by the AI agent.
Blacklist filtering: Blocking outputs that match predefined patterns of undesirable content
02/03/2025
8
Day 6--Securing hashtag#AgenticAI learning alert
Join me and David Linthicum on an journey to secure agentic AI
On Day 6, I'll explore Monitoring and logging
What do we mean by monitoring and logging in agentic AI?
In agentic AI, "monitoring and logging" refers to the continuous tracking and recording of an AI
agent's actions, decisions, and interactions with its environment, allowing developers to understand
how the agent is performing, identify potential issues, and ensure its behavior aligns with intended
goals, often by recording detailed information about the agent's decision-making process within
system logs.
There are different types of data that get logged depending on the kind of agentic AI system being
used. What type of data is logged?
Agent actions--Every action taken by the agent, including the context in which it was taken.
Decision-making process--The reasoning behind each decision made by the agent, including the
factors considered and the calculated probabilities.
Environmental data--Information about the agent's surroundings that influenced its decision-
making.
Performance metrics--Key indicators like success rate, response time, and accuracy.
What are a few example scenarios?
Customer service chatbot--Logging the questions asked by customers, the agent's responses, and
any escalations to human agents to identify areas for improvement.
Autonomous vehicle--Recording the sensor data, steering commands, and braking actions of a
self-driving car to analyze driving behavior and identify potential safety hazards.
Cybersecurity agent--Monitoring network traffic, user activity, and system logs to detect suspicious
behavior and potential threats
What are mechanisms that organizations can put in place to monitor what is being logged?
Input tracking: Log all inputs received by the AI system, including user details and timestamps.
Output analysis: Monitor AI responses for unusual patterns, unexpected behavior, or potential
security risks.
Anomaly detection: Use machine learning techniques to identify deviations from normal
behaviour.
02/03/2025
9
Day 7--Securing hashtag#AgenticAI Learning
Join David Linthicum and I for Day 7 of Securing hashtag#AgenticAI learning as Dave digs into Denial
of Service in Agentic AI
What exactly is Denial of Service in Agentic AI?
In the context of Agentic AI, "denial of service" refers to a malicious attack where an attacker floods an
AI agent with excessive requests or queries, designed to overwhelm its computational resources and
effectively prevent it from performing its intended functions, essentially causing the system to become
unusable for legitimate users by overloading it with traffic
What are some key points to consider about denial of service in an Agentic AI environment?
Key points about denial of service in Agentic AI include resource exhaustion, malicious intent, and
vulnerability of large language models. Let’s be specific.
Resource exhaustion--The primary mechanism is to send a large volume of complex or
computationally intensive requests to the AI agent, causing its processing power, memory, or network
bandwidth to become saturated, preventing it from responding to other requests effectively.
Malicious intent--Unlike accidental overload, a denial of service attack is deliberately designed to
disrupt the system and prevent legitimate users from accessing the AI agent's services.
Vulnerability of large language models--Large language models (LLMs) commonly used in Agentic
AI systems are particularly susceptible to denial of service attacks due to their high computational
demands when processing complex queries.
How can Denial of Service be mitigated in an Agentic AI environment?
There are a few ways to mitigate denial of service in an Agentic AI environment.
Rate limiting--Implementing mechanisms to restrict the number of requests an agent can receive
within a specific timeframe, preventing rapid-fire queries from overwhelming the system.
Input validation--Filtering out malicious or poorly formatted inputs that could trigger excessive
processing load on the agent.
Data anonymization--Removing identifiable information from training data before using it to train
the AI model.
Data filtering--Implementing mechanisms to filter out sensitive information from model outputs.
Access controls--Limiting access to sensitive data within the AI system.
02/03/2025
Prompt engineering--Carefully crafting prompts to guide the AI model towards generating
responses that avoid sensitive information.
10
Regular monitoring and auditing--Continuously reviewing the model's outputs to identify
potential sensitive information leaks
02/03/2025
11
Day 8--hashtag#AgenticAI Security Learning Alert
Working with my colleague David Linthicum, we’re going to review the OWASP Top 10 critical
vulnerabilities for LLMs and provide practical suggestions for mitigation of these vulnerabilities
Today we're serving up Supply Chain Vulnerabilities in Agentic AI
What do we mean when we talk about Supply Chain Vulnerabilities in Agentic AI?
In agentic AI, "supply chain vulnerabilities" refer to security risks that arise when third-party
components, services, or data used to train or operate the AI agents are insecure, potentially allowing
attackers to exploit weaknesses within the broader system through these external dependencies,
leading to data breaches, compromised decision-making, or disruptions in the AI agent's functionality
When we talk about Supply Chain Vulnerabilities in Agentic AI, we’re really talking about 6 key areas:
Third-party libraries and frameworks--Relying on external libraries or frameworks with known
vulnerabilities can introduce security risks into the AI agent.
Data poisoning--Malicious actors injecting tainted data into the training dataset used to train the
AI agent, causing it to produce inaccurate or biased outputs.
Insecure plugin design--If an AI agent uses plugins or extensions from third parties, vulnerabilities
within their design could be exploited by attackers.
Unverified data sources--If the AI agent pulls data from external sources without proper validation,
it could be susceptible to malicious data manipulation.
Lack of provenance tracking--Not being able to trace the origin and modifications of data used in
the AI pipeline can make it difficult to identify potential vulnerabilities.
Impact chain and blast radius--A security breach in one AI agent within a connected system could
cascade and affect other agents, causing widespread disruption.
What are a few pragmatic and proactive ways to mitigate supply chain vulnerabilities in agentic AI?
We think that these 6 recommendations will help you secure your agentic AI supply chain
environment
Strict vendor vetting--Carefully evaluate third-party providers and their security practices before
incorporating their components into the AI system.
Data validation and cleansing--Thoroughly clean and validate data used for training to mitigate
data poisoning attacks.
02/03/2025
Regular security audits--Conduct periodic security assessments of the AI system and its
dependencies to identify potential vulnerabilities.
12
Secure coding practices--Implement secure coding standards when developing custom
components for the AI agent.
Access controls--Limit access to sensitive data and components within the AI system to authorized
users.
Transparency in data lineage--Maintain clear records of data provenance to identify potential
issues in the supply chain.
02/03/2025
13
Day 9--Securing hashtag#AgenticAI Learning Alert
On Day 9, David Linthicum walks you through Sensitive Information Disclosure in Agentic AI
What do we mean when we talk about Sensitive Information Disclosure in Agentic AI?
Sensitive Information Disclosure in the context of agentic AI refers to the unintentional leaking of
confidential or private data, like personal information, proprietary algorithms, or other sensitive
details, through the outputs generated by an AI agent, potentially exposing this information to
unauthorized parties due to flaws in the model's design or training data.
Key points about Sensitive Information Disclosure in Agentic AI:
Unintentional exposure--The AI system may inadvertently reveal sensitive information in its
responses without any malicious intent, simply because it has access to a vast amount of data that
could include private details.
This can lead to significant privacy violations, especially when dealing with personal identifiable
information (PII) like names, addresses, or medical records.
There is an impact associated with training data. The risk arises from both the input data used to
train the AI model and the way the model processes and generates outputs based on that data.
So what are the potential consequences of Sensitive Information Disclosure in Agentic Security?
Sensitive Information Disclosure can result in reputational damage, legal issues, competitive
disadvantage, and financial losses for the organization using the AI system.
Examples of Sensitive Information Disclosure in Agentic AI:
Chatbot revealing user details--A customer service chatbot inadvertently disclosing a user's
personal information in its response to a query.
AI assistant generating sensitive content--A virtual assistant generating text that includes
confidential business data or private medical records.
Model output with identifiable information--An AI-powered text generation tool producing text
that contains personal details from its training data.
How do practically mitigate Sensitive Information Disclosure in Agentic AI?
There are 5 key actions to take:
Data anonymization--Removing identifiable information from training data before using it to
train the AI model.
Data filtering--Implementing mechanisms to filter out sensitive information from model
outputs.
Access controls--Limiting access to sensitive data within the AI system.
02/03/2025
Prompt engineering--Carefully crafting prompts to guide the AI model towards generating
responses that avoid sensitive information.
14
Regular monitoring and auditing--Continuously reviewing the model's outputs to identify
potential sensitive information leaks.
hashtag#aisecurity hashtag#cyberai
02/03/2025
15
Day 10--Securing hashtag#AgenticAI Learning
Working with my colleague David Linthicum, we’re going to review the OWASP Top 10 critical
vulnerabilities for LLMs and provide practical suggestions for mitigation of these vulnerabilities
What is Excessive Agency in the context of Agentic AI?
Excessive agency in the context of agentic AI refers to a situation where an AI system, designed to
act autonomously, is given too much power or freedom, leading to unintended actions or behaviors
that go beyond its intended purpose, potentially causing harm or disruption; essentially, it's when
an AI starts making decisions or taking actions without proper oversight or limitations, exceeding its
designed scope of operation.
What are some key things to consider about excessive agency in Agentic AI?
There are 3 Key points about excessive agency:
Potential risks--An AI with excessive agency could make unauthorized changes to systems, share
sensitive information, or perform actions that could have negative consequences without human
intervention.
Over-reliance on autonomy--When an AI is given too much autonomy without robust safeguards,
it can lead to situations where the AI interprets its goals too broadly, resulting in unexpected
outcomes.
Importance of clear boundaries--To mitigate excessive agency, it's crucial to clearly define the
parameters of an AI's decision-making capabilities and implement mechanisms to prevent
overstepping those boundaries.
What are some example scenarios of excessive agency?
While not an exhaustive list, here are 3 ways examples of excessive agency:
A virtual assistant making unauthorized financial transactions based on a misinterpreted
command
A self-driving car taking unnecessary risks to reach a destination quickly due to a
misinterpretation of "efficiency"
A chatbot engaging in harmful conversations or generating biased content without proper
filtering
Want to learn how to mitigate excessive agency in Agentic AI? Watch the video for 7 timely tips
02/03/2025
16
Day 11--Securing Agentic AI Learning
Join me and David Linthicum as we taking you on a journey to Secure hashtag#AgenticAI
Today Dave covers Insecure Plugin Design in Agentic AI
What is Insecure Plugin Design in Agentic AI?
In the context of agentic AI, insecure plugin design refers to flaws in the development of software
extensions (plugins) that can be used by AI agents, potentially allowing malicious actors to exploit
vulnerabilities within the plugin system, leading to unauthorized access, data breaches, or
manipulation of the agent's behavior due to poorly designed input validation, excessive permissions,
or lack of security controls.
What are some key considerations in this area?
There are 4 Key points about insecure plugin design in agentic AI.
1⃣ Vulnerability through natural language--Because plugins often interact with AI agents using
natural language, malicious actors can easily craft deceptive inputs to exploit vulnerabilities more
readily than in traditional software.
2️⃣ Excessive access control--If a plugin is granted too much access to system resources or sensitive
data without proper restrictions, it can be used to perform unauthorized actions.
3⃣ Lack of input validation--Not adequately validating user input within a plugin can allow malicious
code injection or other harmful commands to be executed.
4⃣ Uncontrolled plugin execution--If the agent does not have sufficient control over when and how
plugins are executed, it can be susceptible to unexpected or unintended behavior.
What are a couple of examples of insecure plugin design?
A web scraping plugin that allows users to input any website URL without validation, potentially
enabling the agent to scrape sensitive data from unauthorized websites .
A system command execution plugin that allows raw shell commands to be entered, potentially
enabling malicious code execution on the host system .
What are a few practical tips to mitigate insecure plugin design?
There are six things to do today
Strict input validation--Implement robust validation mechanisms to filter out potentially
harmful inputs.
Least privilege principle--Grant plugins only the minimum permissions necessary to perform
their intended functions.
Sandboxing--Isolate plugins from the system to prevent unauthorized access to sensitive data.
02/03/2025
Whitelisting--Only allow trusted plugins to be used by the agent.
Regular security audits--Conduct periodic reviews of plugins to identify and address potential
17
vulnerabilities.
02/03/2025
18
