AI penetration testing for tech companies: what to test

AI penetration testing for a tech company should target the whole system in production: the model, the prompts, the retrieval pipeline, the tools, the permissions, the APIs and the runtime controls around them. The goal is not a one-off report. It is finding the paths that change real product risk, then turning those findings into release gates and retests.

Key takeaways

• Testing only the model misses the highest-risk paths, especially when an assistant can read data, call tools or trigger actions.
• Good scoping starts with crown-jewel workflows, then maps what the AI can read, write, disclose and execute.
• Automation gives breadth and regression coverage, but human testers are still needed to prove exploit chains and business impact.
• AI testing belongs in CI/CD once you define risk-based pass/fail gates, owners and a retest cadence.

AI pentesting must target the whole AI system

Why model-only testing misses business risk

The model is the wrong unit of testing. Most of the damage appears when a model sits inside a product that retrieves data, formats output and executes tools on behalf of a user. The OWASP DevSecOps Guideline frames it well: AI application security testing treats the whole system (model, prompts, retrieval pipeline, tools and the permissions behind them) as the attack surface, and probes it repeatedly the way an adversary would.

That matters for a support bot, a sales copilot or an internal assistant. If the bot can see CRM records, search a knowledge base and open tickets, a successful attack is not "bad text". It is unauthorised access, unwanted side effects or disclosure through a legitimate workflow.

The real attack surface in production

The practical surfaces are prompt injection, sensitive information disclosure, improper output handling, excessive agency and retrieval-layer weaknesses, all named in the OWASP Top 10 for LLM Applications. Plan around the system's failure modes, not around a generic checklist.

Take an AI support bot connected to a helpdesk and a CRM. A jailbroken model is one risk. A bot that can also read customer notes, update account fields or expose internal case history because its service account is too broad is the business problem. Teams regularly spend weeks hardening prompt templates while the service account behind the bot keeps read access to every case.

Start with scope, threats and crown-jewel paths

Map assets before you test

• System prompts: hidden instructions, policy text and role separation.
• RAG data stores: which corpora feed answers and which ones are sensitive.
• Vector databases: embedding stores, namespaces and tenant boundaries.
• Model gateways: routing, filtering, logging and provider fallbacks.
• Agent tools: function calls, webhooks, browser actions and code execution.
• APIs and secrets: tokens, service accounts and secret-scoped actions.
• Identity roles: who can approve, trigger or override an AI action.
• Logging paths: what is stored, retained and visible to operators.

Strong engagements start with discovery, perimeter mapping and threat modelling before any payload is sent. MITRE ATLAS gives you a shared catalogue of adversary techniques against AI systems, while OWASP gives you the product-security lens.

Define the abuse cases that matter

Scope two or three crown-jewel journeys first: account recovery, code generation or customer-data lookup. For each one, list what the AI can read, write, trigger and disclose. That keeps the engagement tied to outcomes the business actually cares about.

If an internal agent can summarise a customer account, issue a refund and email a user, the abuse cases are not only jailbreaks. They include unauthorised disclosure, fraudulent actions and escalation through a loosely controlled workflow.

What to test in prompts, RAG, agents and tools

Prompt and output attacks

• Prompt injection: direct jailbreaks, and indirect injection through documents and tool output.
• System prompt leakage: attempts to extract hidden instructions, policies and tool schemas.
• Unsafe function execution: the model must not trigger actions outside the user's intent.
• Improper output handling: downstream systems must not trust unvalidated model output.

A common miss is the "helpful parser" problem: the model returns JSON and a backend treats it as authoritative. If that JSON can steer billing, access or notifications, prompt injection becomes a control-plane issue, not a text issue.

RAG and data-layer attacks

• Retrieval poisoning: hostile content planted to change what gets surfaced.
• Document-borne prompt injection: instructions hidden in PDFs, tickets or wiki pages.
• Overbroad chunk access: retrieval that crosses tenant, project or role boundaries.
• Vector and embedding weaknesses: namespace bleed, similarity abuse and stale indices.

One useful test is a malicious PDF in the knowledge base that tells the assistant to disclose secrets or call an external webhook. If the agent follows the document over the user, or if retrieval returns chunks the user should never see, the exploit chain is real even when the model itself looks safe.

Agent and tool misuse

• Excessive agency: the agent must not act beyond the minimum the task needs.
• Tool permission review: every function checked for scope, authorisation and auditability.
• MCP server trust: external tool servers authenticated and sandboxed.
• Exfiltration channels: email, webhooks, file writes, copy-to-chat and browser actions.

For agents, the question is not "can it answer?" but "what can it do if the inputs are malicious?". The NIST Generative AI Profile (AI 600-1) helps on the control side, while the OWASP GenAI Security Project is more specific about runtime behaviour, implementation and infrastructure. That is the right frame for a production agent stack.

Use automation for coverage, humans for proof

What automation is good at

• Repeatable evals: tools such as promptfoo, garak, Giskard and DeepTeam run the same checks on every build.
• Broad probing: large prompt sets improve coverage across jailbreak styles and policy edge cases.
• Regression detection: you see when a model upgrade or a prompt tweak reopens a known bypass.

The rule is simple: define pass/fail by risk, not by average score, and run the suite on every change that can alter behaviour. That includes model version changes, prompt edits, retrieval updates, tool permission changes and gateway rule changes. Large public benchmark sets give breadth, but they do not tell you whether your specific environment can be chained into a real exploit.

Where manual red teaming is still required

People are needed to validate exploitability and to chain weaknesses across the stack. Automation can tell you that a prompt sometimes leaks policy text. A tester can show that the leak leads to a tool call, which leads to data access, which leads to exfiltration.

That is why the OWASP GenAI Red Teaming Guide, as summarised in the OWASP DevSecOps Guideline, structures an engagement across four areas: model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis. The same logic is behind the split between automated and manual work in classic pentesting, covered in our comparison of automated and manual penetration testing.

Use automation to widen the net. Use people to prove the path.

Make AI testing part of the release pipeline

What to gate in CI/CD

• Critical prompt-injection bypasses: block the release when the model follows hostile instructions.
• Unsafe tool calls: stop the deployment if the agent can invoke forbidden actions.
• Personal data leakage: fail the build when tests show protected data can be exposed.
• Broken authorisation: block when model-connected APIs ignore user, tenant or role boundaries.

OWASP describes a maturity path in three stages: Starter, Intermediate and Advanced. A starter programme inventories the AI features and what each one can reach, runs a generic scanner against staging and strips unnecessary tool permissions. The next stage runs automated red-team evals in CI. The advanced stage combines those gates with periodic external testing whose findings become internal controls.

The last item on the gate list is where a classic pentest still earns its place. The model-connected APIs, the web app around the assistant and the external infrastructure are ordinary attack surface. Fleuret runs agentic pentests on web apps, REST and GraphQL APIs and external infrastructure, with replayable proofs of concept, findings pushed to Jira, Linear or Slack, and a retest after the fix.

The minimum metrics to track

• Attack success rate by category: jailbreak, leakage, tool abuse, exfiltration.
• Eval coverage: which workflows, roles and tool paths are exercised.
• Time to remediate bypasses: measured in days, not in meetings.
• Agent permission footprint: what the system can read or do.
• Regression rate: how often old failures come back after a change.

If those numbers are not visible to engineering, the pentest becomes an annual artifact. If they are, it becomes a release control.

Reports should drive fixes, not just satisfy audits

What a useful deliverable looks like

A good report maps each finding to business impact, the affected workflow, reproduction steps, proof and a concrete fix. A prompt transcript alone is not enough. Security teams need the affected asset, the exact chain and the control that failed.

Use severity scoring where it fits, weakness naming where it helps, and a risk map against the OWASP Top 10 for LLMs. That gives engineering a stable vocabulary without pretending AI risk is identical to classic web risk.

How to prioritise findings

NIST AI RMF 1.0, NIST AI 600-1 and the EU AI Act all raise the weight of evidence, but compliance theatre is a bad target. Prioritise by exploitability, data sensitivity, user impact and whether the issue can be chained into an action. For the web and API layer, a replayable proof of concept and a dated retest are the evidence that holds up, which is also why agentic pentesting is built around them.

The best reports tell you what to fix on Monday morning, and what to retest once the fix lands.

Choose the right engagement for your stage

When a one-off pentest is enough

A one-time specialist engagement works for a pre-launch feature, a post-incident hardening sprint or a procurement deadline. It gives you a sharp external view and a fast read on the highest-risk paths. On its own it is not enough if the product changes every week.

Engagement model	Best for	Strengths	Key limits	Operational fit
Procurement-driven test	Deals that need customer evidence	Artifacts for review and sign-off	Can turn into theatre	Pair with runtime checks
One-time specialist pentest	Pre-launch or incident response	Deep manual exploit work	No built-in retest loop	Good for sharp, narrow findings
AI-assisted testing plus human validation	Teams shipping monthly or faster	Repeatable coverage and retests	Needs good scoping	Best for hybrid security teams
Internal continuous evaluation	Fast-changing agent products	Always-on regression control	Misses novel exploit chains	Pair with periodic external red teaming

If your AI can reach customer data or take actions, the default is periodic manual red teaming plus automated checks in CI. Continuous evals catch drift. People catch the strange chain nobody anticipated.

Treat AI features like any other production control

Start with one crown-jewel workflow, one human-led test and one automated regression suite. Then wire the findings into the same owners and gates you already use for authentication, payments and data access. If the AI can read, write or act on behalf of users, it deserves the same discipline as any other production control.

Fleuret runs agentic pentests on the web apps, APIs and external infrastructure behind your AI features, with results in hours and replayable proofs of concept. See Fleuret in action.

Sources

• AI/LLM Application Security Testing, OWASP DevSecOps Guideline
• OWASP Top 10 for LLM Applications, OWASP GenAI Security Project
• OWASP Top 10 for LLM Applications 2025 (PDF), OWASP, 2024
• AI Red Teaming Initiative, OWASP GenAI Security Project
• MITRE ATLAS, MITRE
• Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1), NIST, 2024