Securing AI workflows before they touch finance and customer data

In IBM’s 2025 Cost of a Data Breach Report, 13 % of organizations reported breaches of AI models or applications, and 97 % of those organizations lacked proper AI access controls.¹ The second figure covers only the organizations that were breached, so it says nothing about how common missing controls are elsewhere. It does show what these breaches had in common. The study was run by Ponemon Institute, sponsored and analyzed by IBM, and covered 600 organizations breached between March 2024 and February 2025.¹

That matters to anyone connecting a language model to invoices, contracts, customer records or an email inbox. Such a workflow takes text from outside the company, passes it to a model and acts on what comes back, and each of those steps is an attack surface. This article uses the OWASP Top 10 for LLM Applications 2025 to name four failure modes,² maps the matching controls to NIST’s AI Risk Management Framework and its generative AI profile, and ends with a checklist you can hand to the team building the workflow.

Four failure modes worth designing against

The OWASP list for 2025 has ten entries, from LLM01 Prompt Injection to LLM10 Unbounded Consumption.² Four of them fit workflows that read documents and then act: LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM05 Improper Output Handling and LLM06 Excessive Agency. The selection is ours, based on the kind of workflow this article covers. The other six, including supply chain and unbounded consumption, deserve a review of their own.

Prompt injection (LLM01)

A prompt injection is text that changes a model’s behavior in a way its operator did not intend. In a back-office workflow that text can arrive inside an invoice PDF, a supplier email or a web page the system retrieves. OWASP notes that injected content does not need to be readable by a human to be processed by the model, and that retrieval-augmented generation and fine-tuning do not fully mitigate the problem.³ Its page on the risk is blunt about prevention:

it is unclear if there are fool-proof methods of prevention for prompt injection

The mitigations OWASP lists limit the damage instead. They are to constrain the model’s behavior, define and validate output formats, filter inputs and outputs, grant least privilege, require human approval for high-risk actions, keep untrusted content separate from instructions and test with adversarial scenarios.³

Sensitive information disclosure (LLM02)

OWASP’s examples of what can leak are personal data, financial details, health records, confidential business data, credentials and legal documents.⁴ For a finance or HR workflow that list reads like a description of the inbox. Its mitigations include sanitizing data before it reaches the model, validating inputs, applying least privilege, restricting data sources, educating users and being clear about how long data is retained.⁴ In practice, if the task needs only the supplier name and the invoice total, the bank details should never be in the prompt.

Excessive agency (LLM06)

Excessive agency is the risk that a model can take damaging actions because its output was unexpected, ambiguous or manipulated.⁵ OWASP’s remedy is to minimize the extensions, the functionality and the permissions the model has, for example read-only access to a products table when the task is only to read it, and to avoid open-ended extensions such as a shell.⁵ A model that drafts a payment approval is a text generator. One that can release the payment is an actor, and the second needs a different level of control.

Improper output handling (LLM05)

This risk is insufficient validation or sanitization of model output before it is passed to other systems. OWASP says the consequences can include cross-site scripting, cross-site request forgery, server-side request forgery, privilege escalation and remote code execution.⁶ Its guidance is to treat the model as an untrusted user: validate its output, encode it for the context where it will be used, use parameterized queries and log what happens.⁶ A model that writes into an ERP field or a database query is a source of untrusted input, however polite its prose.

Where the controls come from

NIST’s Generative AI Profile (AI 600-1, July 2024) is a companion to the AI Risk Management Framework, whose functions are Govern, Map, Measure and Manage.⁷ It lists information security among the risks specific to generative AI and says generative AI expands the attack surface, because the systems are themselves open to attacks such as prompt injection and data poisoning.⁷ It also describes indirect prompt injection, in which the attacker plants prompts in data the system is likely to retrieve, and says researchers have demonstrated stealing proprietary data or running malicious code remotely this way.⁷

Three of the profile’s actions translate directly into a workflow checklist. GOVERN 1.6 asks for an inventory of AI systems. GV-1.5-002 covers incident response and incident disclosure processes. GOVERN 6.1 deals with risks from third parties.⁷ The mapping of these actions to the checklist below is our own, and the profile does not prescribe the controls in this article.

What a failure costs, and what the numbers do not say

IBM puts the global average cost of a data breach at $4.44 million, the first decline in five years.¹ It reports that organizations with high levels of shadow AI paid an average of $670,000 more per breach than those with low or no shadow AI, that one in five organizations reported a breach due to shadow AI, and that 63 % of breached organizations either had no AI governance policy or were still developing one.¹

The same release carries a finding that cuts the other way: organizations that used AI and automation extensively in security saved an average of $1.9 million in breach costs and shortened the breach lifecycle by an average of 80 days.¹ AI also helps defenders, so the sensible position is to secure the workflows and to use the tooling.

What these controls will not fix

Given OWASP’s own statement about prompt injection, no control in this article gives a guarantee. Least privilege, output validation and human approval cap what a successful attack can do. They do not stop the attempt.

The controls also have a price. Human approval adds handling time to every action it covers, and narrow permissions mean some tasks stay manual. That is our reasoning, not a measured figure from the sources above, but it is why the sensible order is to apply the strictest controls to workflows that move money, change records or send messages outside the company, and lighter ones to read-only summarization.

A control checklist before go-live

Each step below names the risk or the NIST action it addresses. None needs a new tool to start.

1. Inventory every workflow that sends company data to a model, including vendor-hosted ones, and name an owner for each (GOVERN 1.6).
2. List the data each workflow sees and remove the fields the task does not need before the call; ask each vendor how long prompts are retained (LLM02, GOVERN 6.1).
3. Treat every document, email and web page the workflow reads as untrusted, keep it apart from instructions and test with adversarial samples before launch (LLM01).
4. Give the workflow the narrowest permissions: read-only unless writing is the task, and no open-ended tools such as a shell (LLM06).
5. Validate model output against a fixed format before it reaches a database, an ERP field or an outgoing message, and use parameterized queries (LLM05).
6. Require human approval for payments, contract changes and external messages, log every call, and write down who responds to an incident and who tells whom (GV-1.5-002).

If you do not yet know which of your workflows are worth automating, the free pre-audit is a short questionnaire that returns a first estimate of where AI could cut cost. It is not a security assessment.