AI security

An AI agent is also an access decision

When an assistant can act on business systems, its permissions and the checks around each action become part of the product design.

Executives in discussion around a boardroom table

An assistant that answers from public information and an agent that can change a customer record may use a similar conversational interface. Their impact is different. Connecting tools gives the system the ability to act, so the scope of those actions deserves the same attention as the quality of its answers.

Separate understanding from authorisation

The model can interpret what a user wants. The application should check whether the user is entitled to perform the requested action. Instructions in a prompt are useful guidance, but they are not a substitute for access checks in the service that executes the action.

Make tools specific to the task. Looking up an order, drafting a reply and issuing a refund should have distinct permissions and checks. A narrowly scoped operation is easier to review than a general connection that can run arbitrary commands or modify unrelated records.

Treat retrieved content as information

Agents may read documents, emails, web pages or tool responses containing text they did not control. That content can include instructions intended to redirect the model. OWASP describes this class of risk as prompt injection.

Keep the authority to act outside those retrieved instructions. Check tool arguments, restrict the resources a tool can reach and avoid exposing credentials to the model or the browser. Give the agent the access needed for the current task, with the same boundaries enforced when a request is unusual.

Match confirmation to the consequence

Every extra question has a cost for the user. Routine information retrieval should remain straightforward. An irreversible or materially sensitive action may need a preview, an explicit confirmation or a human decision. Choose the check according to the action, rather than placing the same interruption in every conversation.

This is also a service design issue. A user correcting an email address should be able to continue the task. A system should not turn a recoverable misunderstanding into a dead end. Good controls protect the action while allowing ordinary corrections and retries.

Test the boundary, not just the dialogue

Check whether an unexpected instruction can cause an unauthorised read, write or disclosure. Exercise failure cases at the API boundary as well as in the conversation. Keep enough operational evidence to understand what happened, while limiting access to records containing personal information.

OWASP's guidance on excessive agency highlights the risk of giving a model more functionality, permissions or autonomy than its task requires. For us, the design question is concrete: what can this agent do, for whom, with which evidence and under whose control?

References: OWASP on prompt injection and excessive agency.

Startit brings cybersecurity and applied AI expertise to European and American clients, alongside brand strategy and marketing partnerships.

More insights

Contact

What do you need to secure or build?

A first conversation costs nothing and usually takes twenty minutes.

contact@startitpartners.com

Geneva · Paris · Nice