AI and LLM Vulnerabilities, and Phishing Controls

What it is

This page covers weaknesses in AI language model applications, and the controls that reduce phishing and social engineering.

A large language model (LLM) generates text from the instructions it is given. Applications wrap it with a system prompt, extra data and tools, and each part can be attacked. Phishing is the older problem, and AI makes it cheaper for attackers, so the controls matter more.

How it works

Most LLM risks come from the model being unable to separate trusted instructions from untrusted text.

Prompt injection

Prompt injection means crafted input changes the behaviour of the model. In direct injection, the user types the malicious instruction. In indirect injection, the instruction is hidden in content the model reads, such as a web page, a document or an email.

Because the model follows text, a hidden instruction can make it ignore its rules, reveal data or call a tool. The impact grows with the permissions the model has been given.

System prompt leakage

The system prompt holds the rules and sometimes secrets, such as keys or internal logic. An attacker can try to make the model repeat it. Do not place credentials in a system prompt, and do not rely on the prompt as the only access control.

Vector and embedding weaknesses

Many applications retrieve documents by turning text into embeddings, which are number vectors, and searching a vector database. This is called retrieval augmented generation.

Weaknesses include poisoned documents added to the store, and weak access control that lets one user retrieve another user data. Embeddings can also leak information about the original text. Control who can write to the store and filter retrieval by the caller permissions.

Supply chain

An AI application depends on models, datasets, libraries and plugins from outside sources. A tampered model, a poisoned training set or a malicious package can compromise the application before it runs. Use trusted sources, verify integrity, and keep an inventory of components.

Mitigations in summary

Human approval for sensitive actions is a useful extra control. Logging prompts and responses helps with detection.

Phishing and social engineering controls

No single control stops phishing, so layers are used. They fall into technical, process and human groups.

• Email authentication: SPF, DKIM and DMARC help receivers detect forged sender domains.
• Email security gateway: filters spam, malicious attachments and links, and can detonate files in a sandbox.
• URL and web filtering: blocks known bad sites at click time and through DNS.
• Multifactor authentication: a captured password alone does not give access.
• Endpoint protection: stops a payload that a user opens.
• Training and reporting: staff learn the signs and have an easy way to report mail.
• Process controls: call back to a known number before changing payment details.

Simulated phishing campaigns measure how people respond and show who needs more training. Pair them with a simple report button, so a suspicious message reaches the security team quickly.