AI and LLM Application Security
Security for the systems you build around a model. The OWASP Top 10 for LLM Applications — prompt injection, improper output handling, system prompt leakage, excessive agency — plus attacks on the models themselves, MLSecOps practice, and the governance layer of NIST AI RMF, ISO/IEC 42001 and the EU AI Act. Remembered with spaced repetition.
- flashcards
- 75
- flashcards
- per day
- ~10 min
- per day
- level
- Intermediate → Advanced
- level
- modules
- 5
- modules
What is AI security?
AI security covers the whole system you build around a model, not the model alone. There are four distinct layers, and treating them as one topic is why teams miss things: the application that puts untrusted text into a prompt and then acts on what comes back; the model itself, which can be poisoned, evaded, inverted or copied; the pipeline that trains, packages and serves it; and the governance that has to answer for all of it.
The single most misread item is prompt injection. It is not a prompting problem with a prompting fix — instructions and data reach the model through the same channel, so the model has no way to tell a developer's rule from a line hidden in a retrieved document. That is architectural, which is why the guidance is layered controls rather than a better filter, and why the interesting questions are about privilege and output handling: what the model is allowed to call, and what happens to its answer when it reaches a browser, a shell or a query.
This track works through the OWASP Top 10 for LLM Applications at the threat level, the classic attacks on models — poisoning, evasion, inversion, membership inference, extraction — the MLSecOps controls that answer them (model and dataset provenance, AI-BOM, artifact scanning), the risks AI assistants bring into your own SDLC such as slopsquatting and context leakage, and the frameworks a regulator or auditor will ask about. It is a security companion to the AI tracks, not a replacement for them.
5 modules, seed to bloom
Each module is a set of flashcards — 75 in total. Answer, review, and watch your knowledge grow from seed to full bloom.
LLM Application Risks
Attacks against the LLM application itself — prompt injection and its channels, improper output handling, system prompt leakage and agent privilege
15 cardsModel Attacks
Attacks against ML models themselves — poisoning, evasion, inversion, membership inference and model extraction
15 cardsMLSecOps
Treating models and datasets as supply-chain artifacts — provenance, AI-BOM, artifact scanning and inference-path protection
15 cardsAI in the SDLC
Risks introduced by using AI assistants and agents to build software — generated code, hallucinated dependencies and context leakage
15 cardsAI Governance & Regulation
The frameworks and law shaping AI risk work — OWASP LLM Top 10, NIST AI RMF, ISO/IEC 42001 and the EU AI Act
15 cardsSample questions
A taste of the real flashcards. Pick an answer, then reveal the explanation.
What distinguishes an indirect prompt injection from a direct one?
- AThe malicious instructions arrive inside content the model retrieves rather than from the user
- BThe malicious instructions arrive in a later conversation turn rather than the opening message
- CThe malicious instructions target the system prompt layer rather than the user prompt layer
- DThe malicious instructions are obfuscated to slip past a filter rather than written in plain text
Where does the real risk lie when an application's system prompt leaks?
- AIn the credentials and rules the prompt contains, rather than in the disclosure itself
- BIn the disclosure itself, because the prompt is the vendor's core intellectual property
- CIn the tokens the prompt occupies, because an attacker can then exhaust the context
- DIn the model's behaviour, because a prompt that has leaked stops being applied to it
What does a training-data poisoning attack do?
- ACorrupts the data a model learns from, so the trained model behaves incorrectly
- BCorrupts the input sent at inference, so a single prediction comes out wrong
- CCorrupts the stored weights after training, so the deployed model is replaced
- DCorrupts the evaluation set, so the model's reported accuracy is overstated
What does an AI-BOM inventory beyond a conventional software bill of materials?
- AModels, datasets and fine-tuning data, alongside the usual software components
- BThe hardware accelerators each model requires in order to run at target latency
- CThe prompts and system instructions the application sends to the model at runtime
- DThe cloud regions in which the model's inference endpoints have been deployed
What is slopsquatting?
- ARegistering a package name an AI model repeatedly invents so victims install it
- BPublishing a package whose name closely resembles a popular library's real name
- CUploading a model to a public hub under the name of a well-known research lab
- DSubmitting generated code to a project so a maintainer merges it without review
On what basis does the EU AI Act impose obligations?
- AThe risk the system's intended use presents, in graded tiers
- BThe size of the model, measured in parameters and training compute
- CThe revenue of the organisation placing the system on the market
- DThe technique used, with neural approaches regulated more strictly
Learn it once, keep it for good
Answer a question
Each card is one practical concept with multiple options. Pick what you think is right.
Get the full answer
See the correct option plus a clear explanation, and a link to deeper docs when one is available.
Review at the right time
A spaced-repetition engine (SM-2 or FSRS) resurfaces each card just before you would forget it.
Why AI security is worth your time
The gap your AppSec review misses
A model turns text into actions, which is a class of vulnerability no traditional review is looking for — and every team shipping an LLM feature now owns it.
Prompt injection as a threat, not a prompting tip
Direct versus indirect, the retrieval channel, multimodal payloads, and why input filtering alone cannot close it.
Traced to primary sources
Cards link to OWASP GenAI, NIST, ISO and the EU AI Act text itself, rather than to secondary summaries that go stale.
Ready for the compliance conversation
NIST AI RMF, ISO/IEC 42001 and the EU AI Act risk tiers — enough to know which obligations attach to what you are building.
Common questions
What is the OWASP Top 10 for LLM Applications? +
It is a catalogue of the risks specific to applications built on large language models — prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. It is versioned, so check genai.owasp.org for the current edition rather than an older blog summary.
What is prompt injection and can it be fixed? +
Prompt injection is input that makes a model follow the attacker's instructions instead of the application's. There is no known complete fix, because instructions and data share one channel — so the practical answer is layered: constrain what the model can do, validate its output before anything acts on it, give it the narrowest privileges, and require human approval for high-impact actions.
What is the difference between direct and indirect prompt injection? +
Direct injection is a user typing the malicious instructions themselves. Indirect injection hides them in content the model later reads — a web page, a file, a retrieved document — so the attacker never touches the conversation. Indirect is the one that matters most for RAG systems and agents, because the attack surface becomes everything the model is allowed to fetch.
What is MLSecOps? +
MLSecOps extends DevSecOps practice to models and datasets, treating them as supply-chain artifacts: recording model and dataset provenance, inventorying them in an AI-BOM, scanning serialized model files before loading them, signing and versioning what gets deployed, and monitoring the inference path in production.
Does the EU AI Act apply to a company outside the EU? +
It can. The Act attaches obligations to the risk tier of a system's intended use, and its scope reaches providers outside the EU when a system is placed on the EU market or its output is used in the EU. The cards cover the mechanisms rather than dates or penalties — verify anything you rely on against the official text, and take legal advice for your own case.
Is it free? +
Yes, completely free. No registration or credit card is required, and all your progress is stored locally in your browser.
Ready to secure what you ship with AI?
Plant your first seed today. Ten minutes a day turns prompt injection, MLSecOps and the EU AI Act into knowledge you can act on.