安全なAIアプリケーションの構築のトレーニングコース

Module 1 — How AI Apps Break

Lab: none — architecture walkthrough & discussion

A builder’s mental model of the attack surface.

Topics:

• LLM, RAG, and agent architectures from the developer’s side
• the request/response lifecycle of an AI feature
• prompt flow: system, developer, user, and tool messages
• where untrusted data enters (and re-enters) the model
• the trust boundaries a developer owns vs. inherits
• why AI attacks are semantic, not syntactic
• mapping the OWASP LLM Top 10 to code you write

Key insight: Every place untrusted text reaches the model — or model output reaches your code — is a boundary you own.

Module 2 — Prompt Injection for Builders

Lab: Lab 01 — 01-Prompt-Injection

The “SQL injection moment” for AI — but you can’t fully escape it.

Topics:

• direct vs. indirect prompt injection
• hidden instructions in documents, web pages, tool output
• jailbreaks and role-confusion
• why instruction/data separation matters
• defensive prompt design (delimiters, structure, minimal authority)
• why prevention is partial — design for containment

Hands-on:

• attack your own chatbot
• bypass a naïve filter
• restructure the prompt to shrink the blast radius

Module 3 — Treating Model Output as Untrusted

Lab: Lab 02 — 02-Output-Handling

The bug class developers underestimate most.

Topics:

• model output as untrusted input to the rest of the app
• insecure output handling (LLM02): XSS, SSRF, command/SQL injection downstream
• never eval/exec/render raw model output
• structured outputs and schema validation
• output encoding and allowlists
• safe rendering in web/UI contexts

Hands-on:

• find and fix an insecure-output-handling vulnerability
• enforce a JSON schema on model responses

Module 4 — RAG Security

Lab: Lab 03 — 03-RAG-Security

One of the biggest new attack surfaces — and it’s yours to build.

Topics:

• vector DB and retrieval threats
• ingestion sanitization
• document provenance and trust scoring
• retrieval scoping and metadata isolation
• hidden instructions in retrieved content (indirect injection)
• data exfiltration via retrieval

Hands-on: - poison a RAG pipeline with a malicious document - add ingestion sanitization and retrieval scoping to defend it

Module 5 — Agent & Tool Safety

Lab: Lab 04 — 04-Agent-Safety

Where a bug becomes an action.

Topics:

• excessive agency (LLM06) and tool abuse
• least privilege for agents
• tool allowlists and argument validation
• approval gates and human-in-the-loop
• sandboxing tool execution
• scoped, short-lived credentials for agents
• limiting autonomous loops and chaining

Hands-on:

• lock down an over-permissioned agent
• add an allowlist + approval gate to a dangerous tool

Module 6 — Secrets, Identity & Cost

Lab: Lab 05 — 05-Secrets-and-Cost

The operational mistakes that hurt fastest.

Topics:

• API key and secret management (never in prompts, code, or logs)
• per-user authentication and authorization for AI features
• propagating user identity to tools and retrieval
• denial-of-wallet: unbounded token/cost consumption
• rate limits, token budgets, and timeouts
• logging without leaking secrets or PII

Hands-on:

• get secrets out of the prompt/code path
• add per-user rate limits and a token/cost budget

Module 7 — Guardrail Libraries

Lab: Lab 06 — 06-Guardrails

Buy vs. build for input/output safety.

Topics:

• what guardrail frameworks do (and don’t)
• input guardrails: injection/PII/topic classifiers
• output guardrails: validation, filtering, grounding checks
• when a guardrail is appropriate vs. your own deterministic check
• layering guardrails with the controls from earlier modules
• performance, false positives, and failure modes

Hands-on:

• add an input/output guardrail layer to an AI feature
• measure what it catches and what it misses

Module 8 — Red-Teaming Your Own App

Lab: Lab 07 — 07-Red-Teaming

Ship it like an attacker already has it.

Topics:

• building an abuse/test suite for AI features
• automated prompt-injection and jailbreak tests
• regression-testing guardrails and policies
• running AI security checks in CI
• model and dependency supply chain (provenance, pinning)
• a pre-ship security checklist for AI features

Hands-on:

• write automated red-team tests for an AI feature
• wire them into a CI check

Module 9 — Scoring AI Security: The SAIS-100 Framework

Lab: none — scoring exercise (uses the Capstone app)

Turn everything you’ve built into a repeatable score.

Topics:

• the AI Security Hexagon: six questions instead of “is it secure?”
• the six scored categories (Data, Prompt, Agent, Supply Chain, Detection, Governance)
• the 100-point rubric and its weightings
• verdict bands and the single-category override rule
• the Elephant Scale Secure AI Score (SAIS-100) as a branded, re-runnable framework
• scoring before/after hardening as a metric

Hands-on:

• score the Capstone app on the 100-point scale
• name the single change that most raises the score

Key insight: The three highest-weighted categories map to the trust boundaries a developer owns — so the score measures exactly what this course taught.

Capstone

Students harden a deliberately vulnerable AI application end-to-end.

The starter app contains:

• an injectable prompt
• insecure output handling
• an unscoped RAG pipeline
• an over-permissioned agent
• secrets in the prompt path
• no cost limits

Students apply the course:

• restructure prompts for containment
• validate and encode model output
• sanitize and scope retrieval
• apply least privilege and approval gates to the agent
• move secrets out and add cost/rate limits
• add guardrails and automated red-team tests

Deliverable: a hardened app plus a short OWASP LLM Top 10 self-assessment.

Module - Lab map

Labs run in lab order, which follows module order. The course has 9 modules and 7 labs: Module 1 is an architecture walkthrough/discussion and Module 9 is a scoring exercise, so neither has its own lab folder.

• Lab 01 - 01-Prompt-Injection: Attack your chatbot & design for containment (Module 2)
• Lab 02 - 02-Output-Handling: Fix an insecure-output-handling bug (Module 3)
• Lab 03 - 03-RAG-Security: Poison then defend a RAG pipeline (Module 4)
• Lab 04 - 04-Agent-Safety: Lock down an over-permissioned agent (Module 5)
• Lab 05 - 05-Secrets-and-Cost: Secure keys + add cost guardrails (Module 6)
• Lab 06 - 06-Guardrails: Add an input/output guardrail layer (Module 7)
• Lab 07 - 07-Red-Teaming: Automated red-team tests in CI (Module 8)

Module 1 (How AI Apps Break) has no lab — it runs as an architecture walkthrough and discussion. Module 9 (Scoring AI Security) has no lab folder — it runs as a scoring exercise against the Capstone app.