The NIST AI Risk Management Framework, a structured approach for managing risks of AI systems, including functions like Map, Measure, Manage, and Govern.

An attack that attempts to bypass a model's safety policies or system instructions, causing it to generate disallowed content.

Technical and policy mechanisms (such as safety filters, access controls, and policy checks) that constrain what a GenAI system can do or say.

Structured, adversarial testing of a system to uncover vulnerabilities and failure modes, often using creative or malicious prompts.

What is Hallucination?

A GenAI output that is confident but false or unsupported by the provided data or reliable sources.

What is Measure function?

The AI RMF function focused on developing and using metrics, tests, and evidence to understand AI risks and performance.

What is Prompt injection?

A technique where a user prompt tries to override or manipulate the system's instructions or context, often to exfiltrate data or trigger unsafe behavior.

What is Information leakage?

When a GenAI system reveals sensitive training data, secrets, or personal information that should not be disclosed.

What is Content safety model?

A classifier or model used to detect and score potentially harmful content, such as toxicity, hate speech, or self-harm.

Measuring GenAI Risks: Evaluations, Red-Teaming, and Guardrails — Operationalizing the NIST AI RMF and Generative AI Profile in Practice

From Map to Measure: What Are We Evaluating?

From Map to Measure

The NIST Generative AI Profile (NIST AI 600-1, 2024) extends the AI RMF to capture GenAI-specific risks. Moving from Map to Measure means turning these risks into observable tests, metrics, and evidence.

GenAI-Specific Risks

For GenAI, we must test toxicity, hallucinations, sensitive information leakage, prompt injection and jailbreaks, unsafe code or tool use, and synthetic media risks such as deepfakes and impersonation.

From Outcomes to Tests

The Profile lists outcomes like "harmful content is rare" or "no sensitive data disclosure." The Measure function turns these outcomes into concrete evaluation plans linked to your specific GenAI use case.

Module Goals

You will practice translating Profile outcomes into evaluation questions, choosing tests and metrics, running GenAI-specific evaluations, and documenting results as risk evidence for governance.

Step 1: Define the GenAI Use Case and Risk Priorities

Start With the Use Case

Before picking metrics, be precise about what the GenAI system does: chatbot, code assistant, image generator, summarizer, or an agent with tools. The use case shapes what you must measure.

Who Can Be Harmed?

Identify possible harms to individuals, groups, organizations, and society, such as bad advice, harassment, data leakage, reputational damage, or large-scale misinformation.

Link to Profile Outcomes

Choose 3–5 priority outcomes from the Generative AI Profile that fit your use case, such as factuality and harm prevention for medical chatbots or security and IP leakage for code assistants.

Why This Matters

These priority outcomes will directly drive your evaluation questions and test selection in the Measure function, ensuring you focus on the most important risks first.

Step 2: Turn Profile Outcomes Into Evaluation Questions

From Outcomes to Questions

After choosing priority Profile outcomes, convert each into evaluation questions that can actually be tested. This makes the outcomes concrete and measurable.

Bank Chatbot Example

For a bank chatbot, outcomes might include: low toxicity, no sensitive data leakage, and resistance to prompt injection or jailbreak attempts.

Sample Questions

Ask: How often does content exceed a toxicity threshold? Can the model be induced to reveal customer data? Can attackers bypass safety instructions with tricky prompts?

Qualities of Good Questions

Good evaluation questions are specific, observable, and measurable. They form the bridge between abstract Profile outcomes and concrete tests you can run.

Step 3: Core GenAI Evaluation Dimensions and Metrics

Key GenAI Dimensions

Most GenAI systems must be evaluated on toxicity and harm, hallucinations and factuality, privacy and leakage, robustness to prompt injection, and misuse or synthetic media risks.

Measuring Toxicity

Use prompt sets and a content safety classifier to score outputs for hate, harassment, or self-harm. Track the percentage of outputs above a toxicity threshold, broken down by category.

Measuring Hallucinations

Use QA benchmarks or reference documents. Score responses as correct, partially correct, or incorrect, and compute factual accuracy and hallucination rates per 100 responses.

Leakage and Prompt Injection

Test privacy with canary strings and adversarial prompts. Test robustness with curated jailbreak prompts and record how often the model follows unsafe instructions or reveals sensitive data.

Step 4: Mini Evaluation Plan for a Course-Help Chatbot

Use Case: Course-Help Chatbot

A university chatbot answers course questions. Priority outcomes: no hateful or harassing content, no disclosure of other students' data, and reliable information about course logistics.

Evaluation Questions

Ask: How often does toxicity occur under stressful prompts? Can the bot reveal other students' grades or emails? How accurate are answers about schedules and grading rules?

Tests and Metrics

Run toxicity tests with 200 prompts, privacy tests with 50 adversarial prompts, and factuality tests with 100 syllabus-based questions. Track toxicity rate, privacy leaks, and accuracy.

From Tests to Governance

Set thresholds, such as zero privacy leaks and >95% accuracy. If results fall short, deployment is limited and mitigations like stronger guardrails or fine-tuning are required.

Step 5: Design Your Own Evaluation Questions

Activity: Pick a GenAI use case and draft evaluation questions.

Choose one use case:

A code-generation assistant for a software engineering course.
An image generator for marketing materials.
A language tutor chatbot for beginner learners.

For your chosen use case, list three priority outcomes (in your own words). Example for a code assistant:

Outputs do not contain known security vulnerabilities for common tasks.
Outputs do not copy large blocks of proprietary or licensed code.
The assistant refuses to generate clearly malicious code (e.g., ransomware).

Now write one evaluation question per outcome. For example:

Security: "Given 100 security-focused coding tasks, how often does the assistant suggest insecure patterns (e.g., SQL injection, hard-coded secrets)?"
IP: "Can the assistant be prompted to reproduce long, verbatim segments from popular open-source projects under restrictive licenses?"
Misuse: "Across a set of 50 malicious prompts, how often does the assistant comply rather than refuse?"

Pause and actually write your three questions in a separate note or document.

Reflection prompt:

Which of your questions are easiest to measure with automated tools?
Which require human expert judgment?
How would you document them so a governance committee can understand the risk?

Step 6: Red-Teaming and Adversarial Testing for GenAI

What Is Red-Teaming?

Red-teaming is structured adversarial testing to uncover failure modes before and after deployment. For GenAI, it has become a standard expectation for high-impact systems since the mid-2020s.

Designing Attacks

Start with threat modeling, then design jailbreak prompts, misuse scenarios, and data-extraction attempts. Use role-play, translation, and multi-step conversations to probe weaknesses.

Executing and Scoring

Use human red teamers and sometimes automated tools. Log prompts and outputs, then score attempts as success, partial success, or failure to compute jailbreak and leakage rates.

Using Results

High success rates trigger stronger guardrails, restricted deployment, or retraining. Red-teaming is iterative and should be repeated after major model or system changes.

Step 8: Quick Check – Linking Tests to Outcomes

Answer this question to check your understanding.

You are evaluating a GenAI email-writing assistant used by a hospital. The Generative AI Profile highlights outcomes about privacy, toxicity, and factuality of medical logistics. Which of the following test sets is **best aligned** with these outcomes?

A: 1) Measure latency and throughput; 2) Check that emails use polite language; 3) Measure token usage costs.
B: 1) Run a benchmark on creative writing quality; 2) Ask users if they like the style; 3) Count the number of emojis generated.
C: 1) Adversarial prompts trying to reveal patient data; 2) Safety scoring of outputs for toxicity; 3) Accuracy tests comparing appointment and procedure details to the official system.
D: 1) Measure the model's BLEU score on a translation dataset; 2) Evaluate rhyme quality in generated poems; 3) Count the number of distinct words used.

Show Answer

Answer: C) C: 1) Adversarial prompts trying to reveal patient data; 2) Safety scoring of outputs for toxicity; 3) Accuracy tests comparing appointment and procedure details to the official system.

Option C directly targets privacy (adversarial prompts for patient data), toxicity (safety scoring), and factuality of logistics (accuracy against the official system). The other options either ignore these outcomes or focus on unrelated metrics.

Key Terms

AI RMF: The NIST AI Risk Management Framework, a structured approach for managing risks of AI systems, including functions like Map, Measure, Manage, and Govern.
Jailbreak: An attack that attempts to bypass a model's safety policies or system instructions, causing it to generate disallowed content.
Guardrails: Technical and policy mechanisms (such as safety filters, access controls, and policy checks) that constrain what a GenAI system can do or say.
Red-teaming: Structured, adversarial testing of a system to uncover vulnerabilities and failure modes, often using creative or malicious prompts.
Hallucination: A GenAI output that is confident but false or unsupported by the provided data or reliable sources.
Measure function: The AI RMF function focused on developing and using metrics, tests, and evidence to understand AI risks and performance.
Prompt injection: A technique where a user prompt tries to override or manipulate the system's instructions or context, often to exfiltrate data or trigger unsafe behavior.
Information leakage: When a GenAI system reveals sensitive training data, secrets, or personal information that should not be disclosed.
Content safety model: A classifier or model used to detect and score potentially harmful content, such as toxicity, hate speech, or self-harm.
NIST Generative AI Profile: NIST AI 600-1, a 2024 profile that extends the AI RMF with GenAI-specific risks, scenarios, and outcomes such as hallucinations, content harms, and data leakage.