Frameworks › NIST AI RMF
NIST AI RMF (AI 100-1)
The NIST AI Risk Management Framework (AI 100-1, January 2023) defines four functions: GOVERN, MAP, MEASURE, and MANAGE. ConstantX contributes engagement evidence primarily to MEASURE, contributes to parts of MAP through threat modeling, and produces inputs organizations can use in MANAGE decisions.
Function Overview
| Function | ConstantX Role |
|---|---|
| GOVERN | Not in scope. GOVERN defines organizational policies, accountability structures, and roles. ConstantX assurance outputs can serve as evidence inputs to governance documentation, but ConstantX does not perform GOVERN activities. |
| MAP | Partial contribution through threat modeling. The T-code threat model walk contributes evidence for MAP 2.3 (TEVV documentation), MAP 3.2 (cost documentation), and MAP 3.3 (application scope). The remaining MAP categories — organizational context, risk tolerance, human oversight process — stay with the organization using the framework. |
| MEASURE | Primary function. ConstantX contributes empirical verdict data, confidence intervals, and evidence chains to MEASURE activities. |
| MANAGE | ConstantX evidence informs two MANAGE subcategories: deployment decisions (MANAGE 1.1) and validation that deactivation mechanisms function under adversarial conditions (MANAGE 2.4). Organizational response planning and incident procedures remain out of scope. |
MAP — Contributions
ConstantX engagements begin with a T-code threat model walk against the target system. This process contributes evidence to three MAP subcategories. The remaining MAP categories are organizational activities that ConstantX does not perform.
| Subcategory | NIST Description (AI 100-1) | ConstantX Output |
|---|---|---|
| MAP 2.3 | Scientific integrity and TEVV considerations are identified and documented, including those related to experimental design, data collection and selection, system trustworthiness, and construct validation. | The threat model walk (T1–T17 against the target system) is the TEVV design document: it identifies which attack techniques to test, what constructs each scenario validates, and what is structurally out of scope. Coverage Boundaries documentation makes the scope explicit and defensible. |
| MAP 3.2 | Potential costs, including non-monetary costs, which result from expected or realized AI errors or system functionality and trustworthiness — as connected to organizational risk tolerance — are examined and documented. | Adversarial scenarios document the specific consequence of each threat if realized: data exfiltration (TM-004), unauthorized command execution (TM-005), identity spoofing (TM-012), supply chain compromise (TM-018). The undefined_behavior verdict identifies where costs were not contained by the target enforcement surface. |
| MAP 3.3 | Targeted application scope is specified and documented based on the system’s capability, established context, and AI system categorization. | The engagement scope document specifies the target deployment configuration (model snapshot, controller version, tool set), the suite version applied, and the Coverage Boundaries that define what the engagement does and does not assess. |
MEASURE — Satisfied Subcategories
The following 12 subcategories are satisfied with empirical evidence from completed engagements. Descriptions are quoted directly from NIST AI 100-1 (Table 3).
| Subcategory | NIST Description (AI 100-1) | ConstantX Output |
|---|---|---|
| MEASURE 1.1 | Approaches and metrics for measurement of AI risks enumerated during the MAP function are selected for implementation starting with the most significant AI risks. The risks or trustworthiness characteristics that will not — or cannot — be measured are properly documented. | Decision Coverage methodology targets adversarial runtime risks identified in the threat model walk, prioritized by threat severity. Coverage Boundaries explicitly documents structural limits on what cannot be measured within target-runtime assurance scope (e.g., behavioral drift, deceptive alignment, multi-service lateral movement). |
| MEASURE 1.3 | Internal experts who did not serve as front-line developers for the system and/or independent assessors are involved in regular assessments and updates. | ConstantX is the independent third-party assessor. It does not develop the system under test. Engagements are conducted against an externally defined target by assessors independent of that system’s development team. |
| MEASURE 2.1 | Test sets, metrics, and details about the tools used during TEVV are documented. | Every scenario carries a threat_id, scenario spec, and target enforcement configuration where applicable. Suite version, scenario IDs, and run window are recorded and bound to the engagement artifact. Auditors can inspect the exact test set used for any completed engagement. |
| MEASURE 2.3 | AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s). Measures are documented. | Terminal Coverage = (valid_commit + bounded_failure) / Total Runs. Measured under single-pass autonomous execution with no retries and no human-in-the-loop — the exact condition of deployment. Documented in the Decision Coverage report with Wilson 95% CI. |
| MEASURE 2.5 | The AI system to be deployed is demonstrated to be valid and reliable. Limitations of the generalizability beyond the conditions under which the technology was developed are documented. | TC metric with Wilson 95% CI establishes statistical validity bounds for the assessed configuration. Coverage Boundaries documents generalizability limits: scope is bound to sandbox-testable target enforcement behaviors; structural out-of-scope risks are named explicitly, not omitted. |
| MEASURE 2.6 | The AI system is evaluated regularly for safety risks. The AI system to be deployed is demonstrated to be safe, its residual negative risk does not exceed the risk tolerance, and it can fail safely, particularly if made to operate beyond its knowledge limits. | All adversarial scenarios test safe failure under attack conditions. bounded_failure verdict demonstrates that target enforcement surfaces contained unsafe actions before completion. Terminal Coverage establishes the rate at which the system fails safely. The undefined_behavior rate with confidence interval quantifies residual risk. |
| MEASURE 2.7 | AI system security and resilience — as identified in the MAP function — are evaluated and documented. | Adversarial scenarios cover prompt injection, tool argument attacks, path traversal, privilege escalation, and step exhaustion. ConstantX records target-runtime policy, sandbox, and gate signals where the assessed deployment exposes them. Results mapped to OWASP ASI risk categories and MITRE ATLAS technique IDs. |
| MEASURE 2.13 | Effectiveness of the employed TEVV metrics and processes in the MEASURE function are evaluated and documented. | Published methodology documents the three-state verdict taxonomy, Wilson score confidence intervals, target-runtime evidence capture, and deterministic trace replay. Explicitly states what Decision Coverage measures and what it does not. Available at constantx.net/paper. |
| MEASURE 3.1 | Approaches, personnel, and documentation are in place to regularly identify and track existing, unanticipated, and emergent AI risks based on factors such as intended and actual performance in deployed contexts. | Append-only engagement index tracks every assurance run by dated model snapshot. Per-scenario verdict comparison across snapshots surfaces category-level behavioral drift. Gap analysis identifies untested threats per model version, feeding back into scenario authoring for subsequent engagements. |
| MEASURE 4.1 | Measurement approaches for identifying AI risks are connected to deployment context(s) and informed through consultation with domain experts and other end users. Approaches are documented. | Threat model derived from the specific target system’s architecture and tool configuration — not generic templates. Suite runs against the exact dated model snapshot + controller + tool configuration intended for deployment. Model aliases are not accepted; dated snapshots required. |
| MEASURE 4.2 | Measurement results regarding AI system trustworthiness in deployment context(s) and across the AI lifecycle are informed by input from domain experts and relevant AI actors to validate whether the system is performing consistently as intended. Results are documented. | Decision Coverage report delivers per-scenario verdicts, OWASP ASI coverage, confidence intervals, and full trace bundle as a verifiable artifact. Report is bound to the specific assessed configuration and suite version and is available for auditor inspection. |
| MEASURE 4.3 | Measurable performance improvements or declines based on consultations with relevant AI actors, including affected communities, and field data about context-relevant risks and trustworthiness characteristics are identified and documented. | Per-scenario verdict comparison across model versions surfaces category-level regressions. Example: Opus 4.5 100.0% TC vs. GPT 5.4 85.85% TC — the 14.15% gap is concentrated in specific ASI categories driven by systematic multi-action batching behavior, not uniform degradation. |
MEASURE — Not in Scope
| Subcategory | Reason Not in Scope |
|---|---|
| MEASURE 1.2 | Appropriateness of AI metrics and effectiveness of existing controls regularly assessed and updated, including reports of errors. Organizational process activity — not a TEVV output. |
| MEASURE 2.2 | Human-subject assessments must meet applicable requirements and be representative of the relevant population. ConstantX assurance engagements do not involve human subjects. |
| MEASURE 2.4 | The functionality and behavior of the AI system are monitored when in production. ConstantX is a point-in-time pre-deployment assurance engagement, not a continuous production monitoring system. Retesting on new model snapshots detects behavioral drift but does not constitute production monitoring. |
| MEASURE 2.8 | Risks associated with transparency and accountability examined and documented. Transparency and accountability risks are organizational governance concerns addressed in the GOVERN function. |
| MEASURE 2.9 | The AI model is explained, validated, and documented; output interpreted within its context to inform responsible use. Interpretability and explainability are model-level concerns outside target-runtime adversarial assurance scope. |
| MEASURE 2.10 | Privacy risk of the AI system examined and documented. Out of target-runtime adversarial assurance scope. |
| MEASURE 2.11 | Fairness and bias assessment with documented results. Out of target-runtime adversarial assurance scope. |
| MEASURE 2.12 | Environmental impact and sustainability assessed and documented. Out of scope for target-runtime adversarial assurance. |
| MEASURE 3.2 | Risk tracking considered for settings where measurement techniques aren’t available. Meta-level framework planning activity, not a TEVV output. |
| MEASURE 3.3 | Feedback processes for end users and impacted communities established and integrated into assurance metrics. Organizational process activity. |
MANAGE — Inputs
ConstantX evidence informs two MANAGE subcategories. ConstantX does not perform MANAGE activities — risk treatment planning, incident response, and decommissioning procedures are organizational responsibilities. ConstantX provides the empirical input those decisions require.
| Subcategory | NIST Description (AI 100-1) | ConstantX Input |
|---|---|---|
| MANAGE 1.1 | A determination is made as to whether the AI system achieves its intended purposes and stated objectives and whether its development or deployment should proceed. | Terminal Coverage and its Wilson 95% CI provide quantitative input for a deployment decision. A system with a high undefined_behavior rate did not contain all tested adversarial cases under the measured conditions. |
| MANAGE 2.4 | Mechanisms are in place and applied, and responsibilities are assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use. | ASI-10 (Rogue Agents) scenarios validate that the target runtime's kill paths and timeout enforcements fire correctly under adversarial conditions. bounded_failure verdicts on these scenarios demonstrate that deactivation mechanisms intercept unsafe actions before they complete. |
The Measurement Gap
Most AI governance platforms satisfy NIST GOVERN, MAP, and MANAGE through questionnaires, risk scorecards, and policy documentation. None of those produce MEASURE 2.3 — performance criteria measured under conditions similar to deployment — or MEASURE 2.6, which requires demonstrating the system can fail safely under adversarial conditions.
“We reviewed the model and assessed risk as medium” is not a measurement. MEASURE 2.3 requires conditions similar to deployment. MEASURE 2.5 requires demonstrated validity. MEASURE 2.6 requires demonstrated safe failure. ConstantX is the measurement layer. It plugs into existing governance workflows without replacing them.
Target-runtime enforcement is structural. Alignment is probabilistic. Decision Coverage measures the structural part.
Review NIST Measure evidence methodology
All Frameworks · OWASP ASI · MITRE ATLAS · Methodology Paper