TL;DR
A pre-deployment AI risk assessment is a structured process that forces an organisation to answer the questions it would otherwise avoid: What exactly could go wrong here? Who would be harmed? How severely? And what would we do about it? This guide provides the specific method — inputs, outputs, and decision criteria.
When to Run This Assessment
Every AI system deployment, regardless of scale or perceived risk, warrants some version of this assessment. The depth of the assessment should be proportional to the stakes — but skipping it entirely is never the right call.
Run a full assessment when:
- The system makes or influences decisions about people (hiring, lending, healthcare, education)
- The system generates content that will be published without human review
- The system handles sensitive personal data
- The system will be used by a large number of people
Run an abbreviated assessment when:
- The system is an internal productivity tool with no external-facing outputs
- The system’s outputs are always reviewed by a human before action is taken
- The system is a well-understood tool in a stable, low-stakes domain
The Assessment Process
01 / Define the System
Before assessing risk, the system must be precisely defined. Vague definitions produce useless risk assessments.
Document:
- Exactly what decision or output does this system produce?
- What data does it consume as inputs?
- Who operates it, and who receives its outputs?
- What actions are taken based on those outputs?
- Are those actions reversible?
The irreversibility question is the most important. An AI system that flags content for human review is fundamentally lower risk than one that automatically removes content. The distinction should shape every other decision in the assessment.
02 / Identify Affected Populations
For every group of people that the system’s outputs could affect, characterise:
- The nature of the potential effect (beneficial, neutral, harmful)
- The severity if harm occurs (minor inconvenience to life-altering)
- The reversibility of the harm
- Whether the group has meaningful recourse if harm occurs
Pay specific attention to groups that are underrepresented in your training data — they are statistically most likely to receive worse outcomes from your system.
03 / Enumerate Failure Modes
For each component of the system, identify the most plausible failure modes:
| Failure Mode | Cause | Consequence | Severity | Probability |
|---|---|---|---|---|
| Hallucination | LLM overconfidence | False information acted upon | High | Medium |
| Demographic bias | Skewed training data | Discriminatory outcomes | High | Medium |
| Prompt injection | Adversarial user input | System prompt bypass | Medium | Low |
| System outage | Infrastructure failure | Service disruption | Medium | Low |
Be honest about severity and probability. The purpose of this exercise is not documentation — it is to surface the risks that require mitigation before deployment.
04 / Assign Risk Scores and Thresholds
For each identified failure mode, assign a composite risk score:
Risk = Severity × Probability × (1 − Reversibility)
This formula forces the assessment to weight irreversible harms more heavily — consistent with good governance practice.
Establish thresholds before scoring:
- Green (deploy): Risk score below threshold, standard monitoring sufficient
- Yellow (deploy with controls): Risk score elevated, additional mitigations required
- Red (do not deploy): Risk score above threshold, deployment not approved until mitigated
05 / Design Mitigations
For each yellow or red risk, define specific mitigations:
- Technical: system prompt constraints, output filters, human review gates
- Operational: training for operators, clear escalation paths
- Monitoring: specific metrics to track, alert thresholds, review cadence
A mitigation is only valid if it meaningfully reduces severity, probability, or irreversibility. “We will monitor it” is not a mitigation for a high-severity risk.
06 / Document and Sign Off
The completed assessment should be:
- Reviewed by someone who did not conduct it
- Approved by the accountable AI Risk Owner
- Stored with version control so future changes can be tracked
- Referenced in the system’s ongoing monitoring plan
The assessment is not filed and forgotten. It is the baseline against which future incidents and monitoring results are compared.
Common Assessment Failures
Scope creep avoidance: Teams tend to assess the system they intended to build, not the one they actually built. Assess what the system does, not what it was designed to do.
Optimism bias: Risk assessors who want the project to succeed underestimate probabilities. Bring in an external reviewer for any high-stakes system.
Missing the indirect effects: The most consequential harms from AI systems are often indirect — effects on people who never interact with the system directly. Mapping affected populations carefully is the only way to catch these.
Supplier Evidence
Added 3 August 2026.
Most of the assessment above examines a system you can inspect. The part you cannot inspect is the supplier’s, and since 2 August 2026 the consequences of leaving that gap undocumented have changed.
The EU AI Act’s enforcement routes became operational on that date, including a channel for downstream providers to escalate alleged breaches by general-purpose AI model providers. Separately, Article 25(4) of the Act — as amended by the Digital Omnibus in July 2026 — now requires that the provider of a high-risk system and any third party supplying an AI system, AI model, tools, services, components or processes specify by written agreement the “necessary information, capabilities, technical access and other assistance” needed for compliance. “AI model” is newly added; model suppliers are explicitly in scope.
Those obligations bite from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I products, so there is time. The reason to start now is that renegotiating a contract takes longer than writing an assessment.
Add a supplier-evidence step to the assessment, and record four things:
- What you asked for. Model documentation, evaluation results, known limitations and failure modes, technical access for testing and validation, incident-notification commitments.
- What you received. Including the date, and whether it was specific to your use case or generic marketing material.
- What is missing, and what it prevents. An unanswered request is evidence — but only if you kept it and recorded the operational consequence. “The supplier did not provide bias testing for our population, so we conducted our own” is a defensible position. Silence is not.
- What you decided. Whether the gap was accepted, mitigated, or blocked deployment, and who made that call.
Two cautions on supplier assurances:
- A signed voluntary code is not evidence of operation. Roughly 190 organisations signed the Commission’s Code of Practice on Transparency of AI-generated Content in July 2026. Signing is not certification, not an audit, and not Commission approval. If anything it gives you something specific to hold a supplier to — not something to rely on.
- A vendor’s safety record describes a configuration. July 2026 disclosures from two major labs showed models behaving very differently when production classifiers and monitoring were absent. If your deployment sits upstream of, disables, or bypasses those controls, the vendor’s evidence does not cover you. The Claude evaluation boundary failures case study sets out how sharply that boundary can move.
Supplier assistance does not transfer the duty. It remains your assessment, your oversight and your evidence.
Jurisdiction-Specific Fairness and Outcome Evidence
Added 10 August 2026.
On 7 August 2026 the US Federal Trade Commission announced it will no longer pursue disparate-impact claims under any of its authorities, nor antidiscrimination claims under Section 5 of the FTC Act. It will continue appropriate disparate-treatment enforcement under the Equal Credit Opportunity Act.
Two details that get lost in summaries:
- The FTC did not merely deprioritise these theories — it asserts it never had the authority, describing its previous disparate-impact claims under both Section 5 and ECOA as ultra vires. That is a stronger and more durable position than an enforcement-priority shift.
- ECOA is not untouched. Disparate-impact claims under ECOA were abandoned too. What continues there is disparate-treatment enforcement.
- The policy statement is non-binding. Footnote 1 states it creates no rights and binds neither the FTC nor the public.
What this changes in an assessment: the regulator map, not the measurement.
It pre-empts no federal, state or local law, binds no court, amends no statute, and leaves deception, unfairness and other statutory theories intact. State attorneys general, sector regulators and private litigants are unaffected. An organisation deploying a credit or decision system in the US has a narrower FTC exposure and the same everything-else.
So keep measuring group outcomes. Measurement surfaces drift, data defects, pretext and exposure under regimes the FTC does not administer — and a statistically neutral system is not thereby free of intentional discrimination. Note the countervailing constraint honestly: collecting and using protected-class data must itself be lawful, and the lawful basis for doing it belongs in the assessment alongside the testing plan.
The wider point for anyone assessing a system deployed across jurisdictions: maintain a regulator-scope register, because a single regulator’s retreat is not a change in the underlying obligation. In the EU, over the same period, the direction has been the opposite.
Reassessment After a Capability-Threshold Change
Added 10 August 2026.
Most assessment processes assume the system under assessment holds still. It does not. Two things move underneath you: the vendor’s model, and the vendor’s own view of what that model can do.
On 7 August 2026 OpenAI announced it could not rule out that its forthcoming Astra model meets the Critical cybersecurity threshold in its Preparedness Framework, and imposed additional internal controls in response. Whatever happens to Astra, the mechanism is the point: a capability finding triggered a restriction automatically, before any incident.
Build the deployer-side equivalent. Decide, at assessment time, the answers to:
- What evidence would make us restrict or stop this deployment? Name it now, while nobody is under pressure.
- Who is authorised to pull that brake, and does it require anyone’s agreement?
- What is the maximum time between a vendor capability disclosure and our reassessment?
- What do we do in the interval — carry on, restrict scope, or suspend?
Two cautions on borrowing a vendor’s framework wholesale. First, a vendor’s threshold is not your threshold — theirs concerns what the model can do, yours concerns what it can do with your permissions, your data and your credentials. Second, a published framework is not necessarily a followed one: OpenAI’s own framework prescribes “halt further development” at Critical, while what was announced was a pause of non-conforming activity. Read what the vendor did, not only what its policy says.
Retesting After Vendor Control Changes
Added 10 August 2026.
A safeguard change can invalidate your deployment evidence while every identifier you track stays exactly the same — same product, same model name, same API, same integration.
On 7 August 2026 Anthropic retrained Fable 5’s biology classifier so more benign health, educational and clinical queries are answered directly rather than falling back to another model. In its own testing, biology-related fallbacks fell by about 85%. Nothing you would notice in a dependency file changed at all.
The retest, when a vendor announces a safeguard or routing change:
- Routing regression tests. Representative and adversarial requests across the affected categories. Which model answers now — and is that different from your baseline?
- Review-threshold revalidation. If content previously reached a human because it was refused or rerouted, and no longer is, your escalation trigger has moved without anyone deciding to move it.
- Output accuracy in the newly permitted range. More answers in a domain where wrongness is invisible to a non-specialist is precisely where a validation layer earns its cost.
- A vendor-change register, with a named owner who watches announcements and holds authority to trigger retesting, rollback or suspension.
And read the announcement critically. Anthropic’s 85% is self-reported internal testing, measured against a baseline it had deliberately set at maximum restriction at launch, and it concedes false positives inevitably remain without quantifying them. All three qualifiers are on the vendor’s own page — none of them appear in a headline.
Testing Cumulative Agent Behaviour
Added 10 August 2026.
Assessment usually asks whether each action an agent can take is acceptable. For agents, that is the wrong unit. Individually permitted actions combine into transactions nobody authorised. Three purchases each under the approval limit. A transfer that uses an account looked up two steps earlier. A sequence of reads that assembles something no single read exposed.
AWS shipped a partial answer on 6 August 2026: temporal policies in Amazon Bedrock AgentCore, which evaluate an agent’s session history before permitting a call, alongside gateway rate limits. Useful, and worth understanding precisely:
- Limits scope on several dimensions — target name, tool name, model ID and identity claims among them. Identity is the headline case, not the only one.
- Token rate limits apply to inference targets only. Requests and connection rate are broader.
- Connection limits are a rate — connections per second — not a ceiling on simultaneous sessions.
- Unmatched callers can fall through to service quotas unless a catch-all is configured. AWS’s own guidance warns about this.
What no platform decides for you: which business invariants matter, what the cumulative threshold should be, who approves an exception, and what happens on denial. A temporal policy enforces a rule consistently — including a wrong one.
Assess it by asking: what is the worst sequence of individually-permitted actions this agent could perform? Then check whether anything would stop it.
Reusable Agent Skills: Ownership, Evaluation and Withdrawal
Added 10 August 2026.
A governed model and a secured tool can still be connected by an unowned, outdated or over-permissioned reusable instruction. Skills, prompt packs and agent templates are shared like code and governed like documentation.
In a Google Cloud engineering account published 3 August 2026, the team described how it manages its own skills repository: structural, link and guardrail checks, evaluation prompts with scoring rubrics, repeated runs, and weekly quality checks running scheduled evaluation jobs across the full library to catch regressions.
Two things to hold in view. This is a practice account by one engineering team, not a standard, policy or certification — it is written in the first person and describes internal working practice. And its scope is Google’s own repository and contributions from Google product teams; nothing in it governs third-party or community-authored skills. Notably, exported public versions strip three things: internal assets, ownership information, and evaluation suites. What an external deployer can independently verify is therefore limited.
Your controls, for any shared skill you rely on:
- A named owner per skill, not a repository.
- Permission mapping — what tools and data does this instruction reach?
- Versioning and an approval baseline, so you can tell what changed.
- Recurring evaluation, because models, APIs, source material and agent frameworks all move after approval.
- A withdrawal route. Deciding to stop using a skill is a control only if someone can actually remove it.
Regulatory-Sandbox Readiness for Legal AI
Added 10 August 2026.
On 3 August 2026 the UK opened applications to the Legal Services AI Growth Lab, a coordinated engagement programme run with the Ministry of Justice and the Department for Business, Innovation, Science and Trade, alongside the Legal Services Board, Solicitors Regulation Authority, Council for Licensed Conveyancers and the Information Commissioner’s Office. Applications close at 11:59pm on 27 September 2026; participation lasts up to nine months.
Read the eligibility bar accurately, because it is lower than it sounds and differently aimed. The programme asks applicants to show readiness to engage with regulators at pace — a sufficiently developed product, partnerships in place where needed, and the organisational capacity and commitment to engage throughout. It states plainly: “We’re not expecting products to be market-ready.” Evidence strengthens an application rather than qualifying it.
Nor does every participant get all six bodies. Engagement is with relevant regulators, and which ones depends on the proposal.
What it does not do, and this is the part worth putting in front of anyone excited about it:
- No approval, endorsement, authorisation, exemption or safe harbour. The guidance says so directly.
- No change to the law. Questions must be answerable within existing regulatory frameworks.
- No transfer of professional responsibility. Participants remain responsible for complying with everything that already applies to them.
Which makes it a useful lens for assessment even if you never apply. The application asks what a good pre-deployment assessment asks: what exactly is the product, what specific regulatory question does it raise, who owns the answer, and can you evidence any of it? Regulatory engagement cannot substitute for an undefined use case, an absent control owner, or an inability to show the product is ready to be examined.
Agent Data and Business-Logic Controls
Added 18 August 2026.
When an agent answers a question about your business, two things can be wrong: the query it wrote, and the definition it applied. Most assessment attention goes to the first. The second is where the durable damage sits, because a wrong metric definition is wrong consistently and invisibly.
Google previewed a mechanism aimed at this on 14 August 2026: support for measures in BigQuery Graph, letting a conversational agent reach relationships, descriptions, synonyms and governed metrics rather than composing aggregations itself. Data modellers define the measure in the graph schema; the engine resolves the relationship paths before evaluating the metric, which avoids the row duplication that makes ordinary SQL joins produce wrong totals during graph traversal.
That is a real control, and worth understanding as a pattern beyond one product: constrain the query surface rather than trusting the model’s arithmetic.
Read the claim at the right strength. Google’s blog says this prevents hallucinations. Google’s own documentation says it reduces them. Where a vendor’s marketing and its documentation disagree, assess against the documentation — and note that incorrect source data, relationships, measures or generated queries still produce confidently wrong answers.
What remains yours, and cannot be delegated to the platform:
- Metric ownership. A named owner for each measure and relationship — who decides what “active customer” means, and who approves a change to it.
- Change control on the semantic model. Version control exists in the Looker/LookML integration, not natively in BigQuery Graph. Git and CI give you a change mechanism; they do not tell you a definition is correct.
- Generated-query testing. Validate the queries the agent writes, against known-good answers, before anyone acts on them.
- Drift monitoring. A definition that was right last quarter is a liability this quarter if the business changed and the metric did not.
- Human approval before consequential action. The platform can enforce a defined metric. It cannot decide whether an answer should trigger a decision.
Preview-stage constraints to factor into any assessment: it is a preview under pre-GA terms with potentially limited support; at most one graph per agent or conversation; tables and graphs cannot be combined as data sources; and GQL on a graph requires Enterprise or Enterprise Plus edition. Preview support runs through a dedicated mailbox — a fair signal of maturity.
The wider assessment question this belongs to: when an agent grounds an answer in your business logic, who owns that logic, and when was it last verified?
Agent Autonomy, Threat Modelling and Blast Radius
Added 24 August 2026.
On 20 August 2026 the UK National Cyber Security Centre published interim practical advice on managing the cyber risk of agentic AI. It is the closest thing yet to a national-authority checklist for this assessment — with one caveat to carry throughout: it is a blog post offering interim advice, not formal NCSC guidance. NCSC says explicitly that it is “working with partners to develop formal guidance which will build upon, and ultimately supersede, this blog.”
Threat model before deployment, and define the red lines. NCSC: “Before deployment, carry out threat modelling to identify the failure scenarios during the agent’s activities.” The modelling should cover the prompt given to the agent and “the networks and services the agent may be able to access and communicate with through the configured sandbox — either directly or indirectly.”
That word indirectly is doing the work. Blast radius is not the systems you connected; it is everything reachable from them. The output of the exercise is a documented scope that highlights “any ‘red lines’ that the agentic AI system must stay within” — and then an explicit decision on whether you accept the risks identified.
Sandboxing is unconditional; the level of isolation is proportionate. These are two different statements and they are easily merged into one wrong one. NCSC’s sandboxing instruction is absolute: “Always run AI agents within a sandboxed environment that controls and manages what resources can and cannot be communicated with, both locally and over a network.” What scales with risk is how far you isolate — for high-risk activities, “the most robust approach is to run your AI agent within an isolated and disconnected environment.” NCSC provides maturity models as assessment aids, not as a menu from which to pick the cheapest tier.
Identity and credentials are separate controls. Also easily conflated. Identity is unconditional — “All agents should be assigned their own unique identity in a class which differentiates them from human or individual systems.” Credential lifetime is hedged and attached to credentials, not identities: “Where possible you should restrict the credentials available to the agent… use credentials with the shortest possible lifetime.”
Logs must be trustworthy, and are themselves attack surface. NCSC asks for chain-of-thought traces, transcripts and sandbox logs — then adds that they “should be protected from modification or deletion” and “where possible… immutable so you can trust them during an investigation.” And the part most designs miss: consider “the attack surface created by your log collection infrastructure and whether an AI agent could abuse it to escape from its sandbox environment.”
Attribution. NCSC devotes a full consideration to making AI activity easy to attribute — distinguishing agent actions from human ones in your telemetry, which is what makes an investigation possible at all.
Clinical GenAI: Configured-Device Evidence and Lifecycle Monitoring
Added 24 August 2026.
On 18 August 2026 the FDA’s Digital Health Center of Excellence, within CDRH, published a discussion paper and request for feedback on regulating generative-AI-enabled medical devices. Feedback goes to a Regulations.gov docket by 19 October 2026.
Read the status before the substance. This is not guidance, draft or final, and CDRH says so — disclaiming that it communicates its “proposed (or final) regulatory expectations, including its expectations for supporting evidence in future marketing submissions.” It is a set of questions put out for comment, and respondents may answer some and ignore others. Anyone describing it as a new FDA requirement has misread it.
What makes it worth assessing against anyway is that the questions describe the shape of an evidence model that clinical deployers will recognise as overdue:
- Evaluate the final configured device — through benchmarking and clinical confirmation — rather than the foundation model underneath it.
- Continue evaluating after launch: periodic re-benchmarking, sample-based clinician review, and monitoring for degradation.
- Control for third-party foundation-model change, the failure mode where your device’s behaviour shifts because someone else updated something.
One floated idea deserves care because it is easy to over-read: a voluntary Foundation Model Master File. It does not exist. CDRH “seeks feedback on the feasibility” of such a programme, and its own Question 25 doubts the premise — asking, given that participation would be voluntary and developers “may have limited incentive to disclose safety-relevant information”, what would make it useful for premarket review. The paper also states such a file “would not constitute authorization of the underlying model for any device intended use.”
For a healthcare deployer, none of this is contingent on FDA settling anything. Qualified clinical review, local performance monitoring, an escalation route and the authority to suspend are your controls regardless of what the final regulatory model looks like. See qualified clinical review and escalation.
Agentic Financial Authority and Transaction Controls
Added 24 August 2026.
AWS made AgentCore Payments generally available on 18 August 2026, letting agents transact using delegated wallets without ever receiving raw wallet credentials. Payment sessions deterministically constrain maximum spend, currency and expiry, with short-lived tokens and transaction telemetry.
It is a genuine control, and it draws a clean line worth stating plainly: protecting payment credentials governs how funds move. It does not decide why, or on whose authority.
A cap of £500 does not know whether the agent chose an approved supplier, bought the right thing, or bought it twice. Everything that determines whether a purchase was legitimate sits on your side:
- Permitted purposes and merchants, defined before delegation.
- Who may delegate spending authority, and who may revoke it instantly.
- Cumulative limits across sessions — noting the constraint in the next section, that session-scoped enforcement cannot pool across sessions.
- The approval point: which purchases require a human before execution.
- Duplicate-payment protection, and what happens on retry.
- Reconciliation, refunds and disputes — an agent can initiate a transaction it cannot resolve.
- Emergency suspension, tested.
Failure cases to exercise deliberately: exceeded or expired authority, repeated retries, wrong merchant, failed approval, refunds, and immediate wallet revocation mid-transaction.
Translating Prose Policies into Enforceable Agent Rules
Added 24 August 2026.
On 20 August 2026 AWS described natural-language authoring of Dogwood policies — turning written requirements into enforceable rules covering tool-argument constraints, prerequisites and sequential ordering, rate limits, cumulative caps, and checks on free-form content via guardrails.
The reason it belongs in a risk assessment rather than a tooling note is AWS’s own honesty about what validation proves. Formal validation can establish that a policy is well-formed and schema-anchored. It cannot establish that the policy still means what its human author intended. AWS positions the tool as translation of rules that already exist in prose, and keeps the interpretation with the person who owns the source document.
Three things the translation does quietly, which an assessment should surface:
- It resolves ambiguity by choosing. On a cumulative cap, AWS notes the source document “says ‘transferred’ without saying whether a blocked or failed attempt counts”, and the translation sums attempts — “the safer reading for a cap”. A vague human rule becomes a specific machine rule by the tool’s judgement, not the author’s.
- It fills in unstated thresholds. “This rule states no threshold, so the translation uses the default for that check.”
- It filters before translating, deliberately — so an inexpressible rule cannot become “a policy that validates cleanly and enforces the wrong thing.”
Four classes of rule cannot be enforced this way at all, and knowing them is more useful than the capability list:
- Judgement rules — “act in the customer’s best financial interest” has no permit-or-deny form.
- Rules demanding an action, not a verdict — “redact it before the note is stored”. As AWS puts it: “A policy engine permits or denies a call. It doesn’t modify one.”
- Rules outside the language — “deny wire transfers on weekends and U.S. federal bank holidays”. There is “no day-of-week accessor and no holiday calendar”.
- Cross-session scope — enforcement evaluates a trajectory within a session, and AWS is blunt that “a cap that pools across sessions isn’t something a different phrasing can recover.”
For each, the tool returns a label saying it cannot be translated — which tells you to rewrite it, move it to a different control, or accept that it stays a human process. That label is the useful output, and an assessment should record which of your policies produced one, and what covers them instead.
Verifying Vendor AI Claims Before You Buy
Added 31 August 2026.
On 27 August 2026 the FTC finalised orders against Cox Media Group and two other firms over claims made for an AI-powered advertising product. The case is worth a section in a risk assessment not because of the penalty but because of what the FTC says the product actually was.
The complaint’s allegation is not that the service used a different dataset than advertised. It is that the service did not collect or use voice data in any manner — that it was, in the FTC’s words, nothing more than consumer email list buying, resold at a significant markup. The alleged AI capability was not overstated. It was absent.
Count I charges three misrepresentations, not two, and the third is the one most likely to be missed: alongside the collection and use of voice data, and consent, the complaint alleges the service generated consumer lists from across the country with only a fraction of consumers located near the small-business customer paying for local advertising. The order’s prohibition expressly covers the geographic targeting capabilities of the firm’s advertising or marketing services.
What this means for a pre-deployment assessment of a purchased AI system:
- Ask what the system does when the AI does not fire. A vendor whose product degrades to a conventional database lookup should be able to say so, and say how often it happens.
- Ask for the mechanism, not the outcome. “We use AI to identify intent” is a claim about a result. What signal, collected how, with what consent, is a claim you can test.
- Test the targeting, not just the accuracy. A system can be accurate about the wrong population. The geotargeting count exists because nobody checked where the matches were.
- Get the capability claims in the contract. Marketing claims sit outside the agreement you sign, which is where they are least useful to you.
Read the settlement language precisely
Two details commonly get flattened when this kind of order is summarised, and both matter if you are citing one.
Issuance and effect are different dates. The CMG order records “ISSUED: August 26, 2026”, while its own Order Effective Dates provision makes it final and effective upon publication on the Commission’s website as a final order — which the FTC’s case timeline dates to 27 August 2026. The order relies on the distinction throughout: some obligations run from the effective date, others from the issuance date. An assessment that treats the two as one will compute the wrong compliance clock.
A settlement is not an admission. The respondents neither admit nor deny the allegations except as specifically stated in the order, and admit the jurisdictional facts only for purposes of this action. Both qualifiers belong in any summary; dropping either changes what the document establishes.
The recordkeeping obligation is the part to plan for
If your organisation ever ends up under an order of this kind, the record-retention provisions are what bind day to day. They are longer and broader than the usual summary suggests:
- Records must be created for 20 years after the order’s issuance date, and each one retained for 5 years.
- Substantiation material, and evidence contradicting a covered representation, runs on its own clock — 5 years from the last dissemination of the representation.
- The categories are not limited to the marketing material. They include accounting records, personnel records, copies of all subpoenas and other communications with law enforcement relating to compliance, and — the provision worth reading twice — records that demonstrate non-compliance or tend to show any lack of compliance.
That last category is an obligation to retain the evidence against yourself. It is a good argument for building the retention schedule before you need it, which is covered in building a tested incident response capability.
Reading a Vendor “Preview” Announcement
Added 31 August 2026.
On 25 and 26 August 2026 Google announced Gemini Enterprise for Legal and Gemini Enterprise for Financial Services. Both are described as available in preview today. Taken together they are a clean worked example of how to read a vertical AI launch before it reaches a risk assessment — the pattern generalises well beyond Google.
Four checks, in order.
01 / What does “preview” actually entitle you to?
In both announcements: no private-or-public designation, no general-availability date, no pricing, no regions, and no eligibility criteria. Nothing on either page indicates that any given customer can obtain the product at all.
“Available in preview” is the vendor’s own hedge and should survive into your notes intact. It should not become “launched”, “generally available”, “shipping” or “in production” anywhere in an internal paper — and it should not be assumed to be a private preview either, because that has not been said. Where a vendor’s own word is the only qualifier available, keep the word.
02 / Are the named customers using this product, or a different one?
This is where the financial-services announcement rewards care. BNY, Citi Wealth, Lloyds Banking Group, Macquarie Bank and Signal Iduna are cited — accurately — as institutions using Gemini Enterprise, the general platform. They are not presented as users of Gemini Enterprise for Financial Services, and the press releases linked for them date from late 2025 and early 2026, before this launch existed. Only Deutsche Bank and CME Group are named in connection with the new product, and as development and design partners.
The legal announcement follows the same shape. Cleary Gottlieb, Freshfields, Weil and Williams & Connolly are firms Google is working closely with and early adopters shaping the product — Weil’s own quotation describes early access to emerging capabilities. None of the four is quoted describing a live production deployment or a measured result.
The check is mechanical: for each named organisation, ask whether the source says they use this product, and whether they describe an outcome. Reference customers for the platform are not reference customers for the vertical built on it.
03 / Does the capability belong to the product, or to one component of it?
Confidence scores, explicit methodologies, data snapshots and citations are attributes of the Financial Research agent — described as a Google-built, Google-managed agent — not of Gemini Enterprise for Financial Services as a whole, and not of the partner agents from D&B, FlowX, Obin and S&P Global that run alongside it.
The legal announcement has two of these. On the blog, ethical walls appear as a property of one connector, NetDocuments, described as preserving each user’s existing permissions and ethical walls — the broader claim that a firm’s ethical walls are inherited platform-wide sits in the separate press release. And the blog does not say the product verifies citations: it says skills enforce a firm’s playbooks, citation rules and house style. Enforcing a house citation format and verifying that a citation is real are different capabilities, and only the first is claimed in the document most people will read.
Attributing a component’s property to the whole product is the most common way a vendor claim gets accidentally hardened — usually by the buyer, not the vendor.
04 / Where the vendor does not hedge, hedge yourself
The financial-services page is the sharper example, and it runs the opposite way to what an assessor might expect.
There is no accuracy disclaimer of any kind on it: no statement that outputs may contain errors, no verify-before-use instruction, no human-in-the-loop requirement, no small print. Confidence scores are stated as a flat capability — the agent exposes its reasoning through confidence scores — with no statement of what they measure, how they are calibrated, or what scale they use. A confidence score whose basis is unstated is not yet evidence of anything, and it cannot be relied on in a control until the vendor says what it represents.
The surrounding language runs to absolutes. The control plane is said to hold every output to verifiable grounding with traceable citations; the product page states twice that every output is traceable through built-in confidence scores. The only sentence touching accuracy is competitive positioning — that general-purpose AI lacks the real-time accuracy, verifiable data lineage and strict security that financial institutions demand.
Universal quantifiers on a preview product are the thing to flag. The assessment note writes itself: the vendor makes unqualified claims about every output, for a product with no stated availability, no calibration detail for its confidence signal, and no published accuracy testing. That is not an accusation of bad faith — it is the gap your own evaluation has to close before deployment, and it is the gap that determines what testing you have to fund.
One terminology point in the same family: “data snapshots for auditing” and “auditable data snapshots” are Google’s phrases. Compressing them to “audit snapshots” implies a formal audit-log feature, and nothing is published on retention, immutability or export. Likewise “fully governed” is vendor wording describing the deployment environment in one document and the underlying platform in the other — quote it as the vendor’s characterisation, attached to the object the vendor attached it to.
“Provably Complete” Is a Claim About Arithmetic
Added 6 October 2026.
On 2 October 2026 AWS published its Adjudicated Query pattern — a bounded conversational layer over a deterministic rules engine, demonstrated by sweeping a synthetic corpus of leases for compliance. The architecture is genuinely careful and the post is unusually candid. It is also a clean worked example of how a precise-sounding guarantee can be precise about the wrong thing.
What the mechanism actually guarantees. Every sweep produces what AWS calls a completeness receipt: an asserted invariant that compliant + in-breach + ambiguous + unreadable must equal the number scanned, computed from counts and asserted before anything persists. A run that cannot account for its population does not finish. Unreadable documents are named rather than dropped.
That is a real control and worth having. Note precisely what it establishes: every record the system was handed landed in exactly one bucket. It says nothing about whether the right records were handed over, whether the facts extracted from them were correct, or whether the rule applied was the right rule.
Two of those buckets contain no determination at all. In AWS’s sample, 689 records are ambiguous and 20 unreadable. Both satisfy the invariant. So a record can be provably counted and still entirely unresolved — the receipt proves accounting, not adjudication. Any management report built on it needs the unadjudicated buckets on the face of it, not in an appendix.
The sample is a known-answer harness. The repository generates a synthetic corpus with counted, seeded violations and a manifest, and is deterministic so a rebuilt stack reproduces identical results. The violations were planted and counted in advance. The sample therefore demonstrates that the engine accounts for a population it was given; it demonstrates nothing about extraction accuracy from real documents — which is the step where real compliance programmes actually fail. AWS publishes no extraction-accuracy figure and does not claim one.
Where the vendor’s own words diverge from the vendor’s own code
This is the part to carry into a supplier assessment, because the pattern recurs far beyond AWS.
- The blog post concludes that the pattern delivers “provably complete, defensible compliance answers”, and the same phrase appears in the post’s social-sharing description.
- The phrase appears nowhere in the sample repository, which says instead that it is a proof of concept “demonstrating mechanism only”, is “Not production-ready”, and should not be deployed “or reuse its patterns” in production without additional security testing and independent legal validation of any rule content.
The marketing surface and the engineering surface of the same artefact disagree, and only one of them is linked from the press coverage. When you assess a vendor claim, read the repository, the README and the licence — not the post. Where they conflict, the one written for operators is the one to believe.
A second gap sits inside the post itself: AWS instructs the reader to verify the receipt by checking that the four buckets sum to the scanned total, but publishes only three of the four numbers. The invariant the vendor tells you to check is not checkable from the material the vendor published. The receipt is asserted, not exhibited. That is a reasonable thing for a blog post to do and an unreasonable thing for an assurance artefact to do, and the difference is the whole point.
Credit where it is due, and the finding inside it
AWS reports, in the first person, generative-AI failures from its own delivery path — a model stripped a caveat prefix; an illustrative citation tag was removed during paraphrase and an invented citation was presented as statute; a model given twenty preview rows inferred a population-wide range that did not exist in the data.
Its stated principle is the transferable one, and it belongs in any assessment of an AI system that summarises its own output: a safeguard in the payload is only as strong as its survival through paraphrase. If your controls live in text that a model later rewrites, they are not controls.
The evidence chain, and its missing leg
AWS defines defensibility as knowing which version of which rule was applied, to which clause text, by what method, on what date, and by whom. It then concedes that the token identifies the application rather than the individual who asked the question, so the final leg must be reconstructed by correlating identifiers, or the buyer must thread an end-user identity through and persist it.
Four of five legs ship; the missing one is the human. For a record intended to answer “who decided this”, that is the leg that matters. Add it to the assessment rather than assuming the architecture supplies it.
Finally, AWS says when not to use the pattern at all, and the honesty is worth repeating: where the call is a genuine judgment rather than a clean comparison — whether a clause is unconscionable, whether a disclosure is adequate — forcing it into this shape hides the subjectivity rather than removing it. A deterministic engine executing a contestable rule produces a confident, reproducible, wrong answer.
Endpoint Permissions Are Now an Agent Question
Added 6 October 2026.
On 2 October 2026 Apple told macOS developers it will tighten how Full Disk Access is granted. The post is about 150 words and it belongs in a risk assessment for two sentences in it.
The platform owner concedes the hole is deliberate. Apple writes that Full Disk Access largely sidesteps its own privacy controls in order to allow backup apps to function properly on the Mac. A documented exception to the consent model, shipped for one use case, now relied on by many others.
The harm reaches people who never consented. Apple notes that for communication apps this can also compromise the privacy of the people users are communicating with. Your employee grants the permission; the counterparties in their mailbox and message history did not, and in most cases will never know.
The AI connection is Apple’s own: as AI agents become increasingly capable and autonomous, it says, the risks associated with this level of access will grow substantially, preceded by the flat sentence “Addressing this is critical.”
Read the announcement precisely, because it says less than the coverage
- Apple names no AI product, no AI vendor and no developer anywhere in the post. Reporting that attaches specific assistants to it is adding them. Do not repeat those names as though Apple had.
- The present cause Apple identifies is developer behaviour, not AI. AI agents are connected to the future growth of the risk.
- There is no date, no macOS version, no beta, no mechanism and no developer action. Apple’s only temporal marker is “going forward”. The way the permission is granted today has not changed.
- Apple’s words are “very explicit user action” — and the clause around it preserves the grant for users who genuinely want it. Apple is raising the consent bar, not removing the capability.
- The stated remedy is informed consent, not least privilege. Apple offers no scoped alternative, no partial access and no audit mechanism. Comprehension-based consent for a whole-disk grant is a weak control, and worth saying so.
What to do now, since Apple asks for nothing
Treat it as advance notice of a dependency you probably cannot currently enumerate.
- Inventory which applications and agents hold Full Disk Access across your managed estate, who approved each one, and when.
- Record the business justification per grant. “It was needed at install” is not one.
- Map what each grant actually reaches — and for communication tools, note explicitly that this includes third-party correspondents’ data.
- Test revocation before Apple forces the question. Which workflows break, and is the breakage acceptable?
- Decide your position on agents holding it at all. An autonomous agent with whole-disk read is the broadest blast radius available on an endpoint, and it is granted by one dialog.
The same assessment applies on every platform that has an equivalent grant. Apple is the one that said it out loud.
Where the Enforcement Sits
Added 6 October 2026.
NVIDIA announced its Open Agent Safety Platform on 28 September 2026. It is worth a section not for the product but for the design principle it states and then, today, cannot yet deliver.
NVIDIA’s own framing is the strongest governance sentence in any vendor material this quarter: an agent in these circumstances cannot be expected to fully govern its own behaviour. Its analogy is apt — the internet did not become safe because web developers promised to be good, but because the browser stopped trusting the code in the page. It defines drift as agent actions departing from the intended task or operating constraints, and says it cannot be trained away while retaining the capability.
From that it derives five design principles, which are directly usable as criteria for assessing any vendor’s agent-safety claim:
- Policy must be verifiable.
- Enforcement must be out of band.
- The path to the model is the control point.
- Agent authority should scale with the ability to inspect its reasoning.
- Responsibility is shared between labs, enterprises and hardware providers.
The inversion to check before you buy
The platform has two halves, and they are at very different stages.
- OpenShell is real and available today: Apache-2.0, a public repository, published documentation, versioned releases. It runs each agent in a sandbox, instruments the kernel to enforce policy on file access, system calls and network connections, and — a genuinely good control — ensures agents never see real credentials, adding them only to requests bound for approved endpoints. Its formal-verification claim is narrower and more credible than the marketing implies: it checks what a policy change would newly allow before approval, so risky changes wait for human review. It is not a proof that an agent is safe.
- Sentry is the out-of-band half — a watchdog on separate silicon, outside the host trust boundary, able to quarantine an agent in milliseconds. It has no repository, no documentation, no licence, no price and no availability date.
But OpenShell’s own supervisor sits, in NVIDIA’s documentation, on the trusted side of the isolation boundary — the same host as the agent.
So the half you can deploy today gives you host-local, out-of-process enforcement. The half that satisfies NVIDIA’s own second principle is not yet a thing you can obtain. That is not a criticism of the architecture, which is sound; it is the distinction a risk assessment has to draw between an announced design and a deployable control, and the vendor’s own principle is the measuring stick.
The question to put to any agent-safety vendor: does your enforcement run inside the trust boundary of the thing it governs? If it does, it constrains an agent that is behaving. If the agent compromises its host, enforcement goes with it.
Note also what is shipping versus what is promised in the surrounding claims: the audit trail NVIDIA describes is of policy allow and deny decisions, not of agent actions generally, and centralised log collection is something the operator wires up rather than a service NVIDIA provides.
California Makes the Decision Not to Audit a Documented One
Added 6 October 2026.
On 9 September 2026 the Governor of California approved and filed four AI bills: SB 813 (chapter 179), AB 1405 (chapter 178), SB 1119 (chapter 190) and SB 867 (chapter 189). An executive order, N-9-26, followed on 18 September.
All four are enacted but not yet in force. Each carries its own operative date, and those dates differ; check the instrument rather than the signing announcement before you plan against any of them. The executive order is not a rule at all — it directs officials to do work.
The structural change worth planning for
SB 813 does not require anyone to be audited. What it does is make an audit legally weighted: performing one is provided to be relevant to, but not conclusive of, questions in AI harm litigation.
That cuts both ways and the second way is the one to plan for. If having an audit is relevant evidence, then not having one becomes a governance decision somebody made, visible in discovery, made at a point in time, by a named person, with or without a recorded reason.
The practical step is small and worth taking now: record the decision. If you have assessed whether to obtain an independent audit of a given system and concluded not to, write down who decided, when, and on what basis. A reasoned “not yet, because X” is a defensible position. Silence is not a position at all, and it will be read as one.
AB 1405 builds the other half: mandatory registration of AI auditors with the Government Operations Agency. The significance for a buyer is that California is constructing a register — which means that, in due course, “we were audited” becomes a checkable claim about a registered person rather than an unverifiable assertion about a consultancy. Ask any prospective auditor what their registration position will be.
SB 1119, known as Adam’s Law, imposes duties on operators of companion chatbots: annual risk assessments on new or updated designs, a crisis protocol for suicidal ideation, parental controls and notifications, an age-determination step, and independent third-party child safety audits, with a private right of action and per-child civil penalties. SB 867 addresses children’s safety alongside it. If you operate anything a user could reasonably treat as a companion — and that definition is broader than products marketed as such — this is the instrument to read in full.
Executive Order N-9-26 directs the Government Operations Agency to accelerate implementation of SB 813 and AB 1405 and to convene national experts to make recommendations on embedding independent verification organisations onsite at frontier AI companies, on independently verifying safety frameworks and risk assessments, and on advancing an emergency shutoff capability. Nothing in it binds anyone yet. It is a signal about direction, and the direction is toward external verification rather than self-attestation.
Assessing the People Who Assess You
Added 6 October 2026.
Every assessment framework asks you to evaluate suppliers. Almost none ask you to evaluate the supplier doing the evaluating.
In May 2026 a frontier model being tested in a contracted evaluation environment authenticated into protected systems at three real companies. The evaluator’s own published account names two causes: internet egress from the environment was enabled, and a fictional target name invented for the exercise collided with a real registered domain. The full entry is the evaluation supply chain.
The failure was in the test harness, not in anything the labs had built — and because that evaluator serves several labs, one misconfiguration reached beyond any single customer.
Four questions to add to supplier assessment wherever an external party tests, audits, red-teams or sandboxes your systems:
- What can their environment actually reach? Not what the design says. Has egress been tested from inside it, through every service it can touch? “No internet access was configured” is a statement about intent — the same point this guide makes about testing your own boundaries, applied to someone else’s.
- Are their synthetic identifiers verified as synthetic? Fictional company names, domains, IP ranges and account numbers used in testing should come from reserved ranges, or be registered by the tester. The evaluator here said it does run background checks and that this name slipped through because the real domain was not widely known. Obscurity is not availability.
- Who else do they serve, and would you know? Concentration risk in assurance is real and almost never on a register. If one supplier’s error would reach you and your competitors in the same week, that is worth recording even though you cannot do much about it.
- What is their disclosure commitment to you? In this case the labs were notified in late July and the first public confirmation came in mid-September, after a journalist asked. Agree in advance what you will be told, how quickly, and what you are free to say.
The counterweight belongs in the assessment too. The only first-party root-cause account of that incident was published by the party at fault, naming its own failed control. An assurance supplier that discloses its own errors candidly is demonstrating the capability you are buying, and a procurement process that penalises that candour will select for suppliers who say less.
Healthcare: Accepting Everything Is Not Delivering Anything
Added 12 October 2026.
On 6 October 2026 the UK government published its response to the National Commission on the Regulation of AI in Healthcare, whose 44 recommendations had been published on 10 September.
It accepted all 44, without qualification. There are no partial acceptances and no acceptances in principle anywhere in the document. That is unusual and it is a genuine signal of direction.
It is also a policy paper, and it commences nothing. No legal duty arises from it. The reforms it points toward would require secondary legislation that has not been made, and the response’s own legislative language is prospective and conditional throughout — the MHRA will then use secondary legislation; it will consider options, including secondary legislation; this may include seeking potential licensing powers.
Read the dates before you plan against them
This is where a healthcare organisation will most easily misread the document.
- 43 of the 44 responses reference Spring 2027. For 42 of them, what is promised by that date is “further details” — not delivery. An implementation plan and roadmap is due then.
- Only two commitments carry a substantive date: draft guidance on Predetermined Change Control Plans by December 2026, and a consultation on how AI-enabled products qualify and are classified by Spring 2027.
- Around a third of the 44 responses commit only to explore, consider or review.
Ownership, which the response does allocate clearly: the MHRA leads 31 of the 44 commitments, the Department of Health and Social Care with the devolved health departments 11, and NICE and the Health Research Authority one each.
The accurate summary is: full acceptance, thin delivery, one real date in 2026. Anything that presents these as incoming duties with a 2027 deadline is reading an implementation roadmap as a compliance calendar.
What a healthcare deployer can usefully do now
The direction is lifecycle evidence rather than point-in-time approval, and that is worth preparing for regardless of when the instruments land. None of this depends on the legislation being made.
- Build the local evidence map. For every AI use in your organisation: the named owner, what it does, the population it affects, the monitoring in place, the thresholds, and the reporting route when a threshold is crossed. This is the artefact every strand of the response eventually asks for.
- Separate the manufacturer’s evidence from yours. The manufacturer evidences the product. You evidence its behaviour in your workflow, on your population, with your staff. No regulatory change will transfer the second to the first.
- Measure across population groups, not in aggregate. Performance that holds overall and degrades for a subgroup is the failure mode this reform is aimed at, and it is invisible to a single accuracy figure.
- Plan for change control. Predetermined Change Control Plans carry the only 2026 date in the document. If your supplier updates a model, you need to know in advance what is allowed to change without re-approval, and what is not.
- Decide what a patient is told, and when. Transparency to patients recurs throughout the recommendations and is entirely within your gift today.
The general method is in the assessment process above; the monitoring and escalation side is in building a tested incident response capability.