Confident errors
A model can produce incorrect details or invented references.
What to design for
Use agreed sources, structured checks, representative tests and human verification before relying on consequential output.
AI responsibility
Useful automation.
Visible limits.
AI outputs require evaluation. The project should make review, exceptions and responsibility explicit.
Incomplete or misleading inputs
Missing fields, poor scans or hostile instructions in documents can affect an output.
What to design for
Define required inputs, flag uncertainty, restrict tool permissions and test malformed or adversarial samples.
Privacy and external processing
Connected services may process or retain information outside the client’s systems.
What to design for
Map the data flow, approve providers and settings, limit access and document retention before sensitive data is used.
Changed tools and model behaviour
API changes, provider outages and model updates can change a working flow.
What to design for
Define monitoring, regression checks, manual fallback and maintenance responsibilities in the project scope.
Public-facing mistakes
A draft post, email or website can contain inaccurate claims or material that does not fit the brand.
What to design for
Keep approval gates for publication and customer communication; verify rights, facts and intended use.
Over-automation
An exception can require context that the workflow does not have.
What to design for
Identify decisions that remain with people, name an escalation owner and provide a way to pause affected actions.
For consequential decisions
Legal, clinical, financial and other high-impact work needs appropriately qualified review, suitable systems and a separate assessment of the proposed use. A general workflow demo does not establish compliance or suitability for regulated information.
Background: NIST’s Generative AI Profile describes risks including confabulation, information integrity and privacy. Our planning approach does not constitute NIST certification.
Discuss data boundaries before access ↗Bring one familiar task.
Use a free 30-minute call to discuss the checks, access and human review your workflow may need.
Request the intro call