**The AI Compliance Checklist Conundrum**
A decade ago, a simple surgical checklist revolutionized healthcare by cutting complications and deaths across eight hospitals worldwide. The list consisted of just 19 items, printed on a single card. Aviation had learned a similar lesson earlier: the pre-flight checklist is concise, not comprehensive. Yet, in the world of AI security, we’re still struggling with an over-reliance on massive frameworks and questionnaires that promise more than they deliver.
The timing couldn’t be more critical. The EU AI Act’s enforcement teeth for general-purpose AI arrive this August, and high-risk obligations are phasing in behind them. NIST’s AI Risk Management Framework has become the default answer for “show me you have an AI risk program” in North America. But what we don’t have is a simple, effective way to assess AI security.
The frameworks themselves largely agree, with substantial overlap between ISO 42001, NIST AI RMF, and the EU AI Act. An organization that builds its program thoughtfully can satisfy all three with a single set of processes and documentation. The problem lies in what happens downstream, when those frameworks get translated into questionnaires, audits, and attestations.
The Questionnaire Problem
If you’ve been on the receiving end of an AI security questionnaire lately, you know the drill. Hundreds of questions, free-text answers, and prompts like “Describe how your AI system ensures fairness” or “Explain your approach to responsible AI.” These questions have three critical flaws. Firstly, they can’t be answered with evidence; only prose is acceptable. And let’s face it: creative writing isn’t compliance; it’s fiction masquerading as fact.
Secondly, these questions ignore the nature of the systems being assessed. Large Language Models (LLMs) are stochastic systems that are extremely difficult to replay and troubleshoot. A point-in-time attestation about model behavior is stale the moment a model version changes or a system prompt is updated. Asking “does your model produce biased outputs?” as a yes/no compliance question fundamentally misunderstands what these systems are.
Thirdly, these questions don’t scale with risk. The same 300-question addendum is sent to vendors running marketing chatbots and clinical decision support systems, regardless of their actual risk profile.
**What Usable Actually Looks Like**
I propose a different bar for AI compliance: one that borrows directly from the checklist lesson. A framework is only as good as its worst question. Before any question makes it into your AI assessment, whether that’s a vendor questionnaire or an internal review gate, it should pass five tests:
1. Answerable with an artifact: every question should map to evidence, such as a log, config, eval report, data flow diagram, or architecture document.
2. Scoped to risk tier: classify the system first, then ask the questions that tier deserves.
3. Context-specific: questions should be tailored to the specific AI system being assessed.
4. Actionable: questions should prompt real actions, not just creative writing exercises.
5. Testable: questions should be designed to elicit evidence of actual practices and policies.
By adopting this approach, we can finally move beyond the compliance paperwork and focus on creating effective, risk-based AI security programs that truly protect users and organizations alike.
Source: SecurityWeek — 2026-07-30