Who this questionnaire is for
Procurement, security, legal/compliance, risk, IT, and product leaders evaluating AI vendors, tools, model providers, or platforms.
What it assesses
Whether a vendor can meet real operational and governance requirements: evidence/log exportability, auditability, access controls, data handling, incident support, evaluation discipline, failure-path behaviour, and contractual clarity on risks.
How it helps
Stops “demo-driven buying.” Results translate requirements into concrete proof points (what the vendor must show, log, export, and support) so you don’t discover gaps after rollout — when failures are expensive and public.
Best used when
- Selecting a vendor/model/provider
- Renewing or expanding contracts
- Introducing AI into regulated/high-stakes workflows
- Comparing vendors beyond feature checklists
AI Vendor & Procurement Readiness
This assessment helps teams avoid demo-driven purchasing. It focuses on evidence exportability, controllability, evaluation discipline, and audit/incident readiness.
Section A — Evidence & Exportability
1) Can the tool export raw logs (inputs, retrieval context, tool calls, outputs) in a structured format?
2) Can you reproduce results later (versioned model/prompt/data + timestamps)?
3) Are costs and rate limits predictable at scale (quotas, per-seat/per-token clarity, alerts)?
4) Can you migrate away without losing critical data (configs, evals, embeddings, logs)?
5) Can the vendor support audits (retention, access logs, incident disclosures, evidence trail)?
Section B — Security & Control Surface
6) Are permissions least-privilege with roles, audit logs, and periodic reviews?
7) Can you enforce groundedness (allowlists, citations, refusal gates) inside the product?
8) Is security posture clear (data handling, training use, retention, encryption) and kept current?
9) Can you define “high-stakes” workflows with human review and escalation paths?
10) Can you tune behaviour (policies/configs) with versioning and rollback?
Section C — Evaluation & Contract Discipline
11) Do you require failure-path demos (empty retrieval, contradictions, injection, tool errors) before purchase?
12) Can you run structured evaluations (not demos) and compare runs over time?
13) Do contracts include incident expectations (SLAs, breach notification, support, postmortems)?
14) Do contracts include evidence expectations (export logs, retention, audit support) rather than “trust us”?
15) Do you have an exit plan before signing (data export, replacement timeline, contingency)?
Tip: If you can’t export evidence, you can’t govern or defend decisions later — and you can’t learn from incidents.
Posture
—
—
Decision
—
—