Skip to content

Worksheet — Vendor selection scorecard

Phase: 1: Pilot; repeated when vendors change materially or yearly in Phase 3. Duration: 90 minutes. Primary output: Completed scorecard + vendor recommendation + exit plan + Decision Record.

Why this meeting matters

Tool selection is the most visible decision in the adoption program, which is why vendors work hardest to influence it. Without a structured scorecard, selection collapses into either (a) "whoever the CTO used at their last job" or (b) "whoever had the best demo." Both decisions tend to survive past their sell-by date.

A scorecard is also what protects you at renewal time. When the vendor asks for a price increase or quietly updates terms, you compare the new offer against the scorecard you used to choose them. If the rationale has eroded, the comparison surfaces it.

Pre-work

Who to invite

Sizing guide

  • Small (< 50): Technical founder + Ops/Legal contact. 45 min. Three vendors max.
  • Mid (50–500): Procurement + CISO + CoE Lead + Eng rep + Legal. 90 min. Three vendors.
  • Enterprise (500+): Procurement + CISO + CoE Lead + Legal + Privacy + Engineering Architect + Finance (per-seat cost modelling). Two sessions: data gathering, then scoring. Three to five vendors.
  • Regulated: add Compliance and sector-specific reviewer (e.g., HIPAA Privacy Officer; DORA ICT risk for EU financial services).

In the room:

  • Procurement: owns contract and commercial terms.
  • CISO / Security: owns risk assessment.
  • Legal: owns IP and liability language.
  • Privacy: owns data-handling terms.
  • CoE Lead or Eng rep: owns fit-for-purpose assessment.

Explicitly not invited: the vendors themselves. Demos happen separately; the scoring room does not host sales.

Materials to circulate 48 hours in advance

  1. Vendor-provided documentation from each candidate (DPA, sub-processors list, security addendum, SOC 2 Type II, ISO certs, IP indemnification language, pricing sheet).
  2. Leadership charter: scope and stop conditions.
  3. Risk classification: which tiers the tool must support.
  4. Scorecard template (below).
  5. Public evidence file: at minimum the IP litigation status (Doe v. GitHub class action context1) and vendor incident history (DeepSeek2, past vendor outages).

Scorecard dimensions

Each vendor scored 0–3 on each dimension (0 = fails minimum, 1 = meets floor with caveats, 2 = meets all stated needs, 3 = exceeds).

Dimension What "meets floor" typically means
Data handling No training on your data by default; clear opt-out; published data retention; regional residency options.
Enterprise controls SSO (SAML / OIDC), SCIM, audit logging, role-based access, tenant isolation.
IP indemnification Contractual indemnity for IP infringement in generated output, with clearly-scoped exceptions.
Security attestations SOC 2 Type II or ISO 27001 at minimum; ISO 42001 credit if present.
Regulatory fit GDPR-compliant processing (if EU); sector-specific attestations (HIPAA BAA if applicable; PCI if applicable).
Agent capabilities and guardrails Documented agent controls: can it write to prod? Can we forbid it? Is there tool-use logging?
Operational reliability SLAs (including during incidents); publicly visible status page with history; RCA process.
Ecosystem and integration IDE support, version-control integration, CI/CD touchpoints, review-tool plugins, matching your stack, not the vendor's ideal stack.
Exit terms Data export format documented; contract termination clause reasonable; no hostage-style renewal terms.
Pricing model and predictability Clear per-seat or usage pricing; caps or predictable escalation; behavior during a license overage.

Minimum-score gate

A vendor that scores 0 on any dimension is disqualified regardless of other scores. The dimension floor is real.

Weighting

Different companies weight differently. Suggested weights:

Company class Highest-weight dimensions
Startup, greenfield Ecosystem, pricing, data handling
Mid-market SaaS Data handling, IP indemnification, security attestations
Enterprise, multi-region Regulatory fit, enterprise controls, exit terms
Regulated (finance/health) IP indemnification, regulatory fit, agent guardrails

Record the weighting in the DR.

Agenda (90 minutes)

Time Topic Owner
0–10 Purpose, rules, confirm disqualification criteria. Facilitator
10–30 Silent scoring round 1. Each participant scores each vendor on each dimension, independently. All
30–55 Discuss divergences. Scores more than 1 apart: talk through evidence; adjust. Facilitator
55–70 Apply weights, compute totals. Any minimum-score gate failures? Facilitator + Procurement
70–85 Recommendation and exit plan. Who is primary? Who is secondary (fall-back)? What would make us switch? All
85–90 Sign off recommendation; owner for Legal review of the chosen vendor's contract. Procurement

Discussion prompts

Data-handling prompts

  • Does the vendor train on our data by default, with a separately-available opt-out? If the opt-out is per account, is it enforceable across our tenant?
  • What happens to our prompts and completions after they are processed? Where are they stored, for how long, and in what jurisdiction?
  • What sub-processors does the vendor use? Are those sub-processors' terms acceptable to Privacy/Legal?
  • Is there a BAA available (for HIPAA contexts)? SCCs / IDTA (for EU)?
  • Named test: if a pilot engineer pasted customer data, what would the vendor tell us when we asked "what did you retain?"

Enterprise-controls prompts

  • Is SSO available and included (not gated by a higher tier)?
  • Is audit logging per user, not just per tenant? Exportable to our SIEM?
  • How fine-grained are role controls?
  • Does SCIM de-provisioning actually revoke token access, or just disable login?

IP-indemnification prompts

  • Is indemnification uncapped for IP infringement, or capped at fees paid?
  • What obligations does indemnification place on us (e.g., "must use current version")?
  • Are there carve-outs (e.g., prompts that ask for a specific open-source project by name)?
  • What does the vendor do if a customer receives a takedown notice for AI-generated code?

Security-attestation prompts

  • Do we have an executed SOC 2 Type II report from the current audit period, not an outdated one?
  • Are the scope boundaries of the attestation aligned with the product we'd use?
  • Has the vendor had an independent pen-test of the AI surface specifically?

Agent-capability prompts

  • Can we disable agent write access to production? In what granularity?
  • Is there tool-use logging we can ingest?
  • What is the default permission set for agents?
  • Does the vendor publish a documented model behavior spec or safety evaluation?

Exit prompts

  • What format is the data export? How long does it take? Can we run a test export under the pilot contract?
  • What does the vendor's contract say about our use of the tool after termination? Our continued use of completions that were generated during the term?
  • If the vendor is acquired or merges, what are our termination rights?

Scorecard template

Vendor Scorecard — AI Coding Assistants

Date: 2026-06-01 Evaluators: roles Candidates: A, B, C

Weighting (must total 100).

Dimension Weight
Data handling 15
Enterprise controls 10
IP indemnification 15
Security attestations 10
Regulatory fit 10
Agent guardrails 10
Operational reliability 10
Ecosystem / integration 8
Exit terms 7
Pricing predictability 5
Total 100

Scores (0–3 per dimension per vendor).

Dimension Vendor A Vendor B Vendor C
Data handling
Enterprise controls
IP indemnification
Security attestations
Regulatory fit
Agent guardrails
Operational reliability
Ecosystem / integration
Exit terms
Pricing predictability
Weighted total

Disqualifications.

  • Vendor X scored 0 on because .

Recommendation.

Primary: Vendor A (score X / 3). Fallback / evaluation-in-parallel: Vendor B (score Y / 3). Not adopted: Vendor C (reasons).

Exit plan. What would cause us to switch from A to B?

  • event / metric / contract change 1
  • event / metric / contract change 2

How long would the switch take, and who owns the runbook?

Named role, rough weeks-to-switch estimate.

# Vendor Scorecard — AI Coding Assistants

**Date:** 2026-06-01
**Evaluators:** <roles>
**Candidates:** <A>, <B>, <C>

## Weighting (must total 100)

| Dimension | Weight |
|---|---|
| Data handling | 15 |
| Enterprise controls | 10 |
| IP indemnification | 15 |
| Security attestations | 10 |
| Regulatory fit | 10 |
| Agent guardrails | 10 |
| Operational reliability | 10 |
| Ecosystem / integration | 8 |
| Exit terms | 7 |
| Pricing predictability | 5 |
| **Total** | **100** |

## Scores (0–3 per dimension per vendor)

| Dimension | Vendor A | Vendor B | Vendor C |
|---|---|---|---|
| Data handling | | | |
| Enterprise controls | | | |
| IP indemnification | | | |
| Security attestations | | | |
| Regulatory fit | | | |
| Agent guardrails | | | |
| Operational reliability | | | |
| Ecosystem / integration | | | |
| Exit terms | | | |
| Pricing predictability | | | |
| **Weighted total** | | | |

## Disqualifications

- <Vendor X scored 0 on <dimension> because <reason>.>

## Recommendation

**Primary:** Vendor A (score X / 3).
**Fallback / evaluation-in-parallel:** Vendor B (score Y / 3).
**Not adopted:** Vendor C (reasons).

## Exit plan

What would cause us to switch from A to B?

- <event / metric / contract change 1>
- <event / metric / contract change 2>

How long would the switch take, and who owns the runbook?

<Named role, rough weeks-to-switch estimate>

Outputs

  1. Completed scorecard.
  2. Decision Record capturing weights, scores, recommendation, dissent.
  3. Exit plan: which fallback is pre-approved, under what conditions, and how fast the switch is feasible.
  4. Legal & Security review list: specific contract clauses and controls requiring deeper review before signature.
  5. Negotiation targets: pre-identified terms (indemnification cap, pricing floor, data residency) where the vendor's current position is below floor and negotiation is planned.

Follow-ups

  • Within 7 days: Legal red-line on the chosen vendor's contract.
  • Within 14 days: CISO sign-off on residual risk.
  • Within 30 days: signed contract or revert to fallback.
  • Every 12 months: re-run scorecard with the same weights; if weights need to change, that is a separate DR.

Facilitator tips

  • Score on evidence, not on demo impressions. Each score should cite a document or observed fact.
  • Separate "cool capability" from "capability that solves our problem." Many AI tools score high on demos but fail on enterprise controls, which is the dimension that matters in practice.
  • Treat the fallback seriously. A fallback that only exists on paper is no fallback. At minimum, the fallback vendor should have an executed NDA and a scoped evaluation plan.
  • Don't renegotiate the weights during scoring. If the weights feel wrong, stop; re-weight; re-score. Don't slide scores to match a preferred vendor; that's how the scorecard becomes a rubber stamp.

Anti-patterns

Don't

  • Don't skip the silent scoring round. Anchoring to the loudest voice is the single most common failure of scorecard sessions.
  • Don't accept a vendor's self-scored answers. Evidence, not responses to a security questionnaire.
  • Don't ignore exit terms. Onboarding is easy; offboarding is where vendors add friction.
  • Don't freeze on one vendor indefinitely. Cursor's rise against Copilot in 2024–2025 surprised procurement teams that had no fallback evaluation. Keep the fallback current.

Next

  1. Doe v. GitHub, OpenAI, Microsoft class action, filed Nov 2022. Most of the claims were dismissed in subsequent rulings, but contractual indemnification language has become standard market practice. ↩

  2. Wiz Research, "Wiz Research Uncovers Exposed DeepSeek Database", Jan 2025. ↩