Worksheet — Pilot scope and success criteria¶
Phase: 1: Pilot. Duration: 60 minutes. Primary output: Pilot scope statement + success criteria + Decision Record.
Why this meeting matters¶
The single biggest determinant of pilot quality is the team and system you pick. Pick an enthusiastic team on a greenfield CRUD service and the pilot will "succeed", and nothing will generalise. Pick the gnarliest legacy system with an uncooperative team and the pilot will fail, and nothing will generalise there either.
A good pilot is representative, bounded, and disprovable. If there is no way for the pilot to fail, you are not running a pilot. You are running a procurement exercise with extra steps.
Pre-work¶
Who to invite¶
Sizing guide
- Small (< 50): Whole company is the "pilot scope" meeting. 30 min.
- Mid (50–500): Exec sponsor + pilot team lead + CoE/Security/DevEx reps. 60 min.
- Enterprise (500+): Exec sponsor + CoE Lead + pilot-team lead + BU leader + Security + DevEx + Data owner (for the system in scope) + Procurement observer. 60–90 min. Parallel BU pilots need parallel meetings.
- Regulated: add Compliance and Audit Liaison. Expect scope conversations to involve DPIAs.
In the room:
- Executive sponsor (from the leadership charter): confirms scope aligns with charter.
- Pilot team lead: will run the pilot day-to-day.
- A candidate pilot team member: not in leadership; bring lived experience of the system.
- Security / AppSec: owns risk view of the pilot system.
- DevEx / Platform: owns instrumentation and metrics.
Materials to circulate 48 hours in advance¶
- Leadership charter: the scope bounds.
- Risk classification output, to pick a system in the right tier.
- Metrics baseline, to anchor success criteria against real numbers.
- 2–3 candidate team + system pairings with pros and cons.
Agenda (60 minutes)¶
| Time | Topic | Owner |
|---|---|---|
| 0–5 | Purpose, ground rules, confirm charter bounds. | Facilitator |
| 5–20 | Pick the team and system. Evaluate the candidates. | Sponsor + Team lead |
| 20–35 | Write success criteria. 3–5 numeric criteria, each with a target. | Sponsor + DevEx |
| 35–45 | Write failure criteria. What would force "no-go"? | Security + Sponsor |
| 45–55 | Bound the scope. Time, people, systems, features explicitly in and out. | All |
| 55–60 | Sign scope statement. | Sponsor |
Discussion prompts¶
Team-and-system selection prompts¶
- Which candidate is representative of the company's normal work? (Not the easiest, not the hardest.)
- Is the candidate team able to run a pilot? (Bandwidth, skills, tooling in place.)
- Is the candidate team willing, or did they volunteer because they already use shadow tools they'd like to legitimise? (Both are OK, but be explicit.)
- Does the candidate system have enough traffic/complexity that 90 days of work generates measurable signal?
- Does the candidate system have existing test coverage? Pilots on untested systems conflate "AI helped" with "any process helps".
- Is the candidate system's risk tier Medium or High? (Critical is usually too risky for a first pilot; Low too unrepresentative.)
| Prefer | Avoid (for first pilot) |
|---|---|
| Brownfield but maintainable | Greenfield with no users |
| Medium-risk classification (not Low, not Critical) | Critical-tier (payments, identity) |
| Team that has complained about toil | Team that's against adoption on principle |
| System with existing test coverage | System with no tests |
| 4–8 week feature of normal scope | "Transformation" of 2+ quarters |
Success-criteria prompts¶
Success criteria should be measurable from the baselines you established.
- Throughput target. Example: "PR lead time reduced by 20% over 90 days on the pilot service, p75, without a concurrent change-failure-rate increase beyond +5%."
- Quality guardrails. Example: "Churn rate on files touched in pilot does not exceed baseline by more than 15%."
- Security guardrails. Example: "No Critical vulnerabilities shipped to production attributable to AI-assisted code."
- Developer experience. Example: "Pilot-team NPS at end ≥ pilot-team NPS at start, with the same survey."
- Cost awareness. Example: "Per-seat spend within budget; per-PR cost not up by more than 10%."
Every criterion must have: metric, target, baseline, and the source of truth.
Failure-criteria prompts¶
- What pattern of incidents would cause an immediate pause?
- What combination of metric movements would convert "continue" to "rollback"?
- What external event (vendor news, regulation) would prompt a separate DR?
State these before the pilot starts. Deferred failure criteria become post-hoc rationalisations.
Scope-bounding prompts¶
- Time: start and end dates. 90 days is the default.
- People: who is in the pilot, who isn't, and how is "accidental expansion" prevented (managers on the pilot team's teams not pulling other teams in)?
- Systems: which repos / services? Explicitly list the ones out.
- Features: which features are in the pilot's scope? Which are out?
- Tools: which tools are approved for the pilot? (Cross-reference Vendor selection.)
- Decisions: who can approve scope adjustments mid-pilot?
Scope statement template¶
Pilot Scope — AI-Assisted Development
Pilot ID: PILOT-0001 Start date: 2026-05-05 End date: 2026-08-04 Mid-pilot checkpoint: 2026-06-13
Team.
- Team name, N engineers, tech lead name.
- Working from office/location/remote.
System in scope.
- Service name, risk tier Medium/High.
- Repository: link.
- Baseline metrics: see link.
Explicitly out of scope.
- Other services / features / roles.
Tools permitted.
- Primary: Tool A, approved tenant.
- Secondary (if any): Tool B, conditions.
Success criteria (all must be met).
- Metric, target, baseline, source
- ...
Failure criteria (any one triggers pause).
- Specific condition
- ...
Decision rights during pilot.
| Decision | Owner |
|---|---|
| Scope adjustments | Pilot team lead + exec sponsor |
| Tool addition/removal | CoE Lead (with CISO sign-off) |
| Pause / rollback | Exec sponsor (or CISO for security) |
Review cadence.
- Weekly metrics pulse (internal to pilot team).
- Fortnightly sponsor readout.
- Mid-pilot checkpoint (1 hour, day 30–45).
- End-pilot retrospective (see Pilot retrospective worksheet).
Sign-off. Exec sponsor, CoE Lead, Pilot team lead, CISO delegate.
# Pilot Scope — AI-Assisted Development
**Pilot ID:** PILOT-0001
**Start date:** 2026-05-05
**End date:** 2026-08-04
**Mid-pilot checkpoint:** 2026-06-13
## Team
- <Team name>, <N> engineers, <tech lead name>.
- Working from <office/location/remote>.
## System in scope
- <Service name>, risk tier <Medium/High>.
- Repository: <link>.
- Baseline metrics: see <link>.
## Explicitly out of scope
- <Other services / features / roles>.
## Tools permitted
- Primary: <Tool A, approved tenant>.
- Secondary (if any): <Tool B, conditions>.
## Success criteria (all must be met)
1. <Metric, target, baseline, source>
2. ...
## Failure criteria (any one triggers pause)
1. <Specific condition>
2. ...
## Decision rights during pilot
| Decision | Owner |
|---|---|
| Scope adjustments | Pilot team lead + exec sponsor |
| Tool addition/removal | CoE Lead (with CISO sign-off) |
| Pause / rollback | Exec sponsor (or CISO for security) |
## Review cadence
- Weekly metrics pulse (internal to pilot team).
- Fortnightly sponsor readout.
- Mid-pilot checkpoint (1 hour, day 30–45).
- End-pilot retrospective (see [Pilot retrospective worksheet](pilot-retrospective.md)).
## Sign-off
<Exec sponsor, CoE Lead, Pilot team lead, CISO delegate>
Outputs¶
- Signed scope statement (~1 page).
- Decision Record citing alternatives considered (other teams / systems), chosen option, and dissent.
- Calendar invites for weekly pulse, sponsor readout, mid-pilot checkpoint.
- Instrumentation checklist: what must be measurable on day 1.
- Dependency list: worksheets and decisions blocked until this ships.
Follow-ups¶
- Within 48 hours: scope statement published to a durable location.
- Within 7 days: instrumentation confirmed by DevEx.
- Within 14 days: training for the pilot team complete (see Training needs).
- Day 30–45: mid-pilot checkpoint.
- Day 85–90: retrospective.
Facilitator tips¶
- Pick deliberately, not by volunteering. If only one team raised their hand, ask "who said no and why?" Their reasons often surface risks.
- Insist on three numeric success criteria. If the group produces five aspirational bullets, ask which three they would measure against.
- Make "do nothing" visible. Add it to the DR: what happens if we don't pilot right now?
- Agree the failure condition with the exec sponsor out loud. The sponsor is the one who will have to pull the cord if it triggers.
Anti-patterns¶
Don't
- Don't pilot with "your best team on the most-appealing system". The results won't generalise.
- Don't let the pilot scope creep. Two services becomes three, becomes five, becomes "we're rolling out". The retrospective then has no clean signal.
- Don't pilot without a timebox. "When it's ready" means "forever."
- Don't run more than 2–3 parallel pilots in Phase 1. Too many pilots swamps the CoE's capacity to respond to incidents.
Next¶
- Vendor selection: sometimes run before, sometimes after pilot scope, depending on whether the tool is a given.
- Training needs: train the pilot team before they touch the tool.
- Pilot retrospective: the end-of-pilot worksheet.