Skip to content

Worksheet — Pilot retrospective and go/no-go

Phase: end of 1: Pilot; gates entry to Phase 2. Duration: 120 minutes. Primary output: Signed go/no-go/extend recommendation + Decision Record.

Why this meeting matters

The retrospective is where the pilot earns its decision. Without a structured retrospective against the scope statement, three things tend to happen:

  1. The program inherits the vendor's framing of success ("lots of acceptances, lots of engagement") instead of its own.
  2. Costs and near-misses get downplayed because nobody wants the pilot to look like a failure.
  3. The scale-up plan starts before anyone has decided whether scaling is the right answer.

A retrospective run well prevents all three. It asks whether the pre-agreed success criteria were met, what the data actually showed, and whether the answer is go, no-go, or extend with a specific new question.

Pre-work

Who to invite

Sizing guide

  • Small (< 50): The whole pilot team + sponsor + Ops/Security contact. 90 min.
  • Mid (50–500): Pilot team + sponsor + CoE Lead + Security rep + DevEx. 120 min.
  • Enterprise (500+): Pilot team + sponsor + CoE + Security + Legal + Privacy + Comms + BU leader. 120 min technical; separate 60 min exec readout.
  • Regulated: add Compliance and Audit. Expect a written report addressable to the regulator.

In the room:

  • Pilot team: every member, not just the lead.
  • Executive sponsor: they own the decision.
  • CoE Lead (or equivalent): carries learnings forward to Phase 2.
  • Security rep: owns the safety readout.
  • DevEx: owns the metrics readout.

Materials to circulate 48 hours in advance

  1. Pilot scope: the pre-agreed success/failure criteria.
  2. Metric snapshot: each success criterion, measured value, baseline, trend. No commentary yet.
  3. Incident log: full, with root cause and mitigation per entry.
  4. Survey results: pre-pilot and post-pilot, same questions.
  5. Cost actuals: per-seat, training, time-to-first-value.

Agenda (120 minutes)

Time Topic Owner
0–10 Purpose, ground rules, restate the scope statement. Facilitator
10–30 Criterion-by-criterion review. Did each success criterion pass? DevEx + Facilitator
30–50 Incidents walk-through. What happened, root cause, mitigation. Pilot lead + Security
50–65 Team voice. What's it actually like to work this way? Pilot team members
65–80 What surprised us. (Good and bad.) All
80–95 Recommendation drafting. Go / no-go / extend. Sponsor + Facilitator
95–115 Phase 2 conditions (if go) or learning outputs (if no-go). All
115–120 Sign, schedule follow-up. Sponsor

Discussion prompts

Criterion-by-criterion prompts

For each criterion in the pilot scope:

  • What was the target? (Restate, don't paraphrase.)
  • What was the measured value? With confidence interval if relevant.
  • Was the criterion met, missed, or ambiguous?
  • If met, is the effect likely to persist at scale? (A 20% lead-time reduction on one team may not replicate to 50 teams.)
  • If missed, by how much? Near-miss vs. clear failure change the recommendation.
  • Were there confounds? (A holiday period, a release freeze, a dependency change: these can move metrics independently of AI use.)

Incidents-walkthrough prompts

For each incident:

  • Severity. Using the IR taxonomy (see Incident tabletop).
  • Detection. How did we find it? How long did it take?
  • Root cause. What was the underlying pattern?
  • Mitigation. What was applied, and is it permanent or temporary?
  • Recurrence risk. Would the same mitigation prevent the same incident at 10x scale?

Then aggregate:

  • What pattern do the incidents reveal? (Prompt-hygiene? Review gaps? Vendor flakiness?)
  • What does the pattern imply about Phase 2 controls?

Team-voice prompts

The numeric data doesn't capture experience. Ask the team:

  • What part of your week changed the most?
  • When did the tool actively help? Was there a category of task that got easier?
  • When did the tool actively hurt? Was there a category where you had to work around it?
  • How did your verification behavior change over the 90 days: got more or less careful?
  • Did you ever ship something you weren't fully confident in because the tool "had it handled"?

Record the answers verbatim. Direct quotes are more useful for the eventual scaled rollout than paraphrases.

Surprise prompts

  • What worked better than we expected?
  • What worked worse than we expected?
  • What did the pilot teach us about the business that wasn't about AI? (Pilots often reveal unrelated friction; note it but don't blame it on AI.)
  • What's the most important thing the exec sponsor needs to know that isn't on the dashboard?

Recommendation-drafting prompts

There are only three honest answers. Name them and defend the choice.

Go.

  • Criteria met with sufficient margin and no unresolved Category 2+ incidents.
  • Phase 2 scaling plan is credible and resourced.

No-go.

  • Criteria missed with clear pattern and no near-term remediation.
  • Or: criteria met but discovered unacceptable risk not in the original model.

Extend with a specific question.

  • Criteria ambiguous because of a confound; extend to resolve.
  • New question is named explicitly. "We're not ready" is not a new question.

Phase 2 conditions

If the answer is go, the retrospective produces explicit conditions for Phase 2, not a blanket approval. Typical conditions:

  • Waves sized to ≤ N engineers for the first M months.
  • CoE stood up with named lead before wave 1 expansion.
  • Incident X mitigation made a pre-merge gate before wave 2.
  • AUP v1.1 incorporating pilot findings before cohort kickoff.
  • Metric refresh cadence monthly (not weekly).
  • Review trigger dates for this DR (e.g., mid-Phase 2).

No-go outputs

If the answer is no-go, the retrospective still produces useful outputs:

  • A learning summary for future re-attempts.
  • Risk register updates: what Phase 1 showed us about risks Phase 0 didn't anticipate.
  • Tool-selection implications: what the next scorecard should weight differently.
  • Timeline for re-attempt (e.g., 6 months, after a specific gating change: vendor update, regulation change).

Retrospective document template

Pilot Retrospective — PILOT-0001

Date: 2026-08-04 Pilot window: 2026-05-05 → 2026-08-04 Scope reference: link to scope statement

Executive summary. Three bullets. Primary recommendation stated.

Criteria outcomes.

Criterion Target Measured Met? Notes
Yes / No / Partial

Incidents.

Date Severity Root cause Mitigation Recurrence risk

Team voice (selected quotes).

  • "quote"
  • "quote"

Surprises.

  • positive
  • negative

Recommendation.

Go / No-go / Extend: recommendation.

If go — Phase 2 conditions.

  1. ...
  2. ...

If no-go — Learning summary.

  1. ...
  2. ...

If extend — New question to resolve. One paragraph. Specific. Time-boxed.

Sign-off.

Role Name Date
Exec sponsor
CoE Lead / Pilot lead
CISO
Legal (for regulated decisions) 
# Pilot Retrospective — PILOT-0001

**Date:** 2026-08-04
**Pilot window:** 2026-05-05 → 2026-08-04
**Scope reference:** <link to scope statement>

## Executive summary

<Three bullets. Primary recommendation stated.>

## Criteria outcomes

| Criterion | Target | Measured | Met? | Notes |
|---|---|---|---|---|
| | | | Yes / No / Partial | |

## Incidents

| Date | Severity | Root cause | Mitigation | Recurrence risk |
|---|---|---|---|---|
| | | | | |

## Team voice (selected quotes)

- "<quote>"
- "<quote>"

## Surprises

- <positive>
- <negative>

## Recommendation

**Go / No-go / Extend:** <recommendation>.

### If go — Phase 2 conditions

1. <...>
2. <...>

### If no-go — Learning summary

1. <...>
2. <...>

### If extend — New question to resolve

<One paragraph. Specific. Time-boxed.>

## Sign-off

| Role | Name | Date |
|---|---|---|
| Exec sponsor | | |
| CoE Lead / Pilot lead | | |
| CISO | | |
| Legal (for regulated decisions) | | |

Outputs

  1. Signed retrospective (2–3 pages).
  2. Decision Record: captures the recommendation and its alternatives.
  3. Phase 2 scope outline (if go) for later detailed planning.
  4. Learning summary (if no-go or extend).
  5. Artifacts delivered to CoE: incident log, survey results, metric snapshots, prompt-library contributions.

Follow-ups

  • Within 7 days: retrospective published (redacted as needed).
  • Within 14 days: Phase 2 kickoff or no-go announcement made.
  • Within 30 days: Decision Record circulated for sign-off.
  • Within 90 days (for no-go): learning summary reviewed and re-attempt criteria discussed.

Facilitator tips

  • Reread the scope statement at the top. Before any numbers. "Here's what we said we'd do, here's what we measured" is the frame.
  • Do not edit the incident log live. If a team member remembers an incident not in the log, capture it in a "missing incidents" note. Editing live erodes trust in the artifacts.
  • Separate facts from interpretations. "PR lead time fell 22%" is a fact. "Because the tool made the team faster" is an interpretation. Call both out.
  • Push the sponsor to state the recommendation aloud. A typed recommendation in a doc is weaker than the sponsor saying "I recommend go, with Phase 2 conditions as drafted".

Anti-patterns

Don't

  • Don't extend to avoid a no. Extensions resolve new questions. Without a new question, an extension is avoidance.
  • Don't attribute every good signal to the tool and every bad signal to the team. Holiday periods, release freezes, infra-change windows, and market events all move metrics.
  • Don't omit the team's voice from the write-up. The dashboard gives you numbers. The team gives you the why.
  • Don't let a single loud voice carry the recommendation. If the room is split, extend to resolve the split with a specific follow-up.

Next