Skip to content

Worksheet — Metrics baselining

Phase: 0: Readiness; referenced throughout every later phase. Duration: 90 minutes. Primary output: Baseline snapshot document + gap list + Decision Record.

Why this meeting matters

"We feel faster" is not a measurement. If you start the pilot without a baseline, the post-pilot retrospective becomes an argument about narrative instead of evidence. That is how programs get continued long past the point where they should have been adjusted.

DORA's 2024 report found AI adoption correlated with a -1.5% throughput change and a -7.2% delivery stability change, even as self-reported productivity rose 2.1%1. That delta (feels better, measures worse) is one of the most important findings in the space, and it is only visible if you measure before and after.

Pre-work

Who to invite

Sizing guide

  • Small (< 50): Senior engineer + Ops contact. 45 minutes. Baseline may be qualitative — say so explicitly.
  • Mid (50–500): DevEx lead + SRE + Engineering Manager + CISO rep. 90 minutes.
  • Enterprise (500+): Head of DevEx / Platform + SRE Lead + AppSec + Finance (for cost baselines) + BU leads. Two sessions: metric selection and baseline production.
  • Regulated: add Audit Liaison so baseline can be used as evidence later.

In the room:

  • DevEx or Platform lead: owns the metrics pipeline.
  • SRE / Production engineer: owns availability and MTTR signals.
  • Engineering Manager: owns team-level measures.
  • AppSec: owns security signals.
  • Finance contact, if cost baselines are in scope.

Materials to circulate 48 hours in advance

  1. Current dashboards: whatever already exists. Pre-work: each invitee brings links to what they can pull in an hour.
  2. Measurement: metrics taxonomy: the canonical definitions of the metrics below.
  3. The last two quarters of incident data: source for stability baselines.
  4. The last two quarters of PR data: source for cycle/throughput baselines.

The metric families

Family What it measures Typical source
DORA four-keys Delivery performance Deployment tooling, CI, incident system
SPACE (optionally) Developer productivity holistically Surveys + telemetry
Quality Code durability and correctness PR system, test suite, GitClear-style churn
Security AppSec posture SCA, SAST, pen-test, vulnerability counts
Developer experience / satisfaction Survey signals eNPS-style, DevEx pulse
Financial (optional) Cost-to-deliver Finance + engineering leadership

Agenda (90 minutes)

Time Topic Owner
0–10 Purpose and rules: "if you don't know, that's a finding". Facilitator
10–30 DORA four-keys — record each with a source and confidence. DevEx + SRE
30–45 Quality — churn, PR review time, defect rate, test coverage trend. Eng Mgr
45–60 Security — open criticals, mean-time-to-remediate, escape rate. AppSec
60–75 Developer experience — last survey date, key score, target frequency. Eng Mgr
75–85 Cost / financial (optional) — spend per engineer on tooling, headcount, output. Finance
85–90 Consolidate. Who commits to the gap-closure plan? Facilitator

Discussion prompts

DORA four-keys prompts

For each of the four keys, ask: definition, current value, source, confidence, date of last refresh, volatility.

  • Deployment frequency: how often do we deploy to production? Per team, per service, or aggregate? Pick the level the pilot will report at.
  • Lead time for changes: commit-to-production for the service in scope.
  • Change-failure rate: % of deployments that cause incident or rollback. If we can't compute this, what would it take?
  • Mean time to restore: from detection to customer-observable recovery.

Capture each as a value and a trend: "14 per week, flat 3 months" is more useful than "14 per week".

Quality prompts

  • Churn: % of code reverted or substantially rewritten within 14 days. GitClear's 2024 reports on AI-era churn are a useful external reference2.
  • PR review time: from first review request to merge.
  • Escape defect rate: defects caught post-release vs. pre-release, over time.
  • Test coverage trend: percentage and trajectory. Coverage alone is not quality, but a declining trend during adoption is a signal.

Security prompts

  • Open criticals: count and trend.
  • Mean time to remediate (critical): from detection to fix.
  • Vulnerabilities per release: over the last 90 days.
  • Pre-merge gates: which scans are blocking, which are advisory? (This becomes a knob in Phase 1; record today's state.)

DX / satisfaction prompts

  • Last survey date and scope.
  • Key question answered. "Would you recommend working here to a friend?" (eNPS) is a recurring signal; DevEx pulse questions are more specific.
  • Frequency target. Quarterly is a reasonable floor during Phase 1.
  • Will the same survey run at pilot start and end? It must.

Financial prompts (optional)

  • Engineering cost per shipped feature / per line of code / per story-point. Be cautious about these; they all have perverse incentives. Still, a baseline lets you notice movement.
  • Tooling spend per engineer.
  • Estimated cost of incidents over the last two quarters.

Baseline template

Each metric follows the same schema. A baseline document is a stack of these blocks.

Metric schema

Metric name:

  • Definition: one sentence
  • Source: system, dashboard, or survey
  • Current value: number, with units
  • As of: date
  • Trend (last 3 months): flat / up / down, with magnitude
  • Confidence: high / medium / low — and why
  • Known caveats: e.g., "includes staging deploys", "3 teams not instrumented"
  • Next refresh: date
## <Metric name>

- **Definition:** <one sentence>
- **Source:** <system, dashboard, or survey>
- **Current value:** <number, with units>
- **As of:** <date>
- **Trend (last 3 months):** <flat / up / down, with magnitude>
- **Confidence:** <high / medium / low — and why>
- **Known caveats:** <e.g., "includes staging deploys", "3 teams
  not instrumented">
- **Next refresh:** <date>

Outputs

  • Baseline snapshot document: one page per metric family.
  • Gap list: metrics we cannot produce today, with a 90-day plan per gap.
  • Decision Record: which metrics are the pilot's primary judgement criteria, and what confidence they carry.
  • Survey plan: instrument the pilot team at start and end with identical questions.
  • Owner for monthly refresh. Baseline is version 1.0; after Phase 0 it needs continuous refresh.

Follow-ups

  • Within 14 days: gaps with "quick fix" status are closed.
  • Within 30 days: survey instrument deployed (pilot start = baseline survey).
  • Within 60 days: mid-pilot refresh.
  • Every month thereafter: monthly pulse (Phase ⅔ cadence).

Facilitator tips

  • Record confidence, not just values. A confident-looking number with "low confidence, CI not instrumented for service X" is more useful than a high-confidence number with unexamined provenance.
  • Resist "we'll start measuring once we have something to show." Baselines must predate the intervention, or they are not baselines.
  • Call out the "feels faster, measures worse" phenomenon upfront. Name it so the team watches for it.
  • Don't chase a 100% complete dashboard. A 60% baseline that ships in Phase 0 beats a 100% baseline that arrives in Phase 2.

Anti-patterns

Don't

  • Don't baseline only productivity metrics. Quality and security counterweights matter most. Without them, Phase 2 can show gains while the underlying codebase degrades.
  • Don't let "acceptance rate" be your pilot measure. Acceptance rate is what vendors track; it does not correlate directly to shipped value.
  • Don't compare teams on baselines. The baseline is a self-comparison tool (team A in June vs. team A in September), not a league table.
  • Don't treat a survey as baseline on its own. Pair it with behavioral data; the gap between the two is your most interesting finding later.

Next

  1. Google Cloud DORA team, "Accelerate State of DevOps 2024", Oct 2024. ↩

  2. GitClear, "AI Copilot Code Quality: 2024 Research Report", 2024. ↩