Skip to content

Worksheet — Training needs analysis

Phase: 1 — Pilot; revised in Phase 2 for scaled rollout. Duration: 60 minutes for pilot; 120 minutes for Phase 2 rollout. Primary output: Role-differentiated curriculum plan + Decision Record.

Why this meeting matters

Most AI-coding training is "here's the tool, here are the buttons." That is product training. It is not enough. Engineers and reviewers need to learn:

  • How to prompt for verifiable output, not just plausible output.
  • How to recognize hallucinated dependencies, fake APIs, and stale patterns.
  • When not to use an assistant.
  • Their specific obligations under the AUP.

Training gaps show up as incidents. The 2023 Stanford study by Perry et al.1 found that developers using AI assistants tended to write less secure code while believing it was more secure — the exact failure mode that training is meant to prevent.

Pre-work

Who to invite

Sizing guide

  • Small (< 50): Tech lead + Security contact. Pragmatic 45-min session; 4 hours of training per engineer is plenty at this size.
  • Mid (50–500): Training / enablement owner + 2 tech leads + AppSec. 60 min.
  • Enterprise (500+): Head of Enablement + Eng Learning team + AppSec + BU learning partners + HR rep (for integration with onboarding). 120 min.
  • Regulated: add Compliance for regulatory-training overlay (annual mandatory training, audit evidence).

In the room:

  • Training or enablement owner — builds and delivers curriculum.
  • One or two tech leads — reality check content for their teams.
  • AppSec or security trainer — owns the security modules.
  • A recent hire — perspective on onboarding integration.

Materials to circulate 48 hours in advance

  1. AUP — training must reinforce it.
  2. Pilot scope — who needs training, on what.
  3. Framework pages — S.C.A.F.F., V.E.R.I.F.Y., C.L.E.A.R., S.H.I.E.L.D. etc. form natural curriculum modules.
  4. Incident log / tabletop findings — the training has to address these.

The role matrix

Different roles need different training. Don't deliver a single "intro to AI coding" to everyone.

Role Focus Depth (hours)
Developer Prompting, verification, when not to use AI, secure patterns 8–12
Tech lead Developer content + reviewing AI-generated code, coaching, metrics literacy 12–16
Security reviewer Developer content + AI-specific threat model, IR participation, red-team basics 14–20
Engineering manager Dashboard literacy, over-trust detection, team coaching 2–4
Executive / board What to ask, what not to commit to publicly, governance overview 1–2

Agenda (60 minutes)

Time Topic Owner
0–10 Purpose, AUP recap, role matrix. Facilitator
10–25 Developer curriculum — modules, depth, delivery, hands-on. Training lead
25–35 Tech lead additions. Training lead + tech lead
35–45 Security reviewer additions. AppSec
45–55 Manager / executive tracks. Training lead
55–60 Confirm cadence, owners, certification strategy (if any). All

Discussion prompts

Developer track prompts

Build the course around six modules. Ask: for each module, what is the desired behavioral change in the engineer, and how will we know it happened?

  1. What AI coding tools do and don't do. Ground engineers in real evidence — GitHub's 55% task-speed study2, METR's 2025 study showing slowdown for expert devs on hard tasks3, Stanford's security study1. All are true. Build a balanced mental model.
  2. The AUP — every clause. Walk through the organization's own AUP with examples. Not a link-dump.
  3. Prompting patterns — S.C.A.F.F.. Hands-on with the team's real code.
  4. Verification — V.E.R.I.F.Y. + C.L.E.A.R.. How to review AI-generated code; how verification level maps to risk tier.
  5. Security — S.H.I.E.L.D.. The classes of AI-specific security bugs (prompt injection, hallucinated dependencies, insecure patterns from training data) and how to catch them.
  6. Incident reporting. Walkthrough of the Slack channel, what to preserve, blame-free framing.

Tech lead track prompts

  • How do tech leads coach verification behavior — not just police it?
  • What does a "good AI-adjacent PR review" look like, concretely?
  • How should tech leads read the team's dashboard? What signals warrant a 1-on-1 conversation vs. a team discussion?
  • How do tech leads handle someone on their team who is clearly disengaged from verification ("it works, ship it")?

Security reviewer track prompts

  • How does the AI-specific threat model differ from the standard app threat model?
  • What are the red-team patterns appropriate for internal use — not to compromise production, but to stress-test assumptions?
  • How do security reviewers maintain their skills against an evolving set of vendor defaults and agent capabilities?
  • What is the security reviewer's authority during an AI-adjacent incident (can they pause a tool? escalate to the exec sponsor?)?

Manager track prompts

  • How do managers avoid the trap of using AI metrics (acceptance rate, completions-per-day) as individual performance signals?
  • How do managers detect over-trust in their teams — engineers shipping AI output without real verification?
  • What language should managers use when an incident happens, to keep reporting blame-free?

Executive track prompts

  • What should executives commit to publicly and what should they not? (Shopify's 2025 "reflexive AI usage is baseline" memo and the subsequent reaction4 is an instructive case study.)
  • What board-level questions should the exec sponsor be prepared to answer quarterly?
  • What board-level questions should the exec sponsor avoid answering without the CoE's input?

Delivery prompts

  • Live cohorts, self-paced, or blended? (Blended is the most resilient.)
  • How is hands-on practice incorporated, without letting it devolve into "free time"?
  • What is the refresh cadence? (At least 6 months; ideally quarterly modules for security reviewers.)
  • How is training integrated with onboarding? What is the Day-1 and Day-30 training picture?

Assessment prompts

  • Is there a certification? Internal (pass/fail on a scenario set) or external ([Copilot GH-300]5, [AWS AIF-C01]6, ISO 42001 Lead Implementer for governance roles7)?
  • Is AI-tool access gated on completing training? (Yes is typical for regulated contexts.)
  • How do we assess that training worked — incident data, survey, observed review quality?

Curriculum plan template

AI Coding Training Plan — v1.0

Effective: 2026-05-15 Owner: Enablement lead Target audience: Pilot team / Phase 2 cohort N

Tracks.

Track Audience Hours Delivery
Developer core All engineers 8 Blended (2h live kickoff + 6h self-paced with exercises)
Tech lead All tech leads +4 Live cohort
Security reviewer AppSec + security champions +6 Live cohort + scenario practice
Manager EM and up 2 Live (single session)
Executive Exec sponsor + board liaison 1 Live briefing

Developer core modules.

  1. Evidence and limits of AI coding (45 min)
  2. AUP walkthrough (45 min)
  3. Prompting (S.C.A.F.F.) — 2 hrs hands-on
  4. Verification (V.E.R.I.F.Y. + C.L.E.A.R.) — 2 hrs hands-on
  5. Security patterns (S.H.I.E.L.D.) — 1.5 hrs hands-on
  6. Incident reporting and recovery — 45 min

Assessment.

  • Pass: complete all modules + scenario exercise + quiz (80% threshold).
  • Fail: re-take; if second attempt fails, 1-on-1 with tech lead and enablement.
  • Access to production-impacting AI tools is gated on assessment pass.

Refresh.

  • Quarterly: 1-hour module on what changed in the last quarter.
  • After each incident: a learning summary is distributed.
  • Annual: full track re-run or equivalent.

External certification.

  • Optional for all engineers.
  • Required for AppSec reviewers within 12 months of role start.

Metrics.

  • Completion rate (%).
  • Time-to-completion (median days).
  • Verification-behavior change (pre/post scenario exercise).
  • Incident rate among trained vs. untrained (lagging indicator).
# AI Coding Training Plan — v1.0

**Effective:** 2026-05-15
**Owner:** <Enablement lead>
**Target audience:** <Pilot team / Phase 2 cohort N>

## Tracks

| Track | Audience | Hours | Delivery |
|---|---|---|---|
| Developer core | All engineers | 8 | Blended (2h live kickoff + 6h self-paced with exercises) |
| Tech lead | All tech leads | +4 | Live cohort |
| Security reviewer | AppSec + security champions | +6 | Live cohort + scenario practice |
| Manager | EM and up | 2 | Live (single session) |
| Executive | Exec sponsor + board liaison | 1 | Live briefing |

## Developer core modules

1. Evidence and limits of AI coding (45 min)
2. AUP walkthrough (45 min)
3. Prompting (S.C.A.F.F.) — 2 hrs hands-on
4. Verification (V.E.R.I.F.Y. + C.L.E.A.R.) — 2 hrs hands-on
5. Security patterns (S.H.I.E.L.D.) — 1.5 hrs hands-on
6. Incident reporting and recovery — 45 min

## Assessment

- Pass: complete all modules + scenario exercise + quiz (80% threshold).
- Fail: re-take; if second attempt fails, 1-on-1 with tech lead and
  enablement.
- Access to production-impacting AI tools is gated on assessment pass.

## Refresh

- Quarterly: 1-hour module on what changed in the last quarter.
- After each incident: a learning summary is distributed.
- Annual: full track re-run or equivalent.

## External certification

- Optional for all engineers.
- Required for AppSec reviewers within 12 months of role start.

## Metrics

- Completion rate (%).
- Time-to-completion (median days).
- Verification-behavior change (pre/post scenario exercise).
- Incident rate among trained vs. untrained (lagging indicator).

Outputs

  1. Curriculum plan — by role, by module, with delivery and assessment.
  2. Decision Record — what is mandatory, what is gated, what is optional, what is paid (e.g., external certifications).
  3. Delivery schedule — when each track runs, per cohort / wave.
  4. Budget estimate — instructor time, content development, external certifications if subsidised. (Reference published benchmarks like Accenture's LearnVantage program spend8.)
  5. Onboarding integration plan — Day-1 training is usually the AUP
  6. a light AI overview; full developer track within Day 30–60.

Follow-ups

  • Within 14 days: content development resourced.
  • Within 30 days: first cohort running.
  • After first cohort: feedback incorporated; v1.1.
  • Quarterly: refresh module published.

Facilitator tips

  • Hands-on trumps slideware. Every module should end with an exercise on the team's real code.
  • Don't outsource the AUP module to the vendor. Your AUP, your clauses, your voice. Vendors can deliver product mechanics; they cannot deliver your policy.
  • Assess behavior, not knowledge. A multiple-choice quiz tells you less than a scenario exercise where the trainee has to choose whether to accept a suggestion that contains a known insecure pattern.
  • Schedule the refresh with the initial rollout. If refresh is "we'll figure it out," it won't happen.

Anti-patterns

Don't

  • Don't deliver a one-hour "intro" and call it training. Engineers need 8+ hours of active engagement to change behavior meaningfully.
  • Don't defer security training to "later". It is exactly the training that prevents the most expensive incidents.
  • Don't gate access purely on training completion. A completed module with a failed assessment is worse than no training — it signals permission without capability. Gate on pass.
  • Don't let training become boilerplate. The material should refresh with every quarter's new incidents, new tools, and new patterns.

Next

  1. Perry, Srivastava, Kumar, Boneh, "Do Users Write More Insecure Code with AI Assistants?", 2023 (arXiv:2211.03622). ↩↩

  2. Peng, Kalliamvakou, Cihon, Demirer, "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot", 2023 (arXiv:2302.06590). ↩

  3. METR, "Measuring the impact of early-2025 AI on experienced open-source developer productivity", Jul 2025. ↩

  4. Tobi Lütke on X (widely reported), "Reflexive AI usage is now a baseline expectation at Shopify", Apr 2025. ↩

  5. GitHub Learning, certifications catalog. ↩

  6. AWS, Certified AI Practitioner. ↩

  7. ISO, ISO/IEC 42001:2023. ↩

  8. Accenture press, "Accenture launches LearnVantage", Mar 2024. ↩