Skip to content

FAQ

The questions below are the ones that recur in leadership conversations about AI coding adoption. The answers are honest about what the data supports, where it is thin, and where the answer is genuinely "it depends."

For the underlying citations, see the evidence pack.

Will AI coding tools replace developers?

Not on the evidence available to date. The published studies show AI assistants accelerating parts of developer work — sandbox tasks, boilerplate, scaffolding, documentation drafting — and showing mixed or negative results on others, particularly debugging in large mature codebases (METR 2025) and team-level delivery throughput (DORA 2024).

The replacement question often comes up because vendor marketing implies unit-of-output equivalence between an "AI engineer" and a human one. That equivalence is not measured anywhere in the public literature; what is measured is acceleration of specific tasks under specific conditions. Treat the replacement framing as a claim to be tested, not a forecast to plan around.

The framework's philosophy page takes a position: augmentation, not replacement. That position is operational, not ideological — accountability for shipped code does not move to the model, and the review, security, and maintenance work that follows generation grows rather than shrinks.

We have a SOC 2 audit next quarter — how does AI coding affect us?

It affects you most where engineers may have been sending company data to AI tools without a written policy. Auditors will ask three things:

  1. Do you have a written policy? A one-page interim Acceptable Use Policy is sufficient to demonstrate management intent, even if the permanent one is not yet drafted.
  2. Do you have evidence the policy is followed? Acknowledgement records, gateway logs, or tool-tenant administrative reports.
  3. Do you have an incident process if it isn't? A pointer to incident response — the AI-specific addition to your existing IR runbook — is enough.

The fastest path to audit-ready: run the leadership charter worksheet, publish the interim AUP, and capture the chain in Decision Records. For the crosswalk to specific SOC 2 trust-services criteria, see regulatory mapping.

Our junior developers are using ChatGPT on personal accounts. What now?

This is the single most common starting state. The answer is not "shut it down on Monday." Banning without an alternative pushes use underground and removes any audit trail; meanwhile, the productivity benefit was real for at least some of the work.

The sequence that has worked:

  1. Publish a one-page interim AUP that names a sanctioned alternative (a single company-tenant chat tool, ideally with training opt-out confirmed) and explicitly disallows personal accounts for company work. See AUP drafting worksheet.
  2. Communicate the why: the prohibited-data clause exists to protect employees from accidentally exposing source code or customer data, not to police behavior.
  3. Stand up the sanctioned alternative in the same week.
  4. Six weeks later, run a use audit and reconvene.

The framework's phase 0 readiness page covers this in operational detail.

Can we just ban AI tools?

You can, but be honest with yourself about what you are buying. A clean ban produces:

  • Lower risk surface for confidential data and IP exposure.
  • Slower time-to-ship on the work that AI assistants accelerate (boilerplate, scaffolds, docs).
  • Recruiting and retention pressure as developers gravitate toward organizations with sanctioned tooling.
  • Shadow-IT pressure unless the ban is enforced technically as well as in policy. Most policy-only bans erode within 12 months.

A targeted ban (e.g. "no AI on these specific data classes / these specific systems") is usually more durable than a blanket one and fits better against the risk classification matrix. If the right answer for your organization is genuinely a blanket ban, that is a defensible choice — document it as a Decision Record and set a re-evaluation date.

How do we know our AI usage isn't accidentally training the model on our IP?

You don't, unless three things are true:

  1. You are on an enterprise-tenant agreement (not a consumer account) where the contract explicitly excludes your prompts and completions from training data.
  2. The vendor's sub-processor list is reviewed and matches your data-handling policy.
  3. You have a technical control — usually an SSO-fronted gateway or vendor admin policy — that enforces enterprise-tenant routing for the tools your team actually uses.

Personal accounts (free-tier ChatGPT, free-tier Copilot personal, free-tier Claude.ai) routinely allow training on user content unless the user has actively opted out. That is the risk path. The mitigation is a sanctioned, contracted alternative, named in the AUP, and gated technically.

For the contract terms to read carefully, see IP, licensing & provenance and data handling.

What's the difference between "code completion" and an "AI agent"?

Three points on a spectrum of AI involvement and risk:

  • Code completion finishes the line or function the developer is typing. Inline suggestion, ghost text, autocomplete on steroids. Risk surface: similar to a smart linter — bad suggestions are usually caught at the keyboard.
  • Code generation produces a block, file, or scaffold from a natural-language prompt. Larger surface area; more chance of fabricated APIs, missed edge cases, and security defaults that don't match your stack.
  • Code agent takes multi-step action — reads the repo, edits files, runs commands, opens pull requests — without per-step approval. Risk surface: every credential, environment, and external service the agent can reach.

The same vendor's product line frequently spans all three modes. What matters operationally is the action surface and approval model, not the marketing label. See tool landscape for the comparison framework and S.H.I.E.L.D. for agent-specific risk controls.

We can't measure productivity — should we even try?

Yes, but measure the right things. Productivity is genuinely hard to measure with one number; that is the central argument of the SPACE paper, and the trap most "AI productivity" reports fall into.

What works:

  • DORA's four delivery metrics at the team level (deployment frequency, lead time for changes, change failure rate, MTTR). Outcomes the business cares about; resistant to gaming.
  • SPACE as a portfolio — combine activity, satisfaction, and efficiency metrics; never report only one.
  • DevEx flow / feedback / cognitive-load surveys to catch the qualitative cost of changes.

What doesn't:

  • Copilot acceptance rate or any single tool-usage metric reported alone. Easy to game; weak correlation with delivery outcomes.
  • Lines of code generated. A lagging indicator of work that may not be valuable.
  • "Hours saved per developer per week" estimates from vendor case studies. Self-reported; rarely match independent measurement.

See metrics taxonomy and anti-patterns (vanity metrics).

Our vendors say their AI gives a 10x speedup. Is that real?

The 10x figure is almost always reverse-engineered from a vendor case study, not measured in a controlled experiment. The peer-reviewed literature shows much smaller and more conditional gains:

  • The Copilot RCT measured 55.8% faster on a sandbox task — large, but not 10x, and on a task type that is unusually friendly to AI assistance.
  • METR 2025 measured a 19% slowdown in a different population on different tasks.
  • DORA 2024 found a small team-level negative correlation between AI-tool adoption and delivery throughput.

A vendor citing 10x is usually:

  • Measuring the time to write a new function (not the time to ship it, including review, testing, and deploy).
  • Sampling a friendly task class (greenfield, well-scoped) and generalizing.
  • Reporting median, not mean, after excluding "no improvement" outcomes.

Ask the vendor for the methodology and the task population. If they cannot provide both, treat the number as marketing and benchmark internally during your pilot.

Do we need to disclose AI use to customers?

It depends on jurisdiction, sector, and what you are using AI for.

  • Customer-facing AI features (a chatbot, an AI-suggested action in your product) almost always warrant disclosure under emerging consumer-AI rules and reasonable customer expectation.
  • Internal AI-assisted development (engineers using AI tools to write code) is generally not separately disclosable — your customers expect you to use the tools you find effective. The exceptions are contracts that specifically restrict third-party processing of their code or data, and regulated sectors (health, finance, government) with sector rules.
  • AI in code that customers see or run (model-generated test fixtures, model-generated SDK examples) is a gray area; some organizations choose to disclose, most do not.

This is one of the questions that requires legal counsel; the regulatory mapping page is a starting point for the conversation, not a substitute for it.

What's a reasonable training budget for our team?

A useful starting point is half a day per developer per quarter on AI-coding-specific training in the first year — about 2% of working time. Smaller organizations frequently spend more in the first quarter, then drop off; larger organizations frequently underspend on training and overspend on tools.

What the budget should cover:

  • Walkthroughs of the team's prompt library and the named frameworks (S.C.A.F.F., V.E.R.I.F.Y., C.L.E.A.R.).
  • Live-code sessions on real tickets, with reviewers calling out AI-specific failure modes.
  • Periodic refresh as the team's chosen tools change behavior.

What it should not cover:

  • Vendor sales sessions repackaged as training.
  • Conference attendance presented as the primary learning vehicle — conferences are a supplement, not a curriculum.

See training needs assessment for the meeting that produces a sized plan.

We tried it and it slowed people down. What did we do wrong?

The most common causes, roughly in order of frequency:

  1. No prompt or review framework. Engineers were left to figure out S.C.A.F.F.-shaped prompts and C.L.E.A.R.-shaped reviews on their own; quality was uneven and reviews caught more issues than benefits saved.
  2. Wrong task population. AI was deployed on debugging in a large mature codebase (METR-style task) rather than on greenfield or scaffold-heavy work (Copilot-RCT-style task). The slowdown is consistent with the published literature.
  3. No measurement baseline. Without a pre-pilot DORA/SPACE baseline, "felt slower" cannot be distinguished from "actually slower." Sometimes the actual delivery numbers were unchanged or improved; the perception was the regression.
  4. Cognitive-load tax. Switching between coding and prompting added friction without enough payoff for the work in scope. DevEx surveys catch this.
  5. Tool-config tax. Default settings produced excessive noise (wrong language, wrong style, wrong patterns) and engineers spent time fighting the tool rather than using it.

Run the pilot retrospective worksheet honestly, identify which of the above applies, and adjust scope, training, or tooling — not the goal — before re-piloting.

What does "vibe coding" actually mean and should we tell the board we're doing it?

The term was popularised by Andrej Karpathy in early 2025 to describe an exploratory practice: directing AI tools to produce code by natural-language intent, without close attention to what each line does, on work that is intended as throwaway or sketch. See the glossary entry.

This framework applies the term more broadly — to any AI-assisted work where a human directs a model to produce executable artifacts — because the practical reality is that organisations are shipping the output, regardless of Karpathy's original sketch-only intent.

For a board conversation, "vibe coding" is the wrong label. It comes in with informal connotations that obscure what is actually being proposed: a controlled, governed, evidence-based program for AI-assisted software development. Use that framing in the boardroom and reserve "vibe coding" for the casual explanatory moment when someone asks "isn't this what people are calling..." — at which point you can say yes and move on.

The framework's site name — Prompt to Prototype — is the deliberate alternative: it captures the same spectrum (intent through working artifact) without the throwaway connotation.

Next

  • Evidence pack — the citations behind the numbers in this FAQ.
  • Roadmap — what to do once leadership has agreed.
  • Glossary — terms defined.