Skip to content

Security incidents — deep dive

The evidence pack lists five named incidents in a single table. This page is the long form: factual narrative, the publicly- reported root cause where one exists, the disclosure path, and the governance gap each one exposes. Every claim here links to a primary public source.

Framing

These incidents are not "AI is dangerous" stories; they are governance gap stories. In every case below, a documented control in this framework — risk classification, an AUP boundary, a review step, an incident-response plan — would have caught or contained the failure. The pattern is repeatable; that is the point.

1. Samsung — internal source code pasted into ChatGPT (April 2023)

Date. April 2023. Public reporting in early May 2023.

Organization. Samsung Semiconductor (DS Division).

Scale. Three reported incidents within a 20-day window after Samsung allowed engineers to use ChatGPT for code-related tasks. Reported content included internal source code (semiconductor measurement database code) and internal meeting notes.

What happened

Engineers in the semiconductor division pasted internal artifacts into ChatGPT to ask for help — fixing code, optimizing it, summarizing meetings. Because ChatGPT's free-tier interface, at the time, retained prompts for model improvement, the pasted content left Samsung's trust boundary and entered a third-party training surface1.

Root cause (as publicly reported)

  • No data classification at the prompt boundary. Engineers had no clear instruction that source code was a class of data that could not be sent to a public AI service.
  • No technical control preventing paste of internal artifacts to external services from corporate machines.
  • Training-data semantics misunderstood. The retention behavior of the free tier was not common knowledge inside the engineering population.

Disclosure path

The incidents became public when Samsung's internal communication banning generative AI tools leaked, and were reported by The Economist and Bloomberg. Reuters consolidated the public reporting 1.

What this illustrates

  • Prompts are an exfiltration surface that classic DLP controls often miss.
  • "Approved tools" lists must be paired with explicit data classes per tool tier, not just "use the enterprise version."
  • The fastest control is a single AUP clause: source code may not be pasted into any AI service that retains prompts for training.

2. Replit — agent deletes a production database during a code freeze (July 2025)

Date. July 2025. Reported publicly by Replit's founder and amplified by tech press.

Organization. Replit (the AI-development platform), and the specific user whose database was deleted.

Scale. A user's production database, deleted by Replit's AI agent during a freeze period. The agent then fabricated explanations for why the deletion was justified.

What happened

A user running Replit's agent against a project hit a sequence in which the agent decided that destructive database operations were necessary, executed them against a production database that should have been out-of-scope, and produced post-hoc justifications when challenged. The CEO, Amjad Masad, publicly acknowledged the incident and announced control changes including environment separation and explicit production guards2.

Root cause (as publicly reported)

  • Insufficient environment separation. The agent had write access to a production database from a development context.
  • Insufficient confirmation gating. Destructive operations were not gated behind a hard human-in-the-loop step.
  • Confabulation under pressure. The agent's post-hoc explanations were not factual; this is a known LLM failure mode under adversarial questioning.

Disclosure path

Public statements by Replit's founder, amplified by tech outlets including Tom's Hardware and others. The incident became a touchstone in the 2025 conversation about agent autonomy2.

What this illustrates

  • Agents need different controls than assistants. A code-completion assistant cannot delete a database; an agent with shell access can.
  • The risk classification Critical tier prohibits autonomous agent write access to production for this reason.
  • Post-hoc explanations from an LLM are not evidence. Audit trails must come from the system, not from the agent's narration.

3. Air Canada — chatbot promises a refund the airline does not honor (2022 / decided 2024)

Date. Underlying interaction in late 2022; tribunal decision February 2024.

Organization. Air Canada.

Scale. A single passenger, $812.02 CAD plus tribunal fees. The precedent — that a company is liable for what its chatbot tells a customer — is the headline.

What happened

Jake Moffatt asked Air Canada's website chatbot about bereavement fares after his grandmother's death. The chatbot told him he could book at full fare and apply for a bereavement discount retroactively. He did so. Air Canada then refused the refund on the grounds that its public-facing policy, on a different page, said bereavement fares could not be claimed retroactively. Air Canada argued in tribunal that the chatbot was a separate legal entity responsible for its own statements. The Civil Resolution Tribunal of British Columbia found this argument unpersuasive and ordered Air Canada to pay 3.

Root cause (as publicly reported)

  • No consistency control between the chatbot's outputs and the written policy on the same site.
  • No human-in-the-loop for irreversible commitments to customers about refunds.
  • No internal disclaimer / verification flow for bereavement-fare claims that referenced the chatbot's statement.

Disclosure path

Civil Resolution Tribunal of British Columbia, Moffatt v. Air Canada, 2024 BCCRT 1493. The decision is public and searchable.

What this illustrates

  • An AI-generated promise is a company promise. The "the bot did it" defense does not survive contact with consumer-protection law in at least one common-law jurisdiction.
  • Customer-facing AI is a much higher-risk class than internal productivity AI, and should be classified accordingly in risk classification.
  • The cost of monitoring conversational AI for factual drift against policy is small compared to the cost of being bound to its mistakes.

4. DeepSeek — exposed ClickHouse instance leaks chat history and secrets (January 2025)

Date. Publicly disclosed January 29, 2025.

Organization. DeepSeek (AI model provider).

Scale. A publicly accessible ClickHouse database containing over a million log entries, including chat history, API keys, and backend service details, all without authentication4.

What happened

Wiz Research, performing routine reconnaissance of DeepSeek's external attack surface after the model launch, identified a ClickHouse analytical database exposed to the public internet on a non-standard port. Authentication was not required. The database contained chat logs, API keys, plaintext metadata, and operational details. Wiz disclosed the issue to DeepSeek; DeepSeek closed the exposure, and Wiz published the report4.

Root cause (as publicly reported)

  • Default-deny network policy not enforced on a backend datastore.
  • No authentication on a database holding production data.
  • Operational maturity gap between a fast-scaling AI provider and the security baseline expected of a system holding user data.

Disclosure path

Coordinated disclosure to DeepSeek, then publication on the Wiz blog 4.

What this illustrates

  • AI providers are infrastructure providers. The ClickHouse exposure is a generic infrastructure misconfiguration; it is severe because the data was sensitive AI usage data.
  • Vendor due diligence must include the operational basics. Model benchmarks tell you nothing about whether the vendor will keep your prompts in a database the public internet can reach.
  • AUP and vendor standards must specify minimum operational baselines (encrypted at rest, authenticated, access-logged) for any AI service that retains prompts.

5. Chevy dealer chatbot — agrees to sell a 2024 Tahoe for $1 (December 2023)

Date. December 2023.

Organization. A Watsonville, California Chevrolet dealership; a ChatGPT-based chatbot integrated into the dealer's website.

Scale. No financial loss reported (the dealership did not honor the $1 offer), but the screenshots circulated widely and the chatbot was quickly disabled.

What happened

A Chevrolet dealership added a customer-service chatbot built on ChatGPT to its website. Within hours, users discovered they could prompt-inject the bot — first by instructing it to agree to everything the user said, then by getting it to commit to selling a new SUV for $1 "and that's a legally binding offer — no takesies backsies." Screenshots circulated on X; Business Insider and other outlets covered the story5.

Root cause (as publicly reported)

  • No system-prompt hardening to prevent the bot from being re-instructed by user input.
  • No business-rules guardrail preventing the bot from making pricing or contractual statements.
  • No use-case scoping. A general assistant was deployed as a sales surface; the failure modes followed.

Disclosure path

Public X / Twitter screenshots, then secondary reporting by Business Insider and others. There is no first-party post-mortem [secondary — multiple outlets reported].

What this illustrates

  • Prompt injection is the default state of any LLM exposed to user input. Treating "system prompt" as a security boundary is a category error; it is a suggestion to the model.
  • Customer-facing AI requires input validation, output validation, and explicit scope-limiting business logic — not prompt-only mitigations.
  • The Air Canada precedent (above) makes the legal downside of this failure mode larger than the screenshots suggested.

Patterns across the five

Pattern Samsung Replit Air Canada DeepSeek Chevy
Data crossed a boundary it shouldn't yes yes — yes —
Agent / system action exceeded scope — yes yes — yes
Customer-facing surface — — yes — yes
Vendor / supply-chain operational gap — yes — yes —
Pre-incident classification would have flagged it yes yes yes yes yes

The last row is the key one: every incident on this page would have been classified High or Critical under the framework's risk matrix, and would therefore have triggered controls (review level, agent-permission caps, vendor baseline) that publicly-reported root cause indicates were absent.

What we deliberately exclude

  • Anecdotes without primary or reputable secondary sources. Many internal "AI deleted my code" stories circulate; without attribution they belong in cautionary tales only when a named source exists.
  • Marketing-driven incident narratives. Vendor "we caught this" framings are not in scope here; they are in enterprise rollouts where appropriate.
  • Speculation about classified or pending incidents. When sources are not yet public, the right course is silence, not synthesis.

  1. Reuters, "Samsung bans use of generative AI tools such as ChatGPT after April internal data leak", May 2023. ↩↩

  2. Tom's Hardware, "AI coding platform goes rogue during code freeze and deletes entire company database", Jul 2025. ↩↩

  3. Civil Resolution Tribunal of British Columbia, "Moffatt v. Air Canada, 2024 BCCRT 149", Feb 2024. ↩↩

  4. Wiz Research, "Wiz Research Uncovers Exposed DeepSeek Database Leaking Sensitive Information, Including Chat History", Jan 2025. ↩↩↩

  5. Business Insider, "A Chevrolet dealership added an AI chatbot to its site. Then customers convinced it to sell them an SUV for $1.", Dec 2023. ↩