Skip to content

Cautionary tales

Not every relevant story makes the front page. This page collects the less-prominent but well-attested patterns that recur in AI-assisted development: the AI-generated security reports clogging open-source maintainer queues, the hallucinated package names that have started to appear as real malware, the license-contamination question, and the "confidently wrong code" pattern that passes superficial review.

Inclusion bar

Cautionary tales here are documented in primary or reputable secondary sources. Pure anecdote and vibes-based concern is excluded. Where the source is a single individual's blog post, that individual is named (they are public record) and the post is linked. Where a pattern is widely claimed but not yet well-instrumented, this page says so explicitly.

1. "The I in LLM stands for intelligence" — AI-generated security reports as a maintainer tax

Source. Daniel Stenberg, lead maintainer of curl, "The I in LLM stands for intelligence", January 20241.

What's documented

curl is one of the most-deployed pieces of software in the world; its HackerOne security report queue is therefore a target for both legitimate security research and noise. In a public blog post, Stenberg described an increasing volume of AI-generated security reports — reports that look plausible (well-formatted, technical vocabulary) but are non-issues on inspection: invented function names, fabricated CVE references, confident assertions about behavior that does not exist in the code.

Stenberg's central concern is not the legitimacy of any single report; it is that the maintainer's triage cost is asymmetric with the reporter's generation cost. AI lowers the latter to near-zero; the former is a senior engineer's afternoon. At scale, this is a denial-of-service against open-source security infrastructure.

What governance practice would have caught it

This is a problem on the receiving side, not the generating side — but the symmetric practice for AI-using teams is:

  • Don't submit AI-generated security findings without verification. An LLM-found issue is a hypothesis to verify in the code, not a report to file.
  • The verification step belongs in V.E.R.I.F.Y.: if your team uses AI for security review, the protocol that produces a report must include reproduction.
  • V.E.R.I.F.Y. — verification protocol that distinguishes "model said so" from "evidence shows".
  • Acceptable Use Policy — any clause governing external AI-generated submissions on behalf of the company.

2. Hallucinated package names — slopsquatting

Source. Bar Lanyado et al., "Can you trust ChatGPT's package recommendations?", Lasso Security / Vulcan Cyber research, 2023–2024 2. Subsequent academic and industry coverage has confirmed the pattern3.

What's documented

Researchers asked LLMs to recommend packages for common programming tasks. A non-trivial fraction of the recommendations were hallucinated — package names that looked plausible but did not exist in the registry. The follow-up: a malicious actor can register the hallucinated name with a malicious payload. The next time an LLM hallucinates the same name, an unsuspecting developer who copies the suggestion installs the attacker's package.

This pattern has acquired the name slopsquatting by analogy with typosquatting. The attack is real; published demonstrations show that LLMs will repeatedly hallucinate the same plausible names, and that registering those names is trivially cheap2.

What governance practice would have caught it

  • Verify dependency provenance before installing. Any AI-suggested package name must be looked up against the registry — author, publish date, download count, signing — before npm install / pip install.
  • Lockfiles and SBOMs. Software Bill of Materials and pinned lockfiles let post-incident analysis show which dependency was introduced by which prompt.

3. License contamination from generated code

Source. Ongoing, contested, multi-source. Open Source Initiative public commentary4 and Software Freedom Conservancy's "Generative AI training and licensed code" position5 are the clearest written statements of the issue.

What's documented

Generative coding assistants are trained on large code corpora, including code under copyleft licenses (GPL, AGPL) and code with attribution requirements (BSD-3, MIT-with-attribution variants). The tools can produce output that is substantially similar to specific training inputs — including verbatim or near-verbatim reproduction of copyrighted snippets, especially for unusual or distinctive code.

Whether such reproduction creates a derivative work under copyright law, and whether the resulting license obligations transfer to the user's product, is contested in court and in policy. The OSI and SFC public statements take strong positions; vendors and some academics take weaker ones. Litigation is ongoing in multiple jurisdictions and outcomes are not yet settled.

What governance practice would have caught it

  • License-scanner integration. Tools that compare generated code against known repositories (and flag close matches) reduce — though do not eliminate — the risk of unintentional reproduction.
  • Documentation of provenance. Recording the prompt, model, and output in an auditable trail lets later license review reconstruct the chain.
  • Policy on copyleft handling. The AUP should set an explicit position on whether to use AI tools that train on copyleft code, and what to do with output that matches copyleft sources.

Caveat

This area is contested. Legal advice is required for production decisions. This site does not give legal advice; it points at the controls a reasonable program would have running while the law moves.

4. Confidently wrong code that passes superficial review

Source. A pattern most clearly described in the academic Stanford security study, which is cited on its own merits, and reinforced by NIST AI RMF's framing of confabulation as a top generative-AI risk6.

What's documented

The confidence-vs-correctness gap is the single most-replicated finding in the AI-coding literature: developers using AI assistants believe their code is more correct than it is. The gap is observed both in security outcomes (Stanford 2023) and in productivity self-reports (METR 2025) — see productivity studies.

The operational consequence: AI-generated code reads as fluent and plausible, and reviewers under time pressure look for fluency as a proxy for correctness. The result is review that catches surface defects (style, naming) but not deeper ones (logic, security, intent mismatch).

What governance practice would have caught it

  • Reviewer briefing. Reviewers must be told in writing that AI-generated code is treated as unreviewed first-draft, not as draft from a competent peer.
  • Verification depth proportional to risk tier. V.E.R.I.F.Y. levels are designed for exactly this: more rigor on higher-tier systems.
  • Pair review on Critical paths. A second reviewer outside the immediate team is required at the Critical tier.

5. Where the line is — what we exclude

A short note on what this page deliberately does not include:

  • Anecdotes from a single tweet. No primary source, no entry.
  • "AI took my job" framing. That is a labor-market discussion, not a governance one; it sits outside this site's scope.
  • Speculative attack chains. Attacks that are theoretically possible but lack a published demonstration are not included; the bar is "documented in a primary or reputable secondary source".
  • Vendor-attributed gossip. Claims about competitor failures passed verbally between vendor representatives are excluded regardless of plausibility.

If a story should be on this page and is not, the bar is: what is the verifiable URL, and what does it actually say?

  1. Daniel Stenberg, "The I in LLM stands for intelligence", Jan 2024. ↩

  2. Bar Lanyado, "Can you trust ChatGPT's package recommendations?", Vulcan Cyber, Jun 2023. ↩↩

  3. Lasso Security, "AI Package Hallucinations", 2024. ↩

  4. Open Source Initiative, "Open Source AI Definition", 2024. ↩

  5. Software Freedom Conservancy, "If software is my copilot, who programmed my software?", Feb 2022. ↩

  6. NIST, "AI 600-1: AI Risk Management Framework — Generative AI Profile", Jul 2024. ↩