Why AI researchers keep building something they think will kill humans
Executive Take
When the people building a product publicly doubt it's safe, that is a governance signal, not just a PR problem. Leaders adopting these tools need their own safety review, not just the vendor's word.
Executive Summary
OpenAI CFO Sarah Friar said the company's largest models now train smaller ones, an example of recursive self-improvement (RSI). Anthropic's Evan Hubinger put over 10% odds AI kills all humans within a decade. Both firms' staff cite commercial pressure and fear of less safety-focused rivals as reasons they keep building despite the risk.
Why It Matters
AI leaders should note that frontier labs themselves are flagging existential risk while racing to ship, which should raise the bar for internal AI governance. Technology executives relying on OpenAI or Anthropic models need to factor this uncertainty into vendor risk assessments.
Bizquad Perspective
The real story isn't doom talk, it's that AI labs now openly admit commercial incentives override their own safety fears, which should worry enterprise buyers more than any apocalypse timeline.