Aquarium Security and Other Organisational Priors

According to a LessWrong post published Tuesday, AI models may be absorbing sensitive details about how companies secure their systems from private chat data. The post notes that while business and API products are typically excluded from training, employees often use personal accounts for work, and those conversations can be used to improve future models. The worry is that models could learn patterns of organizational behavior, making it easier to guess where security is weak and plan attacks. The author points to a recent attack that appears to have been largely model-executed across roughly thirty targets, though the connection to chat data is speculative. The post concludes that testing this link is difficult but worthwhile.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton