RLHF In the Spotlight: Problems and Limitations with Key AI Alignment Technique
A new paper takes a closer look at serious open challenges related to RLHF, and emphasizes that relying solely on it for AI alignment is profoundly risky
AI Safety
A collection of 83 posts
“Sure, I’ll Help You Build a Bomb”: New Research Reveals Alarming Gaps in AI Safety
The new class of adversarial attacks are capable of circumventing the alignment measures designed to prevent the generation of inappropriate or harmful content in multiple LLMs.
Anthropic, Google, Microsoft, and OpenAI Establish Frontier Model Forum to Promote AI Safety
The Forum's establishment is rooted in its members' collective commitment to pooling their expertise for the sake of safer, more responsible AI development.
Anthropic Calls for Stringent Security Safeguards for Frontier AI Models
Given the strategic implications of this technology, Anthropic says frontier models must be secured to levels surpassing standard practices for commercial technologies.
Top AI Companies Meet at White House, Commit to Voluntary Governance Measures
The commitments, which companies say will remain in place until relevant regulations are enacted, address key risks associated with generative AI models surpassing today’s state-of-the-art capabilities.
Google Proposes CoDoC, a Medical AI that Knows When to Defer to Human Experts
The model learns when predictive AI is offering correct information, and when it's better to defer to a clinician.
The Paradox of OpenAI's Superintelligence Alignment Initiative
The company's latest initiative aims to solve superintelligence alignment within four years.