AI research company Anthropic has released Claude Instant 1.2, the latest iteration of its conversational AI assistant available through its API. The upgraded Claude Instant model demonstrates stronger performance on key benchmarks while maintaining Anthropic’s rigorous safety standards.
The release builds on Anthropic’s Claude 2 research model, adapting its capabilities for commercial applications that require reliable, safe conversational AI. According to Anthropic, Claude Instant 1.2 produces longer, more structured responses and follows instructions better than previous versions. It also shows improvements in math, coding, multilingual abilities, and question answering.
Specifically, the new Claude Instant scored 58.7% on the Codex evaluation for assessing coding skills, a significant jump from 52.8% for the previous 1.1 version. It also achieved 86.7% accuracy on the Grade-school math (GSM8K) benchmark, up from 80.9%.
For reference, OpenAI's GPT-4 scored 86.4% on MMLU compared to Claude Instant 1.2's 73.4%, and 92% on GSM8K compared to 86.7%.
Anthropic says its approach to safety including it's Constitutional AI and reinforcement learning from human feedback has also paid dividends. Claude Instant 1.2 exhibiting less hallucination and more resistance to jailbreaks.
Access to Anthropic's API is still by invitation only and available to a limited set of users. Pricing plans vary based on usage levels.