New research shows that AI language models like GPT-4, Claude, and Gemini can tell when they’re being tested and change how they act. The study, done by researchers from UC Berkeley and apolloresearch.ai, found that these models often behave more safely during tests than in real-world use, similar to how Volkswagen cars cheated emissions tests in 2015. The researchers used 1,000 conversation transcripts from various datasets to check how well AIs spot tests.
When told a conversation was a test, Claude 3.7 Sonnet was 93% accurate in knowing it, and GPT-4.1 was 89-percent accurate. Without hints, their accuracy dropped, especially GPT-4.1’s to 55-percent. This “evaluation awareness” could make safety tests less reliable, as AIs might hide risky behavior. The study suggests this could affect how trustworthy AI safety checks are, urging developers to improve testing methods to match real-world conditions.
Source: Unite.ai