When AI Should Say "I Don't Know" - But Doesn't

The results revealed a concerning gap between what these systems claim to understand and what they truly comprehend. The researchers created three types of impossible questions. They removed correct answers from multiple choice options, showed images that had nothing to do with the questions being asked, or provided completely irrelevant answer choices. A reliable AI system should recognize these situations and decline to answer.

But that's not what happened. While these same AI models score impressively on standard tests, they performed dismally when faced with impossible questions. Many open-source models got scores below 6% when they should have said "I can't answer this."

The gap between closed-source models (like GPT-4 Vision) and open-source alternatives proved particularly stark. While GPT-4 Vision managed to identify unsolvable questions about 60% of the time, popular open-source models like CogVLM2 scored below 1% - despite both performing similarly well on standard tests.

"This suggests that our community's efforts to improve performance on existing benchmarks do not directly contribute to enhancing model reliability," the researchers note. In other words, we've been teaching AI to guess even when it shouldn't.

The study uncovers different failure patterns among models. Some struggle specifically with visual tasks, while others have trouble with basic reasoning about whether questions are answerable. The researchers found that adding explicit instructions to consider whether questions were impossible helped some models but made others perform worse.

Looking ahead, the team suggests that future AI development needs to focus not just on getting right answers, but on knowing when getting an answer isn't possible.

Why this matters:

• Current AI visual systems are overconfident - they'll try to answer questions even when they can't possibly know the answer
• We need new ways to measure AI reliability beyond just accuracy scores on standard tests

Read on, my dear:

• The University of Tokyo: Unsolvable Problem Detection: Evaluating Trustworthiness of Large Multimodal Models