Researchers at Los Alamos National Laboratory have developed a new tool called the Prelim Attention Score (PAS) to detect hallucinations in vision-language AI models, systems that combine image analysis with large language models. These models can sometimes generate descriptions of objects or details that are not actually present in an image. PAS works by monitoring how much the AI relies on the visual input versus its own previously generated text while producing a response, helping identify when the model is beginning to make things up.
The method operates in real time and can be integrated into existing vision-language models. By analyzing internal attention patterns, PAS provides a score indicating the likelihood of hallucination, allowing developers and users to assess the reliability of AI-generated outputs. The approach achieves state-of-the-art accuracy in detecting hallucinations and could improve the safety and trustworthiness of AI systems used in applications such as autonomous vehicles, healthcare imaging, robotics, and security monitoring.
More information:
https://interestingengineering.com/ai-robotics/us-tool-hallucinations-machine-vision-model