Testing the blind spots in artificial intelligence

About

By investigating the robustness of deep learning models using a context-based approach, KAUST researchers have developed a means to predict situations in which artificial intelligence might fail.

Artificial intelligence (AI) is becoming increasingly common as a technology that helps automated systems make better and more adaptive decisions. AI is an algorithm that allows a system to learn from its environment and available inputs. In advanced applications, such as self-driving cars, AI is trained using an approach called deep learning, which relies solely on large volumes of sensor data without human involvement. However, these machine learning systems can fail when the training data misleads the decision-making process.

In Bernard Ghanem's team, Abdullah Hamdi and Matthias Müller have been researching the limitations of AI and deep neural networks for safety-critical applications, such as self-driving cars.

“AI and deep learning are very powerful, but the technology can fail in rare "edge" cases, which are likely to eventually happen in real-world scenarios when these deep learning models are used in our daily lives,” he explains.

For many applications, an AI failure is merely an inconvenience as long as the AI works as expected most of the time. However, for applications, such as self-driving cars and medical diagnosis in which failure can result in death or catastrophe, repeated unexpected failures are not tolerated.

“Deep learning is successful in a wide range of tasks but no one knows exactly why it works and when it fails,” says Hamdi. “Because deep learning models will be used more and more in the future, we should not put blind faith in those that are not well understood and that can put lives at risk.”

The team has shown how these very powerful AI tools can fail badly in seemingly trivial scenarios. They developed a method to analyze these scenarios and establish "risk" maps that identify these risks in real applications.

“AI models are commonly tested for robustness using pixel perturbations, that is, adding noise to an image to try to fool the deep-trained network,” says Hamdi. “However, artificial pixel perturbations are actually unlikely to occur in real-life applications. It is much more likely that semantic or contextual scenarios occur that the networks have not been trained on.”

Read the full article