Skip to Main content Skip to Navigation
New interface
Conference papers

SiMOOD: Evolutionary Testing Simulation with Out-Of-Distribution Images

Abstract : Testing perception functions for safety-critical autonomous systems is a crucial task. The reason is that accurate ML models applied in computer vision tasks still fail in scenarios where humans perform well. Out-of-distribution (OOD) images are usually a source of such failures. For this reason, literature usually applies data augmentation techniques or runtime monitors such as OOD detectors to increase robustness. Evaluating such solutions is usually performed by analyzing metrics based on positive and negative rates over a dataset containing several perturbations. However, using such metrics on such datasets can be misleading since not all OOD data lead to failures in the perception system. Hence, testing a perception system cannot be reduced to measuring Machine Learning (ML) performances on a dataset but rely on the images captured by the system at runtime. However, the amount of time spent to generate diverse test cases during a simulation of perception components can grow quickly since it is a combinatorial optimization problem. Aiming to provide a solution for this challenging task, we present SiMOOD, an evolutionary simulation testing of safety-critical perception systems, which comes integrated into the CARLA simulator. Unlike related works that simulate scenarios that raise failures for control or specific perception problems such as adversarial and novelty, we provide an approach that finds the most relevant OOD perturbations that can lead to hazards in safety-critical perception systems. Moreover, our approach can decrease, at least 10 times, the amount of time to find a set of hazards in safety-critical scenarios such as autonomous emergency braking system simulation. Besides, code is publicly available for use.
Document type :
Conference papers
Complete list of metadata
Contributor : Raul Sena Ferreira Connect in order to contact the contributor
Submitted on : Saturday, September 17, 2022 - 3:47:09 PM
Last modification on : Tuesday, October 25, 2022 - 11:58:11 AM


Files produced by the author(s)


  • HAL Id : hal-03779723, version 1


Raul Sena Ferreira, Joris Guérin, Jérémie Guiochet, Hélène Waeselynck. SiMOOD: Evolutionary Testing Simulation with Out-Of-Distribution Images. 27th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC 2022), Nov 2022, Beijing, China. ⟨hal-03779723⟩



Record views


Files downloads