PRECRIME

ABOUT

PRECRIME: Self-assessment Oracles for Anticipatory Testing

Project: European Research Council - Advanced Grant (ERC-AdG n. 787703)
Duration: 2019-2024

Summary:
One of the long-lasting dreams in science fiction is the ability to arrest criminals before they even commit crimes. Software testing researchers have a similar dream: when the context for a bug manifestation occurs in the field, the goal is to discover and fix the bug before it causes any in-field failure. In fact, current practice of pre-release testing is severely limited when dealing with autonomous AI (Artificial Intelligence) systems (such as self-driving cars, robots, automated traders, virtual doctors and customer service chatbots), running in complex, rapidly changing environments, which cause their run-time adaptation, learning and knowledge acquisition, because pre-release testing cannot exhaustively explore all different contexts and states in which the software will be running.

The PRECRIME project introduces a new, disruptive view on testing, called anticipatory testing and aimed at fixing bugs before they even manifest themselves in the field. Anticipatory testing is activated at run-time by a new type of oracles, called self-assessment oracles, which observe and report unexpected execution contexts. A self-assessment oracle is an estimator of the system's confidence in being able to handle a new execution context correctly. The goal of anticipatory testing is to anticipate any failure that might occur in the field due to unexpected execution contexts. Whenever an execution context monitored at runtime by self-assessment oracles is estimated as a low confidence context for the system, anticipatory testing exercises the software automatically and extensively in similar contexts. Timely activation of anticipatory testing by runtime observations results in early, anticipatory fault detection. Combined with automated patch synthesis, anticipatory testing leads to the release of a patch for the fault before any software failure occurs in the field.

TEAM

Project Team

Paolo Tonella

Paolo Tonella
Principal Investigator

Andrea Stocco

Andrea Stocco
Postdoc

Gunel Jahangirova

Gunel Jahangirova
Postdoc

Vincenzo Riccio

Vincenzo Riccio
Postdoc

Matteo Biagiola

Matteo Biagiola
Postdoc

Nargiz Humbatova

Nargiz Humbatova
PhD student

Michael Weiss

Michael Weiss
PhD student

Tahereh Zohdi

Tahereh Zohdi
PhD student

Andréa Doreste

Andréa Doreste
PhD student

MSc (BSc) students

 Marco Calzana, thesis title: "Misbehaviour Prediction in Autonomous Vehicles using Auto-Encoders", September 2019
 André García Gómez, thesis title: "Leveraging Autoencoders for Robustness of Image Classifier based Applications", June 2021
 Brian Pulfer, thesis title: "From Simulated to Real Test Environments for self-driving cars", June 2021
 Nimisha Manjali, thesis title: "Detection of Synthetic Faults in Self-Driving Cars", June 2021
 Rwiddhi Chakraborty, thesis title: "White Box Out of Distribution Supervision in Convolutional Neural Networks", June 2021
 Shubhayu Das, thesis title: "Devising Techniques for the Automated Repair of Trained Deep Neural Networks", June 2021
 Alexandra Willi (BSc), thesis title: "A NeuralWeb Page Embedding for AutomatedWeb Crawling", June 2022
 Davide Mammarella, thesis title: "Predicting Safety-Critical Misbehaviours in Autonomous Driving Systems using Uncertainty Quantification", July 2022
 Giorgio Macauda, thesis title: "Training a Reinforcement Learning Agent in the Real World", September 2022
 Valdet Ismaili, thesis title: "Metamorphic Testing of Autonomous Vehicles", September 2022
 Lodovico Azzini, thesis title: "Opening the Black Box: Heatmap Clustering to Understand the Misbehaviors Exposed by Automatically Generated Test Inputs", September 2022

UROP Summer students

 Simone Giacomelli, Nelson Brochado, project title: "Extreme Testing of Self-Driving Cars", Summer 2019
 Martin Odermatt, project title: "Developing an Interactive Tool to Allow Human Control in MLS Testing", Summer 2020
 Khehra Arshjot Singh, project title: "Developing Self-Driving Car Autopilots using Reinforcement Learning Algorithms", Summer 2021
PUBLICATIONS

Publications

  Michael Weiss, Paolo Tonella: Uncertainty quantification for deep neural networks: An empirical comparison and usage guidelines. Journal of Software: Testing, Verification and Reliability (STVR), 2023. DOI
  Tahereh Zohdinasab, Vincenzo Riccio, Alessio Gambi, Paolo Tonella: Efficient and Effective Feature Space Exploration for Testing Deep Learning Systems. ACM Transactions on Software Engineering and Methodology (TOSEM), 2022. DOI
  Andrea Romdhana, Mariano Ceccato, Alessio Merlo, Paolo Tonella: IFRIT: Focused Testing through Deep Reinforcement Learning. Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST), 2022. DOI
  Matteo Biagiola, Paolo Tonella: Testing the Plasticity of Reinforcement Learning Based Systems. ACM Transactions on Software Engineering and Methodology (TOSEM), 2022. DOI
  Andrea Romdhana, Alessio Merlo, Mariano Ceccato, Paolo Tonella: Deep Reinforcement Learning for Black-Box Testing of Android Apps. ACM Transactions on Software Engineering and Methodology (TOSEM), 2022. DOI
  Andrea Stocco, Paolo Tonella: Confidence-driven Weighted Retraining for Predicting Safety-Critical Failures in Autonomous Driving Systems. Journal of Software: Evolution and Process (JSEP), October, 2021. DOI
  Vincenzo Riccio, Nargiz Humbatova, Gunel Jahangirova, Paolo Tonella: DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score. Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2021. DOI
  Tahereh Zohdinasab, Vincenzo Riccio, Alessio Gambi, Paolo Tonella: DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search. Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2021. DOI
  Nargiz Humbatova, Gunel Jahangirova, Paolo Tonella: DeepCrime: A Mutation Testing Tool for Deep Learning Systems. Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2021. DOI
  Gunel Jahangirova, Andrea Stocco, Paolo Tonella: Quality Metrics and Oracles for Autonomous Vehicles Testing. Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST), 2021. DOI
  Michael Weiss, Paolo Tonella: Fail-Safe Execution of Deep Learning based Systems through Uncertainty Monitoring. Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST), 2021. DOI
  Michael Weiss, Rwiddhi Chakraborty, Paolo Tonella: A Review and Refinement of Surprise Adequacy. Proceedings of the International Workshop on Testing for Deep Learning and Deep Learning for Testing (DeepTest), 2021. DOI
  Michael Weiss, Paolo Tonella: Uncertainty-Wizard: Fast and User-Friendly Neural Network Uncertainty Quantification. Proceedings of the IEEE International Conference on Software Testing, Verification and Validation, Tool Track (ICST-Tool), 2021. Best tool paper award. DOI
  Valerio Terragni, Gunel Jahangirova, Paolo Tonella, Mauro Pezzè: GAssert: A Fully Automated Tool to Improve Assertion Oracles. Proceedings of the IEEE International Conference on Software Engineering, Demonstrations Track (ICSE-Demo), 2021. DOI
  Antonia Bertolino, Pietro Braione, Guglielmo De Angelis, Luca Gazzola, Fitsum Kifetew, Leonardo Mariani, Matteo Orrù, Mauro Pezzè, Roberto Pietrantuono, Stefano Russo, Paolo Tonella: A Survey of Field-based Testing Techniques. ACM Computing Surveys (CSUR), 2021. DOI
  Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, Paolo Tonella: Testing Machine Learning based Systems: A Systematic Mapping. Empirical Software Engineering (EMSE), 2020. DOI
  Vincenzo Riccio, Paolo Tonella: Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing. Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2020. DOI
  Valerio Terragni, Gunel Jahangirova, Mauro Pezzè, Paolo Tonella: Evolutionary Improvement of Assertion Oracles. Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2020. DOI
  Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, Paolo Tonella: Taxonomy of Real Faults in Deep Learning Systems. Proceedings of the 42nd International Conference on Software Engineering (ICSE), 2020. DOI
  Andrea Stocco, Michael Weiss, Paolo Tonella: Misbehaviour Prediction for Autonomous Driving Systems. Proceedings of the 42nd International Conference on Software Engineering (ICSE), 2020. DOI
  Gunel Jahangirova, Paolo Tonella: An Empirical Evaluation of Mutation Operators for Deep Learning Systems. Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST), 2020. Best paper award. DOI
  Mariano Ceccato, Davide Corradini, Luca Gazzola, Fitsum Meshesha Kifetew, Leonardo Mariani, Matteo Orrù, Paolo Tonella: A Framework for In-Vivo Testing of Mobile Applications. Proceedings of the 13th IEEE International Conference on Software Testing, Verification and Validation (ICST), 2020. DOI
  Antonia Bertolino, Guglielmo De Angelis, Breno Miranda, Paolo Tonella: Run Java Applications and Test Them In-Vivo Meantime. Proceedings of the 13th IEEE International Conference on Software Testing, Verification and Validation, Tool Track (ICST-Tool), 2020. DOI
  Gunel Jahangirova, David Clark, Mark Harman, Paolo Tonella: An Empirical Validation of Oracle Improvement. IEEE Transactions on Software Engineering (TSE), 2019. DOI
  Mariano Ceccato, Luca Gazzola, Fitsum Meshesha Kifetew, Leonardo Mariani, Matteo Orrù, Paolo Tonella: Toward In-Vivo Testing of Mobile Applications. Proceedings of the 1st International Workshop on Governing adaptive and Unplanned Systems of Systems (GAUSS), 2019. DOI
TECHNICAL REPORTS

Technical Reports

  Michael Weiss, Paolo Tonella: Uncertainty Quantification for Deep Neural Networks: An Empirical Comparison and Usage Guidelines. TR-Precrime-2023-01
  Matteo Biagiola, Paolo Tonella: Testing the Plasticity of Reinforcement Learning Based Systems. TR-Precrime-2022-01
  Andrea Romdhana, Alessio Merlo, Mariano Ceccato, Paolo Tonella: Deep Reinforcement Learning for Black-Box Testing of Android Apps. TR-Precrime-2022-02
  Andrea Stocco, Paolo Tonella: Confidence-driven Weighted Retraining for Predicting Safety-Critical Failures in Autonomous Driving Systems. TR-Precrime-2022-03
  Tahereh Zohdinasab, Vincenzo Riccio, Alessio Gambi, Paolo Tonella: Efficient and Effective Feature Space Exploration for Testing Deep Learning Systems. TR-Precrime-2022-04
  Andrea Romdhana, Mariano Ceccato, Alessio Merlo, Paolo Tonella: IFRIT: Focused Testing through Deep Reinforcement Learning. TR-Precrime-2022-05
  Michael Weiss, Rwiddhi Chakraborty, Paolo Tonella: A Review and Refinement of Surprise Adequacy. TR-Precrime-2021-01
  Antonia Bertolino, Pietro Braione, Guglielmo De Angelis, Luca Gazzola, Fitsum Kifetew, Leonardo Mariani, Matteo Orrù, Mauro Pezzè, Roberto Pietrantuono, Stefano Russo, Paolo Tonella: A Survey of Field-based Testing Techniques. TR-Precrime-2021-02
  Valerio Terragni, Gunel Jahangirova, Paolo Tonella, Mauro Pezzè: GAssert: A Fully Automated Tool to Improve Assertion Oracles. TR-Precrime-2021-03
  Michael Weiss, Paolo Tonella: Uncertainty-Wizard: Fast and User-Friendly Neural Network Uncertainty Quantification. TR-Precrime-2021-04
  Tahereh Zohdinasab, Vincenzo Riccio, Alessio Gambi, Paolo Tonella: DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search. TR-Precrime-2021-05
  Nargiz Humbatova, Gunel Jahangirova, Paolo Tonella: DeepCrime: Mutation Testing of Deep Learning Systems Based on Real Faults. TR-Precrime-2021-06
  Vincenzo Riccio, Nargiz Humbatova, Gunel Jahangirova, Paolo Tonella: DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score. TR-Precrime-2021-07
  Vincenzo Riccio, Paolo Tonella: Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing. TR-Precrime-2020-01
  Valerio Terragni, Gunel Jahangirova, Mauro Pezzè, Paolo Tonella: Evolutionary Improvement of Assertion Oracles. TR-Precrime-2020-02
  Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, Paolo Tonella: Testing Machine Learning based Systems: A Systematic Mapping. TR-Precrime-2020-03
  Gunel Jahangirova, Andrea Stocco, Paolo Tonella: Quality Metrics and Oracles for Autonomous Vehicles Testing. TR-Precrime-2020-04
  Michael Weiss, Paolo Tonella: Fail-Safe Execution of Deep Learning based Systems through Uncertainty Monitoring. TR-Precrime-2020-05
  Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, Paolo Tonella: Taxonomy of Real Faults in Deep Learning Systems. TR-Precrime-2019-01
  Andrea Stocco, Michael Weiss, Paolo Tonella: Misbehaviour Prediction for Autonomous Driving Systems. TR-Precrime-2019-02
  Gunel Jahangirova, Paolo Tonella: An Empirical Evaluation of Mutation Operators for Deep Learning Systems. TR-Precrime-2019-03
  Gunel Jahangirova, David Clark, Mark Harman, Paolo Tonella: An Empirical Validation of Oracle Improvement. TR-Precrime-2019-04
  Mariano Ceccato, Davide Corradini, Luca Gazzola, Fitsum Meshesha Kifetew, Leonardo Mariani, Matteo Orrù, Paolo Tonella: A Framework for In-Vivo Testing of Mobile Applications. TR-Precrime-2019-05
  Antonia Bertolino, Guglielmo De Angelis, Breno Miranda, Paolo Tonella: Run Java Applications and Test Them In-Vivo Meantime. TR-Precrime-2019-06
  Mariano Ceccato, Luca Gazzola, Fitsum Meshesha Kifetew, Leonardo Mariani, Matteo Orrù, Paolo Tonella: Toward In-Vivo Testing of Mobile Applications. TR-Precrime-2019-07
EVENTS AND TALKS

Events and Invited Talks


  Paolo Tonella: Deep learning testing. Keynote at the 15th International Workshop on Search-Based Software Testing (SBST). May 9, 2022.
  Paolo Tonella: Misconceptions in deep learning testing. Keynote at the IEEE International Conference on Software Testing (ICST). April 5, 2022.
  Paolo Tonella: How can we expose, explain and prevent deep learning faults? Invited talk at the University of Virginia, USA. October 27, 2021.
  Paolo Tonella: How can we expose and explain deep learning faults? Keynote at the Huawei Strategy and Technology Workshop, Beijing, China. October 15, 2021.
  Paolo Tonella: How can we expose and explain deep learning faults? Invited talk at the University of Sheffield, UK. September 21, 2021.
  Paolo Tonella: Precrime: testing deep learning components against unexpected execution conditions. Invited talk at the Technical University of Darmstadt, Germany. July 5, 2021.
  Andrea Stocco, Gunel Jahangirova, Vincenzo Riccio (Workshop Co-organizers): DeepTest: 3rd International Workshop on Testing for Deep Learning and Deep Learning for Testing. June 1, 2020.
  Paolo Tonella: Testing the unexpected: the Precrime way towards dependable systems that include deep learning components. Invited talk at the University of St. Gallen, Switzerland. March 19, 2021.
  Paolo Tonella: Deep Learning Testing: problems, solutions and elephants. Talk at the Software Institute seminar series, USI, Switzerland. March 4, 2021.
  Paolo Tonella: Do we need new testing techniques for systems that include deep learning components? Keynote at the Swedish Association for Software Testing (SAST). November 5, 2020.
  Paolo Tonella: Ensuring quality and dependability of systems based on deep learning. Keynote at the 8th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE). June 29, 2020.
  Andrea Stocco, Gunel Jahangirova, Vincenzo Riccio (Workshop Co-organizers): DeepTest: 2nd International Workshop on Testing for Deep Learning and Deep Learning for Testing. July 1, 2020.
  Paolo Tonella: How to test a system based on deep learning. Invited talk at the CHOOSE Forum 2019 on Software Testing. Zurich, Switzerland, November 22, 2019.
  Paolo Tonella: What is a bug in deep learning applications? Invited talk at the 61st CREST Open Workshop (COW) on Statistics, Learning, Genetic Improvement and Testing for Programs. London, UK, October 21, 2019.
  Andrea Stocco: Black-box Confidence Estimation for Misbehavior Prediction in Autonomous Driving Systems. Invited talk at the 61st CREST Open Workshop (COW) on Statistics, Learning, Genetic Improvement and Testing for Programs. London, UK, October 21, 2019.
  Paolo Tonella: Can we trust software systems that learn? Invited talk at the Swiss Digital Day. Lugano, Switzerland, September 3, 2019.
  Paolo Tonella: Search Based Test Generation: from Programmed Systems to Systems that Learn. Invited talk at Politecnico di Torino, Italy, June 27, 2019.
  Paolo Tonella: The research challenges of the ERC project PRECRIME. Invited talk at Software Engineering for Machine Learning Applications (SEMLA). Polytechnique Montreal, Canada, June 12, 2018.
SOFTWARE

Software and replication artefacts


  DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score
  DeepCrime: A Tool for Mutation Testing of Deep Learning Systems Based on Real Faults
  Replication package for the ISSTA 2021 paper: DeepCrime: Mutation Testing of Deep Learning Systems based on Real Faults
  DeepHyperion: A Tool for Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search
  Uncertainty-Wizard: Fast and User-Friendly Neural Network Uncertainty Quantification
  DeepJanus: A Tool for Model-based Exploration of the Frontier of Behaviours for Deep Learning Systems Testing
  SelfOracle: Misbehaviour Prediction for Autonomous Driving Systems
  Replication package for the EMSE 2020 paper: Testing Machine Learning based Systems: A Systematic Mapping
  Taxonomy of Real Faults in Deep Learning Systems
  GAssert: A Fully Automated Tool to Improve Assertion Oracles
VIDEOS

Conference presentations and invited talks


  Paolo Tonella: Deep learning testing
  Paolo Tonella: Misconceptions in deep learning testing
  Paolo Tonella: Deep Learning Testing: problems, solutions and elephants
  Gunel Jahangirova: Quality Metrics and Oracles for Autonomous Vehicles Testing
  Vincenzo Riccio: Model-Based exploration of the frontier of behaviours for DL System Testing
  Gunel Jahangirova: An Empirical Validation of Oracle Improvement
  Valerio Terragni: Evolutionary Improvement of Assertion Oracles
  Gunel Jahangirova: An Empirical Evaluation of Mutation Operators for Deep Learning Systems
  Andrea Stocco: Misbehaviour Prediction for Autonomous Driving Systems
  Nargiz Humbatova: Taxonomy of Real Faults in Deep Learning Systems
  Paolo Tonella: Can we trust software systems that learn? (in Italian)
SUPPORTERS

The PRECRIME project is funded by the European Research Council, it is supported by Università della Svizzera italiana (USI), and it was granted 12-months of Amazon AWS cloud credits for research in 2019-2020. In Summer 2019 it was granted two Summer student scholarships and in Summer 2020 one Summer student scholarship funded under the program UROP (Undergraduate Research Opportunities Program) of the Faculty of Informatics. The project is hosted inside the Software Institute's labs at USI.

CONTACTS

CONTACTS

We welcome visiting researchers!

Paolo Tonella, Università della Svizzera italiana (USI), Via G. Buffi 13, 6900 Lugano, Switzerland
Phone: +41 58 666 4848
Email: paolo.tonella@usi.ch