PRECRIME

ABOUT

PRECRIME: Self-assessment Oracles for Anticipatory Testing

Project: European Research Council - Advanced Grant (ERC-AdG)
Duration: 2019-2023

Summary:
One of the long-lasting dreams in science fiction is the ability to arrest criminals before they even commit crimes. Software testing researchers have a similar dream: when the context for a bug manifestation occurs in the field, the goal is to discover and fix the bug before it causes any in-field failure. In fact, current practice of pre-release testing is severely limited when dealing with autonomous AI (Artificial Intelligence) systems (such as self-driving cars, robots, automated traders, virtual doctors and customer service chatbots), running in complex, rapidly changing environments, which cause their run-time adaptation, learning and knowledge acquisition, because pre-release testing cannot exhaustively explore all different contexts and states in which the software will be running.

The PRECRIME project introduces a new, disruptive view on testing, called anticipatory testing and aimed at fixing bugs before they even manifest themselves in the field. Anticipatory testing is activated at run-time by a new type of oracles, called self-assessment oracles, which observe and report unexpected execution contexts. A self-assessment oracle is an estimator of the system's confidence in being able to handle a new execution context correctly. The goal of anticipatory testing is to anticipate any failure that might occur in the field due to unexpected execution contexts. Whenever an execution context monitored at runtime by self-assessment oracles is estimated as a low confidence context for the system, anticipatory testing exercises the software automatically and extensively in similar contexts. Timely activation of anticipatory testing by runtime observations results in early, anticipatory fault detection. Combined with automated patch synthesis, anticipatory testing leads to the release of a patch for the fault before any software failure occurs in the field.

TEAM

Project Team

Paolo Tonella
Principal Investigator

Paolo Tonella

Andrea Stocco
Postdoc

Andrea Stocco

Gunel Jahangirova
Postdoc

Gunel Jahangirova

Vincenzo Riccio
Postdoc

Vincenzo Riccio

Matteo Biagiola
Postdoc

Matteo Biagiola

Nargiz Humbatova
PhD student

Nargiz Humbatova

Michael Weiss
PhD student

Michael Weiss

Tahereh Zohdi
PhD student

Tahereh Zohdi

MSc students

 Marco Calzana, thesis title: "Misbehaviour Prediction in Autonomous Vehicles using Auto-Encoders", September 2019
 André García Gómez, thesis title: "Leveraging Autoencoders for Robustness of Image Classifier based Applications", June 2021
 Brian Pulfer, thesis title: "From Simulated to Real Test Environments for self-driving cars", June 2021
 Nimisha Manjali, thesis title: "Detection of Synthetic Faults in Self-Driving Cars", June 2021
 Rwiddhi Chakraborty, thesis title: "White Box Out of Distribution Supervision in Convolutional Neural Networks", June 2021
 Shubhayu Das, thesis title: "Devising Techniques for the Automated Repair of Trained Deep Neural Networks", June 2021

UROP Summer students

 Simone Giacomelli, Nelson Brochado, project title: "Extreme Testing of Self-Driving Cars", Summer 2019
 Martin Odermatt, project title: "Developing an Interactive Tool to Allow Human Control in MLS Testing", Summer 2020
 Khehra Arshjot Singh, project title: "Developing Self-Driving Car Autopilots using Reinforcement Learning Algorithms", Summer 2021
PUBLICATIONS

Publications

  Matteo Biagiola, Paolo Tonella: Testing the Plasticity of Reinforcement Learning Based Systems. ACM Transactions on Software Engineering and Methodology (TOSEM), 2022 (to appear).
  Andrea Romdhana, Alessio Merlo, Mariano Ceccato, Paolo Tonella: Deep Reinforcement Learning for Black-Box Testing of Android Apps. ACM Transactions on Software Engineering and Methodology (TOSEM), 2022. DOI
  Andrea Stocco, Paolo Tonella: Confidence-driven Weighted Retraining for Predicting Safety-Critical Failures in Autonomous Driving Systems. Journal of Software: Evolution and Process (JSEP), October, 2021. DOI
  Vincenzo Riccio, Nargiz Humbatova, Gunel Jahangirova, Paolo Tonella: DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score. Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2021. DOI
  Tahereh Zohdinasab, Vincenzo Riccio, Alessio Gambi, Paolo Tonella: DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search. Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2021. DOI
  Nargiz Humbatova, Gunel Jahangirova, Paolo Tonella: DeepCrime: A Mutation Testing Tool for Deep Learning Systems. Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2021. DOI
  Gunel Jahangirova, Andrea Stocco, Paolo Tonella: Quality Metrics and Oracles for Autonomous Vehicles Testing. Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST), 2021. DOI
  Michael Weiss, Paolo Tonella: Fail-Safe Execution of Deep Learning based Systems through Uncertainty Monitoring. Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST), 2021. DOI
  Michael Weiss, Rwiddhi Chakraborty, Paolo Tonella: A Review and Refinement of Surprise Adequacy. Proceedings of the International Workshop on Testing for Deep Learning and Deep Learning for Testing (DeepTest), 2021. DOI
  Michael Weiss, Paolo Tonella: Uncertainty-Wizard: Fast and User-Friendly Neural Network Uncertainty Quantification. Proceedings of the IEEE International Conference on Software Testing, Verification and Validation, Tool Track (ICST-Tool), 2021. Best tool paper award. DOI
  Valerio Terragni, Gunel Jahangirova, Paolo Tonella, Mauro Pezzè: GAssert: A Fully Automated Tool to Improve Assertion Oracles. Proceedings of the IEEE International Conference on Software Engineering, Demonstrations Track (ICSE-Demo), 2021. DOI
  Antonia Bertolino, Pietro Braione, Guglielmo De Angelis, Luca Gazzola, Fitsum Kifetew, Leonardo Mariani, Matteo Orrù, Mauro Pezzè, Roberto Pietrantuono, Stefano Russo, Paolo Tonella: A Survey of Field-based Testing Techniques. ACM Computing Surveys (CSUR), 2021. DOI
  Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, Paolo Tonella: Testing Machine Learning based Systems: A Systematic Mapping. Empirical Software Engineering (EMSE), 2020. DOI
  Vincenzo Riccio, Paolo Tonella: Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing. Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2020. DOI
  Valerio Terragni, Gunel Jahangirova, Mauro Pezzè, Paolo Tonella: Evolutionary Improvement of Assertion Oracles. Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2020. DOI
  Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, Paolo Tonella: Taxonomy of Real Faults in Deep Learning Systems. Proceedings of the 42nd International Conference on Software Engineering (ICSE), 2020. DOI
  Andrea Stocco, Michael Weiss, Paolo Tonella: Misbehaviour Prediction for Autonomous Driving Systems. Proceedings of the 42nd International Conference on Software Engineering (ICSE), 2020. DOI
  Gunel Jahangirova, Paolo Tonella: An Empirical Evaluation of Mutation Operators for Deep Learning Systems. Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST), 2020. Best paper award. DOI
  Mariano Ceccato, Davide Corradini, Luca Gazzola, Fitsum Meshesha Kifetew, Leonardo Mariani, Matteo Orrù, Paolo Tonella: A Framework for In-Vivo Testing of Mobile Applications. Proceedings of the 13th IEEE International Conference on Software Testing, Verification and Validation (ICST), 2020. DOI
  Antonia Bertolino, Guglielmo De Angelis, Breno Miranda, Paolo Tonella: Run Java Applications and Test Them In-Vivo Meantime. Proceedings of the 13th IEEE International Conference on Software Testing, Verification and Validation, Tool Track (ICST-Tool), 2020. DOI
  Gunel Jahangirova, David Clark, Mark Harman, Paolo Tonella: An Empirical Validation of Oracle Improvement. IEEE Transactions on Software Engineering (TSE), 2019. DOI
  Mariano Ceccato, Luca Gazzola, Fitsum Meshesha Kifetew, Leonardo Mariani, Matteo Orrù, Paolo Tonella: Toward In-Vivo Testing of Mobile Applications. Proceedings of the 1st International Workshop on Governing adaptive and Unplanned Systems of Systems (GAUSS), 2019. DOI
TECHNICAL REPORTS

Technical Reports

  Matteo Biagiola, Paolo Tonella: Testing the Plasticity of Reinforcement Learning Based Systems. TR-Precrime-2022-01
  Andrea Romdhana, Alessio Merlo, Mariano Ceccato, Paolo Tonella: Deep Reinforcement Learning for Black-Box Testing of Android Apps. TR-Precrime-2022-02
  Andrea Stocco, Paolo Tonella: Confidence-driven Weighted Retraining for Predicting Safety-Critical Failures in Autonomous Driving Systems. TR-Precrime-2022-03
  Michael Weiss, Rwiddhi Chakraborty, Paolo Tonella: A Review and Refinement of Surprise Adequacy. TR-Precrime-2021-01
  Antonia Bertolino, Pietro Braione, Guglielmo De Angelis, Luca Gazzola, Fitsum Kifetew, Leonardo Mariani, Matteo Orrù, Mauro Pezzè, Roberto Pietrantuono, Stefano Russo, Paolo Tonella: A Survey of Field-based Testing Techniques. TR-Precrime-2021-02
  Valerio Terragni, Gunel Jahangirova, Paolo Tonella, Mauro Pezzè: GAssert: A Fully Automated Tool to Improve Assertion Oracles. TR-Precrime-2021-03
  Michael Weiss, Paolo Tonella: Uncertainty-Wizard: Fast and User-Friendly Neural Network Uncertainty Quantification. TR-Precrime-2021-04
  Tahereh Zohdinasab, Vincenzo Riccio, Alessio Gambi, Paolo Tonella: DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search. TR-Precrime-2021-05
  Nargiz Humbatova, Gunel Jahangirova, Paolo Tonella: DeepCrime: Mutation Testing of Deep Learning Systems Based on Real Faults. TR-Precrime-2021-06
  Vincenzo Riccio, Nargiz Humbatova, Gunel Jahangirova, Paolo Tonella: DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score. TR-Precrime-2021-07
  Vincenzo Riccio, Paolo Tonella: Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing. TR-Precrime-2020-01
  Valerio Terragni, Gunel Jahangirova, Mauro Pezzè, Paolo Tonella: Evolutionary Improvement of Assertion Oracles. TR-Precrime-2020-02
  Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, Paolo Tonella: Testing Machine Learning based Systems: A Systematic Mapping. TR-Precrime-2020-03
  Gunel Jahangirova, Andrea Stocco, Paolo Tonella: Quality Metrics and Oracles for Autonomous Vehicles Testing. TR-Precrime-2020-04
  Michael Weiss, Paolo Tonella: Fail-Safe Execution of Deep Learning based Systems through Uncertainty Monitoring. TR-Precrime-2020-05
  Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, Paolo Tonella: Taxonomy of Real Faults in Deep Learning Systems. TR-Precrime-2019-01
  Andrea Stocco, Michael Weiss, Paolo Tonella: Misbehaviour Prediction for Autonomous Driving Systems. TR-Precrime-2019-02
  Gunel Jahangirova, Paolo Tonella: An Empirical Evaluation of Mutation Operators for Deep Learning Systems. TR-Precrime-2019-03
  Gunel Jahangirova, David Clark, Mark Harman, Paolo Tonella: An Empirical Validation of Oracle Improvement. TR-Precrime-2019-04
  Mariano Ceccato, Davide Corradini, Luca Gazzola, Fitsum Meshesha Kifetew, Leonardo Mariani, Matteo Orrù, Paolo Tonella: A Framework for In-Vivo Testing of Mobile Applications. TR-Precrime-2019-05
  Antonia Bertolino, Guglielmo De Angelis, Breno Miranda, Paolo Tonella: Run Java Applications and Test Them In-Vivo Meantime. TR-Precrime-2019-06
  Mariano Ceccato, Luca Gazzola, Fitsum Meshesha Kifetew, Leonardo Mariani, Matteo Orrù, Paolo Tonella: Toward In-Vivo Testing of Mobile Applications. TR-Precrime-2019-07
EVENTS AND TALKS

Events and Invited Talks


  Paolo Tonella: Deep learning testing. Keynote at the 15th International Workshop on Search-Based Software Testing (SBST). May 9, 2022.
  Paolo Tonella: Misconceptions in deep learning testing. Keynote at the IEEE International Conference on Software Testing (ICST). April 5, 2022.
  Paolo Tonella: How can we expose, explain and prevent deep learning faults? Invited talk at the University of Virginia, USA. October 27, 2021.
  Paolo Tonella: How can we expose and explain deep learning faults? Keynote at the Huawei Strategy and Technology Workshop, Beijing, China. October 15, 2021.
  Paolo Tonella: How can we expose and explain deep learning faults? Invited talk at the University of Sheffield, UK. September 21, 2021.
  Paolo Tonella: Precrime: testing deep learning components against unexpected execution conditions. Invited talk at the Technical University of Darmstadt, Germany. July 5, 2021.
  Andrea Stocco, Gunel Jahangirova, Vincenzo Riccio (Workshop Co-organizers): DeepTest: 3rd International Workshop on Testing for Deep Learning and Deep Learning for Testing. June 1, 2020.
  Paolo Tonella: Testing the unexpected: the Precrime way towards dependable systems that include deep learning components. Invited talk at the University of St. Gallen, Switzerland. March 19, 2021.
  Paolo Tonella: Deep Learning Testing: problems, solutions and elephants. Talk at the Software Institute seminar series, USI, Switzerland. March 4, 2021.
  Paolo Tonella: Do we need new testing techniques for systems that include deep learning components? Keynote at the Swedish Association for Software Testing (SAST). November 5, 2020.
  Paolo Tonella: Ensuring quality and dependability of systems based on deep learning. Keynote at the 8th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE). June 29, 2020.
  Andrea Stocco, Gunel Jahangirova, Vincenzo Riccio (Workshop Co-organizers): DeepTest: 2nd International Workshop on Testing for Deep Learning and Deep Learning for Testing. July 1, 2020.
  Paolo Tonella: How to test a system based on deep learning. Invited talk at the CHOOSE Forum 2019 on Software Testing. Zurich, Switzerland, November 22, 2019.
  Paolo Tonella: What is a bug in deep learning applications? Invited talk at the 61st CREST Open Workshop (COW) on Statistics, Learning, Genetic Improvement and Testing for Programs. London, UK, October 21, 2019.
  Andrea Stocco: Black-box Confidence Estimation for Misbehavior Prediction in Autonomous Driving Systems. Invited talk at the 61st CREST Open Workshop (COW) on Statistics, Learning, Genetic Improvement and Testing for Programs. London, UK, October 21, 2019.
  Paolo Tonella: Can we trust software systems that learn? Invited talk at the Swiss Digital Day. Lugano, Switzerland, September 3, 2019.
  Paolo Tonella: Search Based Test Generation: from Programmed Systems to Systems that Learn. Invited talk at Politecnico di Torino, Italy, June 27, 2019.
  Paolo Tonella: The research challenges of the ERC project PRECRIME. Invited talk at Software Engineering for Machine Learning Applications (SEMLA). Polytechnique Montreal, Canada, June 12, 2018.
SOFTWARE

Software and replication artefacts


  DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score
  DeepCrime: A Tool for Mutation Testing of Deep Learning Systems Based on Real Faults
  Replication package for the ISSTA 2021 paper: DeepCrime: Mutation Testing of Deep Learning Systems based on Real Faults
  DeepHyperion: A Tool for Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search
  Uncertainty-Wizard: Fast and User-Friendly Neural Network Uncertainty Quantification
  DeepJanus: A Tool for Model-based Exploration of the Frontier of Behaviours for Deep Learning Systems Testing
  SelfOracle: Misbehaviour Prediction for Autonomous Driving Systems
  Replication package for the EMSE 2020 paper: Testing Machine Learning based Systems: A Systematic Mapping
  Taxonomy of Real Faults in Deep Learning Systems
  GAssert: A Fully Automated Tool to Improve Assertion Oracles
VIDEOS

Conference presentations and invited talks


  Paolo Tonella: Deep learning testing
  Paolo Tonella: Misconceptions in deep learning testing
  Paolo Tonella: Deep Learning Testing: problems, solutions and elephants
  Gunel Jahangirova: Quality Metrics and Oracles for Autonomous Vehicles Testing
  Vincenzo Riccio: Model-Based exploration of the frontier of behaviours for DL System Testing
  Gunel Jahangirova: An Empirical Validation of Oracle Improvement
  Valerio Terragni: Evolutionary Improvement of Assertion Oracles
  Gunel Jahangirova: An Empirical Evaluation of Mutation Operators for Deep Learning Systems
  Andrea Stocco: Misbehaviour Prediction for Autonomous Driving Systems
  Nargiz Humbatova: Taxonomy of Real Faults in Deep Learning Systems
  Paolo Tonella: Can we trust software systems that learn? (in Italian)
OPEN PHD POSITION

One PhD position at USI


Are you looking for an exciting research project for your PhD? The ERC project PRECRIME is scouting excellent applicants to fill 1 PhD position in the area of software testing, in particular testing of deep learning based systems. Successful candidates will work at Università della Svizzera italiana (USI) in Lugano, under the supervision of the ERC holder Prof. Paolo Tonella, within the PRECRIME team.

The research project:

Precrime (Self-assessment Oracles for Anticipatory Testing) is a European Research Council - Advanced Grant (ERC-AdG) project that has started in January 2019 and will last for five years. The project was inspired by one of the long-lasting dreams in science fiction: the ability to arrest criminals before they even commit crimes. Similarly, we want to discover and fix bugs before they cause any in-field failure. In particular, we want to automatically create the unexpected execution conditions in which a deep learning based system might experience high uncertainty and eventually misbehaviours. In this PhD, we will investigate reinforcement learning based approaches to evolve a testing agent with the capabilities to challenge the deep learning agent under test.

The research environment:

USI has recently started the Software Institute (SI, web site: http://si.usi.ch), a center of excellence devoted to Software Engineering research and development. SI includes CodeLounge, a R&D center with the mission of pushing research ideas beyond prototypes. The Software Institute and the Faculty of Informatics at USI host a unique set of world-class software engineering researchers, such as Michele Lanza, Mauro Pezzè, Antonio Carzaniga, Carlo-Alberto Furia and Gabriele Bavota.
At USI, PhD salaries are highly competitive, in accordance with the Swiss standards. The selected candidate is expected to start in the first half of 2022. PhD scholarships on PRECRIME last for 4 years. USI is committed to increasing the proportion of female researchers and therefore encourages female candidates to apply.

How to apply:

Applications should include a curriculum vitae and the transcripts of the exams taken so far. PhD candidates should hold a MSc degree in Computer Science by Fall 2022. Interested candidates can send their application to paolo.tonella@usi.ch . The call will remain open until the advertised position is filled, but priority will be given to applications received by June 30th, 2022.

Further Information:

Paolo Tonella: https://www.inf.usi.ch/faculty/tonella/
Università della Svizzera italiana (USI): https://www.usi.ch
Software Institute at USI: https://www.si.usi.ch
Faculty of Informatics at USI: http://www.inf.usi.ch
SUPPORTERS

The PRECRIME project is funded by the European Research Council, it is supported by Università della Svizzera italiana (USI), and it was granted 12-months of Amazon AWS cloud credits for research in 2019-2020. In Summer 2019 it was granted two Summer student scholarships and in Summer 2020 one Summer student scholarship funded under the program UROP (Undergraduate Research Opportunities Program) of the Faculty of Informatics. The project is hosted inside the Software Institute's labs at USI.

CONTACTS

CONTACTS

We welcome visiting researchers!

Paolo Tonella, Università della Svizzera italiana (USI), Via G. Buffi 13, 6900 Lugano, Switzerland
Phone: +41 58 666 4848
Email: paolo.tonella@usi.ch