PRECRIME: Self-assessment Oracles for Anticipatory Testing
Project: European Research Council - Advanced Grant (ERC-AdG)
One of the long-lasting dreams in science fiction is the ability to arrest criminals before they even commit crimes. Software testing researchers have a similar dream: when the context for a bug manifestation occurs in the field, the goal is to discover and fix the bug before it causes any in-field failure. In fact, current practice of pre-release testing is severely limited when dealing with autonomous AI (Artificial Intelligence) systems (such as self-driving cars, robots, automated traders, virtual doctors and customer service chatbots), running in complex, rapidly changing environments, which cause their run-time adaptation, learning and knowledge acquisition, because pre-release testing cannot exhaustively explore all different contexts and states in which the software will be running.
The PRECRIME project introduces a new, disruptive view on testing, called anticipatory testing and aimed at fixing bugs before they even manifest themselves in the field. Anticipatory testing is activated at run-time by a new type of oracles, called self-assessment oracles, which observe and report unexpected execution contexts. A self-assessment oracle is an estimator of the system's confidence in being able to handle a new execution context correctly. The goal of anticipatory testing is to anticipate any failure that might occur in the field due to unexpected execution contexts. Whenever an execution context monitored at runtime by self-assessment oracles is estimated as a low confidence context for the system, anticipatory testing exercises the software automatically and extensively in similar contexts. Timely activation of anticipatory testing by runtime observations results in early, anticipatory fault detection. Combined with automated patch synthesis, anticipatory testing leads to the release of a patch for the fault before any software failure occurs in the field.
The Project activities are organized along the following research lines:
Theoretical foundations: Precrime will define the theoretical framework for anticipatory testing, i.e., for exposing bugs associated with a new execution context before they manifest themselves in the field. The properties that a system should possess to make anticipatory testing effective determine the applicability scope of the proposed approach.
Context analysis and self-assessment: Precrime will introduce a novel context abstraction algorithm that can be used for novelty detection and anticipatory testing. It will also introduce a novel oracle – the self-assessment oracle – to estimate the system’s confidence and possibly trigger anticipatory testing.
Test generation: Precrime will define a novel test adequacy criterion that determines the adequacy of a test suite for a given execution context and will define a novel test generation algorithm, capable of automatically producing a test suite that satisfies the context-aware test adequacy criterion. It will also develop test generation techniques in support to property strengthening and patch validation.
Program properties: Precrime will develop a new interactive algorithm for property adaptation, which can fix any property deficiency that affects the program properties in the given execution context.
Program repair: Precrime will develop a patch synthesis approach based on iterative creation of a candidate patch and search for counterexamples, possibly showing that the patch does not address the fault in all scenarios or has regressions. The iterative counterexample-guided patch-refinement loop terminates when a dependable patch is ready to be released to the user.
Empirical evaluation: Precrime will deliver a series of empirical studies assessing the benefits of anticipatory testing over standard, pre-release testing, both when closed-source/industrial-strength and when open-source/freely available systems are considered.
Annibale Panichella, Fitsum Meshesha Kifetew, Paolo Tonella, Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets.
IEEE Trans. Software Eng. 44(2): 122-158 (2018). DOI
Fitsum Meshesha Kifetew, Roberto Tiella, Paolo Tonella, Generating valid grammar-based test inputs by means of genetic programming and annotated grammars.
Empirical Software Engineering 22(2): 928-961 (2017). DOI
Gunel Jahangirova, David Clark, Mark Harman, Paolo Tonella, Test oracle assessment and improvement.
ISSTA 2016: 247-258. DOI
Paolo Tonella, Roberto Tiella, Duy Cu Nguyen, Interpolated n-grams for model based testing.
ICSE 2014: 562-572. DOI
Alberto Goffi, Alessandra Gorla, Andrea Mattavelli, Mauro Pezzè, Paolo Tonella, Search-based synthesis of equivalent method sequences.
SIGSOFT FSE 2014: 366-376. DOI
Cu D. Nguyen, Alessandro Marchetto, Paolo Tonella, Automated oracles: an empirical study on cost and effectiveness.
ESEC/SIGSOFT FSE 2013: 136-146. DOI
Cu D. Nguyen, Alessandro Marchetto, Paolo Tonella, Combining model-based and combinatorial testing for effective test case generation.
ISSTA 2012: 100-110. DOI
2 Postdoc and 2 PhD positions at USI
Are you looking for an exciting research project for your PhD or Postdoc? The ERC AdG project PRECRIME is scouting excellent applicants to fill 2 Postdoc and 2 PhD positions in the area of software testing, within the topics of self-assessment oracles and test generation. Successful candidates will work at the Università della Svizzera italiana (USI) in Lugano, under the supervision of the ERC holder Prof. Paolo Tonella, who is currently moving from Trento to Lugano.
The research project:
(Self-assessment Oracles for Anticipatory Testing) is a European Research Council - Advanced Grant (ERC-AdG) project that will start in Fall 2018 and will last for five years. The project was inspired by one of the long-lasting dreams in science fiction: the ability to arrest criminals before they even commit crimes. Software testing researchers have a similar dream: When the context for a bug manifestation occurs in the field, the goal is to discover and fix the bug before it causes any in-field failure. PRECRIME will introduce a new view on testing, called anticipatory testing, which is activated at run-time by a new type of oracles, called self-assessment oracles. Combined with automated patch synthesis, anticipatory testing leads to the release of a patch for a fault before any software failure occurs in the field.
The research environment:
USI has recently started the Software Institute (SI, web site: http://si.usi.ch
), a center of excellence devoted to Software Engineering research & development. SI includes CodeLounge, a R&D center with the mission of pushing research ideas beyond prototypes. CodeLounge represents a unique opportunity to facilitate technology transfer and the impact of PRECRIME's research. The Software Institute and the Faculty of Informatics at USI host a unique set of world-class software engineering researchers, such as Michele Lanza, Mauro Pezzè, Antonio Carzaniga and Gabriele Bavota.
PRECRIME will promote collaborative research with other ERC projects and international institutions working in related areas, among which the ERC AdG projects Tune (Prof. Lionel Briand at the University of Luxembourg) and Epic (Prof. Mark Harman at University College London).
At USI, Postdoc and PhD salaries are highly competitive, in accordance with Swiss standards. The selected candidates are expected to start in Fall 2018. Postdoc contracts and PhD scholarships last for up to 5 years. USI is committed to increasing the proportion of female researchers and therefore encourages female candidates to apply.
How to apply:
Applications for Postdoc positions should include a curriculum vitae, a list of publications, one or two references and a research statement. Postdoc candidates should hold a PhD in software testing or related areas by Fall 2018. Applications for PhD positions should include a curriculum vitae and a research statement. PhD candidates should hold a MSc degree in Computer Science by Fall 2018.
If you are interested, send your application to firstname.lastname@example.org
. The call will remain open until all positions are filled, but priority will be given to applications received by June 15th, 2018
We welcome visiting researchers!
Paolo Tonella, Università della Svizzera italiana (USI), Via G. Buffi 13, 6900 Lugano, Switzerland
Mobile phone: +39 380 6898173