Grant: Discovery Grants Program - Individual
McMaster University
$68,000.00
Systematic Evaluation of Assurance Cases
EF 2023-24
Résumé
Background The possibility of complex software-intensive systems contributing to accidents has increased dramatically. In recent years we have seen Assurance Cases (ACs) assume a prominent role in assuring safety of these systems. An AC presents a claim about what we want to assure, and then presents explicit reasoning, backed-up by evidence, that we have adequate confidence that the claim is true. In practice, these claims are not precise and rely on societal norms of what is "good enough". It is thus vital that we develop sound ways of evaluating ACs. There is ongoing research related to "confidence" in ACs, which primarily relates to a "measure" of how sound the conclusion is. "Evaluation" of an AC, in the proposed work, is more general than measuring confidence, and considers multiple aspects of the quality of an AC. Evaluation of ACs has not received the attention it warrants, and should significantly improve system safety . Overarching Goal The 5 year goal of my research is to develop and test evaluation criteria and evaluation processes for ACs. It fits within my research group's mission to develop effective methods for certifying safety-critical software-intensive systems. The current state of the practice for ACs is to use a notation such as Goal Structuring Notation as the basis for the AC. My research on evaluation of the quality of ACs will consider current ACs as well as a new, more formal method for building ACs, that my research group is developing outside of the work in this proposal. Challenges i) ACs are enormous and complex. ii) Most reasoning in ACs is informal. iii) We have few metrics to evaluate ACs. iv) The fundamental claim in an AC is not precise, and relies on societal norms. v) We must achieve incremental assurance, i.e., modify existing ACs to reflect all kinds of changes, without redoing the complete AC - and without "weakening" the AC. vi) An AC may be subject to "confirmation bias". Research Outcomes & Direction We will develop and test criteria that will form the basis of our evaluation methods. In evaluating an AC, we need to distinguish between: 1) the content of the AC; and 2) the documentation structure/notation of the AC. Initial criteria for "content" include: i) convincing basis for the AC, ii) rigour of the argument, iii) support for completeness arguments, iv) repeatability, v) satisfaction of acceptance criteria for evidence. Criteria for "structure" include: i) traceability, ii) robustness with respect to change, iii) understandability, iv) efficiency. External evaluators of an AC may use different evaluation criteria compared with the authors of the AC. We will deal with both cases. For authors of the AC, we will emphasize the use of pair-wise comparison. Impact Effective evaluation of ACs is a necessary step to improve and maintain safety of complex, software-intensive systems. This research will save lives. It will also enhance Canada's reputation in certification of software-intensive systems.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2019-06022
Name nom
Wassyng, Alan
Department département
Computing and Software
Organization id
27
Institution établissement
McMaster University
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2019
Award amount
68000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1206
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Engineering
Area of application fr
Génie
Research subject code
9999
Research subject group en
Not available
Research subject group fr
Not available
Research subject en
Non disponible
Research subject fr
Non disponible
Application title
Systematic Evaluation of Assurance Cases
Keyword
assurance case; assurance case template; automotive software; dependability; safety-critical; security; system safety
Application summary
Background The possibility of complex software-intensive systems contributing to accidents has increased dramatically. In recent years we have seen Assurance Cases (ACs) assume a prominent role in assuring safety of these systems. An AC presents a claim about what we want to assure, and then presents explicit reasoning, backed-up by evidence, that we have adequate confidence that the claim is true. In practice, these claims are not precise and rely on societal norms of what is "good enough". It is thus vital that we develop sound ways of evaluating ACs. There is ongoing research related to "confidence" in ACs, which primarily relates to a "measure" of how sound the conclusion is. "Evaluation" of an AC, in the proposed work, is more general than measuring confidence, and considers multiple aspects of the quality of an AC. Evaluation of ACs has not received the attention it warrants, and should significantly improve system safety . Overarching Goal The 5 year goal of my research is to develop and test evaluation criteria and evaluation processes for ACs. It fits within my research group's mission to develop effective methods for certifying safety-critical software-intensive systems. The current state of the practice for ACs is to use a notation such as Goal Structuring Notation as the basis for the AC. My research on evaluation of the quality of ACs will consider current ACs as well as a new, more formal method for building ACs, that my research group is developing outside of the work in this proposal. Challenges i) ACs are enormous and complex. ii) Most reasoning in ACs is informal. iii) We have few metrics to evaluate ACs. iv) The fundamental claim in an AC is not precise, and relies on societal norms. v) We must achieve incremental assurance, i.e., modify existing ACs to reflect all kinds of changes, without redoing the complete AC - and without "weakening" the AC. vi) An AC may be subject to "confirmation bias". Research Outcomes & Direction We will develop and test criteria that will form the basis of our evaluation methods. In evaluating an AC, we need to distinguish between: 1) the content of the AC; and 2) the documentation structure/notation of the AC. Initial criteria for "content" include: i) convincing basis for the AC, ii) rigour of the argument, iii) support for completeness arguments, iv) repeatability, v) satisfaction of acceptance criteria for evidence. Criteria for "structure" include: i) traceability, ii) robustness with respect to change, iii) understandability, iv) efficiency. External evaluators of an AC may use different evaluation criteria compared with the authors of the AC. We will deal with both cases. For authors of the AC, we will emphasize the use of pair-wise comparison. Impact Effective evaluation of ACs is a necessary step to improve and maintain safety of complex, software-intensive systems. This research will save lives. It will also enhance Canada's reputation in certification of software-intensive systems.