On the Unusual Effectiveness of Type-Aware Operator Mutations for Testing SMT Solvers
Tue 17 Nov 2020 04:00 - 04:20 at SPLASH-I - M-5 Chair(s): Bernardo Toninho, Xiangzhe Xu
We propose type-aware operator mutation, a simple, but unusually effective approach for testing SMT solvers. The key idea is to mutate operators of conforming types within the seed formulas to generate well-typed mutant formulas. These mutant formulas are then used as the test cases for SMT solvers. We realized type-aware operator mutation within the OpFuzz tool and used it to stress-test Z3 and CVC4, two state-of-the-art SMT solvers. Type-aware operator mutations are unusually effective: During one year of extensive testing with OpFuzz, we reported 1092 bugs on Z3’s and CVC4’s respective GitHub issue trackers, out of which 819 unique bugs were confirmed and 685 of the confirmed bugs were fixed by the developers. The detected bugs are highly diverse — we found bugs of many different types (soundness bugs, invalid model bugs, crashes, etc.), logics and solver configurations. We have further conducted an in-depth study of the bugs found by OpFuzz. The study results show that the bugs found by OpFuzz are of high quality. Many of them affect core components of the SMT solvers’ codebases, and some required major changes for the developers to fix. Among the 819 confirmed bugs found by OpFuzz,184 were soundness bugs, the most critical bugs in SMT solvers,and 489 were in the default modes of the solvers. Notably, OpFuzz found 27 critical soundness bugs in CVC4, which has proved to be a very stable SMT solver.
Mon 16 NovDisplayed time zone: Central Time (US & Canada) change
15:00 - 16:20 | M-5OOPSLA at SPLASH-I +12h Chair(s): Jonathan Aldrich Carnegie Mellon University, Leonidas Lampropoulos University of Maryland, College Park | ||
15:00 20mTalk | CAMP: Cost-Aware Multiparty Session Protocols OOPSLA Link to publication DOI Media Attached | ||
15:20 20mTalk | Counterexample-Guided Correlation Algorithm for Translation Validation OOPSLA Link to publication DOI Media Attached | ||
15:40 20mTalk | Multiparty Motion Coordination: From Choreographies to Robotics Programs OOPSLA Link to publication DOI Media Attached | ||
16:00 20mTalk | On the Unusual Effectiveness of Type-Aware Operator Mutations for Testing SMT Solvers OOPSLA Link to publication DOI Media Attached |
Tue 17 NovDisplayed time zone: Central Time (US & Canada) change
03:00 - 04:20 | M-5OOPSLA at SPLASH-I Chair(s): Bernardo Toninho Nova University of Lisbon / NOVA-LINCS, Xiangzhe Xu Nanjing University | ||
03:00 20mTalk | CAMP: Cost-Aware Multiparty Session Protocols OOPSLA Link to publication DOI Media Attached | ||
03:20 20mTalk | Counterexample-Guided Correlation Algorithm for Translation Validation OOPSLA Link to publication DOI Media Attached | ||
03:40 20mTalk | Multiparty Motion Coordination: From Choreographies to Robotics Programs OOPSLA Link to publication DOI Media Attached | ||
04:00 20mTalk | On the Unusual Effectiveness of Type-Aware Operator Mutations for Testing SMT Solvers OOPSLA Link to publication DOI Media Attached |