SPLASH 2020
Sun 15 - Sat 21 November 2020 Online Conference
Tue 17 Nov 2020 09:20 - 09:40 at OOPSLA/ECOOP - T-2
Tue 17 Nov 2020 21:20 - 21:40 at OOPSLA/ECOOP - T-2

Flaky tests are tests that can non-deterministically pass or fail for the same code version. These tests undermine regression testing efficiency, because developers cannot easily identify whether a test fails due to their recent changes or due to flakiness. Ideally, one would detect flaky tests right when flakiness is introduced, so that developers can then immediately remove the flakiness. Some software organizations, e.g., Mozilla and Netflix, run some tools—detectors—to detect flaky tests as soon as possible. However, detecting flaky tests is costly due to their inherent non-determinism, so even state-of-the-art detectors are often impractical to be used on all tests for each project change. To combat the high cost of applying detectors, these organizations typically run a detector solely on newly added or directly modified tests, i.e., not on unmodified tests or when other changes occur (including changes to the test suite, the code under test, and library dependencies). However, it is unclear how many flaky tests can be detected or missed by applying detectors in only these limited circumstances.

To better understand this problem, we conduct a large-scale longitudinal study of flaky tests to determine when flaky tests become flaky and what changes cause them to become flaky. We apply two state-of-the-art detectors to 55 Java projects, identifying a total of 245 flaky tests that can be compiled and run in the code version where each test was added. We find that 75% of flaky tests (184 out of 245) are flaky when added, indicating substantial potential value for developers to run detectors specifically on newly added tests. However, running detectors solely on newly added tests would still miss detecting 25% of flaky tests. The percentage of flaky tests that can be detected does increase to 85% when detectors are run on newly added or directly modified tests. The remaining 15% of flaky tests become flaky due to other changes and can be detected only when detectors are always applied to all tests. Our study is the first to empirically evaluate when tests become flaky and to recommend guidelines for applying detectors in the future.

Tue 17 Nov
Times are displayed in time zone: Central Time (US & Canada) change

09:00 - 10:20: T-2OOPSLA at OOPSLA/ECOOP +12h
09:00 - 09:20
Talk
OOPSLA
Aaron BembenekHarvard University, Michael GreenbergPomona College, Stephen ChongHarvard University
09:20 - 09:40
Talk
OOPSLA
Wing LamUniversity of Illinois at Urbana-Champaign, Stefan WinterTU Darmstadt, Germany, Anjiang WeiPeking University, Tao XiePeking University, Darko MarinovUniversity of Illinois at Urbana-Champaign, Jonathan BellNortheastern University
09:40 - 10:00
Talk
OOPSLA
Yizhou ZhangUniversity of Waterloo, Guido SalvaneschiUniversity of St. Gallen, Andrew MyersCornell University
10:00 - 10:20
Talk
OOPSLA
Hongyu LiuPurdue University, Sam SilvestroUniversity of Texas at San Antonio, USA, Xiangyu ZhangPurdue University, USA, Jian HuangUniversity of Illinois at Urbana-Champaign, Tongping LiuUniversity of Massachusetts Amherst
21:00 - 22:20: T-2OOPSLA at OOPSLA/ECOOP
21:00 - 21:20
Talk
OOPSLA
Aaron BembenekHarvard University, Michael GreenbergPomona College, Stephen ChongHarvard University
21:20 - 21:40
Talk
OOPSLA
Wing LamUniversity of Illinois at Urbana-Champaign, Stefan WinterTU Darmstadt, Germany, Anjiang WeiPeking University, Tao XiePeking University, Darko MarinovUniversity of Illinois at Urbana-Champaign, Jonathan BellNortheastern University
21:40 - 22:00
Talk
OOPSLA
Yizhou ZhangUniversity of Waterloo, Guido SalvaneschiUniversity of St. Gallen, Andrew MyersCornell University
22:00 - 22:20
Talk
OOPSLA
Hongyu LiuPurdue University, Sam SilvestroUniversity of Texas at San Antonio, USA, Xiangyu ZhangPurdue University, USA, Jian HuangUniversity of Illinois at Urbana-Champaign, Tongping LiuUniversity of Massachusetts Amherst