Basic Information
- Publication: ICSE'17
- Authors: Shin Hwei Tan, Jooyong Yi, Yulis, Sergey Mechtaev, Abhik Roychoudhury
- Language: C Program
- Source: Codeforces Programming Contest (Reject/Accept)
- Description: a set of 3902 defects from 7436 programs automatically classified across 39 defect classes
- Dataset Homepage
Summary
Existing benchmarks (like ManyBugs and IntroClass) on automated program repairs do not allow thorough investigation of the relationship between fault types and the effectiveness of repair tools.
Four criterias for a benchmark that allows extensive evaluation of repair tools:
- C1: Diverse types of real defects.
- C2: Large number of defects.
- C3: Large number of programs.
- C4: Programs that are algorithmically complex
- C5: Large held-out test suite for patch correctness verification
Overall, author crawled over 10000 webpages from Codeforces programming contest. For each rejected submission r, they find another accepted submission a by the same user for the same programming problem in the crawled data. Each fault is represented by the submission pair (r, a). In total, they obtain 5544 defects. Then they further exclude 924 defects due to inadequate held-out tests, 677 defects due to non-reproducible bugs, and 41 defects due to a known CIL bugs2 in handling variable sized multidimensional array.
All defects are divided into 39 classes by using Gumtree on AST-level syntactic differences between buggy program and patched program.