Anita Borg Technical Leadership Award Winner
From Anita Borg Institute Wiki
Testing
Looking for Bugs in All the RIGHT places Elain Weyuker, Tom Ostrand, Bob Bell AT&T Research
Goal: To determine which files of a large industrial software system with multiple releases are particularly likely to be fault-prone Why is this Important? - Help testers prioritize testing efforts - Help developers decide what needs to be re-implemented Maybe re-architect is a good idea
Approach Identified properties that were likely to affect fault-proneness, and then built a statistical model to make predictions
Infrastructure Projects use an integrated change management/version control system. Any Change to the softwr requires that a modification request (MR) be opened.
MRs include information such as the reason that the change is to be made, a description of the change, a severity rating, the actual change, development stage during which the MR was initiated.
Sometimes people don’t capture all the bugs or the changes. Use 1-4 rating of severity on when the system should be changed. Severity 1: Fix it now. 2: Fix it soon 3: Fix it sometime. 4: Cosmetic,
Development stage: Who was this initiated by?
Subgoal 1: Determine whether faults are non-uniformly distributed We did an initial large case study using 12 successive releases (about 3 years in the field) of an inventory control system containing roughly ½ million LOC. Data was collected during 9 development stages. About ¾ of the files were written in java, with smaller numbers written in shell script, makefiles, xml, html, perl, c, sql, awk
Considered Structural Characteristics of Files- Size of file (KLOC) - Age of file - New to current release, and if not, whether it was changed during prior release - The number and magnitude of changes made - The number of observed faults during earlier releases - The number of faults observed during early development stages - Programming language used
Findings- Size is a very good predictor of fault count(but a poor predictor of fault density- fault per line of code) - New files and files that had been changed in the last release were significantly more likely to contain faults than files that had not changed - Other weaker factors include file age, number of faults in previous release, programming language, and release number.
Preliminary Findings- Presented them at an in-house software symposium aimed a practiconers - Goal was to alert projects of what we were doing to get feedback and interest other projects in being subjects in future empirical studies.
Moving on to Predictions- Presented findings at research talks - Enlisted a statistician to collaborate and help us do the statistical modeling
Doing the Actual Prediction We used the identified characteristics, properly weighted.
Model Predicts number of faults in each file of the next release. Then sort the bugs in decreasing order. Then as for the worse percent. From the model, can focus on small part and find majority of bugs.
A Project without Regular releases- Automated Voic Response System - Followed for 27 calendar months - Month 1 contailed 61 files and 16 KLOC Month 27 contained 1888 files and 321 KLOC - Very Heterogenous system – 34 languages used in month 27
Dealing with Lack of Regular Releases- Created Synthetic Releases
- 27 “monthly Releases” - 9 “Quarterly Releases”
- Predictions were surprisingly accurate – on average the top 20% of the predicted files contained 75% of the faults
Projects 4, 5, 6 - Three strand-alone subsystems of a maintenance support system. - Developed and mainainted by a different company - Very mature system – 2 of the subsystems had almost 10 years of filed data, the third 7 years.
Look at Different Models- Negative Binomial Regression - Recursive Partitioning - Random Forests - Bayesian Additive Regression Trees
Important Feature of the Research - Dvelop a theory - Get initial interest from practitioners - Present preliminary results to practitioners an dlisten to feedback - Perform multiple case studies using real large systems to validate results and determine generality - Present results to both practitioners and researchers and listen to feedback - Collaborate with other researchrs with complimentary skills - Recognie that the process may be very time-comsuing
ISSUES to CONSIDER - Awareness by practitioners - Relevant to practitioners - Useable by practitioners
