Jump to content

Projects/dataqualitymapp

From ourproject.org wiki
 A Review Protocol
for
"A mapping study1 of data quality in empirical software engineering research"
 Protocol document information:
Authors: Gernot Liebchen, Martin Shepperd, Magne Jørgensen
Contact person: [email protected]
Date: August 2011
Version: 0.7
What’s new
History
 1. Background
 There has been a growing body of empirical software engineering research, partly in response to the proposal that software engineering should become an evidence-based practice [1].  Since such studies are dependant upon the quality of the data, we are concerned with how such judgements can be made and what do other researchers do to tackle this problem? Specifically this investigation is motivated by the offer [to GL and MS] of access to a large industrial data set of software project data but with unknown quality issues. In addition recent evidence [2] indicates that a widely used set of defect data sets from NASA may contain very many errors.
 It is also clear that there are potentially problems related to construct validity, that is, what do we mean by data quality? There are many definitions but a typical one is along the lines of "fitness for purpose" [3]. Unfortunately this is insufficiently concrete for our purposes so we instead use the following definition:
 noise (poor data quality) occurs when data items contain measurement 'errors, i.e. the ''recorded'' value deviates from the, possibly unknown, ''true'' value. 
 DQ need not be viewed as a binary property. DQ could be assessed as the proportion of data items that contain measurement errors.  Where the data item is continuous then DQ might be measured as the size of the difference between the recorded and 'true' value.
 We use poor DQ and noise interchangeably. Nevertheless, we flag up the potential problem of a proliferation of meanings and synonyms that may hinder our proposed Mapping Study. We discuss the notion of construct validity and the detailed inclusion criteria in Section 3.1.
 Finally, a possible threat to the Mapping Study is where researchers have given consideration to DQ but have made no explicit mention of this e.g. if the researcher selects a robust analysis technique because of its ability to tolerate noise but makes no mention of this reasoning. However, it is part of scientific method that methods and rationale is properly reported.
 1.1 Meta-Protocol
 The structure of the Protocol follows (as far as possible for a Mapping Study rather than a Systematic Review) that specified by the Campbell Collaboration (C2)2.
 We intend the Protocol be reviewed by (i) invited researchers who are identified by being active in empirical software engineering research defined as any author of a published paper that cites the original 2008 review [6], (ii) selected experts in the field and (iii) an open process whereby any interested person may comment on the Protocol although editing rights are retained by the Protocol Authors.
 Interested persons may request to join the authoring team.
  2. Objectives
 The objective of the proposed mapping study is to identify current and best practice for addressing data quality problems in the field of empirical software engineering.
 The beneficiaries of the study will be:
  • empirical software engineering researchers because we hope to identify best practice and the limits to this practice
  • it will contribute to evidence-based practice as it will enable us to weigh the strength of empirical evidence in terms of the quality (known or unknown) of the data used
 The top-level research questions are:	
 	RQ1:	Do empirical studies consider DQ at all?
Rationale: so we can understand norms and how seriously the community is
taking DQ
RQ2:	What evidence is there of the extent of DQ problems?
How do these vary:
		RQ2.1	between types of data? Quantitative or qualitative data?
		RQ2.2:	with problem domain?
Rationale: Do we have better than anecdotal evidence concerning the extent of DQ issues and whether this differs between different problem domains (e.g. effort and fault prediction) and types of data. Also different domains may result in different practices in dealing with DQ issues.
	RQ3: 	How and what proportion of empirical studies actually do
something about detecting (and removing) DQ problems?
		RQ3.1: How effective are these DQ detection techniques?
Rationale: to gain a better idea as to what techniques work and in what 			circumstances
	RQ4: 	What if any, techniques are used to polish (replacing the noisy value with the 'true' value) problematic data items?
		RQ4.1:	 How effective are these DQ polishing techniques?
Rationale: to gain a better idea as to what techniques work and in what circumstances
	RQ5: 	What DQ toleration techniques are used (e.g. robust analysis methods)?
Rationale: instead of polishing data analysts may elect to use explicitly use techniques that tolerate noise.
 In terms of PICO:
  • Population or problem of interest – empirical software engineering research studies that explicitly address DQ.
  • Intervention under investigation – the potential impact of noise (and how to prevent or mitigate it) upon performing empirical analysis
  • Comparison of interest – n.a.
  • Outcomes considered most important in assessing results – extent of
    1. DQ being considered
  • noise in data sets – this may be quantitative data or less ideally subjective or other forms of argumentation
    1. techniques to (a) detect (b) polish and / or (c) tolerate noise.
 3. Methods
 The details of the proposed method are informed by Needleman [4], Kitchenham and Chatters [5] and the Campbell Collaboration3
 3.1 Criteria for selecting studies for this review
 a) Types of studies
  1. perform a scholarly analysis of primary or secondary data obtained from the domain of software engineering (e.g. cost analysis, software quality and PDP)
  2. explicitly address the concept of DQ
  3. use data collected from industry-based software engineering i.e. not simulated data or student-related activities unless there is an industrial client
  4. be written in English and refereed
  5. an electronic copy must be available
  6. published after 1990?
b) 	Types of participants (entities)
The entity is the empirical study..
c) 	Types of interventions
The interventions of interest are how the researchers conducted their empirical 		analysis to deal with possible DQ problems.
We categorise these as
  1. detection
  2. polishing and
  3. tolerance techniques.
d) 	Types of outcome measures include:
  1. the prevalence of noise in data sets
  2. frequency of use of noise detection, polishing and tolerance techniques
  3. judgements (where possible) of the impact of such techniques
 3.2 Search methods for identification of studies
 In order to identify candidate studies we propose to:
  1. conduct searches of the major bibliographic databases (ACM, IEEE Xplore, ScienceDirect, Scopus and SpringerLink)
  2. check this with two known Systematic Reviews conducted in 2008 [6] and updated in 2010 [7] and revise search terms and repeat as necessary
  3. hand-review references of selected studies for further potential candidate studies
  4. email all authors of selected studies
 All pdf files of selected papers and a Mapping Study database will be placed on a server accessible to all Study Team members.
 3.3 Data collection and analysis
  1. Application of inclusion/exclusion criteria
  2. Abstract and title screening
    1. Abstract and title screening will ensure that the retrieved documents belong to the empirical software engineering domain.
    2. Full document screening - this will ensure that retrieved documents address data quality according to the definition provided in the background
    3. No quality screening since we already require the papers to be refereed and this is a Mapping Study to characterise the practices of a wide-range of types of study apply to DQ issues.
  3. Characterisation of Included Papers
The characterisation is done carried out independently by two members of the team, 	both reading the papers using keywords and a data collection spreadsheet. Where inconsistencies are detected this will be resolved through discussion, and if need be a 3rd member of the Mapping Study Team.
  1. In depth review of Included Papers
Each paper will be briefly summarised by one member of the Team and checked by another member. This process will capitalise upon the narrative sections of the earlier Systematic Review [7]. Two Team members will independently conduct a thematic analysis of the summaries and the two analyses will be compared and merged (if possible and beneficial).
 4. Timeframe
 The 2010 Systematic Review [7] located 161 relevant studies, we must assume this number will increase, consequently the study will require considerable effort.
Task Target Date
T1: Review and Stabilise the Protocol Sept 2011
T2: Retrieve and Filter Candidate Studies
T3: Validate List of Included Studies
T4: Characterise (keyword) Included Studies
T5: Analysis and synthesis (including thematic analysis) of Included Studies
T6: Write Mapping Study Report
  5. Plans for updating the review 
None at present. We intend to publish the Included Studies and their keywords in an online database similar to the BEST database maintained by Simula Labs, Norway.
 6. Acknowledgements
 7. Statement concerning conflict of interest
None reported.
  References':'
[1]	B. A. Kitchenham, T. Dybå, and M. Jørgensen, "Evidence-based Software Engineering," presented at 27th IEEE Intl. Softw. Eng. Conf. (ICSE 2004), Edinburgh, 2004.
[2]	D. Gray, D. Bowes, N. Davey, Y. Sun, and B. Christianson, "The Misuse of the NASA Metrics Data Program Data Sets for Automated Software Defect Prediction," presented at EASE 2011, Durham, UK, 2011.
[3]	R. D. De Veaux and D. J. Hand, "How to Lie with Bad Data," Statist. Sci., vol. 20, pp. 231–238, 2005.
[4]	I. Needleman, "A guide to systematic reviews," Journal of Clinical Periodontology, vol. 29, pp. 6-9, 2002.
[5]	B. Kitchenham and S. Charters, "Guidelines for Performing Systematic Literature Reviews in Software Engineering, Vesion 2.3," Keele University, UK EBSE Technical Report EBSE-2007-01, 2007.
[6]	G. Liebchen and M. Shepperd, "Data Sets and Data Quality in Software Engineering," presented at PROMISE 2008, Leipzig, 2008.
[7]	G. Liebchen, "Data Cleaning Techniques for Software Engineering Data Sets," in Information Systems and Computing, vol. PhD. London: Brunel, 2011.
              	Other references:
                     	Additional references
                     	Other published versions of this review
	Tables and figures:
             	Additional tables
             	Figures
Supplementary information:
     	Appendices
     	Feedback:
             	Title
             	Summary
             	Reply
             	Contributors
 About the article:
     	Contributions of authors
	Sources of support:
             	Internal sources
             	External sources
 1 	A 	mapping survey is a "form of secondary study intended to 	identify and classify the set of publications on a topic. May be 	used to identify 'evidence gaps' where more primary studies are 	needed as well as 'evidence clusters' where it may be practical to 	perform a systematic literature review". EBSE Project, Durham 	University, 2008 	http://www.dur.ac.uk/ebse/glossary.php?word=mapping%20study.
2 	www.campbellcollaboration.org/
3 	C2 may be found at 	http://www.campbellcollaboration.org/resources/training.php