A Hands-On Approach for Teaching Systematic Review

Maria Teresa Baldassarre, Nicola Boffoli, Danilo Caivano, and Giuseppe Visaggio

Department of Informatics, University of Bari RCOST Bari

{baldassarre, boffoli, caivano, visaggio}@di.uniba.it

Abstract. An essential part of a software engineering education is technology

innovation. Indeed software engineers, as future practitioners, must be able to
identify the most appropriate technologies to adopt in projects. As so, it is
important to develop the skills that will allow them to evaluate and make
decisions on tools, technologies, techniques and methods according to the
available empirical evidence reported in literature. In this sense, a rigorous
manner for analyzing and critically addressing literature is Systematic Review.
It requires formalizing an answerable research question according to the
problem or issues to face; search the literature for available evidence according
to a systematic protocol and retrieve data from the identified sources; analyze
the collected evidence and use it to support decision making and conclusions. In
this paper we report on how Systematic Review has been integrated in the
Empirical Software Engineering Methods course that is taught at the
Department of Informatics at the University of Bari, and how students have
been introduced to this type of literature review through a hands-on approach.
As far as we know, it is the first attempt of including a complex topic like
systematic review in a university course on empirical software engineering. We
have no empirical evidence on the effectiveness of the approach adopted, other
than practice-based experience that we have acquired. Nonetheless, we have
collected qualitative data through a questionnaire submitted to the students of
the course. Their positive answers and impressions are a first informal
confirmation of the successful application of our strategy.

Keywords: Empirical Software Engineering, Systematic Review, Statistical

Process Control, Evidence Based Software Engineering.

1 Introduction
Empirical Software Engineering (ESE) is an important component of any software
engineers curricula as it trains students to evaluate and make decisions on tools,
technologies, techniques and methods according to the available evidence reported in
literature. Indeed this is pointed out in the Guidelines for Software Engineering
Education [3] that shows how SEEK (SE Education Knowledge) can be taught
according to the volumes guidelines. For each knowledge area, there is a short
description and then a table that delineates the units and topics for that area. For each
knowledge unit, recommended contact hours are designated. For each topic, a Bloom
taxonomy level [5] (indicating what capability a graduate should possess) and the
topics relevance (indicating whether the topic is essential, desirable, or optional to

A. Jedlitschka and O. Salo (Eds.): PROFES 2008, LNCS 5089, pp. 415426, 2008.
Springer-Verlag Berlin Heidelberg 2008
416 M.T. Baldassarre et al.

the core) are designated. In this context, ESE can be categorized as part of the
Mathematical and Engineering Fundamental. In particular, within this knowledge
area, it can be seen as empirical methods and experimental techniques topic of the
engineering foundations for software unit. This area is classified, with an Essential
relevance, i.e. the topic is part of the core, and according to the Bloom taxonomy level
it is considered as c comprehension, i.e. students should be capable to understand
information and the meaning of material presented. For example, be able to translate
knowledge to a new context, interpret facts, compare, contrast, order, group, infer
causes, predict consequences, etc. Meyer [19] points out that software engineering
education trains professionals for the industry, while ESE develops the skills for
empirically validating tools, techniques used, developed and aimed for industry.
The above considerations point out how important it is to develop the skills that
will allow software engineering students, also future practitioners, to critically and
systematically evaluate the best available evidence on a specific issue of interest, may
it be a tool, method, technique or other. In order to achieve such goal students must be
able to apply the steps involved in evidence based software engineering [14]:
1. convert a problem or need for information into a research question
2. search the literature for evidence to answer the question
3. critically analyze the evidence
4. combine the evidence with previous knowledge and individual experience
5. evaluate performances and eventually make improvements.
This asks for concepts in empirical software engineering, in that students must acquire
the fundamental elements of empirical methods and techniques used for validating a set
of hypotheses, as well as skills for searching literature and critically addressing research
questions. At the Department of Informatics at the University of Bari our graduate
course in Informatics includes a mandatory course called Empirical Software
Engineering Methods. In accordance to the SEEK guidelines, the course introduces
students to empirical methods such as surveys, case studies and experiments. It also
gives elements of empirical based software engineering (formalized in the above steps)
and trains students on how to empirically evaluate software engineering tools,
techniques, methods and technologies. Within the course, we have achieved step 2 of
the above activities through a Systematic Review. A systematic review is a formal
approach for reviewing research literature [17]. As reviews are often limited to
annotated bibliographies, a systematic review means giving appropriate breadth and
depth, rigor and consistency, let alone effective analysis and synthesis of the literature.
Furthermore, it can be considered as much more effort prone than an ordinary literature
survey. The latter being formally defined as the selection of available documents (both
published and unpublished) on the topic, which contain information, ideas, data and
evidence written from a particular standpoint to fulfill certain aims or express certain
views on the nature of the topic and how it is to be investigated, and the effective
evaluation of these documents in relation to the research being proposed [12]. More so,
it has less scientific value than a systematic review, formally defined as a means of
evaluating and interpreting all available research relevant to a particular research
question or topic area or phenomenon of interest [17].
In this paper we describe how students have been introduced and addressed to
carrying out systematic reviews as part of the above listed EBSE process within the
 A Hands-On Approach for Teaching Systematic Review 417

Empirical SE Methods course at the University of Bari. To make things easier and
more interesting we have used a hands-on approach and actively involved students in
a real systematic review on the topic of Statistical Process Control (SPC) [9]. We
have carried out some type of qualitative evaluation to assess the student opinions.
Results point out positive answers and impressions and therefore confirm the
approach we adopted.
The rest of the paper is organized as follows: section 2 introduces the reader to the
basic concepts of systematic review which we consider part of the EBSE process;
section 3 illustrates the approach used to involve university graduate students in a
review on the topic of statistical process control. Section 4 presents the general
comments on the opinions collected; finally conclusions are drawn.

2 Searching Evidence through a Systematic Review

An important step of the EBSE process is the search of evidence for answering the
research question. The more evidence found (of both positive and negative results on
the topic being investigated), the more support to rational decision making is assured.
Search of evidence can be done informally from various information sources such as
retrieving customer or software user viewpoints, or asking for expert judgment; or
formally as research-based evidence from sources such as scientific journals, books,
grey literature. In our course we emphasize systematically searching evidence through
a rigorous approach like systematic review. As so, step 2 of the EBSE process
consists in searching the appropriate evidence through a systematic review.
Guidelines on systematic review have been defined and are quite stable in contexts
such as medicine, social sciences, education and information sciences and used for
analyzing and synthesizing existing empirical results on a certain topic. Indeed, there are
many existing guidelines in this field that include the Cochrane Reviewers Handbook
[7], Guidelines of the Australian National Health and medical Research Council [1, 2];
CRD Guidelines for those Carrying Out or Commissioning Reviews [16].
Adaptations of these guidelines to software engineering have been made by
Kitchenham in [17]. Also, applications of the procedure for performing a systematic
review are becoming more and more common to the software engineering context in
the past few years [6, 10, 11] and many studies have been carried out on various
topics of interest that range from cost-estimation [15], within and cross company
estimation models [18] to software process improvement [20], to statistical power [8].
We have defined and begun a systematic review on Statistical Process Control
(SPC) [9]. Further details on the study itself can be found in [4].
In this paper we describe how graduate students of our Empirical SE Methods
course have been actively involved in conducting the systematic review.

2.1 Systematic Review Concepts

In spite of the growing importance that systematic review has been achieving in the
past years, it is still a quite new topic to the software engineering community. As so,
before going on, we will provide the reader some preliminary concepts to make the
rest of the paper easier to understand. More details on how to organize a systematic
418 M.T. Baldassarre et al.

review can be found in [17], this section synthesizes the information extracted from
this report.
In general, a systematic review can be seen as a process made up of three main
phases: planning the review, conducting the review, reporting the review (Fig. 1).

1. Specify Research

Plan Review 2. Develop Review Protocol

3. Validate Review Protocol

4. Identify Relevant Research

5. Select Primary Studies

Conduct Review 6. Assess Study Quality

7. Extract Required Data

8. Synthesize Data

9. Write Review Report

Document Review
10. Validate Report

Fig. 1. Systematic Review Process

This first phase of planning involves specifying the research by motivating the
need for information search; developing the review protocol: the review protocol is
what formally specifies the steps and procedures used for carrying out the systematic
review. It is important for the protocol to be defined before starting the review in
order to avoid that results are in any way influenced by researcher expectations and
desiderata. At this point the research questions are formulated and search strings are
defined based on an analysis of the questions. In defining a search string it is
important to keep in mind that: major terms are identified from the topic area,
intervention and outcomes. Search strings should include synonyms, related terms and
alternative spelling for major terms; boolean OR is used to incorporate alternative
spellings and synonyms; boolean AND is used to link major terms. The following
details should be used for constructing search strings:
o Population: Software development and maintenance projects and tasks
o Intervention: Statistical process control
o Outcomes: Reported benefits , reported limitations, task type, software
attribute controlled (e.g. productivity, defect rate)
The search process is divided into two parts. First primary sources are identified from
scientific journals, bibliographical databases, digital libraries, electronic databases on
the Internet. Next, secondary sources are searched. The secondary search phase
 A Hands-On Approach for Teaching Systematic Review 419

consists in checking primary sources, identified in the initial search phase, for other
relevant publications; and in contacting researchers who authored primary sources
who we believe could be working on the topic, to enquire whether they have other
unpublished papers or technical reports (i.e. grey literature).
Specific tables (Data Extraction Forms and Aggregation Tables) must then be
defined for documenting all the outcomes of the search process and for accurately
collecting and recording the information of all reviewed papers. The entire process
should be rigorously documented, for example in spreadsheets, database tables. As
final part of the protocol, selection criteria and procedures have to be defined. They
determine the criteria for including or excluding sources from the systematic review.
In carrying out the review, an initial selection of primary sources is carried out after
examining the title and abstract, in order to exclude primary sources that appear
completely irrelevant to the information need of the research question.
A good rule of thumb is to assign at least two researchers to review the search list
and keep a record of the selected papers. Selected papers are then to be reviewed
against the inclusion/exclusion criteria using the same process used for the abstracts.
Reasons for inclusion/exclusion must also be recorded on the spreadsheet.
The third activity of planning involves validating the review protocol. It is
suggested that reviewers external to the study also be involved in this activity to avoid
biases. Validation aims to make sure that all the information extracted relates to the
research questions the review intends answering.
Once the protocol is finalized and validated, the next step is to conduct the review,
i.e. apply all the steps as they have been formally defined. This points out the
importance of the first phase in making the process replicable and adoptable by other
researchers that may differ from those having defined the protocol itself.
Documentation of the steps in this phase is also crucial to keep track of results. This
phase includes: identifying relevant research through the search strings; selecting
primary studies according to the inclusion/exclusion criteria; synthesizing the data in
data extraction tables defined according to the information needs of each research
question.
The final phase is to document the review. This phase is the conclusive part of the
review, important for communicating the results of the study. It can be done in
formats such as technical reports, journal or conference papers, as well as non
technical articles or web pages.

3 The Hands-On Approach

In this section we will describe how university graduate students have been involved
in a systematic review not only by teaching them the theory, but also by allowing
them to apply the concepts. The graduate students involved in the study are students
at their first year of a MSc degree, all with a BSc degree in informatics or
engineering. They all attended the Empirical SE Methods course held by the
authors of this work. The topic of the review, Statistical Process Control, was part of
their course program, so students were all familiar with it. Given the aim of the paper
and the space available, we will not go into detail on SPC. Further details on the topic,
and on the review results can be found in [4, 9]. We scheduled our classes in order to
420 M.T. Baldassarre et al.

train students first, and then receive feedback before assigning them the papers. In this
sense our major effort was to introduce them to systematic review. Students
participated on a volunteer basis. We defined a schedule similar to the one adopted in
occasion of the International Advanced School on Empirical Software Engineering
(IASESE 2005), with the difference that we had more time available for training
students and receiving feedback before assigning them the reviews. The schedule we
followed is commented below and reported in Table 1.

Table 1. Schedule of the systematic review

Lesson 1 Systematic review We introduced the systematic review methodology

guidelines & Seminar according to the guidelines in [17]; B.Kitchenham held a
Seminar.
Lesson 2 Experiences of Some examples of systematic review carried out in
undertaking a literature were presented to the students.
systematic review
Lesson 3 Revise SPC concepts A general overview of SPC

Lesson 4 Define search terms, Search terms, inclusion/exclusion criteria and data
inclusion/exclusion extraction forms were defined according to the research
criteria, data extraction questions and the search motivation provided to students.
forms The task was assigned as homework.
Lesson 5 Discussion Discussion of proposals.
Final validated version of the protocol. The protocol was
handed out to students so they could familiarize with all
the material for the assignment.
Lesson 6 Search the sources Students were divided into groups, one for each search
source. They selected papers according to the protocol
criteria.
Lesson 7 Group work guided Students were assigned a paper on SPC and were asked
exercise to extract data from the primary source, fill in extraction
forms and aggregate data
Lesson 8 Group work feedback Correctly completed forms were handed out to students.
on guided exercise Obtained results were discussed in groups of 2 and then
with the class.
Lesson 9 Assignment of papers Selected primary sources were assigned to students.
They worked individually at home.
Lesson 10 Group work on assigned Students that worked on the same paper individually
papers confronted their data extraction tables and aggregation
tables in groups with the other students that worked on
the same paper.

First we introduced the students to systematic review and presented the guidelines
illustrated in [17]. Barbara Kitchenham also held a seminar on the guidelines (Lesson 1).
On the next day (Lesson 2), we illustrated some examples of reviews carried out in
literature, supported by published papers and technical reports [8, 18]. Although SPC is
part of the students course program, we thought it was the case to refresh their minds
on the topic, so we dedicated a lesson (Lesson 3) on the concepts that would appear in the
papers to revise. Next, we defined the systematic review protocol on SPC as a class
assignment. It is the case to point out that our research group, in collaboration with
Barbara Kitchenham, had previously set up a preliminary version of the protocol on this
topic. As so, search motivation, research goal and a sketch of the data extraction forms
were clear to us. Consequently, we considered the definition part of the protocol as a
 A Hands-On Approach for Teaching Systematic Review 421

useful training exercise for the students and a manner for receiving feedback on our
behalf. On Lesson 4 we gave students the search motivation and research questions and
asked them to identify possible search terms (in class). We assigned the definition of
inclusion/exclusion criteria and data extraction forms as homework. Individual work and
proposals were then discussed in class during lesson 5. We finally came up with a
definitive version of the protocol which was validated by the researchers working on the
project. We also provided the list of search sources, i.e. 8 digital libraries for retrieving
the papers on SPC.
At this point we presented the complete and validated version of the SPC
systematic review protocol and outlined all the details: tables, extraction procedures,
inclusion/exclusion and quality assessment criteria. In Lesson 6 students were
randomly divided into 8 groups, one for each digital library source, and a person of
our research group was assigned to each group as supervisor. Students searched for
papers according to the search terms and search strings; read titles and abstracts and
adopted inclusion/exclusion criteria to select relevant papers. All decisions were
motivated and reported on a spreadsheet. Lists were handed in by each group and
results were discussed in class. A total of 129 sources were identified. After excluding
duplicates we had a set of 96 relevant titles. Given the number of students in the class
(77) we selected a set of 24 papers to review according to the protocol. This initial set
of papers was identified from the digital libraries that we had access to as University
through an account. Also, we tried to balance the total workload for each student
according to the length of papers and type of journals they were retrieved from and
made sure that each paper was analyzed by at least 3 students. So, we had cases of
students assigned to two papers (i.e. the case of short papers) and cases of papers read
by more than 3 students (i.e. the case of long papers). The assignments are
summarized in Table 2. Note that they are reported with a paperID. Full references
can be found in [4].
Before actually assigning the papers of the review, we decided to carry out a
guided exercise (Lesson 7 & Lesson 8). This was important for allowing students to
familiarize with the documentation they used in their final assignment.
In the guided exercise students were asked to extract data from either of two
primary sources [13, 21] (which we ourselves selected) according to the data
extraction form and aggregation tables defined in the SPC protocol. Students were
handed the following material: data extraction form, data aggregation tables, research
questions of the systematic review, one of the two primary studies. So, for the
exercise, half of the students were assigned to a paper [13] and half to another [21].
Students split into pairs and each pair carried out the following tasks: each pair
member extracted the data from the paper independently; the pair members compared
their data collection forms; any disagreements were solved or noted as disagreements
to discuss with the rest of the class and with the researchers. Then, all pairs that had
worked on the same paper joined together and completed the aggregation tables. Once
the tasks were completed, we handed students the correct complete data extraction
forms for the two papers. We had previously analyzed the papers and completed the
forms. Results were discussed in class with other groups that had worked on the same
paper, and with the researchers. Overall, feedback of the guided exercise was positive.
Students became familiar with the data extraction tables. Most of the data they
422 M.T. Baldassarre et al.

extracted was correct. Some students found it difficult to understand the meaning of
the cells in the tables. Further explanations of the data extraction forms were
provided. Also, we decided to translate the cell content in Italian as well, although
answers were to be reported in English. Due to language problems, two students
decided to not continue with the assignment.

Table 2. Paper Assignment to Students

Nr.Students
Paper ID Comments
assigned

ACM 1 5 --
ACM 2 3 --
CROSSTALK 1 4 --
CROSSTALK 2 4 --
CROSSTALK 3 5 --
CROSSTALK 4 4 --
EMEROTECA1 Two papers per
3
IEEE 5 person
IEEE 10 4 --
IEEE 14 Two papers per
3
SPRINGER 7 person
IEEE 4 Two papers per
3
IEEE 3 person
IEEE 6 4 --
RIF TESI 4 --
SCIENCE DIR 9 3 --
SCIENCE DIRECT 10 4 --
SPRINGER 2 4 --
SPRINGER 3 3 --
SPRINGER 6 4 --
SPRINGER 8 3 --
Students worked in
SPRINGER 1 10 couples due to the paper
length
Used for guided
IEEE_1 0
exercise
Used for guided
IEEE_2 0
exercise

Given the positive results of the presentation part and guided exercise, we
considered it feasible to continue with our schedule (Lesson 9 & Lesson 10). Students
were individually assigned to one of the 24 papers. We ensured that data extraction
from each paper was assigned to at least 3 students. Next, all students reviewing the
same paper met as checkers and confronted their work, in groups, to obtain a unique
version. At this stage a researcher was also assigned as checker, to guarantee that all
pairs (and therefore papers) were controlled by an expert; also, conflicts were solved
by an adjudicator. A PhD student was also recruited as adjudicator.
 A Hands-On Approach for Teaching Systematic Review 423

Given the focus of the paper, we will not discuss or illustrate the results of the
analyses. The reader can refer to [4] for these details.
As it can be seen, the approach adopted has been scheduled in order to allow
students to work individually and in groups, discuss and motivate all their decisions.
Also, it has developed their ability to systematically search for information focused on
the research questions to answer. In this sense, students read paper titles and abstracts
critically addressing the questions and the need for evidence. Their opinions on the
adopted approach were collected through a questionnaire. Details are reported in the
next section.

4 Qualitative Assessment
Once we finished the lessons and carried out the systematic review, students filled in
a questionnaire. The questionnaire is reported in the appendix of this paper. As it can
be seen, its mere objective was to perceive students opinions on this experience,
given it was the first time we included systematic review as course topic and also the
first time that we used a hands-on approach as the one described in the previous
section. It is clear that from the students answers we were able to carry out some type
of qualitative assessment, which cannot allow us yet to generalize the collected
information. In each case, we consider it an important experience.
The answers followed a general trend of positive impressions. In particular, 95% of
our students found the theoretic lessons and examples provided significant for
understanding the tasks carried out; 5% considered it significant although requested
further details. No one considered them useless.
As for the topic chosen (SPC), 98% of the class agreed that the lessons were useful
for understanding and interpreting the concepts of the analyzed papers. In some cases
the papers faced issues that had not been discussed or presented in class. This made
data extraction more difficult for the students having to review those papers. These
comments suggested as improvement (to keep in mind in future courses) the need for
us researchers to briefly read through the papers before assigning them to the class.
The data extraction forms were easier to understand as more examples were
illustrated. We present some of the most interesting comments:
- I found the forms easier to understand after a few examples;
- I understood the data extraction forms after we were given the assignment
to define them for our systematic review. The discussion in class with
classmates and professors also helped a lot;
- the data aggregation forms were a demanding task as it requested to
combine individual work of different students;
- knowing that I had to discuss and support my decisions in groups motivated
the individual task of data extraction
As so, students paid more attention to their individual work knowing they were
asked to discuss it with other classmates in groups and present the results to the entire
class and professors.
Our general impressions on the success of the approach were also confirmed in
question 6, i.e. in most cases the individuals confronting their work agreed on
424 M.T. Baldassarre et al.

everything. This points out that in their individual tasks they interpreted the information
request analogously. In few cases, discussions on minor aspects were necessary.
As it arises from the above opinions and the schedule described in
Table 1, the hands-on approach has given students the chance to apply theoretical
concepts through the assignments on a real systematic review and has enforced the
importance for searching evidence. A student commented: This experience taught
me that searching for evidence focused at answering a specific research question isnt
as easy as it seems.
Students impressions are that the approach is rigorous and allows to retrieve the
necessary information and only focus on the evidence from the research question
perspective, i.e. another research question may have classified the same papers as not
relevant to the search although always on SPC. Finally, many of them expressed their
interest in carrying out another systematic review but on another topic like software
product lines or a specific tool.

5 Conclusions

Software engineers as future practitioners are constantly asked to identify the most
appropriate technologies, methods and tools to adopt in projects. This paper has
focused on the importance of developing skills to allow software engineers to make
decisions and evaluations according to the empirical evidence they are able to retrieve
in literature. In order to do so, we have introduced a hands-on approach based on
systematic review as manner for rigorously searching for evidence, as part of the
EBSE process.
In this sense, we have proposed the approach within our university course on
Empirical SE Methods. As far as we know it is the first attempt to include
systematic review as part of a university course on empirical software engineering.
We have collected qualitative data on this experience by submitting a questionnaire to
students. The considerations presented in the previous section point out the positive
attitude of our students towards the assigned tasks.
Overall, we consider the approach as a useful means for making students perceive
the importance of searching for evidence in a rigorous and systematic manner such as
systematic review. Also, the approach we adopted allowed them to actively
participate to the review both individually and in groups confronting their opinions
and discussing with pairs.
As researchers we consider systematic review as an essential part of EBSE. Indeed
it is only after collecting evidence on a specific issue that decisions can be made. In
this sense, in our opinion systematic review represents such rigor. The qualitative data
that we have collected from the questionnaires submitted to our graduate course
students has in some way confirmed our opinion. Given the results, we have decided
to adopt the same strategy in our next course. As future work we are planning to
extend our experience and collect evidence on the efficacy and effectiveness of the
approach by assigning students to search for evidence with and without systematic
review as part of the EBSE process.
 A Hands-On Approach for Teaching Systematic Review 425

References
1. Australian National Health and Medical Research Council.: How to review the evidence:
systematic identification and review of the scientific literature (2000) ISBN 186-4960329
2. Australian National Health and Medical Research Council.: How to use the evidence:
assessment and application of scientific evidence (February 2000) ISBN 0642432952
3. Bagert, D.J., Jilburn, T.B., Jislop, G., Lutz, M, McCracken, M., Mengel, S.: Guidelines for
Software Engineering Education Version 1.0. Technical report, CMU/SEI CMU/SEI-99-
TR-032, (1999)
4. Baldassarre, M.T., Caivano, D., Visaggio, G.: Systematic Review of Statistical Process
Control: An Experience Report. In: 11th Evaluation and Assessment in Software
Engineering Conference, BCS UK, pp.94-102 (April 2007) ISBN:978-1-902505-86-2
5. Bloom, B.S.: Taxonomy of educational objectives: The classification of educational goals:
Handbook I, cognitive domain. Longmans Green, New York (1956)
6. Brereton, P., Kitchenham, B., Budgen, D., Turner, M., Khalil, M.: Employing Systematic
Literature Review: An Experience Report. Technical Report TR 05/01, School of
Computing & Mathematics, Keele University (2005)
7. Cochrane-Collaboration, Cochrane reviews handbook. Version 4.2.1 (2003)
8. Dyba, T., Kampenes, V.B., Sjoberg, D.: A systematic review of statistical power in
software engineering experiments. Information and Software Technology 48, 745755
(2006)
9. Florac, W.A., Carleton, A.D.: Measuring the Software Process: Statistical Process Control
for Software Process Improvement. Addison-Wesley, Reading (1999)
10. Glass, R., Vessey, I., Ramesh, V.: Research in software engineering: An analysis of the
literature. Information & Software Technology 44, 491506 (2002)
11. Glass, R., Vessey, I., Ramesh, V.: An Analysis of Research in Computing Disciplines.
Communications of the ACM 47, 8994 (2004)
12. Hart, C.: Doing a Literature Review: releasing the social science research imagination.
SAGE Publications, London (1998)
13. Jacob, A., Pillai, S.K.: Statistical Process Control to Improve Coding and Code Review.
IEEE Software 5055 (May/June 2003)
14. Jorgensen, M., Dyba, T., Kitchenham, B.: Teaching Evidence-Based Software Engineering
to University Students. In: 11th IEEE International Software Metrics Symposium. IEEE
Computer Society Press, Los Alamitos (2005)
15. Jorgensen, M., Shepperd, M.: A Systematic Review of Software Development Cost
Estimation Studies. IEEE Transactions on Software Engineering 33(1), 3353 (2007)
16. Kahan, K.S., ter Riet, G., Glanville, J., Sowden, A.J., Kleijnen, J.: Undertaking Systematic
Review of Research on Effectiveness. In: CRDs Guidance for those Carrying Out or
Commissioning Reviews. CRDs Report Number 4 (2nd edn.), NHS Centre for Reviews
and Dissemination, University of York. ISBN 1900640201 (March 2001)
17. Kitchenham, B.: Procedures for Performing Systematic Reviews. Technical Report
TR/SE0401, Keele University, and Technical Report 0400011T.1, National ICT Australia
(2004)
18. Kitchenham, B., Mendes, E., Travassos, G.: A systematic review of Cross vs. Within
company cost estimation studies. In: 10th International Conference on Evaluation and
Assessment in Software Engineering, Keele University Staffordshire, UK, April 2006,
vol. 3, pp. 7988 (2006) ISBN 1-902505-74-3
19. Meyer, B.: Software Engineering in the Academy. IEEE Computer 34(5), 2835 (2001)
426 M.T. Baldassarre et al.

20. Staples, M., Mahmood, N.: Experiences Using Systematic Review Guidelines. In: 10th
International Conference on Evaluation and Assessment in Software Engineering, Keele
University, pp.79-88, BCS UK (2006) ISBN 1-902505-74-3
21. Weller, E.: Practical Applications of Statistical Process Control. IEEE Software, 4855
(2000)

Appendix: Assessment Questionnaire

Paper_ID: _____________________

1. the theoretic lessons and the examples on systematic review were:

a. significant for understanding the tasks to carry out
b. of no use. The task we carried out consisted in data extraction that
could have been done even without the concepts and examples on
systematic review.
c. Significant but needed further investigation.
d. Other.(specify) __________________________________________
2. the training lessons on SPC were:
a. useful for understanding and interpreting the contents of the papers
b. pointless. The contents of the paper were different than those
presented in class
c. useful, but more details would have been better.
d. Other.(specify): ________________________________________
3. Comments on the Data Extraction Forms: ____________________________
4. Comments on the Data Aggregation Forms: __________________________
5. In the group work, how much was taken from your individual work:
a. 20%
b. 40%
c. 60%
d. more than 80%
comments:_____________________________________________________
6. In the group work (more than one answer possible)
a. We always agreed on everything
b. We had to discuss about ____% of times
c. In ____ cases we were unable to agree on a decision
comments:_____________________________________________________
7. would you repeat this experience? YES / NO
motivate your answer: ___________________________________________
8. your impressions on individual work: _______________________________
9. your impressions on group work: __________________________________