Research Methods in Computer Science

Research Methods In Computer Science
1 Introduction
This chapter introduces the subject of this book which is a research concept in computer science. Research activities have been around the academia for a long time as it is the very foundation of the concept in learning. The field of computer science, however, was introduced during the advent of computers at the early of the 20 th century. The concept of research in computer science has been introduced ever since and has evolved as the needs and the environments changed. This book tries to put research in a proper perspective within the computer science context alongside the other fields such as business and engineering. Generally, the research concept is similar despite the differences in the domain of knowledge. The aims, methodologies and analysis should focus on similar characteristics with the emphasis on that specific knowledge area. The students in computer science should produce the research products through such activities so as to gain common understanding from the public. The major aim of a research is to improve and contribute to the well being of the society and all research including from the field of computer science should meet the same target. The word research originally comes from a French word rechercher which means to search closely. It involves a thorough investigation of a subject to produce an insight and relate some phenomenon that explain the problems being studied. Computer science has become a vast field of knowledge which has branched out into numerous domain knowledge. Each of the branch requires different emphasis in research. The chapters in this book will explain some of these differences in detail so the reader can understand what to focus on. The field of software engineering , for example, has become so diverse that the field itself offers its own methodology and analysis. The increasing demand in internet programming has opened up new domain in programming and application development. The field of artificial intelligence has also evolved and produced new emergent technologies to solve problems. These changes demand for research to be
Nordin Abu Bakar 2011, 2012[revised]
done and prove that the proposed method is worthwhile and match the dynamic trends in computer science. Those are some of the reasons why research should be explained in a specific domain knowledge such as computer science. It will help the young researcher to focus their work in the early stage of the project and avoid the mistakes of producing useless and irrelevant results.
1.1 A Research Concept

Researchers, in the field of computer science and engineering, may view the research process in a way depicted by Figure 1. There is an experimenter in a middle of the research field trying to use the experiment to find the optimum result . An experimenter a person who considers to be working towards finding some explanations regarding a specific phenomenon that is happening around him/her. A research field a specific domain knowledge related to the subject being worked on. This field defines the components or parameters related to the study. It provides references for the work that might have been done before. An experimenter must be aware of the contents and boundary of this research field. The research field is the area that a researcher is interested in, it is visible to the researcher and pretty much well aware of what is going on in the area. The experiment the process that a researcher will follow to answer the questions that will take us to a solution. The experiment will be a tool to investigate the problem. This tool contains the methods proposed to solve the problem \ The whole process is driven by a response surface which emerges from the two variables used. A response surface- defines a landscape produced by the data. It is a representation of the problem that might be visible to the researcher at all. The experiment would become a yardstick to drive the researcher to the goal. Variables These are the elements that directly influenced the nature of the landscape. They add dimensions to the response surface
The white flag represents a goal that could be anywhere beneath the surface. It is the answer that researcher is looking for to describe the problem. The quality of the solution increases as it approaches the goal.
Figure 1 : A Research Concept (adapted from [1]
1.2 Why do research ?

Getting into research for the wrong reason is the most common mistake that someone can do. Even if you have registered as a research student ,you need to refine the real intention of
doing research. The goals need to be well understood and revise from time to time so that it will be relevant to you and the people around you. So, what are the reasons to get into research ? 1) So that I will have time to do some other things. This is definitely very wrong. If one intends to do research "part-time" or "whenever I have a free time", then you are planning for trouble. Your commitment to pursue research activities has to be a full-time effort. Although a researcher is free from rigid schedule ;it does not mean that nothing is expected out of him. The freedom to set own timetable is to allow a researcher to "grow" on his own and produce a good, unbiased and undisputable results. 2) Because I have to, this is part of the course. The idea of having to conform with somebody else's needs and expectations without your own interest is the first thing to avoid. A long and hard demand from research environment will not last if you have no personal determination. There might be a promotion involved or a new career move; whatever it is, set your goal and go for it. 3) Looks good in my CV. It really prevails because research achievements represent a high degree of excellence. The ability to find new things or explain new dimensions would be attractive for any field of work; from academic to corporate management. 4) I can keep my present job. The fact that you keen to keep your job gives an indication that doing research or going for a research degree is not your priority. It is a mistake to pursue with this on your mind. A choice has to be made and at the end you might compromise between your job and your intention to do research. Things can be worked out but you have to open the options for any circumstances. 5) I have all the data ready. This statement sounds right because there is a word "data" in it which is directly connected to research. However, having
data without specific research objectives would be meaningless. The data has to be collected according to what we want to do in the research (research objectives) and the answer that we are seeking (research questions). The data might be from your previous research of similar theme or from a database related to your topic. These are considered secondary data and the use of secondary data is very limited in a scientific research project(will be discussed further in the later chapter). The other reasons to get into a research project are : 1) I am interested in the area. 2) I have done this before and now I would like to explore further. 3) I have come to know that the subject is so challenging and practical with huge potential to benefit the community at large. 4) My job is related to this area of research. 5) My supervisor is an expert in this. All of these reasons reflect the intentions and the amount of effort that one will be willing to put forward to pursue research activities.. The intention or aim for doing research has to be reviewed from time to time to avoid it from dying out.
1.3
What makes a good research ?
The main goal of any research is "to contribute something new to the current body of knowledge". Selecting a right research problem is the key to a good research project. With collaborative efforts from your supervisor and support from the faculty, the research will be equipped with momentum and take you to the right direction. It is difficult to pin point the criteria for good research because it depends on the type and objective of the research. One research may be good for one group of people but useless for the other. But to make things a lot easier, there is a list of criteria below that can be used to reflect a good research.
1) Relevant to community at large. The problem that you choose to explore must be important and relevant to a larger community . 2) Inspires You. It is important for the topic to the one that motivates you to address it with intensed passion to authentically engage in the goal of reasoned decision making. 3) Challenging . It challenges you to question your own assumptions about teaching, learning, literacy, and change; i.e., it challenges you to learn. 4) Support multiple perspectives. The problem most be phrased in a way that avoids dichotomies and instead supports the generation and exploration of multiple perspectives. A general rule of thumb is that a good problem is one that would generate a variety of viewpoints from a wide range of audience. Everyone can relate to your topic and contribute their ideas that might shape the analysis and findings of your research. 5) Researchable. Topics that fall into this criteria are the ones that you can deal with one way or another. The data can be collected as scheduled, the equipments are available to you either within the department or somewhere that you can easily go whenever you need them and the topic is proved by previous researchers to be researchable. 6) Doable. It can be completed within your time and material constraints. 7) Not Too General, Yet Not Too Narrow. It is not too general; that would result in a multitude of sub-questions; yet, not too narrow; that would rule out the emergence of other possibilities.
1.4 Research and Development (R & D)

The term R & D is very familiar nowadays that any organisation will struggle to keep their R & D unit active for reasons that they may not fully understand. The economics of R & D is an important factor that support research activities; but we are not discussing that issue in this book. Nevertheless, one might question why do we use these terms together and not without one or another. Which one is better to go into: research or development ? Let's find the relation between these two terms and find out where do we stand the best.
In research, as we mentioned in the earlier part of this chapter, the most important element is to find some thing new. It could be new theory, new solution, new application, new dimension or new description that has been produced at the end of the research period. later in the following chapter there are all sorts of explanation and conceptual views that emphasize the signficant of "new thing" in research. On the other hand, if we develop a system, what would be the most important factor to consider? The answer would probably be" a good system". A good system is defined as a system that meets the user's requirement. It does not necessarily use the most advanced theoretical framework but as long as it works and the clients are happy then we have a good system. So basically, the difference between research and development is the difference between good and new. On one hand you have a good system and on the other hand you have a new method. They are two different perspectives looking at the same thing. Development doesn't have to be new, but it has to be good. Research doesn't have to be good, but it has to be new. These two school of thoughts converge at the peak: the best system will beat the others by using new ideas, and the best research solves problems that are not only new, but actually practical in real life. So ultimately we're aiming for the same destination, just approaching it from different directions.
Research
Development
New findings
Good system
Practical?
Is this the best system?
Figure 2 : Research vs. Development
1.5 Establishing the Idea

Ideas come and go. One minute you might have the whole picture in your mind and the next minute it's gone. That's why it is important to keep a small notepad with you all the time to scribble anything that might come to mind. Ideas could come from various sources, such as Research activities Readings Observations Experience Discussions/Colloquium
Research Activities
Readings
Observatio n Experience
Discussion
1.5.1
Research activities
How do research activities trigger an idea? This is particularly happening in your second research project. The continuation from the first one that you have just finished. Usually a completion of one research will open up several ideas for future research. If you are just starting up and this is your first research then look up this section in the research report or article that you may come across : recommendation for future research. 1.5.2 Readings
Doing a lot of reading can also help you come up with a few questions that will lead to a research topic. Things to read include report or articles related to your interests. Reading an article can sometime be so difficult to understand. It is nor mal to feel that way because the content of the article could be full with technical jargons and mathematical formula. One will be easily put off after a few paragraphs. In understanding an article or more appropriately called technical paper, one has to go through several phases of reading : 1) Surface reading. Read through the article but leave the terms or parts that you don't understand. Go through it several times and see whether you can figure out what the article is trying to tell the reader. If at this phase you have understood the article then it is great; it could be you who have the knowledge or the article is well-written. Either way you are off to a good start because usually that is not the case. 2) Technical reading. Read through the article and stop at the point where you have started to loose the picture. Try to go back a few lines and pin-point the words, phrases or theories that you don't understand. Flip through a dictionary if you are not sure the meaning of a certain word or phrases. Refer to a book to get further explanation of the theory stated in that article. Get onto the internet for some other explanation or description of similar ideas. Somehow you have to get the thing cleared out before proceeding further into the article. This process could last an hour, a day or even a week; but at the end of it you will be satisfied and amazed of the new knowledge that you have just discovered.
3) Guided reading. This is like a bed-time story reading where you sit with your supervisor/lecturer and go through the article paragraph by paragraph. He will stop and explain at places where the narration is too simplistic. Sometime we are lack of background to fully understand one specific paragraph and a little help from someone knowledgeable is required. The phases described above do not have to be followed step by step; but a guideline to follow if one way of reading does not help much. Reading is very important in doing research. The habit to read whatever comes handy is essential as ideas for the next move in research might be available. Lack of reading will result in poor understanding of the current needs and trends in our area of research and minimize our ability to argue and defend our ideas. A problem may arise ,when we have read a lot of books ,articles and journals, that we might loose track of our readings. A few tips to manage your readings are as follows:
1) Try to keep a copy of the article that you are reading, have read, or planning to read. Organize them properly so that you can get them again later. 2) If part of a book is important to you, get a copy of this part only and write down the identification of the book somewhere at the back for your future reference. 3) Keep index cards of the articles that you have read and write down a short summary of the article on each card. This will help you recall which article the card is referring to. Information that need to be put down on the card are : the title, call number, authors, publishers, and edition . 4) Take notes of the things that you have found from your reading practice and keep them on cards for your later use. Differentiate between a summary, quotation, paraphrase and your personal comment. This will avoid unnecessary plagiarism in your writing stage. 1.5.3 Observations
A good idea may pop up from your own observations of the situation. This could happen at a local fair or workshops. A good documentary program on TV could also gives you some ideas on what people have done on the subject. A parking problem at your faculty could open a whole series of research for someone that might be interested to solve it according to his area of expertise. Keep your eyes open as the ideas could be every where. Those are
10
informal observations that could lead to a research question that may start up a full blown research project. If one determined that there is an issue in an organisation that could be used to start a research, a formal observation can be arranged to get the whole picture. This formal observation will be later on known as "qualitative research" is one type of research that observation is the only way to crack a tough research question. However, one can also use this as a preliminary step to establish a need for a subsequent research (or a quantitative research). When we observe a situation, what we actually do is looking at the other end of the spectrum; such as a broken marriage, a ruined child, an empty classroom, a chaotic counter service, a delayed flight, a bankrupt young man, a faulty system, etc. What we can observe is usually the product of bad design or poor foundation and those are the most difficult aspect to be analysed. Since this is the other end, we have to trace backward until we find the root of the problem. This then will be our motive for a research.
1.5.4
Experience
Experience refers to the knowledge that one might have to be used as a research project. This could be related to your line of work where the problems could occur over and over again. The area of management, for example, could use some new techniques all the time as people and the surroundings are rapidly changing. Being a manager or at senior managerial position would trigger some ideas on how to improve the efficiency of the management. There are tools and methods that could be tested that might produce tremendous insight into the new possibilities. It will need a good research project to materialize the concept.
1.5.4.1 How to use your experience in research ?

The experience can be used to determine the direction of your research. The many years on the job gives you an insight of the operational aspect of your line of work. Pick a research project that is in-line with your work and you have unlimited source of information that can be used in your research. The experience will enable you to state the process of your work from the beginning to end and this could be the practical side of your research. If one is a bank manager, then anything related to banking could the area of research. Refine this perspective and you will narrow down the scope of your research. As a bank manager you might expose to Islamic banking practice, for example, and you know this is the future trend in banking, so follow through your instinct and you will end up with a topic. The experience also comes with a network within the banking sector and this will give an
11
advantage during the other phases of research such as data collection, analysis, and professional opinion on the findings. Nonetheless, people with working experience are lack of academic skills such as writing, facilitating argument, critical analysis, etc. These and many more other skills are essential to bring the research forward to a meaning conclusion. However, these skills can be learned and acquired throughout the research period.
1.5.5
Discussions/Colloquium
Attending discussions/colloquium could always bring you some good ideas. Basically one has to participate in the process; either being a presenter or active participants. This kind of activity is held in the faculty and open for all to attend.
1.6 Finding A Research Area

It is a common mistake to go into research without a clear understanding of your subject area. This situation worsens when one changes the topic halfway through the research period for some reasons. If time is a great constraint then you are in trouble because you will not have enough time to define the topic in greater details. The decision to pick a specific area for research maybe based on your own personal enthusiasm, the availability of the supervisory support and it suits your strengths or capabilities. Generally, some "easy" subject areas are always hard to work with; on the other hand, "difficult" subject area is quite easy to deal with . Examples of "easy" subject areas are topics to solve world hunger, eradicate poverty, building computer that can think like human, and the like. Those are good topics but maybe not a good PhD or master's topic for it requires a longer time to find a hard solution. The "difficult" subject areas are very specific and focus, and they are hard to find, but you'll find ample time to test your model. Therefore, a good amount of time should be spent to find the appropriate research area.
12
How to Get Research Ideas from Other Studies
Strategy Replicate the study 1 . Directly replicate the study 2. Replicate an old study in a new time 3. Replicate a study in a new place 4. Replicate a study with new subjects Alter a study 5. Eliminate glitches in the procedures 6. Add control variables 7. Intensify or reduce the treatment
Comments
Use simple duplication to see whether it really works Increase external validity by controlling history Increase external validity by controlling setting Increase external validity by controlling subjects
Control threats to internal validity Control threats to internal validity Treatment may have been ineffective because it was too weak; or it may be useful to know whether less would still work Weak data collection may have missed real effects Multiple operational definitions increases external validity Control external validity by identifying interactions; also enhance internal validity For example, change from experimental to quasiexperimental or qualitative Rule out chance variation
8. Use a more sensitive data collection process 9. Use new operational definitions of one or more research variables 10. Add one or more new moderator variables 11. Use a different research design 12. Use more Subjects
13. Change the timing of data collection For example, use delayed posttest 14. Add levels to the treatment or other variables 15. Alter what the control group does For example, instead of treatment versus no treatment, add intermediate level of slight treatment or other treatment For example, instead of doing nothing, do an alternative activity
13
16. Perform a meta-analysis Where to get research ideas 17. Follow author's suggestions for "further research" 18. Combine information from several studies 19. Use analogical reasoning 20. Relate the study to your own experience 21. Contact the authors of interesting studies
Combine results of several studies to generalize results carefully
These usually come at the end of the article (sometimes they are implied rather than stated) For example, by combining ideas from two different studies, you can have an original study Look for analogies between the study and other knowledge of your own For example, perform an applied variation of a theoretical study For example, write directly or use computer bulletin boards
Table 1 : Some examples on how to get ideas
1.7 Schedule
Keeping the research activities up-to-date is important to make sure that the project can finish on time. This is about time management and anything that deviate you from their main schedule should be seen as a threat . Threats must be dealt with at all costs. A Gantt chart is probably the most familiar tool being used for this purpose. The problem that still exist is the chart remains as a chart without a real sense of purpose. An active schedule is constantly reviewed changed whenever there is a need strictly followed with minimum compromise reflect the real process practical
A general guide line to schedule a research project is to divide the time into three periods where each period may take between six months to a year. The following table illustrates the idea :
14
Table 2 : An Example of Time Slice in Research Periods 1.8 Preparing a Research Proposal
A research proposal is written not only for official purpose of getting the grant but most importantly it's our ideas properly documented on a few sheets of paper. Being able to pen down the whole thing can be considered a huge success towards more glory days ahead. It provides a blueprint by which the research can be directed; and it provides the means by which the study can later be evaluated. There are three basic purposes that a proposal may serve. They are : To understand a deliberate process that the research project will undertake. To present the ideas in a more organized manner so that people can appreciate our intentions to start a study. This is mainly to gain moral as well as monetary support. To facilitate the whole organization of the research project.
1.8.1
Understanding the Research
The proposal acts as an outline which identifies the important aspects of the study. In essence, it forces the researcher to address the major components of a study and describe how they will be carried out. The framework of the study should answer the following questions : Why is the research being proposed ? What are the problems being looked into? How will it be conducted?
15
Who will involve? When will the study be conducted? These questions provide the framework of the study and consequently a detailed plan of a research project. Their answers are reflected in the proposal.
1.8.2
Contents of a Research Proposal
The contents of a research proposal are generally focused on the needs of its intended reader. If the proposal is to seek funds, then the focus would be on the economics and costing of the project. If the proposal is for the masters or PhD thesis then the academic values would be the primary agenda. One has to refine the contents of the proposal to cater for the intended audience. As a general guideline, a list of components below are recommended to be included in the research proposal : 1) Cover Page : It contains the title of the research, the researchers' name, organization and date. This page introduces the research in brief and properly document those involved. 2) Abstract : A summary of the whole research. It should not be longer than a page and contains the essential information of the research. basically an abstract describes the importance of this research, how it is going to be done, what will be the outcome(expected results) and how this give an impact to the current situation. 3) Problem Statement : This part will explain the background of this research. Any research is triggered by a recurring problem and by describing the problem, the researcher has successfully pinpoint the subject of his research. 4) Purpose : The purpose of this research is written the form that formally describe the problem at hand. This is done by presenting hypotheses (the thing that we believe could describe the situation and proved by some quantitative methods that those beliefs are true), research questions ( questions that will lead us to the solution of the problem and support our hypotheses) or research objectives ( some specific aims that we set to achieve by doing this research). 5) Research Methodology : This includes the instruments used, population, sampling techniques, data collection techniques, statistical tests and data analysis used for this research.
16
6) Pilot Study Report : Doing a pilot study is something extra and not necessarily essential. But anyone who would sponsor this research will be convinced if a pilot study is done and reported. The report should be brief and mainly focused on the feasibility of the study. 7) Expected Outcome/product of the research : The proposal should state in detail the kind of outcome that can be expected from this research. The research could produce either a new theorem, a new model, formula, report, evaluation etc. 8) Schedule : The schedule presented in this proposal should be realistic and practical for the type of research proposed. 9) Finance : the budget for this research is quite important to be presented in detail to make the costing just right. 10) References : The name of the articles or books together with their authors should be listed here for references. 11) Researchers' CV : The CV for all researchers involved in the research is also important to convince the prospective sponsor that you are capable to carry out this research.
1.9 Research in Computer Science

Computer Science (CS) has been around for quite some time now. The area has been branched out into so many sub-areas that the original field of computer science has somehow become vague. However the most intriguing fact is that although CS is an established area of study it has not been properly researched into using the proper research methodologies , produced justified results and reported experimentally validated observations. This article looks into this claim in greater details and discusses some methods as a guideline for doing research in computer science. Some literature surveys into this subject show that CS researchers have faced difficulties to establish a strong footing in the area of experimental CS. The crisis has been highlighted in [12,13] and the authors conclude that experimental CS is still unexplored. Denning in [3] gives definition of experimental CS as measuring an apparatus in order to test a hypothesis. He describes in other articles [4,5] that the fields of performance evaluation as a positive example of experimental CS research. Fenton et al. [7] observe software engineering
17
research and states that there are too few examples of moderately effective research in this area. Iyer[10] aggress that experimental CS is still relatively new yet fast developing area. Hooker[8] notes that experimental research is dramatically under-developed in algorithm research states that CS researchers should not publish any report that an algorithm does not perform well in computational tests. In view to those critical observations, Lukowicz et al [13] carried out a survey to explain the scenario of an experimental evaluation in CS. Lukowicz et al [13] surveyed some 403 CS related articles form several journals such as ; Optical Engineering ( OE), Neural Computation (NC), ACM Transactions On Computer Systems(TOCS), ACM Transactions on Programming Languages and Systems (TOPLAS), SIGPLAN Conference on Programming Language Design and Implementation( PLDI), ACM Transactions on Computer Systems (TOCS), IEEE Transactions on Software Engineering ( TSE) and some Random CS samples. The articles were all published in 1993.
TSE TOPLAS PLDI Random TOCS OE NC 0% 20% 40% 60% 80% 100% Theory Design Emp.+ Hyp. Other
Figure 3 : The type of articles in the surveyed journals[Lukowicz et al [13]]
Lukowicz observed that that hypothesis testing is extremely rare in all samples (4 articles out of a total of 403). There is a disproportional high percentage of design & modeling articles without any experimental evaluation in the Random samples as compared to NC and OE (43% vs 14%). In NC and OE there are significantly more design and modeling articles devoting over 20% of their space to experimental evaluation than in the Random
18
samples. The articles related to software engineering has the highest percentage for having 0% space devoted to experimental evaluation.
Design & Modeling Articles 80 70 60 50 40 30 20 10 0

S an do m C PL DI TO PL AS O E TO C TS E N
(%)
>20% 0%
Journals
Figure 4: Percentage of articles in design & modeling devoted space for experimental evaluation ( as in Lukowicz et al [9] )
The definition of research is always the same no matter what area that one is working on. The quest to explore new knowledge is constantly pursued to find the answers for so many questions from the study. The main idea to understand what research is all about is to use proper mechanism and tools so the findings can be accepted and used by the community at large. The benefit of the study should be appreciated within the knowledge area as well as outside; applicable and relevant to the out side community. Although computer science is a very technical subject, its application and usability must be directly concerned with everybody at large.
19
Research is defined as the systematic investigation into and study of materials, sources, etc, in order to establish facts and reach new conclusions. An endeavour to discover new or collate old facts by the scientific study of a subject or by a course of critical investigation.
There are a few approaches to tackle research in general. A researcher can establish a thesis, produce an anti-thesis or making a synthesis. The word thesis is referred to as one's original claim or statement. That's it, not that complicated isn't it. If you said the world is round, then, you've made a claim or you have made a thesis. If you wanted it to be accepted by others then you have to "defend your thesis" or claim. Then we have a word such as " anti thesis"----a statement that goes against a thesis. If some one made a thesis and you claimed otherwise then you've made an "anti thesis". Another word which is much more familiar to us all is "synthesis", it simply means if we take the existing thesis and make it stronger with new data , new proof or new perspectives then we are making a synthesis.
The subject of research in CS varies from one area to another within the discipline of CS itself. The table below summarizes the area of focus for CS.
No 1 2 3 4 5 6
CS subject algorithms Artificial intelligence Programming languages architecture theory systems
Area of focus algorithms Methods, techniques, algorithms Languages, language components or features Instruction sets, memory hierarchies, architectures Models, theorems, proof techniques Systems, components of systems, system architectures
20
It is interesting if we can narrow down which of the area of focus should a researcher goes for. Although it depends on many other factors, the one that we feel within our grab should be the utmost priority. The area of focus (AoF) varies by their size, potential variety and cost of evaluation. The size refers to the scope of study a researcher has to cover in order to come up with a practical and useful observations. (1)Small size : algorithms, language features, architectural components (2) Medium-size size : instruction sets, proof techniques (3) Large size : languages, architectures, machines The potential variety of area of focus gives an indication the volume of similar structures is low or high. If the potential variety is low then there is a good possibility that a lot of research have covered many of the AoF and in order to pursue the researcher needs to find new potential. If the potential variety is high one can rest assure that there are a lot more areas of focus that might not be covered. This is important to a researcher so as not to repeat the same study and claim that it is a new discovery. The examples are as below : (1) Low variety : sorting algorithms, cache designs (2) High variety : AI algorithms, languages, architectures The other factor that characterizes the area of focus is the cost of evaluation. It refers to the level of effort that needs to be realized to establish meaningful end results. Some examples of this factor are as follows : (1) Low cost: algorithms, AI methods (2) Medium cost: theorems, components architecture (3) High cost : languages, architectures, ideal models By having to know such features, one can foresee the kind of research to be pursued. Generally, smaller area of focus exhibits less variety and lower cost of evaluation, therefore they are easier to do research on. Once the type of research has been identified, a researcher needs to come up with suitable research objectives. What are the expected results? A simple but very important question that needs to be explored.
21
Some examples are : 1. A definition of a problem or task; 2. An application for solving a problem; 3. Performing task; 4. Identification of factors influencing the cost; 5. Effectiveness or applicability of a system.; 6. Development of an ideal model; 7. A complete software that can be deployed to users; 8. Measurement of some properties of an application: run time, chip area, representation requirements, reliability, usability ; 9. Identification of problems and shortcomings in an application; 10. A demonstration that one application is better that another
22
1.10 How do you know that your measurement is good ?

There are a few ways to justify that your measurements are good. They are : 1. Validity : The extent to which an instrument measures what it intended to measure. The extent to which differences found with a measuring instrument reflect true differences among those being measured 2. Reliability : Instrument should provide consistent results. Contributes to validity (Note that reliable need not be a valid instrument, but a valid instrument is always reliable). Takes care of non interference of transient and situational factors 3. Practicality :The practical use of the end product. This can be measured through the users acceptance sure. 4. Easy to measure 5. Applicable early in the design process 6. Convincing In CS, some evaluation criteria are as follows: CPU time Cycles per instruction Percentage for correct classification Number of serious interface flaws identified
23
1.11 Research Instruments in Computer Science (CS)

An instrument is a tool to facilitate the process to get us to the output. There are many research instruments available out there; but for CS researchers the following instruments are the most important ones to get familiar with. 1. Programs: A program to a computer scientist is like a novel to a writer. It is a platform to try ideas and address thoughts. This is by far the most reliable tool to test and evaluate performance of an application. Programming skills, therefore, is a must-have ability of a CS researcher, whether one likes it or not. 2. Formal language : The language such as Z is a mathematical scripting language used to prove the hypothesis mathematically. A tradition of axioms and lemma can be effective in avoiding the flaws later in the developmental stages. 3. Systems : Evaluate a system in its context of use 4. Questionnaires : Measure attributes of an application
So the question whether there is research in CS or not; the answer is definitely yes. Even though a large number of CS researches fail to produce the justification for their results; that does not mean there is no requirement for doing so. It is time to come out into the open and argue the worthiness of the systems that we have spent so many hours into designing, building and developing it; when most people hide behind the complex architecture of the system. The methods are there and we just need the courage to use it for our own advantages.
24
1.12 Exercise
a) On a piece of paper, write down your own reason for doing research. b) Write down a research topic of your choice. Is it feasible? Let someone read it and give comments. c) What is the problem that you want to solve with your research? d) Give five other research topics that are similar to yours. In what way are they similar? e) List three areas of computer science that will be relevant to your research? f) What is the area of focus (AoF) for your research? g) Can you identify the potential variety and cost of evaluation for your topic? List them. h) Do you measure something? What kind of measurements will be used in your research? i) j) State the possible output of your research. How can people benefit from your research findings?
25
References
[1] [2] Bailey, D. H., Twelve Ways to Fool the Masses When Giving Performance Results on Parallel Computers, Supercomp. Rec. 54-55 (1991); Supercomputer 4-7 (1991). Baldwin, D., and Koomen, J., Using Scientific Experiments in Early Computer Science Laboratories, ACM SIGCSE Bull. 24, 102-106 (1992).
[3] Denning, P. J., What Is Experimental Computer Science? Commun. ACM 23,543544 (1980). [4] [5] [6] Denning, P. J., Performance Analysis: Experimental Computer Science at Its Best, Commun. ACM 24, 725-727 (1981a). Denning, P. J., Performance Evaluation: Experimental Computer Science at Its Best, ACM Perform. Eval. Rev(. SIGMETRICS) 10, 106-109 (1981b). Feldman, J. A, and Sutherland, W. R., Rejuvenating Experimental Computer Science: A Report to the National Science Foundation and Others, Commun. ACM 22, 491-502. Fenton, N., Pfleeger, S. L., and Glass, R. L., Science and Substance: A Challenge to Software Engineers, IEEE Software 11, 86-95 (1994). Hooker, J. N., Needed: An Empirical Science of Algorithms, Operat. Res. 42, 201212 (1994). STN International, INSPEC: Information Service for Physics and Engineering Communities, IEE, Herts, UK,1994. Iyer, R. K., Experimental Computer Science, IEEE Trans. Software Eng, 16, 109-
[7] [8]
[9] 110.
[10] Langley, P., Machine Learning as an Experimental Science, Mach. Learn. 3, 5-8 (1988). [11] McCracken, D. D., Denning, P. J., and Brandin, D. H., An ACM Executive Committee Position on the Crisis in Experimental Computer Science, Commun. ACM 22, 503-504 (1979). Computer Science and Telecommunications Board, Academic Careers for Experimental Computer Scientists and Engineers, Commun. ACM 37, 87-90 (1994). Lukowicz, P, Heinz,E.A.,Prechelt,L.,Tichy,W.F., Wxperimental Evaluation in Computer Science : A Quantitaive Study, Journal of Systems and Software, January, 1995.
[12]
[13]
26

Research Methods in Computer Science

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Research Methods in Computer Science

Uploaded by

Copyright:

Available Formats

Research Methods In Computer Science

Nordin Abu Bakar 2011, 2012[revised]