In software engineering, the structure of research theses, most notably dissertations, is straightforward: (1) Formulate a research question, choose a method, build a theory, then (2) generate at least one interesting hypothesis, choose a method, and test the hypothesis as part of the theory’s attempted validation. A dissertation can do both parts 1 and 2 or just one of them, relying on or leaving stuff to others. The benefit of this structure is that it will be easily recognized by other researchers and make it eas(ier) to write great papers.Continue reading “How Not to Ask Your Research Question (And What to do About it)”
On a whim, I asked my Twitterverse (which includes a fair number of computer scientists) what they think about the following question:
Continue reading “Anecdotal Evidence on the Method Wars”
When peer-reviewing somebody else’s work submitted for publication, what should you do if you find that the authors have a different belief than you about what can be known?
Most people believe that scientists first perform basic (“fundamental”) research and then perform applied research. Basic research delivers the fundamental insights that then get detailed and refined as they hit reality in applied research. Along with this comes the request that basic research funding should be provided by the country (because few companies would ever pay for it) before industry kicks in and supports applied research. Nothing could be further from the situation in my engineering process research.Continue reading “Inverted Research Funding”
When planning a publication strategy for a dissertation, invariably the question comes up where to submit your papers. Ph.D. students naturally are biased towards conferences, because if a paper gets accepted to a conference they get to travel to a (usually) nice place. I nip this bias in the bud right away: For a journal paper, every Ph.D. student gets a conference to attend for free. This lets us focus then on the economic value of a journal vs. a conference paper and how to best reap the benefits of hard research work. Here, I’m a contrarian (to most colleagues): I’m in favor of journals. It is also the economically smart choice for a Ph.D. student.
Research should be presented with appropriate choice of words to the world. So it bugs me if researchers, maybe unknowingly, overreach and call the evaluation of a theory a validation thereof. I don’t think you can ever fully validate a theory, you can only validate individual hypotheses.
The following figure shows how I think key terms should be used.
Last weekend, I ventured into unchartered territory (for me) and attended the Berliner Methodentreffen, a research conference mostly frequented by social scientists. I participated in a workshop on mixed methods, where the presenter discussed different models of mixing methods with each other (“Methodenpluralität” in German).
She omitted one model that I thought is often used: First to perform exploratory data analysis to detect some interesting phenomenon and then to do some explanatory qualitative research to formulate hypotheses as to why this phenomenon is, was, or keeps happening.
During my question, the temperature in the room dropped noticeably and I was informed that such exploratory data analysis is unscientific and frowned upon. Confused I inquired some more why this is so, but did not get a clear answer.
From my excursion into qualitative research land (the aforementioned Berliner Methodentreffen) I took away some rather confusing impressions about the variety of what people consider science. I’m well aware of different philosophies of science (from positivism to radical constructivism) and their impact on research methodology (from controlled experiments to action research, ethnographies, etc.) I did not expect, however, for people to be so divided about fundamental assumptions about what constitutes good science.
One of the initial surprises for me was to learn that it is acceptable for a dissertation to apply only one method and for that method to only deliver descriptive results (and thereby not really make a contribution to theory). In computer science, it is difficult to publish solely theory development research (let alone purely descriptive results) without any theory validation attempt, even if only selective. The limits of what can be done in 3-5 Ph.D. student years are clear, but this shouldn’t lead anyone to lower expectations.
In our research, we often work with industry. In software engineering research, this is a no-brainer: Industry is, where there the research data is. That’s why we go there. For many research questions, we cannot create adequately, in a laboratory setting, a situation that lets us do our research.
Once a researcher realizes this, they need to decide on whether to charge the industry partner for the collaboration. Many researchers don’t, because sales is not exactly their strength. Also, many shy away from asking for money, because it is an additional hurdle to overcome, once an interested industry partner has been found.
I often discuss with my Ph.D. students how to structure their work and publish the results. There are many pitfalls. It gets more difficult, if we bring in other professors, who may have a different opinion on how to structure the work. Over time, I have found there are really only two main dimensions, though:
- Separate work into theory building and theory validation, and publish one or more papers on each topic
- Merge work on theory building and validation for an important aspect (hypothesis) into one and publish that
A researcher-friend recently complained to me that her research paper had been rejected, because the reviewers considered it “boring”. I recommended that she complain to the editor-in-chief of the journal, because in my book, “boring” is no acceptable reason to reject a paper. (“Trivial” may be, but that is a different issue.)
The reasoning behind rejecting a paper, because it is “boring”, is as follows: Research should be novel and provide new and perhaps even unintuitive insights. Results that are not surprising (in the mind of the reviewer, at least) are not novel and therefore not worthy of publication.