Category: 3.2 Research Methods

Abstract: Pattern discovery, the process of discovering previously unrecognized patterns, is often performed as an ad-hoc process with little resulting certainty in the quality of the proposed patterns. Pattern validation, the process of validating the accuracy of proposed patterns, remains dominated by the simple heuristic of “the rule of three”. This article shows how to use established scientific research methods for the purpose of pattern discovery and validation. We present a specific approach, called the handbook method, that uses the qualitative survey, action research, and case study research for pattern discovery and evaluation, and we discuss the underlying principle of using scientific methods in general. We evaluate the handbook method using three exploratory studies and demonstrate its usefulness.

Continue reading “Pattern Discovery and Validation Using Scientific Research Methods [TPLoP Journal]”

Abstract: Pattern discovery, the process of discovering previously unrecognized patterns, is usually performed as an ad-hoc process with little resulting certainty in the quality of the proposed patterns. Pattern validation, the process of validating the accuracy of proposed patterns, has rarely gone beyond the simple heuristic of “the rule of three”. This article shows how to use established scientific research methods for the purpose of pattern discovery and validation. The result is an approach to pattern discovery and validation that can provide the same certainty that traditional scientific research methods can provide for the theories they are used to validate. This article describes our approach and explores its usefulness for pattern discovery and evaluation in a series of studies.

Keywords: Patterns, pattern discovery, pattern validation, theory codification, theory building and evaluation, research design

Reference: Riehle, D., Harutyunyan, N., & Barcomb, A. (2020). Pattern Discovery and Validation Using Scientific Research Methods. Friedrich-Alexander-Universität Erlangen-Nürnberg, Dept. of Computer Science, Technical Reports, CS-2020-01, February 2020.

The article is available as a PDF file and on FAU’s OPUS server.

Abstract: Interview analysis is a technique employed in qualitative research. Researchers annotate (code) interview transcriptions, often with the help of Computer-Assisted Qualitative Data Analysis Software (CAQDAS). The tools available today largely replicate the manual process of annotation. In this article, we demonstrate how to use natural language processing (NLP) to increase the reproducibility and traceability of the process of applying codes to text data. We integrated an existing commercial machine–learning (ML) based concept extraction service into an NLP pipeline independent of domain specific rules. We applied our prototype in three qualitative studies to evaluate its capabilities of supporting researchers by providing recommendations consistent with their initial work. Unlike rule based approaches, our process can be applied to interviews from any domain, without additional burden to the researcher for creating a new ruleset. Our work using three example data sets shows that this approach shows promise for a real–life application, but further research is needed.

Reference: Kaufmann, A., Barcomb, A., & Riehle, D. (2020). Supporting Interview Analysis with Autocoding. In Proceedings of the 53rd Hawaii International Conference on System Sciences (HICSS 2020), pp. 752-761.

Download: The paper is available a PDF file.

Abstract: In qualitative research, results often emerge through an analysis process called coding. A common measure of validity of theories built through qualitative research is the agreement between different people coding the same materials. High intercoder agreement indicates that the findings are derived from the data as opposed to being relative results based on the original researcher’s bias. However, measuring such intercoder agreement incurs the high cost of having additional researchers perform seemingly redundant work. In this paper we present first results on a novel method of using students for validating theories. We find that intercoder agreement between a large number of students is almost as good as the intercoder agreement between two professionals working on the same materials.

Keywords: Qualitative Data Analysis, Theory Triangulation, Intercoder Agreement, Distributed Coding, Collective Coding

Reference: Andreas Kaufmann, Ann Barcomb and Dirk Riehle. “Using Students as a Distributed Coding Team for Validation through Intercoder Agreement.” Friedrich-Alexander-Universität Erlangen-Nürnberg, Dept. of Computer Science, Technical Reports, CS-2016-01, April 2016.

The paper is available as a local PDF file and also on FAU’s OPUS server.

Abstract: Traceability is an important quality aspect in modern software development. It facilitates the documentation of decisions and helps identifying conflicts regarding the conformity of one artifact to another. We propose a new approach to requirements engineering that utilizes qualitative research methods, which have been well established in the domain of social science. Our approach integrates traceability between the original documentation and the requirements specification and the domain model and glossary and supports adaptability to change.

Keywords: Requirements analysis, requirements traceability, qualitative data analysis

Reference: Andreas Kaufmann, Dirk Riehle. “Improving Traceability of Requirements through Qualitative Data Analysis.” In Proceedings of the 2015 Software Engineering Konferenz (SE 2015). Springer Verlag. Pages 165-170.

The paper is available as a PDF file.