Internal vs. External Validity of Research Funding

So far, most of my research funding has been from industry. Sometimes, I have to defend myself against colleagues who argue that their public funding is somehow superior to my industry funding. This is only a sentiment; they have not been able to give any particular reason for their position.

I disagree with this assessment, and for good reason. These two types of funding are not comparable and ideally you have both.

In research, there are several quality criteria, of which the so-called internal and external validity of a result are two important ones.

  • Internal validity, simplifying, is a measure of how a result is consistent within itself. Are there any contradictions within the result itself? If not so, than you may have high internal validity (which is good).
  • External validity, simplifying, is a measure of how a result is predictive or representative of the reality outside the original research data. If so, your result may have high external validity, which is also good.

Public grants have high internal validity but low external validity. In contrast, industry funding has high external validity, but low internal validity. The following figure illustrates this point:

grants

To understand this, one needs to understand that the allocation of research funds is a social process.

Let’s look at public funding first. Initially, a peer group of (leading) researchers informs the funding agencies of how they think public money should be allocated to research topics. Then, a researcher receives funds because that peer group of researchers believes that the proposed research will amount to something within the defined portfolio allocations. Thus, the more in tune a researcher is with his or her peer’s beliefs of what will be good research, the more likely she will receive public funding. The money being spent is taxpayer money and nobody feels harmed if the project does not deliver anything. Thus, a funding decision by such a peer group reflects that group’s world view (high internal validity) but does say little about any societal relevance (low external validity) beyond that peer group’s beliefs. It is easy for a researcher to toil in obscurity and never have any real-world impact.

Next, let’s look at industry funding. While some large companies manage a portfolio of research funds to be handed out, most research funding is one-off funding. A researcher negotiates with a company about work to be performed and if everything works out, they settle on a contract. The researcher’s professional peer group has no say in this process (no internal validity), but the company is willing to pay significant amounts of money (high external validity). If the research project fails, there will a thoroughly annoyed industry partner, so harm is being done. The danger here is that the work being performed is pure consulting work and may not lead to any relevant research results.

The 2×2 matrix above illustrates this relationship. As the result of social processes, public grants and industry funding cannot and should not be compared in assessing research performance. The only thing that counts is the research output, and that is best evaluated using citations and other measures of societal impact.

There is a larger point to be made here about who is leading who in software engineering: Is academia ahead of industry or is industry leading academia? However, I’ll leave this discussion to another article.

The use of the terms internal and external validity in this article is an analogy rather than a strict quality criterion of a grant proposal. However, it illustrates my point nicely.

Leave a Reply