5th IEEE Workshop on Empirical Studies of Software Maintenance 1999, Keble College, Oxford, UK

Summary of discussion in working group 1: "How can we encourage practitioners to respond to empirical evidence?"

The group agreed that response from practitioners would be welcome and would mean that the practitioners either apply (at least try out) techniques suggested by researchers or respond with constructive criticism, in particular in the form of specific requests for the kind of research they would like to see.

We then compiled two lists of reasons why we do not yet get enough response, separately for two major situations: Research done in cooperation with practitioners and other research.

Why we do not get enough response even when cooperating with practitioners

The following list is ordered by the importance of the reasons as perceived by the working group.

There is usually a lack of management buy-in, i.e., management does not support and encourage attempts to evaluate and adopt new techniques. Companies tend to just "follow the herd" with respect to Software Engineering technology; they may not move towards even a large potential benefit unless they perceive an acute problem.
Practitioners usually do not let us see their real problems.
The reasons are confidentiality issues, embarassment, etc.
It requires an enormous amount of effort to build a really close and trusting relationship.
Practitioners tend to respond only to those problems they perceive as severely painful.
Practitioners tend to resist change, because they hang on to their expertise in the currently used techniques.
Researchers and practitioners come from two quite different professional cultures and also use quite different language.

Why we do not get enough response in general

Again, the list is ordered by the importance of these problems as perceived by the working group.

Researchers often attack the wrong problems.
In particular, much work is based on assumptions that are usually unrealistic, such as: having a precise specification, having complete and up-to-date design documentation, etc.
Worse, often the research is entirely naive in its approach.
There is a frequent lack of evidence of the usefulness of research results.
Sometimes there is no evidence at all. Usually the evidence that is present will stem only from toy problems, or (when viewed from the perspective of a particular practitioner) will be given in the "wrong" application domain, in the "wrong" organisational setting etc.
Practitioners and researchers have very different goals.
This results in hard-to-resolve incompatibilities of their concrete interest.
Even if practitioners would like to respond, time pressure or lack of process maturity often make this almost impossible.
Researchers do not solve problems.
At best they usually contribute small steps towards a solution.
Research prototype software tools usually lack the usability and robustness required for practical use.
Researchers often usually attempt to achieve too much.
This results in a wide gap between goal and actual result, which destroys the actual or perceived usefulness of the result.
Practitioners rarely read research journals.
They may never see what we do. Even if they do, the NIH syndrome ("not invented here") may take effect.
Research contributions often lack a short and simple take-home message.
Researchers fail to perform an appropriate level of marketing and evangelization of their results or lack the authority to get them accepted.

What to do

For some of the above problems, it is fairly obvious how any individual researcher could attempt to avoid them -- assuming s/he is sufficiently interested in getting more response from practitioners.
For others among the above problems it is entirely unclear what the research community could do about them. We therefore decided to single out a single problem, for which believed we could make the most concrete and useful suggestion how to cope with it. The group vote indicated that this was the "lack of evidence" (followed on rank 2 by "attacking the wrong problem" and "lack of simple take-home message").

How to avoid the lack of evidence for the usefulness of research contributions

We identified two underlying problems: First, the lack of awareness (on the part of some researchers) that actual evidence of usefulness is important; second, a lack of pressure to obtain reasonable evidence despite the substantial effort required for doing so.

Consequently, our suggestion presented below has two goals: First, making the consequences of not having good evidence more obvious; second, mildly imposing more pressure towards obtaining evidence. Here is our suggestion: Editors and reviewers of papers that present a Software Engineering model, method, technique, or tool should enforce the following guidelines for making sure that reasonable evidence of usefulness is provided where possible:

Explicitly state both the overall research topic as well as the specific research question addressed in the given article and clearly discriminate between the two.
Explicitly and precisely state your claims about the usefulness and other attributes of interest for your contribution. Clearly delineate the limits of your claims.
For each claim, provide convincing evidence that your contribution fulfils it (or not). Where this is not possible, adjust the extent and formulation of your claims to fit the evidence that you actually provide.
If you want to state claims that are not backed up by evidence at all, completely separate them from the supported claims and explicitly indicate that they are speculative.

The members of this working group were Rachel Harrison, Timothy Lethbridge, Frank Niessink, Heikki Nurmy, Thomas Ostrand, Lutz Prechelt, Austen Rainer, Risto Vehviläinen, Elaine Weyuker.

Lutz Prechelt, prechelt@ira.uka.de, Last modified: Tue Sep 14 16:14:31 MET DST 1999