Tutorial 5C

A Primer on Empirical Studies

Dewayne E. Perry, Adam A. Porter, and Lawrence G. Votta, Jr.

Many researchers and practitioners want to use empirical methods, but don't feel confident doing so. One fear is that the methods are expensive to use and hard to use right. This fear may be fueled by methodology articles that overemphasize the ``statistics'' of empirical methods, without considering cost.

Good methodology and good statistics are clearly important. Nevertheless, we have conducted numerous empirical studies -- in the laboratory and in the field -- and we've never done a perfect study! In every case some factor such as cost considerations, development schedules, or subject availability forced us to deviate from the ideal design. Still we claim that our studies are credible. That is their results have validity, are repeatable, and strive to show causality. Thus the goal of the tutorial is to help attendees assess the credibility of empirical work reported in the research literature and to design and conduct credible studies themselves.

The tutorial will cover: (1) The language of empirical methods, (2) The anatomy of an empirical study, (3) Design alternatives and their effect on credibility, (4) cost-effective strategies for designing and conducting credible studies, and (5) new technology for empirical studies.

Dewayne E. Perry, Adam A. Porter, and Lawrence G. Votta have rich backgrounds in software engineering research and practice, covering the entire range of technical and management aspects of engineering large and small software systems, and a broad range of software engineering research problems. In particular, they have been instrumental in forging a new approach to software engineering experimentation in both in vitro and in vivo contexts.