When and how to evaluate health information systems?

Abstract

Evaluating large scale health information systems (HIS) such as hospital systems can be difficult. This article discusses the reasons we need to evaluate these systems and a range of appropriate methods to carry out evaluations. It is written in non-technical language to assist health policy makers and others commissioning or implementing such systems, with references and a web site containing information for those wishing more detail (http://www.ucl.ac.uk/kmc/evaluation/index.html). A variety of questions relevant to HIS and qualitative and quantitative methods ranging from simple before-after to controlled before-after and fully randomised designs, are discussed. A running example--evaluating the impact of an order communications system on lab requests--is used to illustrate the potential problems, and how they can be resolved. The main types of biases affecting impact studies and methods to reduce them are described. The article then discusses some trade-offs between the low cost, easily conducted before-after study with its unreliable results versus the more complex, expensive but much more rigorous randomised trial. As would be expected, the correct methods to evaluate depend not on what technology is being evaluated--whether an information system or a drug--but on the questions the study is designed to answer, and how reliable the answers must be. Only those commissioning an evaluation study can decide these.