A6:    Evaluation in health informatics

Heather Heathfield PhD. Consultant, IT Perspectives Ltd (UK); Derek Felton Managing Consultant, Secta Group (UK); Susan Clamp PhD.Director, Clinical Information Science Unit, University of Leeds (UK)

Target level: BEGINNERS

The use of the information technology in healthcare is becoming widespread in both primary and secondary care. GP computing and applications such as hospital information systems, electronic patient records, order entry and reporting systems, computer-based guidelines, and even decision support systems are becoming commonplace. The Internet has provided opportunities for a wide-range of clinical and patient-oriented information resources.

Given that the introduction and maintenance of IT in healthcare consumes large amounts of resources, the issue of evaluation becomes increasingly important. Decision-makers and those who are responsible for the procurement or development of IT are expected to demonstrate that resources spent on IT provide some benefits, either in clinical outcomes, cost savings, or to the healthcare process. Furthermore there is a need to understand the effects of IT on the social, professional and organisational context in which they are used. Whilst the need for evaluation is readily accepted by decision makers, those responsible for the design and implementation of evaluation studies are faced with a bewildering and often conflicting array of choices and dilemmas concerning evaluation criteria, study designs, data collection methods and techniques of analysis.

This tutorial will provides an introduction to the design and implementation of evaluation studies of IT in healthcare, with particular emphasis upon the evaluation of electronic patient Records and decision support systems. It is intended to give some insights into common problems and issues that may be encountered in the design of evaluations, the majority of which are practical in nature. The major topics covered will include:

• what is meant by evaluation
• a review of evaluation methods and tools
• the importance of evaluation and the benefits it can provide
• building an evaluation culture
• designing an evaluation
• choosing a study design
• collecting data
• interpreting results
• action planning

Material will be presented mainly as case studies and evaluation design exercises carried out within groups, complimented by an introductory lecture. Participants are encouraged to share their own experiences and evaluation challenges they face with the group.