Skip to content | Change text size

Guide to Evaluating the Effectiveness of Strategies for Preventing Work Injuries:
How to show whether a safety intervention really works

This book aims to provide the tools and concepts required to conduct systematic evaluations of injury prevention initiatives and safety programs.

Preface

Our aim in this book is to provide students, researchers and practitioners with the tools and concepts required to conduct systematic evaluations of injury prevention initiatives and safety programs. Successful evaluations will advance the discipline of occupational safety by building a body of knowledge, based on scientific evidence, that can be applied confidently in the workplace. This knowledge will provide a solid foundation for good safety practice, as well as inform the development of standards and regulations. Building such a knowledge base will help practitioners avoid the temptation of adopting safety procedures simply because they appear “intuitively obvious” when no scientific evidence actually exists for those practices.

Users of the guide are encouraged to demonstrate the strongest level of evidence available for an intervention by measuring the effect on safety outcomes in an experimental design. Even when this level of evidence is not obtained, much useful information can still be gained by following the recommendations in the book. In doing so, the safety field will become current with other disciplines, such as clinical medicine, where evaluation information is increasingly available and allows for evidence-based decision-making.

We hope that this guide will assist safety specialists to meet the challenge of effectiveness evaluations.

Click here to download the complete book (2.5 MB). 
You will need a recent version of Adobe Acrobat Reader to view this document. Click here to download the free reader.

Each of the following sections is a portable document format (PDF). You will need an Adobe Acrobat Reader to view each of these sections.

Table of Contents
Cover Sheet
Click here to download the cover. (740k)
Chapter 1
Introduction: Safety Intervention Effectiveness Evaluation
1.1 What is a safety intervention?
1.2 Effectiveness evaluation
1.3 Overview of the evaluation process and the guide
1.4 Other types of evaluations
Click here to download this chapter. (452k)
Chapter 2
Planning Right from the Start
2.1 Introduction
2.2 Defining the scope of the evaluation
2.3 Who should be involved with the evaluation?
2.3.1 Evaluation committee
2.3.2 Internal vs. external evaluators
2.3.3 Technical or methodological expertise
2.4 Models to assist planning
2.4.1 Conceptual models
2.4.2 Program logic models
2.5 Quantitative vs. qualitative methods for collecting evaluation data
2.6 Choosing the evaluation design
2.6.1 Strength of evidence provided by different evaluation designs
2.6.2 Ethical considerations
2.7 Practical tips
2.7.1 Time management
2.7.2 Dealing with reaction to interim results
2.7.3 Intervention diary
2.7.4 Getting cooperation of workplace parties
2.8 Summary
Click here to download this chapter. (404k)
Chapter 3
Before-and-after design: A simple evaluation design
3.1 Introduction
3.2 Design terminology
3.3 Non-experimental designs
3.4 Before-and-after design
3.5 Threats to internal validity of before-and-after designs
3.5.1 History threat
3.5.2 Instrumentation/reporting threat
3.5.3 Regression-to-the-mean threat
3.5.4 Testing threat
3.5.5 Placebo and Hawthorne threats
3.5.6 Maturation threat
3.5.7 Dropout threat
3.6 Summary
Click here to download this chapter. (507k)
Chapter 4
Quasi-experimental and experimental designs: more powerful evaluation designs
4.1 Introduction
4.2 Quasi-experimental designs
4.2.1 Strategy #1: Add a control group (e.g., pre-post with non-randomized control)
4.2.2 Strategy #2: take more measurements (time series designs)
4.2.3 Strategy #3: Stagger the introduction of the intervention (e.g., multiple baseline design across groups)
4.2.4 Strategy #4: Reverse the intervention
4.2.5 Strategy #5: Measure multiple outcomes
4.3 Experimental designs
4.3.1 Experimental designs with “before” and “after” measurements
4.3.2 Experimental designs with “after”-only measurements
4.4 Threats to internal validity in designs with control groups
4.4.1 Selection threats
4.4.2 Selection interaction threats
4.4.3 Diffusion or contamination threat
4.4.4 Rivalry or resentment threat
4.5 Summary
Click here to download this chapter. (260k)
Chapter 5
Study sample: Who should be in your intervention and evaluation?
5.1 Introduction
5.2 Some definitions
5.3 Choosing people, groups or workplaces for the study sample
5.3.1 How to choose a (simple) random sample
5.3.2 How to choose a stratified random sample
5.4 Randomization - forming groups in experimental designs
5.4.1 Why randomize?
5.4.2 Randomized block design and matching
5.5 Forming groups in quasi-experimental designs
5.6 Summary
Click here to download this chapter. (320k)
Chapter 6
Measuring outcomes
6.1 Introduction
6.2 Reliability and validity of measurements
6.3 Different types of safety outcome measures
6.3.1 Administrative data collection - injury statistics
6.3.2 Administrative data collection - other statistics
6.3.3 Behavioral and work-site observations
6.3.4 Employee surveys
6.3.5 Analytical equipment measures
6.3.6 Workplace audits
6.4 Choosing how to measure the outcomes
6.4.1 Evaluation design and outcome measures
6.4.2 Measuring unintended outcomes
6.4.3 Characteristics of measurement method
6.4.4 Statistical power and measurement method
6.4.5 Practical considerati
6.4.6 Ethical aspects
6.5 Summary
Click here to download this chapter. (688k)
Chapter 7
Qualitative methods for effectiveness evaluation:
When numbers are not enough
7.1 Introduction
7.2 Methods of collecting qualitative data
7.2.1 Interviews and focus groups
7.2.2 Questionnaires with open-ended questions
7.2.3 Observations
7.2.4 Document analysis
7.3 Ways to use qualitative methods in effectiveness evaluation
7.3.1 Identifying implementation and intermediate outcomes
7.3.2 Verifying and complementing quantitative outcome measures
7.3.3 Eliminating threats to internal validity
7.3.4 Identifying unintended outcomes
7.3.5 Developing quantitative measures
7.4 Selecting a sample for qualitative purpose
7.5 Qualitative data management and analysis
7.6 Ensuring good quality data
7.7 Summary
Click here to download this chapter. (400k)
Chapter 8
Statistical Issues: Are the results significant?
8.1 Introduction
8.2 Why statistical analysis is necessary
8.3 P-values and statistical significance
8.4 Statistical power and sample size
8.5 Confidence intervals
8.6 Choosing the type of statistical analysis
8.6.1 Type of data
8.6.2 Evaluation design
8.6.3 Unit of analysis
8.7 Avoiding pitfalls in data analysis
8.8 Summary
Click here to download this chapter. (360k)
Chapter 9
Summary of recommended practices
9.1 Introduction
9.2 Summary of recommended practices
Click here to download this chapter. (160k)
Glossary
Click here to download the glossary. (160k)
Appendix A Some models to assist in planning
A.1 A model for interventions in the technical sub-system
A.2 Models for interventions in the human sub-system
A.3 Models for interventions in the safety management system
Click here to download this appendix. (488k)
Appendix B Examples of statistical analyses
B.1 Analyses for before-and-after designs
B.1.1 Before-and-after design with injury rate data
B.1.2 Before-and-after design with continuous data
B.2 Analyses with pre-post measures and a control group
B.2.1 Pre-post with control group and rate data
B.2.2 Pre-post with control group and continuous data
B.3 Analyses for designs with after-only measures and a control group
B.3.1 After-only measurements with two groups and rate data
B.3.2 After-only measurements with several groups and rate data
B.3.3 After-only measurements with two groups and continuous data
B.3.4 After-only measurements with several groups and continuous data
B.4 Multiple measurements over time
Click here to download this appendix. (108k)
Appendix C Reporting your evaluation results
C.1 Introduction
C.2 Evaluation report
C.2.1 Structure of the report
C.2.2 Audience specificity
C.2.3 Clear language
C.3 Communication beyond the report
C.4 Summary
Click here to download this appendix. (200k)
Bibliography
Click here to download the bibliography. (200k)
This report is also available from the Institute for Work and Health site.