Community Based Injury Prevention Evaluation: The Safe Living Program (1990-1996)

Monash University Accident Research Centre – Report #131 - 1998

Authors: Joan Ozanne-Smith, Graeme Watt, Lesley Day & Voula Stathakis

Full report in .pdf format [3.3MB]

Abstract:

Introduction: Few controlled studies of community-based injury prevention programs have been reported and varying levels of injury reduction outcomes have been demonstrated. Measures of intermediate effects on risk factors and program reach are seldom reported. At the same time expansion of such interventions is occurring across many countries, including Australia.

Aim: To evaluate a community-based injury prevention program, the "Safe Living Program", a WHO accredited Safe Community, conducted over a six year period (1990-1996) in the Shire of Bulla, Victoria.

Methods: A prospective controlled quasi-experimental research design was used. A comparison population (33,600) was matched to the intervention population (37,300) on a number of relevant variables. It was not possible to find a comparison community with a baseline injury rate as low as that of the intervention population, the former Shire of Bulla. Statewide data were used as an additional control.

Process and impact measures of intermediate effects on risk factors and program reach were obtained by three randomised household surveys in the experimental and control populations, conducted pre-, mid- and post-intervention. Separate studies of the impact of selected program components were conducted using observational and survey methods.

Results: The evaluation was complicated by changes to injury data collection systems and populations, particularly increases to hospital admission rates associated with case-mix funding and the transition to a new emergency department injury data collection system. No significant reductions were found in overall rates for deaths, hospitalised injury or emergency department presentations (child ). A significant increase in the proportion of short hospital admissions (a proxy for severity) was observed compared with the control population, though there was no difference in average bed days. Analyses of health sector injury data by specific age groups and injury mechanisms did not reveal any significant differences from the controls.

Self reported injury data showed a statistically significant reduction for households and close to a significant reduction for individuals in households when compared with pre-intervention and control data. Household reported injuries most often required no medical attention nor interruption to normal activities. The possibility of regression to the mean cannot be excluded when interpreting these results. A portion of a reported reduction in occupational injuries among municipal employees appears to be attributable to the intervention program. Process and impact measures demonstrated some significant effects of the program. The reach of interventions to individuals in the community ranged between <1% - 27%. Awareness rasing activities had a wider reach.

Conclusions: The Safe Living Program evaluation was not able to replicate findings of significant reductions in injury rates in health sector data as reported elsewhere in some community based studies. Methodological problems may have contributed to this finding. Some reductions in self-reported, mostly minor injuries, and occupational injuries among municipal employees appear to be attributable to the program.

While substantial changes were made to some risk factors, these often also occurred in the control community. Limited reach appears to have contributed to the outcomes. In future community based injury prevention programs consideration should be given to: fewer, more targeted, interventions; issues of community reach; organisational change; and enhanced evaluation design, including the involvement of several intervention and control communities.

Secure injury data sources are required at all levels of severity. The evaluation was underpowered to detect minor changes to hospital data, though robust changes would have been detected, particularly since some intervention effects would have been expected to increase over time. The theoretical base for community based injury prevention and its evaluation is not yet fully developed and requires further refinement.

Executive Summary

INTRODUCTION

Few controlled studies of community-based injury prevention programs have been reported and varying levels of injury reduction outcomes have been demonstrated. Measures of intermediate effects on risk factors and program reach are seldom reported.

Community-based injury prevention was first described in the research literature in 1987 by Schelp. Significant injury reductions were attributed to a three year intervention program in Falkoping, Sweden compared to Lidkoping, the control municipality21-24. The four published studies of community-based programs that meet the criteria described by WHO for "Safe Communities" have reported varying levels of successful injury reduction outcomes. Despite the somewhat equivocal findings of these evaluations, a rapid expansion of such interventions is occurring across many countries, including Australia.

The rationale underpinning the evaluation reported here was the need to examine whether the injury and risk factor reductions gained in the model project in Sweden could be replicated in Australia; and, if so, to demonstrate the effectiveness or injury or injury risk factor reductions and potential cost effectiveness of community-based injury prevention projects.

AIM

The aim of this study was to evaluate a community-based injury prevention program, the "Safe Living Program", which was modelled on similar Swedish interventions. The former Shire of Bulla was incorporated into the City of Hume in 1994, but this evaluation focuses on the area and population covered by the original program conducted over a six year period (1990-1996) in the Shire of Bulla*, Victoria.

* Since the Shires of Bulla and Melton (comparison municipality) were incorporated into larger amalgamated local government areas in 1994, reference to these Shires should generally be read as "former" Shires in this report.

SPECIFIC OBJECTIVES

1. To evaluate the extent to which the intervention program objectives were met

  • These program objectives were:
  • to increase community awareness of injury prevention and create a 'safer community' environment
  • to develop strategies which provide an overall reduction in the number and severity of injuries within the Shire of Bulla
  • to reduce the number of hospital bed days that result from injuries
  • to reduce the frequency and severity of injuries requiring hospital attendance or admission
  • to reduce the incidence of accidental death
  • to reduce hazards
  • to increase the use of safety devices and equipment
  • to institutionalise the program (added in 1994)

2.    To undertake a formative evaluation to feed back into the program and into wider strategic planning processes.

3.    To identify potential improvements to both the intervention and evaluation methodologies.

METHODS

A prospective controlled quasi-experimental research design was used. A comparison population (33,600) was matched to the intervention population (37,300) on a number of relevant variables. It was not possible, however, to find a comparison community with a baseline injury rate as low as that of the Shire of Bulla. Statewide data were used as an additional control.

Process and impact measures of intermediate effects on risk factors and program reach were obtained by means of three randomised household surveys in the experimental and control populations, conducted pre-, mid- and post-intervention. Separate studies of the impact of selected program components were conducted using observational and survey methods.

RESULTS

The results are summarised for each objective.

  • To increase community awareness of injury prevention and create a 'safer community' environment

This objective relates to program reach and to a culture of safety at various levels of the community. It was particularly addressed by the process evaluation.

Importantly, the community fulfilled the requirements for WHO Safe Community accreditation in 1994 and this was re-confirmed in 1996 for the wider City of Hume, following Council amalgamations.

While there was substantial awareness of the Safe Living Program (46.6% of residents in 1996), and recall of safety messages and displays (eg. 37.4% recall of safety displays at fairs) unfortunately only a moderate amount of action with the potential to reduce injuries appears to have followed from such exposure.

The level of general media coverage was comparable to that of other successful community programs with close to equivalent newspaper articles and information delivered to households compared with the community based Minnesota Heart Health Program102. In 1996, the Safe Living News was read by 13.3% of householders. Up to 12.2% of the population participated in injury prevention courses or activities, though there was generally no significant difference when compared with the comparison community, the Shire of Melton on issues of safety knowledge attitudes and awareness identified in telephone surveys. Support for laws to improve safety showed a significant increase in the Shire of Bulla against a negative trend in the Shire of Melton.

Overall, the reach of interventions to individuals in the community ranged between <1% - 27%, while potentially awareness raising activities had a wider reach. Organisational reach was higher than for the Shire of Melton for primary school utilization of road safety programs, such as Streets Ahead and Bike Ed, from 1991-1993, (86% of schools versus 56% respectively for both programs) though no information is available about the proportion of children reached. Other road safety programs such as Starting out Safely had similar reach in the intervention community versus the comparison community (both 58% in 1996).

In summary, this objective was assessed as reasonably well met at the macro level of WHO accreditation and sustainability and at the micro level of program and safety awareness.

  • To reduce the incidence of accidental death

As may have been anticipated, no significant reduction in injury deaths occurred in the Shire of Bulla compared with the Shire of Melton during the four years of the intervention program for which death data were available (1991-1994). In retrospect, this was probably an inappropriate objective because it is unlikely to be capable of evaluation given the small number of deaths and the long lag time before data become available. A dramatic reduction in the number of deaths would have been required before any statistical significance could be expected to be demonstrated. None of the published successful community based injury prevention programs showed a significant reduction in deaths over a similar time-frame.

  • To reduce the frequency and severity of injuries requiring hospital attendance or admission

No significant reduction was found in the rate for hospital admissions compared with controls. No difference was found for aggregated hospital admission data which generally involve moderate to severe injuries in comparison with the Shire of Melton or Victoria, or when the data were disaggregated by age or mechanism of injury for major causes targeted by the program. Factors influencing this outcome may include: changes to data systems (see page xxiii) , lack of efficacy of program interventions, lack of program reach, issues of council amalgamations, nature of community compared with Scandinavian communities, low base rate of injury, and lack of equivalence of the comparison community.

Available child injury surveillance data from hospital emergency departments showed no evidence of a reduction in the rate of injuries in the Shire of Bulla during the period of the evaluation nor when the Shires of Bulla and Melton were compared. Adult emergency department data were not available post-intervention.

Hospital data generally involves moderate to severe injuries. Although the program objective did not specifically require evaluation of reductions in less severe injuries, this was an obvious direction to investigate, particularly since the gains made in other similar programs were largely for minor injuries. The possibility that an effect may be found in general practice presentations could not be determined as attempts to establish an ongoing data collection were unsuccessful.

Self-reported injury data, collected by the series of telephone surveys, indicated a statistically significant reduction for households in the Shire of Bulla and a non-significant reduction for individuals in households, when compared with pre-intervention and control data. Household reported injuries most often required no medical attention nor interruption to normal activities. The possibility of regression to the mean cannot be excluded when interpreting these results, where a single high estimate was observed pre-intervention for the proportion of households with injuries reported in the Shire of Bulla. A portion of the reported reduction in occupational injuries among municipal employees also appears to be attributable to the intervention program.

  • To reduce the number of hospital bed days as the result of injuries

The proportion of injuries requiring short hospital admissions of <2 days , a proxy for low severity, increased for the Shire of Bulla, when pre-/post- and intervention/control comparisons were made. However, there was no statistically significant difference between average bed days between the Shires of Bulla and Melton. In the Falkoping Program, hospital bed days reduced after 3 years of the program.

As the number of bed days is not necessarily a good measure of severity, alternative, more specific quantified measures, such as the Abbreviated Injury Severity Scale (AIS) and the Injury Severity Score (ISS), should be considered for future studies.

  • To develop strategies which will provide an overall reduction in the number and severity of injuries within the Shire of Bulla

Although a large number of intervention activities were developed and implemented in the course of the Safe Living Program (more than 100), these did not convincingly achieve an overall reduction in the number and severity of injuries.

The international literature generally lacks detail of intervention strategies within community based injury prevention programs. Nevertheless, it seems that there was considerable overlap between the Safe Living Program and successful overseas programs. Because of the large number of Safe Living Program strategies, there may have been dilution of effort compared with other programs.

High levels of program reach, including training of professionals and uptake of safety products or environmental changes appear to be key strategies in successful community interventions. Reach (or penetration) appears to be greater in Falkoping compared with the Shire of Bulla. In Falkoping, study and educational programs were arranged for large numbers of professionals who, in turn, acted directly to intervene in the community in various ways.

Falkoping, for example, provided safety education for 650 home visitors who worked through safety checklists with their clients. Although some similar approaches were adopted in the Shire of Bulla, the scale of direct transfer of safety information and recommendations contrasted with that of Falkoping.

Similarly, effective reach was achieved in Harstad, where, for example, 80% of persons aged 75-79 years were reached by the home visiting intervention program.

In the Minnesota Heart Health Program, 42 of 65 local physicians participated in continuing education and 728 other health professionals attended relevant professional education classes.

Uptake of the countermeasures specifically promoted in the Shire of Bulla has been generally quite small. Hence, even if the interventions were highly effective, only small population reductions in injury frequency and severity would be expected. Furthermore, while the Safe Living Program distributed safety checklists to all households, by contrast, in Falkoping and Harstad30 checklists were used actively by professionals on home visits. Thus, the practical work of implementation was effected primarily via personnel groups who are in contact with the target group in their daily work.

A recommendation of the evaluation report of the Safe Living Program after the first three years of the intervention was to "increase the reach of and community participation in the existing successful interventions".

The quality of the interventions is another important contributor to effective injury prevention. The one successful intervention in the United States SCIPP program, motor vehicle occupant protection, was described by Guyer as highly efficacious.

Social and cultural differences between the Falkoping community and the Shire of Bulla may also have contributed to the lower than expected level of demonstrable outcome. It is possible that the Swedish community is more homogeneous, with stronger common values, than the Shire of Bulla. Some demographic differences between the Shire of Bulla (1991 census) and Falkoping include a lower immigrant population in Sweden and a higher adult proportion of the population. The Shire of Bulla also had higher unemployment and the program was conducted during a recession. Falkoping has local radio, television and daily papers.

It is clear that the translation of the Falkoping model to Australian conditions is not without its difficulties and although some have been identified, other factors, such as availability of resources, may confound the analysis of the two programs making some comparison inappropriate.

  • To reduce hazards

Response by the Shire of Bulla Council to recommendations from the road safety audit (at least 47% implementation), footpath safety audit and Safe Routes to Schools shows a high level of commitment to action. Council also introduced footpath cycling, and occupational safety measures as a result of advocacy by the Safe Living Program. The primary schools implemented the majority of recommendations (56%) arising from a professional safety audit of school playground equipment in 93% of local primary schools. All primary schools participated in Safe Routes to School and $35,000 was spent on minor works. The update and expenditure on Safe Routes to School in unknown for the Shire of Melton.

A State government regulation allowing footpath cycling in the Shire of Bulla was implemented in late October 1991. A statistically significant increase from 47% to 56% in footpath versus road cycling, p<0.01, occurred between October 1991 and October 1992. Several extensions to the legalisation of footpath cycling were subsequently negotiated with the State Government to April 1998.

Telephone surveys indicated significantly more hot tap water temperature reductions and the presence of other safety features in the home, an increase in items purchased for child safety and the presence of electrical circuit breakers in the Shire of Bulla compared with Melton The uptake of some other interventions promoted in the Shire of Bulla was matched by the Shire of Melton (smoke detectors, first aid training).

  • To increase the use of safety devices and equipment

The Shire of Bulla Safe Living Program achieved increased levels of effective sales of children's safety seats and restraints, sales of smoke detectors, usage of the Early Childhood Injury Prevention Program, training conducted by the Train the Trainer Course participants, and an increase in the use of a handyman service for senior citizens. Attempts to measure sales of other safety products were generally unsuccessful.

An accredited child car restraint fitting station was established in 1991 in the Shire of Bulla, which fits approximately 50 restraints per year and checks about 100 more. While the number of visitors to the four sequential safety display homes, which demonstrate safe design, was high, there is little evidence that a large proportion of Shire of Bulla residents visited the houses, nor of large scale on-selling of the safety features within the Shire. This intervention may have had a greater impact outside the Shire.

A limitation to the reduction of hazards was the diminution of effect. Following the supply of household safety information to every household on two occasions during the program, few householders recalled receiving it ( 27.5%, 1991; 9.3% 1996) and less used it, with only 0.3% reporting making safety changes as a result. Attenuation of effect, or the decreasing impact, of an intervention as it diffuses through a community can strongly influence the impact and outcome of education and training activities. This effect may not have been appreciated in designing the prevention programme.

  • To institutionalise the program (objective )

The Safe Living program as a whole survived amalgamation of local government areas, replacement of elected councillors by appointed commisioners, cost cutting, uncertainty and delays in planning and implementation. It has continued beyond the evaluation period as core business of the Hume City Council from June 1997. The response of the program to uncertain future funding and to evidence that a top down approach continued to be required to sustain interventions was to accelerate progress to institutionalisation of the program and its components.

Questions remain, regarding sustainability. For example, is institutionalisation enough to achieve sustainability?; what are the criteria for deciding what is to be sustained?; are the currently available measures of institutionalisation valid for injury programs?

MUARC has been funded by the Victorian Health Promotion Foundation to continue to evaluate the sustainability of the intervention and its component strategies in the relatively short term. This evaluation will be reported seperately. Clearly, the longer term effects of institutionalisation should also be evaluated.

FORMATIVE EVALUATION

This evaluation has aimed to be formative for both the Safe Living Program and for the broader development of community based injury prevention in Australia and internationally.

Close liaison has been maintained with the intervention program and the local government in which it is embedded for approximately 7 years. Updated local injury data and information from new research have been provided regularly to the program. Similarly, feed-back from the evaluation has been reported at all stages, both formally and informally.

A full evaluation report of the first three years of the Safe Living Program was published in 1994. It made recommendations for the ongoing program’s strategic planning processes, including increased reach and specifically more training of professionals, based on the Falkoping success. The introduction of new interventions to address high frequency, high severity injury issues was also recommended, with specific examples.

LESSONS LEARNED

Introduction

An important objective was to identify potential improvements to both intervention and evaluation methodologies. This section identifies and discusses lessons learned from both the intervention program and the evaluation.

Lessons learned from the intervention program

  • A high level of program reach and uptake of countermeasures is essential. The "snowball" effect gained by training professionals and volunteers working in the community to implement the program is a vital tool.
  • It is most efficient to focus on interventions with known benefits. Highly efficacious countermeasures should be chosen.
  • Organisational involvement and change are important factors
  • Education and awareness do not necessarily translate into action
  • Implementation should occur early in the life of the program and be sustained for maximum effect
  • Community based injury prevention may work better in cohesive, homogenous and isolated communities (Falkoping and Lidkoping homogenous; Harstad isolated and homogenous)
  • Few severe injuries would need to be prevented in order for a program such as the Safe Living Program to be cost-effective
  • Institutionalisation may be an important component of community based injury prevention programs. The longer term effects of institutionalisation should be evaluated, particularly in terms of sustainability and injury prevention impacts and outcomes.
  • Statewide implementation appears to be more efficacious than local intervention for some countermeasures

"Planners of future community trials should be confident that they are effective programs that can be delivered to a sufficiently large fraction of their target population to allow them to accelerate the secular trends for the exposures of interest".

Lessons learned from the evaluation

  • It is necessary to use an appropriate number of entire communities to provide valid evidence of the efficacy of a community intervention method. Logistical problems and costs would be a major barrier, perhaps insurmountable, in Australia.
  • Changes which may affect prospective studies may be unpredictable and this possibility should be taken into account in the research design. The Safe Living Program evaluation was complicated by changes to injury data collection systems and populations, particularly increases to hospital admission rates associated with case-mix funding and the transition to a new emergency department injury data collection system. Alternative sources of data would be an advantage.
  • There are many advantages to the research design if programs are implemented in stable populations (Falkoping, Lidkoping and Harstad appear to have stable populations)
  • Reliable measures of low severity injury rates are required. The Falkoping reduction was mostly in minor injuries, at least for home and occupational injuries. Since it was not possible to record minor injuries in the Shire of Bulla and the control municipality, except for self reported data, some gains in minor injury reductions may have been missed. The availability of both emergency department and general practitioners data, is thus desirable. The apparent gains in the Latrobe Valley study50 were also for relatively minor injuries (requiring emergency department treatment, but not hospital admission). While prevention of medically treated minor injury is clearly warranted, Rivara observes that injuries treated in the physician’s office and in emergency departments "probably represent the minimum severity of injury that deserves prevention".
  • Improved measures of severity of hospitalised injuries beyond number of bed-days should be applied in evaluation studies, though accurate coding may be costly
  • Intervention populations with high baseline injury rates are likely to be more susceptible to reductions in injury rates. It was beyond the control of this study to choose such an intervention population nor to find a comparison community with a baseline injury rate as low as that of the Shire of Bulla. It is noteworthy that, in Falkoping the injury rate was also lower than for the control community. Statewide data were used as an additional control.
  • Comparison communities may be non-equivalent in a number of ways (e.g. injury rates, and methods of collecting injury data; geographically; size and nature of community)
  • In the absence of evaluations of many countermeasures, it is difficult to differentiate between the effects, or lack of effects, of an intervention with regard to the actual countermeasure efficacy and the implementation method
  • Case control studies of populations reached versus not reached may show significant gains among those reached, though such gains may not be cost-effective
  • Indicator measures need to be developed for the impact of injury prevention programs (current reporting on all available measures is costly and not efficient)
  • Telephone surveys may decrease in value as a survey tool due to the high prevalence of answering machines and the increasing proportion of households which are not prepared to participate
  • The comprehensive documentation of intervention strategies (preferably standardised), research methods and results is essential to further progress
  • Key informant interviews may contribute to explanations of successful and less successful impacts and outcomes
  • Measures of institutionalisation and its outcomes are required
  • Larger budgets are required to undertake comprehensive well designed evaluation studies, including benefit/cost analyses, to inform further developments in community based injury prevention
  • The publication of results, including process, impact and outcome results, and of negative results, is essential

OTHER ISSUES

Comparison Community

The choice of the Shire of Melton as a comparison community was based on similar demographics to the Shire of Bulla and also the fact that both Shires had a high catchment by VISS hospitals for determining emergency department injury presentations. In contrast with Scandinavia, where the intervention municipalities of Falkoping and Harstad were separated from their controls by 150 and 1000 kilometres, respectively, the Shires of Bulla and Melton are contiguous, although the population centres are separated in various ways.

The possibility of contamination of the Shire of Melton by the Safe Living Program was not ignored. Preliminary investigations were undertaken for potential contamination sources prior to choosing the Shire of Melton. Knowledge of the Safe Living Program in the Shire of Melton was only 2.8% by 1996, indicating a low level of contamination. Alternative possibilities for a comparison community posed even greater methodological problems.

Data quality

The introduction of case-mix funding during the course of this evaluation had a profound effect on the possibility of identifying small changes in hospitalised injury rates, though robust results should still have been evident. Data quality issues were also reported for some of the controlled studies of community based injury prevention reported in the international literature.

Process and impact measures are also dependent on the availability of good quality data. Organisational change and lack of detailed record keeping makes long term evaluation problematic. It is also possible that low priority attention is given to such information requests. Organisations responsible for intervention programs should keep systematic records as part of quality systems management and for specific evaluation purposes.

Evaluation period

Limitations reported by the authors of the SCIPP study included short duration of program (22 months), a lag in the detected benefit of the program, and difficulties in the ambitious task of implementing and managing many complex interventions in many communities simultaneously. They also assert that evaluation should be over an extended period to determine with certainty whether a measurable benefit is achieved. Indeed, the successful Falkoping and Harstad programs ran, in total, over several years.

Statistical methods

Debate continues regarding the appropriate methods for statistical analysis of intervention and control data in quasi-experimental time trend designs for community based injury prevention programs.

Poisson regression and time series analyses were compared by Kuhn in the analysis of a community based injury prevention program and he found both methods to provide similar results. This finding presumably relates to relatively large rates of injury, thus reducing the variance.

For our analyses, linear regression modelling was chosen as the main tool for analysis. While simple regression methods rely on the assumption of normality in the data, which is approximately true in the case of injury rates, they also rely on constant variance, which was not the case in our results. The low rate of injury presentation per person allowed the binomial model to provide a good fit to the data (as good as when the Poisson model was used).

An editorial in Injury Prevention remains salient "although choosing the most appropriate statistical procedure is undoubtedly important, solid findings are usually robust and unlikely to differ greatly regardless of how they are analysed". This view is also supported in Tauber’s review of the limits of epidemiology, where it is suggested that a relative risk of 3 or more is required before an association is believable.

Unanswered methodological questions

Many questions remain regarding community interventions, including: how large a gain in public health is acceptable from community intervention; what size community would best suit a community action approach; how can scientific concerns about standardisation of intervention be married with community realities of needing to be flexible and allow community input120.

Furthermore, does increased awareness of injury facilitate the uptake of new interventions? That is, awareness may be the basis for receptive attitudes to future interventions. This hypothesis could be tested by a controlled implementation study of a countermeasure or series of countermeasures, in established "Safe Communities", new intervention communities and appropriate control communities.

CONCLUSIONS

The Safe Living Program evaluation was not able to replicate findings of significant reductions in injury rates in health sector data as reported elsewhere in some community based studies. Methodological problems may have contributed to this finding. Some reductions in self-reported, mostly minor injuries, and occupational injuries among municipal employees appear to be attributable to the program.

While substantial changes were made to some risk factors, similar changes often also occurred in the control community. Limited reach appears to have contributed to the outcomes. In future community based injury prevention programs consideration should be given to: fewer, more targeted, interventions; issues of community reach; organisational change; and enhanced evaluation design, including the involvement of several intervention and control communities, ideally randomised. Appropriately developed implementation strategies should be required before further funding of community based injury prevention.

Secure injury data sources are required at all levels of severity. The evaluation may have been underpowered to detect minor changes to hospital data due to the confounding effect of case-mix funding, though robust changes would probably have been detected, particularly since some intervention effects would have been expected to increase over time.

The theoretical base for community based injury prevention and its evaluation is limited in its development, and requires further refinement.

Sponsoring Organisation(s): This project was funded by the Victorian Health Promotion Foundation (1990-1996) & VicRoads (1990-1993)