Models and Theories to Promote Health Behavior and Proposed Project Plan
Order Description
here is a video
https://www.youtube.com/watch?v=SWvakOqM3QU

1
Guidelines on Health Promotion Evaluation
Health Promotion
Health promotion is the process of enabling people to increase control over and to improve their health.1 Health promotion represents a comprehensive social and political process, not just actions to improve skills and capabilities of individuals, but also involves changing of the social, environmental and economic conditions which might improve public and individual health. Health promotion interventions often involve different kinds of activities, a long time scale and several partners who may each have their own objectives. Thus evaluating health promotion interventions is not a straight forward task.
Health Promotion Evaluation
Evaluation implies judgment based on careful assessment and critical appraisal of given situations, which should lead to drawing sensible conclusions and making useful proposals for future action.2 In all evaluation, there are two fundamental elements: identifying and ranking the criteria (values and aims); and gathering the kind of information that will make it possible to assess the extent to which they are being met.3 Below are some criteria which can be used to judge the worth of a health promotion intervention:4 ?? Effectiveness – the extent to which aims and objectives are met ?? Appropriateness – the relevance of the intervention to needs ?? Acceptability – whether it is carried out in a sensitive way ?? Efficiency – whether time, money and resources are well spent, given the benefits ?? Equity – equal provision for equal need
2
Outcome Hierarchy
Well defined outcomes are important in evaluating health promotion interventions and facilitate better communication of what constitutes success in health promotion. Below is an outcome model for health promotion by Nutbeam (1996):5
Table 1: Outcome model for health promotion
Health Promotion
Education
Facilitation
Advocacy
Actions
Health Promotion
Health
Social influence
Healthy public
Outcomes
literacy
and action
and organisational
practice
Intermediate
Healthy
Effective
Healthy
Health Outcomes
lifestyles
services
environment
Health and
Mortality, morbidity, disability, quality of life
Social Outcomes
In this model, there are three levels of outcomes. Health promotion outcomes reflect changes in personal, social and environmental factors which may improve people control over health and thereby change the determinants of health (intermediate health outcome). The goal of health promotion actions are to reduce mortality, morbidity and disability of the population (health and social outcome).
3
Evaluation Cycle
Figure 1: Evaluation cycle Figure 1: Evaluation cycle Figure 1: Evaluation cycle Figure 1: Evaluation cycleFigure 1: Evaluation cycle Figure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycle Figure 1: Evaluation cycle Figure 1: Evaluation cycle Figure 1: Evaluation cycle Figure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycle Figure 1: Evaluation cycle Figure 1: Evaluation cycleFigure 1: Evaluation cycle Figure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycle Figure 1: Evaluation cycle Figure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycleFigure 1: Evaluation cycle Figure 1: Evaluation cycle Figure 1: Evaluation cycle Figure 1: Evaluation cycle Figure 1: Evaluation cycle
Figure 1 is a simplified version of the evaluation cycle which has outlined all the important stages in the cycle.6
Needs Assessment
Needs assessments are conducted in order to get a comprehensive picture of the health problems in the community and guide the choices about the type of health interventions required. Needs assessment can be divided into two main stages: Stage 1 Identifying the priority health problem – the purpose is to collect data and canvass a range of opinions to determine the priority health problem. The magnitude of the problem should be clearly specified along with details about the target group having the problem. Stage 2 Analysis of the health problem – the purpose is to collect additional data about the factors that contribute to the health problem.
4
For details of how to perform needs assessment, you may refer to Hawe P, Degeling D and Hall J (1990).7
Evaluability Assessment
Evaluability assessment is a diagnostic and prescriptive tool for improving programmes and making evaluation more useful. It is a systematic process for describing the structure of a programme (i.e. the rationale, objectives, activities and indicators of successful performance); and for analyzing the plausibility and feasibility for achieving objectives, the suitability for in depth evaluation, and the acceptability to programme managers, policy makers and programme operators.8 In evaluability assessment, you check to see whether or not a programme satisfies a number of preconditions for evaluation. To make sure that the programme is ready to be evaluated, you should be able to answer the following questions:9
1. Why will you evaluate?
This sets the evaluation design in motion. You need to identify the primary users of the evaluation information and find out what type of information they require. Success may mean different things to different groups of people or stakeholders who have their own agendas and interests. For example, funders of a project may be looking for efficiency or results which can be interpreted as cost effective. Practitioners may be looking for evidence that their way of working is acceptable to clients and that the objective set has been achieved. It is therefore important to be clear at the outset about whose perspectives are being addressed in any evaluation.
2. Whom will you evaluate?
This refers to the target group of the programme: individuals, groups or community; and the setting: school-based or home-based. For example, in a school-based drug education programme, students, parents, teachers, administrators and community leaders all might be evaluated.
5
3. What will you evaluate?
This relates to the targets of evaluation. For example, in the drug education programme, you might appraise knowledge, attitudes, and behaviours of the students; ask the teachers if the materials were easy to use; evaluate the willingness of the teachers to implement the programme; and assess the cost effectiveness of the programme. You might also ask the parents and community leaders about how the programme has helped the students and community as well as areas that are in need of improvement. Performance indicators will usually be developed to help you evaluate the programme. These indicators should be developed in line with the following rules: a. Identify appropriate standards (as a basis for comparison). i. How did I perform this time compared with last time? ii. How did I perform compared with other people? iii. How well I perform out of a hundred? (consistency) b. Develop quantifiable indicators. c. Establish their relationship(s) with the relevant objective(s). d. Is it relatively easy to collect data for measurement?
4. Where will you evaluate?
This means the place where evaluation is carried out. We should choose a site that is comfortable, both physically and mentally, for the participants being evaluated. For example, a participant may feel more comfortable at a place which is convenient and familiar to him/her. The participants may prefer completing questionnaires by themselves, in the absence of an investigator. It may be more convenient for students to complete evaluation questionnaires in school.
5. When will you evaluate?
The timing of performing evaluation is important. The outcome of a programme will vary at different time periods after the intervention. Some effects are immediate whilst others are slow in emerging. Some effects are transient and others long lasting. Green (1977)10 has highlighted some of the ways in which the evaluation of outcomes of health promotion programmes may be influenced by timing:
6
a. The sleeper effect: if the effects of the programme become apparent after a considerable period of time, evaluation carried out upon completion of the programme will not assess the effects. For example, behaviour change will take time to develop, if we evaluate too early, no effect will be observed. b. The backsliding effect: the intervention will have a more or less immediate effect which decreases over time. If we evaluate too late we will not measure the immediate impact; and even we do observe the early effect, we cannot assume it to be permanent. c. The trigger effect: the programme sparks off a change which would have occurred spontaneously at a later date. This may, of course have real benefits, but we have to be careful not to overestimate the effects of the intervention. d. The historical effect: some or all of the changes could be due to causes other than the programme. For example, the objective of the intervention is to increase the prevalence of a variable, and if this variable is on the increase anyway, we shall overestimate the benefits of the intervention. e. The contrast effect: this may occur when the programme is terminated prematurely, or when the subjects have expectations which are not fulfilled. A consequently embittered group of clients may act in defiance of advice on behaviour, producing a “backlash” effect. Evaluation during, or soon after the intervention would measure the benefits but not the contrasting backlash which occurred after termination of the activity.
6. How will you evaluate?
This is about the evaluation designs that will be used. This will be discussed later.
7. Who will do the evaluation?
If the programme is to be evaluated by the health promotion specialist who are involved in the programme, the evaluation may be biased. It is best for the evaluation to be carried out by an external health promotion specialist. However, this may increase the cost of the programme and is not always feasible.
7
Below are the steps in evaluability assessment:7
Step 1 Identify the primary users of the evaluation information and find out what they need to know. Step 2 Define the programme – define the boundaries of the programme and distinguish the background activities from the programme. Step 3 Specify goals and expected effects – the goals should be realistic and clearly defined. Both the intended effect (i.e. the goals) and the unintended effect (i.e. unexpected effect or “side effect”) should be considered when planning the programme. For example, you may be planning a health promotion programme to promote cervical screening, if your programme is successful, there will be many women attending the Woman’s Clinic requesting a pap smear which is beyond the capacity of the clinic. Step 4 Ensure that the programme is plausible – make sure that the intervention is effective by clearly defining the problem first and searching through the literature for effective interventions. Step 5 Decide on measurable and testable programme activities and goals – not all programme activities are worth measuring or monitoring. Also that not all goals are measurable. One has to decide what should be measured and monitored. Step 6 Decide on what is sufficient in the evaluation – one needs to make sure that there are enough data to supply the users with information they need. Step 7 Make sure that the programme is being implemented as intended – this is the same as process evaluation.
8
Types and Levels of Evaluation in Health Promotion11
There are five types and levels of evaluation:
1. Formative evaluation
Formative evaluation is also called pre-testing and many people would group it under process evaluation. The objective of formative evaluation is to examine how well the intervention is developed to achieve the planned change. In other words, it is used to ensure that health promotion interventions are tailor-made to their particular, defined target group(s) and that the intervention is in fact effective in achieving its aim. It is the testing of the intervention with a sample of the target group. Very often, qualitative methods such as focus groups and interviews are carried out at this stage. It is important to pay attention to characteristics of the target group as well as the language, design and communication channels of the proposed project. Other important issues include relevance of the message, imagery and communication media at personal and group levels, recall and comprehension of the message, credibility, appeal and quality of the messages or imagery. Marketing and mass media theories would need to be applied in this stage. For example, you are producing a pamphlet on senile dementia targeting at the elderly, you may need to hold a focus group for the elderly soliciting their opinions about the pamphlet (whether they understand the pamphlet, like the graphics etc.) so that you can refine the pamphlet which suits the needs of the elderly.
2. Process evaluation
Process evaluation examines the extent to which the programme is delivered as designed.12 It is an essential component of any health promotion programme and is a prerequisite of impact and outcome evaluation. One cannot assess the effectiveness of any programme unless the programme has been implemented as desired. In general, process evaluation employs a wide range of qualitative methods, for example, interviews, diaries, observations and content analysis of documents. These methods tell us a great deal about particular programme and the factors leading to its success or failure, but they are unable to predict what would happen if the programme is replicated in other areas. More information about qualitative evaluation can be found in Table 2.
9
Process evaluation should be able to address the following questions: ?? How well was the programme implemented? ?? Did the intervention reach the intended target recipients? ?? What proportion of the target recipients actually received the intervention? ?? Was the intervention acceptable to the recipients? ?? What was the satisfaction level of the recipients?
3. Impact evaluation
Both impact and outcome evaluation assess the effects of an intervention. Impact evaluation assesses the effects of an intervention on its immediate achievements which will bring about health outcomes (corresponding with the measurement of the programme’s objectives). These achievements can be classified generally into behavioural or non-behavioural dimensions. Achievements in the behavioural dimension are usually changes in awareness, attitudes, knowledge, skills and behaviour among project recipients. Non-behavioural achievements will center on the achievements in organisational and policy changes.
Knowledge measures Knowledge measures aim to assess whether or not the transmission of factual information to the programme recipients is effective and that the information can be understood or recalled. This is usually assessed by quasi-experimental method like pre- and post-intervention knowledge tests. Skill measures Skill measures aim to assess the extent to which the programme recipients can master certain skills or perform actions to promote health. This can be assessed through observations and demonstrations of skills in settings approximating those encountered in real life. Attitude measures Attitude measures aim to assess the changes in values and believes that affect individuals acting in a particular manner. This can be assessed through self-report inventories.
10
Behavioural measures
Behavioural measures aim to assess the changes in behaviour under normal circumstances in real life as a result of the intervention. This can be assessed by observations or self-report inventories.
Environmental and policy measure
Environmental and policy measures aim to assess the changes in policy (e.g. statements, guidelines, rules and regulations) and infrastructure (e.g. participation, networks, committees and facilities) at both organisational and community levels. Organisational support can be measured at different levels.13,14 These dimensions are: purpose, structure, leadership, relationships, helpful mechanisms, rewards, expertise and attitude change. Community capacity can be measured in term of eight dimensions,15,16,17 namely, participation, commitment, self-other awareness, articulateness, conflict containment, management of relationships and social support. Impact evaluation tends to be the more popular choice of evaluation because it is easier to do, less costly and less time consuming than outcome evaluation. Below are examples of some measures of impact evaluation of an anti-smoking programme: ?? Increased knowledge, e.g. effects of passive smoking ?? Changes of attitudes, e.g. less willing to be passive smoker ?? Acquiring new skills, e.g. learning relaxation methods to reduce stress in stead of smoking ?? Introduction of health policies, e.g. funding to enable GPs to prescribe nicotine replacement aids for poor people
11
4. Outcome evaluation18
Outcome evaluation assesses the long term effects of an intervention and usually corresponds with measurement of the goal of the programme. Outcome evaluation aims to improve an individual’s physiological and social aspects of health. It assesses whether changes in risk factors, morbidity, mortality, disability, functional independence, equity and quality of life will occur as a result of the intervention. Functional independence, equity and quality of life can be examined by either a single-item measure or a composite score that is developed based on a number of measures. Outcome evaluation is usually more complex, costly, more time consuming and more resources required than for impact evaluation. However, outcome evaluation is needed as it measures the sustained changes which have stood the test of time. Below are examples of some measures of outcome evaluation of an anti-smoking programme: ?? Reduction in risk factors, e.g. reduction in number of smokers and amount of tobacco consumed per person ?? Reduced morbidity, e.g. reduced hospital admission rates of respiratory illness and coronary heart disease ?? Reduced mortality, e.g. reduced mortality from lung cancer
5. Economic evaluation18
For health promotion practitioners, they often will carry out the above four types of evaluation, but for administrators or managers, they would like to know whether the desired results have been achieved in the most economical way and whether allocating resources to health promotion can be justified. Cost-effectiveness analysis (CEA) and cost-benefit analysis (CBA) (sometimes cost-utility analysis (CUA)) are then carried out to see if the spending in health promotion is justifiable. It is often assumed that prevention is cheaper than cure and that health promotion saves money, but it is not necessarily the case.
12
Evaluation Methods
There are two approaches in evaluation: quantitative and qualitative. Each approach has advantages for answering certain kinds of questions. Table 2 shows the difference between the two kinds of approaches. Table 2: Differences between quantitative and qualitative approaches19
Quantitative evaluation
Qualitative evaluation
Larger number of subjects – generalisable to broader population
Smaller number of subjects/cases
Deductive generalizations – objectivity; strength of the scientific method; experimental/quasi-experimental designs; statistical analysis
Inductive process –phenomenological inquiring; naturalistic, holistic understanding ofthe experience in context; content or case analysis
Valid, reliable instrument used to gather data; specific administration protocol
Researcherprotocol
is the instrument; less rigid
Use of standardized measures; predetermined response categories
Able to study selected issues in depth and in details
Rigor
Flexibility, insight
Results easily aggregated for analysis and easily presented
Understanding of what individual variation means; deepening understanding, insights
Can be perceived as biased, predictable, or rigged to obtain certain results
Offers credibility of an outsider making assessment
Results easily aggregated for analysis and easily presented
Results are longer, detailed, variable in content; difficult to analyse
Data include actual numbers;
Data include group or individual opinions
frequencies/counts of people, events,
or perceptions; relationships, anecdotal
systems changes, passage of
comments, assessment of quality;
policy/legislation, trends
descriptions; case studies; unanticipated outcomes
Experimental conditions and designs to
Openness
to
variation
and
multiple
control or reduce variation in extraneous
directions
variables; focus on limited number of
predetermined measures
To determine which evaluation approach to use, one has to identify the stakeholders – those who determine what questions they want to have answered, and what evidence will convince them that the programme is working and be clear about what type of information is desired by and acceptable to the stakeholders. In general, qualitative approach are used in formative evaluation, and mixed qualitative and quantitative approach used in process, impact and outcome evaluation.
13
Ethical Issues
Whenever evaluation is conducted, ethical standards should be observed: ?? Informed consent must be obtained from the respondents to the evaluation study ?? All data collected must be kept in strict confidence ?? Respondents have the right to withdraw from the evaluation study ?? There must be no collection of unnecessary information from the respondents ?? The respondents of the evaluation study must be free from coercion ?? The researcher/evaluator must be value free and must not be in conflicts of interest ?? The researcher/evaluator must not withhold findings of the evaluation study
Is Evaluation Worth the Efforts?
Ongoing routine work which is based on previously demonstrated effectiveness or efficiency is probably not worthwhile evaluating in depth. However, new or pilot interventions do warrant a more thorough evaluation. It is because without evidence of their effectiveness or efficiency, it is difficult to argue that such interventions should become established work practices. Evaluation is worthwhile only if it makes a difference. This means that the results of the evaluation need to be interpreted and fed back to the relevant audiences in an accessible form so that improvement can be effected in the programme. Evaluation will consume resources. As a general guide, evaluation should cost approximately 10 to 20% of the resources.
14
References
1. World Health Organization. Ottawa Charter for Health Promotion. Geneva: World Health Organization, 1986. 2. World Health Organization. Health Programme Evaluation: Guiding Principles for Its Application in the Managerial Process for National Health Development. Geneva: World Health Organisation, 1981. 3. Peberdy A. Evaluation Design. In: Katz J, Peberdy A (editors). Promoting Health: Knowledge and Practice. Basingstoke, Hampshire: Macmillan/ Open University Press, 1997. 4. Naidoo J, Wills J. Health Promotion: Foundations for Practice. 2nd ed. Edinburgh: Baillière Tindall, 2000. 5. Nutbeam D. Health Outcomes and Health Promotion: Defining Success in Health Promotion. Health Promotion Journal of Australia 1996;6(2): 58-60. 6. Bauman A. Qualitative Research Methods in Health Promotion. Sydney: Australian Centre for Health Promotion Research Unit, University of NSW and University of Sydney, 2000. 7. Hawe P, Degeling D, Hall J. Evaluating Health Promotion: A Health Worker’s Guide. Sydney: Maclennan and Petty, 1990. 8. Smith MF. Evaluability Assessment: A Practical Approach. Boston: Norwell, Mass.: Kluwer Academic Publishers, 1989. 9. McDermott RJ, Sarvela PD. Health Education Evaluation and Measurement: A Practitioner’s Perspective. 2nd ed. Boston: WCB/ McGraw-Hill, 1999. 10. Green LW. Evaluation and Measurement: Some Dilemmas for Health Education. American Journal of Public Health 1977;67(2): 155-61. 11. Tang KC. Health Promotion Evaluation Component Evaluation Protocol. Health Promotion Project Management Lecture, 1 August 2000.
15
12. Moskowitz J. Preliminary Guidelines for Reporting Outcome Evaluation Studies of Health Promotion and Disease Prevention Programs. In Braverman MT (editor). Evaluating Health Promotion Programs. San Francisco, Calif.: Jossey-Bass, 1989. 13. Goodman RM, McLeory KR, Steckler AB, Hoyle RH. Development of Level of Institutionalization Scales for Health Promotion Programs. Health Education Quarterly 1993;20(2): 161-78. 14. Pfeiffer JW, Jones JE. Weisbord’s Organisation Assessment Questionnaire. The Annual Handbook for Group Facilitators. La Jolla, Calif.: University Associates Publishers, 1980. 15. Eng E, Parker E. Measuring Community Competence in the Mississippi Delta: The Interface between Programme Evaluation and Empowerment. Health Education Quarterly 1994;21(2): 199-220. 16. Dixon J. Community Stories and Indicators for Evaluating Community Development. Community Development Journal 1995;30(4): 327-36. 17. Goodman R, Speers M, McLeroy K, Fawcett S, Kegler M, Parker E, et al. Identifying and Defining the Dimensions of Community Capacity to Provide a Basis for Measurement. Health Education and Behaviour 1998;25(3): 258-78. 18. Thorogood M, Coombes Y (editors). Evaluating Health Promotion: Practice and Methods. Oxford: Oxford University Press, 2000. 19. Capwell E, Butterfos FD, Francisco V. Choosing Effective Evaluation Methods. Health Promotion Practice 2000;1(4): 307-13.
Prepared by Central Health Education Unit Department of Health Hong Kong SAR Government 13 June 2005
Evaluation Questionnaire
We hope that the “Guidelines on Health Promotion Evaluation” has provided you and your organisation with useful information on health promotion evaluation. Your feedback will enable us to improve our future production of other guidelines for health promotion practitioners. Please let us have your opinion on the following: 1. Do you find these Guidelines useful? ?? Very useful ?? Somewhat useful ?? Not useful 2. What information do you find most useful in these Guidelines? 3. What additional information would you like to include in future? 4. Do you have any further suggestion or comment on these Guidelines? 5. Any other comment? 6. Your information: Name: Post/Job Nature: Organisation: Tel No.: Email:
Thank you for your suggestion/comment.
Please return by fax (Fax No.: 2591 6127) or mail to Central Health Education Unit Centre for Health Protection Department of Health 7/F, Southorn Centre 130 Hennessy Road Wanchai, Hong Kong