Introduction

Evaluating health and social care programs is important for multiple reasons. These include, but are not limited to:

• ensuring the organisation is delivering the best programs and services while addressing community needs;
• making programs, services, and systems more efficient and effective;
• informing future directions and strategic service planning;
• producing context-specific and robust evidence to support funding applications;
• strengthening and informing communication and marketing efforts;
• and providing feedback to the community about the value of services and programs.

Program evaluation is a systematic process to examine the value of a program or project, including effectiveness, efficiency, and appropriateness. Patton (2012) defined evaluation as “The systematic collection of information about the activities, characteristics, and outcomes of program, services, policy, or processes, in order to make judgments about the program/process, improve effectiveness, and/or inform decisions about future development”.

Significant resources are invested into the development of clinical pathways, guidelines, interventions, and innovations aimed at improving health and social care outcomes; however, systematic and appropriately scaled evaluations of such innovations receive less attention. Evaluation capabilities are required across all levels of health service management and leadership roles to continue to drive innovation, ensure consumer voices are heard, and build evidence-based responsive, appropriate, and effective health services. Several approaches, (frameworks, models, concepts) can be used for evaluation. This presents a challenge to those undertaking evaluations. The approach and methods used for evaluation can depend on factors such as practitioner experience, the discipline, funding body requirements, consumers and community needs, and the setting. It is crucial to ensure the methods used are fit for the problem at hand and generate evidence that can inform future decision-making.

Background

Evaluation is an applied inquiry process for collecting and synthesising evidence before, during, and after the delivery of real-world policies, programs, and interventions that results in passing judgment about the value, merit, worth, significance, or quality of a program, product, policy, proposal or plan against pre-determined goals, aims, objectives or intentions (Scarth, 2005, Bauman and Nutbeam, 2014). Evaluation is most effective when the needs of the users of the system are placed at the forefront (Patton, 2012); however, this needs to be balanced with rigorous design within constrained resources. Investment in designing robust evaluations for existing and new health service policies, programs, and interventions that seek to enhance community health and well-being or health service performance (which contributes to better health outcomes for populations) is fundamental to ensuring we are delivering the intended outcomes. If not, it allows us to answer the fundamental questions of why not and how we can do this better (Bauman et al., 2014). The primary focus of an evaluation is to determine what is working, what does not work, or what can be done to make improvements.

When evaluating change for complex health initiatives, services, and programs it is important for health service managers to keep in mind small incremental changes over time can make a huge impact. Evaluating these incremental changes contributes to identifying the program components driving the change, thus contributing to sustaining longer-term public health and health service impacts (Steckler and Linnan, 2002). When evaluations of complex health service programs are designed at the outset, pathways to change can be better established through multiple evaluation approaches and methodologies (Bauman et al., 2014). Early evaluation can prevent larger-scale implementation barriers and failures and allows for the lived experience of participants to be appreciated through participatory evaluation methodologies (Scarth, 2005, Bauman, 2014, Patton, 2012).

Evaluation theories and approaches

When planning an evaluation, consideration must be given to the paradigm (and theory) along with the evaluation approach that will be adopted. A paradigm refers to the beliefs about the nature of reality and the types of knowledge required to facilitate an understanding of reality (Fox, Grimm, and Caldeira, 2016). The paradigm being used for an evaluation will inform methodological and data collection choices.

The literature concerning paradigms, theories, and approaches in evaluation is extensive, and continues to be an arena for considerable debate. Lucas and Longhurst (2010) provided a useful introduction to different perspectives that can be brought to evaluation and the approaches associated with these. In a widely cited monograph, Stufflebeam (2001) provided a description of 22 program evaluation approaches. These approaches are distinguished by factors such as their purpose, scope, engagement with stakeholders, methods, timing, and applications. Most can be seen to have foundations in the evaluation paradigms described in this section. This section introduces two commonly used paradigms, and their associated evaluation approaches that assist in operationalising these paradigms when planning a health and social care evaluation.

Theories of change and the realist evaluation approach

The theories of change perspective is described as an expression of the critical realist paradigm; that is, it sets out to understand the complexity and contextual dependence of programs and services. Poland, Frohlich and Cargo (2009, p. 307) described critical realism as: ‘…a logic of inquiry that privileges neither “objective” facts nor subjective lived experience or narrative accounts, but rather seeks to situate both in relation to a theoretical understanding of the generative mechanisms that link them together, as a basis for interpreting the empirical or observable world’.

Based on the critical realist paradigm, the realistic evaluation approach shares many elements of the theory of change perspective (Pawson and Tilley, 1997). Those adopting this approach operate from a position whereby the interventions delivered are considered working (or real world) theories concerning how a set of activities will function in given contexts to create change and achieve the desired program and service objectives.

A key role for evaluators is to work with program managers and other stakeholders to make the theory of change inherent within a program explicit, often represented as logic models, and to use this to guide the evaluation. This approach is characterised by the purposive sampling of a wide variety of quantitative and qualitative data to shed light on the multiple mechanisms of change that take place in the program, and the contextual factors that act as enablers or inhibitors of change, both intended and unintended (Fox, Grimm, and Caldeira, 2016). Harris et al. (2020) provided a practice example of realist evaluation stages and how this paradigm can be applied to the evaluation of a recovery-oriented program in Australia for those with complex, severe, and persistent mental illness.

Within a theories of change paradigm and a critical realists approach the primary focus is on learning about how change is achieved focusing on context and unintended and intended outcomes. This makes it an appealing approach for planning an evaluation of complex health and social services.

Pragmatic perspective and the utilisation-focused evaluation approach

The pragmatic perspective (or paradigm) is guided by collecting information useful for stakeholders and communities rather than being concerned with objective measurement or proving causal pathways of change within programs and services (Patton, 2012). This approach to evaluation focuses on the practical use of evaluation findings to make decisions and improve programs. For pragmatists, questions of truth and validity are of far less importance than obtaining information that ‘works’ for funding bodies, policy makers, service managers, and program beneficiaries.

The utilisation-focused evaluation approach is firmly ground in the pragmatic perspective. Patton (2012) argued that the utilisation-focused evaluation approach allows evaluations to be undertaken with specific users in mind. In this approach, the role of an evaluator is to undertake an analysis of program stakeholders, to identify the primary users of the evaluation findings, and determine the needs and associated questions they have concerning the program. The utilisation-focused approach is suitable for quantitative, qualitative, or mixed methods – these decisions are guided by the interests of stakeholders and their views about credible and useful evidence and data. The evaluator can facilitate this decision making by presenting a menu of context appropriate and rigorous methods to stakeholders and expert advice about the utility, validity, and cost-effectiveness of different options.

In practice, most evaluations will draw on several paradigms to ensure each evaluation question can be addressed and to recognise the interests of various stakeholders, known as a pluralist perspective (Lucas and Longhurst, 2010).

Planning for evaluation

When planning for an evaluation, decisions about the questions you want your evaluation to answer need to be developed and clarified. This will help you develop a statement of purpose or purposes.

The statement of purpose for an evaluation should address the following:

• What is the overarching rationale for evaluating this program?
• Who stands to benefit from the evaluation findings?
• How will this evaluation meet the needs of stakeholders and partners?
• What are the key aspects of the service or program that are to be evaluated and why? Note: this is linked to all previous questions.

When considering the purpose of the evaluation, it is worth reflecting on how the evaluation will meet the following needs:

• the funding requirements;
• the need for evidence in the field;
• your organisation’s need to improve practice, process, and outcomes;
• the community’s needs and interests;
• the interests and needs of decision makers that might be the focus for future advocacy efforts or funding applications.

Numerous frameworks and models can be used for program and evaluation planning. Some are generic in nature and can be used regardless of the paradigm and evaluation approach adopted. Regardless of which framework is adopted to assist you to plan your evaluation, it should include the following elements to ensure your evaluation is well considered:

• a description of the program;
• an evaluation preview or overview (this will include your evaluation purpose and key questions);
• the evaluation design (based on the paradigm, approach, purpose, and questions);
• data collection (the data collection plan, including data sources, indicators, and timelines);
• data analysis and interpretation (a plan for how data will be analysed and interpreted);
• dissemination plan (a plan for how you will communicate, report, and share the findings).

Levels of evaluation and the program planning cycle

Following on from decisions about the nature and reality of knowledge (paradigms and approaches), evaluators then turn their attention to further refinement of the evaluation purpose. That is, what do we want to know from the evaluation and what evidence do we want this evaluation to produce? Being clear on the purpose of the evaluation will help in deciding what level/s of evaluation, or level of change your organisation is most interested in, as this will assist in deciding the frameworks most appropriate to draw on for your evaluation.

Four levels of evaluation are commonly used when assessing the value, merit, or worth of health and social care programs; however, it should be noted that across the literature other terms are often used to describe the levels of evaluation presented here. Each level of evaluation contains several key elements that can be investigated throughout an evaluation, depending on the program or service under consideration.

Levels of evaluation and program relationship

Evaluation Level	Evaluation purpose description	Program/project or service delivery component
Outcome evaluation	Determines whether the health or social service long term program or project goals/long term aims have been reached.	Goals/long term aims. The long-term measurable changes in a health, social or health service issue.
Impact evaluation Also known as summative evaluation	Investigates whether the health or social service program or project or intervention objectives/aims have been achieved.	Objectives/short terms aims. Short-term changes required to achieve the goals/long term aims.
Process evaluation Also known as auditing and monitoring; however, auditing and monitoring do not capture the full range of process evaluation activities.	Monitors the implementation of strategies. Strategy components assessed in process evaluation include: delivery (recruitment, fidelity, dose delivered) exposure reach dose received context.	Strategies/activities The actions and activities implemented to achieve the stated objectives/short term aims
Formative evaluation	Pre-tests strategies/activities before they are fully implemented. This is often undertaken with participants and partners.

 

Choosing evaluation indicators

Once decisions about the level/s of evaluation and the evaluation questions aligned with the level/s have been determined, indicators need to be identified. The United States Centers for Disease Control and Prevention (CDC) described evaluation indicators as:

Measurable information used to determine if a program is implementing their program as expected and achieving their outcomes. Not only can indicators help understand what happened or changed but can also help you to ask further questions about how these changes happened (CDC, n.d).

Indicators provide a framework for data collection within an evaluation plan. These data collected against each indicator allow evaluators to determine an answer to the posed evaluation question. For example, if a program has introduced a new model of care to reduce hospital readmissions, a logical indicator might be pre- and post-data on the number and nature of readmissions to the hospital. Indicators can be quantitative or qualitative, and both are often drawn on in an evaluation. Table 2 provides a detailed example evaluation plan for a program delivered with a residential aged care setting to increase physical activity participation among residents.

Study designs and data collection options

Following identification of indicators, study design/s and data collection methods for measuring the outcomes, impacts, and processes of an evaluation must be determined.

Study designs provide a framework and set of methods and procedures used to determine whether a change in an impact or outcome of interest occurred, and to what extent (Greenhalgh, 2019). Broadly, study design can be categorised as either experimental or observational and quasi-experimental. Experimental designs have clear strengths for determining the causal effects of interventions, yet are less frequently used in the evaluation of health and social care policies, services, programs, and projects. They are most common in large, well-funded evaluations, where the intervention is novel and/or has the potential for future use at the population-wide level (for example, randomised controlled trials (RCT) and cluster RCT).

More commonplace health and social care evaluations are observational and quasi-experimental designs. These include the pre- and post-test, post-test only, controlled pre-and post-test (without random allocation to groups), and time-series designs. To understand the alignment between evaluation level, indicators, study design, and data collection methods, refer to Table 2.

The health research methods literature contains extensive descriptions, applications, and discussions of study designs and methodologies, including the strengths and limitations of each option. It is beyond the scope of this chapter to address each individual study design and methodology available for data collection within an evaluation. Head to Better Evaluation, a global interdisciplinary network that provides practitioners and researchers with accessible evidence-based and robust evaluation resources, including examples of the application of paradigms, evaluation approaches, methodologies, and frameworks. The Evaluation Journal of Australasia publishes a wealth of interdisciplinary peer reviewed papers on evaluation theory, research, and evaluation practice.

Example Evaluation Plan Preview/Overview