Sambodhi Blogs

Impact evaluation has become an important part of public-policymaking. Thousands of impact evaluation studies are conducted across the world every year, and most of them ask a straightforward question, “What really works?”. However, usually the question is followed by another one that “Is it meaningful to ask such a question? Is it answerable at all?” The realist evaluation approach tries to make the evaluation question comprehensive, and in that effort makes it answerable.

The Realist Approach

In 1997, Ray Pawson and Nick Tilley proposed the realist approach, urging the evaluators to ask a more comprehensive question. Pawson and Tilley argued that in order to be more useful for decision makers, evaluators need to ask, “What works, for whom does it work, in what respects, to what extent, in what contexts, and how?”. They drew ideas from the works of Karl Popper and Donald Campbell, two of the most celebrated thinkers of the modern age. Karl Popper’s ideas on ‘Piecemeal Social Engineering’ and his promotion of trial and error, learning to refine policymaking has significant influence on realist view. Similarly, they also drew upon Donald Campbell’s work on ‘Reforms as Experiments’. The basic premise is that reforms should be small and incremental, if they have to be effective as large scale interventions lead to disasters. After Pawson and Tilley, many interpretations of the realist approach has surfaced, however, the basic realist question remains the same.

The Realist Understanding of How the Programmes Work

The realist approach has a distinctive take on the way programmes bring about change. There are four basic assumptions about the nature of the programmes that realist evaluation is based upon.Firstly, programmes are theories that incarnate, multiple theories of change are at work for every programme and the effectiveness of the programme depends upon combined efficacy of these theories. The second assumption is that programmes are embedded into different layers of social and cultural reality, influencing and influenced by any changes in behaviours, events and social conditions. Thirdly, programmes require active engagement of those who are touched by them to effect changes. Finally, programmes are open systems and they cannot be isolated from its surroundings. Realists accept that externalities always impact on the delivery of a programme and this entails that they are never quite implemented in the same way.

For any programme, realists try to identify the generative mechanisms that bring about change. The next stage is to identify the contextsrelevant to the operation the programme mechanisms. Outcome-patterns comprise the intended and unintended consequences of programmes, resulting from the activation of different mechanisms in different contexts. The next step is to identify the appropriate Context mechanism outcome pattern configuration (CMOCs), indicating how programmes activate mechanisms amongst whom and in what conditions, to bring about change. The context-mechanism-outcome configuration (CMOC) is used as the main structure for realist analysis. It uses an iterative research cycle which tries to identify all the relevant CMOCs that are at work for the programme. The evaluation output are in terms of CMOCs, i.e. statements like “This has worked under these circumstances for these target beneficiaries in this way”.

The scope of realist enquiry is broad, and it is applicable to any social science discipline. It has an added advantage of flexibility. It may be used prospectively (in formative evaluations), concurrently (in summative evaluations) or retrospectively (in research synthesis). Moreover, it has no particular preference for either quantitative or qualitative methods. Usually, both quantitative and qualitative data are collected in a realist evaluation, often with quantitative data being focused on context and outcomes and qualitative data on generative mechanisms.

Realist Response to policy questions

Having outlined the basic principles of realist evaluation, one might ask what would be the response of a realist evaluator when posed with a tough policy question. If asked whether an intervention worked, the realist is going to say, “It depends”. It might sound useless as a response, but the realist will detail out everything it depends on, i.e. “What worked and for whom, in what respects, to what extent, in what contexts, and how? He is also going to provide the detail of what did not work, why it did not work and for whom it did not work. After the success of a pilot, the realist is not likely to recommend that the programme should go all out. The table below gives frequently occurring policy questions and typical realist responses to them.

The Policy Question

The Realist Response

Did that intervention work?

It depends (in what respects?)

Does that intervention work?

It depends on the conditions.

Does that programme work?

Parts only, in some places and at some times

Should we fund X rather than Y?

Check first to see if they are commensurable

The pilot was great, should we go large?

Unlikely, but you’d have to wait and see

Will it have a lasting effect?

No, play only to its strengths

Can you let us known before the next

spending round?

Sorry, not in all hone

Realist Approach: Pragmatism or Pessimism

Its reluctance to offer universal policy prescription on most of the issues may be termed as a defeatist approach by the host of experts. But before dismissing the realist approach, it should be kept in mind that there exists a very thin line between pessimism and pragmatism. To understand the realist’s refusal to scale up any intervention, we must trace back to its ideological roots, which may be found in the works of Karl Popper. Popper always favoured ‘Piecemeal Social Engineering’ in the form of small-scale interventions. It is no wonder that realists refuse to generalize their findings.

In a world where billions remain in abject poverty, there is a tendency among policymakers and evaluators to generalize the findings, especially when the findings are positive. Therefore, it is not hard to find instances of interventions which worked well in Asia but failed miserably when introduced in Africa. The realists advise caution before such mistakes, and promote a better understanding on how the programmes bring changes.