Context Analysis: Need for a shift in Evaluation Paradigm

Context refers to the environment or background; the situations which form the setting of any idea, event or program. Every program would be working in a specific context, with regards to demographic factors, socio-cultural environment and existing infrastructure.

Experimental and Quasi-experimental studies provide information regarding attribution of the observed impact to the studied program. However, the basis of such analysis is the counterfactual. A counterfactual refers to what would have been, had the program not existed. To put it simply, a comparison between the project area and this counterfactual post the implementation of the program should give us a measure of changes brought about by the program. In essence, we assume that the context or the environment for our programme areas and counterfactual is the same and would also show a similar path of evolution over the period when the studied program is being implemented.

However, with differing demographic and socio-cultural factors, existence of several other similar programs and scaling up of the programs into non-project areas, the context within the programme areas as well as in the counterfactual or comparison areas is bound to evolve in differing ways. Thus a correct measurement of the context becomes very important. In cases where impact of the program is observed, it will be necessary to rule out the possibility that the same is due to contextual factors and not the studied project. Moreover, in case no impact is observed, evaluators might want to check if the contextual factors are masking the actual effects of the program.

Measurement of impact of the contextual factors becomes particularly important for the programs within the realm of the health sector. With the adoption of the Millennium Development Goals, the developing world has placed special emphasis on maternal and child health. There are numerous organizations and programs working towards development of maternal and child health, nutrition and water and sanitation. These areas of work are closely related. For instance, a WASH program which works towards improving behavior and attitude with regards to sanitation practices is likely to affect the outcomes of another program operational in the same region, which works towards diarrhea control. In another situation, presence of the WASH program in the comparison areas only is bound to modify the results of an experimental study for the diarrhea program.

While there does not exist any particular framework for measuring accurately the impact of the contextual factors for any program, there have been few studies which tried to capture the degree to which contextual factors modified the actual impact of a program.

A correct measure of the impact of the context on the outcomes can help us determine the isolated impact of the program under study. This becomes crucial when organizations are considering scaling up of successful programs. What if the success of the program in a particular area was majorly due to the interplay of context and the program? Such a program is likely to fail in a different region or in a different context. It thus becomes extremely important to consider context variables while conducting evaluation studies.