Use of mixed methods in long-term retrospective evaluation. We found the importance of a participatory
component essential to conducting analysis and framing narratives, though more in-depth qualitative
work could have strengthened findings. An important consideration in the original design of the
study was the need to go beyond the confines of program and intervention logic which had defined
development assistance throughout the 40-year period under review and understand from the outset
how people living in the area had themselves experienced and perceived change. The RCA findings
are important, in that they are the only source that provides an indication of the subjective experiences
and views of people living in the region, and provide often vivid accounts and impressions of
those experiences and views. But the fieldwork was limited in time, space, and scope and was undertaken
after the extensive documentary review. It was not possible therefore to feed the results of the
fieldwork into an ongoing process of data collation and quantitative analysis as part of the delivery
of other individual research components; RCA studies while insightful and relevant cannot provide
firm conclusions regarding the effects of development interventions.
Rather, RCA insights were used to frame causal narratives grounded in the experience and perceptions
of people living in the area rather than as a primary source of qualitative data for the analysis
as a whole. This approach provided insights into how people themselves viewed change,
removed from the usual externally imposed program or sectoral lenses. It was able to report on how
some people in the area thought change came about and their view of the interplay of outside interventions
and local knowledge, and the role of government, development partners and private sector,
as well as their perceptions of the varying significance of different factors. It was able to distinguish
differences between intended benefits and actual benefits and differences between stated change and
actual behavioral change.
A nonprescriptive method such as the RCA proved an effective tool for framing narratives and
illustrating the views and experiences of local people. It also provided an effective means of identifying
alternative avenues of investigation and inquiry, causing us, as researchers, to question previously
assumed truths or realities and return to the data with a more critical appraisal. However, it
does need to be combined more systematically with other sources of data to generate more rigorous
and generalizable findings.
Managing interdisciplinary teams. The management of the research process itself needs to be strong in
order to ensure integration (both coherence and continuity) and adherence to a clear program timetable.
The team involved must be able to function as a whole, with individual specialisms harnessed
to address a well-defined set of issues and produce a defined set of outputs.
Throughout the data collection and compilation process, several analytical ‘‘check-ins’’ were
conducted, particularly in sharing the results of the documentary review to frame the data collection
activities of the RCA study in an effort to encourage interdisciplinary exchange among researchers.
12 American Journal of Evaluation
Downloaded from aje.sagepub.com at Naresuan University on January 23, 2016
However, the extensive period of documentary review was carried out for the most part on a sectoral
or disciplinary basis, with individual members of the team tending to work on their own area of
expertise and sector rather than in a fully integrated manner. Furthermore, the description of the
major findings from the documentary review tended to precede the analysis rather than for the two
to proceed together. The analysis was, in effect, largely undertaken as a later phase in the project.
This led to a number of weaknesses in the study that should be considered by future researchers.
Effective working relationships, coordination, and opportunities to exchange and analyze data in
a sequential manner throughout the study are essential to success. Future researchers should consider
investigating organizational tools that can be used to improve this critical process, including the
right use of general and technical steering committees, or holding regular plenary sessions and/or
refocusing workshops at regular intervals with key stakeholder groups.