Clear-aa.co.za

Evaluations that make a
Evaluations that make a difference-EN-20Sep15.indd 1 21/09/2015 13:14:17

Evaluations that make a difference-EN-20Sep15.indd 2 21/09/2015 13:14:18

extent possible, from the perspective The story of this project
of the users and beneficiaries who have The issue of evaluation use is gaining been involved in the profiled evaluations What are evaluations that make a
currency, presuming a positive correlation – a perspective that, strangely, has between evaluation use and evaluation rarely been heard, even in discussions When we first set out to collect stories, little value. But on the fundamental issue of of evaluation utility. It is our hope that did we know how challenging it would be to determining the value of evaluation, the stories will inspire policymakers, describe what we meant by ‘evaluations that the literature has been strangely silent. managers, and programme staff to use made a difference'. What are the benefits? How are they evaluation by demonstrating the potential expressed? Can they be measured? Can We were looking for examples of evaluations benefits in a concrete and engaging way.
they be described in economic or other that have contributed to social betterment terms that make sense to citizens and to We have also analysed the factors that in some way. Many evaluations use policymakers? What factors contribute contributed to making the evaluations good (even innovative) methods, with to making an evaluation more or less useful, yielding insights about the participation by important stakeholders. In enablers of a valuable evaluation. We some cases, those evaluations get used to learned about specific actions evaluators, inform decisions about programmes and To help answer these questions, we've policymakers, managers, and programme policies. But for this collection, we were prepared this collection of short stories staff can take to enhance the benefits looking for something more. We were about evaluations from around the world of an evaluation. These actions are seeking stories describing how an evaluation that have made a difference to the lives of led to positive changes in people's lives. people. These stories collectively represent a range of approaches to evaluation, This proved a difficult concept to making it clear that there is no one grasp, so with the help of Chris Lysy of and hope that readers will be able to ‘right' way to do evaluation that leads to FreshSpectrum, w apply these insights to enhance the value improvements in people's explain the concept. of their evaluations.
lives.
This collection is intended for policymakers and other potential evaluation users, as well as for evaluators. The stories are told, to the Evaluations that make a difference-EN-20Sep15.indd 3 21/09/2015 13:14:18 the North American story, a process that • European Evaluation Society (EES) Potential stories were solicited through culminated in a set of story development • Canadian Evaluation Society (CES) an international call for stories between guidelines, which were then made available • African Evaluation Association (AfrEA) January and May 2014. There was a high to the other regions.
• Australasian Evaluation Society (AES) level of interest, with many inquiries and a The completed stories were professionally • Red de Seguimiento, Evaluación y total of 64 submissions. The Editorial Board edited and then translated into English, Sistematización en América Latina y el then assembled Regional Review Teams to French and Spanish.
review the submissions.
• Sri Lanka Evaluation Association (SLEvA) The main selection criterion used by reviewers was evaluation impact – Spark and core funding
The Editorial Board provided strategic evidence that the evaluation led to positive This project was supported through an oversight for the project, advising about changes in people's lives. Reviewers also EvalPartners-Innovation Challenge grant, the selection criteria, the story formats, etc. looked for stories that would provide which aimed to strengthen the demand They leveraged their own networks to get maximum opportunities for learning, as for and use of evaluation to inform policy the word out about the project, to solicit well as submissions that would simply be making during the International Year of submissions, and to oversee the regional engaging as stories.
Evaluation (2015).
review process. As a group, they made the The Regional Review Teams each chose up We are grateful to Martha McGuire for final story selections. They then recruited to three submissions to recommend to the encouraging us to take on this Innovation local story writers and oversaw the writing Editorial Board. From these, the Editorial Challenge, to Jim Rugh for his ongoing process. Members of the Editorial Board Board agreed on eight submissions to support and guidance throughout, and develop into stories, including at least one to the European Evaluation Society, and Burt Perrin (co-chair) – EES
from each major region of the world. in particular Eva Petrová, its Secretariat Manager, for significant administrative Rochelle Zorzi (co-chair) – CES
The Editorial Board then recruited a professional story writer for each Pablo Rodriguez-Bilella – ReLAC
story. The writers reviewed the original This project was a strategic partnership of submission documents and interviewed evaluators from all major regions of the Scott Bayley – AES
key stakeholders in the evaluation (usually world. The partners were: about five). This process was piloted with Evaluations that make a difference-EN-20Sep15.indd 4 21/09/2015 13:14:18 Serge Eric Yakeu – AfrEA
Developing and piloting the story
Spanish translation was done by Pablo Rodriguez-Bilella and reviewed by Ramon Soma De Silva – SLEvA
The story development guidelines were created by a subcommittee that included Project coordination, editing and
Yasser Ismail (chair), Dayna Albert, Jessica Regional Review Teams
Sperling, and Rochelle Zorzi. This group Finally, we wish to recognize the Each Regional Review Team reviewed the worked closely with Stephanie Potter and contributions of two individuals who story submissions from their regions and Sara Pederson, members of the North played key roles in the project. The recommended several to the Editorial American story team who piloted the project. first of these is our editor, Eric McGaw
Board. Members of the Regional Review ([email protected]), who took The African Development Bank provided responsibility for the formatting of the AfrEA: Serge Eric Yakeu Djiam (chair),
funding for French translation of the final collection of stories (in all three Samuel Kouakou, Ousseni Kinda, Awuor evaluation stories, lessons learned and languages), in addition to editing the Ponge, Djelloul Saci project materials.
English version.
AES: Scott Bayley (chair), Vanessa Hood,
The Inter-American Development Bank Last but certainly not least, this project April Bennett, Kim Grey, Jessica Kenway provided funding for Spanish translation of could not have been completed without the ongoing coordination provided by EES: Ramon Crespo (chair), Burt Perrin,
the evaluation stories, lessons learned and Dayna Albert ([email protected]),
Maria Bustelo, Murray Saunders project materials.
who managed just about everything that CES: Rochelle Zorzi (chair), Pierre-Marc
Teleconferencing costs were contributed in- needed managing, from the distribution Daigneault, Tracy Fiander Trask, Marie kind by Cathexis Consulting Inc.
of the call for proposals through to signing Gervais, Kathryn Parker, Hallie Preskill off on the last of the translation contracts.
ReLAC: Pablo Rodriguez-Bilella (chair),
French translation was done by Edmond Both Dayna and Eric went above and Thomaz Chinanca, Ignacio Irarrázaval, Kembou and reviewed by a committee of beyond the call of duty, and their Claudia Maldonado, Inka Mattila, Rafael volunteers, including Pierre-Marc Daigneault efforts and commitment are very much (chair), Ann Royer, Raimi B. Osseni, Helene SLEvA (Asia): Soma De Silva (chair),
Wirzbah, Ruth Chamberlain, Claire Bard, Shobhini Mukerji, Ada Ocampo,Mallika Véronique Dugas, Nora Habafy, and Alexandre Evaluations that make a difference-EN-20Sep15.indd 5 21/09/2015 13:14:18 What these stories tell us about collaborative effort.
Ways to enhance evaluation impact
how to do evaluations that can
How the evaluation make a difference
is undertaken is a big part of it. These stories collectively have a lot to say Evaluators can.
about how evaluations can be undertaken • Ensure users Evaluation users can.
so that they are most likely to have a right, there are evaluation impact • Sincerely care positive impact on people's lives (what Mel several things • Give voice to beneficiaries are Mark, Gary Henry, and George Julnes have evaluators can do described as social betterment1). to enhance impact. • Provide credible • Champion the As we collected and developed the • Use a positive stories, the Editorial Board had several users view, support, opportunities for rich dialogue about the factors that enabled these evaluations to make a difference. The following ideas are based on an analysis of themes in these In the following sections, we II. What evaluators can do
stories, discussions among members elaborate on what evaluators 1. Focus on evaluation impact
of the Editorial Board, and interactive and evaluation users can do to sessions at the 2014 European Evaluation influence evaluation impact, and The focus of this project, and of the stories included, Society conference in Dublin and the 2015 how interlinkages between the has been on evaluations that could be shown, in Canadian Evaluation Society conference in programme and the evaluation some way, to contribute to changes in people's Montreal where preliminary stories were can support evaluation impact. lives – to ‘social betterment' or ‘impact' as it is most shared and discussed.
The distilled success factors typically referred to in the evaluation literature. This described here are not intended is the raison d'être of evaluation, right? Otherwise, I. Evaluators and evaluation users both
to represent a definitive list. what is its value? play important roles
Instead, they are meant to It is important to appreciate that evaluation impact, Doing an evaluation that leads to provide a starting point for as we've defined it, goes beyond ‘utilisation' and improvements in people's lives is a discussion as we build a collective more immediate or intermediary outcomes such as understanding of how evaluations 1 Evaluation: An integrated approach for understanding, changes to policies or programmes. While utilisation guiding, and improving policies and programs. Jossey-Bass can make a difference.
is, of course, important, it must not be the end goal. Evaluations that make a difference-EN-20Sep15.indd 6 21/09/2015 13:14:18 In the end, it is important that the changes work an intention to improve people's lives, this evaluation, which for the first time influenced by evaluation eventually lead and work with evaluation users and other gave voice to people across the country, to improvements in people's lives, as the programme stakeholders to structure the many of whom only spoke their local diagram on page 2 illustrates.
evaluation accordingly. Evaluation users can language and could not benefit from the Nevertheless, a major, indeed startling, help by outlining the desired evaluation radio programmes until the evaluation finding is how difficult it is for evaluators impact in the terms of reference.
identified the need for these to be offered to think in terms of impact. Even as they 2. Give voice to the voiceless
in local languages and tailored to local exhort programmes to think beyond initial contexts. The evaluation led to significant In many instances, those who make outcomes, evaluators find it difficult to changes in the programme, and in turn to decisions about programmes or policies do the same for their own work. Indeed, improved lives (such as reductions in child do not have an opportunity to observe the despite clear guidelines, few of the 64 labour or girls being forced to marry at a needs or contexts of the beneficiaries. They submissions to the project even attempted very early age).
are making their decisions, with the best of to identify a connection between the intentions, based on limited information and The Mexico story similarly indicates evaluation and benefits to people.
assumptions that may be incorrect. how the evaluation identified language We found that some evaluators resisted the barriers that prevented very poor In these situations, evaluation can have notion that they should think about how, or indigenous people from being able to a transformative impact on the lives of if, their work has contributed to changes in benefit from a programme. Changes beneficiaries by giving them a voice, and people's lives. ‘That's not my responsibility, to the programme allowed for carrying their words to decision-makers. that's up to someone else,' they suggested. communication in local languages, which Many of the stories in this collection ‘I carried out a "good" evaluation [perhaps greatly improved people's ability to used evaluation approaches that enabled one that was methodologically rigorous, understand the programme requirements beneficiaries to have their voices – and their or one that influenced what programmes (such as children's regular attendance needs – heard for the first time. were doing]. I can't be held accountable for at school), and thus benefit from the anything beyond that.' This collection includes some dramatic programme's cash transfers.
examples of evaluations that led to Yet as the eight stories in this collection 3. Provide credible evidence
significant changes and improvements in the illustrate, evaluation can have positive lives of people, simply by letting people who As many of these stories demonstrate, impacts on people's lives, and there are were typically unheard tell their stories.
evidence must be seen as credible things that evaluators can do to enhance in order to bring about necessary (or, on the flip side, to diminish) those The title of the Nepal story: Listening to the changes that result in improvements in impacts. Evaluators can bring to their listeners, says a lot about the approach of people's lives. These stories collectively Evaluations that make a difference-EN-20Sep15.indd 7 21/09/2015 13:14:18 indicate that there is not just one right stories from Canada and the Netherlands, the need for an expanded childbirth way to generate credible and persuasive evaluation users' involvement in the emergency phone service. The evaluation information. What is seen as ‘credible' evaluation process helped them relate to of the Positive Sisters programme in evidence depends on the situation and the the findings, making the findings more the Netherlands validated many of the particular actors.
meaningful and easier for them to act approaches of volunteers and staff, In some cases, it was important to assess encouraging and indeed energising them to programme outcomes with perceived 4. Use an approach that supports positive carry on with those approaches that were
thinking and action2
shown to have a positive impact.
In the story from Papua New Guinea, ‘hard' These stories also illustrate the value of III. What evaluators and evaluation users
data about lives saved by a pilot childbirth a positive focus. Documenting what does can do together
emergency phone service was an important and can work and should be continued, 5. Ensure users and intended beneficiaries
factor in convincing the authorities to expanded, or modified, rather than are engaged through a participatory
invest in the continuation of this service on focusing mainly on the inevitable glitches approach to evaluation
an ongoing basis.
and shortcomings, avoids the quagmire of Evaluators often try hard to keep their In other situations, a credible evaluation blame and defensiveness and moves the distance in order to guard their ‘objectivity'. involved capturing the perspectives of discussion into solutions. The experience of The danger of evaluation done in this these stories is consistent with the findings way is that it may become remote from from the evaluation literature: evaluation The Mexico story for example, illustrates – and irrelevant to – those who would is much more useful when it can provide how a qualitative evaluation approach act upon the evaluation findings to make evidence about approaches that do work was needed to demonstrate how the improvements and to bring about change. and need to be continued, expanded, or Oportunidades programme, due in large adapted. While many of the stories in the By keeping their distance from the part to language barriers, needed to collection have documented the need programme and its people, evaluators also change to address the needs of very poor for change, they all have gone beyond pass up valuable opportunities to make indigenous people (something that was this, documenting what form of change is a difference through process use. The not apparent in previous quantitative needed, and why.
evaluation literature demonstrates, very forcefully, that benefits follow as much, For example, the story from Papua New In still other cases, credibility was achieved or even more, from how the evaluation is Guinea illustrates the value of documenting through active engagement of the users in done, as from the evaluation findings.3 2 This is the theme, for example, of a special section and the evaluation process. For example, in the collection of articles in a recent issue of the Canadian Journal 3 For example: J. Brad Cousins, ed., Process use in theory, of Program Evaluation (vol.29, no.2, 2014).
research and practice, New Directions for Evaluation, No. Evaluations that make a difference-EN-20Sep15.indd 8 21/09/2015 13:14:18 By being involved in an evaluation, people about her HIV/AIDS status before ShivA, and Similarly, the collaborative approach taken can gain insights, get to know others, begin now recognises that being interviewed was in the Kenya community sanitation project to see things from a new perspective, and an extremely positive experience. ‘Being led to the community taking over the strengthen their abilities. The accumulation interviewed helped me feel important and evaluation, reflecting their belief that if of these changes, among a group of people, part of something good.' positive action was going to take place, it can have a strong positive effect that is has to ‘come about from the grass roots, For Liako, being interviewed meant quite independent from any evaluation not from outsiders.' becoming more aware of the impact of her work since ShivA: ‘The evaluation had a big Importantly, active engagement in the One way to engage users and beneficiaries impact on me. It was a Wow! moment for evaluation process helps develop a in an evaluation is by having them tell their me. We are really doing a great job.' better understanding of evaluation, and stories. By sharing stories with one another, contributes to commitment and buy- Community participation in gathering and people form relationships, strengthen in. Those who have been involved in an using evaluative data is another powerful networks, and set up informal knowledge evaluation have more commitment to act way of engaging users and beneficiaries. transfer channels that need not go through upon its findings and implications. The story from Canada and the Kenya ‘experts'. In addition, they can also develop community sanitation story demonstrate In summary, these stories collectively a sense of community and a feeling of the transformative impact of community demonstrate that participation yields being part of something larger than involvement in data collection. In both strong benefits, regardless of what form cases, the communities felt increasing it takes. This requires an openness by The evaluation of the Positive Sisters ownership of their data, which empowered evaluation users to actively engage programme in the Netherlands not only them to take charge of and make use of in evaluation, and it also requires a provided direction to the project to willingness of evaluators to adopt increase its impact; the process energised approaches to evaluation that allow for As the evaluator in the Canadian story its volunteers and participants in a way that active engagement and participation. observed, ‘Whose project is it? Whose data? traditional approaches to evaluation rarely There has to be room for the people who 6. Embed evaluation within the
are affected. It's not just the government's Millie, for example, had been quite shy story.' In this example, being involved in the An interesting characteristic of three of 116, Jossey-Bass, 2007.
data collection gave nurses the evidence the stories is that evaluation and the For broader consideration about participatory approaches they needed to ‘overhaul the community's programme were so closely linked that it to evaluation: Elisabeth Whitmore, Understanding and prenatal and postnatal health care, setting sometimes was difficult to say where the practicing participatory evaluation, New Directions in Evaluation, No. 80, Jossey-Bass, 1998.
the stage for a healthier future.' programme stopped and the evaluation Evaluations that make a difference-EN-20Sep15.indd 9 21/09/2015 13:14:19 began. Both stories from Kenya featured time goes on, evaluation can help users sustainably funded by the provincial health evaluations that were embedded in the find out what effects the changes had, authority Naidi headed, he needed proof.' and further (better-informed) refinements In the story from the Netherlands, we The Community and Progress Youth can be made if needed. These stories also heard that ‘Inga was hungry for insights and Empowerment Institute (CAP YEI) made use illustrate how evaluators often worked feedback and quick to apply them.' of internal and external evaluation to aid in very closely with programme staff to And in the story from Mexico, the ongoing improvements to the programme, support implementation of the findings.
coordinator of the Oportunidades contributing to better employment IV. What users can do
programme, Salvador Escobedo, despite and financial outcomes for its youth 7. Really care about the evaluation
having intimate knowledge of the indigenous Sometimes, people do evaluations only communities and their reality, knew that A more dramatic example is the community to fulfil accountability requirements. They without evidence that both the language sanitation project in the village of Murihi go through the motions, get the findings, barrier and the targeting procedures wa bibi in rural Kenya. The evaluation was, and submit the report to their funder. were serious impediments to successful in truth, part of the intervention and not The report sits on a shelf unused, and the implementation of the programme, he would a separate activity, and it played a key role programme continues on undisturbed.
be unable to make and sustain the needed in achieving a dramatic reduction in open changes in the programme. defecation practices. As part of the project, That is not what happened with the This wanting of the evaluation, we believe, members of the community gathered to evaluations in this collection of stories, made it possible for the evaluation to make analyse and interpret the data to assess where someone really wanted the a difference in people's lives. As Michael their progress. Their day-to-day interaction evaluation for the learning and guidance Patton4 has indicated: with the evaluation findings inspired a spirit that it could provide. Each evaluation was of friendly competition, so that people valued because it enabled users to make The personal factor is the presence of an soon wanted to achieve open defecation better decisions or advocate for change. identifiable individual or group of people who personally care about the evaluation free status before their neighbouring Here are just three examples from among and the findings it generates. Where such a person or group was present, evaluations were When evaluations are designed to collect In Papua New Guinea, Billy Naidi had used; where the personal factor was absent, research on evaluation use consistently shows a data and feed results back on a regular heard positive things about the emergency correspondingly marked absence of evaluation basis, stakeholders can make changes phone service, but he ‘needed some long before the final report is written (if hard evidence that the emergency line indeed there even is a final report). As was working. For an initiative to become 4 Developmental evaluation: applying complexity concepts to enhance innovation and use, Guildford Press, 2011, p.56.
Evaluations that make a difference-EN-20Sep15.indd 10 21/09/2015 13:14:19 8. Champion the evaluation with decision
impoverished households. The story tells us 1. What is needed to create enabling that it is ‘nothing less than astonishing' that environments where high-quality credible Programme staff and managers who have they were successful. ‘Never before had evaluations influence decisions at all been involved in the evaluation might this government body granted permission levels? Future work in this area might be committed to making changes to the to anyone to access timber.' Such is the build on what is already being done with programme. But often, the decision makers power of having committed evaluation parliamentarians5 as well as the efforts of with real authority are external to the evaluation associations to educate both programme and haven't been engaged in the The champions in these stories were evaluators and users.
evaluation process.
people who cared passionately about the 2. What are the expected competencies What happens when evaluation users need families and communities affected, and evaluators need to make a difference in to convince other stakeholders that a change who had influence with those who could people's lives? For example, maybe it is make decisions. And they clearly played time for the discipline to make an effort an instrumental role in getting changes to better describe what is really meant by Well, that's when it's helpful to have a made. The stories do not suggest that the upholding democratic values in conducting champion who can influence key decision evaluators deliberately cultivated these and reporting evaluations (a widely makers and push for the needed changes. We champions, but it may well be worthwhile recognized disposition for an evaluator can see the power of champions in several of for evaluators and users to identify and collected in several ‘evaluation capability the stories.
nurture such champions more intentionally. frameworks') that goes beyond but In Sri Lanka, the secretary of the small V. Moving forward
certainly includes the need for evaluators enterprise ministry ‘seized on the findings… to take an impact orientation to their work He took the evaluation to all relevant cabinet These stories are just the beginning of and to facilitate good communication ministers to secure their support. With the understanding how evaluations can make a and indeed engagement of stakeholders cabinet on side, there was no way the issues difference. Beyond the 2015 International throughout the evaluation process.
raised by the evaluation could be ignored.' Year of Evaluation, it will be important to This paved the way for significant changes continue the research and test some of the to the Industrial Estates, which improved theories that have emerged through the conditions for small and medium businesses. stories in this collection.
In Kenya, monitoring committees brought their progress data to forest authorities We pose two questions to provide to request timber to build latrines for inspiration for future efforts: For example, through the Global Parliamentarian Forum for Evaluation that is being formally launched in November 2015 at the EvalPartners Global Forum (www.pfde.net).
Evaluations that make a difference-EN-20Sep15.indd 11 21/09/2015 13:14:19

SRI LANKA
Salvaging Sri Lanka's small and
some more than doubling year after R.S. Balanathan, the managing director year. And even when rents were paid, of handloom silk exporter Ko Lanka, says how an evaluation led to rapid change
there seemed to be no correlation to the all the industries in the estates were quality of infrastructure. Roads were more undeveloped. ‘The roadways were bad, Since the end of Sri Lanka's long civil potholes than asphalt, electricity was and the rents were very high,' he says.
war in 2009, the nation's economy has unreliable, and coordinated marketing recovered at a startling rate. GDP growth simply lacked focus. Although the In 2013, the situation began to change. The now hovers between 6% and 8%. Peace government was investing heavily in Ministry of Traditional Industries and Small has certainly brought dividends – tourism national infrastructure, smaller businesses Enterprise Development commissioned a has rebounded, China has made large appeared to have been overlooked.
comprehensive evaluation of the sector in investments in infrastructure, and trains once again run the length of the pearl- shaped island in the Indian Ocean, just off the southern coast of India.
But it's not all good news. As peace returned, small and medium enterprises (SMEs) found to their chagrin that China, India, and Bangladesh had all but cornered the world's export markets in mass manufacturing and textiles – at their expense. To add to their woes, the national government had paid little attention to the sector over the years. For the hundreds of business owners renting land in the 18 government-owned industrial estates managed by the Industrial Development Board (IDB), the economic boom seemed to have all but passed them by.
The list of complaints was long and detailed. Rents were out of control, with Handloom industry at Pallekelle Industrial Estate
Evaluations that make a difference-EN-20Sep15.indd 12 21/09/2015 13:14:21

rural poor – who provided much of the workforce in the estates – also suffered.
Silk showroom at Pallekelle
The evaluation report gave voice to longstanding grievances and recommended sweeping changes. What was needed, argued Ekanayaka, was direct intervention by the national government. The secretary of the small enterprise ministry, Velayuthan Sivagnanasothy, seized on the findings. He The evaluation showed that the IDB was a mere rent collector and not improving the estates. With an evaluation like this, change would be far easier to generate. The evaluation process was a platform serving to carry the voices of the voiceless to the highest level of policy makers.
He took the evaluation to all relevant 2013. A team of independent evaluators broad problems that came up again cabinet ministers to secure their support. led by Ajith Ekanayaka began travelling to and again were decaying infrastructure, With the cabinet on side, there was no way each of the 18 estates across the country exorbitant rents, and expensive loans. the issues raised by the evaluation could be to examine the issues plaguing the sector Some had special concerns. Metal and to suggest direct solutions. Ekanayaka industrialists, for instance, could not lay And so it came to pass. The ministry drafted made it clear that this evaluation would their hands on raw materials because a national policy on SME development, not be relegated to a filing cabinet but almost all scrap metal was being exported. consulting widely and drawing on the would be put to immediate use.
The evaluation also found that some expertise and advice of other ministries For five months, the evaluators gathered estates were being used to build housing and agencies. But even before the policy their material, forming focus groups for by the owners of failed SMEs. The long- could be adopted, change began to ripple industrial associations and interviewing term weakness of the sector meant that outwards. The evaluation's actionable managers and district officers. The three employment for ethnic minorities and the recommendations had buy-in. Why? Evaluations that make a difference-EN-20Sep15.indd 13 21/09/2015 13:14:22

Shoes at Baddegama Industrial Estate
Aluminium ware at Kaludwela Industrial Estate
Because policymakers had given frequent periods meant that SMEs could now expand in order to shift from a reactive to a input through scoping sessions, and the their operations more easily. In each of proactive stance. The IDB now focuses evaluation was built on a broad, credible the estates, factory production is up, and on training, access to raw materials (such base of evidence and data as well as strong with it, SME profits, leading to more jobs. It's trickled down to the rural villages.
as previously scarce scrap metal), and connections to stakeholders. According access to credit. Not only that, but the to Sivagnanasothy, a great deal has come A tangible shift in attitude amongst IDB rental income earmarked for infrastructure from this single evaluation. He recalls: management towards the SMEs was improvement around the estates was confirmed by Ratnamalala, who praised When the rents went up, industrialists increased tenfold, from as little as 2–3% used to shout and cry How can we afford the evaluation for what it revealed. percent to 20–30% of the rent paid.
it? We were able to convince the central ‘We expanded our service delivery and To address the problem of access to government to cap rent increases at 7% created a shift in mindsets of our officers annually. That's a predictable environment, to be more development oriented.' credit, the government rolled out a special which is an enabling environment. Not The shift for the IDB was significant, with scheme with concessionary rates to boost only that, but the introduction of targeted the agency undergoing a restructure access. Joint marketing schemes were loans with lower interest and longer grace introduced to give potential buyers a Evaluations that make a difference-EN-20Sep15.indd 14 21/09/2015 13:14:22 streamlined way of examining the range million), with more on the way.
For silk exporter Balanathan, the result and quality of products. Ailing SMEs can For IDB director general Ratnamalala, has been more cash in his pocket – and now gain access to a ‘nursing programme' another benefit of the evaluation is simply that means expansion into his key markets to bring them back to health, while the being heard. ‘We have direct access to in Europe and the Middle East. From 100 rules for gaining entry have been made central government,' he says. ‘Now, with employees in 2013, he now employs 130. more stringent to cut down on residential these improvements, we will see growth in ‘We have strong competition with China use of industrial estates. And the metal our industrial estates.' and Bangladesh,' he says, ‘but these industries were given a lifeline at the On the ground, results of these rapid changes make us more competitive. Now, expense of the scrap merchants, with a evaluation-led changes have been we have higher silk production than they ban on the export of scrap. The amount impressive. The voices of small and do. We can beat them!' injected into infrastructure upgrades medium industrialists have been heard, immediately following the evaluation was and the lines of communication with Velayuthan Sivagnanasothy,
305 million Sri Lankan rupees (US$2.3 government are open.
Secretary, Ministry of Traditional Industries and Small Enterprise hotmail.com) commissioned the evaluation of the SME Industrial Estate Programme and used the evaluation recommendations that led to policy changes and programmatic actions that made changes in people's lives.
Ajith Ekanayake, Independent M&E
Consultant, led the evaluation team.
Vaidehi Anushyanthan, Assistant
Director, Ministry of Finance, co- chaired the evaluation as focal point of the Government's National Evaluation Department.
Story writers: Jessica Kenway
and Doug Hendrie.
Newly constructed road at Pallekelle Industrial Estate. No potholes!
Evaluations that make a difference-EN-20Sep15.indd 15 21/09/2015 13:14:24 KENYA
Tumekataa kula mavi tena!
Let us step back a few years to World Toilet We refuse to eat shit!
Day in 2011. We're in Murihi wa bibi, a village in the highlands of Kwale county Of the world's seven billion people, only along Kenya's low-lying coastal strip. 4.5 billion have access to toilets or latrines. Small children chase cockerels, which in The remaining 2.5 billion, most of whom a few hours will form part of the meal for live in rural areas, lack proper sanitation. the celebrations. Excitement is in the air. And nothing spreads disease faster than Why are the villagers of Murihi wa bibi open defecation. Indeed, Millennium celebrating? Quite simply, they are proud Development Goal Number 7 is to halve and exultant that there are no longer heaps the population of people living without of human excrement in the bushes. The adequate sanitation. Hence the idea for community here, convinced that their age- World Toilet Day, which takes place on 19 old tradition of shitting in the bush can no November each year. longer be tolerated, has achieved a new level of freedom – freedom from disease.
The practice had to stop; there was Field monitoring at Ngerenya village, Kilifi county.
Njoroge (in striped polo shirt) with Prof Robert
continue. Not with Chambers of IDS Sussex and other colleagues.
such champions as practice open defecation in areas used Kingi Mapenzi, Peter for living and farming. Kingi and Mbith are local community health workers Josephine Mbith, who while Mwambaka is the local government went from household administrator. The three of them are typical of the network of local volunteers who reminding residents work tirelessly to convince villagers to stop of the need to stop open defecation.
eating shit, which is what happens in Not that they are the first to do this. CLTS follow-up in Mazumalume village, Kwale county. Kingi Mapenzi is on
the far left.
communities that Sanitation has long been recognised as Evaluations that make a difference-EN-20Sep15.indd 16 21/09/2015 13:14:24 behind – to let The Alliance raises funds from a the communities membership of 10 core individuals. These individuals are regular employees of various organisations and agencies. efforts to solve Although their expertise ranges from monitoring and evaluation to epidemiology, the common denominator is community health. Acting as individuals, each of the 10 members takes on remunerated consultancies from which they donate time and funding to Alliance activities. And since they work through and with community Health Advocates groups, the costs are minimal.
It all started in 2007. An international Sanitation Monitoring Committee, Kafuduni village, Kwale county
NGO had triggered the project at Jaribuni a serious health problem in Kenya. For a legal entity with the government, but village in Kilifi, a neighbouring county years, dozens of local and international from the beginning the idea was to eschew north of Mombasa. During the triggering organisations have attacked the scourge branding and notoriety. The thinking session, a skilled facilitator, using various of open defecation from a variety of was: If sanitation is going to work, it has directions. The results have been varied, to come about from the grass but more often than not, the daunting task roots, not from outsiders. After confronting these organisations put paid to all, villagers don't care about their good intentions.
the names of organisations. Realising that the task was simply beyond To emphasise the primary the reach of any single organization, role of the community in this a number of concerned individuals venture, the alliance adopted decided to pool their efforts – and more the term community-led total importantly, to position themselves sanitation (CLTS) from a project as partners, not leaders, of the target in Bangladesh, where CLTS had Usinye msituni – ‘Do not
proved successful in addressing shit in the bush'
communities. The idea was to lead from open defecation. Evaluations that make a difference-EN-20Sep15.indd 17 21/09/2015 13:14:24 achievement, the residents and volunteers Monitoring is essential if a triggered village organized a simple ceremony. They invited is to attain ODF status. This is a lesson local health officials, who were extremely that CLTS practitioners in Kwale and Kilifi impressed by the simplicity of the approach counties learned the hard way. Initially, – empowering communities to analyse they thought that a triggered village would their own sanitation profile and to make automatically translate into ODF status. decisions based on the realisation that they With time, the importance of monitoring were literally eating each other's shit due – and monitoring committees – gained to open defecation. And the cry went up: recognition. NGO workers designed Tumekataa kula mavi tena! (‘We refuse to monitoring tools that the committees could eat shit any longer!' in Swahili).
use to gauge progress and identify which As the CLTS movement spread across households required special attention.
Kenya, especially the coastal region, Through this process, female-headed communities remained at the forefront. households and households with very old It doesn't work any other way. Passionate people or people living with disabilities community workers like Kingi, Mbith, and were identified. In many places, the Mwambaka, who tirelessly monitored the monitoring committees mobilised young situation every day, provided the spark that people to contribute labour by constructing participatory methodologies, led the ignited the process. But they knew that to toilets for households with occupants who villagers to understand the terrible achieve total sanitation consequences of open defecation. The the community had residents of Jaribuni collectively resolved to to truly embrace the stop open defecation by building and using idea. By explaining why latrines in a time-bound campaign led by a ODF was necessary, local committee. They gave themselves 90 they provided the days to have everyone in the village using a spark. But once the latrine and the local monitoring committee engine was running went around documenting the progress.
with the community in The village achieved open defecation the driver's seat, their free (ODF) status just 67 days after the job was confined to triggering session. To showcase this Recognizing the monitoring team.
Evaluations that make a difference-EN-20Sep15.indd 18 21/09/2015 13:14:24 lacked mobility. In cases where a household forget to wash their hands after latrine could not afford construction materials, the use, introduced hand washing facilities committee approached forest authorities with soap or ash next to the latrine. Rival with the progress data to show that these villages insisted on assigning participants few households required special support to in the verification missions sent out to access such construction materials as logs. inspect communities for ODF status. They That they were successful in this endeavour even visited the bushes previously used for is nothing less than astonishing. Never open defecation to verify the absence of before had this government body granted faeces. In one case, where all households in permission to anyone to access timber. a Penda Nguo, a village in Kilifi county were Such is the power of placing data in the found to have latrines, the sanitation and hands of aggressive community members! hygiene promotion experts had decided Once a village attained ODF status, amongst themselves to confer ODF status the residents, to show pride in their on the village. But when a member of the achievement, erected signposts proclaiming committee from a rival village discovered Administrator Peter Mwambaka addressing ODF
their achievement to the whole world a pile of fresh faeces in the bush, the celebrations at Muririhi wa Bibi village.
and warning visitors that open defecation committee was left powerless to act. The celebrate their collective achievement. It would not be tolerated in their turf.
villagers eventually decided to build a public latrine and they were eventually was the first time that such a large number Villages even started competing with declared ODF when no more signs of fresh of villages would celebrate collectively. CLTS one another. To outdo each other, they faeces were found. These are not the practitioners call it ‘strategic noise'. When a kept raising the bar. They said that the actions of complacent people.
village – in this case seven villages – come monitoring indicators put together by together to formalise their refusal to eat hygiene promotion experts were not Without the active participation of shit, their neighbours have no option but comprehensive enough to truly stop communities in monitoring total sanitation, to follow suit – to come and hear about the residents from eating shit. They pointed the hard work and dedication of health achievement and join in the strategic noise out that merely building and using a workers would result in nothing but tired of celebration.
latrine was inadequate. One community bodies and disillusionment.
No less than the regional director of health fabricated aperture covers to prevent Which brings us back to World Toilet Day is invited to witness their joy of saying no flies from breeding in the pits. Another 2011. A total of seven villages of Kwale to unsanitary living. The strategic noise community, to ensure the users did not county agreed to meet in Murihi wa bibi to includes testimonials from villagers. Mzee Evaluations that make a difference-EN-20Sep15.indd 19 21/09/2015 13:14:24 Hamadi recalls how squatting out in the bush robbed him of his dignity. The fear of stepping on snakes at night and the nightmare of finding a dry spot in the grassy patch during the rainy season was too much for him. When the movement swept his village, he was more than glad to join and stop open defecation.
His neighbour, Yusuf Ali, tells a different story. ‘At first,' he said ‘I was not so glad. I didn't see the importance of wasting effort and resources to build a house for shit.' However, his wife noticed that their 2-year-old daughter Fatuma, who suffered chronic diarrhaea before Yusuf reluctantly built the family latrine, was now active and healthy. For the last 3 months, she informs As a roving community health worker Celebrating ODF achievement
the cheering gathering, Fatuma has not had he knows his energy will be tested in diarrhaea. Even the health workers in the Katangini, a village on the other side of the Mazumalume dispensary say they miss her hill. But for now, he can relax and enjoy a because she no longer has to visit them. Redempta Muendo, Kwale
sumptuous meal of pilau and chicken stew. County Public Health Officer Siku hizi, twaenda hospitali kwa chanjo za Remember those cockerels? Haron Njiru, Programs
kakake mdogo. (‘These days, we only visit The revelry is well deserved. And what's Director, HEADS Alliance the clinic for immunising Fatuma's little more? Unlike the last 2 years when Kingi Kingi Mapenzi, Peter Mwambaka, and
had to travel to the next county, this year's The ceremony ends up with awarding Josephine Mbith: on the ground
World Toilet Day celebration is happening of certificates to the ever-committed right here in his village. What an honour, Story writers: Njoroge Kamau
monitoring committee. Expressing his what a great way to put the finishing and Eric McGaw
excitement, Kingi exclaims ‘Even my most touches on a successful year – and to look troublesome set of villages are finally ODF!' forward to an even better one! Evaluations that make a difference-EN-20Sep15.indd 20 21/09/2015 13:14:25 PAPUA NEW GUINEA
Evaluation in action
of 7 million, a third of whom The Milne Bay emergency phone service
are females of child-bearing age, are isolated from the major For expectant mothers in the mountainous population centres. Rural health Pacific nation of Papua New Guinea (known care workers do their best, but popularly as PNG by its inhabitants), if something goes wrong, they giving birth is an exercise in luck. If your may not know what to do.
labour is short, and you experience no The seriousness of the situation complications, luck is in your favour. But if is illustrated by PNG's maternal luck is against you, if your labour is difficult, mortality rate, which is among if the baby is stuck, if you bleed too much – the highest in the world. For you can only hope that you and your baby every 100,000 live births, 733 will survive. Because you are very, very far women die. By comparison, in from the nearest hospital.
Australia – the country's nearest Amanda Watson and Gaius Sabumei being interviewed about
the project at the local radio station in Alotau.
For the tens of thousands of mothers who neighbour – the death rate is only 6.8.
and expertise to repair them were lacking. go into labour in remote villages each year, The Childbirth Emergency Phone These were uncharted waters for PNG. babies are born at home or at the local programme was designed to tackle this Could the promise of mobile technology clinic. Almost all of the country's population grim problem. In 2007, PNG had only really transcend the tyranny of distance? To 50,000 phone lines for the entire country find out, a pilot project was paired with a – one phone for every 140 persons. Seven years later, 2.7 million people had mobile The idea for the project began with a phones – one for every 2.6 persons.
medical specialist at the University of This was real progress. It meant that for Papua New Guinea named Glen Mola. the first time rural health workers could Professor Mola believed that the best be reliably connected to specialists in the place to run and evaluate a time-limited labour wards of major towns. A previous pilot project would be Milne Bay Province, initiative to link rural clinics to major where the maternal mortality rate was centres via radio had achieved some even higher than the national average. Alice Siwawata pauses to catch her breath at success, but this became unreliable when
Because the province constitutes the Alotau Provincial Hospital. the radios broke down because the parts
mountainous eastern peninsula of PNG's Evaluations that make a difference-EN-20Sep15.indd 21 21/09/2015 13:14:25 were also provided to calls. ‘It was very difficult previously, recharge the mobile communications-wise,' she says. ‘During phones. Calls started emergencies now, it's like having a labour coming in almost ward doctor in the rural clinics.' immediately. Reports of The Australian government made a breach births, excessive comprehensive evaluation part of the pilot. bleeding, eclampsia, ‘That's not common in development work,' ectopic pregnancies, says Watson. ‘But for this one, evaluation retained placentas – was very important. Australia wanted to all potentially lethal know what could be taken from it.' without the right information at the While the initial anecdotal feedback was right time – could positive, Billy Naidi needed some hard now be addressed in response to isolated, Alotau Provincial Hospital labour ward staff practising answering calls.
workers who needed main island, as well as hundreds of remote help in managing difficult deliveries. Each islands, health care workers faced the month, an average of 17 calls were being challenge of extreme isolation.
made – each concerning a potentially life- After discussion with Mola, Australian threatening issue.
researcher Dr Amanda Watson set up ‘This project is saving two lives, mother and the pilot project in the labour ward of child, and we are thankful,' says one rural the Alotau Provincial Hospital in the health worker.
provincial capital in late 2012. She and Billy Naidi, the chief executive officer of ‘Once, I saw a mother die in front of me, the province's health authority, set up a simply because of lack of communication,' dedicated landline phone in the middle of says another. ‘Today, I can call anywhere, the bustling labour ward and sent the toll- free number to rural health care workers in Alotau health extension officer Alice Billy Naidi and Rob Brink mark the handover of the
remote locations. Importantly, solar panels Siwawata fielded many of these project from Australia to the local health authority.
Evaluations that make a difference-EN-20Sep15.indd 22 21/09/2015 13:14:26 Sabumei travelled across the province, interviewing rural health workers, recent mothers and village leaders in remote areas, as well as the labour ward staff at Alotau Provincial Hospital where the phone line was based. He returned from his foray through the province with 44 interviews in his pocket. These interviews almost universally praised the pilot.
One rural worker heard of the free call service just before a woman presented with a retained placenta. ‘If this project was not here and I didn't have any means to communicate, we could have lost this mother because she was already septicaemic,' said the worker. ‘From anywhere, at any point, I can seek advice Gaius Sabumei interviewing rural health worker Rose Elliot at Omarakana Health Centre.
and help the patients.' evidence that the emergency line was proof that would come through a detailed working. Ever since Watson had pitched the evaluation of the pilot. With a thorough Almost all the interviews reflected positive idea to him, he'd become interested in the evaluation of the project in his pocket, he responses. And even when a few of the health care possibilities opening up with could justify taking over the funding of the respondents had something critical to say, the rapid expansion of PNG's mobile phone phone line. ‘We wanted to see what has they readily admitted that things were network. Australia's aid programme was happened,' says Naidi. Watson understood working better than before. For example, funding the pilot project, but these funds his position and agreed to switch roles and one labour ward staff member said would soon dry up. (Aid from Australia, evaluate the project she'd set up.
‘Answering an emergency call when you are PNG's former colonial overseer, still busy attending to a mother in delivery is accounts for around 13% of the nation's Watson began the evaluation process by sometimes frustrating, but it's part of our revenues.) But for an initiative to become hiring a local field officer and evaluator work.' Beyond that, criticism was muted. sustainably funded by the provincial health named Gaius Sabumei, who came to know Another labour ward member said that authority Naidi headed, he needed proof – Milne Bay Province well. For months, the initiative helped build relationships Evaluations that make a difference-EN-20Sep15.indd 23 21/09/2015 13:14:26 and distaste for seeking help phone line was superior to the ailing from clinics amongst some radio network. While it was impossible to cultures on outlying islands, estimate how many lives had been saved, it the phone line had saved lives. was clear from the clinical evaluation that Dr Bintol wrote: ‘There is very some women would have died without the good evidence from the cases service, and that rural health care workers that lives of both mothers and felt the initiative reduced their isolation. babies have been saved. It was ‘I wanted to know that it was effective. also evident that information Eventually we were convinced, and we exchanges in the phone made a commitment to sustain it.' calls were very educational, And with that, the phone line's future was especially for the rural health secured. After two handover ceremonies, staff who may encounter a in Alotau and in the nation's capital of Port similar problem in the next Moresby, the Childbirth Emergency Phone event.' The evaluation also line at last became an ongoing, sustainable demonstrated that the advice given over the phone was useful and appropriate in most cases, What has surprised Watson has been the Rose Elliot of Omarakana Health Centre is delighted that she
can charge her mobile phone with a solar charger provided by and inappropriate (and thus
unexpected spin-offs from the evaluation. potentially harmful) in 16% of The lengthy report led to national and Australian media attention on maternal between the metropolitan staff and the the cases. This led Watson and Sabumei to mortality. Newspapers, radio, and rural areas.
recommend follow-up training to address television all reported the evaluation's the types of cases where bad advice had In addition, Watson sought external findings. ‘The evaluation gave people ideas validation from a health professional from for the use of phone lines in health more a different province. Dr Derick Bintol, a PNG When the evaluation was given to Naidi, he broadly,' she says. In the mountainous national, listened to a sample of recorded examined it carefully. ‘From the beginning, I Western Highlands Province, for example, a phone calls and analysed the clinical notes liked the idea. I was excited,' he says. What broader scheme is now up and running – a from a sample of 68 calls.
he wanted to know, though, was whether call centre staffed by trained nurses who the line worked as well as the anecdotal The evaluation found that despite the answer health questions from the general feedback suggested, and whether the challenges of phone network downtime Evaluations that make a difference-EN-20Sep15.indd 24 21/09/2015 13:14:27 The Childbirth Emergency Phone is a project of the Milne Bay Provincial Health Authority. Establishment and the evaluation were funded by the Government of Australia.
This story was written by Jessica Kenway
and Doug Hendrie. Drafts were reviewed
by Billy Naidi, Chief Executive Officer
of the Milne Bay Provincial Health Authority, and Amanda Watson, Mobile
Communications Research Consultant, Economic and Public Sector Program.
Photographs were taken by Rawena
Russell and Amanda Watson.
More information on the project
is available at www.pngepsp.
Performance group celebrating project launch in Alotau.
For Billy Naidi, the value of the evaluation confronting a challenging situation, and has also been demonstrated by proving you can be in an intimate conversation – as the effectiveness of breaking down Papua if you're standing next to someone in the New Guinea's millennia-old isolation using labour ward,' he says. ‘After the evaluation, modern technology. ‘With this phone we were overwhelmingly happy!' number, you can be in a remote area, Evaluations that make a difference-EN-20Sep15.indd 25 21/09/2015 13:14:28 THE NETHERLANDS
Positive Sisters: a transformative surprise people who first meet her over the
phone. ‘I started doing this work in 1996 when my gay friends, all white men, were When you find the right direction in life dying from the disease. I felt a sense of it can be contagious. Inga didn't realise it though. To her, she was doing what was After working with various religious right, but little did she know how many organisations for a few years, Inga felt that people would blossom from being in the restrictions and expectations placed on contact with her. Her clarity of purpose also her diluted her sense of purpose. She felt helped build the support she needed.
a need to choose her work based on what Inga Mielitz is an atypical Christian minister was important to her.
in the Netherlands who works with people They didn't want to hear that discrimination affected by HIV. ‘I'm a dyke,' she says of people with HIV has its roots in the with unabashed candour. Her razor-short negative way our churches talk about bleached hair and assertive demeanour sexuality; they didn't like my openness. But Volunteers receive their certificates from project
leader Inga Mielitz.
life is precious, and I wanted to do what I consider important in my life instead of doing what others expect from me. I believe that God wants every person to be who they are instead of pretending to be someone else. I still wanted to work with HIV-positive people, but I wanted to do it my way. Put together, sexuality and faith make a connection to something bigger than oneself.
Shortly thereafter, ShivA – an abbreviation for Spirituality, HIV, and AIDS – was born. The aim of this Netherlands organisation Taking a break on the beach during a training workshop.
was to improve the quality of life of HIV- positive people and their loved ones Evaluations that make a difference-EN-20Sep15.indd 26 21/09/2015 13:14:29 through empowerment and by supporting people seeking meaning and spirituality. The idea was to provide a ready response for people asking Where do I find the strength to go on? Positive Sisters is a ShivA programme that provides support for HIV- positive people, especially migrant African and Caribbean women. Most of the women are referred to ShivA by hospitals where they've gone for treatment.
The gift of evaluation
After 4 years working with African and
Caribbean women and 2 with the Positive Sisters project, having enrolled more Getting the participants together with zumba dancing.
than 150 women and trained 18 Positive Sisters, Inga met two staff members of an the stories to identify the ones that felt Liako is a lively speaker and wears a evaluation firm called Results in Health most meaningful to them. The evaluation broad smile. Her hands and body move at National AIDS day in Amsterdam. team then analysed the data and compiled rhythmically with her words as she speaks. Aryanti Radyowijati and Maaike Esselink, a report. Finally they shared results Her dark skin looks translucent. She exudes impressed by Inga's story and touched by and recommendations. Inga hoped the positivity and joy. Inga offered Liako her ShivA's apparent impact, offered a pro- evaluation would help them secure funding newly started Positive Sisters volunteer bono evaluation using the Most Significant going forward.
training. Liako was enthusiastic, inspired to Change technique, as an opportunity In an interview, Liako, a Positive Sister from find a kindred spirit in Inga, and eager to for their team to gain experience in that Lesotho, shares her first conversation with help support others to live positive lives.
methodology. Given the powerful stories Inga: ‘I explained to Inga how my life is and After Liako had been a Positive Sister for and small sample size, Most Significant how open I am about HIV. Inga said, "Vive ShivA for 11 months, she met a young Change would be perfect.
la Vie [one of ShivA's early projects serving woman named Millie, aged 30 and recently Maaike's team interviewed the women and Afro-Caribbean women] is not for you, it's diagnosed with HIV. Knowing that Liako was then facilitated a collaborative workshop for women who are closed and who don't HIV-positive, Millie expected a thin, weak where different stakeholders reviewed know other women."' woman, her face creased in pain.
Evaluations that make a difference-EN-20Sep15.indd 27 21/09/2015 13:14:30 whatever she wanted to do with her life, and who are affected by it. This evaluation was neither should I. I should go on and do it.
the voice of the people infected.… Being [You] need a doctor or a nurse when you're interviewed made me want to be involved sick, [but you also] need someone who has more. I didn't want to only be a patient been through that pain, who knows what it after that. Being interviewed helped me feel feels like. Without Liako I would not be who I important and part of something good.
After the interview, Millie decided to Clearly, being at ShivA had positive effects become a Positive Sister, and she is now on the women's lives. So did the evaluation, enrolled in university.
but in different ways.1 For Liako, being interviewed meant becoming more aware of the impact of her Each of the five women interviewed work since ShivA: ‘The evaluation had a big for this article (two positive sisters, a impact on me. It was a Wow! moment for project coordinator, a lead evaluator, and me. We are really doing a great job.' a referring nurse) spoke about different Maaike's team prepared the interviews into aspects of the evaluation process, but they stories so they could be read at a half-day all mentioned the positive impact it had on workshop to see which stories were more them and on the project as a whole. In the individual and which represented shared evaluation interviews, the women reflected experiences.
on their experience with HIV/AIDS and how much they had changed since ShivA came The nurse, Lia, and the evaluator, Maaike, into their lives. Millie had been quite shy described the collaborative workshop day My body is my best friend, I will take care of her – about her HIV/AIDS status before ShivA,
as a lively experience. The atmosphere even with HIV.
and now recognises that being interviewed that Positive Sisters brought was one was an extremely positive experience.
where ‘the floor felt like it was vibrating.' The environment was joyful: good food, Whenever people talk about HIV they talk I was shocked that she looked so beautiful. about how many people are infected in laughter, music. The women talked, It was reassuring for me to talk with her; she the world and how many people are taking laughed, and sang spontaneously despite motivated me to push forward. She told me medication, but it's never the voice of those the intense sadness of their stories.
how she found out when she was my age that HIV/AIDS shouldn't stop her from doing Lia remembers a shy, somber woman she Judgments about the impact of the evaluation are based on the perceptions of the five respondents interviewed.
supported. At the evaluation workshop, she Evaluations that make a difference-EN-20Sep15.indd 28 21/09/2015 13:14:31 was pleased to see this woman stand up from the project, by hearing other stories, can Liako agrees. ‘We did short videos and share her perspective: now talk about the project as a whole.' online. They are being shown on the When Positive Sisters are in the room, it is The stories grew from individual narratives, internet and at conferences on HIV. alive; I saw women become empowered. One to project stories, and finally to a national Mine is about having a husband and a woman transformed in front of me when she daughter who are negative. Healthy. got up to speak. With Positive Sisters, women dialogue. Inga says that the evaluation And people get interested in who we are finding meaning in life. Everyone searches empowered African and Caribbean women are and what we do.' for meaning but when you are diagnosed with to speak up on the national stage: ‘It was a HIV, you face it more.
snowballing process. Once it got started there Evaluation spurs action
With the collaborative workshop, the was no stopping it. Policy makers are now As a result of the evaluation, individual stories started to blend into a contacting our women to contribute to their Positive Sisters was invited to deliver collective story. After the workshop, Inga research, and there is a group of African women a workshop at a National AIDS told Maaike, ‘It is amazing that the women contributing who never spoke up before.' conference. Inga says: Positive Sisters present themselves at World AIDS Day.
Evaluations that make a difference-EN-20Sep15.indd 29 21/09/2015 13:14:31 Many Positive Sisters became more confident. do more. It made me There was a very small woman from Ethiopia. feel very important, like The candle with an AIDS ribbon burns throughout
She was very quiet but there was a deep I'm fulfilling a dream I every ShivA activity.
inner power in her. She did an Ethiopian coffee ceremony, sitting on that little low chair didn't even know I had. just by herself. But she raised her head and I feel like I'm more than contributed. Her voice was not loud, but her I thought I could be. confidence shone through and everyone paid For me, success is not attention to her because they were hearing getting paid – it's doing the voice of a free woman! Although most of the stories collected by For ShivA, expanding Results in Health were positive, they also roles are also important offered insights into where improvements given that referrals are could be made. One suggestion was that increasing too. Lia, the Inga was doing too much. ‘The project nurse, says that the was leaning too strongly on me,' Inga evaluation increased recognises. ‘If I wasn't there, nothing her referrals from one happened.' But having more coordinators in three to all women meant investing more money and Inga diagnosed. ‘Because of knew money was tight. the evaluation I have a The evaluators suggested she assume more professional view a coordinator role to help mediate of Positive Sisters. After the evaluation I project. She really wanted to learn from it interactions with participants (now talked with my colleagues and we decided and get results. She always made time for numbering several hundred) and Positive we would refer more people to Positive Sisters (now 31). Two Positive Sisters offered to take on a coordinating role One of Inga's primary concerns for ShivA without pay, including Liako, who with The evaluation team felt that working is, as with most community projects, to ShivA's help is now pursuing a professional with Positive Sisters was an opportunity ensure continued funding. So when the vocation as a counsellor and a coach. At to work with an exemplary client. Inga evaluators proposed a workshop with the time Liako was interviewed, this new was hungry for insights and feedback and multiple stakeholders to look at results and coordinating role had just begun. She quick to apply them. Maaike says, ‘Inga was think about funding opportunities going explains: ‘The evaluation made me want to extremely interested in learning about the forward, Inga was on board.
Evaluations that make a difference-EN-20Sep15.indd 30 21/09/2015 13:14:32 While steady funding for ShivA is not quite there yet, it is certainly on the way. The snowballing effect continues. At the workshop there were many ideas about how to get health providers to pay for the support participants get from ShivA. For example, a large foundation supported by the royal family of the Netherlands recently committed to help find the needed funding.
But the most important outcome from the workshop was something intangible and transformative.
In Inga's own words: It was very special to me because I do a lot of work alone and this time I was not alone. Everyone was trying to find a future for Positive Sisters. Through the evaluation, I was able to tap into the experience of professionals who also do good quality work. They could evaluate the quality of what I'm doing. It gave me a new surge of self-confidence, a stable ground to stand on. Now I can see how to make ShivA grow further and give it a brighter future.
Each bead on each bracelet represents a blessing given by the women to each other.
We would like to credit the tellers of this story: Story writer: Rita Fierro
Inga Mielitz, ShivA Foundation
Short video on the effect of the Positive Sisters link to Liako Lekhooa-Oude Lansink and Millie, Positive Sisters
the evaluation report: Link to ShivA: www.shiva-positief.nl C. (Lia) Meerkerk, nurse practitioner
Link to ResultsinHealth: www.resultsinhealth.org Maaike Esselink, Programme Officer, Public Health
On the Positive Sisters project: www. and all respondents of the evaluation who made this evaluation so valuable. shiva-positief.nl/a-letter-to-you Evaluations that make a difference-EN-20Sep15.indd 31 21/09/2015 13:14:33 KENYA
Learning and earning: training
and vocational training. Now CAP was keen to learn how this approach could work in Kenya. Funded by The MasterCard Like many countries, Kenya has an Foundation as one of its Learn, Earn and enormous population of unemployed Save initiatives, CAP YEI, in partnership young people. Depending on one's with the University of Minnesota, evaluates perspective, this is either a ticking time its performance in both the short and the bomb or an opportunity for affordable long term to make changes as it learns vocational training – relevant training that what works. The ongoing evaluations have enables young people to get jobs, earn and helped to equip Kenyan participants with save money, and move out of poverty.
technical and psychological skills to get a job or to start their own business.
The Community and Progress Youth Empowerment Institute (CAP YEI) opened Douglas Moseti, CAP YEI Nairobi regional its first training centre in Nairobi in 2011. coordinator, has been with the programme In India, CAP had been successful in giving since it began. ‘We target people between vulnerable youth a chance to acquire 18 and 25 years old who can demonstrate useful workplace skills through technical that they come from vulnerable backgrounds. Either they are orphans or they Students filling Equity Bank savings account
come from single-parent applications in the CAP YEI training centre.
families or large families where a large number not pay his school fees. He finally finished of people depend on a high school, thanks to a lucky encounter single breadwinner.' at his church. But with a high school diploma, he could only find a job paying the Moseti knows what it equivalent of $1 a day. A 2-year vocational means to need a leg up in training course was out of his reach, at life. He grew up poor and least for a time.
was forced to drop out of school several times CAP YEI provides 3 months of training CAP YEI youth in a Unilever training course on how to access products when his mother could
adapted to labour market demand. To for their small enterprises.
Evaluations that make a difference-EN-20Sep15.indd 32 21/09/2015 13:14:33 the proposed course He then goes on to describe the evaluation content, to mentor that follows the placement of each CAP YEI trainees, graduating class.
and to offer them Once a batch has completed everything, internships when we review what we've finished before they graduate from starting a new batch. Facilitators, the the programme.
regional coordinator, and the programme director describe the whole process. We ask Importantly, the questions: Did we get the right people at the training cycle doesn't road shows? Did we give them the right skills? end with the classes We look at student attendance at training as in most vocational and at work and ask: Was placement good? training programmes. What course gave good placement? Where Rather, each cycle training does not lead to jobs, we change and get more industry people to help us, to tell culminates when the us what's missing and we supply that. In one graduates are placed centre where students had a hard time getting in internships or jobs jobs in electrical and electronics, a new or are starting their market scan was carried out and a curriculum was designed to offer training in building This trainee obtained employment with a floriculture firm in Naivasha.
Moseti describes and construction. Elsewhere, garment manufacturing was replaced with training in what happened with security and management systems.
determine which courses to offer, CAP the first group of trainees.
YEI staff scan the local market, contacting James Chepyegon, CAP YEI project There were 141 students in batch 1 and we companies and entrepreneurs to see what manager, uses a management information were 6 staff who were very new. We were told entry-level jobs are available. In its first that we'd get at least 80% of those 141 into system developed by CAP to collect data centre in Nairobi – it now has nine there – employment opportunities. We didn't hit the on course enrolments, on graduates per target for the first batch because the industry they found that many hotels needed staff cohort, on the absorption rate of graduates didn't believe that a new employee could be and security guards, and that construction into the job market, and on changes in trained in just 3 months, and we had no way firms need builders and electricians. As CAP students' attitudes and lives after the to show potential employers that the trainees YEI engages with potential employers to could do the work. But we did manage to see what jobs they might have to offer their place 67% of the trainees, including 5% who Students are surveyed before and after graduates, they also ask them to look at went to start their own businesses.
training to see how their attitudes have Evaluations that make a difference-EN-20Sep15.indd 33 21/09/2015 13:14:33 changed, how they made the transition to jobs, One of the most striking findings of the Dr Heidi Eschenbacher, another member of how they've created their own businesses, how evaluations has been the critical role the team, confirms: ‘Our findings suggest their views of life have changed as a result of the of life skills in preparing young people that life skills combined with technical training and employment, and how well they for jobs. Professor Joan DeJaeghere, skills help youth to build up sufficient understand working relationships with employers.
who leads the University of Minnesota confidence to feel as though they can use This project data blends with the data team, cites her team's annual on-site their technical skills. In other words, they collected annually by the University of visits, surveys, and interviews with need life skills to build confidence to be Minnesota team that comes to Kenya to efficacious in their technical skills.' survey stakeholders, including participants Year after year, CAP does life skills, a mix and employers. Their data is compiled into The life skills range from mentoring by of engendering self-confidence, teaching a longitudinal study to see how the training successful business people, many of whom students to be assertive, teaching them changes the trajectory of participants' about job markets and how they work, started out with little but managed to livelihoods and lives. This impact evaluation where they are in the job market and what succeed, training in financial literacy, and is designed to emphasise learning in the they can expect, and financial literacy. training for young entrepreneurs looking monitoring, evaluation and learning portion Youth say that the life skills training is to start their own businesses. Chepyegon what makes the difference. They say ‘I of the programme to help CAP learn during confirms that the evaluations clearly show can get into something, start off, move to implementation and respond to the needs of that the links between employers and life something new and manage finances.' targeted youth.
skills training work. ‘They are,' he says, ‘the strongest point of success for CAP YEI in getting youth access opportunities.'Their strength lies in part in the way that the life skills module has changed in response to the evaluations of participants' comments over time. Adds Chepyegon: The entrepreneurship department got stronger, to better account for entrepreneurship activities from the time individual potential entrepreneurs were identified to helping them start up their small business, including the creation and registration of entrepreneurship groups and support for the success of their small Carpentry training class at the Athiriver Centre.
Evaluations that make a difference-EN-20Sep15.indd 34 21/09/2015 13:14:33 businesses. Nearly 200 entrepreneurs are 30,000 shillings per month from these mobile David Chapman, co-lead of the external currently functioning well on a full- and part- banking services, which include Mpesa, Equity Agent, and Cooperative Bank Agent. Another There have been some surprises going into Moseti understands the confidence graduate imports clothes and shoes from Dubai and sells them in Kenya.
the fourth year of the 5-year field data building that comes from having a mentor collection. Where conventional wisdom because he experienced it. He recalls: The learning and evaluation partnership might suggest that participants go through at CAP YEI is working to help youth get the training and then find or create a job, A prominent person gave me electrical the reality is that they are multi-tasking to installation work at a very big building. As an ahead, while simultaneously learning what earn money. In their efforts to start their own electrical person God granted me favour with stands in their way. Reports Professor the boss. Whenever there was need for a driver and since I was qualified, I was the one to buy materials and run errands that required a driver. I became his point man on the construction site and ended up becoming the foreman of his building with both electrical and other construction responsibilities.
Moseti later joined his boss' ICT firm, was promoted, and got a good raise.
Quite recently, Moseti was able to take the measure of the project's success when he met with the first cohort of graduates in Nairobi. This cohort was the group followed by the University of Minnesota for its longitudinal study.
Seventy-eight of the 141 students came. All but one of them were working. I discovered that most of these young people (almost 40%) had moved into self-employment with big businesses. One lady had become an agent for large banks seeking to extend services to rural areas. She says that she's earning Kenya's Industrialization and Enterprise Development Cabinet Secretary Adan Mohamed inspecting a
demonstration by CAP YEI Athiriver Centre building and construction students.
Evaluations that make a difference-EN-20Sep15.indd 35 21/09/2015 13:14:33 businesses, these youths have trouble getting released, average start-up capital, average the necessary capital. That is slowly changing. earning post start-up, and average savings Christopher Johnstone, Acacia
Banks in Kenya have been in the forefront per client in order to make it easier for Nikoi, and University of Minnesota
of youth bank products and have found that research fellows conducted annual
group lending programmes have a better graduates to get the financial support they external evaluations of the project repayment rate than they get with their more need to start their own businesses.
and presented data that helped The ongoing programme improvement refocus project programming and CAP YEI, in response to this finding, has process is extremely helpful in assisting improve services for youth.
been pursuing a number of partnerships youths to get and keep good jobs, with financial service providers who are particularly in dynamic situations such as James Chepyegon, monitoring and
tracking key data points such as number labour markets into which many youths are evaluation executive, was responsible for of accounts opened, amount of funding ready to enter.
conducting internal project monitoring, interpreting implications of M&E data, and building capacities of project staff.
Ashok Ankathi (project director at
the time) provided programmatic evolutions influenced by evaluation.
Douglas Moseti (Nairobi regional
coordinator at the time) provided the context of the beneficiaries that helped shape the story.
Story writer: Deborah Glassman
We did it! Nyeri Centre Batch 1 trainees marching on their graduation day.
Evaluations that make a difference-EN-20Sep15.indd 36 21/09/2015 13:14:33 MEXICO
If you don't ask, you won't see it! of poverty that typifies many rural
The indigenous communities were communities. It does so by using a among the intended beneficiaries of the Contemplating the indigenous women of conditional cash transfer (CCT) approach, programme since its inception, and they the Sierra Tarahumara, a mountain area in where families are provided with payments also participated in their evaluations the state of Chihuahua in northern Mexico, that are conditional upon undertaking from 1999 to 2006. But although these evokes bucolic scenes from the time of the certain activities, such as ensuring regular evaluations confirmed that indigenous Spanish conquest. These communities have attendance of their children in school or people were effectively participating in the maintained themselves for centuries far obtaining certain health services. These programme, it was unclear if they were from cities, clustered in small villages and incentives assist in the achievement of achieving the stated objectives in terms of scattered family groups, cultivating a few higher standards of education, health, health, education, and ultimately, economic seasonal crops and raising chickens, goats, and nutrition, and also provide necessary and cattle. Almost all live in poverty, and in support to the people of the Sierra many cases they are semi-nomadic.
Tarahumara to undertake economic Access to the Sierra Tarahumara is difficult. activities that enable them to increase their Sometimes it takes several days to reach family income and quality of life.
the settlements. It is therefore quite difficult to make contact with them to interview and select candidate families for the Programa de Desarrollo Humano Oportunidades (Human Development Opportunities Programme, hereafter referred to as Oportunidades). But the effort is well worth it because the programme is quite remarkable.
This programme, implemented in Mexico since 1997, aims at nothing less than breaking the seemingly endless cycle The evaluators learned to ask the right questions – in the right language!
Evaluations that make a difference-EN-20Sep15.indd 37 21/09/2015 13:14:35 performance. According to Mercedes Gonzalez de la Rocha, anthropologist and head of the qualitative impact evaluation of Oportunidades, the absence of a clear focus on indigenous communities had left a blind spot in the programme's knowledge base. Or as she puts it, If you don't ask, you won't see it!This statement constituted a starting point for raising the influence of an ethnicity variable in the new qualitative evaluation to be conducted in 2008. By then, the programme had a decade of experience in implementation in Mexico, which allowed for a thorough evaluation. Evaluators designed a strategy for field work comprising 11 indigenous intercultural regions in the states of Chiapas, Chihuahua, The women of Comunidad San Ignacio, Municipio de Bocoyna, Chihuahua.
Oaxaca, and Sonora. In each location, the programme's coverage and operations vocales were bilingual. Although the technical information they provided. This were analysed. The evaluation identified programme had been operating since 1997 situation was exacerbated in the case of the main obstacles to programme with more impact in indigenous areas elderly women, who spoke no Spanish at implementation with particular attention to than in non-indigenous areas (as had been all. The promotores assigned to the region the relationship between the extensionists documented by previous evaluations), did their best to overcome the language (promotores) and the women representing the important problem of communication barrier, but the results were unsatisfactory.
the indigenous communities, the vocales.
had not been sufficiently addressed. For instance, the evaluation found that young In some areas, the majority of indigenous What they found was quite unexpected: indigenous women – most of whom were women did not understand what the there were serious communication bilingual, although their Spanish was often programme was for. They couldn't problems with language. Virtually none limited – did not accurately understand understand what good it did to spend of the promotores and only a few of the the Oportunidades employees and the hours listening to medical specialists who Evaluations that make a difference-EN-20Sep15.indd 38 21/09/2015 13:14:35 spoke about issues they could barely in some indigenous areas, in the Sierra comprehend. Sometimes the information Tarahumara, a whopping 30% of the that was being communicated conflicted population remained outside the with their traditional customs. For example, programme. It was clear that the lack of when indigenous women participated in access to health services and education This arrangement training on the importance of a physical was brought about by the problem of examination for possible breast cancer, monolingual families who could not benefit further positive it was clear that the idea of a stranger from the written and oral information touching them in their private parts made conveyed by Oportunidades staff members providing jobs them very uncomfortable. These women to the women.
and wages to the do not even undress in front of their It was urgent to take measures to solve husbands! Thus, a practice intended to this serious problem, and Oportunidades own living space, save their lives was totally unacceptable for did just that. The qualitative evaluation cultural reasons.
Jaime Holguín, one of the
suggested that bilingual promotores be bilingual promotores, in
The evaluation found that although the recruited from indigenous youth alumni so learning they had Comunidad Cerocahui,
programme's coverage had improved as to contribute to better communications acquired right Municipio de Bocoyna.
own communities.
The then general coordinator of Oportunidades, Dr Salvador Escobedo, We managed to make the change in rules of operation to include bilingual extensionists, and that was the first step. Then in parallel we ran a training programme with the National Institute of Indigenous Languages (INALI) in order to generate a mechanism to evaluate the indigenous youths we wanted to hire as extensionists who speak an indigenous language. We have sought to work with the 13 most spoken languages such as Tzotzil, Maya, When their parents can understand the promotores, they are happy to support their children's education.
and Totonac.
Evaluations that make a difference-EN-20Sep15.indd 39 21/09/2015 13:14:38 The women of San Ignacio receiving food aid during the drought crisis of May 2012.
Another important aspect that was pointed out by the qualitative evaluation of 2008 concerned the inefficient (and even absurd) survey of each household to assess whether or not it was eligible for the programme. In the case of the communities of the Sierra Tarahumara, this was redundant because every household was unquestionably poor – not to mention the difficulty and cost of getting there just to conduct the survey.
Escobedo, who understood the realities of indigenous communities in Mexico, supported the proposals arising from the qualitative evaluation that were eventually endorsed by the President.
To do this, INALI began training a group of The process of bringing bilingual The first step was to convince 15 young men and women with diplomas promotores into the programme unfolded Oportunidades collaborators such as as trainers of social programmes in steadily over the next 2 years. As their government officials and international indigenous languages. They were the first understanding increased about the organisations like the Inter-American to reach the remotest areas of the country importance of sending their children to Development Bank and the World Bank of to carry the voice of the programme to school, feeding them properly, and learning the need to change the rules for coverage where it was most needed. Says Escobedo, how to use resources, indigenous women in isolated areas. That involved a major ‘The project trained 350 promotores in became increasingly willing to participate challenge for the programme managers order to achieve almost total coverage of and interact with the promotores and because, beyond being convinced of the monolingual indigenous populations in with each other. In some areas they even the real value of this recommendation, Yucatan, Oaxaca, Chiapas, Jalisco, and the formed groups for early child education. they needed to effect those changes Sierra Tarahumara. By the time I left the The role played by the qualitative without appearing to refute the spirit of programme, we had trained a total of 250 evaluation with its remit to address the Oportunidades, which was historically extensionists, and awaited evaluation on marginalisation caused by monolingualism focused on the poor while requiring certain the success of this implementation.' was key to this transformation.
Evaluations that make a difference-EN-20Sep15.indd 40 21/09/2015 13:14:38 responsibilities as sending children to school An important spin-off of the programme has been
or attending health talks. The figure was to instill a spirit of entrepreneurship amongst the
worrisome, and returning these people to women and girls of the Sierra Tarahumara.
the programme would be difficult, if not impossible, given the rules of operation.
Reaffirming this contention, Escobedo The World Bank and Inter-American Development Bank opposed eliminating the requirement of the co-responsibilities in these areas, in part because they would undermine the programme, thus generating a new programme. That´s the reason why it could not be done overnight. It is sad and painful, but very However, given the magnitude of the problem in the context of the severe drought affecting indigenous people in the Sierra Tarahumara, the situation was resolved conditionalities of the beneficiaries such as That report was published in a Chihuahua after the intervention of the President health checks and school attendance.
newspaper and generated an immediate response from the federal government, which Felipe Calderón himself, who ordered the In late 2011 and early 2012, a severe decided to intervene in various regions with immediate reinstatement of 8,000 families. drought hit the Sierra Tarahumara. the Secretary of Social Development. It was This was accomplished by integrating the Rumours were rife about indigenous then that they communicated the decision reinstatement to the Food Support Program, people committing suicide for lack of food, that Oportunidades should intervene to solve a transfer programme that did not require although these were subsequently found to the problem precisely in the area of the Sierra the fulfillment of co-responsibilities that be false. Dr Iliana Yaschine, former director was also operated by Oportunidades. The of evaluation of the programme from 2002 The important mobilisation and attention indigenous promotores brought the good to 2006, coordinated a study documenting to the area made it possible to detect 8,000 news to their communities, while a renewed the work of Oportunidades in the Sierra families (40,000 indigenous people) who training of bilingual extensionists was Tarahumara during the drought crisis, had been dropped from the programme encouraged to address immediately the because they had not fulfilled such co- Evaluations that make a difference-EN-20Sep15.indd 41 21/09/2015 13:14:39 for the evaluation's strategy of full coverage in the Sierra Tarahumara, the onset of drought favoured the implementation of a recommendation for the best care of indigenous communities. The history of the qualitative evaluation Neglected no
shows the relevance of cultural sensitivity more! The
to the evaluation of social projects. In the women of
words of Gonzalez de la Rocha: may be hard to
If you don't ask, you won't see it!
reach, but well
worth the effort.
Co-authors Mercedes González de
In addition, the rules of the Oportunidades of this decision in the field at a time of crisis. la Rocha and Agustín Escobar Latapí
programme were changed to implement Special efforts were made to deploy support directed the qualitative external the strategy of full coverage (as it had for the affected families, not at the household evaluation of the programme and been recommended by the qualitative level but in the care centres where groceries made the recommendations that led to evaluation). This allowed incorporating were delivered. I remember what the improvements in beneficiaries' lives.
qualitative evaluation said on this subject, and families into the programme from small how that had prompted the changes during Iliana Yaschine was Director of Evaluation
and distant locations without undertaking a in the programme from 2002 to 2006.
home survey. This change helped broaden The findings and recommendations of the Pablo Rodríguez-Bilella and Omar
the programme's coverage and, in addition qualitative evaluation in the indigenous Zevallos helped shape the story.
to the reincorporation of the families communities provided crucial information mentioned above, it made serving the Salvador Escobedo (Program Director
for decision making. Managers made entire indigenous population of the Sierra at the time) provided helpful insight. changes that impacted the participants of the programme directly. In the case of Photographs were taken by Agustín
According to Yaschine: the bilingual promotores, the evaluation Escobar Latapí and Iliana Yaschine.
Note that the programme was originally
Without the qualitative evaluation, it would noted a dimension of cultural adaptation have been impossible to make that change, that had been neglected in the original called Progresa, then Oportunidades even with the drought crisis. That is my design of the programme, and once it was (the name in use when this story took perception. I observed the implementation incorporated the benefits were obvious. As place), then Prospera, the present name.
Evaluations that make a difference-EN-20Sep15.indd 42 21/09/2015 13:14:40 NEPAL
Listening to the listeners
accurate picture of the crude reality of child Just before dawn Ashish nervously waits marriage that prevails for Rama at the village bus stop. As soon across the country's as he spots his girlfriend, they cautiously remote villages. Nearly board the bus toward the city planning 3 of every 10 girls their future as a couple. But their plans are between 15 and 19 are thwarted when a police officer interrupts presently married, and 4 their journey and starts interrogating them. of 10 are married before ‘We just want to get married,' Ashish tells the age of 18, according the officer. ‘Rama's parents were forcing to national statistics.
her to marry a stranger.' Radio dramas like this ‘We have done nothing wrong,' Rama adds. one reflect the ethos The police officer then intervenes as the and pathos of Nepalese teenagers, both 16, request him to let them society. One programme Collecting a voice recording on effects of child marriage with an
in particular, a weekly adolescent girl for the SSMK radio programme.
called Saathi Sanga Man Ka Kura ‘It's a crime,' he says. ‘You need to be at Narrowing the focus
(SSMK), which means ‘Chatting with least 20 years old to get married. Child My Best Friend', has been reaching Before the introduction of the electronic marriage is a criminal offense – your out to young girls and boys for 15 media, street dramas and stage shows were parents can be prosecuted if they're years. SSMK educates young people popular throughout Nepal. Mostly mythological forcing you to marry young and you can be about issues that are generally kept and sometimes satirical skits in local dialects prosecuted too if you marry now.' quiet in conservative Nepalese emerged as a major form of entertainment in The legal age for marriage in Nepal is 20 society. Started in 2001 with support years without parental consent and 18 from the United Nations Children's While this trend might have declined in cities, years with consent.
Fund (UNICEF) and managed by Equal Ghanshyam Kumar Mishra, a radio producer Access Nepal for the past decade, the While Rama and Ashish are fictional from Nepal's southern town of Janakpur in 45-minute show has 7.2 million loyal characters from a weekly radio drama Dhanusa district, says dramas in local languages listeners and receives more than 1,200 aired on more than 40 stations across the are still popular and preferred in Nepal's rural letters and 2,000 text messages each Himalayan republic, their story paints an pockets. Most people in these areas are low- Evaluations that make a difference-EN-20Sep15.indd 43 21/09/2015 13:14:41 income farmers with little educational backgrounds and no source of information other than radio, which is also one of the cheapest forms of entertainment. Therefore radio dramas, according to Mishra, are one of the best media through which to inform the masses.
In 2013, SSMK conducted an evaluation using the Ethnographic Action Research method to assess and evaluate the listenership trend and impact of its programme on its target audience. Community-based evaluators in Nepal's rural districts conducted surveys and focus group discussions with community members and stakeholders.
One of the major findings from the evaluation was the need for a local Collecting opinion from a listener on sexual harassment.
language radio show.
The survey during the evaluation process addressed topics on a macro level, lacked says. ‘The listeners were demanding a highlighted the language conflict among discourse on community-specific problems. programme in a local language.' SSMK's listeners; the central version show The SSMK team therefore wanted to ensure To better equip rural youth with the was not able to fully capture the spirit meaningful participation of youth from knowledge and skills they need to of local communities. With the central across the country by starting local versions participate in local and national policy programme produced in Nepali language, of the programme, which would give them development, SSMK launched local versions listeners in other parts of the country a sense of ownership, according to Ayush of the show in 15 marginalised districts. such as Dhanusa where Maithali is widely Joshi, senior programme officer at Equal Equally important, its purpose was to spoken, sometimes felt disconnected.
Access Nepal.
provide necessary knowledge to the young Also, the issues varied from region to ‘It was important to start a local version people so that they could control their lives region and a national programme, which to sustain SSMK in the long run,' Joshi by applying what they learned.
Evaluations that make a difference-EN-20Sep15.indd 44 21/09/2015 13:14:42 Nepali, they might just think that it is not A group photo with adolescent listeners at school.
meant for them.'In the 26 episodes of SSMK's local offshoot, the show raised area-specific issues such as child marriage, dowry-related problems, and violence against women.
Mishra says the team also incorporated experts and law enforcement officials from the community to make episodes relevant locally. He says it adds extra credibility and makes the show more trustworthy. Upon listening to the show on child labour broadcasted in Maithali, Mishra The local programmes are customised Raising awareness of child labour
says people have become fairly hesitant according to their listenership with tailor- in recruiting children. This episode also made content that helps the audience At Radio Mithila in Dhanusa, Sangi Sang prompted authorities to at least warn connect with prevailing issues in their Manat Baat became an offshoot of SSMK. businesses that employed children.
communities. In a short span, Mishra says It was targeted at the Maithali-speaking ‘It takes time for these practices to end,' SSMK's local version has been effective in people living in the district's 90 Village Mishra says. ‘But I'm sure our radio highlighting issues like child marriage and Development Committees. A show in show has started that much-needed child labour, both of which are considered Nepali, according to Mishra, simply would conversation. It's a small revolution of socially acceptable in many rural not resonate with the locals. A Maithali communities. While the local-language programme, on the other hand, pulls in listeners primarily interested in the Changing attitudes about women's issues
radio dramas have played a pivotal role in educating the public, segments problems that dominate their region.
SSMK's evaluators note that local incorporating local authorities and ‘When you're communicating with people programmes help represent community stakeholders have opened up a non-formal in their own language, the audience feels problems more accurately, and Manauda communication channel for discussion. associated with the show,' Mishra says. ‘It Ka Kura, SSMK's local adaptation, is doing It has also helped in holding authorities is also easier to make them understand the just that in the far western district of accountable for these issues.
topic. When the programme is strictly in Evaluations that make a difference-EN-20Sep15.indd 45 21/09/2015 13:14:43 Through radio drama, using an Manauda Ka Kura receives considerable Auji says that their show in Acchameli informational and emotional approach, feedback and comments from its listeners, language has increased the engagement the show has drawn attention toward child a few apologetic but most applauding and interaction level with the community marriage, which is highly prevalent in the the programme's efforts. The letters and members. It has also made it easier to text messages, according to programme disseminate information on topics that Sarita, one of the characters, is forcefully producer Bidhutma Auji at Radio matter the most in their native tongue.
married at an early age. She is deprived Ramaroshan, motivate them to continue ‘When the programme is in a local of her will to study and dies of pregnancy the show and also help them ascertain the language, listeners feel that someone they complications. On the other hand, her show's impact.
know and trust is talking about their issues,' friend Renuka, who was against Sarita's One of the letters the show received Auji says. ‘It's easier to communicate using wedding, finishes her school and pursues after the episode on child marriage local words and metaphors that might be her education to become a nurse.
summarises the scale of Manauda Ka Kura's difficult to convey in Nepali.' ‘My daughter will never forgive me,' effectiveness. A 16-year old girl writes: Connecting youth, the elderly, and
Sarita's father later apologises to Renuka My parents who are very conservative were for dismissing her idea to at least wait for a forcing me to get married. But one day when For almost three years, listening to SSMK I was listening to the radio programme, the hosts were talking about the hazards of child and now its local version has become a marriage. I requested regular affair for Sagar Bhandari's family in Community dialogue on child marriage
my parents to listen Accham's Mangalsen village. Every week, to the show, and the family sits by the radio set in the living after listening they became aware for room and listens to the show. the first time about With higher mobile and internet the realities of child penetration, young people have alternative marriage. They then decided not to force means of accessing information. Bhandari, me into marriage. 18, says ‘Adolescents have many problems The local language they cannot talk with their parents. These radio show not only issues are always hidden from the parents. helped educate them But when their parents to listen to a radio but also helped them drama, it helps them understand what their to empathise with children are going through.' Evaluations that make a difference-EN-20Sep15.indd 46 21/09/2015 13:14:44 But statistics show that radio is still the popular choice of mass communication for Focus group discussion with listeners.
people aged 15–19, and SSMK is the most important radio show in that age group. According to the National Demographic and Health Survey 2011, 58.5% of men and 49.7% women in the surveyed population listen to SSMK regularly.
While the local programmes have broken the silence between the two generations, SSMK's evaluation also showed that they act as a tool to teach the adolescents about their rights – the legal age of marriage, the use of contraceptives, and charters regarding child labour. The programmes have also helped to initiate discussions between local policymakers and youth.
In village clubs and meetings led by local youth from the community, these episodes help to start conversations on important influence and impact of radio shows in Acchameli also yield positive results for topics. Bhandari says: the local language, particularly when the In our monthly Child Club meetings, we shows reaching areas inaccessible by the ‘Our [government] programmes are only discuss the issues raised in the radio show. mainstream media.
targeted at certain groups. So we cannot The information is not limited to the radio ‘When the programme is in a local incorporate everyone. But programmes like show itself. The dramas serve as catalysts. Once the topics are introduced, they are then language, people feel a sense of ownership SSMK are for everyone. No one is left out discussed and shared continuously at the – they think it's a programme for and by when it comes to accessing information.' workplace, in schools, and at home.
them. And they are right!' Youngsters like Auji and Mishra believe Prem Bahadur Buda, Accham's District Buda added that local language that education and access to information Child Rights Officer, also stressed the programmes like Manauda Ka Kura in are the keys to changing social attitudes. In Evaluations that make a difference-EN-20Sep15.indd 47 21/09/2015 13:14:44 districts like Accham and Dhanusa, young raising community-specific issues. They are people are in the forefront of this slow, yet programmes by the community and for the Yvette Shirinian is the Business
significant, change – making people aware community.
Development Officer for EqualAccess.
of the issues and helping prevent some ‘This has helped grow SSMK's listener base Ayush Joshi was the Program
of the negative aspects of past practices in many hard-to-reach areas,' says Joshi Officer for EqualAccess Nepal.
through the information they receive from Equal Access Nepal. ‘SSMK is building through radio programmes.
a strong rapport in rural communities Story writer: Bibek Bhandari
Shows like Manauda Ka Kura and Sangi as a people-centric and inclusive media Sang Manak Baat have proved relevant in platform that listens to its listeners.' Discussion on adolescent issues with adolescents
Evaluations that make a difference-EN-20Sep15.indd 48 21/09/2015 13:14:45 CANADA
The power of community-owned components of the initiative – as simply
if she and her colleagues could get their part of her busy and demanding job. ‘I'm a hands on first-hand information, they could community health nurse,' she says, ‘and this develop better services for their prenatal Things weren't happening that we
goes back to my nursing background – when clients.
thought were happening. Because of this
you start a project you need to finish it.' Changing River's maternal and child health project, we made changes.
Chesne's partners in this groundbreaking care needs are pressing. For one thing, – Emily Chesne, Community Health Nurse initiative included colleagues, managers, the population of the community is very community leaders, and researchers from young, with a median age of just 21, and Emily Chesne, a public health nurse in Health Canada. The researchers were it's growing quickly. For another, access Changing River, won't take credit for the responsible for coordinating and evaluating to treatment is a major issue because the improvements in her community's prenatal a total of 19 pilot studies in Aboriginal community is located 3 hours from the health services that she helped implement.1 communities across Canada. nearest urban centre.
For years, Aboriginal mothers have had to The research team included the travel long distances for prenatal checkups. project's leader, Judy Watson, a And on the health care side, record keeping 30-year veteran of government was far from what it should be. But thanks health programming and research to a pilot project called Patient Wait Times with Health Canada; Samir Khan, Guarantee and people like Chesne who a senior researcher and Chesne's take their jobs seriously, the situation has main point of contact; and Stephanie changed dramatically for the better.
Potter of the Whetstone Group, the While Chesne is satisfied with her team's programme's lead evaluator.
success in transforming her community's Through supportive collaboration prenatal health services through the with these researchers, Chesne and project, she downplays her role as an agent her colleagues, as well as the nurses of change. She views her participation in in other participating Aboriginal the data collection, tool development, communities, were able to use the and process mapping activities – all key data collected as part of the initiative 1 The events described in this story are true. However, at to make health services more the request of the resource persons involved, the names accessible and effective. The data of people and places have been changed. Suffice to say the events transpired in a First Nations community in northern First Nations people are justifiably proud of their vibrant
were important. Chesne knew that cultures. Above is an artist's rendering of a dream catcher.
Evaluations that make a difference-EN-20Sep15.indd 49 21/09/2015 13:14:45 The initiative – and importantly, its strong their appointments once they were in the community than anyone realised. evaluation component– led to a realisation referred to the larger hospital. Through This is a serious health issue that, that would change the course of everything their involvement in the evaluation, they according to Chesne, could have long- for the pregnant women of Changing River. discovered that many at-risk mothers were term repercussions. By facilitating early Through the evaluation, Chesne and her showing up for delivery without having had diagnosis, such problems can be identified colleagues learned that the reason data prenatal care for several months.
and remedied.
were lacking was because women were Using the evidence at their fingertips
The team continued to use the initiative's not attending their third trimester prenatal data to overhaul the community's prenatal Armed with the evidence they needed, the team took immediate action by and postnatal health care, setting the stage It was not difficult to understand why. implementing on-site doctor visits for for a healthier future. For the first time, Previously, the women attended prenatal prenatal patients. Chesne: ‘We initiated the clinic was governed by a client-centred appointments and delivered their babies getting one of the doctors from the culture. According to Chesne, ‘Now moms at a hospital 45 kilometers away. Recently, delivering hospital to come to our clinic. come in and learn how to take some of however, that hospital had not renewed Having the doctor here worked really well. their own prenatal tests. We have our its license to deliver babies. As a result, all It helped with gathering data.' doctor hold a sharing circle with clients as third trimester appointments and deliveries well as individual sessions.' were referred to a hospital a full 80 Improving third-trimester prenatal care Chesne reflects, ‘We learned things kilometers away – nearly twice as far. rates led to the identification of more gaps in care and unrecognised health risks weren't happening that we thought were Chesne knew that many local moms were among the patients. Better access to the happening. Because of this project, we young, impoverished, and uncomfortable physician's files helped the team realise in unfamiliar environments. She suspected that many prenatal patients were not The initiative had an immediate and that these circumstances made the getting their bloodwork done, including substantial effect on people's lives because 90-minute trek to the distant hospital too the glucose tolerance test for gestational its leadership, especially Watson, chose challenging. ‘We have a lot of young moms diabetes. Says Chesne, ‘We're trying to to prioritise community engagement. in our community. They are shy. They don't better educate our prenatals with what Frontline health workers like Chesne feel comfortable with professionals, with tests they need and why.' were true collaborators in the evaluation, Improving compliance with testing facilitating community ownership of the But until data collection began, the also led to the discovery that rates of data and making needed changes. Once community health team had no way of gestational diabetes were much higher these changes had been made, many of knowing if the women were attending Changing River's women who had been Evaluations that make a difference-EN-20Sep15.indd 50 21/09/2015 13:14:45 Potter recalls another community where nurses were able to use data to effect change. She was awed by the transformation of one group of nurses during the evaluation. At first, Potter found them apprehensive, as if things had gone poorly with past research.
By the end of the initiative, however, the health workers had shown Potter that even people without strong research backgrounds could ensure that their communities got the most out of the data to which they had access. She recalls: They didn't have computer databases, they only had pieces of paper. But by the end of the project, they made those pieces of paper mean something. I will always remember what they said at the end: Data is good. They Many First Nations communities are accessible only by air or boat.
learned that. They could see patterns and at risk for pregnancy-related complications evaluation. But her fears were allayed when make changes in their communities even now had access to diagnosis and treatment. the nurses all arrived for their meeting.
without sophisticated stats.
Chesne's community was not alone in Watson was moved. ‘Two nurses had their Community empowerment through
taking ownership of the initiative's data. study files under their arms – they didn't Watson recalls a 2009 meeting with nurses even have briefcases. One nurse had lost a from a Manitoba community even more family member in the midst of all this, but At every level of participation, stakeholders remote and with fewer resources than they kept the study files organised in spite point to the Patient Wait Times Initiative Chesne's clinic in Changing River. At the of the challenges. They really wanted to stay as a particularly powerful example time, the swine flu crisis was raging and it on track for themselves, for us, but also for of government-funded community was overwhelming the community. Watson the community. That's how important the programming to empower frontline wondered whether the nurses would have community is – counting and being part of workers to effect change. But what made it the time or motivation to continue with the the evaluation was so important to them.' different from other initiatives? Evaluations that make a difference-EN-20Sep15.indd 51 21/09/2015 13:14:45 Health Canada and its Potter found that working with Watson collaborator in the leadership and her team was refreshing, informative, of the project, the Assembly of and educational because they were ‘…so First Nations, as true partners comfortable not knowing necessarily what from the beginning to the end. was going to happen. They gave permission Watson and Khan agree that for evaluators to ask questions and find the the same can be said of the evaluation team, emphasising Questions and answers – from both sides
that with Potter's and her colleagues they were able to At the community level, the data collection form a genuine partnership, tools and procedures were refined based leveraging each other to on the evaluation results and direct A nurse arriving by speedboat.
inform each other.
feedback from the frontline. Both Watson and Khan found that communities were Traditionally, health research in Aboriginal Potter refers to the initiative as her very comfortable coming to them with communities has been characterised by ‘touchstone'. In fact, her own approach questions and suggestions. According to a top-down approach. Khan explains that to evaluation has been molded by her Khan, this feedback was vital to the success usually such research is undertaken by observations of Watson's work in enacting of the initiative: ‘We were getting data all scientists from outside the community. Health Canada's mission of community along the way. We were trying to come Rarely is community feedback sought. empowerment and willingness to take risks. According to Potter, ‘The historical dynamic between the federal government and First Nations communities has not been positive. That's the big context that all this work happens in and we can't pretend it doesn't exist.'This project, however, fostered a truly collaborative spirit. In Khan's words, ‘It's usually hard to get people from different areas to collaborate. But not on this project.' Watson agrees. She characterises Patient Wait Time staff and community members in the community garden, part of wellness activities.
Evaluations that make a difference-EN-20Sep15.indd 52 21/09/2015 13:14:46 up with explanations in Ottawa, but we could be frustrating. Then I'd go to meetings stakeholders: the patients, the community could ask questions directly of the people or do teleconferences, and they'd tell me health professionals, the government providing the data. We could work together what's going on and inspire me to finish researchers, and the evaluation team. to find the answers much more readily and the programme. It's important to have the communication to be able to talk to the Watson, Khan, and Potter all agree easily. We were getting explanations from people processing the data.
that the evaluation affected how each the people who had the answers.' stakeholder engaged in future research This support reflects the collaborative spirit Even more important, for Khan, this with at-risk communities.
that marked the project under Watson's methodology demonstrated how having leadership and the evaluation team's Khan emphasises that the lessons he communities lead, and researchers finding facilitation.
took from this initiative, including the the best ways to support, showed what importance of helping communities to is possible in government-sponsored Turning it around – for good
take ownership of their own health care, research. The evaluation's face-to- The Patient Wait Times evaluation were instrumental in shaping the rest of face group meetings with participating had a long-lasting impact on all the communities at the end of data collection were a particularly powerful learning An annual run comprises an important part of the community's
experience for Khan.
health promotion and prevention activities.
As important as it was to hear first-hand about their challenges – travelling in winter, getting reimbursed for travel, staff turnover – it was critical to get real explanations for what we were learning. The real innovation was figuring out how to have First Nation communities lead, and then finding ways to support them.
In turn, the nurses collecting data found that their work was facilitated by the support of the programme's research team.
According to Chesne: Meeting and speaking with the researchers, especially Khan, inspired you to go back and do better. It was very time-consuming and Evaluations that make a difference-EN-20Sep15.indd 53 21/09/2015 13:14:46 In Potter's words, the project helped her Whose project is it? Whose data? There has stakeholders working together to make learn that successful evaluation ‘…is all to be room for the people who are affected. positive changes for both individuals and about building relationships and working in It's not just the government's story.' communities – provided the researchers that indigenous community development The Patient Wait Times initiative illustrates get out of the way and have the way: community-led, community-paced. that an evaluation can be a story about all communities take the lead.
Stephanie Potter, Lead Evaluator, SP Consulting and the Whetstone Group
Samir Khan, former Senior Research Analyst, First
Nations Inuit Health Branch, Health Canada
Judy Watson, former Project Lead, First Nations
Inuit Health Branch, Health Canada
Sara Pederson conducted interviews and crafted the evaluation story.
Evaluations that make a difference-EN-20Sep15.indd 54 21/09/2015 13:14:46

Source: http://www.clear-aa.co.za/wp-content/uploads/2015/10/Evaluations-that-make-a-difference-ENGLISH.pdf