Evaluations that make a
Evaluations that make a difference-EN-20Sep15.indd 1
Evaluations that make a difference-EN-20Sep15.indd 2
extent possible, from the perspective
The story of this project
of the users and beneficiaries who have
The issue of evaluation use is gaining
been involved in the profiled evaluations
What are evaluations that make a
currency, presuming a positive correlation
– a perspective that, strangely, has
between evaluation use and evaluation
rarely been heard, even in discussions
When we first set out to collect stories, little
value. But on the fundamental issue of
of evaluation utility. It is our hope that
did we know how challenging it would be to
determining the value of evaluation,
the stories will inspire policymakers,
describe what we meant by ‘evaluations that
the literature has been strangely silent.
managers, and programme staff to use
made a difference'.
What are the benefits? How are they
evaluation by demonstrating the potential
expressed? Can they be measured? Can
We were looking for examples of evaluations
benefits in a concrete and engaging way.
they be described in economic or other
that have contributed to social betterment
terms that make sense to citizens and to
We have also analysed the factors that
in some way. Many evaluations use
policymakers? What factors contribute
contributed to making the evaluations
good (even innovative) methods, with
to making an evaluation more or less
useful, yielding insights about the
participation by important stakeholders. In
enablers of a valuable evaluation. We
some cases, those evaluations get used to
learned about specific actions evaluators, inform decisions about programmes and
To help answer these questions, we've
policymakers, managers, and programme policies. But for this collection, we were
prepared this collection of short stories
staff can take to enhance the benefits
looking for something more. We were
about evaluations from around the world
of an evaluation. These actions are
seeking stories describing how an evaluation
that have made a difference to the lives of
led to positive changes in people's lives.
people. These stories collectively represent
a range of approaches to evaluation,
This proved a difficult concept to
making it clear that there is no one
grasp, so with the help of Chris Lysy of
and hope that readers will be able to
‘right' way to do evaluation that leads to
apply these insights to enhance the value
improvements in people's
explain the concept.
of their evaluations.
This collection is intended
and other potential
evaluation users, as well
as for evaluators. The
stories are told, to the
Evaluations that make a difference-EN-20Sep15.indd 3
the North American story, a process that
• European Evaluation Society (EES)
Potential stories were solicited through
culminated in a set of story development
• Canadian Evaluation Society (CES)
an international call for stories between
guidelines, which were then made available • African Evaluation Association (AfrEA)
January and May 2014. There was a high
to the other regions.
• Australasian Evaluation Society (AES)
level of interest, with many inquiries and a The completed stories were professionally • Red de Seguimiento, Evaluación y
total of 64 submissions. The Editorial Board edited and then translated into English,
Sistematización en América Latina y el
then assembled Regional Review Teams to French and Spanish.
review the submissions.
• Sri Lanka Evaluation Association (SLEvA)
The main selection criterion used by
reviewers was evaluation impact –
Spark and core funding
The Editorial Board provided strategic
evidence that the evaluation led to positive This project was supported through an
oversight for the project, advising about
changes in people's lives. Reviewers also
EvalPartners-Innovation Challenge grant,
the selection criteria, the story formats, etc.
looked for stories that would provide
which aimed to strengthen the demand
They leveraged their own networks to get
maximum opportunities for learning, as
for and use of evaluation to inform policy
the word out about the project, to solicit
well as submissions that would simply be
making during the International Year of
submissions, and to oversee the regional
engaging as stories.
review process. As a group, they made the
The Regional Review Teams each chose up We are grateful to Martha McGuire for
final story selections. They then recruited
to three submissions to recommend to the encouraging us to take on this Innovation
local story writers and oversaw the writing
Editorial Board. From these, the Editorial
Challenge, to Jim Rugh for his ongoing
process. Members of the Editorial Board
Board agreed on eight submissions to
support and guidance throughout, and
develop into stories, including at least one to the European Evaluation Society, and
Burt Perrin (co-chair) – EES
from each major region of the world.
in particular Eva Petrová, its Secretariat
Manager, for significant administrative
Rochelle Zorzi (co-chair) – CES
The Editorial Board then recruited
a professional story writer for each
Pablo Rodriguez-Bilella – ReLAC
story. The writers reviewed the original
This project was a strategic partnership of
submission documents and interviewed
evaluators from all major regions of the
Scott Bayley – AES
key stakeholders in the evaluation (usually world. The partners were:
about five). This process was piloted with
Evaluations that make a difference-EN-20Sep15.indd 4
Serge Eric Yakeu – AfrEA
Developing and piloting the story
Spanish translation was done by Pablo
Rodriguez-Bilella and reviewed by Ramon
Soma De Silva – SLEvA
The story development guidelines were
created by a subcommittee that included
Project coordination, editing and
Yasser Ismail (chair), Dayna Albert, Jessica
Regional Review Teams
Sperling, and Rochelle Zorzi. This group
Finally, we wish to recognize the
Each Regional Review Team reviewed the
worked closely with Stephanie Potter and
contributions of two individuals who
story submissions from their regions and
Sara Pederson, members of the North
played key roles in the project. The
recommended several to the Editorial
American story team who piloted the project. first of these is our editor, Eric McGaw
Board. Members of the Regional Review
([email protected]), who took
The African Development Bank provided
responsibility for the formatting of the
AfrEA: Serge Eric Yakeu Djiam (chair),
funding for French translation of the
final collection of stories (in all three
Samuel Kouakou, Ousseni Kinda, Awuor
evaluation stories, lessons learned and
languages), in addition to editing the
Ponge, Djelloul Saci
AES: Scott Bayley (chair), Vanessa Hood,
The Inter-American Development Bank
Last but certainly not least, this project
April Bennett, Kim Grey, Jessica Kenway
provided funding for Spanish translation of
could not have been completed without
the ongoing coordination provided by
EES: Ramon Crespo (chair), Burt Perrin,
the evaluation stories, lessons learned and
Dayna Albert ([email protected]),
Maria Bustelo, Murray Saunders
who managed just about everything that
CES: Rochelle Zorzi (chair), Pierre-Marc
Teleconferencing costs were contributed in-
needed managing, from the distribution
Daigneault, Tracy Fiander Trask, Marie
kind by Cathexis Consulting Inc.
of the call for proposals through to signing
Gervais, Kathryn Parker, Hallie Preskill
off on the last of the translation contracts.
ReLAC: Pablo Rodriguez-Bilella (chair),
French translation was done by Edmond
Both Dayna and Eric went above and
Thomaz Chinanca, Ignacio Irarrázaval,
Kembou and reviewed by a committee of
beyond the call of duty, and their
Claudia Maldonado, Inka Mattila, Rafael
volunteers, including Pierre-Marc Daigneault efforts and commitment are very much
(chair), Ann Royer, Raimi B. Osseni, Helene
SLEvA (Asia): Soma De Silva (chair),
Wirzbah, Ruth Chamberlain, Claire Bard,
Shobhini Mukerji, Ada Ocampo,Mallika
Véronique Dugas, Nora Habafy, and Alexandre
Evaluations that make a difference-EN-20Sep15.indd 5
What these stories tell us about collaborative effort.
Ways to enhance evaluation impact
how to do evaluations that can
How the evaluation
make a difference
is undertaken is
a big part of it.
These stories collectively have a lot to say
about how evaluations can be undertaken
• Ensure users
Evaluation users can.
so that they are most likely to have a
right, there are
• Sincerely care
positive impact on people's lives (what Mel several things
• Give voice to
Mark, Gary Henry, and George Julnes have evaluators can do
described as social betterment1).
to enhance impact.
• Provide credible
• Champion the
As we collected and developed the
• Use a positive
stories, the Editorial Board had several
users view, support,
opportunities for rich dialogue about the
factors that enabled these evaluations to
make a difference. The following ideas are
based on an analysis of themes in these
In the following sections, we
II. What evaluators can do
stories, discussions among members
elaborate on what evaluators
1. Focus on evaluation impact
of the Editorial Board, and interactive
and evaluation users can do to
sessions at the 2014 European Evaluation
influence evaluation impact, and
The focus of this project, and of the stories included,
Society conference in Dublin and the 2015 how interlinkages between the
has been on evaluations that could be shown, in
Canadian Evaluation Society conference in programme and the evaluation
some way, to contribute to changes in people's
Montreal where preliminary stories were
can support evaluation impact.
lives – to ‘social betterment' or ‘impact' as it is most
shared and discussed.
The distilled success factors
typically referred to in the evaluation literature. This
described here are not intended
is the raison d'être of evaluation, right? Otherwise,
I. Evaluators and evaluation users both
to represent a definitive list.
what is its value?
play important roles
Instead, they are meant to
It is important to appreciate that evaluation impact,
Doing an evaluation that leads to
provide a starting point for
as we've defined it, goes beyond ‘utilisation' and
improvements in people's lives is a
discussion as we build a collective more immediate or intermediary outcomes such as
understanding of how evaluations
1 Evaluation: An integrated approach for understanding,
changes to policies or programmes. While utilisation
guiding, and improving policies and programs. Jossey-Bass
can make a difference.
is, of course, important, it must not be the end goal.
Evaluations that make a difference-EN-20Sep15.indd 6
In the end, it is important that the changes work an intention to improve people's lives,
this evaluation, which for the first time
influenced by evaluation eventually lead
and work with evaluation users and other
gave voice to people across the country,
to improvements in people's lives, as the
programme stakeholders to structure the
many of whom only spoke their local
diagram on page 2 illustrates.
evaluation accordingly. Evaluation users can
language and could not benefit from the
Nevertheless, a major, indeed startling,
help by outlining the desired evaluation
radio programmes until the evaluation
finding is how difficult it is for evaluators
impact in the terms of reference.
identified the need for these to be offered
to think in terms of impact. Even as they
2. Give voice to the voiceless
in local languages and tailored to local
exhort programmes to think beyond initial
contexts. The evaluation led to significant
In many instances, those who make
outcomes, evaluators find it difficult to
changes in the programme, and in turn to
decisions about programmes or policies
do the same for their own work. Indeed,
improved lives (such as reductions in child
do not have an opportunity to observe the
despite clear guidelines, few of the 64
labour or girls being forced to marry at a
needs or contexts of the beneficiaries. They
submissions to the project even attempted
very early age).
are making their decisions, with the best of
to identify a connection between the
intentions, based on limited information and The Mexico story similarly indicates
evaluation and benefits to people.
assumptions that may be incorrect.
how the evaluation identified language
We found that some evaluators resisted the
barriers that prevented very poor
In these situations, evaluation can have
notion that they should think about how, or
indigenous people from being able to
a transformative impact on the lives of
if, their work has contributed to changes in
benefit from a programme. Changes
beneficiaries by giving them a voice, and
people's lives. ‘That's not my responsibility,
to the programme allowed for
carrying their words to decision-makers.
that's up to someone else,' they suggested.
communication in local languages, which
Many of the stories in this collection
‘I carried out a "good" evaluation [perhaps
greatly improved people's ability to
used evaluation approaches that enabled
one that was methodologically rigorous,
understand the programme requirements
beneficiaries to have their voices – and their
or one that influenced what programmes
(such as children's regular attendance
needs – heard for the first time.
were doing]. I can't be held accountable for
at school), and thus benefit from the
anything beyond that.'
This collection includes some dramatic
programme's cash transfers.
examples of evaluations that led to
Yet as the eight stories in this collection
3. Provide credible evidence
significant changes and improvements in the
illustrate, evaluation can have positive
lives of people, simply by letting people who As many of these stories demonstrate,
impacts on people's lives, and there are
were typically unheard tell their stories.
evidence must be seen as credible
things that evaluators can do to enhance
in order to bring about necessary
(or, on the flip side, to diminish) those
The title of the Nepal story: Listening to the
changes that result in improvements in
impacts. Evaluators can bring to their
listeners, says a lot about the approach of
people's lives. These stories collectively
Evaluations that make a difference-EN-20Sep15.indd 7
indicate that there is not just one right
stories from Canada and the Netherlands,
the need for an expanded childbirth
way to generate credible and persuasive
evaluation users' involvement in the
emergency phone service. The evaluation
information. What is seen as ‘credible'
evaluation process helped them relate to
of the Positive Sisters programme in
evidence depends on the situation and the the findings, making the findings more
the Netherlands validated many of the
meaningful and easier for them to act
approaches of volunteers and staff,
In some cases, it was important to assess
encouraging and indeed energising them to
programme outcomes with perceived
4. Use an approach that supports positive carry on with those approaches that were
thinking and action2
shown to have a positive impact.
In the story from Papua New Guinea, ‘hard' These stories also illustrate the value of
III. What evaluators and evaluation users
data about lives saved by a pilot childbirth a positive focus. Documenting what does
can do together
emergency phone service was an important and can work and should be continued,
5. Ensure users and intended beneficiaries
factor in convincing the authorities to
expanded, or modified, rather than
are engaged through a participatory
invest in the continuation of this service on focusing mainly on the inevitable glitches
approach to evaluation
an ongoing basis.
and shortcomings, avoids the quagmire of
Evaluators often try hard to keep their
In other situations, a credible evaluation
blame and defensiveness and moves the
distance in order to guard their ‘objectivity'.
involved capturing the perspectives of
discussion into solutions. The experience of The danger of evaluation done in this
these stories is consistent with the findings way is that it may become remote from
from the evaluation literature: evaluation
The Mexico story for example, illustrates
– and irrelevant to – those who would
is much more useful when it can provide
how a qualitative evaluation approach
act upon the evaluation findings to make
evidence about approaches that do work
was needed to demonstrate how the
improvements and to bring about change.
and need to be continued, expanded, or
Oportunidades programme, due in large
adapted. While many of the stories in the
By keeping their distance from the
part to language barriers, needed to
collection have documented the need
programme and its people, evaluators also
change to address the needs of very poor
for change, they all have gone beyond
pass up valuable opportunities to make
indigenous people (something that was
this, documenting what form of change is
a difference through process use. The
not apparent in previous quantitative
needed, and why.
evaluation literature demonstrates, very
forcefully, that benefits follow as much,
For example, the story from Papua New
In still other cases, credibility was achieved
or even more, from how the evaluation is
Guinea illustrates the value of documenting
through active engagement of the users in
done, as from the evaluation findings.3
2 This is the theme, for example, of a special section and
the evaluation process. For example, in the collection of articles in a recent issue of the Canadian Journal 3 For example: J. Brad Cousins, ed., Process use in theory,
of Program Evaluation (vol.29, no.2, 2014).
research and practice, New Directions for Evaluation, No.
Evaluations that make a difference-EN-20Sep15.indd 8
By being involved in an evaluation, people
about her HIV/AIDS status before ShivA, and Similarly, the collaborative approach taken
can gain insights, get to know others, begin now recognises that being interviewed was
in the Kenya community sanitation project
to see things from a new perspective, and
an extremely positive experience. ‘Being
led to the community taking over the
strengthen their abilities. The accumulation interviewed helped me feel important and
evaluation, reflecting their belief that if
of these changes, among a group of people, part of something good.'
positive action was going to take place, it
can have a strong positive effect that is
has to ‘come about from the grass roots,
For Liako, being interviewed meant
quite independent from any evaluation
not from outsiders.'
becoming more aware of the impact of her
work since ShivA: ‘The evaluation had a big
Importantly, active engagement in the
One way to engage users and beneficiaries impact on me. It was a Wow! moment for
evaluation process helps develop a
in an evaluation is by having them tell their me. We are really doing a great job.'
better understanding of evaluation, and
stories. By sharing stories with one another,
contributes to commitment and buy-
Community participation in gathering and
people form relationships, strengthen
in. Those who have been involved in an
using evaluative data is another powerful
networks, and set up informal knowledge
evaluation have more commitment to act
way of engaging users and beneficiaries.
transfer channels that need not go through
upon its findings and implications.
The story from Canada and the Kenya
‘experts'. In addition, they can also develop community sanitation story demonstrate
In summary, these stories collectively
a sense of community and a feeling of
the transformative impact of community
demonstrate that participation yields
being part of something larger than
involvement in data collection. In both
strong benefits, regardless of what form
cases, the communities felt increasing
it takes. This requires an openness by
The evaluation of the Positive Sisters
ownership of their data, which empowered
evaluation users to actively engage
programme in the Netherlands not only
them to take charge of and make use of
in evaluation, and it also requires a
provided direction to the project to
willingness of evaluators to adopt
increase its impact; the process energised
approaches to evaluation that allow for
As the evaluator in the Canadian story
its volunteers and participants in a way that
active engagement and participation.
observed, ‘Whose project is it? Whose data?
traditional approaches to evaluation rarely There has to be room for the people who
6. Embed evaluation within the
are affected. It's not just the government's
Millie, for example, had been quite shy
story.' In this example, being involved in the An interesting characteristic of three of
116, Jossey-Bass, 2007.
data collection gave nurses the evidence
the stories is that evaluation and the
For broader consideration about participatory approaches
they needed to ‘overhaul the community's
programme were so closely linked that it
to evaluation: Elisabeth Whitmore, Understanding and
prenatal and postnatal health care, setting
sometimes was difficult to say where the
practicing participatory evaluation, New Directions in
Evaluation, No. 80, Jossey-Bass, 1998.
the stage for a healthier future.'
programme stopped and the evaluation
Evaluations that make a difference-EN-20Sep15.indd 9
began. Both stories from Kenya featured
time goes on, evaluation can help users
sustainably funded by the provincial health
evaluations that were embedded in the
find out what effects the changes had,
authority Naidi headed, he needed proof.'
and further (better-informed) refinements In the story from the Netherlands, we
The Community and Progress Youth
can be made if needed. These stories also heard that ‘Inga was hungry for insights and
Empowerment Institute (CAP YEI) made use illustrate how evaluators often worked
feedback and quick to apply them.'
of internal and external evaluation to aid in very closely with programme staff to
And in the story from Mexico, the
ongoing improvements to the programme,
support implementation of the findings.
coordinator of the Oportunidades
contributing to better employment
IV. What users can do
programme, Salvador Escobedo, despite
and financial outcomes for its youth
7. Really care about the evaluation
having intimate knowledge of the indigenous
Sometimes, people do evaluations only
communities and their reality, knew that
A more dramatic example is the community to fulfil accountability requirements. They without evidence that both the language
sanitation project in the village of Murihi
go through the motions, get the findings,
barrier and the targeting procedures
wa bibi in rural Kenya. The evaluation was,
and submit the report to their funder.
were serious impediments to successful
in truth, part of the intervention and not
The report sits on a shelf unused, and the implementation of the programme, he would
a separate activity, and it played a key role
programme continues on undisturbed.
be unable to make and sustain the needed
in achieving a dramatic reduction in open
changes in the programme.
defecation practices. As part of the project, That is not what happened with the
This wanting of the evaluation, we believe,
members of the community gathered to
evaluations in this collection of stories,
made it possible for the evaluation to make
analyse and interpret the data to assess
where someone really wanted the
a difference in people's lives. As Michael
their progress. Their day-to-day interaction
evaluation for the learning and guidance
Patton4 has indicated:
with the evaluation findings inspired a spirit that it could provide. Each evaluation was
of friendly competition, so that people
valued because it enabled users to make
The personal factor is the presence of an
soon wanted to achieve open defecation
better decisions or advocate for change.
identifiable individual or group of people
who personally care about the evaluation
free status before their neighbouring
Here are just three examples from among
and the findings it generates. Where such a
person or group was present, evaluations were
When evaluations are designed to collect
In Papua New Guinea, Billy Naidi had
used; where the personal factor was absent,
research on evaluation use consistently shows a
data and feed results back on a regular
heard positive things about the emergency
correspondingly marked absence of evaluation
basis, stakeholders can make changes
phone service, but he ‘needed some
long before the final report is written (if
hard evidence that the emergency line
indeed there even is a final report). As
was working. For an initiative to become
4 Developmental evaluation: applying complexity concepts to
enhance innovation and use, Guildford Press, 2011, p.56.
Evaluations that make a difference-EN-20Sep15.indd 10
8. Champion the evaluation with decision
impoverished households. The story tells us 1. What is needed to create enabling
that it is ‘nothing less than astonishing' that environments where high-quality credible
Programme staff and managers who have
they were successful. ‘Never before had
evaluations influence decisions at all
been involved in the evaluation might
this government body granted permission
levels? Future work in this area might
be committed to making changes to the
to anyone to access timber.' Such is the
build on what is already being done with
programme. But often, the decision makers
power of having committed evaluation
parliamentarians5 as well as the efforts of
with real authority are external to the
evaluation associations to educate both
programme and haven't been engaged in the The champions in these stories were
evaluators and users.
people who cared passionately about the
2. What are the expected competencies
What happens when evaluation users need
families and communities affected, and
evaluators need to make a difference in
to convince other stakeholders that a change who had influence with those who could
people's lives? For example, maybe it is
make decisions. And they clearly played
time for the discipline to make an effort
an instrumental role in getting changes
to better describe what is really meant by
Well, that's when it's helpful to have a
made. The stories do not suggest that the
upholding democratic values in conducting
champion who can influence key decision
evaluators deliberately cultivated these
and reporting evaluations (a widely
makers and push for the needed changes. We champions, but it may well be worthwhile recognized disposition for an evaluator
can see the power of champions in several of for evaluators and users to identify and
collected in several ‘evaluation capability
nurture such champions more intentionally. frameworks') that goes beyond but
In Sri Lanka, the secretary of the small
V. Moving forward
certainly includes the need for evaluators
enterprise ministry ‘seized on the findings…
to take an impact orientation to their work
He took the evaluation to all relevant cabinet These stories are just the beginning of
and to facilitate good communication
ministers to secure their support. With the
understanding how evaluations can make a and indeed engagement of stakeholders
cabinet on side, there was no way the issues difference. Beyond the 2015 International
throughout the evaluation process.
raised by the evaluation could be ignored.'
Year of Evaluation, it will be important to
This paved the way for significant changes
continue the research and test some of the
to the Industrial Estates, which improved
theories that have emerged through the
conditions for small and medium businesses. stories in this collection.
In Kenya, monitoring committees brought
their progress data to forest authorities
We pose two questions to provide
to request timber to build latrines for
inspiration for future efforts:
For example, through the Global Parliamentarian Forum for
Evaluation that is being formally launched in November 2015
at the EvalPartners Global Forum (www.pfde.net).
Evaluations that make a difference-EN-20Sep15.indd 11
Salvaging Sri Lanka's small and
some more than doubling year after
R.S. Balanathan, the managing director
year. And even when rents were paid,
of handloom silk exporter Ko Lanka, says
how an evaluation led to rapid change
there seemed to be no correlation to the
all the industries in the estates were
quality of infrastructure. Roads were more undeveloped. ‘The roadways were bad,
Since the end of Sri Lanka's long civil
potholes than asphalt, electricity was
and the rents were very high,' he says.
war in 2009, the nation's economy has
unreliable, and coordinated marketing
recovered at a startling rate. GDP growth
simply lacked focus. Although the
In 2013, the situation began to change. The
now hovers between 6% and 8%. Peace
government was investing heavily in
Ministry of Traditional Industries and Small
has certainly brought dividends – tourism
national infrastructure, smaller businesses Enterprise Development commissioned a
has rebounded, China has made large
appeared to have been overlooked.
comprehensive evaluation of the sector in
investments in infrastructure, and trains
once again run the length of the pearl-
shaped island in the Indian Ocean, just off
the southern coast of India.
But it's not all good news. As peace
returned, small and medium enterprises
(SMEs) found to their chagrin that
China, India, and Bangladesh had all but
cornered the world's export markets in
mass manufacturing and textiles – at their
expense. To add to their woes, the national
government had paid little attention to the
sector over the years. For the hundreds
of business owners renting land in the
18 government-owned industrial estates
managed by the Industrial Development
Board (IDB), the economic boom seemed
to have all but passed them by.
The list of complaints was long and
detailed. Rents were out of control, with
Handloom industry at Pallekelle Industrial Estate
Evaluations that make a difference-EN-20Sep15.indd 12
rural poor – who provided much of the
workforce in the estates – also suffered.
Silk showroom at Pallekelle
The evaluation report gave voice to
longstanding grievances and recommended
sweeping changes. What was needed,
argued Ekanayaka, was direct intervention
by the national government. The secretary
of the small enterprise ministry, Velayuthan
Sivagnanasothy, seized on the findings. He
The evaluation showed that the IDB was
a mere rent collector and not improving
the estates. With an evaluation like this,
change would be far easier to generate. The
evaluation process was a platform serving to
carry the voices of the voiceless to the highest level of policy makers.
He took the evaluation to all relevant
2013. A team of independent evaluators
broad problems that came up again
cabinet ministers to secure their support.
led by Ajith Ekanayaka began travelling to
and again were decaying infrastructure,
With the cabinet on side, there was no way
each of the 18 estates across the country
exorbitant rents, and expensive loans.
the issues raised by the evaluation could be
to examine the issues plaguing the sector
Some had special concerns. Metal
and to suggest direct solutions. Ekanayaka industrialists, for instance, could not lay
And so it came to pass. The ministry drafted
made it clear that this evaluation would
their hands on raw materials because
a national policy on SME development,
not be relegated to a filing cabinet but
almost all scrap metal was being exported. consulting widely and drawing on the
would be put to immediate use.
The evaluation also found that some
expertise and advice of other ministries
For five months, the evaluators gathered
estates were being used to build housing
and agencies. But even before the policy
their material, forming focus groups for
by the owners of failed SMEs. The long-
could be adopted, change began to ripple
industrial associations and interviewing
term weakness of the sector meant that
outwards. The evaluation's actionable
managers and district officers. The three
employment for ethnic minorities and the
recommendations had buy-in. Why?
Evaluations that make a difference-EN-20Sep15.indd 13
Shoes at Baddegama Industrial Estate
Aluminium ware at Kaludwela Industrial Estate
Because policymakers had given frequent
periods meant that SMEs could now expand
in order to shift from a reactive to a
input through scoping sessions, and the
their operations more easily. In each of
proactive stance. The IDB now focuses
evaluation was built on a broad, credible
the estates, factory production is up, and
on training, access to raw materials (such
base of evidence and data as well as strong
with it, SME profits, leading to more jobs.
It's trickled down to the rural villages.
as previously scarce scrap metal), and
connections to stakeholders. According
access to credit. Not only that, but the
to Sivagnanasothy, a great deal has come
A tangible shift in attitude amongst IDB
rental income earmarked for infrastructure
from this single evaluation. He recalls:
management towards the SMEs was
improvement around the estates was
confirmed by Ratnamalala, who praised
When the rents went up, industrialists
increased tenfold, from as little as 2–3%
used to shout and cry How can we afford
the evaluation for what it revealed.
percent to 20–30% of the rent paid.
it? We were able to convince the central
‘We expanded our service delivery and
To address the problem of access to
government to cap rent increases at 7%
created a shift in mindsets of our officers
annually. That's a predictable environment,
to be more development oriented.'
credit, the government rolled out a special
which is an enabling environment. Not
The shift for the IDB was significant, with
scheme with concessionary rates to boost
only that, but the introduction of targeted
the agency undergoing a restructure
access. Joint marketing schemes were
loans with lower interest and longer grace
introduced to give potential buyers a
Evaluations that make a difference-EN-20Sep15.indd 14
streamlined way of examining the range
million), with more on the way.
For silk exporter Balanathan, the result
and quality of products. Ailing SMEs can
For IDB director general Ratnamalala,
has been more cash in his pocket – and
now gain access to a ‘nursing programme'
another benefit of the evaluation is simply that means expansion into his key markets
to bring them back to health, while the
being heard. ‘We have direct access to
in Europe and the Middle East. From 100
rules for gaining entry have been made
central government,' he says. ‘Now, with
employees in 2013, he now employs 130.
more stringent to cut down on residential
these improvements, we will see growth in ‘We have strong competition with China
use of industrial estates. And the metal
our industrial estates.'
and Bangladesh,' he says, ‘but these
industries were given a lifeline at the
On the ground, results of these rapid
changes make us more competitive. Now,
expense of the scrap merchants, with a
evaluation-led changes have been
we have higher silk production than they
ban on the export of scrap. The amount
impressive. The voices of small and
do. We can beat them!'
injected into infrastructure upgrades
medium industrialists have been heard,
immediately following the evaluation was
and the lines of communication with
305 million Sri Lankan rupees (US$2.3
government are open.
Secretary, Ministry of Traditional
Industries and Small Enterprise
hotmail.com) commissioned the
evaluation of the SME Industrial Estate
Programme and used the evaluation
recommendations that led to policy
changes and programmatic actions
that made changes in people's lives.
Ajith Ekanayake, Independent M&E
Consultant, led the evaluation team.
Vaidehi Anushyanthan, Assistant
Director, Ministry of Finance, co-
chaired the evaluation as focal
point of the Government's National
Story writers: Jessica Kenway
and Doug Hendrie.
Newly constructed road at Pallekelle Industrial Estate. No potholes!
Evaluations that make a difference-EN-20Sep15.indd 15
Tumekataa kula mavi tena!
Let us step back a few years to World Toilet
We refuse to eat shit!
Day in 2011. We're in Murihi wa bibi, a
village in the highlands of Kwale county
Of the world's seven billion people, only
along Kenya's low-lying coastal strip.
4.5 billion have access to toilets or latrines. Small children chase cockerels, which in
The remaining 2.5 billion, most of whom
a few hours will form part of the meal for
live in rural areas, lack proper sanitation.
the celebrations. Excitement is in the air.
And nothing spreads disease faster than
Why are the villagers of Murihi wa bibi
open defecation. Indeed, Millennium
celebrating? Quite simply, they are proud
Development Goal Number 7 is to halve
and exultant that there are no longer heaps
the population of people living without
of human excrement in the bushes. The
adequate sanitation. Hence the idea for
community here, convinced that their age-
World Toilet Day, which takes place on 19
old tradition of shitting in the bush can no
November each year.
longer be tolerated, has achieved a new
level of freedom –
freedom from disease.
The practice had
to stop; there was
Field monitoring at Ngerenya village, Kilifi county.
Njoroge (in striped polo shirt) with Prof Robert
continue. Not with
Chambers of IDS Sussex and other colleagues.
such champions as
practice open defecation in areas used
Kingi Mapenzi, Peter
for living and farming. Kingi and Mbith
are local community health workers
Josephine Mbith, who while Mwambaka is the local government
went from household administrator. The three of them are typical
of the network of local volunteers who
work tirelessly to convince villagers to stop
of the need to stop
eating shit, which
is what happens in
Not that they are the first to do this.
CLTS follow-up in Mazumalume village, Kwale county. Kingi Mapenzi is on
the far left.
Sanitation has long been recognised as
Evaluations that make a difference-EN-20Sep15.indd 16
behind – to let
The Alliance raises funds from a
the communities membership of 10 core individuals.
These individuals are regular employees
of various organisations and agencies.
efforts to solve
Although their expertise ranges from
monitoring and evaluation to epidemiology,
the common denominator is community
health. Acting as individuals, each of
the 10 members takes on remunerated
consultancies from which they donate time
and funding to Alliance activities. And since
they work through and with community
Health Advocates groups, the costs are minimal.
It all started in 2007. An international
Sanitation Monitoring Committee, Kafuduni village, Kwale county
NGO had triggered the project at Jaribuni
a serious health problem in Kenya. For
a legal entity with the government, but
village in Kilifi, a neighbouring county
years, dozens of local and international
from the beginning the idea was to eschew north of Mombasa. During the triggering
organisations have attacked the scourge
branding and notoriety. The thinking
session, a skilled facilitator, using various
of open defecation from a variety of
was: If sanitation is going to work, it has
directions. The results have been varied,
to come about from the grass
but more often than not, the daunting task roots, not from outsiders. After
confronting these organisations put paid to all, villagers don't care about
their good intentions.
the names of organisations.
Realising that the task was simply beyond
To emphasise the primary
the reach of any single organization,
role of the community in this
a number of concerned individuals
venture, the alliance adopted
decided to pool their efforts – and more
the term community-led total
importantly, to position themselves
sanitation (CLTS) from a project
as partners, not leaders, of the target
in Bangladesh, where CLTS had
Usinye msituni – ‘Do not
proved successful in addressing
shit in the bush'
communities. The idea was to lead from
Evaluations that make a difference-EN-20Sep15.indd 17
achievement, the residents and volunteers Monitoring is essential if a triggered village
organized a simple ceremony. They invited is to attain ODF status. This is a lesson
local health officials, who were extremely
that CLTS practitioners in Kwale and Kilifi
impressed by the simplicity of the approach counties learned the hard way. Initially,
– empowering communities to analyse
they thought that a triggered village would
their own sanitation profile and to make
automatically translate into ODF status.
decisions based on the realisation that they With time, the importance of monitoring
were literally eating each other's shit due
– and monitoring committees – gained
to open defecation. And the cry went up:
recognition. NGO workers designed
Tumekataa kula mavi tena! (‘We refuse to
monitoring tools that the committees could
eat shit any longer!' in Swahili).
use to gauge progress and identify which
As the CLTS movement spread across
households required special attention.
Kenya, especially the coastal region,
Through this process, female-headed
communities remained at the forefront.
households and households with very old
It doesn't work any other way. Passionate
people or people living with disabilities
community workers like Kingi, Mbith, and
were identified. In many places, the
Mwambaka, who tirelessly monitored the
monitoring committees mobilised young
situation every day, provided the spark that people to contribute labour by constructing
participatory methodologies, led the
ignited the process. But they knew that to
toilets for households with occupants who
villagers to understand the terrible
achieve total sanitation
consequences of open defecation. The
the community had
residents of Jaribuni collectively resolved to to truly embrace the
stop open defecation by building and using idea. By explaining why
latrines in a time-bound campaign led by a ODF was necessary,
local committee. They gave themselves 90 they provided the
days to have everyone in the village using a spark. But once the
latrine and the local monitoring committee engine was running
went around documenting the progress.
with the community in
The village achieved open defecation
the driver's seat, their
free (ODF) status just 67 days after the
job was confined to
triggering session. To showcase this
Recognizing the monitoring team.
Evaluations that make a difference-EN-20Sep15.indd 18
lacked mobility. In cases where a household forget to wash their hands after latrine
could not afford construction materials, the use, introduced hand washing facilities
committee approached forest authorities
with soap or ash next to the latrine. Rival
with the progress data to show that these
villages insisted on assigning participants
few households required special support to in the verification missions sent out to
access such construction materials as logs. inspect communities for ODF status. They
That they were successful in this endeavour even visited the bushes previously used for
is nothing less than astonishing. Never
open defecation to verify the absence of
before had this government body granted
faeces. In one case, where all households in
permission to anyone to access timber.
a Penda Nguo, a village in Kilifi county were
Such is the power of placing data in the
found to have latrines, the sanitation and
hands of aggressive community members!
hygiene promotion experts had decided
Once a village attained ODF status,
amongst themselves to confer ODF status
the residents, to show pride in their
on the village. But when a member of the
achievement, erected signposts proclaiming committee from a rival village discovered
Administrator Peter Mwambaka addressing ODF
their achievement to the whole world
a pile of fresh faeces in the bush, the
celebrations at Muririhi wa Bibi village.
and warning visitors that open defecation
committee was left powerless to act. The
celebrate their collective achievement. It
would not be tolerated in their turf.
villagers eventually decided to build a
public latrine and they were eventually
was the first time that such a large number
Villages even started competing with
declared ODF when no more signs of fresh of villages would celebrate collectively. CLTS
one another. To outdo each other, they
faeces were found. These are not the
practitioners call it ‘strategic noise'. When a
kept raising the bar. They said that the
actions of complacent people.
village – in this case seven villages – come
monitoring indicators put together by
together to formalise their refusal to eat
hygiene promotion experts were not
Without the active participation of
shit, their neighbours have no option but
comprehensive enough to truly stop
communities in monitoring total sanitation, to follow suit – to come and hear about the
residents from eating shit. They pointed
the hard work and dedication of health
achievement and join in the strategic noise
out that merely building and using a
workers would result in nothing but tired
latrine was inadequate. One community
bodies and disillusionment.
No less than the regional director of health
fabricated aperture covers to prevent
Which brings us back to World Toilet Day
is invited to witness their joy of saying no
flies from breeding in the pits. Another
2011. A total of seven villages of Kwale
to unsanitary living. The strategic noise
community, to ensure the users did not
county agreed to meet in Murihi wa bibi to includes testimonials from villagers. Mzee
Evaluations that make a difference-EN-20Sep15.indd 19
Hamadi recalls how squatting out in the
bush robbed him of his dignity. The fear
of stepping on snakes at night and the
nightmare of finding a dry spot in the
grassy patch during the rainy season was
too much for him. When the movement
swept his village, he was more than glad to
join and stop open defecation.
His neighbour, Yusuf Ali, tells a different
story. ‘At first,' he said ‘I was not so glad.
I didn't see the importance of wasting
effort and resources to build a house for
shit.' However, his wife noticed that their
2-year-old daughter Fatuma, who suffered
chronic diarrhaea before Yusuf reluctantly
built the family latrine, was now active and
healthy. For the last 3 months, she informs As a roving community health worker
Celebrating ODF achievement
the cheering gathering, Fatuma has not had he knows his energy will be tested in
diarrhaea. Even the health workers in the
Katangini, a village on the other side of the
Mazumalume dispensary say they miss her hill. But for now, he can relax and enjoy a
because she no longer has to visit them.
Redempta Muendo, Kwale
sumptuous meal of pilau and chicken stew.
County Public Health Officer
Siku hizi, twaenda hospitali kwa chanjo za
Remember those cockerels?
Haron Njiru, Programs
kakake mdogo. (‘These days, we only visit
The revelry is well deserved. And what's
Director, HEADS Alliance
the clinic for immunising Fatuma's little
more? Unlike the last 2 years when Kingi
Kingi Mapenzi, Peter Mwambaka, and
had to travel to the next county, this year's
The ceremony ends up with awarding
Josephine Mbith: on the ground
World Toilet Day celebration is happening
of certificates to the ever-committed
right here in his village. What an honour,
Story writers: Njoroge Kamau
monitoring committee. Expressing his
what a great way to put the finishing
and Eric McGaw
excitement, Kingi exclaims ‘Even my most
touches on a successful year – and to look
troublesome set of villages are finally ODF!' forward to an even better one!
Evaluations that make a difference-EN-20Sep15.indd 20
PAPUA NEW GUINEA
Evaluation in action
of 7 million, a third of whom
The Milne Bay emergency phone service
are females of child-bearing
age, are isolated from the major
For expectant mothers in the mountainous population centres. Rural health
Pacific nation of Papua New Guinea (known care workers do their best, but
popularly as PNG by its inhabitants),
if something goes wrong, they
giving birth is an exercise in luck. If your
may not know what to do.
labour is short, and you experience no
The seriousness of the situation
complications, luck is in your favour. But if
is illustrated by PNG's maternal
luck is against you, if your labour is difficult, mortality rate, which is among
if the baby is stuck, if you bleed too much – the highest in the world. For
you can only hope that you and your baby
every 100,000 live births, 733
will survive. Because you are very, very far women die. By comparison, in
from the nearest hospital.
Australia – the country's nearest Amanda Watson and Gaius Sabumei being interviewed about
the project at the local radio station in Alotau.
For the tens of thousands of mothers who neighbour – the death rate is only 6.8.
and expertise to repair them were lacking.
go into labour in remote villages each year, The Childbirth Emergency Phone
These were uncharted waters for PNG.
babies are born at home or at the local
programme was designed to tackle this
Could the promise of mobile technology
clinic. Almost all of the country's population grim problem. In 2007, PNG had only
really transcend the tyranny of distance? To
50,000 phone lines for the entire country
find out, a pilot project was paired with a
– one phone for every 140 persons. Seven
years later, 2.7 million people had mobile
The idea for the project began with a
phones – one for every 2.6 persons.
medical specialist at the University of
This was real progress. It meant that for
Papua New Guinea named Glen Mola.
the first time rural health workers could
Professor Mola believed that the best
be reliably connected to specialists in the
place to run and evaluate a time-limited
labour wards of major towns. A previous
pilot project would be Milne Bay Province,
initiative to link rural clinics to major
where the maternal mortality rate was
centres via radio had achieved some
even higher than the national average.
Alice Siwawata pauses to catch her breath at success, but this became unreliable when
Because the province constitutes the
Alotau Provincial Hospital. the radios broke down because the parts
mountainous eastern peninsula of PNG's
Evaluations that make a difference-EN-20Sep15.indd 21
were also provided to
calls. ‘It was very difficult previously,
recharge the mobile
communications-wise,' she says. ‘During
phones. Calls started
emergencies now, it's like having a labour
coming in almost
ward doctor in the rural clinics.'
immediately. Reports of The Australian government made a
breach births, excessive comprehensive evaluation part of the pilot.
‘That's not common in development work,'
says Watson. ‘But for this one, evaluation
retained placentas –
was very important. Australia wanted to
all potentially lethal
know what could be taken from it.'
without the right
information at the
While the initial anecdotal feedback was
right time – could
positive, Billy Naidi needed some hard
now be addressed in
response to isolated,
Alotau Provincial Hospital labour ward staff practising answering calls.
workers who needed
main island, as well as hundreds of remote help in managing difficult deliveries. Each
islands, health care workers faced the
month, an average of 17 calls were being
challenge of extreme isolation.
made – each concerning a potentially life-
After discussion with Mola, Australian
researcher Dr Amanda Watson set up
‘This project is saving two lives, mother and
the pilot project in the labour ward of
child, and we are thankful,' says one rural
the Alotau Provincial Hospital in the
provincial capital in late 2012. She and
Billy Naidi, the chief executive officer of
‘Once, I saw a mother die in front of me,
the province's health authority, set up a
simply because of lack of communication,'
dedicated landline phone in the middle of
says another. ‘Today, I can call anywhere,
the bustling labour ward and sent the toll-
free number to rural health care workers in Alotau health extension officer Alice
Billy Naidi and Rob Brink mark the handover of the
remote locations. Importantly, solar panels Siwawata fielded many of these
project from Australia to the local health authority.
Evaluations that make a difference-EN-20Sep15.indd 22
Sabumei travelled across the province,
interviewing rural health workers, recent
mothers and village leaders in remote
areas, as well as the labour ward staff at
Alotau Provincial Hospital where the phone
line was based. He returned from his foray
through the province with 44 interviews
in his pocket. These interviews almost
universally praised the pilot.
One rural worker heard of the free call
service just before a woman presented
with a retained placenta. ‘If this project
was not here and I didn't have any means
to communicate, we could have lost
this mother because she was already
septicaemic,' said the worker. ‘From
anywhere, at any point, I can seek advice
Gaius Sabumei interviewing rural health worker Rose Elliot at Omarakana Health Centre.
and help the patients.'
evidence that the emergency line was
proof that would come through a detailed
working. Ever since Watson had pitched the evaluation of the pilot. With a thorough
Almost all the interviews reflected positive
idea to him, he'd become interested in the evaluation of the project in his pocket, he
responses. And even when a few of the
health care possibilities opening up with
could justify taking over the funding of the respondents had something critical to say,
the rapid expansion of PNG's mobile phone phone line. ‘We wanted to see what has
they readily admitted that things were
network. Australia's aid programme was
happened,' says Naidi. Watson understood working better than before. For example,
funding the pilot project, but these funds
his position and agreed to switch roles and one labour ward staff member said
would soon dry up. (Aid from Australia,
evaluate the project she'd set up.
‘Answering an emergency call when you are
PNG's former colonial overseer, still
busy attending to a mother in delivery is
accounts for around 13% of the nation's
Watson began the evaluation process by
sometimes frustrating, but it's part of our
revenues.) But for an initiative to become
hiring a local field officer and evaluator
work.' Beyond that, criticism was muted.
sustainably funded by the provincial health named Gaius Sabumei, who came to know Another labour ward member said that
authority Naidi headed, he needed proof – Milne Bay Province well. For months,
the initiative helped build relationships
Evaluations that make a difference-EN-20Sep15.indd 23
and distaste for seeking help
phone line was superior to the ailing
from clinics amongst some
radio network. While it was impossible to
cultures on outlying islands,
estimate how many lives had been saved, it
the phone line had saved lives.
was clear from the clinical evaluation that
Dr Bintol wrote: ‘There is very
some women would have died without the
good evidence from the cases
service, and that rural health care workers
that lives of both mothers and
felt the initiative reduced their isolation.
babies have been saved. It was
‘I wanted to know that it was effective.
also evident that information
Eventually we were convinced, and we
exchanges in the phone
made a commitment to sustain it.'
calls were very educational,
And with that, the phone line's future was
especially for the rural health
secured. After two handover ceremonies,
staff who may encounter a
in Alotau and in the nation's capital of Port
similar problem in the next
Moresby, the Childbirth Emergency Phone
event.' The evaluation also
line at last became an ongoing, sustainable
demonstrated that the advice
given over the phone was useful
and appropriate in most cases,
What has surprised Watson has been the
Rose Elliot of Omarakana Health Centre is delighted that she
can charge her mobile phone with a solar charger provided by and inappropriate (and thus
unexpected spin-offs from the evaluation.
potentially harmful) in 16% of
The lengthy report led to national and
Australian media attention on maternal
between the metropolitan staff and the
the cases. This led Watson and Sabumei to mortality. Newspapers, radio, and
recommend follow-up training to address
television all reported the evaluation's
the types of cases where bad advice had
In addition, Watson sought external
findings. ‘The evaluation gave people ideas
validation from a health professional from
for the use of phone lines in health more
a different province. Dr Derick Bintol, a PNG When the evaluation was given to Naidi, he broadly,' she says. In the mountainous
national, listened to a sample of recorded
examined it carefully. ‘From the beginning, I Western Highlands Province, for example, a
phone calls and analysed the clinical notes liked the idea. I was excited,' he says. What broader scheme is now up and running – a
from a sample of 68 calls.
he wanted to know, though, was whether
call centre staffed by trained nurses who
the line worked as well as the anecdotal
The evaluation found that despite the
answer health questions from the general
feedback suggested, and whether the
challenges of phone network downtime
Evaluations that make a difference-EN-20Sep15.indd 24
The Childbirth Emergency Phone is
a project of the Milne Bay Provincial
Health Authority. Establishment
and the evaluation were funded by
the Government of Australia.
This story was written by Jessica Kenway
and Doug Hendrie. Drafts were reviewed
by Billy Naidi, Chief Executive Officer
of the Milne Bay Provincial Health
Authority, and Amanda Watson, Mobile
Communications Research Consultant,
Economic and Public Sector Program.
Photographs were taken by Rawena
Russell and Amanda Watson.
More information on the project
is available at www.pngepsp.
Performance group celebrating project launch in Alotau.
For Billy Naidi, the value of the evaluation
confronting a challenging situation, and
has also been demonstrated by proving
you can be in an intimate conversation – as
the effectiveness of breaking down Papua
if you're standing next to someone in the
New Guinea's millennia-old isolation using labour ward,' he says. ‘After the evaluation,
modern technology. ‘With this phone
we were overwhelmingly happy!'
number, you can be in a remote area,
Evaluations that make a difference-EN-20Sep15.indd 25
Positive Sisters: a transformative surprise people who first meet her over the
phone. ‘I started doing this work in 1996
when my gay friends, all white men, were
When you find the right direction in life
dying from the disease. I felt a sense of
it can be contagious. Inga didn't realise it
though. To her, she was doing what was
After working with various religious
right, but little did she know how many
organisations for a few years, Inga felt that
people would blossom from being in
the restrictions and expectations placed on
contact with her. Her clarity of purpose also her diluted her sense of purpose. She felt
helped build the support she needed.
a need to choose her work based on what
Inga Mielitz is an atypical Christian minister was important to her.
in the Netherlands who works with people
They didn't want to hear that discrimination
affected by HIV. ‘I'm a dyke,' she says
of people with HIV has its roots in the
with unabashed candour. Her razor-short
negative way our churches talk about
bleached hair and assertive demeanour
sexuality; they didn't like my openness. But
Volunteers receive their certificates from project
leader Inga Mielitz.
life is precious, and I wanted to do what I
consider important in my life instead of doing
what others expect from me. I believe that
God wants every person to be who they are
instead of pretending to be someone else. I
still wanted to work with HIV-positive people,
but I wanted to do it my way. Put together,
sexuality and faith make a connection to
something bigger than oneself.
Shortly thereafter, ShivA – an abbreviation
for Spirituality, HIV, and AIDS – was born.
The aim of this Netherlands organisation
Taking a break on the beach during a training workshop.
was to improve the quality of life of HIV-
positive people and their loved ones
Evaluations that make a difference-EN-20Sep15.indd 26
through empowerment and by supporting
people seeking meaning and spirituality.
The idea was to provide a ready response
for people asking Where do I find the
strength to go on? Positive Sisters is a ShivA
programme that provides support for HIV-
positive people, especially migrant African
and Caribbean women. Most of the women
are referred to ShivA by hospitals where
they've gone for treatment.
The gift of evaluation
After 4 years working with African and
Caribbean women and 2 with the Positive
Sisters project, having enrolled more
Getting the participants together with zumba dancing.
than 150 women and trained 18 Positive
Sisters, Inga met two staff members of an
the stories to identify the ones that felt
Liako is a lively speaker and wears a
evaluation firm called Results in Health
most meaningful to them. The evaluation
broad smile. Her hands and body move
at National AIDS day in Amsterdam.
team then analysed the data and compiled rhythmically with her words as she speaks.
Aryanti Radyowijati and Maaike Esselink,
a report. Finally they shared results
Her dark skin looks translucent. She exudes
impressed by Inga's story and touched by
and recommendations. Inga hoped the
positivity and joy. Inga offered Liako her
ShivA's apparent impact, offered a pro-
evaluation would help them secure funding newly started Positive Sisters volunteer
bono evaluation using the Most Significant going forward.
training. Liako was enthusiastic, inspired to
Change technique, as an opportunity
In an interview, Liako, a Positive Sister from find a kindred spirit in Inga, and eager to
for their team to gain experience in that
Lesotho, shares her first conversation with help support others to live positive lives.
methodology. Given the powerful stories
Inga: ‘I explained to Inga how my life is and After Liako had been a Positive Sister for
and small sample size, Most Significant
how open I am about HIV. Inga said, "Vive
ShivA for 11 months, she met a young
Change would be perfect.
la Vie [one of ShivA's early projects serving woman named Millie, aged 30 and recently
Maaike's team interviewed the women and Afro-Caribbean women] is not for you, it's
diagnosed with HIV. Knowing that Liako was
then facilitated a collaborative workshop
for women who are closed and who don't
HIV-positive, Millie expected a thin, weak
where different stakeholders reviewed
know other women."'
woman, her face creased in pain.
Evaluations that make a difference-EN-20Sep15.indd 27
whatever she wanted to do with her life, and
who are affected by it. This evaluation was
neither should I. I should go on and do it.
the voice of the people infected.… Being
[You] need a doctor or a nurse when you're
interviewed made me want to be involved
sick, [but you also] need someone who has
more. I didn't want to only be a patient
been through that pain, who knows what it
after that. Being interviewed helped me feel
feels like. Without Liako I would not be who I
important and part of something good.
After the interview, Millie decided to
Clearly, being at ShivA had positive effects
become a Positive Sister, and she is now
on the women's lives. So did the evaluation, enrolled in university.
but in different ways.1
For Liako, being interviewed meant
becoming more aware of the impact of her
Each of the five women interviewed
work since ShivA: ‘The evaluation had a big
for this article (two positive sisters, a
impact on me. It was a Wow! moment for
project coordinator, a lead evaluator, and
me. We are really doing a great job.'
a referring nurse) spoke about different
Maaike's team prepared the interviews into
aspects of the evaluation process, but they stories so they could be read at a half-day
all mentioned the positive impact it had on workshop to see which stories were more
them and on the project as a whole. In the individual and which represented shared
evaluation interviews, the women reflected experiences.
on their experience with HIV/AIDS and how
much they had changed since ShivA came
The nurse, Lia, and the evaluator, Maaike,
into their lives. Millie had been quite shy
described the collaborative workshop day
My body is my best friend, I will take care of her – about her HIV/AIDS status before ShivA,
as a lively experience. The atmosphere
even with HIV.
and now recognises that being interviewed that Positive Sisters brought was one
was an extremely positive experience.
where ‘the floor felt like it was vibrating.'
The environment was joyful: good food,
Whenever people talk about HIV they talk
I was shocked that she looked so beautiful.
about how many people are infected in
laughter, music. The women talked,
It was reassuring for me to talk with her; she
the world and how many people are taking
laughed, and sang spontaneously despite
motivated me to push forward. She told me
medication, but it's never the voice of those
the intense sadness of their stories.
how she found out when she was my age
that HIV/AIDS shouldn't stop her from doing
Lia remembers a shy, somber woman she
Judgments about the impact of the evaluation are based on
the perceptions of the five respondents interviewed.
supported. At the evaluation workshop, she
Evaluations that make a difference-EN-20Sep15.indd 28
was pleased to see this woman stand up
from the project, by hearing other stories, can
Liako agrees. ‘We did short videos
and share her perspective:
now talk about the project as a whole.'
online. They are being shown on the
When Positive Sisters are in the room, it is
The stories grew from individual narratives,
internet and at conferences on HIV.
alive; I saw women become empowered. One
to project stories, and finally to a national
Mine is about having a husband and a
woman transformed in front of me when she
daughter who are negative. Healthy.
got up to speak. With Positive Sisters, women
dialogue. Inga says that the evaluation
And people get interested in who we
are finding meaning in life. Everyone searches
empowered African and Caribbean women
are and what we do.'
for meaning but when you are diagnosed with
to speak up on the national stage: ‘It was a
HIV, you face it more.
snowballing process. Once it got started there
Evaluation spurs action
With the collaborative workshop, the
was no stopping it. Policy makers are now
As a result of the evaluation,
individual stories started to blend into a
contacting our women to contribute to their
Positive Sisters was invited to deliver
collective story. After the workshop, Inga
research, and there is a group of African women a workshop at a National AIDS
told Maaike, ‘It is amazing that the women contributing who never spoke up before.'
conference. Inga says:
Positive Sisters present themselves at World AIDS Day.
Evaluations that make a difference-EN-20Sep15.indd 29
Many Positive Sisters became more confident.
do more. It made me
There was a very small woman from Ethiopia.
feel very important, like
The candle with an AIDS ribbon burns throughout
She was very quiet but there was a deep
I'm fulfilling a dream I
every ShivA activity.
inner power in her. She did an Ethiopian
coffee ceremony, sitting on that little low chair
didn't even know I had.
just by herself. But she raised her head and
I feel like I'm more than
contributed. Her voice was not loud, but her
I thought I could be.
confidence shone through and everyone paid
For me, success is not
attention to her because they were hearing
getting paid – it's doing
the voice of a free woman!
Although most of the stories collected by
For ShivA, expanding
Results in Health were positive, they also
roles are also important
offered insights into where improvements
given that referrals are
could be made. One suggestion was that
increasing too. Lia, the
Inga was doing too much. ‘The project
nurse, says that the
was leaning too strongly on me,' Inga
recognises. ‘If I wasn't there, nothing
her referrals from one
happened.' But having more coordinators
in three to all women
meant investing more money and Inga
diagnosed. ‘Because of
knew money was tight.
the evaluation I have a
The evaluators suggested she assume
more professional view
a coordinator role to help mediate
of Positive Sisters. After the evaluation I
project. She really wanted to learn from it
interactions with participants (now
talked with my colleagues and we decided and get results. She always made time for
numbering several hundred) and Positive
we would refer more people to Positive
Sisters (now 31). Two Positive Sisters
offered to take on a coordinating role
One of Inga's primary concerns for ShivA
without pay, including Liako, who with
The evaluation team felt that working
is, as with most community projects, to
ShivA's help is now pursuing a professional with Positive Sisters was an opportunity
ensure continued funding. So when the
vocation as a counsellor and a coach. At
to work with an exemplary client. Inga
evaluators proposed a workshop with
the time Liako was interviewed, this new
was hungry for insights and feedback and
multiple stakeholders to look at results and
coordinating role had just begun. She
quick to apply them. Maaike says, ‘Inga was think about funding opportunities going
explains: ‘The evaluation made me want to extremely interested in learning about the forward, Inga was on board.
Evaluations that make a difference-EN-20Sep15.indd 30
While steady funding for ShivA is not quite
there yet, it is certainly on the way. The
snowballing effect continues. At the workshop
there were many ideas about how to get
health providers to pay for the support
participants get from ShivA. For example, a
large foundation supported by the royal family
of the Netherlands recently committed to help
find the needed funding.
But the most important outcome from the
workshop was something intangible and
In Inga's own words:
It was very special to me because I do a lot
of work alone and this time I was not alone.
Everyone was trying to find a future for Positive
Sisters. Through the evaluation, I was able to tap
into the experience of professionals who also
do good quality work. They could evaluate the
quality of what I'm doing. It gave me a new surge
of self-confidence, a stable ground to stand on.
Now I can see how to make ShivA grow further
and give it a brighter future.
Each bead on each bracelet represents a blessing given by the women to each other.
We would like to credit the tellers of this story:
Story writer: Rita Fierro
Inga Mielitz, ShivA Foundation
Short video on the effect of the Positive Sisters link to
Liako Lekhooa-Oude Lansink and Millie, Positive Sisters
the evaluation report: Link to ShivA: www.shiva-positief.nl
C. (Lia) Meerkerk, nurse practitioner
Link to ResultsinHealth: www.resultsinhealth.org
Maaike Esselink, Programme Officer, Public Health
On the Positive Sisters project: www.
and all respondents of the evaluation who made this evaluation so valuable. shiva-positief.nl/a-letter-to-you
Evaluations that make a difference-EN-20Sep15.indd 31
Learning and earning: training
and vocational training. Now CAP was
keen to learn how this approach could
work in Kenya. Funded by The MasterCard
Like many countries, Kenya has an
Foundation as one of its Learn, Earn and
enormous population of unemployed
Save initiatives, CAP YEI, in partnership
young people. Depending on one's
with the University of Minnesota, evaluates
perspective, this is either a ticking time
its performance in both the short and the
bomb or an opportunity for affordable
long term to make changes as it learns
vocational training – relevant training that
what works. The ongoing evaluations have
enables young people to get jobs, earn and helped to equip Kenyan participants with
save money, and move out of poverty.
technical and psychological skills to get a
job or to start their own business.
The Community and Progress Youth
Empowerment Institute (CAP YEI) opened
Douglas Moseti, CAP YEI Nairobi regional
its first training centre in Nairobi in 2011.
coordinator, has been with the programme
In India, CAP had been successful in giving
since it began. ‘We target people between
vulnerable youth a chance to acquire
18 and 25 years old who can demonstrate
useful workplace skills through technical
that they come from vulnerable
they are orphans or they
Students filling Equity Bank savings account
come from single-parent
applications in the CAP YEI training centre.
families or large families
where a large number
not pay his school fees. He finally finished
of people depend on a
high school, thanks to a lucky encounter
at his church. But with a high school
diploma, he could only find a job paying the
Moseti knows what it
equivalent of $1 a day. A 2-year vocational
means to need a leg up in training course was out of his reach, at
life. He grew up poor and least for a time.
was forced to drop out
of school several times
CAP YEI provides 3 months of training
CAP YEI youth in a Unilever training course on how to access products when his mother could
adapted to labour market demand. To
for their small enterprises.
Evaluations that make a difference-EN-20Sep15.indd 32
the proposed course He then goes on to describe the evaluation
content, to mentor
that follows the placement of each
CAP YEI trainees,
and to offer them
Once a batch has completed everything,
we review what we've finished before
they graduate from
starting a new batch. Facilitators, the
regional coordinator, and the programme
director describe the whole process. We ask
questions: Did we get the right people at the
training cycle doesn't
road shows? Did we give them the right skills?
end with the classes
We look at student attendance at training
as in most vocational
and at work and ask: Was placement good?
What course gave good placement? Where
Rather, each cycle
training does not lead to jobs, we change and
get more industry people to help us, to tell
culminates when the
us what's missing and we supply that. In one
graduates are placed
centre where students had a hard time getting
in internships or jobs
jobs in electrical and electronics, a new
or are starting their
market scan was carried out and a curriculum
was designed to offer training in building
This trainee obtained employment with a floriculture firm in Naivasha.
and construction. Elsewhere, garment
manufacturing was replaced with training in
what happened with
security and management systems.
determine which courses to offer, CAP
the first group of trainees.
YEI staff scan the local market, contacting
James Chepyegon, CAP YEI project
There were 141 students in batch 1 and we
companies and entrepreneurs to see what
manager, uses a management information
were 6 staff who were very new. We were told
entry-level jobs are available. In its first
that we'd get at least 80% of those 141 into
system developed by CAP to collect data
centre in Nairobi – it now has nine there –
employment opportunities. We didn't hit the
on course enrolments, on graduates per
target for the first batch because the industry
they found that many hotels needed staff
cohort, on the absorption rate of graduates
didn't believe that a new employee could be
and security guards, and that construction
into the job market, and on changes in
trained in just 3 months, and we had no way
firms need builders and electricians. As CAP
students' attitudes and lives after the
to show potential employers that the trainees
YEI engages with potential employers to
could do the work. But we did manage to
see what jobs they might have to offer their
place 67% of the trainees, including 5% who
Students are surveyed before and after
graduates, they also ask them to look at
went to start their own businesses.
training to see how their attitudes have
Evaluations that make a difference-EN-20Sep15.indd 33
changed, how they made the transition to jobs,
One of the most striking findings of the
Dr Heidi Eschenbacher, another member of
how they've created their own businesses, how
evaluations has been the critical role
the team, confirms: ‘Our findings suggest
their views of life have changed as a result of the
of life skills in preparing young people
that life skills combined with technical
training and employment, and how well they
for jobs. Professor Joan DeJaeghere,
skills help youth to build up sufficient
understand working relationships with employers.
who leads the University of Minnesota
confidence to feel as though they can use
This project data blends with the data
team, cites her team's annual on-site
their technical skills. In other words, they
collected annually by the University of
visits, surveys, and interviews with
need life skills to build confidence to be
Minnesota team that comes to Kenya to
efficacious in their technical skills.'
survey stakeholders, including participants
Year after year, CAP does life skills, a mix
and employers. Their data is compiled into
The life skills range from mentoring by
of engendering self-confidence, teaching
a longitudinal study to see how the training
successful business people, many of whom
students to be assertive, teaching them
changes the trajectory of participants'
about job markets and how they work,
started out with little but managed to
livelihoods and lives. This impact evaluation
where they are in the job market and what
succeed, training in financial literacy, and
is designed to emphasise learning in the
they can expect, and financial literacy.
training for young entrepreneurs looking
monitoring, evaluation and learning portion
Youth say that the life skills training is
to start their own businesses. Chepyegon
what makes the difference. They say ‘I
of the programme to help CAP learn during
confirms that the evaluations clearly show
can get into something, start off, move to
implementation and respond to the needs of
that the links between employers and life
something new and manage finances.'
skills training work. ‘They are,' he says, ‘the
strongest point of success for CAP YEI in
getting youth access opportunities.'Their strength lies in part in the way
that the life skills module has changed in
response to the evaluations of participants'
comments over time. Adds Chepyegon:
The entrepreneurship department
got stronger, to better account for
entrepreneurship activities from the time
individual potential entrepreneurs were
identified to helping them start up their
small business, including the creation and
registration of entrepreneurship groups
and support for the success of their small
Carpentry training class at the Athiriver Centre.
Evaluations that make a difference-EN-20Sep15.indd 34
businesses. Nearly 200 entrepreneurs are
30,000 shillings per month from these mobile
David Chapman, co-lead of the external
currently functioning well on a full- and part-
banking services, which include Mpesa, Equity
Agent, and Cooperative Bank Agent. Another
There have been some surprises going into
Moseti understands the confidence
graduate imports clothes and shoes from
Dubai and sells them in Kenya.
the fourth year of the 5-year field data
building that comes from having a mentor
collection. Where conventional wisdom
because he experienced it. He recalls:
The learning and evaluation partnership
might suggest that participants go through
at CAP YEI is working to help youth get
the training and then find or create a job,
A prominent person gave me electrical
the reality is that they are multi-tasking to
installation work at a very big building. As an
ahead, while simultaneously learning what
earn money. In their efforts to start their own
electrical person God granted me favour with
stands in their way. Reports Professor
the boss. Whenever there was need for a
driver and since I was qualified, I was the one
to buy materials and run errands that required
a driver. I became his point man on the
construction site and ended up becoming the
foreman of his building with both electrical
and other construction responsibilities.
Moseti later joined his boss' ICT firm, was
promoted, and got a good raise.
Quite recently, Moseti was able to take the
measure of the project's success when he
met with the first cohort of graduates in
Nairobi. This cohort was the group followed
by the University of Minnesota for its
Seventy-eight of the 141 students came. All
but one of them were working. I discovered
that most of these young people (almost 40%)
had moved into self-employment with big
businesses. One lady had become an agent
for large banks seeking to extend services
to rural areas. She says that she's earning
Kenya's Industrialization and Enterprise Development Cabinet Secretary Adan Mohamed inspecting a
demonstration by CAP YEI Athiriver Centre building and construction students.
Evaluations that make a difference-EN-20Sep15.indd 35
businesses, these youths have trouble getting
released, average start-up capital, average
the necessary capital. That is slowly changing.
earning post start-up, and average savings
Christopher Johnstone, Acacia
Banks in Kenya have been in the forefront
per client in order to make it easier for
Nikoi, and University of Minnesota
of youth bank products and have found that
research fellows conducted annual
group lending programmes have a better
graduates to get the financial support they
external evaluations of the project
repayment rate than they get with their more
need to start their own businesses.
and presented data that helped
The ongoing programme improvement
refocus project programming and
CAP YEI, in response to this finding, has
process is extremely helpful in assisting
improve services for youth.
been pursuing a number of partnerships
youths to get and keep good jobs,
with financial service providers who are
particularly in dynamic situations such as
James Chepyegon, monitoring and
tracking key data points such as number
labour markets into which many youths are
evaluation executive, was responsible for
of accounts opened, amount of funding
ready to enter.
conducting internal project monitoring,
interpreting implications of M&E data,
and building capacities of project staff.
Ashok Ankathi (project director at
the time) provided programmatic
evolutions influenced by evaluation.
Douglas Moseti (Nairobi regional
coordinator at the time) provided
the context of the beneficiaries
that helped shape the story.
Story writer: Deborah Glassman
We did it! Nyeri Centre Batch 1 trainees marching on their graduation day.
Evaluations that make a difference-EN-20Sep15.indd 36
If you don't ask, you won't see it! of poverty that typifies many rural
The indigenous communities were
communities. It does so by using a
among the intended beneficiaries of the
Contemplating the indigenous women of
conditional cash transfer (CCT) approach,
programme since its inception, and they
the Sierra Tarahumara, a mountain area in where families are provided with payments also participated in their evaluations
the state of Chihuahua in northern Mexico, that are conditional upon undertaking
from 1999 to 2006. But although these
evokes bucolic scenes from the time of the certain activities, such as ensuring regular
evaluations confirmed that indigenous
Spanish conquest. These communities have attendance of their children in school or
people were effectively participating in the
maintained themselves for centuries far
obtaining certain health services. These
programme, it was unclear if they were
from cities, clustered in small villages and
incentives assist in the achievement of
achieving the stated objectives in terms of
scattered family groups, cultivating a few
higher standards of education, health,
health, education, and ultimately, economic
seasonal crops and raising chickens, goats, and nutrition, and also provide necessary
and cattle. Almost all live in poverty, and in support to the people of the Sierra
many cases they are semi-nomadic.
Tarahumara to undertake economic
Access to the Sierra Tarahumara is difficult. activities that enable them to increase their
Sometimes it takes several days to reach
family income and quality of life.
the settlements. It is therefore
quite difficult to make contact
with them to interview and
select candidate families for
the Programa de Desarrollo
Humano Oportunidades (Human
Programme, hereafter referred
to as Oportunidades). But the
effort is well worth it because the
programme is quite remarkable.
This programme, implemented
in Mexico since 1997, aims
at nothing less than breaking
the seemingly endless cycle
The evaluators learned to ask the right questions – in the right language!
Evaluations that make a difference-EN-20Sep15.indd 37
performance. According to Mercedes
Gonzalez de la Rocha, anthropologist and
head of the qualitative impact evaluation of
Oportunidades, the absence of a clear focus
on indigenous communities had left a blind
spot in the programme's knowledge base.
Or as she puts it, If you don't ask, you won't
see it!This statement constituted a starting point
for raising the influence of an ethnicity
variable in the new qualitative evaluation
to be conducted in 2008. By then, the
programme had a decade of experience in
implementation in Mexico, which allowed
for a thorough evaluation. Evaluators
designed a strategy for field work
comprising 11 indigenous intercultural
regions in the states of Chiapas, Chihuahua,
The women of Comunidad San Ignacio, Municipio de Bocoyna, Chihuahua.
Oaxaca, and Sonora. In each location, the
programme's coverage and operations
vocales were bilingual. Although the
technical information they provided. This
were analysed. The evaluation identified
programme had been operating since 1997 situation was exacerbated in the case of
the main obstacles to programme
with more impact in indigenous areas
elderly women, who spoke no Spanish at
implementation with particular attention to than in non-indigenous areas (as had been all. The promotores assigned to the region
the relationship between the extensionists documented by previous evaluations),
did their best to overcome the language
(promotores) and the women representing the important problem of communication
barrier, but the results were unsatisfactory.
the indigenous communities, the vocales.
had not been sufficiently addressed. For
instance, the evaluation found that young
In some areas, the majority of indigenous
What they found was quite unexpected:
indigenous women – most of whom were
women did not understand what the
there were serious communication
bilingual, although their Spanish was often programme was for. They couldn't
problems with language. Virtually none
limited – did not accurately understand
understand what good it did to spend
of the promotores and only a few of the
the Oportunidades employees and the
hours listening to medical specialists who
Evaluations that make a difference-EN-20Sep15.indd 38
spoke about issues they could barely
in some indigenous areas, in the Sierra
comprehend. Sometimes the information
Tarahumara, a whopping 30% of the
that was being communicated conflicted
population remained outside the
with their traditional customs. For example, programme. It was clear that the lack of
when indigenous women participated in
access to health services and education
training on the importance of a physical
was brought about by the problem of
examination for possible breast cancer,
monolingual families who could not benefit further positive
it was clear that the idea of a stranger
from the written and oral information
touching them in their private parts made
conveyed by Oportunidades staff members providing jobs
them very uncomfortable. These women
to the women.
and wages to the
do not even undress in front of their
It was urgent to take measures to solve
husbands! Thus, a practice intended to
this serious problem, and Oportunidades
own living space,
save their lives was totally unacceptable for did just that. The qualitative evaluation
Jaime Holguín, one of the
suggested that bilingual promotores be
bilingual promotores, in
The evaluation found that although the
recruited from indigenous youth alumni so learning they had
programme's coverage had improved
as to contribute to better communications acquired right
Municipio de Bocoyna.
The then general coordinator of
Oportunidades, Dr Salvador Escobedo,
We managed to make the change in rules of
operation to include bilingual extensionists,
and that was the first step. Then in parallel we
ran a training programme with the National
Institute of Indigenous Languages (INALI) in
order to generate a mechanism to evaluate
the indigenous youths we wanted to hire
as extensionists who speak an indigenous
language. We have sought to work with the 13
most spoken languages such as Tzotzil, Maya,
When their parents can understand the promotores, they are happy to support their children's education.
Evaluations that make a difference-EN-20Sep15.indd 39
The women of San Ignacio receiving food aid during the drought crisis of May 2012.
Another important aspect that was
pointed out by the qualitative evaluation
of 2008 concerned the inefficient (and
even absurd) survey of each household
to assess whether or not it was eligible
for the programme. In the case of the
communities of the Sierra Tarahumara, this
was redundant because every household
was unquestionably poor – not to mention
the difficulty and cost of getting there just
to conduct the survey.
Escobedo, who understood the realities
of indigenous communities in Mexico,
supported the proposals arising from the
qualitative evaluation that were eventually
endorsed by the President.
To do this, INALI began training a group of
The process of bringing bilingual
The first step was to convince
15 young men and women with diplomas
promotores into the programme unfolded
Oportunidades collaborators such as
as trainers of social programmes in
steadily over the next 2 years. As their
government officials and international
indigenous languages. They were the first
understanding increased about the
organisations like the Inter-American
to reach the remotest areas of the country importance of sending their children to
Development Bank and the World Bank of
to carry the voice of the programme to
school, feeding them properly, and learning the need to change the rules for coverage
where it was most needed. Says Escobedo, how to use resources, indigenous women
in isolated areas. That involved a major
‘The project trained 350 promotores in
became increasingly willing to participate
challenge for the programme managers
order to achieve almost total coverage of
and interact with the promotores and
because, beyond being convinced of
the monolingual indigenous populations in with each other. In some areas they even
the real value of this recommendation,
Yucatan, Oaxaca, Chiapas, Jalisco, and the
formed groups for early child education.
they needed to effect those changes
Sierra Tarahumara. By the time I left the
The role played by the qualitative
without appearing to refute the spirit of
programme, we had trained a total of 250
evaluation with its remit to address the
Oportunidades, which was historically
extensionists, and awaited evaluation on
marginalisation caused by monolingualism focused on the poor while requiring certain
the success of this implementation.'
was key to this transformation.
Evaluations that make a difference-EN-20Sep15.indd 40
responsibilities as sending children to school
An important spin-off of the programme has been
or attending health talks. The figure was
to instill a spirit of entrepreneurship amongst the
worrisome, and returning these people to
women and girls of the Sierra Tarahumara.
the programme would be difficult, if not
impossible, given the rules of operation.
Reaffirming this contention, Escobedo
The World Bank and Inter-American
Development Bank opposed eliminating the
requirement of the co-responsibilities in these
areas, in part because they would undermine
the programme, thus generating a new
programme. That´s the reason why it could not
be done overnight. It is sad and painful, but very
However, given the magnitude of the
problem in the context of the severe drought
affecting indigenous people in the Sierra
Tarahumara, the situation was resolved
conditionalities of the beneficiaries such as
That report was published in a Chihuahua
after the intervention of the President
health checks and school attendance.
newspaper and generated an immediate
response from the federal government, which
Felipe Calderón himself, who ordered the
In late 2011 and early 2012, a severe
decided to intervene in various regions with
immediate reinstatement of 8,000 families.
drought hit the Sierra Tarahumara.
the Secretary of Social Development. It was
This was accomplished by integrating the
Rumours were rife about indigenous
then that they communicated the decision
reinstatement to the Food Support Program,
people committing suicide for lack of food,
that Oportunidades should intervene to solve
a transfer programme that did not require
although these were subsequently found to
the problem precisely in the area of the Sierra
the fulfillment of co-responsibilities that
be false. Dr Iliana Yaschine, former director
was also operated by Oportunidades. The
of evaluation of the programme from 2002 The important mobilisation and attention
indigenous promotores brought the good
to 2006, coordinated a study documenting to the area made it possible to detect 8,000 news to their communities, while a renewed
the work of Oportunidades in the Sierra
families (40,000 indigenous people) who
training of bilingual extensionists was
Tarahumara during the drought crisis,
had been dropped from the programme
encouraged to address immediately the
because they had not fulfilled such co-
Evaluations that make a difference-EN-20Sep15.indd 41
for the evaluation's strategy of full coverage
in the Sierra Tarahumara, the onset of
drought favoured the implementation of
a recommendation for the best care of
indigenous communities. The history of the qualitative evaluation
shows the relevance of cultural sensitivity
to the evaluation of social projects. In the
words of Gonzalez de la Rocha:
may be hard to
If you don't ask, you won't see it!
reach, but well
worth the effort.
Co-authors Mercedes González de
In addition, the rules of the Oportunidades
of this decision in the field at a time of crisis.
la Rocha and Agustín Escobar Latapí
programme were changed to implement
Special efforts were made to deploy support
directed the qualitative external
the strategy of full coverage (as it had
for the affected families, not at the household
evaluation of the programme and
been recommended by the qualitative
level but in the care centres where groceries
made the recommendations that led to
evaluation). This allowed incorporating
were delivered. I remember what the
improvements in beneficiaries' lives.
qualitative evaluation said on this subject, and
families into the programme from small
how that had prompted the changes during
Iliana Yaschine was Director of Evaluation
and distant locations without undertaking a
in the programme from 2002 to 2006.
home survey. This change helped broaden The findings and recommendations of the
Pablo Rodríguez-Bilella and Omar
the programme's coverage and, in addition qualitative evaluation in the indigenous
Zevallos helped shape the story.
to the reincorporation of the families
communities provided crucial information
mentioned above, it made serving the
Salvador Escobedo (Program Director
for decision making. Managers made
entire indigenous population of the Sierra
at the time) provided helpful insight.
changes that impacted the participants
of the programme directly. In the case of
Photographs were taken by Agustín
According to Yaschine:
the bilingual promotores, the evaluation
Escobar Latapí and Iliana Yaschine.
Note that the programme was originally
Without the qualitative evaluation, it would
noted a dimension of cultural adaptation
have been impossible to make that change,
that had been neglected in the original
called Progresa, then Oportunidades
even with the drought crisis. That is my
design of the programme, and once it was
(the name in use when this story took
perception. I observed the implementation
incorporated the benefits were obvious. As
place), then Prospera, the present name.
Evaluations that make a difference-EN-20Sep15.indd 42
Listening to the listeners
accurate picture of the
crude reality of child
Just before dawn Ashish nervously waits
marriage that prevails
for Rama at the village bus stop. As soon
across the country's
as he spots his girlfriend, they cautiously
remote villages. Nearly
board the bus toward the city planning
3 of every 10 girls
their future as a couple. But their plans are between 15 and 19 are
thwarted when a police officer interrupts
presently married, and 4
their journey and starts interrogating them. of 10 are married before ‘We just want to get married,' Ashish tells
the age of 18, according
the officer. ‘Rama's parents were forcing
to national statistics.
her to marry a stranger.'
Radio dramas like this
‘We have done nothing wrong,' Rama adds. one reflect the ethos
The police officer then intervenes as the
and pathos of Nepalese
teenagers, both 16, request him to let them society. One programme
Collecting a voice recording on effects of child marriage with an
in particular, a weekly
adolescent girl for the SSMK radio programme.
called Saathi Sanga Man Ka Kura
‘It's a crime,' he says. ‘You need to be at
Narrowing the focus
(SSMK), which means ‘Chatting with
least 20 years old to get married. Child
My Best Friend', has been reaching
Before the introduction of the electronic
marriage is a criminal offense – your
out to young girls and boys for 15
media, street dramas and stage shows were
parents can be prosecuted if they're
years. SSMK educates young people
popular throughout Nepal. Mostly mythological
forcing you to marry young and you can be about issues that are generally kept
and sometimes satirical skits in local dialects
prosecuted too if you marry now.'
quiet in conservative Nepalese
emerged as a major form of entertainment in
The legal age for marriage in Nepal is 20
society. Started in 2001 with support
years without parental consent and 18
from the United Nations Children's
While this trend might have declined in cities,
years with consent.
Fund (UNICEF) and managed by Equal
Ghanshyam Kumar Mishra, a radio producer
Access Nepal for the past decade, the
While Rama and Ashish are fictional
from Nepal's southern town of Janakpur in
45-minute show has 7.2 million loyal
characters from a weekly radio drama
Dhanusa district, says dramas in local languages
listeners and receives more than 1,200
aired on more than 40 stations across the
are still popular and preferred in Nepal's rural
letters and 2,000 text messages each
Himalayan republic, their story paints an
pockets. Most people in these areas are low-
Evaluations that make a difference-EN-20Sep15.indd 43
income farmers with little educational
backgrounds and no source of information
other than radio, which is also one of
the cheapest forms of entertainment.
Therefore radio dramas, according to
Mishra, are one of the best media through
which to inform the masses.
In 2013, SSMK conducted an evaluation
using the Ethnographic Action Research
method to assess and evaluate the
listenership trend and impact of its
programme on its target audience.
Community-based evaluators in Nepal's
rural districts conducted surveys and
focus group discussions with community
members and stakeholders.
One of the major findings from the
evaluation was the need for a local
Collecting opinion from a listener on sexual harassment.
language radio show.
The survey during the evaluation process
addressed topics on a macro level, lacked
says. ‘The listeners were demanding a
highlighted the language conflict among
discourse on community-specific problems. programme in a local language.'
SSMK's listeners; the central version show
The SSMK team therefore wanted to ensure To better equip rural youth with the
was not able to fully capture the spirit
meaningful participation of youth from
knowledge and skills they need to
of local communities. With the central
across the country by starting local versions participate in local and national policy
programme produced in Nepali language,
of the programme, which would give them development, SSMK launched local versions
listeners in other parts of the country
a sense of ownership, according to Ayush
of the show in 15 marginalised districts.
such as Dhanusa where Maithali is widely
Joshi, senior programme officer at Equal
Equally important, its purpose was to
spoken, sometimes felt disconnected.
provide necessary knowledge to the young
Also, the issues varied from region to
‘It was important to start a local version
people so that they could control their lives
region and a national programme, which
to sustain SSMK in the long run,' Joshi
by applying what they learned.
Evaluations that make a difference-EN-20Sep15.indd 44
Nepali, they might just think that it is not
A group photo with adolescent listeners at school.
meant for them.'In the 26 episodes of SSMK's local offshoot,
the show raised area-specific issues such
as child marriage, dowry-related problems,
and violence against women.
Mishra says the team also incorporated
experts and law enforcement officials from
the community to make episodes relevant
locally. He says it adds extra credibility and
makes the show more trustworthy. Upon listening to the show on child
labour broadcasted in Maithali, Mishra
The local programmes are customised
Raising awareness of child labour
says people have become fairly hesitant
according to their listenership with tailor-
in recruiting children. This episode also
made content that helps the audience
At Radio Mithila in Dhanusa, Sangi Sang
prompted authorities to at least warn
connect with prevailing issues in their
Manat Baat became an offshoot of SSMK.
businesses that employed children.
communities. In a short span, Mishra says
It was targeted at the Maithali-speaking
‘It takes time for these practices to end,'
SSMK's local version has been effective in
people living in the district's 90 Village
Mishra says. ‘But I'm sure our radio
highlighting issues like child marriage and
Development Committees. A show in
show has started that much-needed
child labour, both of which are considered
Nepali, according to Mishra, simply would
conversation. It's a small revolution of
socially acceptable in many rural
not resonate with the locals. A Maithali
communities. While the local-language
programme, on the other hand, pulls
in listeners primarily interested in the
Changing attitudes about women's issues
radio dramas have played a pivotal
role in educating the public, segments
problems that dominate their region.
SSMK's evaluators note that local
incorporating local authorities and
‘When you're communicating with people
programmes help represent community
stakeholders have opened up a non-formal in their own language, the audience feels
problems more accurately, and Manauda
communication channel for discussion.
associated with the show,' Mishra says. ‘It
Ka Kura, SSMK's local adaptation, is doing
It has also helped in holding authorities
is also easier to make them understand the just that in the far western district of
accountable for these issues.
topic. When the programme is strictly in
Evaluations that make a difference-EN-20Sep15.indd 45
Through radio drama, using an
Manauda Ka Kura receives considerable
Auji says that their show in Acchameli
informational and emotional approach,
feedback and comments from its listeners, language has increased the engagement
the show has drawn attention toward child a few apologetic but most applauding
and interaction level with the community
marriage, which is highly prevalent in the
the programme's efforts. The letters and
members. It has also made it easier to
text messages, according to programme
disseminate information on topics that
Sarita, one of the characters, is forcefully
producer Bidhutma Auji at Radio
matter the most in their native tongue.
married at an early age. She is deprived
Ramaroshan, motivate them to continue
‘When the programme is in a local
of her will to study and dies of pregnancy
the show and also help them ascertain the language, listeners feel that someone they
complications. On the other hand, her
know and trust is talking about their issues,'
friend Renuka, who was against Sarita's
One of the letters the show received
Auji says. ‘It's easier to communicate using
wedding, finishes her school and pursues
after the episode on child marriage
local words and metaphors that might be
her education to become a nurse.
summarises the scale of Manauda Ka Kura's difficult to convey in Nepali.'
‘My daughter will never forgive me,'
effectiveness. A 16-year old girl writes:
Connecting youth, the elderly, and
Sarita's father later apologises to Renuka
My parents who are very conservative were
for dismissing her idea to at least wait for a
forcing me to get married. But one day when
For almost three years, listening to SSMK
I was listening to the radio programme, the
hosts were talking about the hazards of child
and now its local version has become a
marriage. I requested
regular affair for Sagar Bhandari's family in
Community dialogue on child marriage
my parents to listen
Accham's Mangalsen village. Every week,
to the show, and
the family sits by the radio set in the living
after listening they
became aware for
room and listens to the show.
the first time about
With higher mobile and internet
the realities of child
penetration, young people have alternative
marriage. They then
decided not to force
means of accessing information. Bhandari,
me into marriage.
18, says ‘Adolescents have many problems
The local language
they cannot talk with their parents. These
radio show not only
issues are always hidden from the parents.
helped educate them
But when their parents to listen to a radio
but also helped them
drama, it helps them understand what their
to empathise with
children are going through.'
Evaluations that make a difference-EN-20Sep15.indd 46
But statistics show that radio is still the
popular choice of mass communication for
Focus group discussion with listeners.
people aged 15–19, and SSMK is the most
important radio show in that age group.
According to the National Demographic
and Health Survey 2011, 58.5% of men and
49.7% women in the surveyed population
listen to SSMK regularly.
While the local programmes have broken
the silence between the two generations,
SSMK's evaluation also showed that they
act as a tool to teach the adolescents about
their rights – the legal age of marriage,
the use of contraceptives, and charters
regarding child labour. The programmes
have also helped to initiate discussions
between local policymakers and youth.
In village clubs and meetings led by local
youth from the community, these episodes
help to start conversations on important
influence and impact of radio shows in
Acchameli also yield positive results for
topics. Bhandari says:
the local language, particularly when the
In our monthly Child Club meetings, we
shows reaching areas inaccessible by the
‘Our [government] programmes are only
discuss the issues raised in the radio show.
targeted at certain groups. So we cannot
The information is not limited to the radio
‘When the programme is in a local
incorporate everyone. But programmes like
show itself. The dramas serve as catalysts.
Once the topics are introduced, they are then
language, people feel a sense of ownership SSMK are for everyone. No one is left out
discussed and shared continuously at the
– they think it's a programme for and by
when it comes to accessing information.'
workplace, in schools, and at home.
them. And they are right!'
Youngsters like Auji and Mishra believe
Prem Bahadur Buda, Accham's District
Buda added that local language
that education and access to information
Child Rights Officer, also stressed the
programmes like Manauda Ka Kura in
are the keys to changing social attitudes. In
Evaluations that make a difference-EN-20Sep15.indd 47
districts like Accham and Dhanusa, young
raising community-specific issues. They are
people are in the forefront of this slow, yet programmes by the community and for the
Yvette Shirinian is the Business
significant, change – making people aware community.
Development Officer for EqualAccess.
of the issues and helping prevent some
‘This has helped grow SSMK's listener base
Ayush Joshi was the Program
of the negative aspects of past practices
in many hard-to-reach areas,' says Joshi
Officer for EqualAccess Nepal.
through the information they receive
from Equal Access Nepal. ‘SSMK is building
through radio programmes.
a strong rapport in rural communities
Story writer: Bibek Bhandari
Shows like Manauda Ka Kura and Sangi
as a people-centric and inclusive media
Sang Manak Baat have proved relevant in
platform that listens to its listeners.'
Discussion on adolescent issues with adolescents
Evaluations that make a difference-EN-20Sep15.indd 48
The power of community-owned components of the initiative – as simply
if she and her colleagues could get their
part of her busy and demanding job. ‘I'm a hands on first-hand information, they could
community health nurse,' she says, ‘and this develop better services for their prenatal
Things weren't happening that we
goes back to my nursing background – when clients.
thought were happening. Because of this
you start a project you need to finish it.'
Changing River's maternal and child health
project, we made changes.
Chesne's partners in this groundbreaking
care needs are pressing. For one thing,
– Emily Chesne, Community Health Nurse initiative included colleagues, managers,
the population of the community is very
community leaders, and researchers from
young, with a median age of just 21, and
Emily Chesne, a public health nurse in
Health Canada. The researchers were
it's growing quickly. For another, access
Changing River, won't take credit for the
responsible for coordinating and evaluating to treatment is a major issue because the
improvements in her community's prenatal a total of 19 pilot studies in Aboriginal
community is located 3 hours from the
health services that she helped implement.1 communities across Canada.
nearest urban centre.
For years, Aboriginal mothers have had to
The research team included the
travel long distances for prenatal checkups. project's leader, Judy Watson, a
And on the health care side, record keeping 30-year veteran of government
was far from what it should be. But thanks health programming and research
to a pilot project called Patient Wait Times with Health Canada; Samir Khan,
Guarantee and people like Chesne who
a senior researcher and Chesne's
take their jobs seriously, the situation has
main point of contact; and Stephanie
changed dramatically for the better.
Potter of the Whetstone Group, the
While Chesne is satisfied with her team's
programme's lead evaluator.
success in transforming her community's
Through supportive collaboration
prenatal health services through the
with these researchers, Chesne and
project, she downplays her role as an agent her colleagues, as well as the nurses
of change. She views her participation in
in other participating Aboriginal
the data collection, tool development,
communities, were able to use the
and process mapping activities – all key
data collected as part of the initiative
1 The events described in this story are true. However, at
to make health services more
the request of the resource persons involved, the names
accessible and effective. The data
of people and places have been changed. Suffice to say the
events transpired in a First Nations community in northern
First Nations people are justifiably proud of their vibrant
were important. Chesne knew that
cultures. Above is an artist's rendering of a dream catcher.
Evaluations that make a difference-EN-20Sep15.indd 49
The initiative – and importantly, its strong
their appointments once they were
in the community than anyone realised.
evaluation component– led to a realisation referred to the larger hospital. Through
This is a serious health issue that,
that would change the course of everything their involvement in the evaluation, they
according to Chesne, could have long-
for the pregnant women of Changing River. discovered that many at-risk mothers were term repercussions. By facilitating early
Through the evaluation, Chesne and her
showing up for delivery without having had diagnosis, such problems can be identified
colleagues learned that the reason data
prenatal care for several months.
were lacking was because women were
Using the evidence at their fingertips
The team continued to use the initiative's
not attending their third trimester prenatal
data to overhaul the community's prenatal
Armed with the evidence they needed,
the team took immediate action by
and postnatal health care, setting the stage
It was not difficult to understand why.
implementing on-site doctor visits for
for a healthier future. For the first time,
Previously, the women attended prenatal
prenatal patients. Chesne: ‘We initiated
the clinic was governed by a client-centred
appointments and delivered their babies
getting one of the doctors from the
culture. According to Chesne, ‘Now moms
at a hospital 45 kilometers away. Recently, delivering hospital to come to our clinic.
come in and learn how to take some of
however, that hospital had not renewed
Having the doctor here worked really well. their own prenatal tests. We have our
its license to deliver babies. As a result, all
It helped with gathering data.'
doctor hold a sharing circle with clients as
third trimester appointments and deliveries
well as individual sessions.'
were referred to a hospital a full 80
Improving third-trimester prenatal care
Chesne reflects, ‘We learned things
kilometers away – nearly twice as far.
rates led to the identification of more
gaps in care and unrecognised health risks weren't happening that we thought were
Chesne knew that many local moms were
among the patients. Better access to the
happening. Because of this project, we
young, impoverished, and uncomfortable
physician's files helped the team realise
in unfamiliar environments. She suspected that many prenatal patients were not
The initiative had an immediate and
that these circumstances made the
getting their bloodwork done, including
substantial effect on people's lives because
90-minute trek to the distant hospital too
the glucose tolerance test for gestational
its leadership, especially Watson, chose
challenging. ‘We have a lot of young moms diabetes. Says Chesne, ‘We're trying to
to prioritise community engagement.
in our community. They are shy. They don't better educate our prenatals with what
Frontline health workers like Chesne
feel comfortable with professionals, with
tests they need and why.'
were true collaborators in the evaluation,
Improving compliance with testing
facilitating community ownership of the
But until data collection began, the
also led to the discovery that rates of
data and making needed changes. Once
community health team had no way of
gestational diabetes were much higher
these changes had been made, many of
knowing if the women were attending
Changing River's women who had been
Evaluations that make a difference-EN-20Sep15.indd 50
Potter recalls another community
where nurses were able to use data
to effect change. She was awed by the
transformation of one group of nurses
during the evaluation. At first, Potter found
them apprehensive, as if things had gone
poorly with past research.
By the end of the initiative, however, the
health workers had shown Potter that
even people without strong research
backgrounds could ensure that their
communities got the most out of the data
to which they had access. She recalls:
They didn't have computer databases, they
only had pieces of paper. But by the end of
the project, they made those pieces of paper
mean something. I will always remember
what they said at the end: Data is good. They
Many First Nations communities are accessible only by air or boat.
learned that. They could see patterns and
at risk for pregnancy-related complications evaluation. But her fears were allayed when
make changes in their communities even
now had access to diagnosis and treatment. the nurses all arrived for their meeting.
without sophisticated stats.
Chesne's community was not alone in
Watson was moved. ‘Two nurses had their Community empowerment through
taking ownership of the initiative's data.
study files under their arms – they didn't
Watson recalls a 2009 meeting with nurses even have briefcases. One nurse had lost a
from a Manitoba community even more
family member in the midst of all this, but
At every level of participation, stakeholders
remote and with fewer resources than
they kept the study files organised in spite point to the Patient Wait Times Initiative
Chesne's clinic in Changing River. At the
of the challenges. They really wanted to stay as a particularly powerful example
time, the swine flu crisis was raging and it
on track for themselves, for us, but also for of government-funded community
was overwhelming the community. Watson the community. That's how important the
programming to empower frontline
wondered whether the nurses would have community is – counting and being part of workers to effect change. But what made it
the time or motivation to continue with the the evaluation was so important to them.'
different from other initiatives?
Evaluations that make a difference-EN-20Sep15.indd 51
Health Canada and its
Potter found that working with Watson
collaborator in the leadership and her team was refreshing, informative,
of the project, the Assembly of and educational because they were ‘…so
First Nations, as true partners comfortable not knowing necessarily what
from the beginning to the end. was going to happen. They gave permission
Watson and Khan agree that
for evaluators to ask questions and find the
the same can be said of the
evaluation team, emphasising Questions and answers – from both sides
that with Potter's and her
colleagues they were able to
At the community level, the data collection
form a genuine partnership,
tools and procedures were refined based
leveraging each other to
on the evaluation results and direct
A nurse arriving by speedboat.
inform each other.
feedback from the frontline. Both Watson
and Khan found that communities were
Traditionally, health research in Aboriginal
Potter refers to the initiative as her
very comfortable coming to them with
communities has been characterised by
‘touchstone'. In fact, her own approach
questions and suggestions. According to
a top-down approach. Khan explains that
to evaluation has been molded by her
Khan, this feedback was vital to the success
usually such research is undertaken by
observations of Watson's work in enacting of the initiative: ‘We were getting data all
scientists from outside the community.
Health Canada's mission of community
along the way. We were trying to come
Rarely is community feedback sought.
empowerment and willingness to take risks.
According to Potter, ‘The historical dynamic
between the federal government and First
Nations communities has not been positive.
That's the big context that all this work
happens in and we can't pretend it doesn't
exist.'This project, however, fostered a truly
collaborative spirit. In Khan's words, ‘It's
usually hard to get people from different
areas to collaborate. But not on this
project.' Watson agrees. She characterises Patient Wait Time staff and community members in the community garden, part of wellness activities.
Evaluations that make a difference-EN-20Sep15.indd 52
up with explanations in Ottawa, but we
could be frustrating. Then I'd go to meetings
stakeholders: the patients, the community
could ask questions directly of the people
or do teleconferences, and they'd tell me
health professionals, the government
providing the data. We could work together
what's going on and inspire me to finish
researchers, and the evaluation team.
to find the answers much more readily and
the programme. It's important to have the
communication to be able to talk to the
Watson, Khan, and Potter all agree
easily. We were getting explanations from
people processing the data.
that the evaluation affected how each
the people who had the answers.'
stakeholder engaged in future research
This support reflects the collaborative spirit
Even more important, for Khan, this
with at-risk communities.
that marked the project under Watson's
methodology demonstrated how having
leadership and the evaluation team's
Khan emphasises that the lessons he
communities lead, and researchers finding facilitation.
took from this initiative, including the
the best ways to support, showed what
importance of helping communities to
is possible in government-sponsored
Turning it around – for good
take ownership of their own health care,
research. The evaluation's face-to-
The Patient Wait Times evaluation
were instrumental in shaping the rest of
face group meetings with participating
had a long-lasting impact on all the
communities at the end of data collection
were a particularly powerful learning
An annual run comprises an important part of the community's
experience for Khan.
health promotion and prevention activities.
As important as it was to hear first-hand
about their challenges – travelling in winter,
getting reimbursed for travel, staff turnover
– it was critical to get real explanations for
what we were learning. The real innovation
was figuring out how to have First Nation
communities lead, and then finding ways to
In turn, the nurses collecting data found
that their work was facilitated by the
support of the programme's research team.
According to Chesne:
Meeting and speaking with the researchers,
especially Khan, inspired you to go back and
do better. It was very time-consuming and
Evaluations that make a difference-EN-20Sep15.indd 53
In Potter's words, the project helped her
Whose project is it? Whose data? There has stakeholders working together to make
learn that successful evaluation ‘…is all
to be room for the people who are affected. positive changes for both individuals and
about building relationships and working in It's not just the government's story.'
communities – provided the researchers
that indigenous community development
The Patient Wait Times initiative illustrates
get out of the way and have the
way: community-led, community-paced.
that an evaluation can be a story about all
communities take the lead.
Stephanie Potter, Lead Evaluator, SP Consulting and the Whetstone Group
Samir Khan, former Senior Research Analyst, First
Nations Inuit Health Branch, Health Canada
Judy Watson, former Project Lead, First Nations
Inuit Health Branch, Health Canada
Sara Pederson conducted interviews and crafted the evaluation story.
Evaluations that make a difference-EN-20Sep15.indd 54
Haemato-oncology and stem cell transplant unit for private patients Welcome to The Christie Clinic The Christie Clinic is a joint partnership between made in both treatment and supportive care leading the world famous Christie NHS Foundation Trust to improved survival. It is vital patients are diagnosed and leading healthcare provider HCA International. early, seen by the right specialist and offered the
RESOLUCIÓN 05-03-ARCOTEL-2016 EL DIRECTORIO DE LA AGENCIA DE REGULACIÓN Y CONTROL DE LAS Que, el artículo 226 de la Constitución de la República del Ecuador dispone que: "Las instituciones del Estado, sus organismos, dependencias, las servidoras o servidores públicos y las personas que actúen en virtud de una potestad estatal ejercerán solamente las competencias y facultades que les sean atribuidas en la Constitución y la Ley. Tendrán el deber de coordinar acciones para el cumplimiento de sus fines y hacer