Description
Executive Summary
The purpose of this meta-evaluation is to attempt to identify which sanitation approaches in developing
countries have been effective and sustainable, so that Catholic Relief Services (CRS) and other sector
actors can position themselves for achieving universal access. This work will be divided into two phases:
the desk review and expert consultation (Phase I) and the in-depth country case studies (Phase II). This
report describes the output of Phase I and recommends countries for Phase II.
The sanitation sector has evolved over decades. However, in 2015, the target year for the Millennium
Development Goals (MDGs), much remained to be done: 2.4 billion people lack access to improved
sanitation and almost 1 billion people practice open defecation, 9 out of 10 in rural areas (WHO/UNICEF,
2015). While some attempts have been made to determine sustained solutions, comparable information
is scarce. Furthermore, funders and practitioners in the sector sometimes lack institutional memory,
making them vulnerable to repeating mistakes or investing in unproven approaches. This is an important
gap to overcome – and quickly – because Sustainable Development Goal 6 aiŵs ͞ to ensure availability
and sustainable management of ǁateƌ aŶd saŶitatioŶ foƌ all͟ 1 in just 14 years.
The sanitation-related targets for Sustainable Development Goal 6 aim to, by 2030:
Achieve access to adequate and equitable sanitation and hygiene for all, and end open
defecation, paying special attention to the needs of women and girls and those in vulnerable
situations.
Improve water quality by reducing pollution, eliminating dumping, and minimizing release of
hazardous chemicals and materials, halving the proportion of untreated wastewater and
substantially increasing recycling and safe reuse globally.
Expand international cooperation and capacity-building support to developing countries in
water- and sanitation-related activities and programs, including water harvesting,
desalination, water efficiency, wastewater treatment, recycling and reuse technologies.
Support and strengthen the participation of local communities in improving water and
sanitation management.
It is likely few countries – developed or developing – have achieved successful sanitation when all of the
above elements of sanitation are considered.
While seĐtoƌ aĐtoƌs doŶ͛t ŶeĐessaƌilLJ agree what needs to be done to achieve universal sanitation, shifts
by all actors are required: shifts in mindsets, attitudes, and ways of learning, collaborating and being
accountable (Hueso, 2015b). Thus, it is valuable for CRS and other development organizations to identify
countries moving toward successful sanitation and examine – through the Phase II case studies –the
useful support roles that nongovernmental organizations (NGOs) play. In consultation with CRS, we
subjectively selected the following countries for Phase II case studies based on the limited evidence of
success and other factors discussed above: Bangladesh, India, Nepal, Ethiopia, Lesotho and Rwanda.
1 Sustainable Development Goals: 17 Goals to Transform Our World, United Nations News Center,
www.un.org/sustainabledevelopment/water-and-sanitation.
Country case studies would include urban and rural sanitation. CRS and Improve International envision
that conducting the case studies will be a collaborative process with other development organizations
and donors, and they are open to including other countries based on evidence of success.
Background
The sanitation sector has evolved over decades, from construction of simple pit latrines to ͞ ƌeiŶǀeŶtiŶg
the toilet,͟ fƌoŵ full suďsidLJ to zeƌo suďsidLJ to saŶitatioŶ ŵaƌketiŶg. Directive individual or household
hygiene promotion efforts have given way in many places to community-led approaches to total
sanitation and national social change campaigns. For example, many sector actors have made major
shifts in approaches to sanitation (Table 1).
Table 1: Old and New Approaches to Sanitation
Source: UNICEF, 2014
Old Approaches New Approaches
Building toilets Changing social norms
Individual/family Social/community
Health message focused Economic, social, health, disgust
Top-down and externally driven Community-led – internal, demand driven
Didactic Participatory – natural and traditional leaders
Technologies predetermined Local technologies – community capacity
Subsidized Rewards – pride – celebration
DoŶ͛t mention the s*** word Talk shit – feces, poo, kaka, toilets, latrines
Recent major investments in research have uncovered some useful information on the demand (or lack
thereof) for toilets and aspirations for technologies beyond the pit. We have recognized the need for
fecal sludge management and the critical links between hygienic use and health impacts. However, in
2015, the target year for the MDGs, much remains to be done: 2.4 billion people lack access to improved
sanitation and almost 1 billion people practice open defecation, 9 out of 10 in rural areas (WHO/UNICEF,
2015). Very little information on sustained solutions is available, making funders and practitioners in the
sector vulnerable to repeating mistakes or investing in unproven approaches. This is an important gap to
overcome because Goal 6 of the SustaiŶaďle DeǀelopŵeŶt Goals aiŵs ͞ to ensure availability and
sustainable management of ǁateƌ aŶd saŶitatioŶ foƌ all.͟
Purpose
The purpose of this meta-evaluation was to identify which sanitation approaches in developing
countries have been effective and sustainable, so that sector actors can position themselves for
achieving universal access. This work will be divided into two phases: the desk review and expert
consultation (Phase I) and the in-depth country case studies (Phase II). This report describes the output
of Phase I and recommends countries for Phase II.
Seeking Sanitation Successes, Phase I
3
Methods
Phase I of the evaluation consisted of a desk review and an expert consultation. The desk review
consisted of the following steps:
Identified 195 journal articles, conference presentations, webinars and gray literature reports,
some opportunistically (e.g., through conference attendance), some through online search,
some by investigating the references in documents we found and some referred by the
experts consulted. Appendix A contains an annotated bibliography.
Used key words for the literature search including combinations of the following: community-
led total sanitation (CLTS), ecological, evaluation, international, latrine, marketing, monitoring,
post-project, sanitation, success and sustain.
For total sanitation approaches, focused on evaluations that contained specific data on open
defecation free (ODF) rates with special emphasis on data that showed persistence of ODF
over time.
Looked for studies with data for specific indicators, such as number of communities sustaining
ODF post-triggering, latrines used post-project, or other tangible measures of sustained
behavior change. Appendix B contains the sanitation indicators found in the desk review.
Recorded relevant quantitative data in an Excel spreadsheet.
CRS and Improve International purposefully identified 12 sanitation experts from key sector institutions
to target for interviews. This expert identification was then complemented by snowball sampling,
whereby the interviewees recommended other potential interviewees, or whereby authors from reports
identified in the desk review surfaced as potentially interesting informants. Because Improve
International had very recently conducted a series of interviews related to resolution of sanitation
problems, relevant content from those interviews (e.g., a definition of sanitation success) was included
in the analysis as well.
All told, we contacted 43 sanitation experts and interviewed 28 of them from June to September 2015.
The interviewees included 17 from international development agencies and organizations (Africa
AHEAD, Bill & Melinda Gates Foundation, Catholic Relief Services, Lifewater, Millennium Water Alliance,
SARAR-T, UNICEF, USAID, WaterAid, Water For People, World Bank); six from academia (Emory
University; University of California, Davis; University of Leeds; University of North Carolina, London
School of Hygiene & Tropical Medicine); three from sanitation marketing organizations (Pivot, Sanitation
Solutions Group, World Toilet Organization); and two independent consultants.
We conducted semi-structured, in-depth interviews by telephone, with the exception of one interview,
in which the expert preferred to respond to questions by email. We opened interviews by asking the
edžpeƌts͛ defiŶitioŶ of suĐĐessful sa nitation, discussing the factors that contribute to lasting sanitation
services, and then asking for examples of successful sanitation over five years or more. The interview
guides can be found in Appendix C.
Limitations of Study
We based this study on research and evaluation materials readily available to the researchers and
thought to be relevant to sanitation successes. We do not intend for this report to provide an exhaustive
or definitive record of this topic. We primarily intend to demonstrate the significant variation in
evidence of sustainability across various sanitation projects and programs for which reliable evaluation
material was available. Specific limitations of the study relate to the limited time frame for the study
(two months). Our research looked only at English-language reports. Some reports or articles might not
have been publicly available or online. The key words we used might not have elicited all the useful
information available. We were not able to interview all the originally targeted experts, nor did we
identify all the possible experts on this topic. We only interviewed people who spoke English.
In addition to the challenges with defining sanitation or sanitation successes (described in more detail
below), we encountered challenges related to finding and comparing evidence of results or success,
which include:
The indicators are not consistent, making it difficult to compare across programs, approaches or over
time. The evidence we found is not easily comparable because few common indicators exist within or
across sanitation approaches. For example, some organizations report the number or percentage of
households triggered, trained or building/receiving/buying toilets, while others discuss villages,
communities or number of beneficiaries in general. Indicators found in the desk review are shown in
Appendix B.
Even where common indicators are used, the definitions of the indicators are not consistent. Different
definitions are used for hygienic, unhygienic and improved sanitation facilities (FANSA-BD & WSSCC-B,
2011), as well as for what ODF means (Venkataramanan, 2015b). For example, one study found that
CLTS was 90 percent effective in achieving ODF if ODF was defined as having a functioning latrine.
However, if other factors were included (such as hygiene, presence of ash or soap at the latrine, and no
evidence of open defecation), then ODF levels dropped to 8 percent (Tyndale-Biscoe, Bond, & Kidd,
2013). Figure 1 shows an example of how much rural household latrine coverage results vary when
determined using the Government of Bangladesh and Joint Monitoring Programme (JMP) definitions. In
another evaluation, districts were classified as ODF if only 80 percent ODF, and in one unexplained
anomaly, a community was classified as ODF with only 38 percent ODF status (Africa AHEAD, 2013).
Figure 1: Percentage of Rural Household Latrine Coverage in ODF Declared Unions
Source: Hanchett, Krieger, Kahn, Kullmann, & Ahmed, 2011
For CLTS programs, ODF status is recognized at several different stages that occur sequentially – namely,
declaration, verification and then certification (Tyndale-Biscoe et al., 2013; UNICEF East Asia and Pacific
Regional Office, 2013). A significant period of time often elapses between a community being verified as
ODF and subsequently receiving the formal recognition of ODF status from the government (Tyndale-
Biscoe et al., 2013). Data reported in studies could reflect different stages of this process. Thomas and
Bevan compared CLTS indicators across several sub-Saharan countries and found several variations,
although almost every country surveyed in the region includes an iŶdiĐatoƌ of ͞ use of latƌiŶes .͟ ͞ Use of
latƌiŶes͟ can mean either each household having a latrine and/or evidence of no open defecation
(Thomas & Bevan, 2013).
Approaches are implemented differently. It is also difficult to compare results because even strong
methodologies like CLTS are not implemented the same way, even in the same country. Sanitation
marketing can range from simply training a few artisans or masons to build slabs or better pits for
latrines to setting up a for-profit business. Traditional sanitation promotion (latrine construction) might
encourage use of traditional pit latrines or innovations like ecological sanitation – or both – and might or
might not include hygiene training. Furthermore, even the best approach might not be implemented
well, or it might not be applicable in other contexts (e.g., urban versus rural). Another complication that
arises when trying to determine relative performance is considering the importance of factors (if the
approach/program is not trying to change them directly) like political will, policies, general wealth,
population, population density and cultural norms – often called the enabling environment.
Programs are generally under-studied and results are under-shared. Few NGO sanitation interventions
of any type have been revisited years later; if they have, we did not discover the reports during our
search. Despite ODF sustainability being a major topic of interest, many programs have little written
about them or reports from practitioners remain unpublished, partly because few countries or programs
have invested in this sort of research (UNICEF East Asia and Pacific Regional Office, 2013; Myers, 2015).
Progress may be over-reported. There are strong incentives (for example, additional financing) for
governments and implementing organizations to claim success, and their definitions/indicators of
success might not match those of other countries or monitoring organizations. For example, in a study in
Madagascar, the Ministry of Water had listed every community surveyed as having become ODF.
However, respondents indicated that only 58 percent of the surveyed communities had achieved ODF
(Ryan, 2014).
The collection of data is inconsistent. Sanitation coverage figures that are collected, collated and
compiled by government agencies (in most cases), nongovernment organizations, or even international
agencies vary significantly (FANSA-BD & WSSCC-B, 2011). For example, monitoring is reported to be an
important part of CLTS, but post-intervention monitoring is either not done consistently, not done for
long or not done completed at all (Myers, 2015). “oŵe ƌepoƌts state that ƌeliaŶĐe oŶ loĐal stakeholdeƌs͛
ability to monitor is critical, but skill- and knowledge-building and long-term follow-up are needed to
make this happen. It is also possible that the monitoring is being done by a local entity and thus the
outputs are not intended for global learning.
Findings
Definitions of Sanitation
Many definitions of sanitation exist. UNICEF͛s ϭεεϳ defiŶitioŶ of saŶitatioŶ foĐused oŶ people: ͞ a
process whereby people demand, effect, and sustain a hygienic and healthy environment for themselves
ďLJ eƌeĐtiŶg ďaƌƌieƌs to pƌeǀeŶt the tƌaŶsŵissioŶ of disease ageŶts͟ (UNICEF, 1997). However, a 2002
report focused on the hardware: ͞ Good saŶitatioŶ ĐaŶ ďe aĐhieǀed ƌeŵaƌkaďlLJ siŵplLJ; foƌ edžaŵple, the
ďasiĐ pit latƌiŶe Đoŵpƌises a pit, a Đoǀeƌ slaď, aŶd soŵe foƌŵ of supeƌstƌuĐtuƌe foƌ pƌiǀaĐLJ͟ (Cotton,
Scott, & Venkataraman, 2002).
To monitor progress towards the MDGs, the WHO/UNICEF JMP established a standard set of sanitation
categories. An ͞ improved͟ sanitation facility is one that hygienically separates human excreta from
human contact. Since the MDG targets are based on the use of improved sanitation and drinking-water
facilities, the JMP relies on nationally representative data from national statistics organizations, which
tend to use household surveys and censuses to measure the actual use of sanitation facilities by
household members (JMP, 2016). The definitions used by the JMP are often different from those used
by national governments. Estimates in JMP reports may therefore differ from national estimates
(WHO/UNICEF, 2015).
Since 2008, the JMP has presented data as a ͞ saŶitatioŶ laddeƌ͟ ǁith asĐeŶdiŶg ͞ ƌuŶgs͟ of seƌǀiĐe level:
open defecation, unimproved, shared, and improved (JMP, 2008). Each rung of the ladder is related to
specific technical choices, as shown in Figure 2.
Figure 2: Sanitation Ladder
Source: Adapted from JMP, 2008
Improved sanitation
Improved sanitation facilities
Ensure hygienic separation of human excreta from human contact, through use of the
following facilities:
• Flush/pour flush to:
o Piped sewer system
o Septic tank
o Pit latrine
• Ventilated improved pit (VIP) latrine
• Pit latrine with slab
• Composting toilet
Unimproved sanitation
Shared sanitation facilities
Sanitation facilities of an otherwise acceptable type shared between two or more households.
Only facilities that are not shared or not public are considered improved.
Unimproved facilities
Do not ensure hygienic separation of human excreta from human contact. Unimproved
facilities include pit latrines without a slab or platform, hanging latrines, and bucket latrines.
Open defecation
When human feces are disposed of in fields, forests, bushes, open bodies of water, beaches or
other open spaces or disposed of with solid waste.
As a household ascends the rungs of the ladder, the hygienic quality of the facility is supposed to
improve and thus reduce health risks to the users (Exley, Liseka, Cumming, & Ensink, 2015). However,
there is little evidence to support the ladder concept and little explanation of what actually constitutes a
hygienic latrine (Exley et al., 2015). There is also little evidence on how improvements up the ladder
translate into health benefits. The MDG targets and thus the JMP definition of improved sanitation did
not include the cleanliness, maintenance, or overall quality of the toilet facility, which led some to
suggest that the estimates of people with access to adequate sanitation are significantly overestimated
(Exley et al.
