PTI PERFORMANCE EVALUATION OF USAID/ZAMBIA CDCS (2011–2017) FINAL EVALUATION REPORT CONTRACT NO: AID-OAA-I-15-00023 TASK ORDER NO: AID-611-TO-16-00002 , Inc. November 2016 This publication was possible by the support of the American people through the United States Agency for International Development (USAID). The report was produced for the USAID/Zambia Mission by Dr. Valentine J. Gandhi, Mwila Kanema, Victor Peleka, and Wilfred Manda of IT Shows, Inc. PERFORMANCE EVALUATION OF USAID/ZAMBIA CDCS (2011-2017) FINAL EVALUATION REPORT Program Title: PERFORMANCE EVALUATION OF USAID/ZAMBIA CDCS (2011– 2017) Sponsoring USAID Office: USAID Zambia Contract Number: AID-611-15-00023 Task Order Number: AID-611-TO-16-00002 Contractor: IT Shows, Inc. 1655 N. Fort Myer Drive Suite 625 Arlington, VA 22209 Date of Submission: November 28th, 2016 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGEMENTS The Evaluation Team would like to express their appreciation to respondents from the Implementing Partners, the Government Officials in the Republic of Zambia, the USAID/Zambia staff, particularly the COR Kevin Chilemu and the ACOR Patricia Sitimela, and the various field staff who were essential in data collection process. Special thanks to the beneficiaries for their time in answering our questions. The team also appreciates the inputs of IT Shows Inc. home office staff particularly David Flemmons, Abdi Wardere and Terry Redding. Final Evaluation Study Team Dr. Valentine J. Gandhi, Team Leader Mrs. Mwila Kanema, Evaluation Methods Specialist Mr. Victor Peleka, Indicator Specialist Mr. Wilfred Manda, Training and Evaluation Specialist ACRONYMS ACOR ART CDCS CE CEA CIPP COR CSH CSO DEC DO DRG ECZ ET FACT FGD FTF FY GBV GCC GHI GRZ ICT INESOR IP IR ITS KII M&E MDG MNCH MoGE MOH MPI MSL NDP NRPCD PEPFAR PIR PMP PPR QA RA Alternative Contracting Officer’s Representative Antiretroviral therapy Country Development Cooperation Strategy (USAID) Cost effectiveness Cost effectiveness analysis Context, Input, Product and Process model Contracting Officer’s Representative Communication Strategies for Health Civil society organization Development Experience Clearinghouse Development Objective Democracy, human rights, and governance Electoral Commission of Zambia Evaluation team Fostering Accountability and Transparency Focus group discussions Feed the Future Fiscal year Gender-based violence Global Climate Change Initiative Global Health Initiative Government of the Republic of Zambia Information and communication technology Institute of Economic and Social Research Implementing partners Intermediate results IT Shows, Inc. Key informant interview Monitoring and evaluation Millennium Development Goals Maternal and newborn child health Ministry of General Education (formerly Ministry of Education) Ministry of Health Multidimensional Poverty Index Medical Stores Limited National Development Plan National Registration, Passport and Citizenship Department President’s Emergency Plan for AIDS Relief Project implementation reviews Performance Monitoring Plan Performance plans and reports Quality assurance Research assistant R-SNDP SMAGs SMGL SOW STIP SWOT TB TL TO TPM USAID USG WHO ZDHS Revised Sixth National Development Plan Safe Motherhood Action Groups Saving Mothers, Giving Life Scope of work Science, technology, innovation and partnerships Strengths, weaknesses, opportunities, threats Tuberculosis Team leader Task Order Team planning meeting U.S. Agency for International Development United States Government World Health Organization Zambia Demographic and Health Survey CONTENTS Page EXECUTIVE SUMMARY .................................................................................................... i Evaluation Purpose and Evaluation Questions .................................................................... i CDCS Background............................................................................................................... i Evaluation Methods ............................................................................................................. i Findings and Lessons Learned ............................................................................................ ii Recommendations............................................................................................................... v 1. INTRODUCTION.............................................................................................................. 1 1.1 Evaluation Purpose........................................................................................................ 1 1.2. Background and Problem Statement............................................................................ 1 Development Objective One: Enabling Governance Environment Improved................... 1 Development Objective Two: Rural Poverty Reduced in Targeted Areas ......................... 2 Development Objective Three: Human Capital Improved ................................................. 2 CDCS Areas of Intervention ............................................................................................... 2 1.3. Key Research Questions .............................................................................................. 3 2. EVALUATION DESIGN, METHODOLOGY AND LIMITATIONS......................... 3 2.1 Evaluation Design ......................................................................................................... 4 2.2 Methods......................................................................................................................... 5 2.3 Data Collectors.............................................................................................................. 5 2.4 Training and Data Collection........................................................................................ 5 2.5 Data Analysis................................................................................................................ 6 2.6 Sampling ....................................................................................................................... 7 2.7 Limitations .................................................................................................................... 7 3. FINDINGS AND CONCLUSIONS.................................................................................. 8 3.1. Q1: To what extent is the CDCS Contributing to increasing the prosperity of Zambians in an inclusive manner? ................................................................................ 9 3. 2. Q2: To what extent have crosscutting issues been integrated into the strategy to contribute towards CDCS success?................................................................................... 12 3.3. Q3: Which elements of the project and activities designed toward achieving DO 1 can be strengthened or scaled up in the next CDCS? ............................. 15 3.4. Q4: To what extent are the projects and activities designed under DO 2 contributing to reducing poverty in the targeted areas? .................................................... 17 3.5. Q5: to what extent are the project and activities designed under DO 3 contributing to the improvement of human capital? ......................................................... 27 4. RECOMMENDATIONS................................................................................................. 41 REFERENCES..................................................................................................................... 46 Page FIGURES FIGURE 1: Location of USAID CDCS Projects in Zambia.................................................... 3 FIGURE 2: Components of Stufflebeam’s (2003) CIPP Model ............................................. 4 FIGURE 3: USAID CDCS Development Objectives Budget ............................................... 10 FIGURE 4: President’s Initiative Funding ............................................................................ 11 TABLES TABLE 1: Ensuring Quality in Data Analysis ........................................................................ 6 TABLE 2: USAID and Implementing Partners Interviewed, Overall Sample Size................ 7 ANNEXES ANNEX 1: Scope of Work ............................................................................................. Ann. 1 ANNEX 2: Evaluation Matrix ...................................................................................... Ann. 12 ANNEX 3: USAID/Zambia CDCS Evaluation Common Analysis Framework: CDCS level ................................................................................................ Ann. 26 ANNEX 4: Indicator Tables by DOs............................................................................ Ann. 36 ANNEX 5: Qualitative Tools........................................................................................ Ann. 42 ANNEX 6: USAID/Zambia CDCS Performance Evaluation Respondents ................. Ann. 68 ANNEX 7: Trip Reports: Summary from Eastern Field Data Collection .................... Ann. 73 ANNEX 8: Performance Evaluation Organization Chart............................................. Ann. 80 ANNEX 9: Provinces and Districts Visited by Evaluation Team ................................ Ann. 81 Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) i EXECUTIVE SUMMARY EVALUATION PURPOSE AND QUESTIONS The Country Development Cooperation Strategy (CDCS) of USAID Zambia was evaluated by an IT Shows, Inc.’s external evaluation team from May to October 2016. This Performance Evaluation covered the period 2011–2017. The primary aim of the evaluation was to determine where the Mission is with respect to reaching its development objectives, and to inform the development and design of its next Country Development Cooperation Strategy (CDCS) by identifying what is and what is not working. CDCS BACKGROUND The United States Agency for International Development’s (USAID) Zambia CDCS directly aligns with Zambia’s development goals while supporting U.S. foreign assistance priorities. USAID/Zambia’s current CDCS runs from 2011 to July 2017. The CDCS comprises three development objectives, each with intermediate results (IRs): Enabling Governance Environment Improved; Rural Poverty Reduced in Targeted Areas; and Human Capital (education and health) Improved. USAID/Zambia seeks to achieve its goals by working with implementing partners (IPs). The CDCS activities were directly aligned with Zambia’s Revised Sixth National Development Plan (R-SNDP) goals. EVALUATION METHODS The CDCS is a broad strategy. To track the components within the CDCS in line with the Scope of Work (See Annex 1), the evaluation team developed an evaluation matrix (See Annex 2). The matrix expanded the key evaluation questions with further sub-questions at the CDCS and each development objective (DO) level, which were in line with the implementing partners’ target indicators. This served as a basis for the performance evaluation methodology development and design. The CDCS Evaluation Team (herein referred to as the CDCS ET or ET), used a mixed methods approach and triangulation of data, including document review (see References for full list) and qualitative surveys split into focus group discussions and in-depth key informant interviews. In addition, the ET conducted a semi-structured, web-based, follow-up mini-survey with the IPs and USAID staff. This combination of methods enabled the ET to obtain the data necessary to answer the five key questions and their corresponding sub-questions. The ET Five key evaluation questions framing the design: 1. To what extent is the CDCS contributing to increasing the prosperity of Zambians in an inclusive manner? Which elements of the CDCS are significantly contributing toward achieving the CDCS goal? 2. To what extent have crosscutting issues been integrated into the strategy to contribute towards CDCS success? 3. Which elements of the project and activities designed toward achieving DO1 can be strengthened or scaled up in the next CDCS? 4. To what extent are the project and activities designed under DO2 contributing to reducing poverty in the targeted areas? 5. To what extent are the project and activities designed under DO3 contributing to the improvement of human capital? How have the design aspects of DO3 contributed to improving health and education? Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) ii trained interviewers and collected data in four provinces: Eastern, Luapula, Lusaka and Copperbelt. Specific districts visited include Lusaka, Katete, Lundazi, Chipata, Mfuwe, Ndola, Kitwe, Chililabombwe, Luanshya, Chingola, Mansa, Mbereshi, Kawambwa, Kashikishi and Samfya (See Annex 9). FINDINGS AND LESSONS LEARNED Q1. To what extent is the CDCS contributing to increasing the prosperity of Zambians in an inclusive manner? Which elements of the CDCS are significantly contributing toward achieving of the CDCS goal? The CDCS activities were directly aligned with Zambia’s Revised Sixth National Development Plan (R-SNDP) goals. This was done through a series of consultative stakeholder workshops with partners from government, civil society, and development partners, as well as key USAID/Zambia Mission staff. These workshops helped in identifying key intervention areas both in terms of geographic locations and programs where the Mission could provide assistance. In addition, several independent assessments (a Political Economy Diagnostic Analysis, a Gender Analysis, and an Environmental Threat and Opportunities Assessment) were conducted prior to the design of the CDCS. The focus group discussions (FGDs), key informant interviews (KIIs), and document review confirm that the alignment of CDCS with Zambia’s national priorities was successful. The combination of these assessments also helped achieve targeted interventions at the field level. Among the USAID mini-survey responses, 15 out of 17 respondents (88%) felt that the development hypothesis was clearly articulated, and that the results framework was logical and provided pathways for introducing project interventions. As a qualitative performance evaluation, it is difficult to quantify the extent of USAID Zambia’s contribution to increasing the prosperity of Zambians. However, from the desk reviews, key informant interviews, and focus group discussions with beneficiaries during site visits, it is clear that, while challenges do remain, the CDCS strategy is contributing to improved quality of life, and activities are moving in the right direction to put Zambians on the pathways to decreased hardships. Q2. To what extent have crosscutting issues been integrated into the strategy to contribute towards the CDCS? Evidence from the evaluation suggests that right from the design phase, the results framework and the DOs were designed to be crosscutting and work together to maximize impacts. This can be observed in nearly all of the interventions implemented through the CDCS. The major crosscutting issues have been gender, youth, building partnerships, science, technology and innovation, and working with marginalized communities. Gender differences are given careful consideration in the design, implementation, and monitoring and evaluation of all program components. This is evident through desk reviews and qualitative surveys; Mission staff and implementing partners have been proactive in ensuring there is gender equity in their respective projects. The Mission has successfully applied creative approaches to enhance the focus on the critical development issue of youth. However, more work with youth has surfaced as an increasingly important theme moving forward in the next CDCS. Through USAID assistance, Science, Technology, Innovation and Partnerships (STIP) also emerged as crosscutting Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) iii interventions across all DOs. This is also reflected in the performance plans and reports (PPRs) 2011-2015 and the performance monitoring plan (PMP) on indicators. Through DO3, USAID health activities expand the impact of the United States Government (USG) investments by accelerating efforts to address the needs of marginalized and vulnerable communities, particularly in rural areas where the retention of health workers is low and access to quality health services is often difficult. DO2 also targeted interventions in rural areas in Eastern Province; these focused on strengthening farmers and contributed to gender equity and economic development, which are cornerstones for empowerment. Q3: Which elements of the project and activities designed toward achieving DO1 can be strengthened or scaled up in the next CDCS? Through DO1, USAID assisted in the capacity development of the National Registration, Passport and Citizenship Department (NRPCD) through improved financial management, administration, and technical capacity. It was interesting to note that USAID funding occurred predominantly in an election year (Zambian tripartite elections 2016); after elections, there was no funding. This caused challenges in sustaining such activities as voter registration, monitoring of the voter registration process, and nominations. Citizen demand for accountable service delivery has increased in limited areas of what is essentially proof-of-concept social accountability programming. There is a need to work more on the demand side; this work has already begun. Despite receiving one percent funding, DO1 has achieved a lot; work on the demand side has already started and is expected to be rolled out in the next phase. Zambia has shown a positive governance trajectory since 20111 . Zambia’s progress in overall governance performance in recent years has been underpinned by gains in three of the underlying governance components, most notably human development, to which USAID has contributed. The issues that surfaced of the 2016 electoral cycle show that accountability and transparency are acute needs—and may be consequential for saving competitive democracy in Zambia and safeguarding USG development investments. Q4: To what extent are the projects and activities designed under DO2 contributing to reducing poverty in the targeted areas? Findings indicate that DO2 activities have assisted to improve rural incomes and facilitate rural markets. However, it was also learned that not all IP activities have effectively addressed rural poverty. Respondents highlighted challenges in finding markets for their produce and the need for the next CDCS to exert efforts on corporate credit markets, as access to credit for rural populations has not fared well for both men and women. There is a need to ensure food security through the dry season by addressing water access activities, such as building boreholes and dams in target areas. Considering the dry spells that have hit the Eastern Province, building boreholes and dams would assist in crop irrigation and livestock survival through all seasons. However, it is also fundamental that technical assessments be considered to address ground water depletion issues. Findings also suggest the need to engage traditional leaders and communities in design and implementation. This would facilitate local ownership and promote needs assessment. 1 Ibrahim Index of African Governance (IIAG). Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) iv Within the environment sector, deforestation is a long-standing challenge that is largely driven by urban fuel needs coupled with poor agricultural productivity. Global Climate Change Initiative (GCC) programs emphasize innovative solutions to halt and reverse deforestation by reducing emissions from deforestation and land degradation (REDD+) as a replicable forest management strategy. Feed the Future’s applied learning provided substantial yields in terms of program impact, and an appropriate consideration of climate change increased the potential for resilience and adaptation (climate-smart agriculture). Q5: To what extent are the project and activities designed under DO 3 contributing to the improvement of human capital? How have the design aspects of DO3 contributed to improving health and education? EDUCATION According to the IPs and literature review, there has been progress in literacy as learners are able to read and write; however, a low reading culture and high illiteracy levels are still prominent in Zambia. Reading skills have improved in USAID–supported schools across all Early Grade Reading Assessment (EGRA) sub-tasks for both grades two and three learners2 . Between 2011 and 2015, the Mission supported the training of school administrators and the then-Ministry of Education3 (MOE; now MoGE) officials in the development of laws, policies and guidelines to improve equitable access to quality education. In 2013, there were 63 laws, policies, regulations and/or guidelines developed against a target of 43 (i.e., 147 percent of target). A focus on metrics and assessment guidelines drove the results under this indicator with significant increases in fiscal year (FY) 2013, as new education projects began extensive national efforts in support of reading policies, strategies and guidelines. 4 The Mission had a target of providing in-service support to public and community school teachers to increase their competency in teaching early grade reading, facilitate the development of reading materials, and train Parent Teacher Associations (PTAs). The evaluation team found that USAID had achieved more than 100 percent of the target in service training and intensive coaching in FYs 2012, 2013, and FY 2015. In FY 2011 the Mission achieved 20 percent of targets because the major activities were not procured at the time. The Mission supported education activities included the support of safe water and improved school sanitation facilities that allowed, among other things, adolescent girls to maintain menstrual hygiene, enjoy and stay in safe schools with healthy environments, and devote more time to learning. Health and hygiene education was also provided to teachers and learners. The Mission assisted the MOE to institutionalize and take over financial responsibility for an HIV/AIDS workplace policy program, strengthen HIV/AIDS and life skills education for learners, and integrate interventions. HEALTH The Mission supported one tuberculosis (TB) activity, which resulted in Zambia achieving a TB treatment success rate of 89 percent. This has also increased the proportion of registered TB 2 PPR 2015. 3 Now called Ministry of General Education. 4 PPR 2015. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) v patients testing for HIV from 83 percent in 2011 to 89 percent in 2013. Treatment of people living with HIV also increased from 149,365 in 2011 to a 2016 total of 288,934 adults and children receiving anti-retroviral therapy (ART), denoting an almost double percentage increase of 192 percent. Given the HIV rates in Zambia, this has been a significant achievement. Training in maternal health was another USAID investment. The four-year targets from 2012 to 2015 were achieved and exceeded. The results were above target because more community health workers were trained than anticipated in the original plan through the Saving Mothers, Giving Life (SMGL) and Safe Motherhood Action Groups (SMAGS) activities. To assist in detecting counterfeit medication and ensuring that defective commodities do not enter Zambia, the Ministry of Health (MOH), with support from USAID/Zambia, procured and installed quality control lab equipment. Additionally, Medical Stores Limited (MSL)5 used to face storage capacity challenges for its medicines. The Mission assisted in addressing these storage capacity challenges through outsourcing of warehousing storage space. With USAID support, the MOH has observed a reduction maternal mortality. This is evident in the Zambia Demographic Health Survey (ZDHS) 2013-14, which indicates improvements in maternal mortality from 729 to an estimated 398 per 100,000 lives. 6 The Mission assisted Zambia in making impressive gains in controlling malaria in the last decade, contributing to a 56 percent reduction in under-five mortality since 2001. This is down from 169/1,000 population to 75/1,000, as recently documented in the 2014 DHS7 . Overall, the project has demonstrated success in contributing to the improvement of human capital in health. A number of activities that were planned under this IR were achieved, and many notable successes were recorded in improving human capital in health. DO3 was significantly bigger with the two IRs for health and education. There has been feedback from Mission interviews and partners that DO3 can be split into two: Education and Health. RECOMMENDATIONS The performance evaluation scope of work (SOW) had elements that were specific to projects within the DOs. However, because this evaluation was at the CDCS level, the evaluation team compiled lessons learned that can serve as recommendations for future programming at both CDCS and DO levels. Some DO–level recommendations are applicable at the CDCS level as well. CDCS LEVEL RECOMMENDATIONS  The CDCS planning is highly participatory and considers the views of all stakeholders at the planning stage; this process should continue. USAID has enabled opportunities for local partners other than U.S.–based organizations to engage and participate in addressing Zambia’s development goals. Future CDCS planning should tap into this opportunity.  Increase funding for sectors beyond health (e.g., addressing HIV and poverty rates), particularly agriculture and economic development. There is huge potential for livelihood 5 MSL is the quasi-governmental company responsible for procurement of all the medicines for the government through the MOH. 6 Central Statistical Office (CSO), Ministry of Health (MOH), Tropical Diseases Research Center (TDRC), University of Zambia, and Macro International Inc. 2014. Zambia Demographic and Health Survey 2013-14. Calverton, Maryland, USA: CSO and Macro International Inc. 7 Ibid. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) vi diversification. The Mission can support and encourage increases in funding, however the Evaluation Team is aware that the national Mission does not have control over funding levels. There is potential to push for an integrated approach that realizes a scale up of sectors such as agriculture and economic development, which could see other potential sectors benefiting as well.  Expanding and intensifying activities should be embarked upon and success should be replicated where funds allow. However, these should be based on need assessments to ensure alignment to the priorities of the communities. Evidence is pointing towards the need for intensified investment and scale up within the Eastern province because saturation has not been attained (only four of seven districts reached). Similar to PEPFAR, intensifying approaches are essential in Eastern province to maximize gains.  Performance Monitoring Evaluation and Learning. A more ethnographic approach is needed at the CDCS level, in which the teams are embedded within the Mission to understand the workings and various activities of the CDCS, even before a final SOW for evaluation is drawn. It is recommended that the external teams involved in earlier evaluations be present at the design of future CDCSs, and during the implementation of various DOs. This should ensure that prior findings are carried forward, and that targeted rather than broad indicators are set, per an Implementation Science Approach. This would ensure that the CDCS does not become a static document but is able to adapt to changing environments.  More CDCS–level evaluation was recommended by key informants. There is a need to build flexibility into CDCS designs to allow for maximizing available resources inside and outside of USAID, fluctuations in funding, and changes to the development context and to Agency priorities. “CDCS is not a living document” refers to the fact that the CDCS was not updated, nor was there a mid-term stock-taking, to reflect important changes in the resource envelope and other management considerations.  Expand crosscutting innovation/information and communication technologies (ICT). There have been many successes in the use of ICT technologies across DOs; these can be harnessed and replicated.  Donor coordination should extend beyond meetings to provide, for example, feedback to IPs on government MOUs, planning meetings, and portfolio reviews.  USAID uses national indicators to monitor progress in achieving its development goals. The success of the projects cannot be credited entirely to USAID using these national indicators. USAID should therefore select some important national indicators to measure through surveys at the beginning and end of its projects to help demonstrate project impacts attributable to USAID. This is preferable to relying on national indicators connected to national surveys with different cycles to USAID’s project cycle. These selected indicators can be measured within the specified geographic location(s) of a given project.  The PMP and PPR need to match. The evaluation team reviewed indicators from the evaluation matrix, where there is a mismatch between what is in the PMP and in the PPR. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) vii These need to match in order to ensure ease of access to data. Therefore, the CDCS PMP must be a live document in line with a live CDCS that adapts as frequently as needed, such that a match between, for example, the PMP and PPR, is attained.  There should be room in the future CDCSs to allow for innovation and experimentation. For example, 3 to 10 percent of the budget could be allocated to experiment and innovation, such as piloting new initiatives, rather than following a strict adherence to CDCS processes. DEVELOPMENT OBJECTIVE LEVEL RECOMMENDATIONS Development Objective 1  USAID funding predominates when there is an election. The USAID policy is to have annual funding; however, the respondents feel that it is not sustainable, as the election cycle is 5 years. Funding is needed during all cycles, to include various activities before and during elections such as voter sensitization and registration, and also monitoring of nominations. The successes at the NRPCD headquarters need to be rolled out to provincial and district levels by building capacity and strengthening systems. Future CDCS planning should address both the supply side (government) and a demand side (civil society) of governance.  For the period 2011–2015, the World Bank scored Zambia high on political stability; fair on voice and accountability, regulatory quality, and rule of law; but very low on control of corruption and government effectiveness. An increased focus on anti-corruption activities on both demand and supply sides is recommended.  The DO1 activities had a slow start. Several targets were set in 2015, and results will only show in 2016 beyond this evaluation; therefore, continued funding and support is needed to strengthen DO1. Development Objective 2  Of the 12 FGDs conducted in 12 diverse communities of Eastern Province, 11 reported that communities were facing critical water challenges and in urgent need of access to clean water, both for their livelihoods and their livestock. This need was also stressed in a meeting with a Government of the Republic of Zambia (GRZ) local district official. USAID/Zambia could consider activities aimed at aiding the severe water situation highlighted by evaluation respondents. Therefore, the future CDCS needs to have more focus on addressing challenges in the water sector.  While many activities of DO2 indirectly benefit the poorest of the poor, needs-based interventions targeting small scale and subsistence farmers can help strengthen rural households.  Continue to invest in innovative approaches to maximize the benefits on climate change, natural resource and environment, nutrition, crop diversification, fisheries prominence, and livestock in Eastern province. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) viii  Indicators within DO2 can be developed with stakeholders in the next CDCS (e.g., the value of incremental sales and gross margins per unit for crops and livestock), or build capacity on tracking them, as the partners had challenges in calculating and reporting on some of these indicators.  The evaluation understands that activities have specified funding and limited time frames. However, it is essential that where activities have immensely contributed to reduced rural poverty, it can be the Mission’s interest to replicate successful agro-business models, especially those that improve value chains and make them replicable (e.g., COMACO/BLA, PROFITplus).  To achieve sustainability and long-term gains, explore long-term projects (preferably continuous for 10 years, according to stakeholders) for agricultural activities, as the agricultural sector is seasonal.  Engage with local ministry representatives at national and district levels to ensure authorities are aware of donor efforts and support. Engagement with local representatives is key, as province and district IPs are doing a good job of coordinating development efforts in rural communities.  To increase local ownership, it is critical to involve traditional leaders (e.g., local chiefs) earlier (from inception); they return the favor by assisting the project to succeed. Often, they are involved only after the project is set up and they are then introduced to the activities. Involving them as early as practically possible is recommended.  Move beyond training and include more activities such as asset building (e.g., Feed the Future). Development Objective 3: Education  The success recorded in education should be sustained and the partnerships maintained. However, USAID needs to develop one package that encompasses the entire wellbeing of children by tackling all the life-threatening risks in a child’s life. Alternatively, the focus should be on “early childhood development,” which affects most facets of development. Work has begun in this area and the Mission needs to continue in this path.  Need for transparency. The USAID working relationship with the GRZ has been very cordial and successful; it would help if USAID would be more open with the GRZ by sharing with them the project budgets so that expectations or suspicions are not raised (as was indicated by interviewees within the government).  The success of reading gains cannot be over-emphasised. There is a need to consider all government priorities as stipulated in the R-SNDP, in which case numeracy or mathematic skills should be reconsidered in the new CDCS for the Education IR. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) ix Development Objective 3: Health  Government stakeholders appreciate Government to Government (G2G) assistance to the GRZ. However, they are of the view that there is need to scale it up, as in some instances G2G assistance has had limitations as it cannot support government stakeholders with cash directly, but only through implementing partners (who may have different priorities). There was a request for USAID to play a supportive role and start funding government projects directly rather than through the implementing partners, e.g., “In some cases the Mission should give the funds to government through the Provincial Medical Office.”  The Government of the Republic of Zambia usually has high expectations from IPs, some of which are beyond their capacity. This is partly because when agreements are signed between USAID and GRZ, the IPs are not represented. The have worked well and continue to work. KIIs have indicated that USAID should share the contents of MOUs they sign with the GRZ with the IPs. The Mission could go further by bringing the IPs to the table when signing MOUs so that the IPs are well informed about the contents on the MOUs and know the expectations of the GRZ.  Programs like the TB programs worked well and saved a number of lives. The Mission needs to ensure that there are no gaps in programming; when a program comes to an end there has to be another, or a follow on, starting immediately. The Mission should avoid short-term extensions, which could be more expensive in the long run than if a project is awarded on a long term basis. This will not only help the Mission to cut costs but also help toward better planning by the IPs. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 1 1. INTRODUCTION 1.1. EVALUATION PURPOSE This report presents the final draft of the performance evaluation of USAID/Zambia’s Country Development Cooperation Strategy (CDCS) (2011–2017). This was conducted under Task Order Number AID-611-TO-16-00002 by IT Shows, Inc. (ITS). The purpose of the performance evaluation was to aid USAID/Zambia to assess the performance of its current CDCS, namely: 1) to determine where the Mission is with respect to reaching its development objectives; and, 2) to inform the development and design of its next CDCS by identifying what is and is not working. The broad objectives of the performance evaluation of the CDCS were:  To determine the extent to which each CDCS development objective is being achieved and whether each is on track to meet its intended goal.  To determine the extent to which the CDCS design is appropriate for achieving the intended goal.  To inform the development of the CDCS for fiscal years 2017–2022. 1.2. BACKGROUND AND PROBLEM STATEMENT Zambia’s long-term development strategy is pronounced in its “Vision 2030: A prosperous middle-income nation by 2030.” To reach this objective, the Government of the Republic of Zambia (GRZ) has put into place a series of national development plans. The current Revised Sixth National Development Plan (R-SNDP) encompasses the years 2013 through 2016. The R￾SNDP has three overarching objectives: infrastructure development, rural development, and human development. The United States Agency for International Development’s (USAID) Zambia Country Development Cooperation Strategy directly aligns with Zambia’s development goals while supporting U.S. foreign assistance priorities. USAID/Zambia’s current CDCS expires in July 2017. The CDCS comprises three development objectives, each with Intermediate Results (IRs) and a sound development hypothesis8 : DEVELOPMENT OBJECTIVE ONE: ENABLING GOVERNANCE ENVIRONMENT IMPROVED An effective, accountable, and transparent government is the foundation for growth and prosperity. It is the U.S. Mission’s top priority in Zambia, and is essential to long-lasting results for all development objectives. The objective is to foster an environment in which the Zambian government provides quality services in a transparent manner, and in which Zambian citizens expect high standards of government performance and hold under-performing officials accountable. An enabling governance environment reduces waste and channels public resources and energies toward productive purposes. To operationalize an enabling governance environment, USAID/Zambia works towards two intermediate results: 8 Based on the responses from the mini-survey, in which 15 out of 17 said that the CDCS was based on a sound hypothesis. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 2  IR 1.1: Citizen Demand for Transparent Accountable Service Delivery Increased  IR 1.2: Transparency and Accountability in Government Service Delivery Increased DEVELOPMENT OBJECTIVE TWO: RURAL POVERTY REDUCED IN TARGETED AREAS Inclusive development requires a focus on rural areas to benefit a population that has been largely excluded from Zambia’s recent growth. The CDCS targets specific areas based on need, population density and probability of success, which enables USAID/Zambia to leverage limited resources to a greater effect. Targeting rural-based poverty benefits the greatest number of Zambians and directly supports the GRZ’s own development priorities. As the majority of rural poor are smallholder farmers, approaches to achieving DO2 focus on reducing rural poverty through four intermediate results:  IR 2.1: Smallholder Agricultural Productivity Increased  IR 2.2: Markets and Trade Expanded  IR 2.3: Natural Resource Management Improved  IR 2.4: Resilience of Vulnerable Households Improved DEVELOPMENT OBJECTIVE THREE: HUMAN CAPITAL IMPROVED Human capital is a multi-dimensional concept that merges the knowledge, skills and capabilities that people need for life and work. Human capital in this context refers to education and health levels as they relate to economic productivity. The GRZ places considerable importance on human capital and its role as a prerequisite for Zambia’s development under the R-SNDP. Human capital is a crosscutting constraint in Zambia, and must be addressed holistically rather than through discrete interventions. Human capital requires an educated populace that is able to make sound decisions that affect the health and welfare of families, as well as a healthy populace that is able to participate fully in educational and economic opportunities. The Mission works towards achieving two intermediate results to meet this objective:  IR 3.1: Educational Achievement in Reading Improved  IR 3.2: Health Status Improved CDCS AREAS OF INTERVENTION Figure 1 (next page) is an illustrative map of the Republic of Zambia where CDCS projects were implemented. DO1 was carried out on a national scale. DO2 was focused in the Eastern Province and parts of Lusaka. DO3 was national, with a concentration on Lusaka, Copperbelt, Muchinga, Luapula, Northern and Central Provinces. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 3 FIGURE 1: LOCATION OF USAID CDCS PROJECTS IN ZAMBIA Source: Illustrative map compiled by authors. For a full list of districts, see Annex 9. 1.3. KEY RESEARCH QUESTIONS The performance evaluation sought to answer five key questions: 1. To what extent is the CDCS contributing to increasing the prosperity of Zambians in an inclusive manner? Which elements of the CDCS are significantly contributing toward achieving the CDCS goal? 2. To what extent have crosscutting issues been integrated into the strategy to contribute towards CDCS success? 3. Which elements of the project and activities designed toward achieving DO1 can be strengthened or scaled up in the next CDCS? 4. To what extent are the project and activities designed under DO2 contributing to reducing poverty in the targeted areas? 5. To what extent are the project and activities designed under DO3 contributing to the improvement of human capital? How have the design aspects of DO3 contributed to improving health and education? 2. EVALUATION DESIGN, METHODOLOGY AND LIMITATIONS This performance evaluation, designed as a learning and adaption tool for USAID/Zambia and its stakeholders, integrated mixed methods approaches to engage key stakeholders in evaluation planning, data collection, formulating findings, and recommendations. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 4 2.1. EVALUATION DESIGN Due to their strategic nature, the CDCS goals are broad and often focus on long-term impacts and wide-ranging objectives. Evaluating the CDCS required going beyond a straightforward “checking the box” approach of reviewing indicators and ensuring whether USAID targets for its implementing partners have been met or not. The evaluation process focused on understanding the dynamics behind the design of the CDCS, the results framework and subsequent projects, and eventually tracking the indicators through the Evaluation Matrix (see Annexes 2 and 4). The performance evaluation followed the principle of being useful to its intended users. Therefore, the performance evaluation of USAID/Zambia’s CDCS was planned and conducted in ways that enhances the likely utilization of both the findings and of the process itself, to both inform decisions and improve performance. During the review process, the evaluation team considered several utilization-focused evaluation models. 9 Given the complexity of the CDCS, the team decided to use Daniel Stufflebeam’s “Context, Input, Product and Process” (CIPP) evaluation model (Fitzpatrick, Sanders & Worthen, 2011; Mertens & Wilson, 2012; Stufflebeam, 2003; Zhang, Zeller, Griffith, Metcalf, Williams, Shea & Misulis, 2011). In this decision-oriented approach, program evaluation is defined as the “systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming” (Patton, 1997, p. 23). The CIPP evaluation model (see Figure 2) is a framework for guiding evaluations of programs, projects, personnel, products, institutions, and evaluation systems (Stufflebeam, 2003). FIGURE 2: COMPONENTS OF STUFFLEBEAM’S (2003) CIPP MODEL Source: Stufflebeam 2003. 9 The evaluation team considered several “utilization focused” and management-oriented system models, including the Critical Path Method (CPM) and Program Evaluation and Review Technique (PERT). Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 5 2.2. METHODS Between May and August 2016, the evaluation team used the following tools and methods: primary data collection, desk reviews, and a web-based mini-survey (for details see Annex 5):  Document Review: The evaluation team received various evaluative materials from the USAID/Zambia team: the USAID/Zambia CDCS, indicator data from the Performance Management Plan (PMP), Performance Indicator Reviews Sheets (PIRS), impact and performance evaluations, the USAID Partners in Development booklet, a results framework document and other relevant DO evaluative materials. The team reviewed several documents from the Development Experience Clearinghouse (DEC) relevant during the CDCS cycle. In total, the team reviewed nearly 300 documents and other information sources.  Key Informants Interview (KII): The key informant guide was transmitted at two levels. One was focused on USAID staff, with questions centered on CDCS themes and the results framework. The other KII guide focused on achievements, challenges and recommendations from key implementing partners.  Focus Group Discussion (FGD) Guide: The focus group guide was translated into two native languages (Bemba and Nyanja) spoken by 60 percent or more of the population in the data collection areas. The KII and FGD questionnaires were translated into Nyanja and Bemba. Translation quality was assured using a team translation approach assisted by INESOR. Interview guidelines were further refined based on observations during training, feedback from USAID/Zambia, and initial days of fieldwork. 2.3. DATA COLLECTORS All data collectors were bachelor’s degree holders from the University of Zambia. They received 1 week of training on data quality and the evaluation tools they would administer (i.e., KII, FGD guide). The evaluation team familiarized the research assistants (RAs) on fundamentals of the USAID/Zambia CDCS, evaluation protocols, and ethics. The RAs assisted in data collection in the field with key informants as well as focus groups, and in the transcription of collected data (for a team structure graphic, see Annex 8). 2.4. TRAINING AND DATA COLLECTION Preparation for fieldwork began with thorough training of the research assistants to conduct fieldwork. The ITS Team Leader and co-evaluators trained the field staff primarily from June 15–17, 2016. The Mission Contracting Officer Representative (COR) also was present during trainings to ensure quality control and to conduct a session on confidentiality issues. Training of research assistants included introduction to CDCS, evaluation design and methodology, and training and testing of data tools. A field manual with the compiled information was developed by the evaluation team and given to each research assistant. Field work began in Lusaka with Chiefs of Party of USAID project partners as well as government partners, who in turn gave the evaluation team lists of their field partners. A fieldwork plan was developed jointly by the Team Leader and the team members, and upon approval from the COR, a final field team of 10 Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 6 individuals conducted field data collection from July 21 to August 10, 2016 (for the full list of the respondents see Annex 6). Summary sheets from the key informant interviews and focus group discussions that had been reviewed and approved were uploaded to ITS servers on a daily basis, when possible. When lack of Internet access prohibited this, data were submitted prior to starting work the next day. The program management team at ITS headquarters worked with the team lead to review data and case completion regularly. These reviews informed fieldwork where necessary to improve data quality. 2.5. DATA ANALYSIS The evaluation core team analyzed the performance evaluation qualitative data using NVivo version 11. The focus of analysis was using the CIPP model to identify emerging themes, recurring themes, and to track indicators over the CDCS time period. The evaluation team developed a common analysis framework for data triangulation (see Annex 3). The evaluation team utilized different types of triangulation to validate skills, findings, analysis, and conclusions, including: a) data triangulation, using a variety of data sources; b) investigator triangulation, involving the use of different evaluators who bring diverse perspectives and cultural and analytical skills, all having relevant program experience; and c) methodological triangulation, in which the evaluation team used thematic analysis for the qualitative information. For a thorough description of data analysis methods, sources of data, design, approach, sampling and analysis, see Annex 3. The team developed a logical framework of how each question would be individually analyzed based on the nature of the question, the program structure of the DO, and type of indicators that are tracked by partners, as well as documenting lessons learned in the implementation. Table 1 below indicates the approach that was used by the evaluation team to achieve validity and ensure the quality of the analysis. This will help in addressing the limitation of comparing various DOs with different mandates, goals and intermediate results, as well as the challenges that may emerge in implementing each project in the DOs. The mini-surveys served as a confirmation of data gathered earlier through KIIs and FGDs, and contributed to the data quality described in Table 1. TABLE 1: ENSURING QUALITY IN DATA ANALYSIS Criteria Issues Solution Credibility (=internal validity) Truth value Prolonged and persistent observation, triangulation, peer-debriefing, member checks, deviant case analysis Transferability (=external validity) Applicability Thick description, referential adequacy, prevention of premature closure of the data, reflexive journal Dependability (=reliability) Consistency Dependability audit, reflexive journal Conformability (=objectivity) Neutrality Conformability audit, reflexive journal The quantitative indicators were tracked from desk reviews. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 7 2.6. SAMPLING The evaluation team utilized a three-tier approach in identifying the respondents for study. The sample of respondents in the first tier was provided by USAID/Zambia. This included IP Chiefs of Party participating as key informants; this then snowballed to their provincial managing officers. This second tier of IP provincial managers as well as GRZ partners were also interviewed as key informants, and this in turn snowballed to the third tier of beneficiaries, who participated in the evaluation as focus group discussants (to avoid bias the evaluation team selected the beneficiaries randomly from the list provided by the field implementers). The qualitative data collection conducted included:  67 key informant interviews with staff and key implementing partners of USAID/Zambia including: – USAID/Zambia staff – Ministries of Agriculture, Forestry, Health and Education officials – IAPRI, COMACO, Profit Plus, SAIOMA, Tetratech/Land Alliance, BioCarbon Partners and Mawa staff – Time to Learn, Expanded Church Response, FHI/Sarai Project, Community RISING, FACT, National Democratic Institute, Time To Learn, AIRS-Abt Associates and its sub￾partner Akros Research, PATH – Zambia Rising, FHI360-ZPCT  14 FGDs with beneficiaries from implementing partners in Lusaka, as well as in the field in Eastern and Central provinces (Table 2 below presents the overall sample size). For a full list of participants please refer to Annex 6.  31 respondents answered mini-survey questions (15 USAID staff and 16 IPs). TABLE 2: USAID AND IMPLEMENTING PARTNERS INTERVIEWED, SAMPLE SIZE Sample KIIs Mini Survey FGDs Total Sample Total USAID Total IPs USAID IPs Beneficiaries Overall sample size 9 58 15 16 14 112 For a full list of participants interviewed by the evaluation team please see Annex 6. 2.7. LIMITATIONS Per USAID’s 2011 Evaluation Policy, the evaluation team documented the limitations to the methodological approaches used for this evaluation. The performance evaluation in some ways compares apples with oranges, because each DO is unique; while they are indeed crosscutting and meant to complement each other, they still have different approaches and end project goals. For example, DO1 broadly focuses on the national level and the indicators are based on improvements measured at a national scale through national reports and other donor reports. DO2, on the other hand, has several micro-level interventions which can be easily tracked and can also be attributed directly to USAID intervention. In DO3, the health sector can have measurable and attributable results. However, the education sector can in some cases be at a Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 8 national level, where overall national progress has to be measured but cannot be attributed directly to USAID intervention. The evaluation team, therefore, ensured that each DO was examined and presented on its own merit, based on goals and targets, and lessons learned in implementing them over the past 5 years. At the CDCS level, the goals are strategic in nature and much broader, and can take longer to achieve. Therefore, an evaluation of this magnitude has to individually examine the progress made across all levels, while presenting the uniqueness of each level and how they come together to achieve the common goal (or if they do not, then identify the reasons for the same). This will result in recommendations for future CDCS design and planning. Indicators were provided by USAID; however, the desk review showed that several of the performance plans and reports (PPRs) did not contain indicators that the evaluation team was to track, or these were listed as “0.” In DO2 and DO3, certain indicator targets were set or changed annually, therefore it was difficult to observe trends for the entire cycle. Therefore, in the future more CDCS-level evaluations at all stages of the implementation could be useful for final evaluation teams. This point was alluded to by several key informants as well. Throughout the evaluation process the team worked closely with the USAID team, fine-tuned and finalized the list of indicators, and chose the ones most suitable to answer the key questions. Care was taken to ensure that indicators considered the external factors and the different cycles of measurements, as well as the nature of the indicators before attributing a change to USAID intervention. In addition, most data are secondary and much field data collection depended on the availability of respondents and datasets, and the timely delivery of materials from USAID to the ITS team. During field work, given that this was an election year in Zambia, most government officials were preparing for the general elections and accessing them was a bit difficult. In some cases, the Government and IPs at the provincial level were attending planning workshops and had to postpone some interviews. In some cases the evaluation team had to drop interviews, as they could not be completed within the budgeted time. Furthermore, FGDs were not conducted with education beneficiaries, as most beneficiaries were writing their mock exams at the time of field visits. These issues caused several planned FGDs to be dropped (See Annex 7 for field challenges in detail). Cost Effectiveness Analysis (CEA) was planned and the tools were designed by the team and shared with DO2 teams, however due to shortage of time as well as lack of comparable outcomes within projects, this was dropped. CEA can be done in the future CDCS by comparing several activities within a DO that are aimed towards a single outcome (e.g., increased awareness of condom usage to prevent HIV is most effective through radio, road shows, or television programs). 3. FINDINGS AND CONCLUSIONS The evaluation team reviewed the achievements of CDCS activity targets by triangulating the data from desk reviews of USAID and IP reports, and verified them through key informant interviews and focus group discussions. The USAID/Zambia CDCS performance evaluation findings are presented below and organized by findings about the CDCS, followed by DOs, with responses pertaining to the five key evaluation questions. Key indicators that were tracked and available to the evaluation team have been presented in table format in the annexes. The report follows a discussion format on achievements, activities, and evaluation team findings. The table Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 9 of indicators is based on PMP indicators made available to the team by USAID Zambia and the implementing partners (see Annex 4 for indicator tables). 3.1. Q1: TO WHAT EXTENT IS THE CDCS CONTRIBUTING TO INCREASING THE PROSPERITY OF ZAMBIANS IN AN INCLUSIVE MANNER? WHICH ELEMENTS OF THE CDCS ARE SIGNIFICANTLY CONTRIBUTING TOWARD ACHIEVING THE CDCS GOALS? FINDINGS: OVERVIEW OF CDCS OVERALL PERFORMANCE AGAINST THE SUB￾ACTIVITY TARGETS The major objective of the USAID/Zambia Mission is to bring prosperity to all Zambians; therefore, the CDCS was closely aligned with Zambia’s development needs, while supporting U.S. foreign assistance priorities. The evaluation team found that this alignment was achieved through a series of consultative stakeholder workshops, which included partners from the government, civil society, and development partners, as well as key USAID/Zambia Mission staff. These helped identify key areas for which the USAID Mission could provide assistance. Along with the workshops, a Political Economy Diagnostic Analysis, a Gender Analysis, and an Environmental Threat and Opportunities Assessment were conducted, prior to the design of the CDCS. In addition to a thorough analysis of issues to address, the key elements that contributed towards the achievement of the CDCS goals were the financial commitment, partnerships built, and successful integration of crosscutting issues. “The CDCS reflects on the Zambian development context in line with the Vision 2030, the SNDP (2013–2016), and with Zambia’s SDG targets, and also reflects on the USAID 5 year plan.”10 According to the World Bank, Zambia’s economy grew at an average annual rate of 7 percent between 2010 and 2014. However, global headwinds and domestic pressures have strained the Zambian economy. Consequently, growth in 2015 fell to an estimated 3 percent (compared to 4.9 percent in 2014) following a six-year low in copper prices, increasing power outages, and El Nino-related poor harvests. Growth is expected to remain around 3 percent in 2016, subject to the 2016 harvest, the mining industry’s reaction to softer copper prices, and stabilization of the power situation. The benefits of gross domestic product (GDP) growth have accrued mainly to the richer segments of the population in urban areas. Zambia has a very unequal income distribution (Gini coefficient = 55.6). Falling copper prices, exports and foreign direct investment (FDI) have weakened the economy. Copper prices declined by almost a third from their peak in February 2011 to $4,595/ton in February 2016 (LME) and are forecast to remain soft until 2018, as the global supply currently exceeds demand. Mine closures in 2015 led to the loss of over 7,700 jobs. Sixty percent of the population lives below the poverty line, and 42 percent are considered to be in extreme poverty. Moreover, the absolute number of poor has increased from about six million in 1991 to 7.9 million in 2010, primarily due to a rapidly growing population. 11 Social services have certainly improved over time, for example access to quality health care, access to schools, improved literacy among the general population, and an increased awareness on governance issues. 10 CDCS Document: https://www.usaid.gov/sites/default/files/.../USAIDZambiaCDCS30Sept2011.pdf. 11 World Bank report 2015. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 10 “There are tangible results that can be pointed to, for example, farmers have been able to improve production through economic growth interventions; many Zambian have accessed ART (antiretroviral therapy) through our working health, and millions of children (have been) reached with reading interventions in education. But for purposes of measuring a high level goal such as that of the CDCS, it is important to allow sufficient time for actual impact to be measured.” USAID key informant interview. As the majority of the Zambian population is youth, the current CDCS missed a core development opportunity, specifically for the education sector to scale up education interventions. The CDCS did align to the Zambian development context at the design time, and it was understood that the economic growth experienced during the last decade has not translated into significant reductions in poverty and improved general living conditions for most Zambians. Moreover, agriculture has not diversified, a weak education system is still a major drag on long￾term growth, and citizen voices remain rather weak. The CDCS adequately captures the intersection of sectors and the importance of crosscutting activities and cross-office collaboration. It does not, however, adequately capture the challenges presented by PEPFAR planning cycles that limit how the health office interacts with other parts of the portfolio. “The CDCS is strongly aligned to Zambia’s SNDP and has similar development objectives, except on infrastructural development, which is outside USAID’s remit. Overall, the CDCS supports Zambia’s development agenda.” USAID key informant interview. Financial Commitment: Overall, between 2011 and 2015, USAID invested a total US$1.3 billion in Zambia, which was distributed through the three DOs. In addition, another US$1.2 billion was distributed through the Presidential Initiative funding through Feed the Futures, Global Climate Change, and Global Health Initiative. Figure 3 presents the breakout of funding through the various DOs. Figure 4 presents the breakout of Initiative funding. FIGURE 3: USAID CDCS DEVELOPMENT OBJECTIVES BUDGET (In US$000s) Source: Desk review/USAID Zambia Program Office figures. 241,481 240,581 225,001 225,001 225,001 - 50,000 100,000 150,000 200,000 250,000 300,000 FY11 FY12 FY13 FY14 FY15 DO1 DO2 DO3 Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 11 Between 2011 and 2015, DO3 received 88 percent of the total funding. DO2 received 10 percent, and 2 percent went to DO1. Within DO3, health received 96 percent of funding, and education received 4 percent. FIGURE 4. PRESIDENT’S INITIATIVE FUNDING (In US$000s) Source: Desk review/USAID Zambia Program Office figures. The Global Health Initiative received 90 percent of funding, while Feed the Future and Global Climate Change received 7 percent and 3 percent, respectively. Partnerships built: Between 2011 and 2014, the USAID Mission supported 45 development partners, who helped to implement the activities under each DO. In addition, USAID also partnered with the donors’ forum in Zambia as well as with the GRZ through its partners. Crosscutting issues: CDCS was designed with purposive recognition of the crosscutting nature of Zambia’s development challenges, and these challenges were addressed through an integrated and focused approach that targets the greatest opportunities for impact and sustainable success. The crosscutting issues addressed by the CDCS were Science and Technology, Innovation and Partnerships, Gender, HIV/AIDS, and Environment. These are discussed in detail in the next section. CONCLUSIONS The well-established results framework with a measurable development hypothesis contributed to the overall success of CDCS. Realization of the project goal is premised on the hypothesis that two aspects to governance, a supply side (government) and a demand side (civil society), must both be addressed simultaneously to improve the enabling environment However, there have been areas that needed improvement, particularly strengthening of DO1, which has played a crucial role in ensuring transparency in government. The health portfolio got the major share; - 50,000 100,000 150,000 200,000 250,000 FY11 FY12 FY13 FY14 FY15 Feed the Future Global Climate Change Global Health Initiative Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 12 while it did address crucial health needs of Zambia, in the past years falling copper prices, exports and foreign direct investment have weakened the economy. This provides an opportunity for the next CDCS to invest more in economic growth and agricultural productivity to ensure livelihood diversification. The scale up of antiretroviral therapy (ART) in Zambia is impressive. This has both sociological and economic benefits as it impacts having a healthy population overall. The CDCS is set to meet most developmental goals, and is moving towards eradication of malaria and lowering the infection rate for HIV. On rural poverty reduction, to some extent there is improvement in food security in Eastern Province only, but poverty reduction remains a challenge in the country as whole. Education has also made some notable achievements. One of the most important developments is an increased focus on the quality of education, from a shift from counting the number of children who enter the school system and the number of classrooms built to what is happening to children while in the classroom. The launch of the Revised Curriculum and the National Literacy Framework (NLF) in 2013 were positive steps to this effect. However, a weak education system is still a major drag on long-term growth; and citizen voice remains rather weak. The indicators at the CDCS level were too broad to be attributed directly to USAID interventions. For example, the life expectancy at birth, which is a CDCS–level indicator, has improved from 57.5 years in 2011 to 60.1 years in 2014. 12 While this is an improvement, it cannot be attributed to USAID or its interventions alone, as there are several factors, such as other donors, government initiatives, and the private sector, that have contributed to this. However, this is an opportunity for future CDCSs to develop specific indicators, working closely with DOs, the Government, and implementing partners to develop specific indicators that can be attributable to USAID funding. Therefore, it would good in the future to relate attribution at the USAID/CDCS/DO level at the Intermediate Result level, since everything at this level is in control of USAID programing and funding. Overall however, the intermediary results have been achieved and they hypothesis were proven to be valid. 3.2. Q2: TO WHAT EXTENT HAVE CROSSCUTTING ISSUES BEEN INTEGRATED INTO THE STRATEGY TO CONTRIBUTE TOWARDS CDCS SUCCESS? FINDINGS USAID/Zambia has long recognized the crosscutting nature of the HIV/AIDS epidemic in Zambia, and has used PEPFAR resources to target HIV within other development assistance sectors. Mitigating the impact of HIV on the education sector, helping households affected by HIV participate in agricultural development, and advocating an end to the marginalization of people living with HIV and AIDS are key components of various sector strategies. The CDCS continues with this approach and, where appropriate, has incorporated PEPFAR funds into DO1 and DO2 activities. Furthermore, crosscutting issues such as governance, HIV/AIDS, gender, disability, nutrition, environment and disaster risk management have been mainstreamed in the SNDP. This entails that USAID development objectives closely align with Zambia’s development priorities and approaches, and be based on synergistic approaches that effectively link sectors, reflecting the intersecting nature of Zambia’s development challenges. As part of Serving the Poorest of the Poor, USAID health activities expand the impact of the United States 12 Human Development Report 2015. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 13 Government (USG) investments by accelerating efforts to address the needs of underserved communities, particularly in rural areas where the retention of health workers is low and access to quality health services is often difficult. Evidence from the evaluation suggests that from the initial design phase, the results framework and the DOs were designed to be crosscutting, and aiding each other to maximize impacts. This can be observed in nearly all of the interventions implemented through the CDCS. The major crosscutting issues have been gender, youth, building partnerships, science, technology and innovation, and working with marginalized communities. Gender differences are given careful consideration in the design, implementation, monitoring and evaluation of all program components. This is evident through desk reviews and qualitative surveys, where Mission staff and implementing partners have been proactive in ensuring that there is gender equity in their respective projects. The Mission has successfully applied creative approaches to enhance the focus on the critical development issue of youth. However, additional work with youth has surfaced as an increasingly important theme moving forward in the next CDCS. Through USAID assistance, science, technology, innovation and partnerships (STIP) also emerged as crosscutting interventions across all DOs. This is also reflected in the PPRs 2011-2015 and the PMP on indicators. However, there is need to strengthen STIP by improving it through innovative ways. Through DO3, USAID health activities expand the impact of USG investments. DO2 also had targeted interventions in rural areas in Eastern Province, which focused on such cornerstones for empowerment as strengthening farmers and contributing to gender equity and economic development. Following is a list of some achievements from various DOs that contributed to the success of CDCS across STIP:  In 2014, even in the absence of sustained funding, democracy and governance activities strengthened the linkages between citizens and government by working with government entities, elected officials and civil society groups. Through a novel partnership with the United Nations Development Programme, and with an eye toward the 2016 elections, USAID/Zambia supported the Electoral Commission of Zambia (ECZ) to develop and test an electronic system to transmit election results. The evaluation team found that democracy and governance activities promote equal and targeted opportunities for women and men. For example, the Zambian Elections and Political Processes Activity advanced women’s political engagement, and trained over 1,100 members of civil society (of which nearly 40 percent were women) to serve as election monitors.  Additionally, from 2012–2015, USAID/Zambia funded six research and development agriculture activities; these provided the foundation for growth in the sector by developing, adapting, field-testing, and disseminating new, locally appropriate technologies that contributed to household and national food and nutrition security. USAID/Zambia research and development activities addressed the orange maize activity and reached 1,953 lead farmers with 500 gram seed packs and fertilizer to promote the use of Vitamin A-rich orange maize seed. The produced was also tested with millers and supermarkets13 .  In FY 2014, five GCC activities were implemented by USG–supported Zambian organizations, and the private sector promoted appropriate technologies to implement energy 13 PPR 2014. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 14 efficient measures, reaching 1,269 households. These technologies included the use of fuel￾efficient stoves and higher-efficiency kilns for charcoal production, as well as Zambia’s first gel cooking fuel, which is produced from agricultural waste by a local business.  In FY 2015, Mission assistance through the CASH14 activity assisted in developing private sector partnerships (like MRI/Syngenta and Green2000) with horticulture producers for the provision of in-kind loans for small scale irrigation equipment, including treadle, solar, and diesel pumps, and drip irrigation, which maximizes water efficiency for vegetable production throughout the year.  Under the Better Life Alliance, COMACO15 worked with communities to construct 198 shallow water wells for the production of high value vegetables, orange flesh sweet potato vines, and Gliricidia seedlings. COMACO farmers produced over 20 million of these Gliricidia seedlings for dissemination to farmers in order to fix nitrogen into soil and provide mulch and firewood.  In education, the SMS Gateway aims to overcome infrastructure-related communication challenges by electronically linking district and school-level managers via a platform for uploading and sending key data and statistics from schools to districts, and vice versa.  “mLearning” is facilitated through Stepping Stone, a mobile multimedia teacher training course to increase the teacher’s contact time with students and cut down on the cost and time associated with travel to essential training courses. The lessons package has text, audio, interactive skill-building activities, and video instructions into best practices. It also encourages teachers to observe each other and give feedback on how to become more effective instructors. The Water and Sanitation Data Collection facility assessment tool is a real-time, tablet-based data collection system for baseline and ongoing monitoring.  With USAID/Zambia’s support, Zambia’s Ministry of General Education (MoGE) is now using empirical data to justify increased support of facilities and strengthen the Education Management and Information Systems (EMIS) database with relevant school water, sanitation, and hygiene data.  An electronic Early Grade Reading Assessment (eEGRA) utilizes mobile technology to address critical academic needs and bring the classroom to life. Teachers in community schools access eEGRA instructional videos and scholastic games via a simple cell phone. The technology also includes simple digital reading tests to help teachers understand their students’ reading levels and tailor remedial work for those falling behind.  Reading Without Books is an exciting tool to teach reading through vocabulary that learners already use at home. Based on the pronunciation learners use in their local language, teachers are able to lead pupils from sounds, to syllables, to words, to sentences, and then paragraphs without referring to a single textbook. When local teachers need a story, they use tales from the community or narratives written by teachers themselves. 14 Commercial Agribusiness for Sustainable Horticulture. 15 Community Markets for Conservation. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 15  The Locally Developed Reading Materials approach utilizes local culture as a source of inspiration. In partnership between USAID and the MoGE, teachers are trained to produce reading materials by writing grade-appropriate stories based on local myths, history, politics, and religion. Students easily relate to and identify with these stories, carefully edited to comply with the Ministry’s new reading curriculum. To address these concerns, USAID mobilized traditional leaders, empowered communities and established systems to tackle gender barriers and increase access to services for those at risk of HIV/AIDS.  USAID health activities expanded the impact of the USG investments by accelerating efforts to address the needs of underserved communities, particularly in rural areas where the retention of health workers is low and access to quality health services often difficult. Additionally, achieving gender equity is a major goal in Zambia for many sectors, including the health sector.  The health portfolio represents the largest funding at USAID/Zambia and therefore progress in achieving gender equity is of fundamental importance to USAID and the people of Zambia. Even though the Mission focuses much attention on women and children, men are also an important group in achieving gender equity through their meaningful involvement; they are also targeted for HIV prevention, such as through voluntary medical male circumcision efforts. Gender differences are given careful consideration in the design, implementation, and monitoring and evaluation of all program components. 3.3. Q3: WHICH ELEMENTS OF THE PROJECT AND ACTIVITIES DESIGNED TOWARD ACHIEVING DO1 CAN BE STRENTHENED OR SCALED UP IN THE NEXT CDCS? FINDINGS An effective, accountable, and transparent government is the foundation for growth and prosperity. This is the U.S. Mission’s top priority in Zambia, and it is essential to long-lasting results for all development objectives. However, while this is the top priority, the funding streams do not reflect this. The reality is that life-saving HIV care and treatment is the Mission’s Number One priority. This is a fact, based on the funding levels within DOs as shown in Figure 3 above. Despite accountable and transparent government being a priority and a U.S. foreign policy interest, this has not been supported by budget levels; the next CDCS needs to resource this aspiration and make it an operational priority for the USAID Mission. The flaws and distortions of the 2016 electoral cycle Activities under DO1  Worked with CSOs through NDI and FACT to increase their institutional capacity and capacity for advocacy.  Awareness programs with citizens to increase awareness of their rights, and encouraging them to expect responsible performance in government services.  Capacity building to ensure transparency and accountability in government service delivery  Worked directly with GRZ service delivery institutions to improve financial management, administration, and technical capacity. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 16 show that accountability and transparency are acute needs—and may be consequential for saving competitive democracy in Zambia and safe-guarding USG development investments. “The elections in Zambia being only funded during an election year poses a big challenge in building capacities and ensuring that systems and personnel are in place right from voter registration; nominations, monitoring of campaigns and voting, and most importantly post-election monitoring can greatly contribute to the gains so far achieved,” said an IP key informant during an interview. According to the 2015 Ibrahim Index of African Governance (IIAG), Zambia shows a positive governance trajectory since 2011. Zambia’s progress in overall governance performance seen in recent years has been underpinned by gains in three underlying governance components, most notably human development (to which USAID has contributed). The World Bank scores Zambia highly on political stability, fair on voice and accountability, regulatory quality, and rule of law, but very low on control of corruption and government effectiveness. USAID has built the capacity of the Department of National Registration, Passport and Citizenship Department (NRPCD) through improved financial management, administration, and technical capacity. Additionally, a national referendum, constitutional reforms, legislative amendments and elections are scheduled during the project’s period of performance. During implementation, opportunities arose to forge partnerships with local civil society organizations (CSOs) through NDI and Fostering Accountability and Transparency in Zambia (FACT), including the strengthening of parallel voter tabulation. However, a challenge was that funding was only given during the election year and not the full cycle of the election period (5 years). Partnering with the United Nations Development Programme and the Electoral Commission of Zambia, U.S. assistance reached over 1,400,000 citizens through civic and voter education activities to promote citizen participation in elections in 2015. Please refer to Table 3 in Annex 4 for details. In addition, USAID continued its support for the Electoral Commission to design and improve its electronic results transmission system. The Commission field-tested the electronic system in targeted, remote locations using mobile devices, with positive results. However, USAID funding occurs predominantly when there is an election; after elections there is no funding. This caused challenges in sustaining activities such as voter registration, monitoring of the voter registration process, and also nominations. Additionally, with USAID assistance 1,129 ministry staff were trained against a target of 500 between 2011 and 2014. Furthermore, an enabling governance environment provides the foundation for consolidating and sustaining overall gains from U.S. foreign assistance in health, education, and economic growth. Despite positive trends in electoral processes, Zambian democracy requires strengthening. The government struggles with corruption and maladministration in the delivery of public-sector services; key institutions are weakened by funding or policy constraints; and voter turn-out in recent elections has been at a record lows. However, in the August 2016 elections, the average voter turnout was 56 percent, compared to 41 percent in 2011.16 This can be an indirect benefit of activities under DO1. 16 Source: Election Commission of Zambia, 2016. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 17 Fostering Accountability and Transparency in Zambia works across health, education and environmental conservation and draws on Mission-wide funding. It is implementing advocacy activities on progressive policies through government line ministries and CSOs. However, it has yet to engage government officials at higher levels (as it has managed to on the demand side) and it has yet to scale up its activities for 2016-2017. CONCLUSIONS  DO1 is a significant component of the CDCS and through cross cutting efforts with other DOs. Some governance related activities were implemented.  The successes at the NRPCD headquarters need to be rolled out to provincial and district levels by building capacities and strengthening systems.  There is a need for sector-specific funding for DO1 to enable Mission work directly with government on vibrant democracy/governance targets and indicators. As noted by a USAID key informant, “We have a DO focused on improving government systems but we are not working directly with them on clear indicators.”  Activities that were supposed to be undertaken during the review period were not implemented, especially on the supply side relating to government delivery of services. However, some activities were conducted under the demand side through awareness meetings and training of election observers, including the contribution to strengthening voter parallel tabulation.  It was noted by the implementing partners that USAID funding occurs predominantly in election years. This is not sustainable, as the election cycle is 5 years. Therefore, there is a need to fund for all cycles. This includes various activities before and during elections such as voter sensitization and registration, as well as monitoring of nominations. DO1 continues to be an important component of the CDCS despite other DOs contributing to addressing governance issues in their respective activities, as it can address specific policy advocacy issues whose attention may not be a priority in other DOs. 3.4. Q4: TO WHAT EXTENT ARE THE PROJECTS AND ACTIVITIES DESIGNED UNDER DO2 CONTRIBUTING TO REDUCING POVERTY IN THE TARGETED AREAS? FINDINGS, REDUCED RURAL POVERTY This question has been answered by measuring high-level results that are attributed to the role played by the intermediate results in realizing outcomes. The evaluation team found that DO2 activities fall under two U.S. Presidential initiatives, Feed the Future (FTF) and Global Climate Change (GCC), and are mostly strategically located in the Eastern Province (EP) geographic target area, which makes up 85 percent of the province.17 17 PPR 2015. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 18 DO2 activities increased investments of and engagement in a market-led, interrelated approach at the household, community and landscape levels that challenges the core drivers of poverty. At the end of 2015, FTF activities had completed their third or fourth year of working in EP. The GCC, in their first or second years, are centered on the Luangwa Valley of EP, and complement FTF quotas. IP activities assisted farmers, households and communities to diversify their crops and enhance productivity, form linkages to markets, and improve beneficiary resilience. This included the use of advanced seed, labor reducing inputs, post-harvest storage, improved business and financial practices, and climate smart technologies that are able to improve yields, and reduce diseases. Implementing partners reported improvements in their reporting systems and the way they run their businesses since they engaged with USAID. USAID funding has also assisted them in facilitating farmer cooperatives. REDUCED RURAL POVERTY Overall findings indicate that there has been an increase in the number of rural households benefiting from USG assistance; the number of rural households benefiting has grown by 99 percent from 2011 to 2015 (See text box on page 15). Most communities reported they were still lacking access to clean and safe drinking water. Of the 12 FGDs conducted in 12 diverse communities of EP, 11 FGDs reported they were in urgent need of access to clean water, both for their livelihood and for their livestock. “The water situation in our community is quite terrible, oftentimes we share the same well with fiver other villages and drawing water is rotational, if one village draws today another Activities under DO2  To improve household-level resilience, the Better Life Alliance and the Global Development Alliance promoted land management practices through conservation farming, paying a premium price for commodities produced through conservation agriculture, while targeting vulnerable households and small holder farmers.  Health and nutrition activities by the Mawa Project bolstered resilience by improving the nutritional status of women and children. Activities also worked to increase assets and access to credit through Savings and Internal Lending Communities, which, with combined savings and loans, totaled US$310,581, benefitted 368 groups totaling 8,967 members.  Partnerships and policy strengthening with private sector partners to increase linkages to input and output markets and promote trade policies that encourage agriculture sector growth.  CFP jointly collaborated with communities and police to curb illegal forest activities, enabled forest patrols, provided community sensitization and training on the use of maps and geographic information systems (GIS).  Enhancing household resilience, benefiting 203,320 of Zambia’s most vulnerable households. Nutrition and gender activities were integrated to reduce stunting and improve women’s participation and empowerment. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 19 will draw tomorrow, meaning you cannot go back until your turn comes, as if that is not enough we also share drinking water with our animals, something has to be done soon to help us.” Focus group discussion, Beneficiary, Chipata. This need was also stressed in a meeting with a GRZ local district official. “Our request to USAID is to urgently assist us as a district with critical water shortage. Hardly a day goes by without us receiving a request to assist with water facilities in the rural areas of Eastern Province.” Senior government official key informant interview, Chipata. Water is fundamental in these predominantly agricultural communities, as it assists in crop irrigation through the dry season. With improved access to water, the livestock of these beneficiaries will better flourish, even when their territories experience dry spells. Sub-Intermediate Result 1: Smallholder Agricultural Productivity Increased IP activities employed new approaches to assist households to diversify crops and improve productivity. In particular, most IP activities were capacity-building and training related to new and labor-reducing inputs and agronomic techniques that increase yield, and improve agricultural productivity and post-harvest storage. From 2012 to 2015, there was an 81 percent increase in short-term training for improved agricultural productivity. The total number of beneficiaries trained was 304,626 in 2015, with the majority being small-hold farmers (see Annex 4). IPs reported that activities and approaches included: improved seed use, labor-reducing inputs, improved post-harvest storage, agronomic techniques that increase yield and decrease disease, and improved business and financial practices. Beneficiaries reported that IP activities by Strengthening Agricultural Input and Output Markets in Africa (SAIOMA) have assisted them with building post-harvest storage sheds to facilitate private and public market buyers. “After we as a community contributed to the construction of a shelter for storage of agricultural produce, SAIOMA also assisted us by purchasing more cement, metal rods and roofing sheets for our big storage shed, which can handle produce of the entire community; however, additional support is still required to complete the building.” Focus group discussion on DO2, Chipata. Research and development is another significant vehicle for raising agricultural productivity. IP research activities focused on the development of nutritionally high and drought-resistant seed varieties of maize, and supported community-based seed multiplication for new varieties of soybeans, groundnuts, cowpeas and legumes. According to one IP, the country has seen a shift toward greater reliance on research findings as a result of USAID funding; prior to this, policy makers used more ad-hoc policy decisions in agriculture. “With support from USAID we can say that agriculture policy research has become more evidence based and factual; there is a greater reliance on research findings in Zambia. For Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 20 instance, we have developed the electronic voucher which is way of marking input subsidies. We advise the House of Chiefs based on rural livelihoods statistics; the National Parliament and Ministry of Agriculture constantly consults our findings before embarking on most major decisions. As such we have seen a reduction in ad-hoc policy making.” IP DO2, Key informant interview, Lusaka. IPs also alluded to the improved yields in the target areas, such as working with vulnerable farmers to train them on the use of composite manure as fertilizer. “We combine a farmer support approach for training, and provide inputs with our business that sells value-added food products made from food crops we buy from the farmer members. From the sale of these products we are able to reward farmer commitment, improve farming and land use practices by offering premium prices for these surplus commodities when commitment is noticed, and destructive farming practices are abandoned by the farmer. We have also built and renovated a number of farming output storage sheds which act as storage points for farm commodities, a cooperative meeting place, and farmer shops.” Key informant interview on DO2, Chipata. Through the use of these practices in conservation farming, farmers’ yields for various crops have increased. As a validation of these positive changes, in the 2013/2014 season, the IP recorded an increase in production and sales for groundnuts and soybeans by 28 percent and 138 percent, respectively.18 Challenges were also noted by evaluation respondents. IPs noted that donors promote new technologies and believe that farmers can adopt them in order to increase productivity and food security. On the contrary, farmers decline by saying they cannot afford to buy reapers, fertilizer and certified seed due to limited finances. “Beneficiaries indicate they don’t have money for new technologies and often the challenge is working in a community where another donor is working and also giving them support in form of cash and we come in only with training; there is a tendency for the beneficiaries to shy away and opt for tangible benefits.” IP key informant interview on DO2, Chipata. Respondents indicated that they understand and appreciate accountability, however with USAID funding there are high restrictions on fund on fund allocation, leaving little or no room for innovation. Others highlighted the need to devise appropriate interventions and allow for thoughtful activities: “…if PMP says 200 farmers, you have to go and get that no matter what.” “Restrictions on money that cannot be used for tangible things; this restriction must be re￾visited. For example if a farmer is trained to be an aggregator, why not support them with the aggregation facility? By striking a 50-50 arrangement however with USAID funding, this is not the case; tangible development is avoided in such scenarios.” IP key informant interview on DO2. Chipata One IP indicated that there is some level of waste of resources from the donor side, as there is a tendency of not building on other efforts and often there is a lack of understanding of how rural communities and private sectors work. According to the IP, it is not a question of providing new 18 PPR 2015. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 21 farming technologies to farmers; there is need to realize that farmers are required to make money, and not simply go from one training to another. Another IP noted that viable positive outcomes have been forgone regarding agricultural productivity because of fund disbursement delays from USAID to IPs. “As essential as it is to promote new technologies, consideration of farmers needs is cardinal. I can give you feedback from the farmers but when you speak with them they will tell you that themselves; there are too many trainings provided by donors. One can even obtain a bachelor’s degree from the numerous trainings, if you ask the farmers they will tell you. ‘I don’t need training.’ Execute one good trade and the profits from that trade will never be matched by new trainings and technologies.” IP key informant interview on DO2, Lusaka. Sub-Intermediate Result 2: Markets and Trade Expanded USAID–supported activities worked with private sector partners to increase linkages to input and output markets and promote trade policies that encourage agriculture sector growth. These were investments in storage, linking farmers to private sector grain buyers and connected smallholder vegetable farmers to buyers at supermarkets, restaurants and hotels. However, evidence suggests that these connections could do better as agreements were not consolidated. From 2012 to 2014 there was a 99.8 percent increase in the value of incremental sales. However, for 2015, the indicator targets were not met due to a combination of factors, including difficulties in capturing an indicator (which relies on farmer recall, and has variables outside of IP control such as seasonal/weather aspects and pricing fluctuations). Another contributing factor was that one IP set very high targets for the year.19 IP activities assisted smallholder farmers by providing ready market for the sale of farm produce. IPs such as PROFIT+ built long term partnerships with two major private sector companies (SeedCo and Syngenta) to support smallholder farmers through linkages to input and output markets. “After 4 years of working with rural entrepreneurs we can say that we have been successful in building assets, we improved farmers’ access to markets and managed to link smallholder farmers to markets which were never in Eastern Province, and we developed a successful entrepreneurship model. We work with the chiefs and the churches. We have built resilience and diversification with the farmers that we work with.” USAID key informant interview on DO 2, Lusaka. The evaluation team’s focus was in understanding high level activities. However, FGDs also provided fundamental information on activity-level dynamics, which are equally key to future programming. Information obtained from FGDs reveals that, especially for those crops where markets were guaranteed, they are inclined and motivated to produce more; special reference was made to groundnuts, where IPs provided ready, prime markets. Beneficiaries reported that when 19 PPR 2014. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 22 sales are made to COMACO, they make optimal savings on time and transport, as collection is done from within the village. “USAID supported activities have really improved my wellbeing and that of my community members. When we sell to COMACO we save a lot on the cost of transport and maximize on profits. Instead of spending time and money searching for markets, we can now focus our energy on planning well for next farming season” Focus group discussion on DO2, Chipata. “Through USAID funding, COMACO has been buying our groundnuts and soybeans at premium prices. Consequently, we are inspired to produce more than before. Aside from that, we are also empowered with beehives to promote honey production, harvesting kits, built shallow walls, and provided reapers and seed inputs on a loan basis.” Focus group discussion on DO2, Lundazi. Household income derived from the sale of crop commodities and other income generating activities in beneficiary target areas of EP has almost doubled, from a value of $246 in 2011 to $401 in 2015.20 Food security at the household level has also been cumulative. Key informants indicated that they have built diversification with the farmers and improved access to markets. “After four years of working with rural entrepreneurs we can say that we have been successful in building assets, have improved farmers’ access to markets and the predictability of markets, entrepreneurship model. We work with the chiefs and the churches. We have built resilience and diversification with the farmers. As a result of USAID funding we have also been able to build partnerships with sustainability effects.” Key informant interview on DO2, Lusaka. This success can also be validated by IAPRI, which indicated that, compared to 2012 in Eastern Province alone, sales from groundnut production increased by almost 52 percent in 2015.21 Both key informants and beneficiaries noted several impediments to the expansion of markets and trade in USAID target areas, including inconsiderate terms and conditions given by private sector companies to smallholder farmers, and delays in processing of funds from USAID to implementers. One key informant noted that they link farmers to the markets and encourage them to sign forward contracts during the planting period of November-December. This commits them to guarantee markets to the producers; however, when the time comes there is a lack of commitment from the supposed buyers. “We connect the private companies to our farmers and CADs, however when it comes to doing business they are hesitant to give CADs inputs on consignment.” IP key informant interview on DO2, Chipata. Beneficiaries also indicated that IPs connect them to the markets; however, there is no follow up by IPs on the unfavorable conditions given to them. Furthermore, beneficiaries indicated that 20 COMACO, 2016. 21 Chapoto. A., 2016. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 23 although the markets for groundnuts and soybeans are certain, there is no definite market for the staple crop of maize. As some government entities are tasked to buy maize from the farmers, the price is pre-determined by the buyer and is not competitive, considering the costs spent on inputs. Consequently, they opt to use Malawian “briefcase buyers” who buy at a better price, although there are some minor losses incurred depending on the currency exchange rates. Some beneficiaries also highlighted that they spend a lot of time going to markets for their farm produce, thereby reducing on their productive time on the farm. “Transporting of produce in search for markets is a huge burden. For instance, in a farming season we may produce about 100 plus bags of maize, but for one to transport two bags of maize to Chipata town we can only use a bicycle and spend almost 2 days going and another 2 days coming back, meaning 4 days in total. In order to sell all my 100 bags, it means I will spend more than a month just transporting my produce.” Focus group discussion on DO2, Chipata. They also indicated that there is a great need for the markets to be brought closer to their communities as they travel long distances on very rough roads. Sometimes when they sell in towns, they are asked to return in two weeks for payment; in the end they lose out as most of the sale is spent on transport. Sub-Intermediate Result 3: Natural Resources Management There have been significant improvements in natural resources management. From 2012 to 2015, IPs trained farmers in conservation agriculture and agroforestry, resulting in 1,129,517 hectares under improved natural resource management. Beneficiaries reported that USAID–supported trainings have assisted their communities to transition from traditional to modern ways of farming. In 2015, environment activities focused on innovative activities under a three-pronged strategy. First, activities target reducing emissions from deforestation through community- and performance-based participatory natural resource management of globally, biologically significant, forested landscapes. In particular, activities demonstrate the viability of reducing emissions from deforestation and land degradation (REDD+) as a replicable forest management strategy. Second, GCC activities prioritize reducing poverty through the development and implementation of sustainable, community-based livelihoods and forest-based enterprises, expansion of sustainable farming systems and promotion of compatible markets and value chains that reduce threats to biodiversity and conservation objectives. From 2011 to 2015 there has been a number of activities directly aimed at benefitting rural households. The evaluation team found that IPs, through the Land Alliance, supported pilot program activities to strengthen tenure under customary land administration; this was mainly to advocate for rural, poor households to have ownership of and security for their lands. Key informants reported that, with USAID support, they succeeded in mapping 6,305 fields in EP, which are ready for certification.22 Consequently, 22 Key informant interview, Chipata. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 24 beneficiaries also reported that, because of the certified land demarcations, they are now able to live in peace in their communities. “Before the USAID–funded activity, they experienced a number of disputes, especially before the farming season, but with the authenticated demarcations, “we all know our boundaries and can focus on land preparations and farming.”23 Key informant interview, Beneficiary, Chipata Third, activities work to build local and national capacity of government and non-government stakeholders and institutions to implement Community Based Natural Resource Management (CBNRM) and REDD+ intervention. These activities target the development or strengthening of legal frameworks, policies and sub-national processes and partnerships. IP activities worked to address hunger and poverty through a business model, which rewards people living around forestry areas with improved ways of farming that motivate them to produce more. IP key informant interviews also mentioned that alternative programs, such as honey production projects and conservation farming, have been introduced in partnership with conservation IPs. “We work using initiatives with incentives such as cash payments to conservative famers as a motivation for the farmers to produce more with improved methods of farming.” IP key informant interview on DO2, Mfuwe. The GRZ and USAID have been engaged in coordinating the joint management of natural resources. The Forestry Department and the Department of National Parks and Wildlife have been engaged in joint policy developments; however, consensus and agreement are still ongoing. In 2015, a memorandum of understanding (MOU) was signed with the Zambian Wildlife Authority to clarify the roles and responsibilities of various partners in implementing forest and biodiversity conservation through REDD+ implementation and a first draft of a Participatory Forest Management Plan (PFMP) for the 73,531 hectares that comprise the Mwanya Chiefdom was submitted to the Zambian government for review. Additionally, an innovative Free Prior and Informed Consent (FPIC) process, called the “Flying FPIC,” was piloted in two participatory forest management areas, allowing 223 chiefs, advisors, community ambassadors and resource managers to conduct aerial mapping of the Mwanya Chiefdom’s Upper Lupande Block, which covers 280,675 hectares. Additionally, successful site selection processes were completed elsewhere in the Mfuwe and Nyimba sites and needs assessments that will inform FY 2016 activities were conducted in five chiefdoms in three districts. Quantitative data show the percentage of full conservation agriculture adopters in EP moved from 0.6 percent in 2012 to 8.0 percent in 2015. Partial adopter figures increased from 1.9 percent to 9.6 percent, respectively24. Beneficiaries embraced the concept of conservation agriculture and appreciated the guidance received from IPs. According to evaluation team findings, communities have seen, through IPs like CFP, significant improvements in treating and setting up community conservation areas (CCA) in order to protect the environment and bring 23 Focus group discussion, Chipata. 24 COMACO, 2016. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 25 about better farming practices. IPs reported that, concerning natural resource management, there is no GRZ policy in place, although it is in process of being implemented. Sub-Intermediate Result 4: Resilience of Vulnerable Households Improved The prevalence of exclusive breastfeeding of children under six months of age was at 23 percent in USAID target areas. The evaluation team found that a number of IP activities (Mawa) focused on increasing assets and access to credit through Savings and Internal Lending Communities (SILCs), reaching 255 groups with 5,443 members for combined savings and loans. Beneficiaries acknowledged the various efforts by the IP to improve livelihood of the most vulnerable households. USAID/Zambia-supported activities worked with the District Nutrition Coordination Committee (DNCC) and the Nutrition Food and Nutrition Commission (NFNC) to encourage the dietary diversity and household nutrition of parents and grandparents. Key achievements in improving resilience of vulnerable households were noted through positive changes in dietary diversity within the households involved, according to annual surveys conducted by the IP. “Through our work we are able to show that, even as farmers produced less, and actually had longer hunger periods, they had higher dietary diversity, which is a big achievement, because in years where there is poor rainfall and poor production, knowing very well that little can be done about the rainfall pattern. We were able to see households sustaining themselves due to better planning as a result of our work.” IP key informant interview, Chipata. Despite the poor rainfall patterns, key informants were comfortable that they trained people on dietary diversity and have noted behavior changes in that regard. CONCLUSIONS The evaluation concluded that USAID–supported activities improved the agriculture and environment sectors; however, more can be done to decrease rural poverty levels and further improve natural resource management In conclusion:  DO2 investments in agriculture were only 10 percent of the current CDCS budget. However, more finances need to be channeled to the sector to significantly attain reduced rural poverty levels and increase rural incomes. Considering Zambia’s abundant natural resources, there is significant potential growth for agriculture and wildlife tourism.  Agriculture needs to diversify from an over-reliance on maize at the household level, and adjustments related to diversity are essential at national levels. From field data collection, it is clear that small scale farmers have explored avenues of cotton, rice and groundnuts production. However, scaling up could be of great benefit as the markets for these may be not saturated in comparison to maize. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 26  Under current CDCS funding and programming, activities have minimal durations of one/two years. However, there is a need to consider longer-term activities, perhaps 10 years in agriculture: as a seasonal activity and with the complexity surrounding agricultural interventions, the activities take longer periods to yield results.  There is a need for the next CDCS to exert efforts on corporate credit markets, as access to credit for rural populations has not fared well for both men and women. The evaluation understands that the majority of the commercial banks require land titles as collateral from the small-scale farmers. However, most of the land is traditionally owned. It would be useful for Mission to engage the Ministry of Traditional Affairs, corporate credit markets and small scale farmers to amicably ponder the available options, such as lending through cooperatives. The Mission could also consider expanding on other successful credit options for rural areas such as the Savings and Internal lending model.  There is a critical need for numerous activities focused on improving access to water and sanitation in the Eastern Province. Beneficiary communities are in desperate need of resources, as indicated in all focus group discussions.  If USAID facilitates borehole creation, beneficiaries would benefit from crop irrigation and livestock survival through the dry seasons.  Evaluation findings indicate that most traditional leaders were not involved at the project design levels and were brought on board when the activity is fully operational. Consequently, as they indicated a lack of ownership, involving them at the design level can be beneficial. Key informants suggested that the next CDCS could include common areas management commitments (i.e., get commitments from chiefs to set aside agriculture land for conservation farming). Such commitments could leverage the way USAID rolls out programs to the communities.  USAID could encourage more needs assessment (i.e., sit down with communities before program selection; it could be that a community prioritizes moving with an environment component rather than an agriculture one).  Within the environment sector, deforestation has been a growing challenge that is largely driven by urban fuel needs coupled with poor agricultural productivity.  There has been substantial but recent progress on environmental aspects such as the drafting of the participatory forest management plan between IPs and the Zambian Wildlife Authority. Essentially, land was piloted in two participatory forest management areas, allowing chiefs, advisors, community ambassadors and resource managers to conduct aerial mapping of the Mwanya Chiefdom, which covers 280,675 hectares. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 27 Activities under DO3 (Education)  Improved education system management: Focused on strengthening institutional capacity to support and sustain improved learning.  Provision of in-service support to public and community school teachers to increase their competency in teaching early grade reading, facilitating the development of reading materials, and train PTAs.  Provision to disadvantaged and vulnerable children opportunities to participate and excel through access to quality education focused on early academic foundational skills in reading.  Help the Ministry institutionalize and take over financial responsibility for HIV/AIDS workplace policy program, strengthen HIV/AIDS and life skills education for learners, and integrate interventions.  Cross cutting activities included gender, USAID Forward, and USAID Education and Strategy requirements. 3.5. Q5: TO WHAT EXTENT ARE THE PROJECT AND ACTIVITIES DESIGNED UNDER DO 3 CONTRIBUTING TO THE IMPROVEMENT OF HUMAN CAPITAL? HOW HAVE THE DESIGN ASPECTS OF DO3 CONTRIBUTED TO IMPROVING HEALTH AND EDUCATION? FINDINGS: (EDUCATION) DEVELOPMENT OBJECTIVE 3.1: HUMAN CAPITAL IMPROVED (EDUCATION) According to the Mission, human capital is defined as a multi￾dimensional concept that merges the knowledge, skills and capabilities that people need for life and work. In the CDCS, human capital refers to education and health levels as they relate to economic productivity. The argument is that the long-term economic growth and political, social and environmental stability of nations depend on a population that is healthy, productive, and literate. The Mission further argues that aspects of human development are inherent to any integrated, long-term development program. Well-educated, healthy citizens are a national asset and a prerequisite for sustainable growth and prosperity. 25 Hence, USAID’s third development objective—Human Capital Improved—is to be achieved through attaining two intermediate results; the first focuses on improving educational outcomes, the second on improving health outcomes. IR 3.1: Educational Achievement in Reading Improved The Mission set an ambitious target and partnered with the MOE to improve education system management, with a focus on strengthening institutional capacity to support and sustain improved learning. The partnership with the Ministry has culminated in the implementation of new management practices in schools, and linked these interventions to transparent financial and management information systems, in order to improve decision making and resource allocation. The Mission also supported activities preparing a new generation of young Zambians with the academic and critical-reasoning skills necessary to achieve Zambia’s own strategic vision. In particular, improving learner performance in the core subject of early-grade reading provides the platform to launch a skilled workforce and a new generation of Zambian leaders. 26 25 PPR 2015. 26 Ibid. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 28 During key informant interviews and focus group with teachers, they indicated that there has been progress in literacy, as learners are better able to read and write. Although it has been slow, it is still a step in the right direction, within the backdrop of a low reading culture and high illiteracy levels in Zambia. According to the triangulated results from the EGRA assessment report, reading skills have improved in USAID–supported schools across all EGRA sub-tasks for both grades two and three learners in all six provinces. Grade two learners’ performance showed a more substantial percentage change than their counterparts in grade three.27 Furthermore, the FY 2015 results indicate that, against a target of 540,000, an actual number of 548,778 (102 percent) students were demonstrating reading gains at the primary level, an achievement that saw the Mission exceeding its target in FY 2015. In addition, findings revealed that USAID activities facilitated a notable change in the mindset and practices of individual education officers to overcome certain management norms that have undermined performance in Zambia’s education system. USAID IPs’ provincial advisors guided all 10 provinces and 94 out of 103 districts through the strategic planning process, a process that put learner performance at the core of planning and resource allocation. “Initially, before the Mission’s activities started, the planning was done from the national headquarters to the districts. The district and provincial offices never used to make strategic plans, but now the planning is starting from the districts to the national level; the DEBs and PEO offices are initiating the planning, something that has never happened before.” IP key informant interview, Lusaka. Advisors from the IPs provided support; Provincial Education Officers (PEOs) and District Education Officers (DEOs) were empowered to lead the effort. PEOs and DEOs set their own reading targets and the practical steps that were needed to progress towards their goals. Through this process, the Ministry of Education, Science, Vocational Training and Early Education (MESVTEE) officers at all levels are now able to pinpoint actions they could and should implement to effect change, including identifying necessary policy reforms and determining how to reallocate existing resources to better focus on learners. The FYs 2011 and 2012 had no results or targets, because three new education mechanisms were slated to begin in FY 2011 and were launched in Quarter 2 of FY 2012.28 These activities placed an emphasis on decentralized planning, programming, and training at the lower levels of the education system. This emphasis was consistent with the Government of Zambia’s commitment to decentralization and the empowerment of provincial offices to plan and execute education reform. In keeping with the Mission’s focus and concentration mandate, activities were limited to the Eastern Province, with implementing partner project staff embedded in provincial and district education offices. The education activities in FYs 2011 and 2012 facilitated over￾achievements, like the 102 percent increase in literacy that was witnessed in FY 2015.29 Additionally, in FY 2014, USAID’s continued partnership with the MESVTEE resulted in the finalization and launch of the revised national education curriculum and, subsequently, the “Let’s 27 Ibid. 28 PPR 2013. 29 PPR 2014. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 29 Read, Zambia” campaign. However, after the launch of the campaign, nothing happened; the program was never rolled out, citing financial constraints by the implementing partners. The stakeholders lamented the failure to roll out the campaign. “We called the Minister and he launched the “Let’s Read, Zambia” Campaign. Unfortunately, it was not rolled out because the IPs claimed they had no money to continue.” GRZ key informant interview, Lusaka. The Mission resolved to drop the indicator, “Proportion of students who, by the end of the primary cycle, are able to read and demonstrate understanding as defined by a country curriculum, standards, or national experts,” with the argument that the proposed measure and time frame (end of the primary cycle) did not accurately reflect the emerging Development Objective Monitoring and Evaluation needs (early grade reading). The Mission started focusing on outcome indicators in order to measure progress against baselines established in grades two and three. The conclusion was that the most logical outcome indicator for the USAID/Zambia approach to improved reading achievement is, “Proportion of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text.”30 The Mission had planned to activate this indicator in FY 2013; at the time, the baseline was anticipated to be established, and concrete FY 2014 targets with a clear rationale would have been in place. The FY 2013 National Assessment student testing data (administered at the grade five level in all Zambian public schools) in reading was used as the basis for this indicator, to show overarching progress under Goal One of the USAID Education Strategy. Results under this indicator were driven by USAID–sponsored interventions with the MOE. These interventions included policies, assessments, teacher training, provincial-level planning, and direct classroom interventions for improved student performance in reading as measured by annual progress on Zambia’s National Assessment examinations in grade five. The impact of these interventions was projected to result in a three percent annual improvement in student mean performance in English on the National Assessment. As of FY 2014, no activities were reported against this indicator. Thereafter, a 10 percent target was set for FY 2014. This target was based on the 2012 baseline, which indicated that close to 90 percent of children in grade two could not read at grade level. In FY 2015, USAID/Zambia set a target of 15 percent, which was achieved as more children were able to read and demonstrate an understanding of what they read by the end of grade two. The results of the November 2014 EGRA midline assessments (conducted by ongoing USAID–supported activities) showed that oral reading fluency at grade two improved from 7 percent at baseline (2012) to 15 percent at midline (2014) in target community schools; and from 10 percent at baseline to 20 percent at midline in target government schools.31 The survey results further indicated that overall, grade two pupils were struggling to read fluently; the average oral reading fluency rate for the local languages ranged from 1.84 to 8.40 words per minute. On the basis of results from the external baseline survey and those conducted by ongoing USAID activities, the MOE set a national 30 PPR 2012. 31 Early Grade Reading Assessment (EGRA). Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 30 reading (oral reading fluency) benchmark of 20 correct words per minute for an emergent reader.32 The Mission achieved a great deal in systems strengthening. Most of the indicators set under this sub-IR were over their targets, with achievements recorded over 100 percent. In FYs 2011, 2013, 2014 and 2015, the number of school administrators and MoGE officials successfully trained with USG support was achieved and exceeded plans. In addition to the improvements observed, systems strengthening of institutions’ strategic plans at the provincial levels were the other big achievement. “Systems strengthening of institutions’ strategic plans at provincial levels was a very big achievement.” GRZ key informant interview, Lusaka. The IPs consolidated partnerships and consensus with the Ministry facilitated a planning process that helped education managers at all levels to better focus and track the existing resources toward learning outcomes. This has been seen as the most effective way of MOE by the Mission has it has guaranteed sustainability of the intervention. The Mission supported the development of laws and policy regulations. A focus on metrics and assessment guidelines drove results under this indicator, with significant increases in FY 2013 as new education projects began extensive national efforts in support of reading policies, strategies and guidelines. In FY 2013, 63 laws, policies, regulations or guidelines developed against a target of 43 (147 percent above target), which was a significant achievement. However, numbers were narrowed in FYs 2014 and 2015 as the national literacy framework and other associated policies were finalized, and MoGE and USAID practitioners transitioned from policy development to execution. The number of policies and guidelines in the out years assumed MoGE motivation, planning and leadership of sector reform initiatives, and a renewed commitment to policy and planning exercises in support of reading outcomes. With the knowledge gained through the education activities, some PEOs started doing initiatives like the development of local policies (e.g., homework policies, remedial work and various other initiatives). Some districts started partnering with the private sector under the PPP, as in Chililabombwe, where some schools partnered with the Lubambe mine to build resource centers, libraries, and literacy centers. The Mbala district had schools that partnered with World Vision. This has been as result of the capacity that was built in the provinces and districts33 . The Mission, through its education activities, had a target of providing in-service support to public and community school teachers in order to: increase their competency in teaching early grade reading; facilitate the development of reading materials; train Parent Teacher Associations (PTAs); provide assistance for them to implement interventions that support reading; and support schools to plan and implement strategies to promote reading. Implementing partners worked with the MoGE to: strengthen school management and governance; institute assessments and performance standards for quality assurance; and engaged PTAs to encourage community 32 PPR 2015. 33 Key informant interview, IP Education Lusaka. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 31 involvement and oversight. Improved reading instruction, the availability of reading materials, and the use of assessments to feed back into the teaching and learning process, along with school management and oversight reforms, combined to improve learner outcomes in reading. The Mission achieved more than 100 percent in FYs 2012, 2013, and 2015 of the target in service training and intensive coaching. In FY 2011 the Mission achieved just 20 percent, because the major activities were not procured at the time. Similar significant results were recorded under the indicator for the number of PTAs or similar school structures supported. The results show that in FYs 2012, 2013 and 2015, 100 percent of PTAs were supported; FY 2011 recorded only a 47 percent achievement in support. The positive performance is an indication that the Mission was going to achieve all its set targets in the respective FYs. Had the procurement of the activities been completed in time, there would be no FY with low achievement (as observed in FY 2011). The Mission-supported education activities provided disadvantaged and vulnerable children with opportunities to participate and excel through access to quality education, which was focused on early academic foundational skills in reading. These activities included the support of safe water and improved school sanitation facilities that, among other things, allowed adolescent girls to maintain menstrual hygiene, enjoy and stay in safe schools with healthy environments, and devote more time to learning. Health and hygiene education was also provided to teachers and learners. A range of interventions were made available to support orphans and vulnerable children (OVC) school attendance, including scholarships and remedial instruction to help the most vulnerable learners go to school and complete their education. According to the Mission, USAID provided 45,000 students with scholarships. Equitable and inclusive academic institutions (with environments conducive to learning) increased both learner and teacher attendance rates, prevented student drop outs, and contributed to increased learner performance in foundational subjects such as reading and mathematics. In addition to a strong academic foundation in early grade reading, USAID’s gender strategy provided financial and psycho-social support to address critical gender concerns that often compel girls to drop out of school. The Mission addressed gender bias and inequitable learning conditions in education by awarding scholarships to girls and other vulnerable children. Scholarships covered school fees, uniforms, remedial instruction, and guidance counseling services. The Mission’s assistance also improved the quality of school learning environments for female teachers and girls by constructing water and sanitation facilities. In FY 2012, for instance, 39,265 girls and female teachers gained access to clean drinking water at school, increasing academic time on task (as girls were no longer required to walk long distances to fetch water, and were less likely to miss school due to water-borne diseases). Evidence indicates that girls are more likely to enroll and complete their education in schools with adequate sanitation facilities. This is more justification for USAID–funded activities to provide separate wash facilities for adolescent girls, which allows them to maintain menstrual hygiene throughout their teenage years.34 34 PPR 2012. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 32 The indicator results revealed that USAID had far exceeded its targeted level for most fiscal years. The Mission implemented two school water sanitation and hygiene education (WASH) activities in the Northern and Eastern provinces respectively as part of its school and learner improvement strategy and gender equity programming. The indicator targets were exceeded for learners enrolled: in FY 2012 targets were exceeded by 110 percent; in FY 2013 the figure was 177 percent, and in FY 2015 it was 105 percent. However, for FY 2011, only 1.8 percent was achieved because the activity was not implemented as anticipated. In addition, the existing school water and sanitation program missed targets for three reasons: 1) The USG negotiated new latrine designs and building standards with the MoGE. The MoGE’s final endorsement of the designs took longer than anticipated and delayed construction; 2) Efforts made to mobilize local communities in new districts exceeded the planned timeframe. Identifying laborers and other community contributions was a thorough and methodical process critical for local buy-in and sustainability; 3) the purchase and storage of construction materials in new sites was complicated by an absence of secure storage facilities. Logistical plans to identify storage facilities resulted in further delays. Despite the delay, all latrine construction planned for FY 2011 was completed by January 2012.35 Hence, the FY 2013 targets rose as the school WASH activity in the Northern Province was extended through 2013, and the WASH activity in the Eastern Province schools started its full implementation. FY 2013 included contributions from both school WASH activities (Eastern and Northern provinces) and saw a spike in results. The school WASH activity in the Northern Province was phased out in FY 2013, leaving only one school WASH activity in the Eastern Province for FY 2014. The results indicated that, from FYs 2013 to 2015, fewer people gained access to improved drinking water; this is because the targets were based on prior year enrollment figures in the MoGE’s statistical bulletin. The results did not include the number of people using the water points from the surrounding communities. In addition, the targets for FY 2012 were reduced as the school water sanitation program in the Northern Province phased out and the new school water sanitation activity in the Eastern Province began. As “WASH Plus” scaled up in the Eastern Province and reached out to the North Western Province in FY 2013, it was anticipated that the program would reach 100,000 individuals. In FY 2014 under the mechanism, targets were not met. Planning called for 50,000 people to gain access to an improved water supply by providing 180 improved water points. The implementing partners only managed to provide 100 water points, representing 56 percent of the planned total. This benefitted 84 percent of the people targeted. This was primarily due to an IP’s internal management challenges, which were partly attributed to additional levels of reporting and decision-making and an atmosphere in which the Mission had very limited access to the sub￾grantee responsible for construction. This led to the IP’s slow and inflexible response on issues that affected implementation and target achievement. The lack of various construction materials, such as cement and spare parts for the hand pumps, posed a challenge for the IP to reach its targets. 35 PPRs 2011-2015. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 33 According to respondent accounts, with USAID support education activities managed to conduct health fairs encompassing HIV/AIDS and other diseases. The findings revealed concerns that, before the USAID interventions, the HIV/AIDS pandemic had devastated the Zambian education sector, and had a resoundingly negative impact on teacher and learner performance. The interview respondents felt that, with the coming of USAID support, there has been a notable change. The Mission assisted the MOE in institutionalizing and take over financial responsibility for an HIV/AIDS workplace policy program, strengthening HIV/AIDS and life skills education for learners, and integrating interventions (such as counseling and testing , palliative care, and HIV prevention) for those affected by the disease. Despite this achievement, key informants in education felt there was not much collaboration between the health and education sectors. An example is adolescent health education, which was moved from education to health, and may make youth shun sex education. Education key informants would have preferred strengthening the linkages between the education and health sectors.36 CONCLUSIONS: DO3–Education  The education sector at the DO level did not have strong collaboration with other sectors in the current CDCS. This meant that DOs rarely engaged with the other teams (e.g., education did not have many joint plans with the health team, which has a similar target group of 8 to 24 years). Therefore, teams did not work together holistically.37 This was different with PEPFAR, as teams planned together because they integrated HIV into education activities, and this worked well. Unfortunately, they have stopped allocating to education the funds for HIV integration. The funding that was removed for PEPFAR was typically used for scholarships to OVCs and peer-to-peer and health clubs for a targeted group. Hence, different DOs and teams need to work and plan together as opposed to working in isolation. Despite these challenges the design has been steadily contributing to the development of human capital.  The design of the education portfolio has been narrowly focusing on basic education, excluding the youth who do not necessarily need basic education but rather tertiary education or practical skills. This investment in education further excludes trainee teachers in tertiary institutions who teach the children.  Overall, there is evidence that USAID–supported activities designed under DO3 are to a greater extent contributing to human capital through “Educational Achievement in Reading Improved.” A majority of the activities were accomplished and some exceeded the target. The Mission has made progress towards “Educational Achievement in Reading Improved.” These activities benefited a number of learners and teachers, and MoGE buy-in has been extremely positive, as the activities were done within the Ministry structures where PEOs were actively involved. However the Mission needs to assist the stakeholders to understand what it defines as quality education, as the stakeholders are of the view that providing materials like books and desks is not enough for a quality education. Therefore, there is a 36 Key informant interview, IP. 37 Key informant interview, USAID. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 34 Activities under DO3  Increasing access to health services, improving services quality, and improving utilization of clinical services.  Improving human resource capacity and management, drug logistics, monitoring systems, and capacity to conduct research and develop new interventions.  Working with community organizations to reach citizens and increase their knowledge of preventive behaviors and healthy practices.  Crosscutting activities included serving the poorest of the poor and gender. need to go into the class and see what is being done with the books, for example (Are the books being read?). Furthermore, some activities are too general; the standards officers need to refocus and look at standards set by the government.  The IPs also felt that there has been much attention given to the pupils, which was highly commended. However, there was supposed to be a balance of attention between the learners and the teachers: “There is also a mismatch; the Mission pays so much more attention to the pupils than the teachers, instead of looking at the curricula for colleges and universities” 38 . The Mission can invest an equal proportion in the learners and in the in-service and out of service for the teachers who teach, so as to have balanced investment to yield positive results.  The Mission dropped numeracy from its education IR because of its global strategy and the fact that other partners are tackling numeracy. However, government stakeholders did not seem to understand the Mission’s position. The evaluation team is of the view that, while the Mission has scored success in reading under IR, the success could have contributed more to human development if mathematics was maintained; the GRZ plan is to include numeracy, since improvements in reading alone cannot work in isolation.  The Mission put much effort into encouraging the vulnerable and disadvantaged to have equal access to education. This was successful as more girls and OVCs received scholarships. Despite this effort and success, most activities that were designed to improve access were procured at different time intervals. This created gaps with some activities that were ending while others were starting. (e.g., a water and sanitation project and an education project started and ended at different times. When the water and sanitation activities ended it compromised the equal access and quality for education.) DEVELOPMENT OBJECTIVE 3.2: HUMAN CAPITAL IMPROVED (HEALTH AND HIV/AIDS) IR 3.2: Health Status Improved (Health and HIV/AIDS) Improving health outcomes is the second intermediate result necessary to improve human capital in Zambia. It is argued that health status is a strong determinant of the strength of human capital and is directly linked to the productive capacity of a population. Improving individuals’ health status, in addition to improving quality of life, is vital to increasing the productive capacity of smallholder farmers, necessary for improving learners’ performance in school, and key to reducing household and government expenditures on health care, 38 KII IP Education Lusaka. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 35 freeing resources for other investments.39 The contribution of the health sector to human development was assessed through the sub-IR’s and the indicator tracked under the IR and the sub IRs. Tuberculosis The Mission supported one TB activity and provided direct support to the MOH. While the country’s overall positive results cannot all be attributed to the Mission, the success and contribution of USAID’s activity was significant as Zambia achieved a TB treatment success rate of 89 percent, and increased the proportion of registered TB patients testing for HIV from 83 percent in 2011 to 89 percent in 2013.40 Other achievements included: raising the percentage of USG–supported laboratories that perform correct TB microscopy from 80.6 percent in 2012 to 83 percent in 2013; training 1,127 health care workers in Directly Observed Therapy-Short Course (DOTS); and funding the first TB prevalence survey in 25 years in Zambia. Data collection, collation and analysis have been completed, and the following preliminary results have been released: Zambia’s TB prevalence is 560/100,000 (urban 664/100,000; rural 206/100,000); the prevalence of TB among HIV-positive patients is 1,088/100,000 and 194/100,000 for HIV-negative patients. In FY 2015, three activities were supported: 1) intensified TB case finding, 2) TB infection control, and 3) isoniazid preventive therapy. The Mission’s activity trained 138 health care workers against a target of 20, and implemented infection control plans in 28 districts against a target of 27.41 In FY 2014, the TB case detection rate was 68 percent, and the treatment success rate was 89 percent. Of all TB patients in FY 2014, 89 percent were tested for HIV, and 65 percent of all co￾infected patients were on ART. A full 83 percent of USAID–supported lab facilities correctly performed TB microscopy 95 percent of the time. Furthermore, Zambia has introduced and scaled up the use of the Xpert MTB/RIF diagnostic technology to increase TB case detection among people living with HIV/AIDS (PLWHA). The Mission has supported and/or funded Xpert MTB/RIF activities by training health workers, developing and producing guidelines and an algorithm for implementing the technology, and procurement. Zambia has also committed to addressing childhood TB by bolstering the diagnostic skills of health providers. With the Mission’s assistance, four central-level staff were trained in childhood TB. Set against these successes, the TB control program in Zambia also faces challenges. The national TB control program did not have an adequate number of staff at the national level to effectively and promptly engage the U.S government-supported implementing partner. The Zambian government has, however, recently increased staffing at the National TB Control Unit, which holds promise that the issue can be resolved. There has not been a Zambian TB survey for 25 years, which makes it challenging to follow trends at a national level. The country depended on global reports done by the World Health Organization (WHO) and other organizations, which at times did not provide specifics for national planning. 39 PPR 2015. 40 World Health Organization, Global Tuberculosis Report, 2015. 41 PPR 2015. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 36 In FY 2015, USAID did not support any TB control activities in Zambia with child survival and health funding, because the global mechanism through which USAID had implemented TB control activities since 2010 had reached its ceiling. At the time of the evaluation, a new mechanism was being procured. This created a gap in the activities under the TB program and it is not clear whether the IPs were communicated to when a mechanism has reached a ceiling. The IP respondents expressed great concern over the non-availability of a TB mechanism after it ended. “The only TB project came to an end and there is nothing happening; as a result people are dying of TB.”IP key informant interview for DO3, Lusaka Antiretroviral Therapy The Mission worked hard with the MOH to improve the quality of care provided by institutions, as well as their ability to reach Zambians. According to key informant interviewees and the triangulated data from the literature review, under this intermediate result the main focus was on increasing access to health services, improving service quality, and improving utilization of clinical services. In this regard, USAID demonstrated significant achievements in scaling up HIV treatment and prevention, surpassing planned targets. Information compiled revealed that a total of 288,934 adults and children eligible for antiretroviral therapy (ART) were receiving it during the FY 2016 second quarter. This is compared to 149,365 receiving ART in FY 2011. The result brought the number of adults and children on ART to almost double (192 percent) between FYs 2011 and 2015.42 The Mission surpassed its targets for the number of individuals who received HIV comprehensive prevention messages (more than 1.5 million) and the number who received HIV counseling and testing (more than 2.6 million). The current numbers for FY 2016 were incomplete for FY 2016, and most mechanisms promoting behavior change communications were closed. Male Circumcision Male circumcision has proved to provide up to 60 percent protection in the acquisition of HIV in HIV–negative males. It is further believed that optimally deployed, male circumcision can reduce the incidence of HIV in both male and females. The Mission’s quest to prevent HIV through male circumcision has seen success as set targets were over-achieved in the respective FYs, particular between FYs 2012 and 2015. Table 15 in Annex 4 shows the highlights of the achievements. The mobile program largely contributed to this record success. Family Planning The number of counseling visits for family planning and reproductive health in USAID–assisted service delivery sites in FY 2011 exceeded the target because the activities had transitioned from start-up activities to full implementation in support of new agency strategies. The FY 2012 target was 33,600 and the result was 111,090 visits (an excess of 330 percent). The target was exceeded because of an increase in the number of referrals through HIV counseling and testing services. Subsequently, the target was increased to 130,000 for FY 2013. However, in FY 2014 the target was underachieved (a target of 200,000 and a result of 98,462 visits). The target was (over-) estimated for the year without considering the fact that two IPs that were implementing the 42 PPR 2015. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 37 activity were closing out.43 The social marketing activity, which was previously tracking this indicator, had closed out during this period and the subsequent award was pending. Hence the project reporting on this indicator was closed in FY 2014 and there were no targets beyond 2014. Training in maternal health has been another major achievement for the Mission: All targets from FYs 2012 to 2015 were achieved. The results were attributable to more community health workers being trained than originally planned through the Saving Mothers, Giving Life (SMGL) activities; another reason is that training for community health workers (CHWs) was done in their respective locations, which came out to be cheaper than the central type of training for regular health workers. The FY 2014 target of 750 was exceeded by 30 percent, with 995 CHWs trained. This was because the sub-grantees trained more community volunteers as safe motherhood action group members than expected. This was also evident in the 2013-2014 Zambian Demographic and Health Survey (ZDHS 2013-2014), which revealed a decline in maternal mortality from 729 to an estimated 398 per 100,000 lives. 44 Sub-IR 3.2.2: Health System and Accountability Lab and Drugs Procurement The Mission procured and installed quality control lab equipment for the MOH, and trained MOH staff in the equipment’s use. The procurement of medical commodities and training of staff was to ensure that medical commodities entering Zambia met necessary WHO standards. The training of staff has assisted in detecting counterfeit medications, thus ensuring defective commodities do not the reach the supply chain. Furthermore, USAID procured US$71 million worth of essential medicines and other health commodities, including ARVs, HIV tests, laboratory commodities, opportunistic infection drugs, and male circumcision kits. Using validated data on commodity consumption rates from the USAID–supported national logistics systems, USAID helped the GRZ to quantify and forecast their commodity needs, which strengthened their capacity to lead the exercise in future years. Resources were used to support the MOH in the ForLab Quantification Tool training to enable them to conduct the national laboratory forecasting and quantification using four different methodologies. To manage commodities at central and facility levels, a total of 3,747 staff were trained in logistics systems. Logistics Management Information System The Mission also assisted the GRZ to develop an Electronic Logistics Management Information System (ELMIS) that has been rolled out to 100 high-volume health facilities, and trained 750 staff in its use. The system has improved supply chain visibility and assisted real-time commodity-consumption facility data, which has enhanced inventory management in the sector. Additionally, MSL used to face storage capacity challenges for its medicines. USAID has assisted in addressing these storage capacity challenges through the outsourcing of warehousing storage space. Sub-IR 3.2.3: Community Health Practices Improved 43 PPR 2014. 44 Central Statistical Office (CSO), Ministry of Health (MOH), Tropical Diseases Research Center (TDRC), University of Zambia, and Macro International Inc. 20014. Zambia Demographic and Health Survey 2013-14. Calverton, Maryland, USA: CSO and Macro International Inc. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 38 Gender-Based Violence In Zambia, nearly half (47 percent) of ever-married females ages 15-49 report ever experiencing physical, sexual and/or emotional violence from their current or most recent partner. Some 43 percent of all females experienced physical violence at least once since age 15, and 17 percent ever experienced sexual violence (Zambia Demographic and Health Survey).45 The Mission assisted in the fight against gender-based violence (GBV). It works with a mechanism that comprises three components working simultaneously toward a GBV theory of change (TOC), with the expected impact to reduce GBV and child marriage in Zambia. To achieve the desired project impact, STOP GBV is working to provide: (1) GBV survivor support services; (2) access to justice; and (3) prevention and advocacy. An additional objective, engaging men and boys through sports, was added to six scale-up sites as part of the first component in 2014.The mechanism was well designed, and was rooted in international best practices and lessons learned from its predecessor mechanism. This mechanism provides a clear and comprehensive multi￾sector prevention and response roadmap toward the expected impact of reduced GBV and early child marriage.46 According to the key informant interviews, triangulated with the evaluation report, reported cases of violence had more than quadrupled from the first quarter of STOP GBV operations to the latest quarter for which there were data (2015), indicating significant success of the mechanism. Reported cases of child neglect and abuse (non-GBV cases), physical assault and emotional abuse were increasing exponentially. Gains were also made in increased reporting of more stigmatized sexual and gender-based violence (SGBV) cases, such as rape and sexual assault, as well as early child marriage and denial of resources, but not at the same rate as the aforementioned case types. Despite the success recorded, GBV is still under-reported, and official statistics tend to underestimate the problem. Many forms of GBV, such as emotional and economic abuse, are not recorded, as there are no official police categories for these abuses. The police, courts and clinics do collect data, but there is a need to standardize and strengthen systems for reporting GBV cases, to both ensure that standard case definitions are used and to avoid double-counting of reported cases. In addition, some cases take very long to settle, which discourages victims.47 Medical Procurements and Distribution The Mission, in collaboration with the National Malaria Control Center (NMCC) and Ministry of General Education, leased storage space for the Long Lasting Insecticide-treated Nets (LLIN) and procured transportation services for their distribution to MCH facilities as part of the National Malaria Control Program’s mass distribution campaign. Furthermore, interviewees acknowledged that USAID supported the GRZ to develop, review and implement national health policies, program tools and guidelines. The Mission, along with other stakeholders, funded technical support to revise the National AIDS Strategic Framework, which was officially launched in April 2015. The Mission further supported the midterm review of the National Health Strategic Plan 2011-2015. It continues to support the development of the national Health Financing Strategy and the Community-Based Distributers Communication Strategy. The 45Central Statistical Office (CSO), Ministry of Health (MOH), Tropical Diseases Research Center (TDRC), University of Zambia, and Macro International Inc. 2014. Zambia Demographic and Health Survey 2013-14. Calverton, Maryland, USA: CSO and Macro International, Inc. 46 Midterm Evaluation of Stamping Out and Preventing (Stop) Gender-Based Violence (GBV) Zambia, 2015. 47 IP key informant interview, Lusaka. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 39 achievements for stock outs did not reach 100 percent, although the results were impressive. The percent of health facilities with laboratories experiencing stock outs of an indicator CD4 reagent recording an over-achievement of six percent in in FY 2011 and nine percent in FY 2015. Key informants attested to these achievements. “There has also been increased central level visibility and improved availability of critical data for supply chain decision making. This has led to accurate forecasting and quantification for health commodities. The F&Q process has continued to improve over time; as forecast accuracy has improved, stock outs at the facility level have decreased, therefore improving commodity availability in the country.” Key informant interview, GRZ Mansa. The evaluation team found that with successes recorded and capacity building complete, there are still some challenges that cause forecast accuracy fluctuations, such as a lack of issuance of commodities from MSL, slow implementation of guidelines, and central level availability. Overall, with USAID support, the forecast accuracy has been high in FY 2015 (77 percent) compared to FY 2014 (71 percent) and FY 2013 (61 percent); this correlates with decreased stock out rates from FY 2015 (16 percent) compared to FY 2014 (17 percent) and FY 2013 (30 percent).48 The evaluation team found that community health practices have slightly improved with the support from the Mission. For example, strides have been made towards reducing stunting rates, which were at 45 percent in FY 2011 and were reduced to 40 percent in FY 2015. According to the ZDHS 2013-2014, 40 percent of children under age five are stunted in Zambia, and 17 percent are severely stunted. Analysis by age groups shows that stunting is highest (54 percent) in children aged 18-23 months, and lowest (14 percent) in children less than 6 months of age. Severe stunting shows a similar pattern, with the highest proportion among children age 18-23 months (25 percent). This suggests that early child feeding practices contribute to the stunting that occurs among children under five in Zambia. The five percent reduction is small; there is still much work to be done, as the Mission’s goal is to reduce stunting to 15 percent by 2017. Maternal Mortality There has also been a reduction in maternal mortality, a significant achievement towards eradicating the maternal mortality rate. Despite this, there is still much room for improvement. The Zambia Demographic Health Survey (ZDHS) 2013-14 indicates improvements in maternal mortality from 729 to an estimated 398 per 100,000 lives. 49 Despite these achievements, Zambia still fell short of meeting the Millennium Development Goal (MDG) targets of maternal mortality ratio of 162/100,000 and infant mortality ratio of 35/1,000 by the end of 2015. Many recorded deaths are preventable, as evidenced by the 35 percent reduction observed in maternal deaths achieved in Saving Mothers, Giving Life districts through the implementation of maternal and newborn child health (MNCH) activities targeted at reducing the delays to reduce preventable deaths. 48 Health Office Annual Portfolio Review November 2015. 49 Central Statistical Office (CSO), Ministry of Health (MOH), Tropical Diseases Research Center (TDRC), University of Zambia, and Macro International Inc. 2014. Zambia Demographic and Health Survey 2013-14. Calverton, Maryland, USA: CSO and Macro International Inc. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 40 Malaria With support from USAID, Zambia has made impressive gains in controlling malaria in the last decade, contributing to the 56 percent reductions in under-five mortality since 2001 (down from 169/1,000 population to 75/1,000), as recently documented in the 2014 DHS. This achievement was acknowledged by the respondents. “There is a significant reduction in the malaria prevalence in the province. There have not been adequate measures put in place especially for the past 10 years, but it hasn’t been the case for the last 3years when USAID came to our aid.” Key informant interview, GRZ Mansa The mission through the President’s Malaria Initiative (PMI) partners with the Ministry of Health in general and the National Malaria Control Program in particular to undertake activities in four technical areas: 1) insecticide-treated nets (ITNs), 2) indoor residual spraying (IRS), 3) prompt diagnosis and treatment of malaria cases, and 4) intermittent preventive treatment in pregnant women. Insecticide-treated nets: In FY 2015, PMI procured 800,000 ITNs for the National Malaria Control Program’s routine distribution through antenatal and under-five clinics. Of these, 70,000 ITNs were distributed in communities in the border areas between Zambia and Angola. PMI is also supporting durability monitoring of nets that were distributed during the FYs 2014–2015 campaign. Similar to previous net durability monitoring efforts, data are being collected by Peace Corps volunteers at the provincial and local levels. Indoor residual spraying (IRS): The Mission targeted IRS in 40 high-malaria-burden focus districts. Spraying covered 409,544 structures and protected approximately 1.5 million people. The Mission supported environmental compliance inspections and insectary and resistance monitoring. Activities monitored insecticide resistance and continue to develop a more complete map of insecticide resistance in the country. During the FY 2015 spray season, USAID supported the use of satellite imagery to map structures in 15 PMI-supported IRS districts, define IRS target areas, and inform insecticide procurement. GIS technicians identified structures and recorded GPS coordinates for roof types and house sizes. Prompt diagnosis and treatment of malaria cases: In malaria case management, the USG assisted in the procurement and distribution of over two million rapid diagnostic tests and trained 564 health workers in their use. Assistance was given to national, provincial, and district staff, who received outreach training and supportive supervision to assure the quality of malaria diagnostics. Over 1.8 million treatment courses of artemisinin-based combination therapy (ACT) were procured with U.S. assistance, and 464 health workers were trained in the case management of malaria. In FY 2014, the National Malaria Control Program and partners revised the Guidelines for the Diagnosis and Treatment of Malaria in Zambia to identify injectable artesunate as the first-line treatment for severe malaria. In FY 2015, PMI procured 150,000 vials of injectable artesunate and partnered with the Clinton Health Access Initiative for a phased roll￾out of the drug. Additionally, PMI continued to assist the ministry in the forecasting and quantification of ACTs, ITNs, rapid diagnostic tests and sulphadoxine-pyrimethamine, which strengthened the distribution of malaria commodities at all levels. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 41 Intermittent preventive treatment in pregnant women (IPTp): With regard to malaria in pregnancy, the national program has updated guidelines for IPTp to recommend preventative treatment as early as possible during the second trimester of pregnancy, with subsequent doses at every planned antennal visit and when possible, monthly doses given up to the time of delivery. Zambia’s 72 percent national coverage rate of at least two doses of IPTp is among the highest in Africa. According to the findings, the malaria burden among children under 5 years was reduced from 39.9 percent in FY 2014 to 30.7 percent. Also, the USG continues to support the development of the National Health Financing Strategy to improve the sustainability of health resources in Zambia. Still, gains in addressing malaria burden are uneven across the country. Preliminary data from the 2015 Malaria Indicator Survey (MIS) indicate that three of the Program for the Advancement of Malaria Outcomes (PAMO) target provinces—Luapula, Muchinga, and Northern—have among the highest under-5-year-old rapid diagnostic test (RDT) prevalence rates of malaria in the country. CONCLUSIONS Overall, the development objective has demonstrated success and positive strides in contributing to improvement of human capital in health. A number of activities that were planned under this IR were achieved and there was much notable success recorded. There was significant achievement in all the eight national indicators which the project is measured against, as planned in the project appraisal document (PAD). 50 4. RECOMMENDATIONS This performance evaluation SOW had elements that were specific to projects within the DOs; however, given that this evaluation was at the CDCS level, the evaluation team has compiled lessons learned that may be applicable to individual DOs but can also contribute to the next CDCS at the design level. This is a list of recommendations at both CDCS level and DO level, and some DO level recommendations are applicable at the CDCS level as well. 4.1. GENERAL RECOMMENDATIONS  The CDCS planning is highly participatory and considers the views of all stakeholders at the planning stage; this process should continue. USAID Forward has enabled opportunities for local partners other than U.S.-based organizations to engage and participate in addressing Zambia’s development goals. Future CDCS planning should tap into this opportunity.  Increased funding for sectors beyond health (e.g., HIV rates dropping, poverty rates remain high), particularly agriculture and economic development. There is huge potential for livelihood diversification.  Expansion based on need assessments (e.g., DO2’s reach beyond Eastern Province as in the earlier CDCS, but much more targeted). Successes can be replicated. 50 PAD 2014-2019. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 42  Evaluation and learning: a more ethnographic approach is needed at the CDCS level, where the teams are embedded within the Mission to understand the working of the CDCS and its various activities even before a final scope of work for evaluation is drawn. Evaluation teams at the time of design of CDCS and during implementation of various DOs is recommended to ensure findings are carried forward as well as targeted indicators rather than broad indicators are set; the Implementation Science Approach. More CDCS level evaluations are recommended by the key informants.  Expand innovation/ICT, which are crosscutting. There have been many successes in the use of Information and Communication Technologies across DOs, these can be harnessed and replicated.  USAID uses national indicators to monitor the progress towards achieving its development goals. The success of the projects cannot be credited entirely to USAID using these national indicators. USAID should therefore select some important national indicators that will be measuring through surveys at the beginning and end of its projects to show the impact of the project attributed to the USAID, as opposed to depending on national indicators that in turn depend on national surveys with different cycles to USAID’s project cycle. These selected indicators can be measured within the specified geographical locations of a given project.  Future CDCS evaluations focusing on social network analysis or organizational capacity assessment at all levels. This recommendation comes from feedback on human resource issues that were raised during the interviews (e.g., reward mechanisms for staff involved in higher responsibilities).  The CDCS is a living document; as such it has to be updated and revised, and always aligned with National Development Plan (NDP) revisions. It was established that the R-SNDP and PMP were revised at some point, but the CDCS was not revised. The Mission needs to ensure that the CDCS is always revised to match the revisions in the R-SNDPs and the PMP if there are revisions at the national priority planning in the life of the CDCS. The PMP and PPR also need to match. The evaluation team reviewed indicators from the evaluation matrix, where there is a mismatch between what is in the PMP and the PPR. These need to match in order to ensure ease of access to data. Therefore, the CDCS PMP must be a live document in line with a live CDCS that adapts as frequently as needed such that a match between, for example, the PMP and the PPR is attained.  There should be room in the future CDCS to allow for innovation and experimentation. For instance, 3 to 10 percent of the budget allocated to experiment and innovation, such as piloting new initiatives rather than having strict guidelines on CDCS processes.  “USAID needs to be forthcoming about country operational plans and targets being set. Some of them come out in presentations, but there is need for clarity. For example, some of them being geographical issues, it will be really more helpful if USAID is more transparent about those things, so the implementing partners understand where the donors are coming from.” While the evaluation team found that the Mission did its best to ensure that partners Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 43 understood the Mission objectives, there is still a need for further capacity building of partners and room for learning. 4.2. DEVELOPMENT OBJECTIVE LEVEL RECOMMENDATIONS Development Objective 1  Governance strengthening took place in other Mission programming and can be leveraged in next CDCS. Sectoral governance mechanisms such as neighborhood health committees and school community partnership committees received significant technical support and strengthening in the CDCS period through non-DRG (democracy, human rights, and governance) programming. As a result, planned DRG cross-sector programming will seek to scale up these channels for meaningful citizen-government collective action.  According to the IPs, they receive USAID funds when there is an election. This is not sustainable, as the election cycle is 5 years. There is a need to fund all cycles, including various activities before and during elections such as voter sensitization and registration, and also monitoring of nominations.  Future CDCS planning should address both the supply side (government) and demand side (civil society) of governance. Hence the need to consider decentralization.  For the period 2011-2015, the World Bank scores Zambia high on political stability; fair on voice and accountability, regulatory quality, and rule of law; but very low on control of corruption and government effectiveness. An increased focus on anti-corruption activities on both the demand and supply sides is recommended.  The successes at the NRPCD headquarters need to be rolled out to provincial and district levels by building capacities and strengthening systems.  Some DO1 activities had a slow start. However several other activities were beginning in FY 2015/16 and results will only show in 2016 and beyond this evaluation; therefore continued funding and support is needed to strengthen DO1. Development Objective 2  Of the 12 FGDs conducted in 12 diverse communities of Eastern Province, 11 FGDs reported they were facing critical water challenges and were in urgent need of access to clean water, both for their livelihood and livestock. This need was also stressed in a meeting with a GRZ local district official. USAID/Zambia could consider activities aimed at aiding the severe water situation. Therefore, a future CDCS needs to have more focus on addressing challenges in the water sector.  Agricultural interventions should target the poor in addition to existing activities; while many activities of DO2 indirectly benefit the poorest of the poor, need-based interventions targeting small scale and subsistence farmers can help strengthen rural households. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 44  Continue to expand the projects on climate change, natural resource and environment, nutrition, crop diversification, fisheries prominence, and livestock.  Indicators within DO2 can be developed with stakeholders in the next CDCS (e.g., the value of incremental sales and gross margins per unit for crops and livestock), or build capacity to track them, as the partners were not able to calculate some of these indicators.  Continue to expand on successful agro-business models, especially those that improve value chains and make them replicable (e.g., COMACO/BLA).  To achieve sustainability and long term gains, consider long-term projects for agricultural activities, as the agricultural sector is seasonal.  Engage early with local ministry representatives at national and district levels to ensure authorities are aware of donor efforts and support. Engagement is key as province and district teams are doing a good job coordinating development efforts in rural communities, but are not fully engaged by donors.  It is critical to involve traditional leaders earlier (from program inception); they return the favor by helping the project to succeed. This promotes local ownership. At the moment they are involved only after the project is set up and they are introduced to the activities.  Move beyond trainings, and include more activities such as asset building (e.g., FTF). Development Objective 3: Education  The success recorded in education should be sustained and the partnerships maintained; however, USAID needs to come up with one package that encompasses the entire wellbeing of a child by tackling all the life threatening risks in a child’s life. Alternatively, the focus should be on “early childhood development,” which affects most facets of development. For example, the Mission can focus on stunting, immunization, education and water sanitation as one package. The evaluation team acknowledges that the Mission has started planning and moving towards a comprehensive approach to looking at the wellbeing of the child; the Mission should continue on this path that encompasses the entire development of a child.  The Mission can should broaden the scope and focus for education and include tertiary education or practical skills for the youth who do not necessarily require basic education. This will help all age groups, and will in turn foster human capital development.  There is a need for transparency. The USAID working relationship with the GRZ has been very cordial and successful. The government shares its budget with the Mission, but the Mission does not reciprocate. It would help much more if USAID could be more open with the government by sharing with them the budgets for the projects so that they do not raise expectations and suspicions (as indicated by interviewees within the government). Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 45  The success of the reading gains, though slow, cannot be over emphasized. There is a need to consider all the government priorities as stipulated in the revised sixth national development plan (R-SNDP). In this case, the numeracy under the education IR that was removed should be reconsidered in the new CDCS, as reading achievement would not contribute to improvements of human development in isolation.  The Mission needs to assist the stakeholders understand what it defines as quality education as the stakeholders are of the view that providing materials like books and desks is not enough for quality education. There is a need to go into the class and see what is being done with the books (e.g., are the books being read?). Furthermore, some activities are too general; the standards officers need to refocus and look at standards set by the government.  The Mission should invest in curricula for the teachers in the colleges and universities that produce the teachers, in order to balance the mismatch between the teachers and the pupils. Development Objective 3: Health  Expand the Government to Government (G2G) assistance to the GRZ. USAID should play a supportive role, and start funding government projects directly rather than through the implementing partners. “In some cases the Mission should strengthen systems and give the funds to government through the Provincial Medical Office.” Alternatively, the Mission should share the contents of MOUs they sign with the government with the IPs and vice versa. MOUs signed by the Mission and the government have worked well and continue to work. The Mission could go further by bringing to the table the IPs when signing MOUs so that the IPs are well informed about the contents on the MOUs and know the expectations from the GRZ.  There were mixed feelings by the IPs as to the best way of partnering with the Mission. Some partners were for the cooperative agreements and others were for contracts. There is a need for the Mission to conduct orientation workshops to explain why a contract or an agreement is used for a mechanism.  Programs like the TB programs worked well and saved many lives. The Mission needs to ensure that there are no gaps in programs; when a program comes to an end there has to be another one starting or following on immediately. It would be even better if a program starts before the other one ends. The Mission should avoid short-term extensions, which could be more expensive in the long run than if a project was awarded on a long-term basis. This will not only help the Mission cut costs, but will also help the IPs to improve their planning. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 46 REFERENCES 1. Chapoto A, Using Data to Inform Programs, Indaba Agricultural Policy Research Institute (IAPRI), Lusaka, Zambia. 2. Better Life Alliance Project/COMACO Final Results Report to USAID, 2016. 3. Central Statistical Office (CSO), Ministry of Health (MOH), Tropical Diseases Research Center (TDRC) University of Zambia, 2015. 4. Communications Support for Health (CSH) Project, Enhancing the Capacity to Influence Behaviors Final Report, 2014. 5. Feed the Future FEEDBACK. 2016. Feed the Future Zambia 2015 Zone of Influence Interim Assessment Report. Rockville, MD: Westat. 6. Feed the Future, Situational and Livelihood Analysis, 2012. 7. Fitzpatrick, J., Sanders, J., & Worthen, B. (2011). Program evaluation: Alternative approaches and practical guidelines (4th Ed.). New York: Allyn & Bacon. Canadian Publisher: Pearson. ISBN: 978-0-205-57935-8. 8. Government of Zambia (GRZ) Roadmap for Accelerating Reduction of Maternal, Newborn and Child Mortality 2013-2016. 9. Government of Zambia (GRZ) Central Statistics Office, The Living Conditions Monitoring Survey 2015. 10. Government of Zambia (GRZ) National Malaria Indicator Survey 2012. 11. Government of Zambia (GRZ) Maternal and Child Survival Program: Zambia, JHPIEGO 2014–2015. 12. Government of Zambia (GRZ), Ministry of Education, Science, Vocational Training and Early Education, Formative Assessment of Teenage Pregnancy in Zambian Primary Schools Final Report, 2015. 13. Government of Zambia (GRZ) The Implications of Treatment Scale-Up Strategies on National Health Systems in Zambia, 2014. 14. Government of Zambia (GRZ) Ministry of Community Development Mother and Child Health. 15. (MCDMCH) Family Planning Services Costed Eight-Year Integrated FP Scale-up Plan 2013- 2020. 16. Government of Zambia (GRZ) Roadmap for Accelerating Reduction of Maternal, Newborn and Child Mortality 2013-2016. 17. Government of Zambia (GRZ), Ministry of Education, Science, Vocational Training and Early Education, HIV/AIDS Knowledge, Attitude, Behaviour and Practice (KABP) Survey in Six Provinces of Zambia: Eastern, Luapula, Muchinga, Northern, Northwestern and Western Provinces, 2015. 18. Mertens, D. & Wilson, A. (2012) Program evaluation theory and practice: A comprehensive guide. New York: Guilford Press. ISBN: 9781462503254. 19. Patton, Q. M. (1997). Utilization focused evaluation: The new century text (3rd Ed.). London: Sage Publications. 20. Program for the Advancement of Malaria Outcomes (PAMO) Work Plan, 2015. 21. PEPFAR Zambia Country Operational Plan (COP) 2016. 22. USAID/Somalia Final Performance Evaluation of the Somalia Program Support Services, 2014. 23. Time to Learn Performance Evaluation Report, 2016. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 47 24. Indaba Agricultural Policy and Research (IAPRI) Rural Agricultural Livelihood Survey Report (RALS), 2015. 25. Indaba Agricultural Policy Research Institute (IAPRI) Technical Compendium: Descriptive Agricultural Statistics and Analysis for Zambia, 2013 http://www.aec.msu.edu/fs2/zambia/index.htm. 26. Stamping Out and Preventing (Stop) Gender-Based Violence (GBV) Zambia, Mid Term Evaluation Report, 2015. 27. Stufflebeam, D. (2003). The CIPP model of evaluation. In T. Kellaghan, D. Stufflebeam & L. Wingate (Eds.), Springer international handbooks of education: International handbook of educational evaluation. Retrieved from http://www.credoreference.com.ezproxy.lib.ucalgary.ca/entry/spredev/the_cipp_model_for_e valuation. 28. Situational Analysis of Programs to Meet the HIV Prevention, Care and Treatment Needs of Persons with Disabilities in Ghana, Uganda and Zambia, 2013. 29. Tenure and Global Climate Change: Global 30. http://www.usaidlandtenure.net/project/tenure-and-global-climate-change-global. 31. USAID/Malawi, Malawi Country Development Cooperation Strategy (CDCS) Impact Evaluation Baseline Report (2015). 32. USAID Zambia CDCS Budget and Dollars to Results Document (2011–2015). 33. USAID Zambia CDCS Document (2011) (https://www.usaid.gov/sites/default/files/documents/1860/USAIDZambiaCDCS30Sept2011. pdf. 34. USAID/Zambia End of Project Evaluation USAID/Zambia Economic Growth Program, 2011. 35. USAID/Zambia Performance Plan Report (PPR) 2011–2015. 36. USAID/ Zambia Project Implementation Reviews (PIRs) for Health, Economic Development. 37. USAID/Zambia Project Appraisal Document (PAD) PEPFAR. 38. USAID/Zambia Project Appraisal Document (PAD) GCC. 39. USAID/Zambia Project Appraisal Documents (PAD) Feed the Future 2014–2019. 40. USAID/Zambia Improving the Health Status of Zambians 2014–2019. 41. USAID/Zambia, Global Health Initiative Strategy 2011-2015. 42. USAID/ZAMBIA Mission-Wide Youth Assessment, 2016. 43. USAID/Zambia, Zambia Best Action Plan, Best Practices at Scale in Home, Community and Facilities (2011‐ 2015). 44. USAID/Zambia, Zambia CDCS Gender Analysis, 2011. 45. USAID/Zambia: Partnership for Integrated Social Marketing (PRISM) Program Midterm Evaluation, 2013. 46. USAID/ZAMBIA-Led Prevention Initiative (ZPI), End of Project Performance Evaluation, 2014. 47. USAID/Zambia, Health Office Annual Portfolio Review, 2015. 48. World Health Organization (WHO) Global Tuberculosis Report, 2015. 49. Zambia Integrated Systems Strengthening Program (ZISSP), End of Project Report, 2014. 50. Zambia Integrated Systems Strengthening Program (ZISSP), Mid-Term Performance Evaluation, 2013. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) 48 51. Zambia National AIDS Council, Gender Assessment of the National HIV Epidemic and Response in Zambia, 2014. 52. Zhang, G., Zeller, N., Griffith, R., Metcalf, D., Williams, J., Shea, C. & Misulis, K. (2011). Using the context, input, process, and product evaluation model (CIPP) as a comprehensive framework to guide the planning, implementation, and assessment of service-learning programs. Journal of Higher Education and Outreach Engagement 15(4), 57–83. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 1 ANNEX 1: SCOPE OF WORK SECTION C -DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK C. l INTRODUCTION AND BACKGROUND Zambia's long-term development strategy is articulated in its own "Vision 2030: A prosperous middle-income nation by 2030." To reach this objective, the Government of the Republic of Zambia (GRZ) has put into place a series of national development plans. The current, Revised Sixth National Development Plan (R-SNDP) encompasses 2013 through 2016. The R-SNDP has three overarching objectives: infrastructure development, rural development, and human development. The United States Agency for International Development's (USAID) Zambia Country Development Cooperation Strategy (CDCS) directly aligns with Zambia's development goals while supporting U.S. foreign assistance priorities. The CDCS recognizes the crosscutting nature of Zambia's development challenges through an integrated and focused approach that targets the greatest opportunities for impact and sustainable success. USAID/Zambia's current Country Development Cooperation Strategy (CDCS) expires in July 2017. The CDCS comprises three development objectives, each with intermediate results (IRs), namely:  Development Objective One: -Enabling Governance EnvironmentImproved An effective, accountable, and transparent government is the foundation for growth and prosperity. It is the U.S. Mission's top priority in Zambia and is essential to long-lasting results in all development objectives. The objective is to foster an environment in which the Zambian government provides quality services in a transparent manner, and Zambian citizens expect high standards of government performance and hold under-performing officials accountable. An enabling governance environment reduces waste and channels public resources and energies toward productive purposes. To operationalize an enabling governance environment USAID works towards been using two intermediate results, namely: IR 1.1 Citizen Demand for Transparent Accountable Service Delivery Increased IR 1.2 Transparency and Accountability in Government Service Delivery Increased Project Description1 Lack of transparency and accountability in Zambia is a well-documented constraint to development. It negatively affects government service delivery, saps government effectiveness, and impedes progress by facilitating waste and corruption at the point of public service delivery, precisely where most Zambian citizens interact with government institutions to receive the most essential and basic services, like health care and education. 1 Source: Democracy and Governance PAD Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 2 Realization of the project goal is premised on the hypothesis that two aspects to governance, a supply side (government) and a demand side (civil society), must both be addressed simultaneously to improve the enabling environment. If citizen demand for transparent and accountable service delivery increases, the government, if able, will respond by taking appropriate steps to meet the demand. In this way, the overall governance-enabling environment improves. "Enabling Governance Environment Improved," is critical for successfully laying the governance foundation necessary for growth and prosperity and achieving sustained improvements across sectors. For example, evidence suggests that countries with good governance- with transparent and accountable public health spending￾enjoy lower child mortality than countries that lack the same level of transparency and accountability. Similarly, public spending on primary education increases education attainment more effectively in countries with good governance. More generally, public spending has virtually no impact on health and education outcomes in poorly governed countries. 2 Such evidence is particularly relevant for developing countries, such as Zambia, where the overall levels of public spending on health and education is low compared with the need. The project intends to initiate a virtuous cycle in which increased demand for transparent service delivery results in focused work by the government to improve its ability to meet the demand, ultimately improving service delivery, which in tum results in greater demand, focused work to meet the new demand, improved service delivery, and so on. In sum, when citizens organize and demand better service delivery from government and government can respond, the governance environment will improve. Zambia has a strong record of democratic and free elections and peaceful transition of power. Opportunity exists, however, to improve governance and reinforce democratic practices. By July 2015, there were a total of five USAID activities under development objective one. A brief overview of these activities under development objective one is provided in the accompanying USAID/Zambia Partners in Zambia’s Development document.  Development Objective Two:- Rural Poverty Reduced in Targeted Areas Inclusive development requires a focus on rural areas to benefit a population that has largely been excluded from Zambia's recent growth. The CDCS targets specific areas based on need, population density and probability of success, which enables USAID/Zambia to leverage limited resources to a greater effect. Targeting rural-based poverty does the most good for the greatest number of Zambians and directly supports the GRZ's own development priorities. As the overwhelming majority of rural poor are smallholder farmers, approaches to achieve DO2 focus on improving agricultural livelihoods through work towards four intermediate results, namely: IR 2.1 Smallholder Agricultural Productivity Increased IR 2.2 Markets and Trade Expanded IR 2.3 Natural Resource Management Improved 2 Rajkumar, A., Swaroop, V., August 2007, World Bank, Public spending and outcomes: Does governance matter? http://origin-www.unicef.org/socialpolicv/files/Public_spending_and_outcomes_govemance.pdf Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 3 IR 2.4 Resilience of Vulnerable Households Improved Project Description 3 : USAID/Zambia's country-tailored Global Climate Change (GCC) Initiative project-Abating Climate Change and Enabling Sustainable carbon-Sequestration (ACCESS) -links field￾based activities with national efforts to: 1) improve Zambia’s capacity to manage natural resources sustainably, in particular, to reduce deforestation and forest degradation; and 2) improve the shared benefits of natural resource management between the Government of Zambia and local communities. The Project contributes to furthering Intermediate Result 2.3, Sustainable Natural Resource Management Improved. ACCESS is expected to result in poverty reduction and improved management of natural resources, which should reduce emissions and increase carbon sequestration. The Feed the Future (FTF) project focuses predominantly in Zambia’s Eastern Province and focuses on promoting growth in the agricultural sector. The agriculture sector’s contribution to Zambia's GDP has been trending downwards, declining from 16 percent in 2001 to 7.1 percent in 2014. While the agricultural sector accounts for less than 20 percent of GDP, a majority of Zambians, in particular those residing in rural areas, depend on agriculture for income, employment and food. Zambia's FTF project adheres to the global principles of the greater-U.S. government's FTF initiative and implements a set of geographically-targeted activities focused on a limited selection of agricultural commodity value chains. Eastern province is the key focus area because of its 1) high number of smallholder farmers; 2) high poverty levels; and 3) high number of underweight children. Within the province, the project targets five districts: Chipata, Katete, Lundazi, Nyimba and Petauke. Together these districts cover 85 percent of Eastern Province, with more than 234,368 rural households and a population of 1.52 million. By July 2015, there were a total of nine USAID activities under development objective two. A brief overview of these activities is provided in the accompanying USAID/Zambia Partners in Zambia's Development document.  Development Objective Three: - Human Capital Improved Human capital is a multi-dimensional concept that merges the knowledge, skills and capabilities that people need for life and work. Human capital refers to education and health levels as they relate to economic productivity. The GRZ places considerable importance on human capital and its role as a prerequisite for Zambia's development under the R-SNDP. Human capital is a cross- cutting constraint in Zambia, and must be addressed holistically rather than as discrete interventions. Human capital requires an educated populace that is able to make sound decisions that affect the health and welfare of families, and a healthy populace that is able to participate fully in education and economic opportunities. The mission works towards achieving two intermediate results to meet this objective, namely: IR 3.1 Educational Achievement in Reading Improved 3Source Global Climate Change and Feed The Future PADs Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 4 IR 3.2 Health Status Improved. Project Description 4 : The Education Project's contribution to D03 is premised on the hypothesis that education is the foundation and engine of human development and is critically linked to broad-based economic growth and democratic governance. Zambia's education system faces daunting challenges in its ability to provide adequate and quality education services. A high pupil-teacher ratio (57:1), high pupil-textbook ratio (I0 pupils per book), and extremely poor learning outcomes in critical areas, such as reading and math are compounded by a weak education management system and the HIV/AIDS pandemic, which has resulted in an influx of HIV orphans and other vulnerable children, as well as teacher deaths. Zambia's primary education system comprises approximately 8,000 schools, of which nearly 5,000 are government owned and operated, with the remainder being private or community schools. The Ministry of General Education counts 100,813 teachers in primary and secondary schools, of which 52 percent are female. Zambia's primary school system is one of the lowest performing in the Southern Africa Development Community (SADC) region, as measured by learning achievement on standardized tests. The 2008 Grade Five National Assessment reported national mean performance of 35.3 percent in reading in English; 39.4 percent in Mathematics; 40.2 percent in Life Skills; and 39.4 percent in Zambian languages. Improving the quality of education, as measured by learner performance, is the critical challenge for Zambia's education sector. Only 20 percent of children who enter the school system manage to complete the education cycle through grade twelve. The poor quality of education has exacerbated inequities for girls and disadvantaged children in general. The Health sub-component of D03 focuses on a number of health programs namely: family planning; maternal and child health; nutrition; HIV/AIDS; tuberculosis; malaria and other cross- cutting, health-related issues. The long-term development of Zambia depends on a healthy and productive population. Without significant improvement in health, Zambia cannot achieve its goal of inclusive prosperity by 2030. The health project is premised upon the hypothesis that improved health service delivery, combined with stronger institutions that are more accountable and an increased practice of healthy community behaviors and healthy lifestyles, will improve the health status of Zambians. At the DO level, each of the above mentioned intermediate results are tracked by several indicators. Nearly 50 indicators are used to track the performance of D03, 24 indicators for D02 and approximately 10 indicators for DO1. Some activities are on course, others have closed or are closing and others have only recently commenced. By July 2015, there were twenty-one health and seven education activities under development objective three. Brief overviews of these activities are provided in the accompanying USAID/Zambia Partners in Zambia's Development document. 4Source: Education and Health PADs Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 5 C.2 PURPOSE OF PERFORMANCE EVALUATION USAID/Zambia seeks to assess the performance of its current CDCS for two primary purposes: 1) to determine where the mission is with respect to reaching its development objectives; and 2) to inform the development and design of its next CDCS by identifying what is working and what is not working. Anecdotal information collected from isolated studies and routine monitoring systems show that progress has been made in meeting the CDCS development objectives; however, this progress needs to be reviewed, validated and determined definitively. The broad objectives of the performance evaluation of the CDCS are: 1. To determine the extent to which each development objective of the CDCS is being achieved and whether each is on track to meet its intended goal; 2. To determine the extent to which the design of the CDCS is appropriate for achieving the intended goal; and 3. To inform the development of the fiscal year 2017-fiscal year 2022 CDCS. C.3 EVALUATION DESIGN AND METHODOLOGY The performance evaluation will focus on higher-level, key performance indicators of development objectives utilizing a traffic light approach where targets are assessed on whether they have been achieved (green); are on track of reaching targets or milestones (amber); or not achieved at all (red). These will be determined based on established baselines and targets identified in the current CDCS. In addition, the contractor will need to explain the status of the results against targets. This entails a critical review and analysis within the parameters defined by questions in the above section. The contractor will use the following methods but will not be limited by these. Where necessary to the extent allowable, the contractor will be expected to propose alternative modus operandi. Phase One - This phase will be achieved through desk review of relevant key resource documents. It is worthwhile to note that document review will involve a fair level of effort reviewing a large volume of information that covers the period from 2011 to the present date. All documents will be provided USAID/Zambia Program Office. Documents to be Provided to the Contractor may include, but not limited to:  USAID/Zambia CDCS;  Indicator data from the Performance plan and report for FY 2010, 2011, 2012, 2013 and 2014 (if able to clear with Washington);  Project Implementation Reviews (PIRs);  Impact and performance evaluations conducted by implementing partners;  Independent Evaluations conducted by USAID on implementingmechanisms;  Synopses of relevant project appraisal documents;  USAID's most current Partners in Development booklet;  CDCS results framework;  Mission-wide performance management plan; Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 6  Relevant USAID and third party analyses; and  Other relevant evaluative material. Since some of the documents listed above are internal to the Mission and may contain procurement and sensitive but unclassified information (SBU), the resulting contractor will be required to sign confidentiality and non-disclosure agreements prior to receiving that information. The resulting contractor will also be required to describe what systems/procedures will put in place to ensure the physical and electronic security of procurement sensitive and SBU materials during Task Order implementation. The contractor will also be responsible for ensuring that its sub-contractors comply with all requirements related to the use and proper handling of SBU and procurement sensitive information. Phase Two - In addition to the above-mentioned desk review, the contractor will conduct selected field site visits and undertake key informant interviews with activity implementing partners. Selection of the field sites will be determined in collaboration with USAID and in consultation with relevant team leads. For example, for development objective two, consultation will be made with the Economic Development team and field sites visits made in Eastern Province. Other site visits will be made at relevant government entities including Cabinet Office; Ministry of Finance; Ministry of National Planning; Ministry of Agriculture; Ministry of Health; Ministry of Community Development; and Ministry of General Education. Interviews will be carried with key informants, with information gathered used to supplement the desk review and to bridge information gaps identified. Furthermore, key informant interviews will be undertaken with relevant USAID staff to gain insights in the performance of the development objectives relevant to the research questions. Care will be taken to avert all forms of bias. Therefore selected USAID be oriented on the need to provide accurate information to ensure a credible evaluation. All interviews will be confidential and anonymous. The timeline for executing the performance evaluation will be a fragmented period of up to four months covering an estimated period of 95 days, with the exception of Zambian public holidays and weekends, encompassing desk review, field site visits, data analysis and report writing and feedback loops between USAID and the contractor. The contractor will be expected to draw a detailed schedule or evaluation plan outlining evaluation activities and milestones. The performance evaluation is expected to be fielded by February 2016, subject to the pace of required procurement processes. C.4 TASKS AND RESPONSIBILITIES The Contractor will perform the following tasks: a) Hold inception meeting with USAIS/Zambia to review and, if necessary, clarify responsibilities and expectations; b) Draft inception report detailing evaluation methodology that will be used; c) Hold consultative meetings with relevant team leads under each development objective. The contractor will meet with a total of four office chiefs: Democracy and Governance office chief (DO1); Economic Development office chief (D02); and the Health and Education office chiefs (D03); Develop key informant guides relevant for Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 7 each targeted group of stakeholders; d) Submit key informant guides to USAID/Zambia for review; e) Develop training materials that will be used for training data collectors (with the support of USAID); f) Hold debriefing meetings with USAID/Zambia; g) Provide cleaned quantitative data set for USAID to review, if quantitative data are collected; h) Submit draft evaluation report to USAID/Zambia for review; and i) Submit a final report and clean dataset where necessary. USAID/Zambia will perform the following tasks: a) Provide all relevant documents for desk review; b) Review training materials prepared by the Contractors; c) Participate in the training of interviewers and monitor fieldwork (conducting random site checks throughout the fieldwork); and d) Review and approve all deliverables from the contractor. C.5 EVALUATION QUESTIONS High Priority Evaluation Questions: High priority evaluation questions for addressing the above objectives for each development objective will include: CDCS Level: 1. To what extent is the CDCS contributing to increase the prosperity of Zambians in an inclusive manner? This question addresses the CDCS's success inclusively increasing the prosperity of all Zambians at the higher level (CDCS level) which elements of the CDCS are significantly contributing towards the achievement of the CDCS goal? This question will investigate the role(s) played by the arrangement of the development objectives as a whole and designs of each development objective. The Mission is interested in a high level cost evaluation analysis embedded in the scope of the evaluation that would show whether what we spent produced reasonable result returns in reaching our IRs/DOs, or not. For example, did we spend too much to get/not get results; are we reasonable in the cost effectiveness of our approach; etc. 2. To what extent have crosscutting issues (environmental, sustainability and gender analyses) integrated into the strategy contributed towards CDCS success? This question aims to investigate whether crosscutting issues identified during CDCS formulation have helped realize or facilitate meaningful development in Zambia across the three development objectives. Development objective one: Enabling Governance Environment Improved 3. Which elements of the project and activities designed towards achieving DO l can be strengthened or scaled up in the next CDCS? Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 8 Development objective two: Rural Poverty Reduced in Targeted Areas 4. To what extent are the project and activities designed DO 2 contributing towards reducing poverty in the targeted areas? This question measures high-level results that can be attributed to the role played by the intermediate results in realizing the outcomes. Development objective three: Human Capital Improved 5. To what extent are the project and activities designed under D03 are contributing to the improvement of human capital? How the design aspects of this development objective have contributed to improving health and education? Same as development objective two, this question measures high-level results that can be attributed to the role played by the intermediate results in realizing the outcomes. Evaluation findings will be used to determine the performance of the current CDCS, and to draw lessons, as well as to inform the design of the subsequent CDCS that the mission will develop in fiscal year 2017. In addition, the evaluation will create an opportunity for articulating the role of new and emerging issues, such as biodiversity and resilience, in the new CDCS. C.6 TEAM COMPOSITION This assignment requires one international consultant as team leader (preferably with experience in development issues in Zambia and strong familiarity with USAID/Zambia's development agenda) and up to three local consultants with experience in strategic development, social and economic development and evaluation. Gender considerations should be considered in the composition of the evaluation team. Collectively, the evaluators should have the following skill set and expertise: a. Extensive experience in conducting programmatic evaluations (essential). This experience should include review and development of strategic frameworks for large scale entities, including government, bilateral and multilateral institutions; b. Experience working in program spheres that address gender inequalities and women's empowerment; c. Experience in the field of social and economic development (essential); d. Skills in assessing value for money, in terms of both cost efficiency and cost effectiveness (essential); and e. Excellent written and verbal communication skills (essential). Evaluation Advisor, senior level (Team Leader) The senior level Evaluation Advisor who will serve as team leader must have: i) a Master's degree in social and economic development, public health or related field, a PhD is strongly preferred; and ii) at least twelve years of relevant work experience if holder of a master's degree, or eight years of relevant work experience if holder of a PhD. In lieu of a Degree, the Team Leader must have 25 years of relevant work experience. The Evaluation Advisor must Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 9 have a demonstrated record of successfully supervising other local experts and producing high-quality analytical reports. Particular expertise and experience should include:  Strong analytical skills to collect, synthesize and analyse data;  Knowledge of Zambia's health and socio-economic status;  Expertise in quantitative and qualitative methods;  Strong team management and supervisory skills;  Demonstrated success in working with a cross-cultural team;  Strong technical English writing and communication skills; and  Experience working with a range of government officials, donors, local NGOs, and academia. Local Expert(s) The following local or resident permit holders are expected for this activity: 1. Evaluation Methods Specialist 2. Evaluation/Performance Monitoring/Training Specialist 3. Performance Monitoring/Indicator Specialist All the above positions are expected to be mid-career level, and collectively, the team must have a strong background in Social and Economic Development with degrees in i) Public Health, Statistics, Demography or other relevant areas of study; and ii) the years of relevant work experience should corresponding to the academic degree as per the terms of the IQC. In lieu of academic degree, local experts must have 18 years of relevant work experience. In addition, it is essential that evaluation team have excellent written and verbal communication skills and extensive experience conducting programmatic evaluations. The local expert(s) will work under the supervision of the Evaluation Advisor, and must possess the necessary skills set to successfully conduct a mixed-methods performance evaluation. Logistics and Research Assistants A logistics person, at a qualification level below that of the mid-career level, as defined in the M&E IQC, will provide logistic support to the core team. It is also anticipated that the contractor will hire a pool of eight junior level research assistants for the field data collection. The research assistants should be recent university undergraduates, preferably with (BA/BSc) qualifications in Demography, Public Health, Statistics and/or other relevant disciplines, and should be at a qualification level below that of the mid-career level, as defined in the M&E IQC. Home Office Support The Evaluation Advisor will also receive administrative and managerial support in the execution of the performance evaluation. This function will receive oversight from the home office Senior Director. USAID/Zambia Staff The Contracting Officer will formally appoint a Task Order Contracting Officer's Representative (TOCOR) and an Alternate TOCOR in a formal letter to the contractor. The TOCOR and Alternate will participate in the evaluation to facilitate institutional learning and Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 10 their capacity building through coaching and mentoring provided by the contractor. To prevent conflict of interest, the evaluation will be externally led by an individual without fiduciary interest in USAID as per USAID evaluation policy. In addition, USAID designated staff will participate actively in the evaluation process as indicated above in section (C4) but not engage themselves substantively. Estimated Level of Effort (Detailed version highlighting roles and responsibilities to be submitted by the resulting contractor) Personnel # of Days A. Home office support Home office Director 20 Evaluation Advisor (Team Leader) 95 Administrative support 30 B. Local personnel Evaluation Methods Specialist 85 Evaluation/Performance Monitoring/Training Specialist 85 Performance Monitoring/Indicator Specialist 85 Logistics support 45 Junior level Research Assistants 35 The offeror shall propose the labor category based on the requirements stated above which should also be in line with the labor categories per the IQC. C.7 SCHEDULING AND LOGISTICS The contractor will be responsible for all logistics, including scheduling of appointments with government officials and all relevant evaluation participants. In addition, the contractor will be responsible for arranging all travel arrangements, printing, equipment, office space, communication costs, car hire, costs of fuel and any other utilities. This evaluation will predominantly comprise desk review and limited field site visits. It is expected that most of the site visits will be undertaken in Lusaka and Eastern Provinces where most of the health and economic development programs are located. USAID/Zambia's contracting officers' representative and Alternate for the award will assist in the selection of field site visits and the coordination where necessary, of appointments with government officials and USAID implementing partners. In addition, USAID/Zambia and the contractor will schedule meetings for oral presentations, debriefs and updates. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 11 Activities and Deliverable Schedule # Deliverable Timeframe for Delivery after award (estimated and not inclusive of weekends and national holidays) 1 In brief with mission 10 days 2 Team planning meeting 21 days 3 Inception report 21 days 4 Data Collectors training 31 days 5 Draft performance evaluation report 71 days 6 Team presentation to USAID 73 days 7 USAID provides written feedback 83 days 8 Final performance evaluation report and dataset 95 days Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 12 ANNEX 2: EVALUATION MATRIX Evaluation Question Sub- Question(s) Result Sought Indicator #/ Title Data Source Data Collection Mechanism Question 1: To what extent is the CDCS contributing to increase the prosperity of Zambians in an inclusive manner? Which elements of the CDCS are significantly contributing towards the achievement of the CDCS goal? 1.1: Were USAID/Zambia activities appropriate for field conditions? Were all the stakeholders and beneficiaries consulted during design? 1.1.1: How strong is the convergence of the host country, USAID planning activities and funding priorities to enable successful realization of a prosperous Zambia?” 1.1.2: How are the strategies of individual DOs and their geographic targeting and focused program approaches optimal for realizing a prosperous Zambia vis a vis Zambia’s development agenda?” 1.2: What were the costs of individual programs, what were the outcomes/targets reached? (Cost Effectiveness Analysis will be conducted at CDCS level comparing various programs costs and the Contribute to Inclusive Prosperity for Zambians by 2030 Multidimensional Poverty Index (MPI) ranking; identifies deprivations and their intensity at the household level in education, health, and standard of living. Rather than measuring standard of living by per capita GNI, the MPI measures a range of deprivations with which poor households typically contend. Life Expectancy at Birth USAID/Zambia Technical / program personnel Project M&E documents SNDP indicator data Indicator analysis Review of key documents identified by USAID and IP staff Human Development Index Democracy Index MCC scorecard MPI Data source would be budgets from AID. It is agreed that CEA can only be measured if outcomes from individual programs are identified in liaison with USAID Zambia and Washington DC office KII, Document review Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 13 outcomes). Key Evaluation Question Sub- Question(s) Result Sought Indicator #/ Title Data Source Data Collection Mechanism Question 2: To what extent have crosscutting issues integrated into the strategy contributed towards CDCS success? 2.1: Were the beneficiaries purposively included in the project (men, women, youth and children and vulnerable groups)? 2.2: To what extent were environmental factors considered in the design of the approach? What measures for sustainability of impacts were planned at the time of design and implementation of programs? # of women in parliamentary seats National Gender Parity Index Existing policy and Legal framework for effective environmental management and protections USAID personnel Project design documents Project M&E documents Document review KIIs Review of key documents identified by USAID and IP staff USAID/Zambia Technical / program staff ETOA 2010 & 2016 Youth Analysis 2016 CDCS Gender Analysis 2011 & 2016 KII, Document review Key Evaluation Questions Sub-Question Result Sought Indicator #/ Title Data Source Data Collection Mechanism Question 3: Which elements of the project and activities designed towards achieving DO1 can be strengthened or scaled up in the next CDCS? 3.1: What is the level of awareness for demanding transparency and accountability of government services by the general public and civil society? 3.2: How does the DO 1 respond to identified challenges or opportunities and to the host government’s DO 1- Enabling Governance Environment Improved 1-World Bank Voice and Accountability Indicator Outcome/annual World Bank Document Review IR 1.1- Citizen Demand for Transparent and Accountable Service Delivery Increased 1.1a-Number of USG￾supported citizens’ action groups meeting minimum standards of effectiveness (Custom) Project Reports/PMP/PP RS KII, Document Review 1.1b- Number of citizens’ actions carried out by USG-supported civil society organizations (Custom) SIR 1.1.1- 1.1.1a- Number of civil Project KII, Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 14 development plans and priorities? 3.3: What policy or legal reforms are necessary to achieve DO 1, and how does DO 1 take into account the political and economic dynamics that help determine development outcomes, including the institutional strengths and weaknesses of the state and the state’s relationship to its citizens? 3.4: Does the development hypothesis explains how this DO and its intermediate results contribute to achievement of the Goal Statement? (This question will apply to other DOs as well) 3.5: What steps were put in place to ensure sustainability of DO1 activities? (This question will apply to other DOs as well) 3.6: How can evidence from lessons learned, best practices, and evaluations inform the strengthening of DO 1 or how did Institutional Capacity of Civil Society Organizations Increased society organizations using USG assistance to improve internal organizational capacity (2.4.1-1) reports/PMP/PP RS Document Review 1.1.1b- Number of direct awards to local organizations (USAID Forward definition of “local”) (Custom) SIR 1.1.2- Capacity of Service Delivery CSOs for Advocacy Increased 1.1.2a- Number of Civil Society Organizations receiving USG assisted training in advocacy (2.2.1-1) Project reports/PMP/PP RS NDI Project reports/PMP/PP RS KII, Document Review 1.1.2b- Number of civil society organizations (CSOs) receiving USG assistance engaged in advocacy interventions. (2.4.1-9) I.R 1.2- Transparency and Accountabilit y in Government Service Delivery Increased 1.2a Percentage of operating unit program funds obligated through partner country systems (CBLD-2) Project reports/PMP/PP RS SIR 1.2.2- Improved financial control and audit function at specific ministry level 1.1.2b Number of people trained in financial standards and procedures within USG-assisted MPSAs. (Custom) Project reports/PMP/PP RS KII, Document review Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 15 they help to determine the appropriate strategies to achieve them? 3.7: How does the DO address aid effectiveness principles that have been endorsed by the U.S. Government? 3.8: How will the programs that support the DO be implemented, including consideration of use of host country systems and leveraging resources and expertise through partnerships and the Diaspora (noting that approaches may change in coming years)? Evaluation Questions Sub- Question(s) Result Sought Indicator #/ Title Data Source Data Collection Mechanism Development Objective 2: Rural Poverty Reduced (USAID spending 2011= $4.2M: 2015 $13.7M) and Environment 2011=$1.4M : 2015=$2.9M) Question 4: To what extent are the project and activities designed in DO2 contributing towards reducing poverty in the targeted areas 4.1: Has yield, income and consumption of beneficiary households increased? 4.2: Has the agricultural value chain and market improved or expanded? 4.3: Are the beneficiaries having better access to the market structures? DO 2 Rural Poverty Reduced in Targeted Areas FTF 3-5 Prevalence of households with moderate or severe hunger COMACO, ZERS, R & D, CIFOR KII, Document Review, FGDs DO 2 Rural Poverty Reduced in Targeted Areas FTF 4-1 Percent of people living on less than $1.25/day PROFIT+, COMACO, ZERS, R & D, CIFOR, CFM, ECLEDS KII, Document Review, FGDs DO 2 Rural Poverty Reduced in Targeted Areas FTF 4 Women’s Empowerment in Agriculture Index (WEAI) PROFIT+, FSRP, COMACO, Commercial Horticulture Alliance/ASNAP P, ZERS, R & D, KII, Document Review, FGDs Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 16 4.4: What are the strengths? What are the existing constraints? 4.5: What are the opportunities for scaling up? CIFOR, CFM, ECLEDS DO 2 Rural Poverty Reduced in Targeted Areas FTF 4.5-1 Per capita expenditures (as a proxy for income) of USG targeted beneficiaries FSRP, Commercial Horticulture Alliance/ASNAP P, ZERS, R & D, CIFOR, CFM, ECLEDS KII, Document Review, FGDs DO 2 Rural Poverty Reduced in Targeted Areas FTF 4.5.2-13 Number of rural households benefiting directly from USG Assistance PROFIT+, FSRP, COMACO, Commercial Horticulture Alliance/ASNAP P, ZERS, R & D, CIFOR, CFM, ECLEDS Implementing Partner’s evaluation report, Annual reports KII, Document Review, FGDs IR 2.1 Smallholder Agricultural Productivity Increased FTF 4.5-4 Gross margin per unit of land, kilogram, or animal of selected product (crops/animals selected varies by country) PROFIT+, FSRP, COMACO, Commercial Horticulture Alliance/ASNAP P, ZERS, R & D, CIFOR, CFM, ECLEDS KII, Document Review, FGDs Sub-IR 2.1.1 Improved Access to Technologies, Practices and Inputs FTF4.5.2-8 Number of new technologies or management practices in one of the following phases of development as a result of USG assistance. PROFIT+, FSRP, COMACO, Commercial Horticulture Alliance/ASNAP P, ZERS, R & D, CIFOR, CFM, ECLEDS KII, Document Review, FGDs Sub-IR 2.1.2 Increased Application of Technologies, Practices and Inputs FTF4.5.2-5 Number of farmers or others who have applied new technologies or management practices as a result of USG assistance PROFIT+, FSRP, COMACO, Commercial Horticulture Alliance/ASNAP P, ZERS, R & D, CIFOR, CFM, KII, Document Review, FGDs Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 17 ECLEDS Sub-IR 2.1.2 Increased Application of Technologies, Practices and Inputs FTF 4.5.2-7 Number of individuals who have received USG-supported short-term agricultural productivity or food security training PROFIT+, FSRP, COMACO, Commercial Horticulture Alliance/ASNAP P, ZERS, R & D, CIFOR, CFM, ECLEDS KII, Document Review, FGDs IR2.2 Markets and Trade Expanded FTF 4.5.2-23 Value of incremental sales (collected at farm-level) attributed to FTF implementation PROFIT+, FSRP, COMACO, Commercial Horticulture Alliance/ASNAP P, ZERS, R & D, CIFOR, CFM, ECLEDS KII, Document review, FGDs Sub-IR2.2.1 Increased Private Sector Investment in Agriculture FTF4.5.2-38 Value of new private sector investment in the agricultural sector or food chain leveraged by FTF implementation PROFIT+, FSRP, COMACO, Commercial Horticulture Alliance/ASNAP P, ZERS, R & D, CIFOR, CFM, ECLEDS KII, Document Review, FGDs Sub- IR2.2.3 Markets and Trade Expanded FTF Stage 1: 4.5.1-9 Stage 2: 4.5.1-10 Stage 3: 4.5.1-13 Stage 4: 4.5.1-12 Stage 5: 4.5.1-11 Number of policies/regulations/admi nistrative procedures in each of the following stages of development as a result of USG assistance in each case: Stage 1: Analyzed Stage 2: Drafted and presented for public/stakeholder consultation Stage 3: Presented for legislation/decree Stage 4: Prepared with USG assistance PROFIT+, FSRP, COMACO, Commercial Horticulture Alliance/ASNAP P, ZERS, R & D, CIFOR, CFM, ECLEDS KII, Document Review, FGDs Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 18 passed/approved Stage 5: Passed for which implementation has begun IR2.3 Natural Resource Management Improved GCC4.8-7 Quality of greenhouse gas emissions, measured in metric tons of CO2 equivalent, reduced or sequestered as a result of USG assistance. CIFOR, CFM, ECLEDS KII, Document Review, FGDs Sub-IR2.3.2 Improved Joint Management of Natural Resources, Particularly Forests GCC 4.8.1-26 Number of hectares of biological significance and/or natural resources under improved natural resource management as a result of assistance. CIFOR, CFM, ECLEDS KII, Document Review, FGDs Sub-IR2.3.2 Improved Joint Management of Natural Resources, Particularly Forests Custom 2.4.2 Number of communities participating in joint management of natural resources or pay-for￾performance programs. CIFOR, CFM, ECLEDS KII, Document Review, FGDs Sub-IR2.3.3 Strengthened Government Capacity to develop Policies and Plans with Stakeholders for REDD+ and LEDS GCC 4.8.2-28 Number of laws, policies, strategies, plans, agreements, or regulations addressing climate change officially proposed, adopted, or implemented as a result of USG assistance. CIFOR, CFM, ECLEDS KII, Document Review, FGDs Sub-IR2.3.4 Technologies for Addressing Climate Change Adopted Custom 2.4.4 Number of people adopting technologies and methodologies as a result of USG assistance. CIFOR, CFM, ECLEDS KII, Document Review, FGDs IR2.4 Resilience of Vulnerable Households Improved Custom 2.4.2 Number of communities participating in joint management of natural resources or pay-for￾performance programs. CIFOR, CFM, ECLEDS KII, Document Review, FGDs IR2.4 Resilience of Vulnerable Households GCC 4.8.2-28 Number of laws, policies, strategies, plans, agreements, or CIFOR, CFM, ECLEDS KII, Document Review, Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 19 Improved regulations addressing climate change officially proposed, adopted, or implemented as a result of USG assistance. FGDs IR2.4 Resilience of Vulnerable Households Improved FTF 3.1.94 and 3.1.9.1- 4 Prevalence of exclusive breastfeeding of children under six months of age PROFIT+, FSRP, COMACO, Commercial Horticulture Alliance/ASNAP P, ZERS, R & D ( IAPRI), CIFOR, CFM, ECLEDS KII, Document Review, FGDs Evaluation Questions Sub-Question(s) Result Sought Indicator #/ Title Data Source Data Collection Mechanism Development Objective 3: Human Capital Improved (Education) (USAID SPENDING 2011=$6.6M : 2015=$11.3M) Question 5: To what extent are the project and activities designed under DO3 are contributing to the improvement of human capital? How the design aspects of this development objective have contributed to improving health and education? 5.1: Measuring reduction in drop outs in schools in study locations, increased enrollment, and increased literacy among selected beneficiaries? DO 3 Human Capital Improved 3 -Number of students demonstrating reading gains at the primary level (Female, Male) Implementing partner (RTS, TTL) data on reading assessment. Step up Zambia, Time to Learn Read to Succeed KII, Document Review IR 3.1 Educational Achievement in Reading Improved by 2017 3.1a-% of primary school learners achieving minimum level of performance on National Assessment (Female, Male) PMP/ Implementing partner (RTS, TTL) data on reading assessment and MOE National Assessment Survey Report. MOE Step up Zambia, Time to Learn Read to Succeed KII, Document Review SIR 3.1.1 MOE Systems Strengthened 3.1.1a- # of administrators and officials successfully trained with USG support (Female, Male) PMP /Partner quarterly and annual reports Step up Zambia Time to Learn KII, Document Review 3.1.1b- # of laws, Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 20 policies, regulations or guidelines developed to improve equitable access to or quality of education services Read to Succeed School Wash Splash SIR 3.1.2 Public and Community School Performance Improved 3.1.2a - Number of teachers/educators/teachi ng assistants who successfully completed in- service training or received intensive coaching or mentoring with USG support PMP/ Partner quarterly and annual reports Time to Learn Read to Succeed School Wash Splash KII, Document Review 3.1.2b - Number of PTA s or similar school governance structures supported with USG assistance SIR 3.1.3 Equitable Access to Education Increased 3.1.3a -# of Learners enrolled in USG supported primary school or equivalent non- school settings (Female, Male) PMP/ Partner quarterly reports Step up Zambia Time to Learn Read to Succeed KII, Document Review 3.1.3b- # of people in targets areas with access to improved drinking water supply as a result of USG assistance Partner quarterly reports Implementing partner records School Wash 3.1.3c- # of people in Splash target areas with access to improved sanitation facilities as a result of USG assistance SIR 3.1.4 Impact of HIV/AIDS on Education Mitigated 3.1.4a- # of active beneficiaries served by PEPFAR OVC programs for children and families affected by HIV/AIDS Partner Quarterly, SAPR and Annual Reports KII, Document Review 3.1.4b- Percentage of individuals from priority populations who completed a standardized HIV prevention intervention, including the specified minimum components, during the Partner Quarterly, SAPR and Annual reports USAID/RTI/part ner data KII, Document Review Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 21 reporting period 3.1.4c- Proportion of students who, by the end of the second grade of primary schooling, demonstrate that they can read and understand the meaning of grade level text Partner Quarterly, SAPR and Annual reports USAID/RTI/part ner data KII, Document Review 3.1.4d- # of eligible children provided with education and/or vocational training (female, male) Partner quarterly reports Implementing partner Step up Zambia Time to Learn Read to Succeed KII, Document Review 3.1.4e- # of target population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required. Partner quarterly reports Implementing partner Step up Zambia Time to Learn Read to Succeed KII, Document Review Evaluation questions Sub questions Result Sought Indicator #/ Title Data Source Data Collection Mechanism Development Objective 3: Human Capital Improved (Health) (USAID spending 2011=$150M : 2015=$126.3M) Question 5: To what extent are the project and activities designed under DO3 are contributing to the improvement of human capital? How the design aspects of this development objective have 5.2: Have improved health systems measured through improved ante natal care and immunization coverage? 5.2.1: What is the level of improved access to HIV/AIDS care, Maternal and Child Health, Nutrition and Government to Government support systems? DO 3.2- Health Status Improved 3.2.1-National Tuberculosis Prevalence TB Prevalence Survey, Measure & Evaluation, Zambia HIV/AIDS Prevention (ZPCT II), THANZI (TB Activity) Dr. Nathan Kapata) and MOH PS KII, Document review SIR 3.2.1: - Health Service Delivery Improved 3.2.1.1-Number of people trained in maternal/newborn health through USG supported programs ZISSP reports, (Maternal and Child Health Integrated Program (M￾CHIP), LINCHPIN) KII, Document review 3.2.1.2-Number of MOH Reports KII, Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 22 contributed to improving health and education? individuals who received T&C services for HIV and received their results Facility Registers ZPI , STEPS OVC ZPCT SFH/PRISM CHAMP, COH III SHARe CSH USAID DELIVER Project, GFATM, PMTCT Operations Research, STOP GBV Survivor Support, S3 ZPRS Document review 3.2.1.3-Number of HIV+ adults and children eligible for anti￾retroviral therapy (ART) currently receiving it. DATIM/Dev Results, (ZPRS), ZPCT II (Mike Walsh) (Comprehensive HIV/AIDS Management Program, PMTCT Operations Research KII, Document review 3.2.1.5-Number of counseling visits for family planning and reproductive health in USG assisted service delivery sites IP quarterly and annual reports PPR’s and PMP KII, Document review 3.2.1.6-Number of males circumcised as part of the minimum package of male circumcision for HIV prevention services IP quarterly and annual reports PPR’s and PMP KII, Document review SIR 3.2.2: Health Systems and Accountabilit y Strengthened 3.2.2.1-Percent of ART health facilities experiencing stock out of first-line ARV drug, TDF/FTC USAID/DELIVE R project KII, Document review 3.2.2.2-Percent of facilities that did not experience a stock out of indicator commodity, Coartem (ACT), during the defined reporting period. LMU at MSL Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 23 3.2.2.3-Percent of health facilities with laboratories experiencing stock outs of an indicator CD4 reagent Supply Chain Management System Project (SCMS) 3.2.2.5-Stage of implementation of laws, policies, regulations, and guidelines for improved access to quality and utilization of health services ZISSP SHARe II SIR 3.2.3 Community Health Practices Improved 3.2.3.1-Percent of children under age five years who slept under a net the night before visit during the nationwide Malaria Indicator Survey House-to-House Survey The survey is conducted by the National Malaria Control Centre (NMCC) with oversight of the Malaria Control and Evaluation Partnership in Africa (MACEPA). (World Vision: STEPS OVC) Document review 3.2.3.2-Percent of individuals who have experienced physical or sexual violence and sought help from medical personnel, police, or social service organization Stamping Out and Preventing (STOP) Gender￾based Violence (GBV) project Staff, MEASURE DHS Project DHS (STOP GBV Survivor Support, STOP GBV Access to Justice, STOP GBC – Prevention and Advocacy) 3.2.3.3-Contraceptive Prevalence Rate DHS, PS (MOF and MOH) (Communication Support for Health (CSH), Partnership for Integrated Social Marketing in Zambia (PRISM), Central Contraceptive Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 24 Procurement (CCP), USAID DELIVER Project, GFATM) 3.2.3.4-Number of orphans and vulnerable children (OVC) reached with a minimum of one care services annually Program reporting, STEPS/OVC, Chiefs of Party) Acting Chief of Party, T2L. 3.2.3.5 Percent of men between 15-49 with 2 or more sexual partners in the last 12 months who used a condom during their last sexual intercourse DHS (Communication Support for Health (CSH), Corridors of Hope FHI 360 – Local Partner Capacity Building) 3.2.3.6- Number of clinically under nourished clients who received therapeutic or supplementary food program reporting, PEPFAR funded projects (ZAMNACS, ZISSP) will report into ZPRS as per established reporting procedures and the Economic Growth team partner like ZERS (Nascent Solutions – Weave Program, STEPS Orphans and Vulnerable Children follow￾on, Zambia Nutrition Assessment, Counseling and Support) SIR 3.2.3 Community Health Practices Improved 3.2.4-Maternal mortality ratio DHS Survey SPAA Health Sector 2, (M￾CHIP), Measure & Evaluation KII, Document review SIR 3.2.3 Community Health Practices Improved 3.2.5-Prevalence of stunted children under five years of age DHS Survey Nutrition surveillance HMIS, Measure & Evaluation Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 25 SIR 3.2.3 Community Health Practices Improved 3.2.6-Percent of children under age five years who had a positive malaria test NMCC survey ZISSP (NMCC) Survey MACEPA USAID (ZISSP). 3.2.3.8- Prevalence of exclusive breastfeeding of children under 6 months of age DHS Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 26 ANNEX 3: USAID/ZAMBIA CDCS EVALUATION COMMON ANALYSIS FRAMEWORK: CDCS LEVEL Evaluation Questions Type of Answer/ Evidence (Check one or more, as appropriate) Performance Evaluation Design/Specific Methods for Data Collection Sampling or Selection Approach (Random Sample, Success Case) Data Analysis Methods (e.g., Frequency Distribution s, Trend Analysis, Cross￾Tabulations Content Analysis, Regression) Data Source(s) (Ministry, Households or Firms, Project Records, Elected Officials Evaluation Designs (Before and after, Comparison to normal, Time Series, Panel Study, Case Study, Causal Tracing, Contribution Analysis) Specific Methods(Ex iting data series, Structured Observation, Key Informants, Mini￾Survey, Focus Groups) 1.0: To what extent is the CDCS contributing to increase the prosperity of Zambians in an inclusive manner? Which elements of the CDCS are significantly contributing towards the achievement of the CDCS goal? Yes/No POCs in GOZ ministries (Finance) IP Project Mangers working on DG All USAID/ Zambia DG staff (FS, FSN, PSCs) Representative project Project M&E Documents Project beneficiaries in Lusaka Case study, Contribution analysis Key Informants, Desk review Purposive Content analysis X Description X Comparison X Explanation 1.1: Were USAID/Zambia activities appropriate for field conditions? Were all the Yes/No POCs in GOZ ministries (Finance) IP Project Mangers Case study Key Informants, Focus Groups, Desk review Purposive Content analysis X Description X Comparison X Explanation Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 27 stakeholders and beneficiaries consulted during design? working on DG All USAID/ Zambia DG staff (FS, FSN, PSCs) Representative project Project M&E Documents 1.2: What were the costs of individual programs, what were the outcomes /targets reached? Yes/No POCs in GOZ ministries (Finance) IP Project Mangers working on DG All USAID/ Zambia DG staff (FS, FSN, PSCs) Representative project Project M&E Documents Case study, Contribution analysis Key Informants, Desk review Purposive Cost effective X Description analysis X Comparison X Explanation 2: To what extent have crosscutting issues integrated into the strategy contribute towards CDCS success? Yes/No POCs in GOZ ministries (Finance) IP Project Mangers working on DG All USAID/ Zambia DG staff (FS, FSN, PSCs) Representative project Project M&E Documents Case study Key Informants, Desk review Purposive Content analysis X Description X Comparison X Explanation 2.1: Were the beneficiaries purposively Yes/No POCs in GOZ ministries Case study Key Informants, Purposive Content analysis X Description Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 28 included in the project (men, Gender inequality index women, youth and children and vulnerable groups)? To what extent environmental factors were considered in the design of the approach? What measures for sustainability of impacts were planned at the time of design and implementation of programs? X Comparison (Finance) IP Project Mangers working on DG All USAID/ Zambia DG staff (FS, FSN, PSCs) Representative project Project M&E Documents Desk review Gender analysis X Explanation Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 29 USAID ZAMBIA CDCS Evaluation Common Analysis Framework DO 1: Enabling Governance Environment Improved Evaluation Questions Type of Answer/ Evidence (Check one or more, as appropriate) Performance Evaluation Design/Specific Methods for Data Collection Sampling or Selection Approach (Random Sample, Success Case) Data Analysis Methods (e.g., Frequency Distributions, Trend Analysis, Cross￾Tabulations, Content Analysis, Regression) Data Source(s) (Ministry, Households or Firms, Project Records, Elected Officials) Evaluation Designs (Before and after, Comparison to normal, Time Series, Panel Study, Case Study, Causal Tracing, Contribution Analysis) Specific Methods (Exiting data series, Structured Observation, Key Informants, Mini￾Survey, Focus Groups) 1.0 :Which elements of the project and activities designed towards achieving DO 1 can be strengthened or scaled up in the next CDCS? Yes/No POCs in GOZ ministries (Finance) IP Project Mangers working on DG All USAID/ Zambia DG staff (FS, FSN, PSCs) Representative project Project M&E Documents Project beneficiaries in Lusaka Case study, Contribution analysis Key Informants, Focus Groups, Desk review Purposive Content analysis X Description X Compariso n X Explanatio n 1.1: What is the level of awareness for demanding transparency and accountability Of government services by the general public and civil Yes/No POCs in GOZ ministries (Finance) IP Project Mangers working on DG All USAID/ Zambia DG staff (FS, FSN, PSCs) Representative Case study Before and after, Comparison to normal Key Informants, Focus Groups, Desk review Purposive Content analysis Comparative analysis X Description X Compariso n X Explanatio n Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 30 society? project Project M&E Documents Project beneficiaries in Lusaka Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 31 USAID ZAMBIA CDCS Evaluation Common Analysis Framework DO 2: RURAL POVERTY REDUCED IN TARGETED AREAS Evaluation Questions Type of Answer/ Evidence (Check one or more, as appropriate) Performance Evaluation Design/Specific Methods for Data Collection Sampling or Selection Approach (Random Sample, Success Case) Data Analysis Methods (e.g., Frequency Distributions, Trend Analysis, Cross￾Tabulations, Content Analysis, Regression) Data Source(s) (Ministry, Households or Firms, Project Records, Elected Officials Evaluation Designs (Before and after, Comparison to normal, Time Series, Panel Study, Case Study, Causal Tracing, Contribution Analysis) Specific Methods (Exiting data series, Structured Observation, Key Informants, Mini￾Survey, Focus Groups) 1.0: To what extent are the project and activities designed DO2 contributing towards reducing poverty in the targeted area? Yes/No POCs in GOZ ministries (Agriculture) IP Project Mangers working on DG All USAID/ Zambia DG staff (FS, FSN, PSCs) Representative project Project M&E Documents Project beneficiaries Case study, Contribution analysis Key Informants, Focus Group discussions, Desk review Purposive Content analysis X Description X Comparison X Explanation 1.1: Has yield income and consumption of beneficiary households increased? X Yes/No POCs in GOZ ministries (Agriculture) IP Project Mangers working on DG All USAID/ Zambia DG staff (FS, FSN, PSCs) Case study, Before and after comparison to normal Key Informants, Focus, Mini HH survey Groups, Desk review Purposive Content analysis, Frequency distribution X Description X Comparison X Explanation Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 32 Representative project Project M&E Documents 1.2: Has the agricultural value chain and market improved or expanded? Are the beneficiaries having better access to the market structures? What are the strengths? What are the existing constraints? What are the opportunities for scaling up? Yes/No POCs in GOZ ministries (Agriculture) IP Project Mangers working on DG All USAID/ Zambia DG staff (FS, FSN, PSCs) Representative project Project M&E Documents Case study, Before and after comparison to normal Key Informants, FGD’s, Desk review Purposive Content analysis X Description X Comparison X Explanation 1.3: Have the beneficiary households developed improved farm practices and efficient irrigation marketing and post-harvest practices that yield higher impacts while preserving their natural resources? X Yes/No POCs in GOZ ministries (Finance) IP Project Mangers working on DG All USAID/ Zambia DG staff (FS, FSN, PSCs) Representative project Project M&E Documents Case study, Before and after comparison to normal Key Informants, Focus, Mini HH survey Groups, Desk review Purposive Content analysis, X Description X Comparison X Explanation 1.4: Have the beneficiaries developed better coping and adaptive Yes/No POCs in GOZ ministries (Finance) IP Project Case study, Before and after comparison Key Informants, FGD’s Desk review Purposive Content analysis X Description X Comparison Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 33 strategies as a result of USAID’s intervention? Have their safety nets and assets increased? Have their knowledge of access to credit, saving and practices and social capital improved? X Explanation Mangers working on DG All USAID/ Zambia DG staff ( FS, FSN, PSCs) Representative project Project M&E Documents Project beneficiaries analysis Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 34 USAID ZAMBIA CDCS Evaluation Common Analysis Framework DO 3: HUMAN CAPITAL IMPROVED Evaluation Questions Type of Answer/Evidence (Check one or more, as appropriate) Performance Evaluation Design/Specific Methods for Data Collection Sampling or Selection Approach (Random Sample, Success Case) Data Analysis Methods (e.g., Frequency Distributions, Trend Analysis, Cross￾Tabulations, Content Analysis, Regression) Data Source(s) (Ministry, Households or Firms, Project Records, Elected Officials Evaluation Designs (Before and after, Comparison to normal, Time Series, Panel Study, Case Study, Causal Tracing, Contribution Analysis) Specific Methods (Exiting data series, Structured Observation, Key Informants, Mini￾Survey, Focus Groups) 1.0: To what extent are the project and activities designed under DO3 are contributing to the improvement of human capital? How the design aspects of this development objective have contributed to improving health and education? Yes/No POCs in GOZ ministries (MOE,MOH, MCDMCH) IP Project Mangers working on DG All USAID/ Zambia DG staff (FS, FSN, PSCs) Representative project Project M&E Documents Project beneficiaries Case study, Contribution analysis Key Informants, Focus Groups, Desk review Purposive Content analysis X Description X Comparison X Explanation 1.1: Measuring reduction in drop outs in schools in study locations, increased enrollment, and increased literacy among selected X Yes/No POCs in GOZ ministries (MOE) IP Project Mangers working on DG All USAID/ Zambia DG Case study Key Informants, survey Groups, Desk review Purposive Comparative analysis X Description X Comparison X Explanation Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 35 beneficiaries? staff (FS, FSN, PSCs) Representative project Project M&E Documents 1.2: Have improved health systems ante natal care and immunization coverage? What is the level of improved access to HIV/AIDS care, maternal and Child Health, Nutrition and Government to Government support systems? Yes/No POCs in GOZ ministries (MOH,MCDM CH) IP Project Mangers working on DG All USAID/ Zambia DG staff (FS, FSN, PSCs) Representative project Project M&E Documents Project beneficiaries Case study, Before and after comparison to normal Key Informants, FGD’s, Desk review Purposive Content analysis X Description X Comparison X Explanation Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 36 ANNEX 4: INDICATOR TABLES BY DOs Table 1.1: Citizen Demand for Transparent Accountable Service Delivery Increased Citizen Demand for Transparent Accountable Service Delivery Increased INDICATOR 2015 TARGET 2015 ACTUAL Number of individuals receiving voter and civic education through USG-assisted programs 225,000 1,413,000 Source: PPR 2011-2015 Table 1.2: Citizen Demand for Transparent Accountable Service Delivery Increased Citizen Demand for Transparent Accountable Service Delivery Increased INDICATOR 2015 TARGET 2015 ACTUAL 2.3.2-11 Number of Domestic Election Observers and/or Party Agents Trained with USG assistance 500 1,129 Source: PPR 2011-2015 Table 2.1: Reduced Rural Poverty Reduced Rural Poverty INDICATOR FY2011 Target FY2011 Result FY2012 Target FY2012 Result FY2013 Target FY2013 Result FY2014 Target FY2014 Result FY2015 Target FY2015 Result FTF 4.5.2-13 Number of rural households benefiting directly from USG Assistance 921 1341 - 58546 105500 133217 188247 228579 272707 314542 Source: PPR 2011-2015 Table 2.2: Agricultural Productivity Increased Sub IR: Agricultural Productivity Increased INDICATOR FY2011 Target FY2011 Result FY2012 Target FY2012 Result FY2013 Target FY2013 Result FY2014 Target FY2014 Result FY2015 Target FY2015 Result FTF4.5.2-5 Number of farmers or others who have applied new technologies or management practices as a result of USG assistance 30394 47964 60768 124390 154403 221970 281686 FTF 4.5.2-7 Number of individuals who have received USG-supported short-term agricultural productivity or food security training 57401 113664 144466 194341 198906 257046 304626 Source: PPR 2011-2015 Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 37 Table 2.3: Markets and Trade Expanded Sub IR: Markets and Trade Expanded INDICATORS FY2011 Target FY2011 Result FY2012 Target FY2012 Result FY2013 Target FY2013 Result FY2014 Target FY2014 Result FY2015 Target FY2015 Result FTF 4.5.2-23 Value of incremental sales (collected at farm-level) attributed to FTF implementation 0 0 0 57964 2345553 727835 4562610 31323575 56790304 -33403504 FTF4.5.2-38 Value of new private sector investment in the agricultural sector or food chain leveraged by FTF implementation - - 1,600,000 1528000 19900000 19900000 31300000 8743812 43300000 51570103 Source: PPR 2011-2015 Table 2.3: Natural Resources Management Sub IR: Natural Resources Management INDICATORS FY2011 Target FY2012 Result FY2012 Target FY2012 Result FY2013 Target FY2013 Result FY2014 Target FY2014 Result FY2015 Target FY2015 Result GCC 4.8.1-26 Number of hectares of biological significance and/or natural resources under improved natural resource management as a result of assistance. 0 0 11680 25680 20453 92800 671357 148150 1129517 Custom 2.4.4 Number of people adopting technologies and methodologies as a result of USG assistance 0 0 0 30394 47964 60768 1145066 60768 221970 Source: PPR 2011-2015 Table 2.4: Resilience of Vulnerable Households Improved Sub IR: Resilience of Vulnerable Households Improved INDICATOR FY2011 Target FY2011 Result FY2012 Target FY2012 Result FY2013 Target FY2013 Result FY2014 Target FY2014 Result FY2015 Target FY2015 Result FTF 3.1.94 and 3.1.9.1-4 Prevalence of exclusive breastfeeding of children under six months of age 43 55 23 Source: PPR 2011-2015 Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 38 Table 3.1: Education Achievement in Reading Improved IR 3.1:Education Achievement in Reading Improved INDICATOR FY2011 Target FY2011 Results FY2012 Target FY2012 Result FY2013 Target FY2013 Result FY2014 Target FY2014 Result FY2015 Target FY2015 Result Number of students demonstrating reading gains at the primary level 540,000 548,778 (102%) Proportion of students who, by the end of the second grade of primary schooling, demonstrate that they can read and understand the meaning of grade level text 10% baseline 15% 15% Source: PPR 2011-2015 Table 3.2: MOE Systems Strengthened Sub IR 3.1.1: MOE Systems Strengthened INDICATOR FY2011 Target FY2011 Results FY2012 Target FY2012 Result FY2013 Target FY2013 Result FY2014 Target FY2014 Result FY2015 Target FY2015 Result Number of administrators and officials successfully trained with USG support 2,400 5,119(213%) 2,400 4,744 (197%) 2,719 3,670 (135%) 2,354 3,185 (135%) Number of laws, policies, regulations or guidelines developed to improve equitable access to or quality of education services 43 43 (100%) 5 43 63 (147%) 33 25 (75%) 6 28 (467%) Source: PPR 2011-2015 Table 3.3: Public and Community School Performance Increase Sub IR 3.1.2: Public and Community School Performance Increase INDICATOR FY2011 Target FY2011 Results FY2012 Target FY2012 Result FY2013 Target FY2013 Result FY2014 Target FY2014 Result FY2015 Target FY2015 Result Number of teachers/educators/teaching assistants who successfully completed in- service training or received intensive coaching or mentoring with USG support 3,300 691 (20%) 935 1,470 (157%) 5,466 6,717 (123%) 4,832 5,757 (119%) 3300 3653 (111%) Number of PTA s or similar school governance structures supported with USG assistance 875 400 (47%) 520 1,296 (249%) 1,162 2,956 (254%) 2,222 1,768 (80%) 875 1,261 (144%) Source: PPR 2011-2015 Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 39 Table 3.4: Equitable Access to Education Increased Sub IR 3.1.3: Equitable Access to Education Increased INDICATOR FY2011 Target FY2011 Results FY2012 Target FY2012 Result FY2013 Target FY2013 Result FY2014 Target FY2014 Result FY2015 Target FY2015 Result Number of Learners enrolled in USG supported primary school or equivalent non- school settings 940,000 17,362 (1.8%) 25,000 27,721 (110%) 519,828 918,715 (177%) 9,300,000 1,009,906(11%)940,000 989,815 (105%) Number of people in target areas with access to improved drinking water supply as a result of USG assistance 100,000 77,388 (77%) 60,000 82,606 (138%) 160,900 62,098 (36%) 50,000 41,941 (84%) 48,000 48,973 (102%) Number of people in target areas with access to improved sanitation facilities as a result of USG assistance 60000 16000 (27%) 43,600 53,043 (122%) 105,550 110,737 (104%) 40,000 49,415 (124%) 35,500 39,609 (112%) Source: PPR 2011-2015 Table 3.5: Health Service Delivery Improved Sub-IR 3.2.1: Health Service Delivery Improved INDICATOR FY 2011 Target FY 2011 Results FY 2012 Target FY 2012 Result FY 2013 Target FY 2013 Result FY 2014 Target FY 2014 Result FY 2015 Target FY 2015 Result FY 2016 Target FY 2016 Result Number of people trained in maternal/newborn health through USG supported programs 200 58 (29%) 250 1,084 (433%) 1,554 2,899 (187%) 750 995 (127%) 250 403 (161%) 750 Number of individuals who received T&C services for HIV and received their results 1,056,12 5 1, 036, 000 1,459,8 12(141 %) 1,102,00 0 Blank 1,160,000 786,100 (67%) 1,200,0 00 851,450( 71%) 470,783 Number of HIV+ adults and children eligible for anti￾retroviral therapy (ART) currently receiving it 1,200,00 0baselin 149,365 174,000 165,520 (95%) 205,000 Blank 240,000 213,653 (89%) 270,000 260,133( 96%) 288,934 Number of counseling visits for family planning and reproductive health in USG assisted service delivery sites 2400 36,707 (1529%) 33,600 111090 (331%) 130,000 222,074 (171%) 200,000 98,462 (49%) 40,000 0 Number of males circumcised as part of the minimum package of male 15,400 baseline 63,444 Blank 43,741 Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 40 circumcision for HIV prevention services 100,000 Blank 100,000 141,417 (141%) 100,000 71,370 (71%) 22,085 Source: PPR 2011-2015 Table 3.6: Health Systems & Accountability Improved Sub-IR 3.2.2.: Health Systems & Accountability Improved INDICATOR FY2011 Target FY2011 Results FY2012 Target FY2012 Result FY2013 Target FY2013 Result FY2014 Target FY2014 Result FY2015 Target FY2015 Result Percent of ART health facilities experiencing stock out of first-line ARV drug, TDF/FTC 90% 99% 90% 90% 90% Percent of facilities that did not experience a stock out of indicator commodity, Coartem (ACT), during the defined reporting period 70% 92% Percent of health facilities with laboratories experiencing stock outs of an indicator CD4 reagent 70% 76% 80% 71% Source: PPR 2011-2015 Table 3.7: Sub-IR 3.2.3: Community Health Practices Improved Sub-IR 3.2.3: Community Health Practices Improved INDICATOR FY2011 Target FY2011 Results FY2012 Target FY2012 Result FY2013 Target FY2013 Result FY2014 Target FY2014 Result FY2015 Target FY2015 Result Percent of children under age five years who slept under a net the night before visit during the nationwide Malaria Indicator Survey 59.9% 65% 57% 58.9% Percent of individuals who have experienced physical or sexual violence and sought help from medical personnel, police, or social service organization 15% 43% 30% Contraceptive Prevalence Rate (CPR) 49% 49% 49% Number of orphans and vulnerable children (OVC) reached with a minimum of one care services annually 121,588b aselin 150,000 205,000 250,000 300,000 Percent of men between 15-49 with 2 or more sexual partners in the last 12 months who used a condom during their last sexual intercourse 49% Number of clinically under nourished clients who received therapeutic or supplementary food 17,000 65,000 (382%) OVCs Maternal mortality ratio 591 0.74% (398) Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 41 Prevalence of stunted children under five years of age 45% 40% 40% 40% Percent of children under age five years who had a positive malaria test 32.2% 39.9% 39.9% 30.7% Prevalence of exclusive breastfeeding of children under 6 months of age 43 55 73% Source: PPR 2011-2015 Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 42 ANNEX 5: QUALITATIVE TOOLS PERFORMANCE EVALUATION OF USAID/ZAMBIA CDCS (2011-2017) KEY INFORMANT INTERVIEW GUIDE FOR IMPLEMENTING PARTNERS DO1 INTERVIEWER INSTRUCTIONS Greet the key informant. Introduce yourself if you have not had previous personal contact with him or her. Thank you for agreeing to talk with us, and for your participation in this evaluation. IT Show Inc. (ITS). ITS has been contracted by the United States government through the United States Agency for International Development (USAID) to carry out a performance evaluation of its strategic plan. Observe demographic characteristics of the group. Clarify information as necessary with him or her, asking only for information that cannot be determined from observation. Do not use a checklist with the informants. PROBE for additional info and take notes, this is only a guide. We anticipate the KII will last for about an hour and appreciate any information you can provide. Your answers to the questions we will ask are completely confidential and will be coded and reported without names. Demographic Information Gender  Male  Female Age  20–34 years  35–44 years  45 years and over [INTERVIEW QUESTIONS START ON NEXT PAGE] Thank you again for taking the time to talk to me today. As indicated when this interview was arranged, we are interested in learning more and evaluating the Performance of the USAID/Zambia CDCS (2011-2017). All of your answers will be kept confidential and only used to inform planning and implementation of USAID activities. But if you are not comfortable with any of the questions, you don’t have to answer them or can choose to end the interview at any time. We thank you sincerely for your generous time and valuable thoughts. We will first focus on the broad level and later move to more specifics.  Describe your role with USAID funding. Discuss:  Specific activities (appropriateness to Zambia`s context, target areas)  Partnerships built, General Information Staff Name: Position: Date of Interview / / Start Time:______________ Interview Site/Setting: Type of Respondent  Implementing Partner Official  Zambian Government Official  USAID/Zambia official  Other (Specify)………………………………………… Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 43  Gender considerations in selection of beneficiaries,  Cross cutting issues  Sustainability plans and  Other initiatives  What are the key achievements with USAID funding?  What are the challenges you faced during implementation of USAID funded activities?  What are the best practices and key lessons that were learned in implementing these activities that can be useful for new USAID funding?  What could donors like USAID be doing better to foster development in Zambia? REQUEST FOR INDICATORS and CONTACTS (VERY IMPORTANT)  Please list indicators you are tracking on partner reports.  Please give us contacts of individuals you are working with? At field and government level. DO 1: Secondary Data Indicators (Ask relevant respondents to tick indicators they are reporting on to USAID)  1.1a-Number of USG-supported citizens’ action groups meeting minimum standards of effectiveness (Custom)  1.1b- Number of citizens’ actions carried out by USG-supported civil society organizations (Custom)  1.1.1a- Number of civil society organizations using USG assistance to improve internal organizational capacity (2.4.1-1)  1.1.1b- Number of direct awards to local organizations (USAID Forward definition of “local”) (Custom)  1.1.2a- Number of Civil Society Organizations receiving USG assisted training in advocacy (2.2.1-1)  1.1.2b- Number of civil society organizations (CSOs) receiving USG assistance engaged in advocacy interventions. (2.4.1-9)  1.2a Percentage of operating unit program funds obligated through partner country systems (CBLD-2)  1.1.2b Number of people trained in financial standards and procedures within USG-assisted MPSAs. (Custom)  Number of citizens’ actions carried out by USG-supported civil society organizations (Custom)  Number of civil society organizations using USG assistance to improve internal organizational capacity (2.4.1-1)  Number of Civil Society Organizations receiving USG assisted training in advocacy (2.2.1-1).  Number of civil society organizations (CSOs) receiving USG assistance engaged in advocacy interventions. (2.4.1-9) Thank you for your time! End Time: Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 44 PERFORMANCE EVALUATION OF USAID/ZAMBIA CDCS (2011-2017) KEY INFORMANT INTERVIEW GUIDE FOR IMPLEMENTING PARTNERS DO 1 INTERVIEWER INSTRUCTIONS Greet the key informant. Introduce yourself if you have not had previous personal contact with him or her. Thank you for agreeing to talk with us, and for your participation in this evaluation. IT Show Inc. (ITS). ITS has been contracted by the United States government through the United States Agency for International Development (USAID) to carry out a performance evaluation of its strategic plan. Observe demographic characteristics of the group. Clarify information as necessary with him or her, asking only for information that cannot be determined from observation. Do not use a checklist with the informants. PROBE for additional info and take notes, this is only a guide. We anticipate the KII will last for about an hour and appreciate any information you can provide. Your answers to the questions we will ask are completely confidential and will be coded and reported without names. Demographic Information Gender  Male  Female Age  20–34 years  35–44 years  45 years and over [INTERVIEW QUESTIONS START ON NEXT PAGE] Thank you again for taking the time to talk to me today. As indicated when this interview was arranged, we are interested in learning more and evaluating the Performance of the USAID/Zambia CDCS (2011-2017). All of your answers will be kept confidential and only used to inform planning and implementation of USAID activities. But if you are not comfortable with any of the questions, you don’t have to answer them or can choose to end the interview at any time. We thank you sincerely for your generous time and valuable thoughts. We will first focus on the broad level and later move to more specifics.  Describe your role with USAID funding. Discuss:  Specific activities (appropriateness to Zambia`s context, target areas)  Partnerships built,  Gender considerations in selection of beneficiaries,  Cross cutting issues  Sustainability plans and  Other initiatives General Information Staff Name: Position: Date of Interview / / Start Time:______________ Interview Site/Setting: Type of Respondent  Implementing Partner Official  Zambian Government Official  USAID/Zambia official  Other (Specify)……………………………… Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 45  What are the key achievements with USAID funding?  What are the challenges you faced during implementation of USAID funded activities?  What are the best practices and key lessons that were learned in implementing these activities that can be useful for new USAID funding?  What could donors like USAID be doing better to foster development in Zambia? REQUEST FOR INDICATORS and CONTACTS (VERY IMPORTANT)  Please list indicators you are tracking on partner reports.  Please give us contacts of individuals you are working with? At field and government level. DO 1: Secondary Data Indicators (Ask relevant respondents to tick indicators they are reporting on to USAID)  1.1a-Number of USG-supported citizens’ action groups meeting minimum standards of effectiveness (Custom)  1.1b- Number of citizens’ actions carried out by USG-supported civil society organizations (Custom)  1.1.1a- Number of civil society organizations using USG assistance to improve internal organizational capacity (2.4.1-1)  1.1.1b- Number of direct awards to local organizations (USAID Forward definition of “local”) (Custom)  1.1.2a- Number of Civil Society Organizations receiving USG assisted training in advocacy (2.2.1-1)  1.1.2b- Number of civil society organizations (CSOs) receiving USG assistance engaged in advocacy interventions. (2.4.1-9)  1.2a Percentage of operating unit program funds obligated through partner country systems (CBLD-2)  1.1.2b Number of people trained in financial standards and procedures within USG-assisted MPSAs. (Custom)  Number of citizens’ actions carried out by USG-supported civil society organizations (Custom)  Number of civil society organizations using USG assistance to improve internal organizational capacity (2.4.1-1)  Number of Civil Society Organizations receiving USG assisted training in advocacy (2.2.1-1).  Number of civil society organizations (CSOs) receiving USG assistance engaged in advocacy interventions. (2.4.1-9) Thank you for your time! End Time: Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 46 PERFORMANCE EVALUATION OF USAID/ZAMBIA CDCS (2011-2017) KEY INFORMANT INTERVIEW GUIDE FOR GOVERNMENT OFFICIALS DO 2 INTERVIEWER INSTRUCTIONS Greet the key informant. Introduce yourself if you have not had previous personal contact with him or her. Thank you for agreeing to talk with us, and for your participation in this evaluation. IT Show Inc. (ITS). ITS has been contracted by the United States government through the United States Agency for International Development (USAID) to carry out a performance evaluation of its strategic plan. Observe demographic characteristics of the group. Clarify information as necessary with him or her, asking only for information that cannot be determined from observation. Do not use a checklist with the informants. PROBE for additional info and take notes, this is only a guide. We anticipate the KII will last for about an hour and appreciate any information you can provide. Your answers to the questions we will ask are completely confidential and will be coded and reported without names. Demographic Information Gender  Male  Female Age  20–34 years  35–44 years  45 years and over [INTERVIEW QUESTIONS START ON NEXT PAGE] Thank you again for taking the time to talk to me today. As indicated when this interview was arranged, we are interested in learning more and evaluating the Performance of the USAID/Zambia CDCS (2011-2017). All of your answers will be kept confidential and only used to inform planning and implementation of USAID activities. But if you are not comfortable with any of the questions, you don’t have to answer them or can choose to end the interview at any time. We thank you sincerely for your generous time and valuable thoughts.  Describe your role with USAID funding. Discuss:  Specific activities (appropriateness to Zambia`s context, target areas)  Partnerships built,  Gender considerations in selection of beneficiaries,  Cross cutting issues  Sustainability plans and General Information Staff Name: Position: Date of Interview / / Start Time:______________ Interview Site/Setting: Type of Respondent  Implementing Partner Official  Zambian Government Official  USAID/Zambia official  Other (Specify)………………………………………… Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 47  Other initiatives  What are the key achievements with USAID funding?  What are the challenges you faced during implementation of USAID funded activities?  What are the best practices and key lessons that were learned in implementing these activities that can be useful for the new USAID funding?  What could donors like USAID be doing better to foster development in Zambia? REQUEST FOR INDICATORS AND CONTACTS (Very important)  Kindly share with us the indicators you are tracking on your reports. (SEE LIST BELOW)  Kindly give us contacts of your stakeholders and partners (specifically, government, other NGOs, Field implementers and end users)? DO 2: Secondary Data Indicators (Request these from relevant respondents)  FTF 3-5 Prevalence of households with moderate or severe hunger  FTF 4-1 Percent of people living on less than $1.25/day  FTF 4 Women’s Empowerment in Agriculture Index (WEAI)  FTF 4.5-1 Per capita expenditures (as a proxy for income) of USG targeted beneficiaries  FTF 4.5.2-13 Number of rural households benefiting directly from USG Assistance  FTF 4.5-4 Gross margin per unit of land, kilogram, or animal of selected product (crops/animals selected varies by country)  FTF4.5.2-8 Number of new technologies or management practices in one of the following phases of development as a result of USG assistance.  FTF4.5.2-5 Number of farmers or others who have applied new technologies or management practices as a result of USG assistance  FTF 4.5.2-7 Number of individuals who have received USG-supported short-term agricultural productivity or food security training.  FTF 4.5.2-23 Value of incremental sales (collected at farm-level) attributed to FTF implementation.  FTF4.5.2-38 Value of new private sector investment in the agricultural sector or food chain leveraged by FTF implementation.  FTF  Stage 1: 4.5.1-9  Stage 2: 4.5.1-10  Stage 3: 4.5.1-13  Stage 4: 4.5.1-12  Stage 5: 4.5.1-11  Number of policies/regulations/administrative procedures in each of the following stages of development as a result of USG assistance in each case:  Stage 1: Analyzed  Stage 2: Drafted and presented for public/stakeholder consultation  Stage 3: Presented for legislation/decree  Stage 4: Prepared with USG assistance passed/approved  Stage 5: Passed for which implementation has begun Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 48  GCC4.8-7 Quality of greenhouse gas emissions, measured in metric tons of CO2 equivalent, reduced or sequestered as a result of USG assistance.  GCC 4.8.1-26 Number of hectares of biological significance and/or natural resources under improved natural resource management as a result of assistance.  Custom 2.4.2 Number of communities participating in joint management of natural resources or pay-for￾performance programs.  GCC 4.8.2-28 Number of laws, policies, strategies, plans, agreements, or regulations addressing climate change officially proposed, adopted, or implemented as a result of USG assistance.  Custom 2.4.4 Number of people adopting technologies and methodologies as a result of USG assistance.  Custom 2.4.2 Number of communities participating in joint management of natural resources or pay-for￾performance programs.  GCC 4.8.2-28 Number of laws, policies, strategies, plans, agreements, or regulations addressing climate change officially proposed, adopted, or implemented as a result of USG assistance.  FTF 3.1.94 and 3.1.9.1-4 Prevalence of exclusive breastfeeding of children under six months of age Thank you for your time! End Time: Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 49 PERFORMANCE EVALUATION OF USAID/ZAMBIA CDCS (2011-2017) KEY INFORMANT INTERVIEW GUIDE FOR IMPLEMENTING PARTNERS DO 2 INTERVIEWER INSTRUCTIONS Greet the key informant. Introduce yourself if you have not had previous personal contact with him or her. Thank you for agreeing to talk with us, and for your participation in this evaluation. IT Show Inc. (ITS). ITS has been contracted by the United States government through the United States Agency for International Development (USAID) to carry out a performance evaluation of its strategic plan. Observe demographic characteristics of the group. Clarify information as necessary with him or her, asking only for information that cannot be determined from observation. Do not use a checklist with the informants. PROBE for additional info and take notes, this is only a guide. We anticipate the KII will last for about an hour and appreciate any information you can provide. Your answers to the questions we will ask are completely confidential and will be coded and reported without names. Demographic Information Gender  Male  Female Age  20–34 years  35–44 years  45 years and over [INTERVIEW QUESTIONS START ON NEXT PAGE] Thank you again for taking the time to talk to me today. As indicated when this interview was arranged, we are interested in learning more and evaluating the Performance of the USAID/Zambia CDCS (2011-2017). All of your answers will be kept confidential and only used to inform planning and implementation of USAID activities. But if you are not comfortable with any of the questions, you don’t have to answer them or can choose to end the interview at any time. We thank you sincerely for your generous time and valuable thoughts.  Describe your role with USAID funding. Discuss:  Specific activities (appropriateness to Zambia`s context, target areas)  Partnerships built,  Gender considerations in selection of beneficiaries,  Cross cutting issues  Sustainability plans and General Information Staff Name: Position: Date of Interview / / Start Time:______________ Interview Site/Setting: Type of Respondent  Implementing Partner Official  Zambian Government Official  USAID/Zambia official  Other (Specify)………………………………………… Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 50  Other initiatives  What are the key achievements with USAID funding?  What are the challenges you faced during implementation of USAID funded activities?  What are the best practices and key lessons that were learned in implementing these activities that can be useful for the new USAID funding?  What could donors like USAID be doing better to foster development in Zambia? REQUEST FOR INDICATORS AND CONTACTS (Very important)  Kindly share with us the indicators you are tracking on your reports. (SEE LIST BELOW)  Kindly give us contacts of your stakeholders and partners (specifically, government, other NGOs, Field implementers and end users)? DO 2: Secondary Data Indicators (Request these from relevant respondents)  FTF 3-5 Prevalence of households with moderate or severe hunger  FTF 4-1 Percent of people living on less than $1.25/day  FTF 4 Women’s Empowerment in Agriculture Index (WEAI)  FTF 4.5-1 Per capita expenditures (as a proxy for income) of USG targeted beneficiaries  FTF 4.5.2-13 Number of rural households benefiting directly from USG Assistance  FTF 4.5-4 Gross margin per unit of land, kilogram, or animal of selected product (crops/animals selected varies by country)  FTF4.5.2-8 Number of new technologies or management practices in one of the following phases of development as a result of USG assistance.  FTF4.5.2-5 Number of farmers or others who have applied new technologies or management practices as a result of USG assistance  FTF 4.5.2-7 Number of individuals who have received USG-supported short-term agricultural productivity or food security training.  FTF 4.5.2-23 Value of incremental sales (collected at farm-level) attributed to FTF implementation.  FTF4.5.2-38 Value of new private sector investment in the agricultural sector or food chain leveraged by FTF implementation.  FTF  Stage 1: 4.5.1-9  Stage 2: 4.5.1-10  Stage 3: 4.5.1-13  Stage 4: 4.5.1-12  Stage 5: 4.5.1-11  Number of policies/regulations/administrative procedures in each of the following stages of development as a result of USG assistance in each case:  Stage 1: Analyzed  Stage 2: Drafted and presented for public/stakeholder consultation  Stage 3: Presented for legislation/decree  Stage 4: Prepared with USG assistance passed/approved  Stage 5: Passed for which implementation has begun  GCC4.8-7 Quality of greenhouse gas emissions, measured in metric tons of CO2 equivalent, reduced or sequestered as a result of USG assistance. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 51  GCC 4.8.1-26 Number of hectares of biological significance and/or natural resources under improved natural resource management as a result of assistance.  Custom 2.4.2 Number of communities participating in joint management of natural resources or pay-for￾performance programs.  GCC 4.8.2-28 Number of laws, policies, strategies, plans, agreements, or regulations addressing climate change officially proposed, adopted, or implemented as a result of USG assistance.  Custom 2.4.4 Number of people adopting technologies and methodologies as a result of USG assistance.  Custom 2.4.2 Number of communities participating in joint management of natural resources or pay-for￾performance programs.  GCC 4.8.2-28 Number of laws, policies, strategies, plans, agreements, or regulations addressing climate change officially proposed, adopted, or implemented as a result of USG assistance.  FTF 3.1.94 and 3.1.9.1-4 Prevalence of exclusive breastfeeding of children under six months of age Thank you for your time! End Time: Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 52 PERFORMANCE EVALUATION OF USAID/ZAMBIA CDCS (2011-2017) KEY INFORMANT INTERVIEW GUIDE FOR GOVERNMENT OFFICIALS DO 3 INTERVIEWER INSTRUCTIONS Greet the key informant. Introduce yourself if you have not had previous personal contact with him or her. Thank you for agreeing to talk with us, and for your participation in this evaluation. IT Show Inc. (ITS). ITS has been contracted by the United States government through the United States Agency for International Development (USAID) to carry out a performance evaluation of its strategic plan. Observe demographic characteristics of the group. Clarify information as necessary with him or her, asking only for information that cannot be determined from observation. Do not use a checklist with the informants. PROBE for additional info and take notes, this is only a guide. We anticipate the KII will last for about an hour and appreciate any information you can provide. Your answers to the questions we will ask are completely confidential and will be coded and reported without names. Demographic Information Gender  Male  Female Age  20–34 years  35–44 years  45 years and over [INTERVIEW QUESTIONS START ON NEXT PAGE] Thank you again for taking the time to talk to me today. As indicated when this interview was arranged, we are interested in learning more and evaluating the Performance of the USAID/Zambia CDCS (2011-2017). All of your answers will be kept confidential and only used to inform planning and implementation of USAID activities. But if you are not comfortable with any of the questions, you don’t have to answer them or can choose to end the interview at any time. We thank you sincerely for your generous time and valuable thoughts.  Describe your role with USAID funding. Discuss:  Specific activities (appropriateness to Zambia`s context, target areas)  Partnerships built,  Gender considerations in selection of beneficiaries,  Cross cutting issues  Sustainability plans and General Information Staff Name: Position: Date of Interview / / Start Time: ______________ Interview Site/Setting: Type of Respondent  Implementing Partner Official  Zambian Government Official  USAID/Zambia official  Other (Specify)………………………………………… Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 53  Other initiatives  What are the key achievements with USAID funding?  What are the challenges you faced during implementation of USAID funded activities?  What are the best practices and key lessons that were learned in implementing these activities that can be useful for the new USAID funding?  What could donors like USAID be doing better to foster development in Zambia? REQUEST FOR INDICATORS AND CONTACTS  Kindly share with us the indicators you are tracking on your reports. (SEE LIST BELOW)  Kindly give us contacts of your stakeholders and partners (specifically, government, other NGOs, Field implementers and end users)? EDUCATION  3 -Number of students demonstrating reading gains at the primary level (Female, Male)  3.1a-% of primary school learners achieving minimum level of performance on National Assessment (Female, Male)  3.1.1a- # of administrators and officials successfully trained with USG support (Female, Male)  3.1.1b- # of laws, policies, regulations or guidelines developed to improve equitable access to or quality of education services  3.1.2a - Number of teachers/educators/teaching assistants who successfully completed in- service training or received intensive coaching or mentoring with USG support  3.1.2b - Number of PTA s or similar school governance structures supported with USG assistance  3.1.3a -# of Learners enrolled in USG supported primary school or equivalent non- school settings (Female, Male)  3.1.3b- # of people in targets areas with access to improved drinking water supply as a result of USG assistance  3.1.3c- # of people in target areas with access to improved sanitation facilities as a result of USG assistance  3.1.4a- # of active beneficiaries served by PEPFAR OVC programs for children and families affected by HIV/AIDS  3.1.4b- Percentage of individuals from priority populations who completed a standardized HIV prevention intervention, including the specified minimum components, during the reporting period  3.1.4c- Proportion of students who, by the end of the second grade of primary schooling, demonstrate that they can read and understand the meaning of grade level text  3.1.4d- # of eligible children provided with education and/or vocational training (female, male)  3.1.4e- # of target population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required. HEALTH  3.2.1-National Tuberculosis Prevalence  3.2.1.1-Number of people trained in maternal/newborn health through USG supported programs  3.2.1.2-Number of individuals who received T&C services for HIV and received their results  3.2.1.3-Number of HIV+ adults and children eligible for anti-retroviral therapy (ART) currently receiving it. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 54  3.2.1.5-Number of counseling visits for family planning and reproductive health in USG assisted service delivery sites  3.2.1.6-Number of males circumcised as part of the minimum package of male circumcision for HIV prevention services  3.2.2.1-Percent of ART health facilities experiencing stock out of first-line ARV drug, TDF/FTC  3.2.2.2-Percent of facilities that did not experience a stock out of indicator commodity, Coartem (ACT), during the defined reporting period.  3.2.2.3-Percent of health facilities with laboratories experiencing stock outs of an indicator CD4 reagent  3.2.2.5-Stage of implementation of laws, policies, regulations, and guidelines for improved access to quality and utilization of health services  3.2.3.1-Percent of children under age five years who slept under a net the night before visit during the nationwide Malaria Indicator Survey  3.2.3.2-Percent of individuals who have experienced physical or sexual violence and sought help from medical personnel, police, or social service organization  3.2.3.3-Contraceptive Prevalence Rate  3.2.3.4-Number of orphans and vulnerable children (OVC) reached with a minimum of one care services annually  3.2.3.5 Percent of men between 15-49 with 2 or more sexual partners in the last 12 months who used a condom during their last sexual intercourse  3.2.3.6- Number of clinically under nourished clients who received therapeutic or supplementary food  3.2.4-Maternal mortality ratio  3.2.5-Prevalence of stunted children under five years of age  3.2.6-Percent of children under age five years who had a positive malaria test  3.2.3.8- Prevalence of exclusive breastfeeding of children under 6 months of age Thank you for your time! End Time: Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 55 PERFORMANCE EVALUATION OF USAID/ZAMBIA CDCS (2011-2017) KEY INFORMANT INTERVIEW GUIDE FOR IMPLEMENTING PARTNERS DO 3 INTERVIEWER INSTRUCTIONS Greet the key informant. Introduce yourself if you have not had previous personal contact with him or her. Thank you for agreeing to talk with us, and for your participation in this evaluation. IT Show Inc. (ITS). ITS has been contracted by the United States government through the United States Agency for International Development (USAID) to carry out a performance evaluation of its strategic plan. Observe demographic characteristics of the group. Clarify information as necessary with him or her, asking only for information that cannot be determined from observation. Do not use a checklist with the informants. PROBE for additional info and take notes, this is only a guide. We anticipate the KII will last for about an hour and appreciate any information you can provide. Your answers to the questions we will ask are completely confidential and will be coded and reported without names. Demographic Information Gender  Male  Female Age  20–34 years  35–44 years  45 years and over [INTERVIEW QUESTIONS START ON NEXT PAGE] Thank you again for taking the time to talk to me today. As indicated when this interview was arranged, we are interested in learning more and evaluating the Performance of the USAID/Zambia CDCS (2011-2017). All of your answers will be kept confidential and only used to inform planning and implementation of USAID activities. But if you are not comfortable with any of the questions, you don’t have to answer them or can choose to end the interview at any time. We thank you sincerely for your generous time and valuable thoughts.  Describe your role with USAID funding. Discuss:  Specific activities (appropriateness to Zambia`s context, target areas)  Partnerships built,  Gender considerations in selection of beneficiaries,  Cross cutting issues  Sustainability plans and General Information Staff Name: Position: Date of Interview / / Start Time:______________ Interview Site/Setting: Type of Respondent  Implementing Partner Official  Zambian Government Official  USAID/Zambia official  Other (Specify)………………………………………… Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 56  Other initiatives  What are the key achievements with USAID funding?  What are the challenges you faced during implementation of USAID funded activities?  What are the best practices and key lessons that were learned in implementing these activities that can be useful for the new USAID funding?  What could donors like USAID be doing better to foster development in Zambia? REQUEST FOR INDICATORS AND CONTACTS  Kindly share with us the indicators you are tracking on your reports. (SEE LIST BELOW)  Kindly give us contacts of your stakeholders and partners (specifically, government, other NGOs, Field implementers and end users)? EDUCATION  3 -Number of students demonstrating reading gains at the primary level (Female, Male)  3.1a-% of primary school learners achieving minimum level of performance on National Assessment (Female, Male)  3.1.1a- # of administrators and officials successfully trained with USG support (Female, Male)  3.1.1b- # of laws, policies, regulations or guidelines developed to improve equitable access to or quality of education services  3.1.2a - Number of teachers/educators/teaching assistants who successfully completed in- service training or received intensive coaching or mentoring with USG support  3.1.2b - Number of PTA s or similar school governance structures supported with USG assistance  3.1.3a -# of Learners enrolled in USG supported primary school or equivalent non- school settings (Female, Male)  3.1.3b- # of people in targets areas with access to improved drinking water supply as a result of USG assistance  3.1.3c- # of people in target areas with access to improved sanitation facilities as a result of USG assistance  3.1.4a- # of active beneficiaries served by PEPFAR OVC programs for children and families affected by HIV/AIDS  3.1.4b- Percentage of individuals from priority populations who completed a standardized HIV prevention intervention, including the specified minimum components, during the reporting period  3.1.4c- Proportion of students who, by the end of the second grade of primary schooling, demonstrate that they can read and understand the meaning of grade level text  3.1.4d- # of eligible children provided with education and/or vocational training (female, male)  3.1.4e- # of target population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required. HEALTH  3.2.1-National Tuberculosis Prevalence  3.2.1.1-Number of people trained in maternal/newborn health through USG supported programs  3.2.1.2-Number of individuals who received T&C services for HIV and received their results  3.2.1.3-Number of HIV+ adults and children eligible for anti-retroviral therapy (ART) currently receiving it.  3.2.1.5-Number of counseling visits for family planning and reproductive health in USG assisted service delivery sites Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 57  3.2.1.6-Number of males circumcised as part of the minimum package of male circumcision for HIV prevention services  3.2.2.1-Percent of ART health facilities experiencing stock out of first-line ARV drug, TDF/FTC  3.2.2.2-Percent of facilities that did not experience a stock out of indicator commodity, Coartem (ACT), during the defined reporting period.  3.2.2.3-Percent of health facilities with laboratories experiencing stock outs of an indicator CD4 reagent  3.2.2.5-Stage of implementation of laws, policies, regulations, and guidelines for improved access to quality and utilization of health services  3.2.3.1-Percent of children under age five years who slept under a net the night before visit during the nationwide Malaria Indicator Survey  3.2.3.2-Percent of individuals who have experienced physical or sexual violence and sought help from medical personnel, police, or social service organization  3.2.3.3-Contraceptive Prevalence Rate  3.2.3.4-Number of orphans and vulnerable children (OVC) reached with a minimum of one care services annually  3.2.3.5 Percent of men between 15-49 with 2 or more sexual partners in the last 12 months who used a condom during their last sexual intercourse  3.2.3.6- Number of clinically under nourished clients who received therapeutic or supplementary food  3.2.4-Maternal mortality ratio  3.2.5-Prevalence of stunted children under five years of age  3.2.6-Percent of children under age five years who had a positive malaria test  3.2.3.8- Prevalence of exclusive breastfeeding of children under 6 months of age Thank you for your time! End Time: Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 58 PERFORMANCE EVALUATION OF USAID/ZAMBIA CDCS (2011-2017) KEY INFORMANT INTERVIEW SUMMARY SHEET USAID STAFF INTERVIEWER INSTRUCTIONS Greet the key informant. Introduce yourself if you have not had previous personal contact with him or her. Thank you for agreeing to talk with us, and for your participation in this evaluation. IT Show Inc. (ITS). ITS has been contracted by the United States government through the United States Agency for International Development (USAID) to carry out a performance evaluation of its strategic plan. Observe demographic characteristics of the group. Clarify information as necessary with him or her, asking only for information that cannot be determined from observation. Do not use a checklist with the informants. PROBE for additional info and take notes, this is only a guide. We anticipate the KII will last for about an hour and appreciate any information you can provide. Your answers to the questions we will ask are completely confidential and will be coded and reported without names. Demographic Information Gender  Male  Female Age  20–34 years  35–44 years  45 years and over [INTERVIEW QUESTIONS START ON NEXT PAGE] Thank you again for taking the time to talk to me today. As indicated when this interview was arranged, we are Thank you again for taking the time to talk to me today. As indicated when this interview was arranged, we are interested in learning more and evaluating the Performance of the USAID/Zambia CDCS (2011-2017). All of your answers will be kept confidential and only used to inform planning and implementation of CDCS activities. But if you are not comfortable with any of the questions, you don’t have to answer them or can choose to end the interview at any time. We thank you sincerely for your generous time and valuable thoughts. We will first focus on the broad level and later move to more specifics. SECTION 1: CDCS Level 1. Describe your position/role? 2. Describe the CDCS designing process and the involvement of your office/department in the design process? General Information Staff Name: Position: Date of Interview / / Start Time:______________ Interview Site/Setting: Type of Respondent  Implementing Partner Official  Zambian Government Official  USAID/Zambia official  Other (Specify)………………………………………… Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 59 3. Has CDCS described the Zambian development context, does it highlight the most important development challenges and opportunities that exist? Does CDCS discuss overarching US Foreign policy and national security considerations in the country?  Yes / No/ Not sure  Thoughts/Observations with examples/evidence a) b) c) 4. Is the CDCS based upon a sound development hypothesis that describes the theory of change, logics, and causal relationships and principal development results? Is hypothesis clearly articulated? Is there an identified theory of change that underlies the development hypothesis?  Yes/No/Not sure  Thoughts/Observation with examples/evidence  a) b) c) 5. Does the Results Framework demonstrate mission is progressing towards CDCS goal? Do the Indicators demonstrate that there is movement towards achievement of the goals? Is there progress towards sustainability? Are roles of partners and USAID helping achieve the goals? Does CDCS demonstrate progress towards its goals, are they measurable and achievable?  Yes/No/Not sure  Thoughts/Observation with examples/evidence a) b) c) Section II: General Development Objectives Level (Please ask only high level KIs, skip section if not high level) 7. To what extent is the CDCS contributing to increase the prosperity of Zambians in an inclusive manner? Which elements of the CDCS are significantly contributing towards the achievement of the DO1/DO2/DO3 (choose appropriate)?  Thoughts/Observation with examples/evidence 8. Were USAID/Zambia activities appropriate for field conditions? Were all the stakeholders and beneficiaries consulted during design?  Thoughts/Observation with examples/evidence 9. What were the costs of individual programs, what were the outcomes/targets reached? (Cost Effectiveness Analysis will be conducted at CDCS level comparing various programs costs and the outcomes).  Thoughts/Observation with examples/evidence 10. To what extent have crosscutting issues integrated into the strategy contributed towards CDCS success?  Thoughts/Observation with examples/evidence Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 60 11. Were the beneficiaries purposively included in the project (men, women, youth and children and vulnerable groups)? To what extent were environmental factors considered in the design of the approach? What measures for sustainability of impacts were planned at the time of design and implementation of programs?  Thoughts/Observation with examples/evidence Section III: DO 1 Enabling Governance Improved ((Please ask only DO1 KIs, skip section if not related to DO1) 12. What is the level of awareness for demanding transparency and accountability of government services by the general public and civil society? 13. How does DO 1 respond to identified challenges or opportunities and to the host government’s development plans and priorities? 14. What policy or legal reforms are necessary to achieve DO 1, and how does DO 1 take into account the political and economic dynamics that help determine development outcomes, including the institutional strengths and weaknesses of the state and the state’s relationship to its citizens? 15. Does the development hypothesis explain how DO 1 and its intermediate results contribute to achievement of the Goal Statement? 16. What steps were put in place to ensure sustainability of DO1 activities? 17. How can evidence from lessons learned, best practices, and evaluations inform the strengthening of DO 1 or how did they help to determine the appropriate strategies to achieve them? 18. How does DO 1 address aid effectiveness principles that have been endorsed by the U.S. Government? 19. Describe ways in which DO1 projects and activities have contributed to improved Transparency and Accountability? What was the percentage allocation of budget for DO1 in CDCS?  Thoughts/Observation with examples/evidence 20. Did other DOs affect your DO in any way? (Positive/Negative) 21. Could you name the targeted projects and areas directly working towards DO1? Would you say the design aspects of this development objective have contributed to improving transparency and accountability?  Yes/No/Not sure  Thoughts/Observation with examples/evidence 22. Would you say project interventions have increased the level of awareness for demanding transparency and accountability of government services by the general public and civil society?  Yes/No/Not sure  Thoughts/Observation with examples/evidence 23. What are some other things that USAID or implementing partner think should be included for GRZ to build its capacity in transparency and Accountability? To what extent do Government institutions pass USAID financial risk assessment?  Thoughts/Observation with examples/evidence DO1: Probing questions How would you rate Good governance in the public sector management and institutions? In what other ways has transparency and Accountability improved with the current DO1? Would you assist with any polices that have been developed and implemented to compare baseline and current Governance environment? Would you assist with any documents we can look at compare baselines and current figures? Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 61 DO 1: Secondary Data Indicators (Request these from relevant respondents)  Number of USG-supported citizens’ action groups meeting minimum standards of effectiveness (Custom)  Number of citizens’ actions carried out by USG-supported civil society organizations (Custom)  Number of civil society organizations using USG assistance to improve internal organizational capacity (2.4.1-1)  Number of direct awards to local organizations (USAID Forward definition of “local”) (Custom)  Number of Civil Society Organizations receiving USG assisted training in advocacy (2.2.1-1)  Number of civil society organizations (CSOs) receiving USG assistance engaged in advocacy interventions. (2.4.1-9)  Percentage of operating unit program funds obligated through partner country systems (CBLD-2)  Number of people trained in financial standards and procedures within USG-assisted MPSAs. (Custom)  Number of citizens’ actions carried out by USG-supported civil society organizations (Custom)  Number of civil society organizations using USG assistance to improve internal organizational capacity (2.4.1-1)  Number of Civil Society Organizations receiving USG assisted training in advocacy (2.2.1-1).  Number of civil society organizations (CSOs) receiving USG assistance engaged in advocacy interventions. (2.4.1-9) Section III: DO 2 Rural Poverty Reduced ((Please ask only DO2 KIs, skip section if not related to DO2) 24. How does DO 2 respond to identified challenges or opportunities and to the host government’s development plans and priorities? 25. What policy or legal reforms are necessary to achieve DO 2, and how does DO 2 take into account the political and economic dynamics that help determine development outcomes, including the institutional strengths and weaknesses of the state and the state’s relationship to its citizens? 26. Does the development hypothesis explain how DO 2 and its intermediate results contribute to achievement of the Goal Statement? 27. What steps were put in place to ensure sustainability of DO2 activities? 28. How can evidence from lessons learned, best practices, and evaluations inform the strengthening of DO 2 or how did they help to determine the appropriate strategies to achieve them? 29. How does DO 2 address aid effectiveness principles that have been endorsed by the U.S. Government? 30. Describe ways in which DO2 projects and activities are contributing towards reducing poverty? What was the percentage allocation for DO2 in CDCS?  Thoughts/Observation with examples/evidence 31. Did other DOs affect your DO in any way? (Positive/Negative) 32. Could you name the targeted projects and areas directly working towards DO2? Would you say project interventions increased income and consumption of beneficiary households?  Yes/No/Not sure  Thoughts/Observation with examples/evidence DO2: Probing questions In what ways has CDCS/DO2 contributed towards beneficiaries having better access to the market structures? In what ways have the beneficiary households improved farm practices and efficient irrigation, marketing, and post-harvest practices that yield higher impacts while preserving their natural resources? In what ways have Beneficiaries safety nets and assets increased? Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 62 33. In what ways has the agricultural value chain and market improved or expanded with the current CDCS/DO2?  Thoughts/Observation with examples/evidence  34. In what ways have USAID interventions assisted beneficiaries develop better coping and adaptive strategies?  Thoughts/Observation with examples/evidence DO 2: Secondary Data Indicators (Request these from relevant respondents)  FTF 3-5 Prevalence of households with moderate or severe hunger  FTF 4-1 Percent of people living on less than $1.25/day  FTF 4 Women’s Empowerment in Agriculture Index (WEAI)  FTF 4.5-1 Per capita expenditures (as a proxy for income) of USG targeted beneficiaries  FTF 4.5.2-13 Number of rural households benefiting directly from USG Assistance  FTF 4.5-4 Gross margin per unit of land, kilogram, or animal of selected product (crops/animals selected varies by country)  FTF4.5.2-8 Number of new technologies or management practices in one of the following phases of development as a result of USG assistance.  FTF4.5.2-5 Number of farmers or others who have applied new technologies or management practices as a result of USG assistance  FTF 4.5.2-7 Number of individuals who have received USG-supported short-term agricultural productivity or food security training.  FTF 4.5.2-23 Value of incremental sales (collected at farm-level) attributed to FTF implementation.  FTF4.5.2-38 Value of new private sector investment in the agricultural sector or food chain leveraged by FTF implementation.  FTF  Stage 1: 4.5.1-9  Stage 2: 4.5.1-10  Stage 3: 4.5.1-13  Stage 4: 4.5.1-12  Stage 5: 4.5.1-11  Number of policies/regulations/administrative procedures in each of the following stages of development as a result of USG assistance in each case:  Stage 1: Analyzed  Stage 2: Drafted and presented for public/stakeholder consultation  Stage 3: Presented for legislation/decree  Stage 4: Prepared with USG assistance passed/approved  Stage 5: Passed for which implementation has begun  GCC4.8-7 Quality of greenhouse gas emissions, measured in metric tons of CO2 equivalent, reduced or sequestered as a result of USG assistance.  GCC 4.8.1-26 Number of hectares of biological significance and/or natural resources under improved natural resource management as a result of assistance.Custom 2.4.2 Number of communities participating in joint management of natural resources or pay-for-performance programs.  GCC 4.8.2-28 Number of laws, policies, strategies, plans, agreements, or regulations addressing climate change officially proposed, adopted, or implemented as a result of USG assistance.  Custom 2.4.4 Number of people adopting technologies and methodologies as a result of USG assistance.  Custom 2.4.2 Number of communities participating in joint management of natural resources or pay-for-performance programs. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 63  GCC 4.8.2-28 Number of laws, policies, strategies, plans, agreements, or regulations addressing climate change officially proposed, adopted, or implemented as a result of USG assistance.  FTF 3.1.94 and 3.1.9.1-4 Prevalence of exclusive breastfeeding of children under six months of age Section IV DO3 Human Capital Improved (Please ask only DO3 KIs, skip section if not related to DO3) 35. How does DO 3 respond to identified challenges or opportunities and to the host government’s development plans and priorities? 36. What policy or legal reforms are necessary to achieve DO 3, and how does DO 3 take into account the political and economic dynamics that help determine development outcomes, including the institutional strengths and weaknesses of the state and the state’s relationship to its citizens? 37. Does the development hypothesis explain how DO 3 and its intermediate results contribute to achievement of the Goal Statement? 38. What steps were put in place to ensure sustainability of DO3 activities? 39. How can evidence from lessons learned, best practices, and evaluations inform the strengthening of DO 3 or how did they help to determine the appropriate strategies to achieve them? 40. How does DO 3 address aid effectiveness principles that have been endorsed by the U.S. Government? 41. Describe ways in which DO3 projects and activities are contributing towards the improvement of human capital? What was the percentage allocation for DO3 in CDCS? (Education/Health)  Thoughts/Observation with examples/evidence 42. Did other DOs affect your DO in any way? (Positive/Negative) 43. Could you name the targeted projects and areas directly working towards DO3? 44. Would you say the design aspects of this development objective contributed to improving health and education among selected beneficiaries?  Yes/No/Not sure  Thoughts/Observation with examples/evidence 45. What is the level of improved access to HIV/AIDS care, maternal and Child Health, Nutrition and Government to Government support systems?  Thoughts/Observation with examples/evidence Section V: Lessons Learned (Please ask all KI respondents) 46. What are the key lessons you have learned from your involvement in this project?  Thoughts/Observation with examples/evidence DO3: PROBING QUESTIONS What were the roles of partners were they clearly defined? In what other ways has Education/Health improved with the current DO3? Would you say project interventions reduced drop outs in schools in study locations, increased enrollment, and increased literacy among selected beneficiaries? In what ways has CDCS/DO3 contributed towards beneficiaries having better access to HIV/AIDS care, maternal and child health Nutrition and Government to Government Support system? Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 64 47. Do you have any recommendations moving forward? (What could have CDCS done better, what should be repeated, what should be stopped altogether?)  Thoughts/Observation with examples/evidence DO 3: Secondary Data Indicators (Request these from relevant respondents) EDUCATION  3 -Number of students demonstrating reading gains at the primary level (Female, Male)  3.1a-% of primary school learners achieving minimum level of performance on National Assessment (Female, Male)  3.1.1a- # of administrators and officials successfully trained with USG support (Female, Male)  3.1.1b- # of laws, policies, regulations or guidelines developed to improve equitable access to or quality of education services  3.1.2a - Number of teachers/educators/teaching assistants who successfully completed in- service training or received intensive coaching or mentoring with USG support  3.1.2b - Number of PTA s or similar school governance structures supported with USG assistance  3.1.3a -# of Learners enrolled in USG supported primary school or equivalent non- school settings (Female, Male)  3.1.3b- # of people in targets areas with access to improved drinking water supply as a result of USG assistance  3.1.3c- # of people in target areas with access to improved sanitation facilities as a result of USG assistance  3.1.4a- # of active beneficiaries served by PEPFAR OVC programs for children and families affected by HIV/AIDS  3.1.4b- Percentage of individuals from priority populations who completed a standardized HIV prevention intervention, including the specified minimum components, during the reporting period  3.1.4c- Proportion of students who, by the end of the second grade of primary schooling, demonstrate that they can read and understand the meaning of grade level text  3.1.4d- # of eligible children provided with education and/or vocational training (female, male)  3.1.4e- # of target population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required. HEALTH  3.2.1-National Tuberculosis Prevalence  3.2.1.1-Number of people trained in maternal/newborn health through USG supported programs  3.2.1.2-Number of individuals who received T&C services for HIV and received their results  3.2.1.3-Number of HIV+ adults and children eligible for anti-retroviral therapy (ART) currently receiving it.  3.2.1.5-Number of counseling visits for family planning and reproductive health in USG assisted service delivery sites  3.2.1.6-Number of males circumcised as part of the minimum package of male circumcision for HIV prevention services  3.2.2.1-Percent of ART health facilities experiencing stock out of first-line ARV drug, TDF/FTC  3.2.2.2-Percent of facilities that did not experience a stock out of indicator commodity, Coartem (ACT), during the defined reporting period.  3.2.2.3-Percent of health facilities with laboratories experiencing stock outs of an indicator CD4 reagent Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 65  3.2.2.5-Stage of implementation of laws, policies, regulations, and guidelines for improved access to quality and utilization of health services  3.2.3.1-Percent of children under age five years who slept under a net the night before visit during the nationwide Malaria Indicator Survey  3.2.3.2-Percent of individuals who have experienced physical or sexual violence and sought help from medical personnel, police, or social service organization  3.2.3.3-Contraceptive Prevalence Rate  3.2.3.4-Number of orphans and vulnerable children (OVC) reached with a minimum of one care services annually  3.2.3.5 Percent of men between 15-49 with 2 or more sexual partners in the last 12 months who used a condom during their last sexual intercourse  3.2.3.6- Number of clinically under nourished clients who received therapeutic or supplementary food  3.2.4-Maternal mortality ratio  3.2.5-Prevalence of stunted children under five years of age  3.2.6-Percent of children under age five years who had a positive malaria test  3.2.3.8- Prevalence of exclusive breastfeeding of children under 6 months of age Thank you for your time! End Time: Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 66 FOCUS GROUP DISCUSSION GUIDE BENEFICIARIES* *Please note that there are several beneficiaries to consider in this evaluation due to the nature of the interventions in the field. The end users can be the farmers, women, men, youth or community as a whole, government personnel or other stakeholders, please verify before you go for interview and specify in the box below. The answers are critical in verifying the changes reported by USAID Partners, so ensure you get clear, specific and detailed answers to questions below. INTERVIEWER INSTRUCTIONS Greet the key informant. Introduce yourself if you have not had previous personal contact with him or her. Thank you for agreeing to talk with us, and for your participation in this evaluation. IT Show Inc. (ITS). ITS has been contracted by the United States government through the United States Agency for International Development (USAID) to carry out a performance evaluation of its strategic plan. Observe demographic characteristics of the group. Clarify information as necessary with him or her, asking only for information that cannot be determined from observation. Do not use a checklist with the informants. PROBE for additional info and take notes, this is only a guide. We anticipate the FGD will last for about an hour and appreciate any information you can provide. Your answers to the questions we will ask are completely confidential and will be coded and reported without names. DEVELOPMENT OBJECTIVES (This is for the interviewer to be pre ‘tick’ based on prior knowledge for categorization during analysis) Tick applicable DO  DO 1- Enabling Governance Environment Improved  DO 2- Rural Poverty Reduced in Targeted Areas  DO 3 Human Capital Improved PROJECT ACTIVITIES List the activities implemented as described by the Implementing Partner (IP) with USAID funding. (Instruction: Do not read this to participants but mention the activities to participants) GENERAL INFORMATION  Please describe yourself: Name, Age and Occupation.  Please highlight three challenges of the wellbeing of people in your community, briefly share on how they are being addressed?  How were they addressed by this project (Specify USAID Funded project or project activity)? Province: District: Interviewer: Number of Participants: Male: Female: Date: Start Time: Venue: Name of Organization/Government Body/Person/Other Group (specify): If it’s a group, (specify group composition, e.g: Male, Female, Mixed, Age range, and occupation, etc.) Type of Group:  USAID Technical level  Implementing Partner Staff  Beneficiaries (Specify): Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 67  What is the role of the government in addressing these challenges?  What is the role of other non-governmental / private sector organizations in addressing these challenges? CHANGES/IMPACTS  What was the situation before these activities were implemented? What major changes has happened in your community since the activity commenced (positive, negative, or general observable change. PROBE)  How did you personally benefit from the above mentioned activities funded by USAID? (PROBING Questions: Lessons learned, economic growth, awareness, knowledge, new skills, capacity, assets) BEST PRACTICES/CHALLENGES/LESSONS LEARNED  What worked? What did not work? in this project activity? (Specify with examples)  Probing question: Describe the challenges you faced, how was it countered/addressed? RECOMMENDATIONS FOR FUTURE PROGRAMS  Anything that needs to be changed?  What could donors like USAID/The Project implementer be doing better to foster development in Zambia?  What more can be done in your region to address (specify: probe from their answers) challenges to you/your area/sector, that can help future projects? Key observations: (Anything that was unique to this respondent/group) Thank you for your time! End Time: Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 68 ANNEX 6. USAID ZAMBIA CDCS PERFORMANCE EVALUATION RESPONDENTS Sampling: While nearly most partners are contacted, the ET will reach at least one sample from each DO (e.g., 1 COP, 1 Field Partner, 1 Government counterpart and 1 or more beneficiaries). The majority of the tools used will be KIIs, with FGDs specifically for beneficiaries and field partners. Identification: This list is TIER 1. The main contact persons were given by USAID/Zambia. The follow-on interviews will be through a ‘snowball’ method. They will form TIER 2 (Field partners) and TIER 3 (Beneficiaries) DATA COLLECTION METHOD EVALUATION TEAM LEVEL(DO/CDCS/IP) LOCATION DATE STATUS CDCS Level KII Dr. Gandhi, Victor, Mwila, Wilfred CDCS (Deputy head programs Stella Mutale) DO2, (M Damaseke) USAID/Lusaka 10/06/2016 Completed KII Dr. Gandhi, Mwila CDCS (Head of programs Debra Mosel & Chris Foley) USAID/Lusaka 07/06/2016 Completed DO1 ENABLING GOVERNANCE ENVIRONMENT APPROVED KII Dr. Gandhi, Wilfred DO1 (Democracy& Governance officer Jenny Neville’s & Edward) USAID/Lusaka 06/06/2016 Completed KII Dr. Gandhi, Wilfred DOI, IP (Fostering Accountability and Transparency in Zambia) NDI Office/Lusaka 14/06/2016 Completed KII Dr. Gandhi, Wilfred DOI, IP (Counterpart International) Contact not available KII Dr. Gandhi, Wilfred DOI, IP (National Democratic Institute COP, Tracy D Cook) NDI Office/Lusaka 14/06/2016 Completed KII Victor, Wilfred DOI, and DO2 Dr. Sinyangwe and Mr. Graig Saili USAID 28/06/2016 completed KII Wilfred D0 1 Project Manager CCMG Kabulonga/Lusaka 28/06/2015 The KII was busy with election preparations DO 2: RURAL POVERTY REDUCED IN TARGETED AREAS KII Dr. Gandhi, Mwila DO2, IP (Profit Plus COP Alex Pavlovic) 14- 15 PP Office/Lusaka 15/06/2016 Completed KII Dr. Gandhi, Mwila, Wilfred DO2, IP (IAPRI COP C Kabaghe & A Chapoto) IAPRI Office/Lusaka 16/06/2016 Completed KII Dr. Gandhi, Mwila, DO2, IP (Better Life Alliance COP Lewis BLA Office/Lusaka 16/06/2016 Completed Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 69 Dale) KII Mwila, Wilfred DO 2 IP COMACO Dr. Dale Lewis Industrial area, Lusaka 22/06/2016 Completed KII Dr. Gandhi, Mwila DO2 IP (Community￾based Forest Management Program, COP, P. Cowles ) Lusaka 2016 Completed KII Dr. Gandhi, Mwila DO2 IP (Mawa Project, COP E Baldridge ) Mawa Office/Chipata 2016 Completed KII Dr. Gandhi, Mwila DO2 IP (SAIOMA COP P. Manda/ Nyambe) SAIOMA Office/Chipata 2016 Completed KII Dr. Gandhi, Mwila DO2 IP (Feed the Future Harmonised Seed Regulations Project HSRP COP B. Kingsley) HSRP Office/Lusaka 2016 Contact left project, in process of new appointment with project team KII Wilfred, Victor DO2 IP (Tenure and Global Climate Change Project COP M Sommerville ) TGCC /Tetratech Office/Lusaka 17/10/2016 Completed KII Dr. Gandhi, Mwila DO2 IP (Frankfurt Zoological Society COP K Laurenson ) FZS Offices/Lusaka 17/06/2016 Completed KII Dr. Gandhi, Mwila DO2 IP ( Game Rangers International COP K Rademeyer) GRI Offices/Lusaka /2016 No response KII Mwila DO2 IP Paul Cowles Paul Cowles 21/06/2016 Completed KII Dr. Gandhi, Mwila DO2 IP (Anna Torres, Jeromy) USAID/Lusaka 02/06/2016 Completed KII Mwila, DO2 IP Erin Baldridge CRS /Chipata 21/07/2016 Completed KII Mwila, DO 2 IP Nyambe Manyando SAIOMA/Chipata 22/07/2016 Completed KII Mwila, DO 2 IP Alfonso Kahalawe District Agriculture Office 25/07/2016 Completed KII Mwila, DO 2 IP Noreen Miti Land Alliance 25/07/2016 Completed KII Mwila, DO 2 IP Botany M Hang’ombe Profit-Plus/Chipata 26/07/2016 Completed KII Mwila, DO 2 IP Zavier Tembo CSO-SUN-Chipata 27/06/2016 Completed KII Mwila, DO 2 IP Whiteson Daka COMACO-Chipata 28/06/2016 Completed KII Mwila, DO 2 IP Micheal Ngulube DACO/Chipata 28/06/2016 Completed KII Mwila, DO 2 IP Mr Nkhuwa FGD Facilitator/ SAIOMA-Chisitu Agric Camp 29/07/2016 Completed KII Mwila, DO 2 IP Sarah Fransica Mwanza Phiri FGD Facilitator Mawa -Caritas Program Manager 02/08/2016 Completed KII Mwila, DO 2 IP Erin Baldridge CRS /Chipata 21/07/2016 Completed Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 70 DO3 HUMAN CAPITAL IMPROVED(EDUCATION) KII Dr. Gandhi, Victor DO3 (Head Health Melanie Luick-Martins & Jessica Healy) USAID/Lusaka 03/06/2016 Completed KII Dr. Gandhi, Victor DO3 (Chief, Education, Iris Young) USAID/Lusaka 09/06/2016 Completed DO 3 Stefan M. McLetchie Time to Learn KII Dr. Gandhi, Victor DO3 IP (Education Step up COP Justin Lupele )11-12 Step up Office/Lusaka 15/06/2016 Completed KII Dr. Gandhi, Victor DO3, IP (Read to Succeed-Creative Associates COP T Zewdie) CA Office/Lusaka 20/06/2016 Completed KII Dr. Gandhi, Victor DO3 IP (Time to Learn Project-EDC)14:30 EDC Offices/Lusaka 20/06/2016 Completed KII Dr. Gandhi, Victor DO3 GRZ (Examination Council, Director C. Michael ) ECZ/Lusaka 2016 No response KII Dr. Gandhi, Victor DO3 GRZ (Ministry of General Education MoGE C Sakala ) MOE/Lusaka 17/06/2016 Completed KII Dr. Gandhi, Victor DO3 GRZ (Ministry of General Education MoGE J Chilufya) MOE/Lusaka 2016 No response KII Dr. Gandhi, Victor DO3 GRZ (Ministry of General Education MoGE J Nthele) MOE/Lusaka 2016 No response KII Dr. Gandhi, Victor DO3 IP (Water & Sanitation for the Urban Poor Country manager R Sipuma) Lusaka 2016 No response KII Wilfred DO3 Nathan Kayombo Time to Learn Ndola 02/08/2016 Completed KII Wilfred DO 3 Felistas Moono/ Provincial Coordinator Ndola 02/08/2016 Completed KII Wilfred DO 3 Mr. Chandel A D. District Resource Centre Coordinator Mufulira DC 02/08/2016 Completed KII Wilfred DO 3 Mr. Kebby Siyuya District Resource Centre Coordinator Kalulushi 29/07/2016 Completed KII Wilfred DO 3 Mr. Mwanza. District Resource Centre Coordinator Masaiti 02/07/2016 Completed KII Wilfred DO 3 Mr. Ngosa C District Resource Centre Coordinator Kitwe 28/07/2016 Completed KII Wilfred DO 3 Ms. Cleopatra Chisenga Phiri District Resource Centre Coordinator Chililabombwe 02/06/2016 Completed KII Wilfred DO 3 Ms. Liseli Maswana Ndola 29/06/2016 Completed Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 71 District Resource Centre Coordinator Dr. Tassew Zewdie COP Read to Succeed/Lusaka Tassewz@crea￾rts.com Lusaka DO3 HUMAN CAPITAL IMPROVED(HEALTH) KII Wilfred DO3, IP (COP, AIRS) Dr P Mumba AIRS/Lusaka 20/06/2016 Completed KII Dr, Gandhi, Victor DO3, IP (COP, ZPCT 2, Bridge), Michael Welsh ZNFU/Lusaka 2016 Completed KII Dr, Gandhi, Victor DO3, IP (COP, Zambia Rising, STC), Cornelia Van Zyl Save the Children/Lusaka 2016 Completed KII Dr Gandhi, Victor DO3, IP (COP, Safe Motherhood 360), Dally Menda, Golden Mwila, A Kamenga CHAZ/Lusaka 2016 Completed KII Mwila Mathew DO3, IP (COP Stop GBv), Annie Banda COP WVI/Lusaka 2016 Completed KII Mwila DO3, IP (ZAMFAM, ECR), Beatrice Simona DCOP DAPP/Lusaka 2016 Completed KII Mwila DO3, IP PATH Dr. Chimumbwa COP PATH/Lusaka 29/06/2016 Completed KII Victor DO 3 IP Cornelia Van Zyl Zambia Rising COP ZAM Rising/Lusaka 25/06/2016 completed KII Mwila, Wilfred DO 3 IP JHPIEGO Chikusela Sikazwe JHPIEGO/Lusaka 21/06/2016 completed KII Victor DO 3 Dr. Cheswa Vwalika COP Sarai Project Office, SFH 22/06/2016 Completed KII Mwila, Dr. Lingenda Godfrey District Medical Officer Chipata 02/08/2016 Completed KII Mwila DO3 IP Wendy Nicodemus COP John Snow Inc 28/06/2016 completed KII Mwila, DO3 Erin Baldridge CRS /Chipata 21/07/2016 Completed KII Wilfred DO3 Sulwe Chalwe/ Program Manager FHI360/Ndola 27/07/2016 Completed KII Wilfred DO3 Benny Njovu - DCOP ZAMFAM-ECR 28/06/2016 Completed KII Wilfred DO3 Dr. Chikafuna Banda District Medical Officer/Kitwe 29/06/2016 Completed KII Wilfred DO3 Ireen Soko- Sister￾In-Charge Buchi Clinic/Kitwe 02/08/2016 Completed KII Victor Rick Henning Ed. D. DCOP The Thrive project￾PATH/Lusaka 18/06/2016 Completed KII Victor Joffrey Zulu COP Luapula Foundation 29/06/2016 Completed KII Victor Micky Kalobwe - Program Manager Luapula Foundation 25/06/2016 Completed KII Victor Suzgo Kapanda￾Program Officer ZPCT /Mansa 21/06/2016 Completed Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 72 KII Victor Dr. Mathews Ng'ambi Medical Officer Ministry of Health/Mansa 22/06/2016 Completed KII Victor Dr. Kabuswe /Medical Officer Ministry of Health/Mansa 02/08/2016 Completed KII Victor 28/06/2016 Completed KII Victor Dr. Michel welsh, PH.D COP fhi360/Lusaka 21/07/2016 Completed KII Victor Mr. steven ngoyi -MoE Ministry of Health/Mansa 27/07/2016 Completed KII Victor Thomas tembo- DMO Ministry of Health/Mansa 28/06/2016 Completed KII Victor Bangwe Joseph -In￾Charge Ministry of Health/Mansa 29/06/2016 Completed KII Victor Chanda m. Kabwe - Clinician Ministry of Health/Mansa 25/06/2016 Completed Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 73 ANNEX 7 – TRIP REPORTS SUMMARY FROM EASTERN FIELD DATA COLLECTION Field work started on Friday and focus was on Key informants while making appointments for focus group discussions. Field team was led by Mwila Kanema and 3 Research Assistants namely Mathews Banda; Augustine Bwalya and Macdonald Lwando Team left Lusaka by road using a hired vehicle on 21st July, 2016 and returned to Lusaka on 1st August, 2016. Team lodged in Chipata for 10 days and spent a night in Katete. While in Chipata KIIs and FGDs were conducted in four districts namely Lundazi, Mfuwe, Katete and Chipata. Majority of the FGDs were in rural areas and team had to spend close to or slightly over 2 hours on gravel road. Our target was to interview all the current USAID implementing partners and conduct at least 2 FGDs with their beneficiaries in different locations within the province. Snow ball sampling recommendations were adopted with the tier running from USAID head of departments to heads of implementing organizations. The main objective of the team was to focus on DO2 IPs however we also managed to interview 2 KIIs under D03 Health. Overall we managed to reach our target by conducting twelve Key informant interviews with Land Alliance, Profit plus, Mawa, SAIOMA, COMACO, Ministry of Agriculture partners and the District Medical Officer. Eleven Focus group discussions were conducted with facilitation assistance from the provincial key informants. Team conducted 11 focus group discussions 2 were done with Land Alliance beneficiaries in Chipata rural, I with Mawa beneficiaries and the third with SAIOMA end users. In Lundazi, team had 3 FGDs with Mawa, Profit plus and COMACO beneficiaries. In Mfuwe, team had 1 KII and I FGD with Bio carbon matters beneficiaries. In Katete, 3 FGDs were conducted with beneficiaries from SAIOMA, COMACO and Profitplus. Team was also joined by the Contracting Officer Representation Mr. Kevin Chilemu who accompanied the team to almost all the focus group discussions. With COR team also paid a courtesy call to the District Administrative officer who provided team with valuable information, appreciated the efforts of the USG and highlighted on the overarching needs of the people in eastern province with access to water being a major challenge as will be seen in findings and majority of the FGDs. Schedule of Work Day 1 Thursday 21/7/2016: KIIs in Chipata Day 2 22/7/2016 Friday: KIIs in Chipata Day 5 25/7/2016 Monday: 2 FGDs in Lundazi Day 6 26/7/2016 Tuesday: 2 FGDs in Chipata & KIIs Day 7 27/7/2016 Wednesday: KII & FGDs in 1 Chipata and 1 FGD Katete rural Day 8 Thursday 28/7/2016: 3 KIIs & 1 FGDs Chipata Day 9 Friday 29/7/2016: 2 FGDs Chipata and Mfuwe KII & 1 FGD Day 10 30/7/2016 Saturday: 1 FGD – Katete Day 11 31/7/2016 Sunday: 1 FGD – Katete Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 74 Challenges/Possible Limitations Choice of focus group discussions was made by implementing partner contact and there is a high possibility of choosing only the preferred ones but to ensure we received full information from groups the IP facilitator was always asked to leave politely so that beneficiaries can be comfortable to discuss all issues in the FGD guide. Introduction IT Shows Inc. has been contracted by the United States government through the United States Agency for International Development (USAID) Zambia to carry out a performance evaluation for its Country Development Cooperation Strategy(CDCS) which has been implemented from 20011- 2015 and it is going up to 2017. Purpose of the Performance Evaluation USAID/Zambia seeks to assess the performance of its current CDCS for two primary purposes: 1) to determine where the Mission is with respect to reaching its development objectives; and, 2) to inform the development and design of its next CDCS by identifying what is working and what is not working. Therefore, the broad objectives of the performance evaluation of the CDCS are: 1. To determine the extent to which each CDCS development objective is being achieved and whether each is on track to meet its intended goal; 2. To determine the extent to which the CDCS design is appropriate for achieving the intended goal; and 3. To inform the development of the CDCS fiscal years 2017 – 2022. Activities Conducted Copperbelt province Field team was led by Wilfred Manda and 2 Research Assistants namely Felix Mwachilenga and; Mercy Nalumbwe. Team left Lusaka by road using a hired vehicle on 24th July, 2016 and returned to Lusaka on 4th August, 2016. Team lodged in Ndola and Kitwe for 11 days. The team planned to conduct 16 KIIs and 2 FGDs, however it only managed to conduct 13 KIIs and 0 FGDs. The other three KIIs were postponed beyond the teams schedule. The team spent most of the time traveling to various districts within Coppeblet and it covered 8 districts out of 8 Districts. Several provincial and District meetings that were taking place affected the data collection process in some places and some of the interviews were rescheduled or postponed to another day. The team further managed to pay courtesy calls on the District Commissioners in 7 Districts they visited. This included Provincial medical office, District health Offices, Health facilities, Provincial Education Office and District Education Offices, Schools including USAID implementing partner’s offices in particular, Time to Learn, ZAMFAM and ZPCT. The District Commissioner offices were happy that we had to pass through their offices and they pledged to give maximum support to the team. The target was to interview all the current USAID implementing partner, Government officials and conduct at least 2 FGDs with their beneficiaries in different locations within the province. Snow ball sampling recommendations were adopted with the tier running from USAID head of departments to heads of implementing organizations. The main objective of the team was to focus on DO 3 IPs and Government Officials in the province. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 75 Document Review The evaluation team thoroughly reviewed background documents and available reports on USAID/Zambia’s programming. USAID/Zambia CDCS; Field Data Collection Following a thorough document review, the evaluation team undertook an 11-day field visits to gather data from USAID/Zambia stakeholders in Lusaka, Central, Copperbelt, Luapula, and Eastern provinces regarding various aspects of CDCS performance. Challenges Some key informants that were scheduled to be interviewed from both Ministry of Education and Health were not available on appointment day and in some cases at the agreed time hence there was a lot of flexibility in the timing. Traveling from one district to another was also challenge as some of the time was spent driving before reaching the participants in some cases traffic jams especially between Kitwe and Chingola project and ZPCT. The timing of data collection coincided with the election campaigns which were at their peak and some respondents were apprehensive to be interviewed and this led to get clearance from District Offices as they did not want to be misquoted. Conclusion All in all, the team successfully conducted field data collection under the circumstances and the resources available. The Provincial medical office and Provincial education office and the implementing partners were very helpful and cooperative. Date Activities conducted Names of People contacted. Comments 24th July,2016 Traveled from Lusaka to Ndola N/A Arrived safely in Ndola 25th July 2016 Courtesy Ndola District Commissioner Office. Made appointment with Proncial Medical and Education Officers. Visited the Ndola District Education Officer￾Dr. Consist Mwale, Mr. Mapulanga, Ms. Liseli Masawana Names were provided for people to see at in Ndola. The team did most work transcribing the Lusaka KIIs Date Planned Interviews Achieved Location 26th July,2016 2 KIIs 1 FGDs 1 KIIs 0 FGD Ndola 27thJuly, 2016 3 KIIs 3 KIIs Ndola 28th July, 2016 4 KIIs 2 KIIs Kitwe /Muful;ira 29th July, 2016 3 KIIs 2 KIIs Kitwe/Luanshya Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 76 1 FGD 0 FGD 30th July,2016 3 KIIs 2 KIIs Chingola/Chililabombwe 31st July 2016 Transcribing and Reviewing transcriptions Kitwe 1 st August, 2016 Transcriptions continued and appointments for Kalulushi Kitwe 2 nd August, 2016 3 KIIs 0 FGDs 2 KIIs 0 FGDs Ndola/ Kalushi 3 rd August,2016 2 KIIs 1 KII Ndola 4 th August, 2016 Travel back to Lusaka Ndola Introduction IT Show has been contracted by the United States government through the United States Agency for International Development (USAID) Zambia to carry out a performance evaluation of its strategic plan and interventions in Zambia for the period 2011- 2015. Purpose of the Performance Evaluation USAID/Zambia seeks to assess the performance of its current CDCS for two primary purposes: 1) to determine where the Mission is with respect to reaching its development objectives; and, 2) to inform the development and design of its next CDCS by identifying what is working and what is not working. Therefore, the broad objectives of the performance evaluation of the CDCS are: 1. To determine the extent to which each CDCS development objective is being achieved and whether each is on track to meet its intended goal; 2. To determine the extent to which the CDCS design is appropriate for achieving the intended goal; and 3. To inform the development of the CDCS fiscal years 2017 – 2022. Document Review The evaluation team thoroughly reviewed background documents and available reports on USAID/Zambia’s programming. USAID/Zambia CDCS; Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 77 Field Data Collection Following a thorough document review, the evaluation team undertook a 10 field surveys to gather data from USAID/Zambia stakeholders in Lusaka, Central, Copperbelt, Luapula, and Eastern provinces regarding various aspects of CDCS performance. Luapula Team The team that went to Luapula was composed of three people the performance indicator specialist and two research assistants. The team was in Luapula Province from 24 of July to 4th August 2016 to cover the whole province in 10 days talking to both key informants from USAID implementing partners as well as government staff. Day One: The team paid a courtesy call on the Provincial medical officer who welcomed the team in the Province made them feel at home, thereafter a key informant interview was conducted with him. On the same day the team had key informant interviews with Mansa district Medical Officer and the Provincial Program Manager for ZPCT Bridge. Day Two: The team conducted key informant interviews with the Public health Specialist for Mansa District, Clinical care specialist and the Malaria focal person for the Province. After meeting the personnel from Ministry of Health (MOH) the team conducted a KII with field coordinator for Community Rising project under the Cupula foundation, while the interviews were taking place the team lead was making appointments for the coming day’s meetings. Day Three: The team conducted interviews with the nurse from SARAI project under society for family health and later in the afternoon proceeded to interview some community volunteers under the same project about 30 kilometers away from the business district. Day Four: The team conducted interviews with the field coordinator for Read to Succeed in the morning and the Provincial Education Officer (PEO) for the Province in the afternoon. Day Five Team conducted interviews with District Education Secretary (DEBs), the head teacher for Mabumba School and two focus group discussions one for children and another with teachers and parents from the same school. Day Six: The team worked despite it being a holiday (Saturday) and traveled to Kawambwa which is about 250 kilometers from Mansa and about 850 kilometers from Lusaka. The team managed to meet the District Social Welfare officer for Nchelenge and conducted a KII. Day Seven: The team traveled to Mbereshi which is 200 kilometers from Mansa and 800 Kilometers from Lusaka. The team did not manage to meet the Social Welfare officer as he had rushed for an in the emergency. However the team managed to speak with Provincial coordinator for the Zambia Rising Project under Save the Children. Day Eight: The team took time to work on the transcriptions taking advantage of the fact that it was a public holiday (August 1st Farmers day). Day Nine: Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 78 The team traveled to Samfya about 100 kilometers from Mansa and about 700 from Lusaka. In Samfya the team paid a courtesy call on the district commissioner, thereafter the team interviewed the District Medical officer and the district public health coordinator. Day Ten: The team interviewed the provincial pharmacist and the program coordinator who was in the field ,but the team had to follow him into the field about 300 kilometers from business district of Mansa. Challenges The team didn’t not face major challenges, however the team was in the Province at the time when the government stakeholders were finalizing their plans so that they could be beat the August 11 elections in case they would be change of government they would not want to do new plans. This made it very hard to meet the government staff and organizing focus group discussions (FDG’s). The district in the Province are far apart making it difficult to interview a big number of people. The communication from the IP’s HQ in Lusaka didn’t reach some of the field implementing partners about the evaluation. There was so much expectation by the community beneficiaries that the evaluation team would provide rewards to them after interviews. The period when the team was in the field was a campaign period for the general elections, therefore it was not easy to find some government staff in their offices. Conclusion The team successfully conducted field data collection under the circumstances and the resources available. The Provincial medical office and Provincial education office and the implementing partners were very helpful and cooperative. This was the case with the community beneficiaries. The team returned safely from the field on the 4th of August to continue transcribing and drafting the final report. Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 79 Schedule Activities Date Planned Accomplished 24/07/2016 Travel from Lusaka to Luapula 25/07/2016 2 KII’s 3 KII’s 26/07/2016 3 KII’s 3 KII’s 28/07/2016 2 KII’s 2 KII’s 29/07/2016 2 KII’s 2 KII’s 30/07/2016 2 KII’s,3 FDG’s 2 KII’s, 2 FGD’s 31/07/2016 2 KII’s 2 FGD’s (Kawambwa) 1 KII’s, 0 FDG’s 01/08/2016 2 KII’s 2 FDG’s (Mbereshi) 1 KII’s, 0 FDG’s 02/08/2016 Transcribing 03/08/2016 2 KII’s, 2 FDG’s (Samfya) 2 KII’s, 0 FDG’s 04/08/2016 2 KII’s Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 80 ANNEX 8: EVALUATION TEAM ORGANIZATIONAL CHART Final Evaluation Report, September 2016 Performance Evaluation of USAID/ZAMBIA CDCS (2011-2017) Annexes: 81 ANNEX 9: PROVINCES AND DISTRICTS VISITED BY EVALUATION TEAM PROVINCE COPPERBELT LUAPULA EASTERN LUSAKA DISTRICTS Ndola, Kitwe, Chililabombwe, Luanshya, Mufulira Chingola Mansa, Mbereshi, Kawambwa, Kashikishi and Samfya. Lusaka, Katete, Lundazi, Chipata, Mfuwe Lusaka DO DO 1 and DO 3 DO 3 DO 2 DO1, 2 and 3