Pastoralist Areas Resilience Improvement Through Market Expansion (PRIME), Ethiopia MID-TERM EVALUATION REPORT February 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Tufts University. The authors of the reports are Sean White, Getachew Gebru and Yacob Aklilu. Agriculture Knowledge, Learning, Documentation and Policy Project (AKLDP), Ethiopia Pastoralist Areas Resilience Improvement Through Market Expansion (PRIME), Ethiopia EXTERNAL MID‐TERM PERFORMANCE EVALUATION REPORT USAID Contract Number: 663‐13‐000006 Agriculture Knowledge, Learning, Documentation and Policy (AKLDP) Project Implemented by: Feinstein International Center Friedman School of Nutrition Science and Policy Tufts University Africa Regional Office PO Box 1078 Addis Ababa Ethiopia Tel: +251 (0)11 618014 www.fic.tufts.edu Disclaimer: The views expressed in the report do not necessarily reflect the views of the United States Agency for International Development the United States Government. CONTENTS EXECUTIVE SUMMARY ....................................................................................................................................I 1. INTRODUCTION............................................................................................................................................ 1 1.1 Background to PRIME and the evaluation .......................................................................................... 1 1.2 Project objective and intermediate results ........................................................................................ 1 1.3 Theory of change ........................................................................................................................................... 3 1.4 Project management, coverage and target groups .......................................................................... 3 1.5 Evaluation questions ................................................................................................................................... 4 1.6 Evaluation design .......................................................................................................................................... 4 1.7 Report limitations ......................................................................................................................................... 5 2. EVALUATION FINDINGS............................................................................................................................... 6 2.1 Evaluation question 1: Performance against objectives ............................................................... 6 2.2 Evaluation question 2: Effectiveness of PRIME leadership ..................................................... 16 2.3 Evaluation question 3: PRIME support to FTF nutrition objectives ..................................... 18 2.4 Evaluation question 4: Contributions to gender equity and female empowerment ...... 18 3. SUMMARY AND CONCLUSIONS ..................................................................................................................19 3.1 General conclusions .................................................................................................................................. 19 3.2 Evaluation question 1: Performance against planned objectives .......................................... 19 3.3 Evaluation question 2: Effectiveness of PRIME leadership ...................................................... 22 3.4 Evaluation question 3: PRIME support to FTF nutrition objectives ..................................... 22 3.5 Evaluation question 4: Contributions to gender equity and female empowerment ...... 23 4. RECOMMENDATIONS AND NEXT STEPS ...................................................................................................23 4.1 Priority recommendations ..................................................................................................................... 23 4.2 Evaluation question 1: Performance against objectives ............................................................ 24 4.2.1 IR1 Productivity and competitiveness of livestock and livestock products ............ 24 4.2.2 IR2. Enhance pastoralists adaptation to climate change ................................................. 24 4.2.3 IR3 Strengthened alternative livelihoods for households transitioning out of pastoralism ............................................................................................................................................................. 25 4.2.4 IR4 Enhanced innovation, learning and knowledge management .............................. 25 4.2.5 IR5 Nutrition status of HHs improved through evidence based interventions ...... 26 4.2.6 Recommendations on sub questions under evaluation question 1 ............................ 26 4.3 Evaluation question 2: Effectiveness of PRIME leadership ...................................................... 27 4.4 Evaluation question 3: PRIME support to FTF nutrition objectives ..................................... 28 4.5 Evaluation question 4: Contributions to gender equity and female empowerment ...... 28 4.6 Other recommendations ......................................................................................................................... 28 List of tables Table 1. Project targets and progress to date for FTF indicators ................................................................. 8 Table 2. Project targets and progress to date against PPR indicators ..................................................... 11 List of acronyms ACPA Aged and Children Pastoralist Association AEMFI Association of Ethiopian Microfinance Institutions AGP‐LMD Agricultural Growth Program – Livestock Market Development AH Animal Health AISDA Afar Integrated and Sustainable Development Association AMP Award Monitoring Plan BCC Behavior Change Communications BDS Business Development Service CAHW Community Animal Health Worker CBA Cost‐Benefit Analysis CCA Climate Change Adaptation CIAFS Capacity to Improve Agriculture and Food Security Project COP Chief of Party CRGE Climate Resilient and Green Economy CVCA Climate Vulnerability and Capacity Assessment ECDD Ethiopian Center for Development and Disability EEAHN Eastern Ethiopia Animal Health Network EIAR Ethiopian Institute of Agricultural Research ELMIS Ethiopian Livestock Market Information System EMD Economic and Market Development EMDTI Ethiopia Meat and Dairy Technology Institute EMMA Emergency Market Mapping and Analysis EMMP Environmental Mitigation and Monitoring Plan ENGINE Empowering New Generations in Improved Nutrition and Economic Opportunities FOG Fixed Obligation Grant FTF Feed the Future FTFMS Feed the Future Monitoring System GCC Global Climate Change GoE Government of Ethiopia GRAD Graduation with Resilience to Achieve Sustainable Development HAVOYOCO Horn of Africa Voluntary Youth Committee HEW Health Extension Worker IIF Innovation Investment Fund ILRI International Livestock Research Institute IR Intermediate Result JESH Jijiga Export Slaughter House LAND Land Administration to Nurture Development LKM Learning and Knowledge Management M&E Monitoring and evaluation MFI Microfinance institution MoA Ministry of Agriculture MSE Micro and small‐scale enterprise NBE National Bank of Ethiopia NLMIS National Livestock Market Information System NRM Natural Resource Management PLI Pastoralist Livelihood Initiative PMP Performance Monitoring Plan PPR Performance Plan and Report PRIME Pastoralists Areas Resilience Improvement and Market Expansion PVP Private Veterinary Pharmacy RAIN Revitalizing Agricultural/Pastoral Incomes and New Markets RFP Request for Proposals RuSACCO Rural Savings and Credit Cooperatives SACCO Savings and Credit Cooperatives SCIP Strategic Climate Investment Program SLA Savings and Loan Association SME Small and medium enterprise SMFI Somali Microfinance Institution TOP Transitioning Out of Pastoralism TVET Technical and vocational education and training USAID United States Agency for International Development VSLA Village Savings and Loan Association i EXECUTIVE SUMMARY This external mid‐term performance evaluation of the Pastoralist Areas Resilience Improvement through Market Expansion (PRIME) project was carried out from November 2014 to January 2015 in line with the Statement of Work (SOW) provided by USAID/Ethiopia and four specific evaluation questions. The project goal is to reduce hunger and poverty in selected pastoralist areas of Oromia, Somali and Afar regions, and the project objective is “to increase household incomes and enhance resilience to climate change through market linkages”. PRIME is a five‐year project, launched in 2012. The project design uses the Push‐Pull model involving increased production (the Push) and developing market linkages (the Pull), but has a broad intervention approach that includes increasing resilience to climate change, providing alternative livelihood options for those transitioning out of pastoralism, and improving nutrition for the most vulnerable. The PRIME Results Framework consists of five intermediate results (IR), each consisting of a number of key results, outcomes and outputs: . IR1: Improved productivity and competitiveness of livestock and livestock products; . IR2: Enhanced pastoralists adaptation to climate change; . IR3: Strengthened alternative livelihoods for households transitioning out of pastoralism; . IR4: Enhanced innovation, learning and knowledge management: . IR5: Nutrition status of households increased through evidence based interventions. The evaluation team reviewed project documentation, plans and progress reports and visited the three regions of Ethiopia where PRIME is being implemented. The team visited implementation sites and interviewed a large cross section of people involved in the project including pastoralists, participants in livestock and livestock products value chains, government officials and consortium implementing partners. The project work‐plans and progress reports provide quantitative data in the format of the 15 Feed the Future (FTF) and Performance Plan and Report (PPR) indicators, but more detailed progress reports on achievements against planned outputs in the format of the project results framework were not available, and this was a significant constraint to the team. To overcome this and to provide a quantitative basis for assessing progress against objectives, the team drew on discussions in the field and data provided by implementing staff and beneficiaries. These data, combined with a qualitative assessment based on field observations enabled the team to triangulate the data and assess project progress to date, and provided the basis for the teams findings, conclusions and recommendations. The findings, analysis and recommendations are intended to inform and improve implementation of PRIME during the remainder of the project. Key findings and conclusions This evaluation report presents 22 specific findings in section 2, structured around the four evaluation questions. The general conclusions drawn from these findings across IRs 1 to 5 are as follows:  The data presented in the progress reports in the format of the 15 FTF and PPR indicators lack the level of detail required to an provide an accurate quantitative assessment of progress in reaching the project objectives, outcomes and outputs.  Some reorientation is required to achieve the right balance between increasing production (the “Push” components) and improving market chains (the “Pull” components) with more focus required on increasing production.  Because of delays in start‐up, implementation is behind schedule and will require a substantial effort to catch up in the remaining years of the project. Adjustments are needed to the management systems and organization of the project to achieve the targets for years 2015‐2017. ii  Despite delays in start‐up there is good progress in some areas, notably IR1.2 improving market linkages. There is also good progress on IR2 on building capacity in rangeland management but this now needs to translate into increased production and lead to increased household incomes and enhanced resilience to climate change.  The nutrition component (IR5) needs to establish closer linkages between nutrition components (IR5.2) and production (IR1.1) to achieve the objective of increasing access to dairy products for children under 5. The key challenges to be addressed at present are:  Lack of a clear project results framework with revised project outputs and associated work‐ plans with clearly defined responsibilities for achieving targets. This is a major concern.  The PRIME organizational structure that centralizes control at Addis Ababa level and does not provide sufficient management control at Regional Cluster level to achieve the ambitious targets in years 3‐5. Key recommendations The evaluation produced 46 specific recommendations, as detailed in section 4 of the report. The priority recommendations are as follows:  Finalize the Performance Monitoring Plan, with a revised results framework, clear targets at the output level, clear responsibilities for implementation and a management information system that provides managers with regular reports on progress against targets.  The 15 FTF and PPR indicators have limited value for project level monitoring and management. More detailed progress reporting is required, including progress against outputs in work plans, or against custom project level indicators or a combination of both. The achievements reported in detail at the project level can then be consolidated and expressed as achievements at the FTF and PPR indicator level.  Recognize that field level staff capacity has increased sufficiently to enable management control to be increasingly decentralized to the Regional level along with responsibility for achieving results.  Increase production activities to achieve a better balance between the Push and Pull project components.  Maintain the primary focus on IR 1 (improving production and competitiveness of livestock and livestock products) as this is the key area of focus of the project.  Re‐focus IR2 on enhancing adaptation to climate change (rather than on natural resources management).  Establish functional linkages between the nutrition component (IR5) and increasing production (1.1) aimed at facilitating increased availability and access to milk products for children under 5 (IR5.2).  Consider increasing the Innovation Investment Fund from the current level of US$5million, but improve the assessment procedures, including more rigorous business case assessment to reduce risk. Provide technical assistance to grantees where appropriate.  Work with OFDA to improve operation of the crisis modifier fund to prevent a reoccurrence of the delays experienced in 2014.  Continue climate adaptation (IR2) activities on rangeland management planning, rehabilitation of water ponds (but in the context of rangeland management plans) and implementing the management plans.  Pilot climate adaptation interventions and assess them for suitability as resilience activities and for scaling‐up. 1 1. INTRODUCTION 1.1 Background to PRIME and the evaluation This external mid‐term performance evaluation of the USAID/Ethiopia‐funded Pastoralist Areas Resilience Improvement through Market Expansion (PRIME) project was carried out from November 2014 to January 2015 following the Statement of Work (SOW) provided by USAID. The evaluation aimed to assess: the performance of PRIME at the mid‐point of implementation; whether assumptions made during project design were still valid; how PRIME is being implemented and how it is perceived and valued; whether expected results are occurring or are likely to occur before the end of the project; and assess the management and operation of the project. The findings, analysis and recommendations are intended to inform and improve implementation of PRIME during the remainder of the project. In addition, the evaluation assessed the contribution of PRIME to overall Feed the Future and USAID’s Horn of Africa Resilience objectives. The PRIME project is a Cooperative Agreement (AID‐663‐A‐12‐00014) implemented over a five‐year period from 2012‐2017 in three region – Oromia, Afar and Somali. The project budget is US$52,972,799. The project is one of several being implemented under the Feed the Future (FTF) Strategy, which is a component of DO1 (“increase economic growth with resiliency in rural Ethiopia”) of USAIDs Country Development Cooperation Strategy for Ethiopia. The FTF Strategy aims to sustainably reduce poverty and hunger through investments in the performance of the agricultural sector, in improved nutrition and improving the capacity of vulnerable households to meet their food needs. The strategy includes the potential of market‐based agricultural development to reduce poverty and promote sustainable livelihoods for chronically food insecure households. 1.2 Project objective and intermediate results The PRIME project was designed to contribute to the Feed the Future (FTF) strategic objective of “Linking the vulnerable to markets”. The PRIME goal is “Reduced hunger and poverty” and the project‐level objective is “To increase household incomes and enhance resilience to climate change through market linkages”. Under this project objective there is a results framework consisting of five Intermediate Results (IRs), with related Key Results, Outcomes and Outputs. The IRs, Key Results and Outcomes are listed below (in addition there were 192 specific outputs in the original results framework): IR1. Improved productivity and competitiveness of livestock and livestock products Key Result 1.1 Productivity in key livestock market systems increased - Improved animal health services and utilization - Availability of quality feed and fodder for animals improved - Increased availability of pastoral livestock resources - Better livestock feeding practices adopted - Herd profiles improved (for productivity and climate resilience) Key Result 1.2 Market linkages improved - Technologies and information better utilized to effectively link to markets - More buying/selling opportunities (market integration) - HHs engaged in 'commercial' livestock production models - Infrastructures for value addition, phyto‐sanitary, and Halal improved - Private sector investment in livestock markets increased Key Result 1.3 Enabling environment improved - Appropriate financial services more available and accessible - Policies, laws and practices for industry growth more responsive, understood and utilized 2 - Improved public and private extension services utilized by HHs and other commercial actors - Enhanced collaboration for livestock production and management Key Result 1.4 Effective emergency response protects pastoralists assets - Early warning information utilized timely and effectively - Producers' attitude change towards asset management. - Working market linkages (and access) during emergencies IR2. Enhance pastoralists adaptation to climate change Key Result 2.1 Improved science and information sharing for decision making - Availability and access to relevant climate change and natural resource information increased - Access to and availability of information on adaptive technologies - Stakeholders understand and able to analyze climate and natural resources information - Systems for dissemination of relevant climate and NRM information, including early warning, developed and enhanced Key Result 2.2 Increased capacity for effective governance for climate resilience - Effective local governance systems for NRM and climate resilience improved - Federal and Regional institutions and policies effectively support NRM governance and climate resilience - Institutions capable of developing and updating early warning response plans Key Result 2.3 Implementation of climate solutions (ensured integration/utilization of climate solutions by IRs 1 and 3 - Climate resilient adaptive technologies accessible and utilized by TOPs and Pastoral Communities - Access and management of pastoral natural resources improved - Stakeholders demonstrate capacity to implement appropriate responses to early warning information IR3. Strengthened alternative livelihoods for households transitioning out of pastoralism Key Result 3.1 TOPs employability increased through life skills, financial literacy and entrepreneurship training - TOP's literacy and numeracy improved - Life skills and financial literacy improved - Small business skills enhanced - Technical/vocational training and mentoring opportunities increased Key Result 3.2 Increased income opportunities for TOPS - Access and availability to appropriate financial services increased - Access and availability to appropriate technologies and market information via private sector actors increased - Capacity by individuals and firms to initiate/better operate MSEs increased - Viable alternative income streams developed Key Result 3.3 Market access expanded to increase employment opportunities - Decent jobs created as a result of businesses expansion or start‐up - TOPs desire and obtain meaningful public and private sector employment - Investment effectively used to create jobs - Business associations support market access and expansion IR4. Enhanced innovation, learning and knowledge management Key Result 4.1 Program effectiveness improved through participatory monitoring and evaluation - Project performance enhanced through effective monitoring - Project decision making based on strong evidence 3 - Project activities rapidly adapt to lessons learnt , evidence base and changing conditions Key Result 4.2 Policy information base strengthened - Information base for decision‐making on pastoral areas is strengthened Key Result 4.3 Information dissemination and coordination improved - Improved consortium coordination through information sharing and enhanced management tools - Strengthened information linkages with pastoral networks and forums - Federal and regional government benefits from PRIME information resources Key Result 4.4 Two‐way communication channels strengthened - Program beneficiaries are fully informed, aware of PRIME - PRIME effectively communicates key message, best practices and research outputs to beneficiaries - Program effectively listens to the voice of the community IR5. Nutrition status of HH’s improved through evidence based interventions Key Result 5.1 Improved knowledge attitudes and practices - Improved knowledge of nutrition supporting practices through expanded communication and accessible counseling on essential nutrition action with a focus on IYCF - Sustaining, promoting and improving dietary diversity - Improving water sanitation and hygiene Key Result 5.2 Increased availability of and access to milk and livestock products for children under 5 - Improved animal health - Increased dry season milk production and availability - Improved preservation of livestock products - Improved equity in household decision making Key Result 5.3 Strengthened local capacities for supporting improved nutrition outcomes - Action research on nutrition impacts of value chain development - Public health systems strengthened for supporting nutrition outcomes 1.3 Theory of change The project design and implementation strategy draws on the “Push‐Pull “ model that aims to a) build the capacity of pastoralists to increase production and participate in markets (the “push”), and b) develops markets and improves market chains to generate demand for livestock products (the “pull”). While the main strategy focuses on increasing production and improving market linkages, the project has a broad intervention approach which includes related objectives including, increasing resilience to climate change, providing alternative livelihood options for those transitioning out of pastoralism, and ensuring that nutrition is improved for the most vulnerable. IR1 focuses on increasing productivity and improving value chains for livestock and livestock products while IR2 focuses on enhancing adaptation to climate change. IR3 is aimed at increasing alternative livelihood options for those transitioning out of pastoralism while IR5 focuses on increasing nutrition. IR4 provides the knowledge framework and information base for effective project management. These five IRs constitute a broad‐based approach to addressing the overall project objective involving a multiplicity of inter‐related activities requiring an integrated approach to implementation. 1.4 Project management, coverage and target groups The project is implemented by a consortium of nine partners led by Mercy Corps. The other partners are CARE International, Kimetrica, and local partners Haramaya University, Aged and Children Pastoral Association (ACPA), Afar Integrated and Sustainable Development Association (AISDA), SoS Sahel, the Horn of Africa Voluntary Youth Committee (HAVOYOCO), and the Ethiopian Centre for Development and Disability (ECDD). Mercy Corps is responsible for overall project management and strategic direction, and provides technical leadership on IRs 1, 3, 4 and 4 5 while CARE provides leadership on IR2. In terms of management leadership at the Regional Cluster level, Mercy Corps takes the lead in Oromia and Somali regions while CARE takes the lead in Afar. Kimetrica provides technical inputs on M&E, and Haramaya University undertakes research for policy development and for management planning. The project is implemented in three regions ‐ Oromia, Somali and Afar. Its target over the five years of the project life is to impact on 250,000 households in the three regions. 1.5 Evaluation questions The evaluation addressed four evaluation questions as specified in the SOW (Annex 1). The evaluation questions, listed in order of priority, were as follows: Question 1. To what extent is the PRIME Project progressing against overall project objectives as embedded in its Performance Monitoring, and Work Plan? Are the assumptions made in the original results framework and design still correct and valid? Areas of specific interest under this question include the following:  Has PRIME implemented any of the recommendations of the Push‐Pull Report?1  What are the promising or not so promising resilience actions and activities of PRIME in relation to resilience building? Specifically what was the impact of the US$1.5 million crisis modifier funds and commercial drought insurance triggered in August‐September, 2014 for livestock feed supplementation, commercial destocking and animal health services?  To what extent have climate change adaptation interventions been mainstreamed into IR1 and IR3 interventions and influenced/improved performances? To what extent have the Climate Vulnerability Capacity Assessment (CVCA) findings, community scenario planning’s, community dialogue findings, etc. under IR2 been utilized to improve: a) Productivity and competitiveness of livestock and livestock products under IR1; b) Alternative livelihoods for households transitioning out of pastoralism under IR 3  To what extent has PRIME effectively coordinated its activities with other FTF implementing partners, particularly the Livestock Market Development Project (LMD) and ENGINE? In evaluating this question, make recommendations about opportunities for greater future collaboration that could be further explored, including how PRIME should coordinate with the forthcoming USAID Lowlands Water Activity. Question 2. How effectively is the project being implemented by the PRIME leadership? Question 3. How have the PRIME activities supported FTF nutrition objectives specifically to achieve IR5 “increase nutritional status amongst pastoralists and TOPS with focus on women and young children and in particular diet diversity”? Question 4. To what extent has this project contributed to gender equity and female empowerment and specifically addressed the role of gender in decision making on the use of resources for livelihoods, maternal and infant feeding and increased women’s access to resources and services that will improve their nutrition and that of their children? 1.6 Evaluation design The evaluation involved four main activities: 1) A review of project documentation (a list of documents reviewed is provided in Annex 5) 2) A review and analysis of quantitative data provided in the project work‐plans, progress reports and other M&E data provided by the project implementers 3) Discussions in Addis Ababa with staff of Mercy Corps, CARE, Kimetrica and other implementing partners during which project data and other information provided by the partners was discussed with IR teams and senior management 1 This report is an internal analysis of the FTF Push‐Pull strategy in Ethiopia commissioned by USAID. 5 4) Field visits to the three regions of Oromia, Somali and Afar (and to Dire Dawa) to observe project activities and review progress During the design of the evaluation, quantitative data in the project work‐plans and progress reports were assumed to be a key resource for addressing the evaluation questions. However, PRIME progress reports were in the format of the 15 FTF and PPR indicators, and more detailed progress reports in the format of the project results framework were not available. To overcome this constraint and to provide a quantitative basis for determining progress against objectives, the evaluation team held detailed discussions with each of the five IR teams in each of the three regions, during which progress against the original planned outputs was discussed and quantitative data provided by the teams was compiled. This provided the evaluation team with an indication of progress in quantitative terms against each of the 190 original planned outputs in the 2012 PMP. During the field visits the team carried out a thorough review of project activities including: reviewing work‐plans and progress to date in detail with PRIME field staff in Oromia, Somali and Afar regions; interviewing regional and local government partners and project beneficiaries; visiting all Innovation Investment Fund (IIF) grantees; holding discussions with rangeland councils; visiting a Prosopsis clearing site and water pond rehabilitation sites; visiting Private Veterinary Pharmacies (PVPs) and other small grants recipients; visiting livestock markets and traders; visiting MFI’s and holding focus group discussions with Village Savings and Loan Associations (VSLAs); visiting Technical Vocational Education and Training (TVETs) centers and talking to TOPs trainers and trainees. The field observations and discussions with project staff and beneficiaries provided a qualitative assessment of progress to date. The team triangulated information from these sources, analyzed the information in conjunction with implementers and beneficiaries, and formulated the recommendations based on the findings and analyses. The feasibility of implementation during the remainder of the project was an important consideration for the team when formulating the recommendations, as the team aimed to provide recommendations that can significantly improve efficiency and effectiveness in implementation without unduly disrupting the current momentum of implementation. Further details on the evaluation design are provided in Annex 3 and CVs of the evaluation team are provided in Annex 8. 1.7 Report limitations As explained above, the project planning and reporting was based on the 15 FTF and PPR indicators and did not use the results framework of IRs, Key Results, Outcomes and Outputs as the format for work plans and progress reports. The lack of quantitative planning and reporting data in the results framework format was a significant constraint to the team, but the information on achievements collected during the field visits helped to overcome this constraint. In addition to assessing the quantitative data provided, the evaluation team was able to draw on field observations and discussions with project staff and beneficiaries to provide a qualitative assessment of progress to date. Triangulating these sources of information provided the basis for the teams findings, conclusions and recommendations. 6 2. EVALUATION FINDINGS The findings presented below are structured according to the four evaluation questions in the SOW. 2.1 Evaluation question 1: Performance against objectives Finding #1 There were delays project start‐up, partly due to delays in signing project implementation agreements with Regional Governments which were completed towards the end of the first year of implementation. This was due to inadequate consultation with Regional Governments at the project planning stage and as a result the implementing agencies took considerable time after the project was launched to explain the project’s market focused approach and to get their support and commitment to implementation. Other factors contributing to the delayed start‐up were the revisions to project targets and indicators and associated work‐planning activities, and training the field staff to use the “Concept Note” system for activity level planning. The main focus in year 1 was on data collection including baseline surveys and studies on key aspects of value chains, and Emergency Market Mapping and Analysis (EMMA) studies to provide the basis for detailed planning of project interventions. Apart from these studies and baselines, some training activities and mapping were done in the first year. Activities on the ground started in year 2 and have accelerated towards the end of 2014. Compared to achievements to date, the work‐plans for 2015‐17 show a substantial increase in activities and budget allocations. Finding #2 The project objectives and targets were specified in the Award Monitoring Plan (AMP) in the Cooperative Agreement (modification 2 signed in May 2013) and in the more detailed PMP/Results Framework developed at the start of the project. The AMP contains 57 indicators and associated targets for the Goal, Objective, IRs and Key Results. The more detailed results framework elaborates outcomes and outputs for each Key Result and specified a total of 192 outputs for the project. At an early stage of implementation, a process of revising the PMP began that focused on revising the project indicators and replacing them with higher level FTF and PPR indicators. The 57 project level indicators in the AMP were replaced by 12 FTF and 3 PPR indicators with associated targets and these were then used as the framework for project plans and progress reports. The revised PMP is still in draft form. Both the targets for the revised list of indicators and the reported progress against those indicators have been the subject of on‐going changes due to differing interpretations of the indicators and how project level achievements are contributing to those indicators. Therefore the project did not have a clear PMP and associated Results Framework at the time of the evaluation. As the project plans and progress reports are not in the format of the original results framework, the evaluation team was not provided with quantitative data showing progress against targets in the format of the original results framework, and so was unable to provide a quantitative assessment of progress in that format. The quantitative data presented in the tables below shows targets and progress to date in the format of the FTF and PPR indicators now in use. The FTF indicator data in the table was extracted from the FTF management information system in mid‐January 2015 and the PPR indicator data was provided by PRIME Q1. To what extent is the PRIME Project progressing against overall Project objectives as embedded in its Performance Monitoring, and Work Plan? Are the assumptions made in the original results framework and design still correct and valid? 7 project staff. The data is presented below in Tables 1 and 2 along with comments on achievements to date. As explained in the tables, the FTF indicators do not provide a satisfactory measure of progress towards the project objective. Finding #3 The end‐line targets for the indicators currently in use have not yet been finalized. The data on reported progress for each indicator were also subject to change depending on the on‐going changes in interpretation of the meaning of the indicators and how the activities undertaken on the project contribute to progress as measured by those indicators. Finding #4 The studies carried out in the first year covered all the areas of project interventions including; milk and meat value chains, cross border trade, animal health, marketing systems for veterinary drugs, best practices for rangeland management, traditional early warning systems, TOPs literature review, consultations with regional research institutions and universities, TVET profiles, insurance products and financial services, and EMMA reports. There is a wealth of relevant information already available from pre‐existing studies and the evaluation team noted some duplication with pre‐existing information. However, the studies provide relevant additional information that needs to be taken into account when planning project interventions, but the team also noted that the studies were not being utilized sufficiently by project staff for that purpose. Finding #5: Progress against objectives on IR1 ‐ Improved productivity and competitiveness in livestock market systems increased IR1.1 Increased productivity in livestock market systems The main focus of this Key Result has been on improving animal health but there has been little progress on the other important components of improving productivity i.e. increasing availability of feed and fodder for animals and improving livestock feeding practices (also note that improving production by improving rangeland management is covered under IR2). The project is improving animal health by facilitating expansion of Private Veterinary Pharmacies (PVPs) to remote pastoral areas through the project’s small grants component (discussed below). These are all new branches of pre‐existing PVP businesses and to‐date 23 have been established. Those visited indicated that they would not have expanded without the grant support. They address a need identified in the year 1 study on animal health and in the Climate Vulnerability Capacity Assessment (CVCA) report and are in appropriate locations, helping to improve animal health and productivity. Livestock keepers are willing to buy veterinary medicines and the intervention is financially self‐sustaining, market‐focused and appropriate for PRIME. IR1.2 Improving market linkages The two main mechanisms being used to improve market linkage are the Innovation Investment Fund (IIF) and the small grants fund. The IIF is a key component of IR1.2 and is aimed at supporting livestock value chains by encouraging new private sector investment through grants of up to 30 per cent of the investment cost. The grants are intended to reduce the risk for investors in high risk areas and leverage new investment. The team finds that the strategy is highly appropriate for PRIME and has the potential to have a strong positive impact on livestock and livestock products value chains. The numbers of applicants for grants was initially small but the entire grant budget of US$5million is now fully committed. 8 Table 1. Project targets and progress to date for FTF indicators (data sourced from the FTF management information system, 13 January, 2015; comments by evaluation team) Performance Indicator Target 2013 +2014 Achievement (Latest data 13 Jan 2015 ) Cumulative endline target Comments on the data, progress to date; potential to reach targets during the remainder of the project, and relevance of indicator against project objective Number of farmers and others who have applied new technologies or management practices as a result of USG assistance 6,687 13,604 29,937 The year 2 progress report compiled in Oct 2014 showed cumulative progress to date at 5,610. The data on progress shown here (13,604) is the revised figure that comes from the annual survey carried out by Kimetrica which was done after the 2014 annual report. In terms of the project objective of improved incomes and resilience, this performance indicator has limited value as the use of new technologies or management do not necessarily lead to higher incomes or resilience. Number of hectares under improved technologies or management practices as a result of USG assistance 1,717,110 3,435,214 27,080,097 The reported achievement gives the impression of substantial progress but only mapping has been done on most of the area reported. As further improved technologies and management practices are implemented on the same area, the reported achievement will remain the same (this illustrates the need for more detailed project level indicators to measure achievement at the project level). The year 2 Annual Report showed cumulative progress to date at 4,862,773. It is not clear on why the FTF MIS is now showing the lower figure. The end‐line target of 27m ha was incorrectly compiled and is due to be revised to about 8.5 million ha as this is the total area of the 24 or so rangeland units the project is working with. In terms of the project objective of improved incomes and resilience, the performance indicator has limited value. The project would have to show that livestock production had improved as a result of the technologies or management, and that this translated into higher incomes or resilience. Number of individuals who have received USG supported short‐term agricultural sector productivity or food security training 193,215 37,553 194,064 IR4 staff say the 2014 target of 193,215 is incorrect and the end‐line target is also unsatisfactory and due to be revised. PRIME had asked FTF to revise targets earlier in FY14 (to 18,640) but the MIS system did now allow it. The estimated end of project target should be about 50,000. In terms of the project objective of improved incomes and resilience, the performance indicator has limited value. The project would have to show that the training led to higher incomes or resilience. Value of incremental sales (collected at farm level) attributed to the FTF implementation. 5,250,131 4,916,866 24,004,305 The annual report showed US$6,267,146 achieved for year 2. The figure was subsequently adjusted due to a change in the interpretation of the indicator. The end‐line target is now satisfactory and considered achievable. 9 Performance Indicator Target 2013 +2014 Achievement (Latest data 13 Jan 2015 ) Cumulative endline target Comments on the data, progress to date; potential to reach targets during the remainder of the project, and relevance of indicator against project objective (Baseline US$3.5m) Value of new private sector investment in the agricultural sector or food chain leveraged by FTF implementation 10,500,000 10,316,987 17,000,000 The Yr 2 Annual report stated the same figure achieved to date. The process for calculating the amount of new private investment leveraged needs to be revised to ensure that only new investment is included (and that existing assets are not included in the calculation). If that is done, the leveraged amount to date would be lower than the amount currently reported. Number of food security private enterprises (for profit), producers organizations, water users associations, women's groups, trade and business associations, and CBOs receiving USG assistance 1,411 646 7,196 PRIME staff found fewer enterprises available to work with than originally envisaged and this accounts for the low achievement to date. Project staff consider that the endline target is too high and should be reduced to about 4,000. In terms of the project objective of improved incomes and resilience, the performance indicator has limited value. The project would have to show that receipt of USG assistance translates into improved income or resilience at hh level. Number of stakeholders implementing risk reducing practices/ actions to improve resilience to climate change as result of USG assistance 19,000 26,824 42,000 The figure reported as achieved to data comes from the annual impact survey carried out by Kimetrica. The Yr 2 annual report shows achievement of 5,610 to date and a target of 18,820. PRIME staff say the increase in achievement since October 2014 is due to changes in interpretation of the indicators. Number of jobs attributed to the FTF Implementation activities 1196 462 9,909 Yr 2 Annual report shows 462 jobs created and 115 obtained. This is attributed to late start of Belcash/SMFI. The end‐line target is likely to be met if IIF investments create the numbers of jobs expected. This a relatively strong indicator, but needs qualifying against the type of household/people accessing new jobs, and levels of income. 10 Performance Indicator Target 2013 +2014 Achievement (Latest data 13 Jan 2015 ) Cumulative endline target Comments on the data, progress to date; potential to reach targets during the remainder of the project, and relevance of indicator against project objective Number of agricultural enabling environment policies completing the following processes/steps of development as a result of USG assistance 12 17 21 The Yr 2 Annual report indicates cumulative achievement of 19 (this is attributed to double counting of mobile banking which went through 2 stages and therefore is counted twice, but has been corrected since then.) Number of children under five reached by USG‐supported nutrition programs (S). 31,892 32,192 165,000 The Yr 2 Annual Report shows achievement of 49,517 but notes overlaps of 32.5% within IR5 activities which when eliminated brings down the value to 32,193. The indicator has limited value as a measure of the projects success in improving HH nutritional status. The project would need to show if children receiving USG assistance have improved nutritional status relative to children not receiving this assistance. Number of people trained in child health and nutrition through USG supported programs 6,650 4,341 24,500 Achievement is lower than expected due to delay in approvals from Somali Regional State to implement nutrition activities in PRIME areas The indicator has limited value as the project objective is not training but improved child health and nutrition. So the project would need to show that training led to improved child health and nutrition. 11 Table 2. Project targets and progress to date against PPR indicators (Years 1 and 2 targets and achievements from the Y2 annual report; 2015‐17 targets were provided by PRIME IR4 staff) Indicator Target Yr 1+2 (Annual Rpt) Achieved to date. Yr 1+2 (Annual Rpt) 2015‐17 Target (from MC data) Comments on data, progress, potential to reach targets during the remainder of the project, and relevance of indicator against project objective Number of person hours of training in natural resources management and/or biodiversity conservation supported by USG assistance 55,000 101,473 78,000 IR4 staff explained that over achievement in year 2 due to expansion of capacity building and NR rehabilitation activities to 24 grazing systems. In terms of the project objective, the project will need to show how training led to improved NR management, which in turn led to improved livestock production and then better incomes or resilience. Number of stakeholders with increased capacity to adapt to the impacts of climate variability and change as a result of USG assistance 20,000 20,041 66,500 The results for progress to date are based on annual survey responses, although sub‐activity reports indicated that there are 13,857 people with increased capacity to adapt. But as this indicator is an outcome level indicator, PRIME decided to use the number from their annual survey which is a better measure of outcome. The indicator has limited meaning unless the level of capacity needed to handle climate variability is qualified e.g. assume a household needs a capacity score of 20 to manage climate variability well. If a project improves capacity of 20,041 stakeholders from 0 to 1, then the achievement is minimal. Total number of clients (households and/or microenterprises) benefiting from financial services provided through USG‐assisted financial intermediaries including non‐ financial institutions or actors. 16,880 12,662 66,560 No progress in Yr1 (target 8,400) but the Yr2 target was exceeded. 197 VSLA were established. SMFI‐Belcash mobile and agent banking is now approved by National Bank of Ethiopia. Also the entry of Rays MFI and Afar MFI will contribute to this This will help in achieving the ambitious end‐line target. The indicator has limited value as an indicator of progress towards the project objective. The project would need to show income increases as a result of financial services from PRIME, and relative to other ongoing and previous services. 12 A vetting process for IIF grant allocation is in place, but needs strengthening to overcome observed weaknesses. The team found that in some cases the investors do not have sufficient knowledge or experience of the value chains in which they are investing. In the case of the grant to the Addis Kidan Milk Processing factory in Afar, the team was of the view that the grant level may be too high and may distort the market, and be a barrier to other entrepreneurs entering the market. The IIF is expected to leverage new private sector investment in livestock and livestock products value chains with the project contributing 30 per cent while the investors contribute 70 per cent. It was observed that in some cases, investors’ existing assets are being taken into account in calculating their 70 per cent contribution giving a misleading indication of additional private sector funds leveraged. The majority of the IIF grant fund is allocated to enterprises in Somali and Oromia regions. The challenge in Afar has been lack of appropriate enterprises to support in comparison with the other two regions. The project also has a small grants component aimed at incentivizing investments that address identified gaps in value chains. The grants of up to US$25,000, referred to as Fixed Obligation Grants (FOGs), are provided for up to 30 per cent of the investment cost with the balance provided by the investor. The grants are allocated on a competitive basis following vetting according to selection criteria. To date 50 grants have been allocated. The range of grant recipients includes dairy processing and distribution businesses, 23 PVPs, women’s traders associations, agricultural cooperatives, agri‐input suppliers, solar energy distributors and livestock holding grounds. The original project results framework provided for engaging HHs on commercial livestock production and small‐scale dairy and fattening models, but there is no progress yet in this area. IR1.3 This is now covered under IR3. IR1.4 Emergency preparedness and responsiveness The crisis modifier funds released in 2014 aimed to prevent livestock deaths due to drought. The report on this intervention is not yet completed but interim data provided by the project indicates that 12,693 HHs benefited and 25,074 livestock were provided with supplementary feed amounting to 26,048 quintals of hay/grass and 11,676 quintals of concentrate. The aim was to sustain livelihoods by reducing livestock mortality and so the report should show the impact of the intervention on livestock mortality. There were delays in release of the funds and this resulted in livestock losses. In the case of Afar, the crisis modifier tracking triggers identified a problem in early July 2014, a rapid assessment was done and a Concept Note developed and sent to OFDA requesting USUS$1 million. OFDA responded immediately with a pre‐authorization of USUS$200,000, but the funds were not utilized until October 2014 when the final authorization was received, because procurement could not be done using a partial amount. In Liben zone of Somali region, the funds for water trucking also arrived late because of delays due to OFDA funding approval. Ultimately this activity went ahead using regular PRIME funds. Despite this delay, the activity was deemed by the Livestock Bureau in Moyale Woreda to have had critical impact in saving livestock. More precise data on livestock mortality reductions due to the intervention were not available. Finding #6: Progress against objectives on IR2 ‐ Enhanced pastoralists adaptation to climate change In general the evaluation team noted that the focus in IR2 is more on improving natural resources management rather than on enhancing pastoralist’s adaptation to climate change (which is the objective of IR2). 13 IR2.1 Improved science and information sharing There is good progress to date on resource mapping, information generation and climate forecasting and scenario planning. The Climate Vulnerability and Capacity Assessment study was completed and integrated into implementation plans. IR2.2 Capacity building and rangeland management planning Activities with rangeland councils are on‐going and building on lessons from PLI1 and 2. There is strong community support for these activities. IR2.3 Implementation of climate solutions The team found no progress yet on climate resilient adaptive technologies (2.3.1). The main focus is on Prosopsis clearing in Afar, which is a rehabilitation activity under IR2.3.2 and is being done on a pilot basis to assess feasibility and suitability for scaling‐up. The area cleared so far (250 ha) is small in relation to the total area affected. The method of clearance seems to be effective as the machines being used are successful in uprooting the trees rather than cutting them (which would result in coppicing), but should be monitored to determine the level of regrowth or reinvasion, if any. According to the Livestock Bureau in Samera, 1.7 million ha of land is already invaded by Prosopis and it would like to see a substantial expansion of Prosopsis clearing by PRIME. However, the data on the experimental area cleared so far has not yet been analyzed to provide the necessary information on the cost effectiveness of this approach and the feasibility of scaling up. There is progress on rehabilitating water sources in rangelands which increases resilience and also contributes to increasing production (IR1.1). The activity has strong community support. Progress is slow on piloting other adaptive technologies (2.3.1) and on implementation of the land use plans developed under IR 2.2. Finding #7: Progress against objectives on IR3 ‐ strengthening livelihoods for people transitioning out of pastoralism (TOPs) IR3.1 Work on increasing employability of TOPs Work is on‐going with skills and entrepreneurship training observed in Jijiga and Negele. Numbers benefiting to date are small and those benefiting seemed to be mainly in urban areas. The requirement for cost sharing is an obstacle to poorer HH’s and those from remote areas are not being reached. TOPs that have completed short‐term training do not readily get jobs and do not have start‐up capital to start in business In the Somali cluster, an employment agency “Hallo jobs” is sponsored by PRIME to help TOPS find jobs in Jijiga. It is operational for three months and has registered 1,900 job seekers and does not yet have information on whether any of those registered have got jobs. Five offices have been opened in Jijiga and additional offices are being considered for Dire Dawa, Harshin and Kebribeyah. PRIME covers all costs including salaries for Hallo Jobs. The project progress reports show 116 jobs have been obtained to date (the team was not able to verify this figure). IR3.2 Increase income opportunities The main focus is on making financial services more available through VSLAs, MFIs and Ru/SACCOs where progress is good. 197 VSLAs were established in 2014 and are working well in the initial year with PRIME support to facilitators. Afar MFI is receiving support from PRIME in the form of the CEO salary. Somali Micro Finance Institute (SMFI) is receiving support from PRIME including mobile banking training through BelCash for SMFI's management staff plus would be agents from Jigjiga, Degehabur and Fik. Haramaya University carried out a study on financial services in project areas (completed in November 2014) but it is not clear if this had any influence on planning for this component as it did not provide any recommendation to guide project planning. 14 The National Bank of Ethiopia has recently given approval to PRIME to pilot mobile money in Somali Region. This is a significant achievement for the project and has potential to have substantial positive impacts during the remainder of the project but it is not clear from existing plans how this facility will be utilized on the project. Finding #8: Progress against objectives on IR4 ‐ Enhanced innovation and learning IR4.1 Effectiveness increased through participatory M&E The PMP has been sent to USAID for approval but is still not finalized and a detailed revised implementation plan for the project is not available in the format of the project results framework (IRs/Key Results/outcomes and outputs). These are critical tools for effective project monitoring and implementation. Without these, it is not clear what the project targets are or how they are prioritized. A web‐based database and reporting system has been developed (Ki‐projects) which uses information from the Concept Notes to report on progress. The system has been up and running since mid‐2014 but is still evolving and not yet fully operational. It has the potential to provide rapid progress reports and analysis for project monitoring and management but its potential is not being adequately exploited at present. Annual and multi‐year plans are available but are geared to the FTF indicators and are not available in the format of the project results framework. Planning for implementation of activities is done using the “Concept Note” system. This involves elaborating details of activities to be implemented using a web‐based system. Concept Notes must be approved a various levels by project staff before implementation. Approvals are generally received within days but there are reports of long delays from some quarters. The same web‐based system is used for reporting on progress. There are some constraints due to poor internet connectivity at field level, which were not sufficiently taken into account at the planning stage. IR4.2 Policy information Haramaya University is an implementing partner and carries out studies and provides information both for improving planning on the project and for policy development. A substantial amount of research was done in the first year and is still on‐going. The research reports are available but the team found that the information being generated is not being utilized sufficiently by PRIME staff, who feel that the reports are too academic. Consequently, there is limited feedback from IR leads, technical advisors, team leaders and cluster leaders on studies already completed by Haramaya University. The team found that the research reports are of variable quality and also noted some overlaps between current research activities and information already available from earlier projects and other studies, indicating that there is duplication in the research work being done. IRs 4.3 and 4.4 Information generation and strengthening two‐way communication channels A regular schedule of standing meetings for coordination across IRs and partners exists but otherwise there is little progress on the planned activities under these outcomes. Finding #9: Progress against objectives on IR5 ‐ Nutrition status of HHs improved Evaluation Question 3 specifically focuses on the implementation of IR5 and the findings are discussed under that heading below (section 2.3). Finding #10: Extent of implementation of the “Push‐Pull” report recommendations The Push‐Pull theory is embedded in the project design and the team considers there is a good balance between Push and Pull components in the design. However in implementation, there is much more progress on the Pull aspects (such as development of market chains, processing etc.) than on the Push aspects. More emphasis is needed on the Push aspects (i.e. on increasing production) to achieve the right balance between Push and Pull. 15 Finding #11: Promising and not so promising resilience actions and activities of PRIME project in relation to resilience building The findings on the impact of the crisis modifier funds are already discussed above under Q1, IR1.4. As regards other promising and not‐so promising resilience actions on PRIME, the promising resilience actions are considered to be VSLA’s, Wet/Dry season grazing areas and mobility enhancement, climate forecasting and scenario planning, and capacity building of NRM institutions. One not‐so‐promising activity at present is the support for the employment agency “Hello Jobs” under IR3 which has made slow progress to date. Finding #12: Extent of mainstreaming climate change adaptations into IR1 and IR3 interventions and if it has influenced/improved performance The recommendations of the CVCA study are being integrated into IR2 but more is required on integrating the recommendations into IR1 and IR3. Two recommendations the team considers are not being addressed at present are conflict as a factor restricting access to resources and security of tenure over land and water. The team noted that some key members of staff were not aware of the CVCA study or its recommendations even though it is a key guiding document for planning activities in IR1 and IR3, as well as in IR2. Finding #13: Extent of coordination with other FTF implementing partners There are periodic meetings facilitated by AKLDP aimed at sharing experiences and exploring potential for collaboration and synergies between the FTF projects including ENGINE, LAND, GRAD, and LMD. There are also examples of field level coordination including the following: - PRIME is collaborating with LAND on sharing research and learning on tenure, ownership and management, and on practical field activities on rangeland demarcation and certification in Borena - PRIME is collaborating with LGP‐LMD on integration of pastoral livestock producers in the value chains (dairy, meat and live animals and hides) - PRIME is collaborating with ENGINE on training and awareness aimed at behavioral change and on harmonizing the approach to behavior change communication (BCC). - PRIME is collaborating with GRAD (Graduation with Resilience to Achieve Sustainable Development) on sharing experience on best practices of VSLA development and technical assistance to MFIs - LMD and PRIME jointly operate the Livestock Market Information System. LMD and PRIME also cooperate on field activities in areas where the two projects overlap. Finding #14: Whether the assumptions made in the original results framework and design are still correct and valid The assumptions in the original design are that the five IRs and their associated Key Results, Outcomes and Outputs will lead to increased HH incomes and enhanced resilience to climate change, and that these objectives will be reached through market linkages. The key feature of this approach (i.e. the project strategy) is the focus on developing and strengthening market linkages and this is what distinguishes this project from many previous projects. Increased production is essential for increasing trade so the two central components of this project are ‐ increasing production and improving market linkages. These are the first two Key Results in IR1 and the core components of the project. For that reason, it may have been better to have elevated the production and market linkages Key Results to IR level, with other components of the project as supporting Key Results. This would have put the focus firmly on the two most important components – production and market linkages (under such a design the resilience components IR2 could have been included under the production IR). In the current design, there is an assumption that the IR2 activities on capacity building with rangeland councils will lead to better rangeland management which will in turn lead to increased resilience and/or increased household incomes over the longer term. This 16 assumption will be proved valid if these activities lead to better management of the rangeland resources and ultimately to increased production. The current design and the indicators being used don’t recognize how pastoral households from different wealth groups engage in markets in different ways i.e. how richer HH’s can readily engage in the market whereas for poorer HH’s the priority is asset (livestock) accumulation, with limited sales until herds have reached a sufficient size. It is unclear whether the activities related to livestock development or rangeland management are targeted at poorer pastoral HH’s or all wealth groups. IR5 (the nutrition component) was not part of the original project design but was included in the project in year 1. There are clear linkages between this and increased production (IR1.1) and it can help to contribute to the PRIME goal of reduced hunger and poverty and therefore can be considered relevant to PRIME. In the current design, there are several overlaps and duplication at the outcome and output levels in the results framework (e.g. 1.2 and 2.3, or 1.3.2 and 4.2, or 1.4.1 and 2.1.3) and these overlaps should be rectified in revising the PMP and results framework to eliminate duplication and to clarify areas of responsibility. In the design, the M&E function is embedded as a Key Result under IR4. As this is a cross‐cutting function (like gender, financial management or disability inclusiveness) it would have been better placed as a cross‐cutting function across all IRs instead of under a specific IR. In terms of regional balance, the original design focused more on Somali and Oromia Regions than on Afar both in geographic coverage and scope of activities. In revising and updating the results framework as part of planning for the remainder of the project, the problems with duplication in outcomes and outputs and a poor regional balance can be rectified. 2.2 Evaluation question 2: Effectiveness of PRIME leadership Finding #15 PRIME is a complex project with a broad range of activities being implemented by a consortium of nine implementing partners working over a wide geographical area. The organizational structure for managing the project is complex. There is a dual system (technical/admin) where field based technical advisors report to IR leads at HQ while other staff including the partner organizations in the field report to the Field Managers and Cluster Managers (and according to one of the field managers, leaving some areas of responsibility in “no mans land”). As the Area Managers and Regional Cluster Managers do not have a line of command with field technical advisors and IR leads, their role is more a program coordinator role than a management role involving responsibility for achieving results and targets. While the Regional Cluster Managers do have a say on all activities carried on in the region as all Concept Notes must be approved by them, they do not have sufficient management control as the technical staff in their regions do not report to them (they report to IR leads in Addis). This means that control is centralized at the level of Addis based senior management and IR leads. Q2. How effectively is the project being implemented by PRIME leadership? 17 Figure 1. PRIME Organizational Structure (source: PRIME project management) Finding #16 The consortium initially experienced challenges in developing a team approach to implementation. The lead implementer Mercy Corps has been strengthening the team approach through “co‐location” of the main partners in the project office for three days, holding frequent coordination meetings and progressively empowering implementing partners. These measures have had some impact and consortium members are indicating a stronger team approach evolving. Finding #17 High‐level PRIME technical staff are concentrated in the Addis Ababa office, far removed from the field sites where problems must be analyzed and solutions found. The field based technical staff would benefit from more engagement with Addis based technical staff in developing concept notes and implementing appropriate and effective project interventions. Finding #18 Relations between PRIME staff and local authorities are mixed. This is partially due to the focus on market linkages that is new to Regional Administrations and the lack of “hand‐outs” sometimes associated with development projects. Another factor is the lack of adequate consultation with Regional Governments at the project planning stage and the current the lack of clear information on project plans and targets which local administrations need to ensure that donor projects are in line with local plans and priorities. All local government partners in the three regions complained of lack of clear PRIME project plans. Finding #19 Managers both at the Regional Cluster level and at senior management level do not have appropriate project plans and progress reports to enable them to know what their targets are and how well they are progressing against their targets. 18 2.3 Evaluation question 3: PRIME support to FTF nutrition objectives Finding #20 IR5 aims to improve the nutrition status of HHs through three key results. Progress on implementation is as follows: IR5.1 Improved knowledge attitudes and practices There are many activities being implemented in conjunction with regional and local level development agents aimed at behavior change, including training and drama shows, “edutainment” activities, posters etc. The impact on actually changing behavior however is not clear and a tool is being developed to assess the impact. IR5.2 Access to livestock products for children under 5 years old There is little evidence of progress on this key Result. It is closely linked to increasing availability of milk i.e. increasing production which is the role of IR1.1. IR5.3 Strengthened local capacities for improved nutrition There are many training and awareness activities on this Key Result in schools and other areas. It includes demonstrations of keyhole gardens and food preservation and storage technologies. The evaluation team noted that in general the activities in this IR are implemented in line with local priorities and plans and in conjunction with the local government Health Bureau. In Somali Cluster, the IR5 staff experienced reluctance initially on the part of the local health bureau as they felt that as well as training on improved nutrition attitudes and practices, people also needed food but this was not part of the project plan as PRIME is not an emergency relief project. 2.4 Evaluation question 4: Contributions to gender equity and female empowerment Finding #21 The project has one full‐time gender officer. To date most focus has been on gender planning, training and awareness activities and ensuring gender concerns are integrated into project planning. The gender officer works with all IR leaders but to‐date most focus has been on IR5 (Nutrition) and IR3 (TOPs). The project gender strategy was developed in June 2013 based on relevant gender information provided in the EMMA studies. The gender analysis of operational areas was scheduled but was not done as it was considered that the EMMA studies provided sufficient information for the strategy. Gender planning guidelines are currently in the process of development and a gender analysis on sheep and goats sector is planned for year 3. All project Concept Notes go through a gender vetting process. The planning and reporting process provides gender‐disaggregated data on beneficiaries. Q3. How have the PRIME activities supported FTF nutrition objectives specifically to achieve IR5 “increase nutritional status amongst pastoralists and TOPS with focus on women and young children and in particular diet diversity”. Q4. To what extent has this project contributed to gender equity and female empowerment and specifically addressed the role of gender in decision making on the use of resources for livelihoods, maternal and infant feeding and increased women’s access to resources and services that will improve their nutrition and that of their children? 19 Finding #22 The project has been making good progress in promoting the role of gender in decision making on the use of resources for livelihoods through many activities such as: promoting inclusion of women on rangeland councils; providing FOG grants to milk marketing cooperatives whose members are mostly women; providing grants to a Women Traders Association for participation in a trade fair in Dire Dawa; establishing 197 VSLA’s the members of which mostly women (aimed at empowering women and increasing resilience in times of food stress). Finding #23 Promotion of gender equity and female empowerment is a key component in IR5 as the role of women is central to improving child nutrition. Gender aspects are well integrated into the IR5 training, “edutainment” and other awareness raising activities, that are aimed at improving attitudes and practices on nutrition and diet diversity, and at improving equity in household decision making. In addition, the gender officer has developed a specific Concept Note for implementation in year 3 dealing specifically with gender aspects of infant and youth child feeding. 3. SUMMARY AND CONCLUSIONS 3.1 General conclusions The general conclusions of the evaluation across IRs 1 to 5 are as follows:  The data presented in the progress reports in the format of the 15 FTF and PPR indicators lack the level of detail required to provide an accurate quantitative assessment of progress in reaching the project objectives, outcomes and outputs.  Some reorientation is required to achieve the right balance between increasing production (the “Push” components) and improving market chains (the “Pull” components) with more focus required on increasing production.  Because of delays in start‐up, implementation is behind schedule and will require a substantial effort to catch up in the remaining years of the project. Adjustments are needed to the management systems and organization of the project to achieve the ambitious targets for years 2015‐2017.  Despite delays in start‐up there is good progress in some areas, notably IR1.2 improving market linkages. There is also good progress on IR2 on building capacity in rangeland management but this now needs to translate into increased production and lead to increased household incomes and enhanced resilience to climate change.  The nutrition component (IR5) needs to establish closer linkages between nutrition components (IR5.2) and production (IR1.1) to achieve the objective of increasing access to dairy products for children under 5. The key challenges to be addressed at present are:  Lack of a clear project results framework with revised project outputs and associated work‐ plans with clearly defined responsibilities for achieving targets. This is a major concern.  The PRIME organizational structure that centralizes control at Addis Ababa level and does not provide sufficient management control at Regional Cluster level to achieve the ambitious targets in years 3‐5 3.2 Evaluation question 1: Performance against planned objectives IR 1 is the main focus of PRIME as it includes the two most important Key Results – increasing production and improving market linkages. These are the two components of the Push‐Pull model. There is very good progress on the Pull aspects (market linkages) but very little progress on the Push aspects (increasing production). As both are essential components, the project needs to focus more on increasing production. (IR2 also has climate change resilience components that will result in increasing production). 20 There are three essential components for increasing production – water, feed and fodder, and animal health. To date the project has made good progress on the animal health, some progress on water (under IR2), some progress towards improving rangeland grazing in IR2 and eventually production, but very little has been done on feed and fodder and this aspect needs special attention. In the revision of the Results Framework, the responsibility for this activity should be clear (IR1.1) and target outputs included that will give sufficient weight to this component. There is potential to build on already existing commercial fodder production initiatives e.g. in Dolo Ado in Somali region, where agro‐pastoralists are engaged in producing green fodder that is hauled to market on donkey backs daily. This is a clear indication that pastoralists/agro‐pastoralists are willing to invest in commercial fodder production. Such activities could be supported by the small grants component to encourage expansion of commercial scale production, or through technical assistance. The Innovation Investment Fund The investment fund is an appropriate initiative and highly appropriate for PRIME and has the potential to have transformative impacts on improving livestock value chains. The entire fund (S5m) is already committed and will be disbursed as investors meet the disbursement criteria. In assessing grant applications, existing businesses should be favored over start‐ups as the investors know their market and therefore the risk of failure is less. In the case of JESH meat export slaughterhouse grant, the enterprise is a start‐up and there is no existing trade in frozen meat from Somali region to the intended markets in the Middle East. From discussions with the investors, the team formed the view that the investors would benefit from technical assistance to help them better understand the market they are expecting to penetrate and the challenges the enterprise will face. The project could consider providing appropriate technical assistance to the enterprise to help it understand the business challenges and market better and improve the chances of success. PRIME undertook a cost benefit analysis of the JESH as part of the project assessment which included a detailed assessment of the costs and benefits and sensitivity analysis but did not include an assessment of the market the product was expecting to penetrate or the feasibility of penetrating that market. This should have been included as part of the assessment of the feasibility of the business case for the investment. In the case the Berwaqo Milk Processing Plant in Jijiga, the team noted a lack of understanding of important technical aspects in milk processing (i.e. processing camel and cow milk in the same lines) that could have an important impact on the businesses. PRIME has already provided some technical assistance to this investor but should consider providing additional technical assistance to this and other IIF grantees to reduce the risk of failure. In the case of grants to milk processers, PRIME should carefully assess the potential impact of grants on others operating in the same value chains and ensure that the grants do not distort the market and provide unfair advantages for grantees, have negative impacts on other market participants, or create monopolies or barriers to entry for others. In assessing the value of leveraged funds, the team noted that in the case of some of the grantees, existing assets are taken into account leading to over valuation of leveraged funds. The team understands that a poultry venture is being considered for a grant from the IIF fund. The links between the poultry business and the PRIME objectives are not clear and should be critically assessed during the grant vetting process. Small grants (Fixed Obligation Grants or Competitive Cost Share Expansion Grants) The project is providing small grants to enterprises in livestock and livestock product value chains on a cost‐sharing basis. The grants target existing businesses where investors already have a track record. The examples visited during the evaluation were clearly viable businesses which would not have expanded into pastoralist areas without the small grants provided by PRIME. The initiative is appropriate for PRIME and is working well. To date PRIME has disbursed US$378,017 (with a further US$307,000 in the pipeline) and have leveraged private 21 investment of US$808,337. In the examples visited the team found the leveraged funds did reflect genuine additional private investment. Most grantees are in Somali (US$253,965) Region, with fewer in Oromia (US$125,000) and none in Afar. More focus is required in Afar where there are fewer small businesses than in the other regions. The small grants could provide the stimulus needed to leverage new investment in the value chains. The project does not have a specific financial limit for the small grant fund and should set a target when revising the results framework outputs. As this is an appropriate and key initiative in improving livestock and livestock product value chains, it should remain a high priority for the remainder of the project. The project should maintain a strong emphasis on rigorous assessment of applications, pitching the grant at the right level to encourage private investment while ensuring that it does not distort competition in the market. To minimize distortion in the market, the project should explore the potential for reduced grant allocations or for full cost recovery by the project. Although the project does have manual for the IIF, it was noted that the project does not have a manual describing the processes and procedures for operation of the small grant system. It is important, for transparency purposes, that this manual is developed and available to all. Enhancing adaptation to Climate Change The current focus of IR2 activities is more on natural resources management activities rather than on adaptation to climate change and needs to refocus. Progress is good on the capacity building activities with rangeland councils which aim to increase resilience and lead to improved production. These activities have good community support. The project should try to quickly move to implementation of rangeland management plans to show tangible results in terms of improved rangeland productivity and improved livestock production. The water‐pond rehabilitation activities can have impacts both on increasing resilience and on increasing production. Management of the ponds (including controlling access and maintenance) is an important consideration for sustainability, and therefore pond rehabilitation should only be done in the context of rangeland management plans where seasonal grazing has been identified and where maintenance and management of the ponds is controlled. In those situations, expansion pond rehabilitation activities could be considered when the outputs in the results framework are being revised. The project needs to look at cost efficiency on this activity however and ensure that water pond rehabilitation is being done in the most cost effective way. Use of local contractors could be explored and compared to current practices in terms of cost. An expansion of this activity would require additional engineering expertise in each Region. This would help accelerate progress but also ensure cost efficiency in implementation. The original plan for IR2 provides for identifying and piloting climate change adaptive technologies. No progress has been made on this yet although the IR2 staff have made some proposals (e.g. small‐scale irrigation for fodder production) that were rejected because they were not sufficiently market focused. As learning exercises, pilot testing adaptive technologies should be regarded as research activities rather than market activities. The Prosopsis clearing activity in Afar is appreciated by the local administration as it is tangible in contrast to capacity building activities that may have more long‐term benefits but are often less appreciated by communities and local administrations. The indications so far are that the clearing method is technically satisfactory and likely to lead to sustained clearance, but it needs further monitoring to confirm if re‐invasion will occur and under what circumstances. The subsequent management of the cleared areas is a key factor in determining if re‐invasion will occur but at present there is no long‐term plan for utilization of the cleared areas. As the cost effectiveness of the operation has not been analyzed, it is not possible to determine if the activity is suitable for scaling–up. The cost of clearance should be analyzed and the operation fine‐tuned to maximize efficiency. 22 The activity needs better planning to determine cost effectiveness, scalability (little land has been cleared to‐date) and relevance and should include a plan for the subsequent use of the cleared areas. Strengthening alternative livelihoods The indicator data shows slow progress in alternative jobs being created. However this may change as a result of the investments in the value chains through the IIF and small grants. In revising the PMP, an estimate should be made of the number of new jobs expected via businesses receiving IIF and small grants. “Hallo Jobs” operates as a job center to link job seekers with job providers, but has not registered much success to date. It is in operation for just three months but it would be prudent for the project to monitor progress and if it seems unlikely to deliver, the project should review its arrangement with this agency. 3.3 Evaluation question 2: Effectiveness of PRIME leadership There have been challenges in managing this complex project. In many cases the consortium approach makes it challenging to develop a team approach as competing organizations have to work together, overcome organizational rivalries and different organizational cultures and management styles. In the case of PRIME, the main constraints to management have been the organizational structure that has complex reporting lines and centralized control, and the lack of clear project targets and associated responsibilities for delivery. In the early stages, the consortium struggled to achieve a team approach despite frequent coordination meetings and co‐location measures. There were delays in project start‐up due to reluctance of regional governments to sign implementation agreements and this affected staff morale. There are indications however that as implementation has accelerated, there is greater delegation of authority and a better team approach has been developing. 3.4 Evaluation question 3: PRIME support to FTF nutrition objectives The main focus to date on improving nutrition has been on changing attitudes and practices through awareness and training activities. Progress in that area has been good. 6,519 heath care workers have been trained to date in the GoE’s curricula for nutrition. The tools now need to be developed to assess the impact of these activities. As regards progress on IRs 5.2 and 5.3, increasing access to milk and livestock products for children under 5, and strengthening local capacities for supporting improved nutrition, there is no apparent progress to date. Increasing production of milk (IR1.1) is key to increasing access to milk for children, which in turn requires increasing feed and fodder production. These important linkages between IR 5.2 and IR 1.1, and the ways in which they need to collaborate are articulated in the cooperative agreement but these linkages are weak at present. . The original results framework for IR5 should be revisited to ensure that activities under 5.2 and 5.3 are addressed in the remainder of the project. USAID‐funded research shows that pastoralist women are already very knowledgeable on child nutrition, and the importance of milk and dietary diversity. This indicates that the focus should be very much on livestock milk production rather than awareness and training. This indicates a need to work with the Bureau of Livestock on nutrition sensitive activities including increasing fodder production. The project also needs to make a clear distinction between wasting and stunting e.g. for stunted children in urban households, the issue is less about livestock production and more about access to milk which is affordable and available. At present there are linkages between IR5 and IR3 but in general IR5 activities appear to be implemented in isolation from the other IRs. A more integrated approach that involves collaboration with IR1.1 is required. Generally, there is good coordination on planning and implementation between the project staff and the Health Bureau, but there were some difficulties in Somali Region as the 23 project approach at present focuses on 'behavioral change', while the health Bureau insists that the priority issue is that people do not have enough food. Again this emphasis the important link between achieving success on IR5 and increasing production. 3.5 Evaluation question 4: Contributions to gender equity and female empowerment PRIME has a clear commitment to ensuring that gender equity and female empowerment issues are well integrated into the project plans and implementation. The gender integration strategy provides guidance and lists specific actions for each IR and provides a good basis for gender integration. However, the gender analysis of operational areas that was originally planned was not done. This would provide a better foundation for gender integration and should be done. 4. RECOMMENDATIONS AND NEXT STEPS The evaluation recommendations are drawn from the evaluation findings in section 2, and conclusions in section 3. All the recommendations fall well within PRIME’s capacity to implement during the second half of the project and will not unduly affect the momentum of project implementation that has been building in the latter half of 2014. 4.1 Priority recommendations The priority recommendations are summarized as follows:  Finalize the Performance Monitoring Plan, with a revised results framework, clear targets at the output level, clear responsibilities for implementation and a management information system that provides managers with regular reports on progress against targets  The 15 FTF and PPR indicators have limited value for project level monitoring and management. More detailed progress reporting is required, including progress against outputs in work plans, or against project level indicators or a combination of both. The achievements reported in detail at the project level can then be consolidated and expressed as achievements at the FTF and PPR indicator level  Recognize that field level staff capacity has increased sufficiently to enable management control to be increasingly decentralized to the Regional level along with responsibility for achieving results  Increase production activities to achieve a better balance between the Push and Pull project components  Maintain the primary focus on IR 1 (improving production and competitiveness of livestock and livestock products) as this is the key area of focus of the project  Re‐focus IR2 on enhancing adaptation to Climate Change (rather than on natural resources management)  Establish functional linkages between the nutrition component (IR5) and increasing production (1.1) aimed at facilitating increased availability and access to dairy products for children under 5 (IR5.2)  Consider increasing the Innovation Investment Fund from the current level of US$5million, but strengthen the assessment procedures, including more rigorous business case assessment to reduce risk. Provide technical assistance to grantees where appropriate  Work with OFDA to improve operation of the crisis modifier fund to prevent a reoccurrence of the delays experienced in 2014  Continue climate adaptation (IR2) activities on rangeland management planning, rehabilitation of water ponds (in the context of the rangeland management plans) and implementing the plans  Pilot climate adaptation interventions and assess for suitability as resilience activities and for scaling‐up Specific recommendations on each of the evaluation questions are detailed in the following sub‐ sections. 24 4.2 Evaluation question 1: Performance against objectives 4.2.1 IR1 Productivity and competitiveness of livestock and livestock products i. IR1.1. Revisit the planned activities and outputs on increasing production and revise the results framework to show targets for production outputs for the remainder of the project. - Increasing commercial fodder production should be prominent with clear targets and responsibility clearly assigned. Commercial fodder production is already being done by pastoralists in Dolo Ado in Somali region and there is potential to scale‐up this and expand to other areas in all three regions. There are opportunities for spate irrigation for fodder production where perennial rivers flow in Afar and Somali. - The original project plan envisaged engaging households in commercial livestock activities (small scale fattening and dairy farms) as a way of increasing production. This is still relevant and should be implemented. - The project can facilitate these production activities through technical assistance, awareness and training, and through the small grants facility. - Continue promoting PVPs and improving health services as this remains an important strategy for increasing production. ii. IR1.2 This Key Result is the core focus of PRIME. - Continue to support enterprises through IIF and small grants. If possible increase funds available for the IIF, although there may be constraints in the absorptive capacity. However, the procedures for grant assessment need to be strengthened with more rigorous business case assessment. The project should also consider technical assistance to the enterprises to reduce risk of failure. - Revise the process for assessing the amount of leveraged funds to ensure the amounts calculated represent genuine additional investment and exclude existing assets. - Actors in milk market chains are being supported in some areas but there is little progress yet on support for equipment vendors, making linkages between vendors and suppliers, delivering hygiene and handling messages, and developing and strengthening dairy marketing and processing value chains. More focus is required in these areas. - There is potential to scale up the small grant component especially in Afar which has benefited little to date but absorptive capacity may be a constraint. iii. IR1.4 The crisis modifier funds were disbursed and utilized late. Drawing on experiences from 2014, the emergency response procedures should be revised to overcome delays in future. 4.2.2 IR2. Enhance pastoralists adaptation to climate change i. Re‐focus IR2 on enhancing pastoralists’ adaptation to climate change rather than on natural resources management. This can be achieved when revising the results framework outputs. ii. IR2.1 and 2.2. Continue to strengthen capacity building activities with rangeland councils but take care to maintain a light touch, using a facilitative approach to ensure community ownership. Continue community dialogue and joint assessment of priorities. The outputs listed in the original results framework need to be reviewed to make them relevant to current needs. - Continue the work with communities on early warning information utilization, linking traditional weather forecasting and modern forecasting as a basis for weather based scenario planning. - More focus is required on feed supply as an adaptation strategy. There is potential for increasing feed supply through irrigated pasture, fodder and stover production particularly in Somali and Afar Regions, where the natural landscape is suitable for spate and permanent irrigation. (There is an overlap with IR1 on this activity which serves to both increase production and increase climate change resilience). iii. IR2.3 In year 3 PRIME should aim to deliver tangible impacts from the investment made in strengthening the rangeland councils, developing the maps and management plans. The 25 project should now move without delay to implementing the plans for improving rangeland quality. - Expand rehabilitation of water sources (ponds) but in the context of management plans which provide for maintenance and control of pond usage. Consider assigning fulltime engineering technical assistance at the Cluster level to support these activities. - Pilot activities for adaptive technologies should be identified and implemented in conjunction with pastoralists. This could include infrastructure adaptations as well as NRM adaptation activities. As research and demonstration activities, the requirements for cost sharing should not apply. Before implementation, plans should be developed which incorporate tools to measure impact, cost effectiveness and suitability as adaptive strategies and mechanisms. Policy briefs should be produced (in conjunction with the IR4 team) once the adaptive technologies have been pilot tested. - The Prosopsis clearing activity requires more attention to planning and monitoring aspects to increase cost effectiveness and determine feasibility as an adaptation strategy. This does not require a large‐scale research intervention. Rather, an action research approach by project staff on the ground and collection and analysis of relevant data to determine cost effectiveness and scalability is what is needed. Analyze the available information on the activity to date and if found cost effective, expand the activity to increase impact. - The platform on rangeland management jointly led by PRIME and the Ministry of agriculture needs to continue as a forum for information exchange, for informing policy, harmonization approaches and for capacity building. 4.2.3 IR3 Strengthened alternative livelihoods for households transitioning out of pastoralism i. Review and revise the results framework for IR3, which had 47 separate outputs in the original plan. Re‐strategize for improved impact based on experience in implementation to date and new information available from the research studies already undertaken including the recommendations of the EMMA study on TOPs training. ii. Ensure training is in line with the jobs market. Revisit the findings of the EMMA that researched skills supply and demand and identified skills training needs, including the current lack of skills to meet the demands of the construction industry. iii. The role of “Hello Jobs” should be expanded to advertising available vacancies and linking job seekers to available vacancies instead of just registering job seekers. Monitor progress and discontinue the arrangement with Hello Jobs if it is not succeeding. iv. Revisit the procedures used to identify TOPs for training so that TOPS from remote areas can benefit. v. Maintain a balance between woredas when identifying TOPs vi. Inability to pay for training is a barrier for many. Increase provision of scholarships or cost share study grant for attendance at long term training in TVET colleges to overcome current barriers vii. Consider linking trainees to MFIs and consider providing start‐up capital (e.g. tools) as part of the intervention design. viii. Continue to expand the VSLAs program. Monitor VSLAs ability to sustain activities after PRIME withdraws support to the facilitator 4.2.4 IR4 Enhanced innovation, learning and knowledge management i. IR4.1 M&E - Revise the list of outcomes and outputs in the original results framework. The outcomes need to be revised to eliminate overlaps in the original results framework. The outputs should be revised to provide a clear indication of project targets for the reminder of the project. - In revising the results framework, review the findings and recommendations of the research studies that were done in the first year of the project as they were intended to inform the planning process. 26 - Develop project plans for the reminder of the project to implement the targets in the revised and updated results framework. - Report on progress in achieving the outputs outputs in the format of the revised results framework. - Finalise the Performance Monitoring Plan (or M&E plan), that includes the 15 FTF and PPR indicators, custom project level indicators and/or output targets and a clear explanation of how the outputs contribute to the indicators. - Make the project plans available to project staff, implementing partners, USAID, other FTF projects. - The Concept Note system should be more integrated with the broader project planning framework and reporting system. CNs should be used as tools to elaborate plans for implementing and reporting on specific activities. Some modifications are required to improve efficiency including limiting the minimum financial threshold for CNs (e.g. to US$1,000) and allowing Cluster level of approvals below a certain value (e.g. below US$5,000). - Restructure the M&E function within the overall project management framework. M&E should be a cross cutting function and not a Key Result under IR4. Review the role of Kimetrica to improve its capacity to serve the M&E needs of the project. ii. IR4.2 Research: - Utilise the findings and recommendations of the research studies done to date in the revision of the PMP and project results framework. - Revise the procedures for identifying and serving the research and information needs of the project to ensure research is serving the needs of the project implementers. This will require flexibility to seek out and utilize the best quality skills available either from Haramaya University or from other sources within Ethiopia or internationally. - Link Haramaya with the AKLDP’s grants under contract to promote peer review and increased levels of accountability. iii. IR4.3. Begin to address Key results on information dissemination and 2‐way communication where very little progress has been made to date. This is an important function of IR4 and progress in this area is urgent. 4.2.5 IR5 Nutrition status of HHs improved through evidence based interventions i. Develop the tools for assessing effectiveness of awareness and education activities in changing behavior. This should be done in conjunction with other FTF partners (in particular ENGINE) to standardize the process and ensure findings can be collated. ii. Continue planning and coordination with Woreda level in planning years 3 and 4 activities. More coordination is recommended with the Zonal Office. Continue to align IR 5 activities with regional health, agriculture and livestock bureau plans as well as with national level plans. iii. Recognizing the linkages between increasing livestock production (1.1) and increasing availability and access to diary products for children under 5 (5.2), the project should seek to establish practical linkages and find synergies between these two Key results in implementation of activities. iv. Draw on the recommendations of the “Milk Matters” to improve effectiveness of this component and work with ENGINE to achieve synergies. 4.2.6 Recommendations on sub questions under evaluation question 1 i. Implementation of the “Push‐Pull” report recommendations: - More focus is recommended on Push elements, in particular on increasing production in order to achieve a better balance between Push and Pull. This balance can be achieved when revising the list of outputs in the project results framework. - Project staff who are planning and implementing activities in the field are not sufficiently aware of the Push‐Pull model and should be made aware of its relevance to the project’s design and implementation strategy. 27 ii. Progress in resilience building and Impact of crisis modifier funds in Aug‐Sept 2014 - Work with OFDA to overcome delays in release of funds, drawing on lessons from experience of implementation in 2014. - Finalise the report on impact of the 2014 crisis modifier funds which is currently in preparation. iii. The recommendation to review the procedures for managing the crisis modifier funds aimed at reducing delays has already been mentioned under IR1.4 above iv. Progress in mainstreaming climate change adaptations into IR1 and IR3. - Project senior management should ensure Cluster Managers and others planning interventions are aware of the CVCA findings and include in the revision of the results framework that is being recommended by this evaluation. - IR1 and IR3 staff should incorporate the CVCA recommendations into IR2 and IR3 in the course of revising the results framework. - Consider expanding AKLDP’s role as an external facilitator to include facilitating learning and information sharing within the FTF suite of projects. This could include facilitating joint planning on collection of baseline data and impact survey data by the FTF projects to standardize the approach to reporting on FTF indicators and to eliminate duplication and overlaps in data collection, especially where projects are geographically co‐located. - This role of AKLDP in coordinating learning among the FTF projects could include production and dissemination of regular policy briefs on findings from pilot activities and other lessons from the FTF projects. 4.3 Evaluation question 2: Effectiveness of PRIME leadership i. Revising the project planning and reporting system as recommended under question 1 above is a pre‐requisite for improving effectiveness of PRIME leadership as this is an essential tool for management. ii. Simplify the management structure and decentralize more control to the regional and area managers aimed at making them more responsible for achieving targets and providing them with the management control to achieve their targets. Introduce a line management structure whereby all staff in the region (both technical and administrative) report to regional managers. iii. Assign a cross cutting role to the M&E function. iv. Clarify roles and responsibilities for achieving outputs. After revising the list of outputs in the results framework and eliminating overlaps and where possible as already recommended above, assign clear responsibilities to individuals, units and teams to implement those activities and achieve outputs. This will help to avoid potential conflicts and unnecessary overlaps and duplication. v. Strengthen the team approach. If feasible, the main partners and staff who are full time on the project should be co‐located full time instead of returning to their offices for 2 days a week. The parent organizations’ administrative and other requirements should be modified to enable this approach. vi. Local organizations that are grant recipients should be sub contracted to implement activities, through sub grants based on CNs (preferably with substantial dollar value rather than in a piecemeal fashion). vii. High level technical expertise currently based in Addis should spend as much time as possible in the field, working with regional based staff, become familiar with problems and be able to generate solutions based on local knowledge and analysis with local staff. viii. Improve communication and partnership with Government and Regional administrations starting with improved consultation at the planning stage. Align planning with GoE and regional government plans while retaining the ability of the project to test and deliver innovative approaches. Provide revised project plans and progress reports to local government partners. These measures will increase local and national government 28 partners’ support for PRIME, increase sustainability of interventions and ensure project plans are in line with local priorities and plans. 4.4 Evaluation question 3: PRIME support to FTF nutrition objectives i. Continue the awareness and training activities aimed at changing attitudes and practices. Develop a tool to assess the impact of those activities. ii. Focus more on IRs 5.2 and 5.3 where little progress is evident to date. iii. Establish linkages and find synergies between the 5.2 activities and IR1.1 (increasing production), which is key to success on increasing access to milk. iv. Continue to strengthen collaboration with local government development agents and Health Bureau through which much of the IR5 activities are implemented. Align revised project results framework with the local government plans. 4.5 Evaluation question 4: Contributions to gender equity and female empowerment i. The gender analysis of operational areas as originally planned needs to be done for selected operational areas to provide a solid basis for effective integration of gender and female empowerment issues into project activities. ii. Gender planning guidelines for project staffs are being developed at present and this should be completed. iii. As gender and female empowerment is key to success in IR5 and in IR3, the focus of the gender officer’s work on those IRs at present should continue. iv. A constraint on the gender officer’s work has been the lack of information on planned activities and outputs from each of the IRs. The revision of the project results framework recommended throughout this report will provide the gender officer with the information required to overcome this constraint. 4.6 Other recommendations The project has a disability inclusiveness component which was not specifically mentioned in the mid‐term evaluation questions but it is an important cross cutting component on the project. It is implemented through the consortium partner ECDD that has one full time staff member based in Addis Ababa and one in each for the three regions. Like gender, this is a cross cutting component working with all IRs and has made good progress to date, focusing mainly on raising awareness among PRIME project staff aimed at ensuring that the project plans and interventions do not marginalize people with disabilities in pastoral communities but rather promotes inclusive development and increased self reliance among people with disabilities. The activities for the remainder of the project will include awareness and training, policy development and specific commercial activities aimed at economically empowering people with disabilities. The component should be recognized as an important component of PRIME and included as a component on similar projects in the future.