September 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by International Business & Technical Consultants, Inc. under IQC Number AID-RAN-I-00-09-00016, Task Order Number AID-623-TO-11-00002. Evaluation Report Mid-Term Performance Evaluation of the Somalia Partnership for Economic Growth Program COVER PHOTO Somalia, 2014, by Herman Albert Sanchez MID-TERM PERFORMANCE EVALUATION OF THE SOMALIA PARTNERSHIP FOR ECONOMIC GROWTH PROGRAM: MID-TERM EVALUATION REPORT September 14, 2014 This publication was produced for review by the United States Agency for International Development. It was prepared under IQC Number AID-RAN-I-00-09-00016, Task Order Number AID-623-TO-11-00002 by International Business & Technical Consultants, Inc. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government CONTENTS EXECUTIVE SUMMARY ...................................................................................................................................... iii BACKGROUND ................................................................................................................................................ iii FINDINGS AND CONCLUSIONS ..................................................................................................................... iii EVALUATION PURPOSE AND EVALUATION QUESTIONS .............................................................................. 1 EVALUATION PURPOSE ................................................................................................................................. 1 EVALUATION QUESTIONS ............................................................................................................................. 1 PROGRAM BACKGROUND ................................................................................................................................. 1 PROGRAM DESCRIPTION .............................................................................................................................. 1 PROGRAM PURPOSE, OBJECTIVE AND DEVELOPMENT HYPOTHESIS ................................................. 3 PEG AS DEMONSTRATION PROGRAM ........................................................................................................ 3 EVALUATION METHODS, TEAM AND LIMITATIONS ........................................................................................ 4 EVALUATION METHODS ................................................................................................................................ 4 EVALUATION TEAM ........................................................................................................................................ 5 KNOWN LIMITATIONS TO THE EVALUATION METHODOLOGY ................................................................. 5 FINDINGS, CONCLUSIONS & RECOMMENDATIONS ...................................................................................... 6 HAS PEG ACHIEVED THE OBJECTIVES AS PRESCRIBED IN THE PROGRAM’S PMP AND IN THE 2011 CONTACT AND THREE SUBSEQUENT MODIFICATIONS? ......................................................................... 6 DO THE PROGRAM’S RESULTS SHOW THAT INCLUSIVE ECONOMIC GROWTH HAS BEEN ACHIEVED BY ITS ACTIVITIES?................................................................................................................... 13 WAS THE DEVELOPMENT HYPOTHESIS AND ACCOMPANYING ASSUMPTIONS THAT SHAPED THE PROGAM DESIGN VALID? AND DOES IT REMAIN valid FOR THE CURRENT SOMALIA AND SOMALILAND CONTEXT AND USAID OBJECTIVES? ................................................................................ 15 HOW HAVE PEG’S OPERATIONAL STRUTURES AND IMPLEMENTATION PRACTICES PERFORMED IN SOMALIA’S CHALLENGING PROGRAMMING ENVIRONMENT? HOW HAVE THEY AFFECTED RESULTS? HOW CAN THEY BE IMPROVED? ............................................................................................ 17 WHAT HAVE BEEN THE MOST IMPORTANT PROGRAM ACCOMPLISHMENTS DURING IMPLEMENTATION? WHAT ARE LESSONS LEARNED FROM THE PROGRAM TO DATE THAT CAN BE APPLIED TO THE PROGRAM’S EXTENSION? HOW SHOULD THESE LESSONS BE APPLIED? .......... 24 ANNEXES ........................................................................................................................................................... 27 ANNEX 1: ADDITIONAL QUOTES SUPPORTING FINDINGS AND CONCLUSIONS ..................................... 28 improved stability ............................................................................................................................................ 28 PRIVATE SECTOR ENTITIES STRENGTHENED ........................................................................................ 28 PRIORITY VALUE CHAINS IMPROVED ....................................................................................................... 28 peg’s flexible implementation .......................................................................................................................... 29 peg’s PARTICIPATORY APPROACH TO PLANNING AND IMPLEMENTATION ......................................... 29 peg’s OTHER SOUND OPERATING AND IMPLEMENTATION PRACTICES .............................................. 29 NEED FOR LONGER TRAINING ................................................................................................................... 30 ANNEX 2: EVALUATION STATEMENT OF WORK ........................................................................................... 31 ANNEX 3: BIBLIOGRAPHY ................................................................................................................................ 41 ANNEX 4: LIST OF KEY INFORMANT INTERVIEWS ....................................................................................... 45 ANNEX 5: LIST OF FOCUS GROUP DISCUSSIONS ....................................................................................... 46 ANNEX 6: KEY INFORMANT INTERVIEW DICUSSION GUIDE ...................................................................... 47 ANNEX 7: FOCUS GROUP DISCUSSION GUIDES ......................................................................................... 54 ANNEX 8: GUIDELINES FOR THE STRUCTURED FOCUS GROUP DISCUSSIONS (FGDS): ENTREPRENEURS’ SPOUSES ......................................................................................................................... 58 ANNEX 9: DISCLOSURE OF CONFLICTS OF INTEREST............................................................................... 61 ANNEX 10: THE M4P APPROACH .................................................................................................................... 63 Page i – PEG Performance Evaluation ACRONYMS ADESO African Development Solutions AOR Agreement Officer’s Representative BE Business Environment BOCAT Business Organization Capacity Assessment Tool CAHW Community Animal Health Worker CEO Chief Executive Officer CLHE Candlelight for Health, Education & Environment CNFA Citizens Network for Foreign Affairs (legally changed to CNFA in 2007) COR Contracting Officer’s Representative CSO Civil Society Organization DAI Development Alternatives Incorporated DCOP Deputy Chief of Party DEC Development Experience Clearinghouse DFID Department for International Development EA East Africa EU European Union FAO Food and Agriculture Organization FGD Focus Group Discussion FSNAU Food Security and Nutrition Analysis Unit GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit HATI Horn of Africa Training Institute IBTCI International Business and Technical Consultants, Inc. IFAD International Fund for Agricultural Development IPP Independent Power Producer IR Intermediate Result KII Key Informant Interview LPC Limited Presence Countries M4P Making Markets Work for the Poor M&E Monitoring and Evaluation MEPS Monitoring & Evaluation Program for Somalia MFI Micro Finance Institution MoERT Ministry of Environment and Rural Development MoLAH Ministry of Livestock and Animal Husbandry MSE Micro and Small Enterprise NGO Non-Governmental Organization PEG Partnership for Economic Growth PMP Performance Management Plan PPD Public-Public Dialogue PSD Private Sector Development RFTOP Request for Task Order Proposal RSA Research Solutions Africa SATG Somali Agriculture Technical Group Page ii - PEG Performance Mid-Term Evaluation SBCI Somaliland Business Confidence Index SME Small and Medium Enterprise SMS Short Message Service SOW Statement of Work STA Senior Technical Advisor SWSPPA Somaliland Wind and Solar Power Producers Association TA Technical Assistance USAID United States Agency for International Development Page iii - PEG Performance Mid-Term Evaluation EXECUTIVE SUMMARY BACKGROUND This report presents the mid-term performance evaluation of the Partnership for Economic Growth (PEG) program operating in Somaliland and Puntland State in Somalia. It aims to inform USAID decisions regarding future economic growth programs in Somalia, including activities that fall under PEG’s contract extension. The USAID/EA/Somalia Office is the primary audience for this mid-term evaluation, with secondary audiences including USAID Washington and other program stakeholders, including Development Alternatives Inc. (DAI), Somali authorities, implementing partners, donors, and other development organizations. PEG is a private sector development (PSD) program funded by USAID/EA/Somalia. It began on April 15, 2011 and on August 20, 2013 was granted a two-year cost extension, allowing program activities to run until August 31, 2015. PEG works closely with private sector businesses, government ministries, non-governmental organizations (NGOs), and civil society organizations (CSOs) to promote economic growth and stabilization in Somaliland and Puntland. Program activities focus on two areas: private sector development and strengthening specific productive value chains. The PEG Partnership Fund also provides matching grants to promising local micro and small enterprises (MSEs) and to NGOs. It is hoped that the lessons learned from PEG will contribute directly to the legitimacy of the PSD approach (more specifically the Making Markets Work for the Poor, or M4P approach) within the Somali context and its subsequent replication and expansion. Annex 10 presents a brief summary of the M4P approach. DAI has sole implementation responsibility and oversight for the program. In this role, it worked with several sub-contractors on different aspects of the program. The PEG mid-term evaluation used a qualitative approach consisting of three primary data collection methods: (1) document review, (2) key informant interviews (KIIs), and (3) focus group discussions (FGDs). The evaluation team reviewed documents either produced by PEG, by other entities about PEG or on topics related to PEG; conducted 33 key informant interviews with diverse program stakeholders drawn from program staff, government officials, implementing partners, program beneficiaries, and other market actors in Somaliland and Puntland; and held 13 FGDs with program beneficiaries and, in some cases, their spouses. The evaluation methodology has a number of limitations that have important implications for the types of findings and conclusions that can be drawn from the evaluation. The limitations of the mid-term evaluation methodology include the following: (1) it is not capable of attributing observed results to program interventions in any quantitative way, (2) it relies on small samples and covers a sub-set of program interventions, (3) it does not measure indirect effects but only direct effects, (4) it does not measure the longer-term sustainability of program interventions, and (5) it relies heavily on subject measurements based on perceptions of KII and FGD participants. FINDINGS AND CONCLUSIONS This section of the Executive Summary summarizes the major conclusions and recommendations from the PEG mid-term evaluation, organized by the five questions the evaluation sought to address. While the evaluation also generated a number of findings related to each evaluation question, on which the conclusions and recommendations were based, these have been omitted from the Executive Summary for the sake of brevity. Readers are encouraged to read the entire report for a much more detailed presentation of the evaluation findings, conclusions, and recommendations. 1. Has ‘The Partnership for Economic Growth Program’ achieved the objectives as prescribed in the program’s PMP and in the 2011 contract and subsequent three modifications? PEG aims to improve stability through inclusive economic growth. Stability is a multi-dimensional concept that, for the purposes of this mid-term evaluation, includes the two dimensions (1) increased security and (2) Page iv - PEG Performance Mid-Term Evaluation improved economic conditions, the latter of which includes the two sub-dimensions (2a) private sector entities strengthened and (2b) priority value chains improved. The mid-term evaluation found no evidence that PEG made a substantial contribution to increased security in Somaliland or Puntland. The program’s scale/outreach was too small and there is neither qualitative nor quantitative evidence that shows that security is a pertinent issue among the PEG beneficiaries. At the same time, PEG’s interventions were effective in strengthening the knowledge and capacity of both private and public sector entities, improving the performance of assisted value chain actors, and strengthening the horticulture, livestock and fodder value chains. Due to the small scale of PEG’s interventions, these improvements are limited to the relatively small number of market actors who participated in program interventions, and there is no evidence that any interventions produced widespread improvements in economic conditions. In spite of the above, there are signs of potentially significant outcomes in the energy value chain, where different market actors are making or planning large investments in wind-generated energy. There are also signs of significant outcomes following the livestock intervention in Puntland, although they are not nearly as broad￾based as in the energy value chain. Recommendations Based on the above conclusions, the evaluators offer the following recommendations:  If PSD programming aims to achieve broad-based change, USAID/EA/Somalia should fund PSD programs with the appropriate scale and over a long enough period to achieve this objective. PEG successfully demonstrated the feasibility of the PSD approach in Somalia; therefore PSD programs in Somalia need not take such a limited, experimental approach but should instead aim to achieve more broad-based results.  To achieve larger scale, and thus broader-based impact, USAID should consider adopting the M4P approach to private sector development.  USAID/EA/Somalia should continue to combine market (value chain) interventions with business environment (BE) interventions, as the latter can be a particularly effective strategy for achieving scale and broad-based impact.  USAID/EA/Somalia should invest additional resources to expand the community animal health worker (CAHW) standard curriculum. Along with expanding the curriculum, additional resources would need to be invested in monitoring its implementation to ensure that its application (in addition to its content) is standardized as well. 2. Do the program’s results show that inclusive economic growth has been achieved by its activities? Once again, PEG aims to improve stability through inclusive economic growth. This question asks whether this particular aspect of the program objective has been achieved. Conclusions Page v - PEG Performance Mid-Term Evaluation PEG prioritized the inclusion of women, youth, poor and different clans/sub-clans in its intervention strategy and implementation. In the process, it managed to achieve significant female representation in a number of interventions and sub-grants and, in the case of Shaqodoon, reach hundreds of youth with employment assistance. The actual share of different clans and sub-clans or poor among intervention participants and beneficiaries could not be determined, although PEG did consciously seek to achieve clan/sub-clan diversity and poverty inclusion through its geographic targeting strategy and its selection of sectors critical to the livelihoods of poor rural households. As to whether PEG’s interventions produced ‘inclusive economic growth,’ the evaluators were not able to answer this question directly. Given the relatively small scale of program interventions, it is doubtful that they contributed to any broader-based ‘economic growth’, although evaluators did find evidence that interventions contributed to improved performance results for the farmers, Community Animal Health Workers (CAHWs), pharmacies, enterprises, etc. directly benefitting from program activities. It is fair to assume that women benefited from these improvements according to the extent of their participation in the program. Recommendations Based on the above findings and conclusions, the following recommendations can be made:  PSD programs should include an explicit strategy for achieving inclusivity and creating inclusive results for each group targeted. It should ensure that this strategy is closely monitored to ensure adherence and to make adjustments given developments in the field.  As a strategy to promote inclusivity, PSD programs should consider the following approaches: o Target multiple levels in a particular value chain from the farm/firm level to end-consumers where the targeted group is active. Women, for example, often play defined roles within the production and post￾production process and/or within support markets (e.g., vendors), and interventions may seek to strengthen their capacity to play those roles. o Focus on key sectors that cut across social groups and locations as a means of benefitting a broad range of social groups at both the farm/firm and household levels.  PSD programs that use participatory methods to select intervention participants should use well-articulated and agreed selection criteria, including criteria for selecting members of targeted social groups. 3. Was the development hypothesis and accompanying assumptions that shaped the program design valid? And does it remain valid for the current Somalia and Somaliland context and USAID’s objectives? PEG’s development hypothesis is as follows. As so many external factors outside of PEG’s control contribute to instability, the sequential and unidirectional progression of achievement from outputs to outcomes to impact does not always take place. The iterative development approach should, to a significant degree, enable PEG to be flexible – to respond to variances in the geography where the program will work, as well as to the changing environment of development in those areas. Conclusions The development hypothesis was accurate when PEG was designed and it remains accurate today— up to a point. It accurately assumed that PSD interventions can successfully be implemented in the Somali context and that they contribute to improved outcomes, particularly improved practice and performance outcomes. Taking this conclusion further, PEG also demonstrated that a PSD approach of working through diverse private, public, NGO and CSO actors is not only possible in the existing Somali context, but also contributes to successful program implementation. None of the research suggests that a PSD approach cannot work in Somalia. Page vi - PEG Performance Mid-Term Evaluation Evaluators also conclude that the development hypothesis is accurate in assuming that PEG would use a flexible and iterative approach and in stating that this flexible approach would contribute to improved outcomes. Evidence shows that PEG did in fact pursue a flexible implementation strategy that allowed it to adapt to circumstances it encountered in the field and that this flexible approach enabled it to improve its outcomes, in addition to helping create a more collegial and collaborative relationship between PEG and its many sub￾contractors. One cannot, however, conclude that a PSD approach is an effective means of improving stability within Somalia, as evaluators were unable to verify whether PEG’s PSD interventions contributed to improved stability. It remains unclear whether this part of the development hypothesis is accurate. More broadly, as the development does not articulate a hypothesis as to how economic growth contributes to stability, evaluators were unable to assess whether the hypothesized causal mechanisms (intermediate outcomes) are accurate either. Recommendations Based on the above findings and conclusions, evaluators offer the following recommendations:  In articulating development hypotheses, USAID/EA/Somalia should specify more clearly a theory of change describing how program interventions will contribute to the program Purpose and Goal.  If improved stability is to remain a principal mission goal and an objective of its PSD programming, USAID/EA/Somalia should verify just how important an issue stability is, what its causes are, how it manifests itself, and what its specific connections to economic growth are, and then incorporate this knowledge into future PSD programming, program design, and statements of work.  USAID/EA/Somalia should accept the viability of the PSD approach and focus on methods to replicate it and expand it. PEG’s successes—particularly its ability to mobilize multiple stakeholders across the private, public, NGO and CSO sectors to work together on private sector strengthening—lay to rest any questions regarding the validity of a PSD approach in the Somali context.  USAID/EA/Somalia should consider conducting a rigorous impact evaluation of a future PSD program to enable it to measure whether and how PSD programs contribute to improved stability or other high-level Mission objectives.  PSD programs should incorporate a significant degree of flexibility into program design and implementation given the dynamic nature of market systems where they work and the inevitability that program outcomes, and all of the causal mechanisms leading to those outcomes, are unlikely to be known in advance. 4. How have PEG’S operational structures and implementation practices performed in Somalia’s challenging programming environment? How have they affected results? How can they be improved? Conclusions PEG’s operational structures and implementation practices performed well in the Somali programming environment. The soundness of these structures and practices played a key role in PEG’s success, both in terms of program implementation and results.  The extensive due diligence work undertaken by PEG prior to launching interventions helped ensure, on one hand, that the reach of PEG’s market interventions reflected the geographic, economic, social and other factors characterizing specific market systems and, on the other hand, that enterprises offering the substantial potential for contributing to enhanced economic outcomes were selected for grant financing.  PEG’s commitment to transparency and participation demonstrated that this approach works in Somalia. It helped foster an environment of trust and cooperation among the program’s sub-contractors, sub-grantees and beneficiaries, and it purchased goodwill that helped smooth over any rough patches. Page vii - PEG Performance Mid-Term Evaluation  PEG was able to operate successfully in Somalia while adhering to strict procurement rules, holding sub￾contractors to high standards of performance.  PEG’s flexible implementation approach combined with its high quality training/TA assistance was widely appreciated by all stakeholders and contributed substantially to program achievements.  PEG took sustainability seriously and attempted to build it into interventions, although with mixed success. This mid-term evaluation also concludes that training/capacity development activities were probably not sufficient and could have benefitted from follow-up reinforcement efforts. Where PEG did encounter implementation challenges, it was able to deal with them and go on to implement its interventions successfully. While there was some dissatisfaction expressed by different stakeholders with PEG’s operating structures and implementation practices, these were the exceptions rather than the norm. On the whole, PEG had sound operational structures and implementation practices, which were instrumental in contributing to the many successful outcomes described by program stakeholders. Recommendations Based on the aforementioned findings and conclusions, recommendations are found below. These recommendations summarize the answers to the question as to how PEG can improve its operational structures and implementation practices.  USAID/EA/Somalia should continue insisting that PSD programs always adhere to accepted best operational/implementation practices. PEG’s success demonstrates that there is no need to sacrifice sound operating/implementation practices in order to implement market interventions successfully. These best operational/implementation practices should include, as a minimum, strict adherence to the principles and practices of due diligence, transparency, participation, flexibility and sustainability.  PSD programs should include an explicit plan to achieve sustainability. Sustainability is more the exception than the norm, and unless PSD programs plan for sustainability and build it into their intervention design and implementation strategies, gains achieved are unlikely to survive much beyond the program.  PSD programs should budget and plan for sufficiently long training/capacity development activities, including after activity follow-up. The training/capacity development assistance provided by PEG was too short in many cases, leaving significant capacity gaps remaining. 5. What have been the most important program accomplishments during implementation? What are the lessons learned from the program to date that can be applied to the program’s extension? How should these lessons be applied? Important accomplishments The most important program accomplishments are summarized below.  Some of the most important program accomplishments had to do with PEG’s internal operating structures and implementation practices that demonstrated the feasibility of its approach in the Somali context, including the following: o PEG demonstrated that a PSD approach, using sound and transparent operational/implementation practices, can succeed in Somalia. o PEG successfully brought together diverse market actors from multiple sectors, with no previous history of working together, to demonstrate that working collaboratively toward mutual benefits is feasible and can be an effective market development strategy. Page viii - PEG Performance Mid-Term Evaluation o PEG demonstrated that public sector officials are willing and can be engaged in market development activities. o More generally, PEG demonstrated that donors and other development organizations need not feel bound by ‘traditional’ development approaches and can experiment using other innovative approaches.  PEG fundamentally challenged the prevailing development practice in Somalia, which relied heavily on, among other things, direct or subsidized service/good provision. On many occasions, PEG met with obstacles set up by these traditional development practices, but in most cases was able to break through these barriers.  One of PEG’s most successful interventions was the energy intervention, including the wind monitoring stations and the wind farm at the Hargeisa airport. Not only did this intervention achieve its immediate goals, but it has also laid the groundwork for a significant expansion of wind-generated energy in Somaliland.  Interventions in the agriculture and livestock sub-activities were largely successful within their (limited) scope. Producing improved the practices and performance of different market actors, both public and private. Although relatively small in scale, a number of these interventions have laid the groundwork for building on their success and the market structures and connections they have left in place.  PEG has laid the foundation in the agriculture and livestock markets for additional interventions based on sound M4P principles that can build on PEG’s work to generate large scale and sustainable improvements in the relevant market systems. Lessons learned The primary lessons learned are summarized below. Some of the lessons learned reflect recommendations made earlier in this Executive Summary.  Development stakeholders in Somalia are open to new development approaches that do not rely on traditional development practices, although it may require persistence, patience, education, and adherence to sound principles and practices to overcome initial skepticism, mistrust, or other obstacles.  Market actors are open to improved ways of doing business; in fact, there exists substantial demand for such things as organizational strengthening; adoption of new practices, technologies or business models; and forming market connections. Government ministries also want to improve their role in promoting and strengthening private sector activity.  Building relationships with public and private sector counterparts is critical both to generate support for the program and its interventions and to facilitate implementation.  Transparency in awarding contracts and grants and in implementing interventions builds trust and goodwill, as does using participatory implementation methods.  Successful interventions combine in-depth due diligence with flexibility during implementation. Sub￾contractors, beneficiaries, and other program stakeholders will appreciate the flexibility, which will also help build trust and goodwill.  Building a sound legal and regulatory framework is critical to strengthening the private sector. Public-private dialogues that involve multiple private and public sector stakeholders are a particularly effective method for facilitating legal and regulatory reform.  Both private and public sector entities need to have strong capacity to enable them to play their roles within the relevant market systems. Thus programs should allocate sufficient time and resources to develop local private and public sector capacities, including post assistance follow-up to increase the likelihood that any capacity and performance gains achieved are sustained.  Achieving broad-based improvements in market outcomes and in high-level Mission objectives require programs of sufficient size and outreach. Identifying and intervening in markets which have the greatest Page ix - PEG Performance Mid-Term Evaluation potential for achieving scale (such as the energy sector in Somaliland) is another effective means of achieving broad-based impacts. Applying lessons learned To apply the lessons learned from PEG’s experience described above, USAID/EA/Somalia should adopt an M4P approach to its PSD programming. It is worth noting that the M4P approach is a specialized application of a general USAID policy recommendation, as expressed in its 2010 US Global Development Policy: “We will also strive to help increase the capacity of our partners by investing in systemic solutions for service delivery, public administration and other government functions where sufficient capacity exists; a focus on sustainability and public sector capacity will be key to how the United States approaches humanitarian assistance and our pursuit of the objectives set out in the Millennium Development Goals.”1 As described in Annex 10, a distinguishing characteristic of the M4P approach is a systems approach to private sector development, with the aim of strengthening the functioning of market systems characterized by large numbers of small-scale farmers, pastoralists and MSEs. Or to put it simply, the M4P approach focuses on the system as a whole—the actors, their interrelationships and their guiding incentives. PEG implemented many elements of an M4P approach in its agriculture, livestock, energy, and business environment sub-activities, but it was hampered by the limited size and scope of its interventions from achieving any significant systemic changes in the relevant market systems. In the evaluators’ opinion, significantly expanding its PSD interventions using the M4P approach offers USAID/EA/Somalia the best opportunity for achieving the development objectives outlined above. 1 USAID, (2014), “Local Systems: A Framework for Supporting Sustained Development,” p. 3. PAGE 1 – PEG Mid-Term Performance Evaluation EVALUATION PURPOSE AND EVALUATION QUESTIONS MID-TERM EVALUATION PURPOSE This report presents the final mid-term performance evaluation of the Partnership for Economic Growth (PEG) program operating in Somaliland and Puntland State in Somalia. PEG is the first development initiative targeting economic growth for the USAID/EA/Somalia Office. The mid-term performance evaluation was requested by USAID under the Monitoring and Evaluation (M&E) Program for Somalia (MEPS) mechanism. It aims to inform USAID decisions regarding future economic growth programs in Somalia, including activities that fall under PEG’s contract extension. The USAID/EA/Somalia Office (hereafter USAID) is the primary audience for the evaluation, with secondary audiences including USAID/Washington and other program stakeholders, including Development Alternatives Inc. (DAI), Somali authorities, sub-contractors, donors, and other development organizations. EVALUATION QUESTIONS As specified in the evaluation Scope of Work (SOW, see Annex 2), the PEG mid-term performance evaluation seeks to answer the following five evaluation questions: 1. Has ‘The Partnership for Economic Growth Program’ achieved the objectives as prescribed in the program’s Performance Management Plan (PMP) and in the 2011 contract and subsequent three modifications? 2. Do the program’s results show that inclusive economic growth has been achieved by its activities? 3. Was the development hypothesis and accompanying assumptions that shaped the program design valid? And does it remain valid for the current Somalia and Somaliland context and USAID’s objectives? 4. How have PEG’S operational structures and implementation practices performed in Somalia’s challenging programming environment? How have they affected results? How can they be improved? 5. What have been the most important program accomplishments during implementation? What are the lessons learned from the program to date that can be applied to the program’s extension? How should these lessons be applied? PROGRAM BACKGROUND PROGRAM DESCRIPTION PEG was designed as a private sector development (PSD) program funded by USAID/EA/Somalia. PEG began on April 15, 2011 and on August 20, 2013 was granted a two-year cost extension, allowing program activities to run until August 31, 2015 and increasing the total estimated cost from $12.4 million to $20.4 million. PEG works closely with private sector businesses, government ministries, non-governmental organizations (NGOs), and civil society organizations (CSOs) to promote economic growth and stabilization in Somaliland and Puntland. PEG aims to help local authorities and private sector groups improve the enabling environment for investment and generate more productive employment. The program began in Somaliland and then expanded to Puntland in mid-2012. Program activities focus on two areas: private sector development, including women’s business development, and strengthening specific productive value chains. As a result of the comprehensive assessments conducted PAGE 2 – PEG Mid-Term Performance Evaluation during its first quarter (April-June 2011), PEG designed three sub-activities in the agriculture and livestock value chains and a set of cross cutting business environment (BE) activities. The energy component of PEG’s BE sub￾activity became its own sub-activity during the program’s fourth quarter (January-March 2012), and the program launched the livestock sub-activity in Puntland during its sixth quarter (July-September 2012). PEG also includes an activity that provided matching grants to promising local small and medium enterprises (SMEs) and to NGOs: “the Partnership Fund.” All grants required some level of cost sharing by partners, in the form of quantifiable in￾kind contributions. Through August 2013, PEG awarded 13 grants with a total partnership contribution of $911,885. DAI had sole implementation responsibility and oversight for the program. In this role, it worked with sub￾contractors on different aspects of the program, including the Citizen’s Network for Foreign Affairs (CNFA), which is the international subcontractor responsible for the livestock sub-activity and the Somali Agricultural Technical Group (SATG), which was responsible for the agriculture sub-activity. PEG’s primary sub-contractors are listed in Table 1 below. TABLE 1. PEG INTERVENTIONS COVERED BY THE MID-TERM PERFORMANCE EVALUATION Intervention Location Implementing Partners Agriculture Sub-Activity Horticulture Amoud, Baki, Ruqi, Somaliland SATG Ministry of Agriculture Amoud University Livestock Sub-Activity Fodder Production Somaliland (various locations) CNFA Candlelight for Health, Education & Environment Community Animal Health Workers (CAHW) Standardized Curriculum Somaliland (various locations) CNFA Somaliland Veterinary Board CAHW & Vet/Pharmacist Training Somaliland (various locations) CNFA VetAid CAHW & Vet/Pharmacist Training Garowe, Puntland Kaalo Aid and Development Organization Vet Laboratory Garowe, Puntland Kisima Peace & Development Organization Ministry of Livestock and Animal Husbandry Energy Sub-Activity Airport Wind Farm Hargeisa, Somaliland Ministry of Energy Ministry of Aviation Golis Energy Company Somaliland Energy Law Hargeisa, Somaliland Ministry of Energy Business Environment Sub-Activity Somaliland Investment Guide Hargeisa, Somaliland Ministries of Commerce, Fisheries, Agriculture, Livestock, Planning, Foreign Affairs, and Energy Diaspora Agency Chamber of Commerce Technical Specialists in: Agriculture, Energy, Livestock, Fisheries, and Salt Partnership Fund Horumar Camel Dairy Milk Farm Burao, Togdheer, Somaliland N/A Red Sea Fishing and Shrimp El Sheikh, Bulahar, Somaliland N/A Tayo Energy Enterprises Company Berbera, Somaliland N/A Kaaba MicroFinance Institution Gabile, Somaliland N/A Shaqodoon Youth Employment Hargeisa, Somaliland N/A PAGE 3 – PEG Mid-Term Performance Evaluation PROGRAM PURPOSE, OBJECTIVE AND DEVELOPMENT HYPOTHESIS PEG’s primary objectives are as follows: Purpose: PEG aims to help local authorities and private sector groups improve the enabling environment for investment in order to generate more productive employment and improve other livelihood activities. Objective: PEG aims to improve stability through inclusive economic growth. Development Hypothesis: Because so many external factors outside of PEG’s control contribute to instability, the sequential and unidirectional progression of achievement from outputs to outcomes to impact does not always take place. The iterative development approach should, to a significant degree, enable PEG to be flexible – to respond to variances in the geography where the program will work, as well as to the changing environment of development in those areas. In addition to the above, PEG has articulated two discrete, yet inter-related, intermediate results (IRs). IR1: Private Sector Entities Strengthened. To strengthen the environment for private sector development, the public sector authorities in each targeted region of Somalia must not only actively participate in assessing and analyzing technical areas and baseline data, but must also build positive working relationships with private sector entities. As more private sector entities are engaged in activities that build their capacity, increase their access to services and resources, expand employment, and develop a positive relationship with relevant public sector actors to improve policies and regulations, the resulting stability will encourage further economic growth in the form of more investment and trade. IR2: Priority Value Chains Improved. Assessments and improved data collection in the private sector and key value chains, with the participation of government authorities and other stakeholders in analysis of data and joint prioritization of activities, lay the foundation for IR2’s achievement. As capacities are strengthened for different value chain actors, they will increase their productivity, sales, and income and expand employment opportunities. Furthermore, as a result of increased income and employment in targeted value chains, stability should improve among vulnerable populations and within affected geographic areas. PEG AS DEMONSTRATION PROGRAM Promoting economic strengthening as a means of promoting stability and growth in Somalia is a new approach. PEG consists, in effect, of a series of pilot interventions in a variety of important economic sectors intended to test the feasibility and results of the PSD (generally) and value chain (specifically) approaches in Somalia. According to USAID’s Microenterprise Development Office, value chain programs aim “to drive economic growth with poverty reduction through the integration of large numbers of micro and small enterprises (MSEs) into increasingly competitive value chains. By influencing the structures, systems and relationships that define the value chain, USAID helps MSEs to improve (or upgrade) their products and processes, and thereby contribute to and benefit from the chain’s competitiveness.” 2 As typically used in USAID documents, the value chain approach is more or less akin to the Making Markets Work for the Poor (M4P) approach. Within a value chain/M4P (hereafter M4P) approach, the program’s role is primarily catalytic, to enable others to be self-supporting by stimulating changes in the market system without becoming a part of it.3 (See Annex 10 for more on the M4P approach.)4 It is hoped that the lessons learned 2 See http://www.microlinks.org/good-practice-center/value-chain-wiki. 3 See, for example, “A Synthesis of the Making Markets Work for the Poor (M4P) Approach,” www.deza.admin.ch/ressources/resource_en_172765.pdf. PAGE 4 – PEG Mid-Term Performance Evaluation from PEG will contribute directly to the legitimacy of the M4P approach within the Somali context and its subsequent replication and expansion. Thus, an important indicator of success for PEG is the extent to which it has produced this expected ‘demonstration effect.’ EVALUATION METHODS, TEAM AND LIMITATIONS EVALUATION METHODS The mid-term evaluation work took place over five distinct phases, including (1) desk review; (2) fieldwork in Somaliland and Puntland; (3) analysis, synthesis, briefing, and reporting; (4) additional field work to prepare the revised Evaluation Report; and (5) preparation of the Final Report based on comments received and new data collected. A teleconference presentation of the initial evaluation findings took place on April 9, 2014 followed by the submission of a draft Evaluation Report on May 5, 2014, and then the submission of a revised Evaluation Report on July 14, 2014. The PEG mid-term evaluation used a qualitative approach consisting of three primary data collection methods: (1) document review, (2) 33 key informant interviews (KIIs) with diverse program stakeholders and (3) 13 focus group discussions (FGDs) with program beneficiaries and their spouses. A triangulated analysis of the data derived from these three data sources is presented in this report. A bibliography of documents reviewed is presented in Annex 3 to this report, while a list of KII and FGD participants, along with their institutional affiliations (where relevant) are presented in Annexes 4 and 5. KII and FGD participants were selected in consultation with USAID, PEG, and/or other program stakeholders, as relevant. Both used discussion guides consisting of a set of pre-determined questions derived from the five key evaluation questions as befitting the respondent’s involvement and/or exposure to PEG efforts or specific economic growth activities. FGDs typically comprised between 8-12 individuals of the same sex, age group, and community, although a small number of mixed-sex FGDs were also conducted. Participants were recruited from the main target group based on a list of beneficiaries provided by DAI. Research Solutions Africa (RSA), the evaluation team’s local research partner, conducted all FGDs. Once fieldwork began, the Horn of Africa Training Institute (HATI) worked with the evaluation team to supervise/monitor the implementation of the KIIs and FGDs so as to ensure quality and consistency. Due to the nature of Somali local dialects, specialists who spoke the dialect for a specific area facilitated the FGDs there; note-takers who also spoke the dialect were assigned to each FGD wherever possible. To analyze the qualitative data, the evaluation team produced detailed summaries of the documents reviewed and the KII and FGD transcripts. With these summaries to hand, the team then went through the summaries looking for recurrent themes and sub-themes emerging from the data related to the five evaluation questions and, when identifying these themes and sub-themes, going back to the data to break them down and identify the relevant pieces of information. Through an iterative process, the team arrived at a final set of themes and sub￾themes and assembled the evidence supporting them, organized by each of the five evaluation questions. The findings, conclusions and recommendations presented in this report draw on these identified themes. If a 4 USAID resources and documents frequently refer to a ‘value chain’ approach to market development, which is more or less the same thing as the M4P approach. Because M4P is the standard term used in the development community outside of USAID, it is likewise used in this report. PAGE 5 – PEG Mid-Term Performance Evaluation particular comment or observation was made by a single or small set of respondents and/or was not connected to a common theme emerging from the data, it was not used to draw conclusions. Certain comments or observations may have been made by a smaller sub-set of respondents, but if they reflected an important sub￾theme, they are used in this report. To sum up, the evidence upon which this report draws reflects a theme or sub-theme identified by a preponderance of KII and/or FGD respondents. EVALUATION TEAM The core evaluation team consisted of an international consultant Team Leader, a Somali-speaking Economic Growth Expert, the local research partner (RSA), and a Senior Technical Advisor (STA). The evaluation team worked closely with IBTCI/MEPS staff to ensure that the FGD methodology followed the overall evaluation design. The IBTCI Home Office provided off-site assistance to the evaluation team, particularly in terms of staffing, planning, logistics, and publication support in finalizing the evaluation report. KNOWN LIMITATIONS TO THE EVALUATION METHODOLOGY The evaluation methodology has a number of limitations that have important implications for the types of findings and conclusions that can be drawn from the evaluation, as discussed below. Attribution: The qualitative nature of the evaluation—which in turn involved small sample sizes and the absence of comparison measurements—means that the evaluation methodology is not able to generate sufficient evidence to attribute observed results to PEG’s interventions. At the same time, however, the evaluation methodology seeks to mitigate this limitation through the triangulation of data and by drawing on multiple sources of data to reach informed conclusions about the program’s likely ‘contribution’ to observed outcomes. Aside from the attribution question, the evaluation methodology answers these other evaluation questions with a reasonably high level of credibility. Sampling: Owing to time and resource constraints, the evaluation team was not able to cover all of PEG’s interventions and instead evaluated a sample of interventions (see Table 1). While the evaluation team worked with USAID and PEG to identify a set of ‘representative’ interventions, these may not give a fully comprehensive picture of the program, its interventions, and its results. Indirect Effects: The evaluation methodology and timeframe meant that evaluators were obliged to focus primarily on the program’s direct beneficiaries, defined as those farmers, pastoralists, MSEs, and sub-contractors, who participated directly in and benefited directly from program interventions. At the same time, evaluators recognize that there are potentially farmers, pastoralists, MSEs, etc. who benefitted indirectly from program activities. In this case, the evaluation methodology did not allow us to determine what or how substantial these indirect benefits were. Sustainability: This mid-term performance evaluation was done while certain interventions were ongoing or shortly after other interventions had finished, which made it difficult for evaluators to determine whether observed outcomes have endured or are likely to endure once program assistance has ended. In the evaluators’ experience, however, many positive outcomes do not in fact endure for long after a program concludes, for a variety of reasons. The team tried to find evidence suggesting whether certain interventions are likely to endure after the program ends, drawing both on our evaluation findings and our previous experience evaluating other PSD programs. Subjective Measurements: A final implication of the qualitative approach used is that the measurement of results largely consists of subjective perceptions of KII and FGD participants. Thus, the mid-term evaluation lacked specific quantitative measurements of certain results, such as changes in stability, knowledge and capacity acquisition, on-farm and enterprise performance, income, or employment. For specific quantitative program results, the findings that follow draw on the results against targets set in the program monitoring system. PAGE 6 – PEG Mid-Term Performance Evaluation FINDINGS, CONCLUSIONS & RECOMMENDATIONS This section presents the findings, conclusions, and recommendations from the PEG mid-term performance evaluation. The presentation of the findings, conclusions and recommendations is organized by the five evaluation questions listed above. Note that the quotes presented below to demonstrate the findings and conclusions are not exhaustive of all statements made by KII and FGD respondents, or all cases demonstrating a particular finding or conclusion but are instead intended to illustrate such statements and cases. Quotes and examples were selected to represent both different stakeholders and different perspectives. For the sake of brevity, however, no more than four quotes are provided to demonstrate a particular point. Additional quotes, where they are available, are provided in Annex 1. HAS PEG ACHIEVED THE OBJECTIVES AS PRESCRIBED IN THE PROGRAM’S PMP AND IN THE 2011 CONTACT AND THREE SUBSEQUENT MODIFICATIONS? FINDINGS Evaluators have reviewed PEG documents together with USAID/EA/Somalia documents and web content and were unable to find a single, clear and operationalized definition of ‘stability.’ To help inform our understanding of this concept, evaluators reviewed other US State Department and USAID documents on the topic. USAID Afghanistan defines stability as ‘a reduction in the means and motivations for violent conflict, increased capacity to resist sudden change or deterioration and socioeconomic predictability.” The US Department of State’s Strategic Plan for 2014-2017 says that ‘regional stability must be built on a foundation of responsive and accountable governance, inclusive economic growth, and rights-respecting and capable security institutions.” Finally, according to the USAID Fragile States Strategy, “Research indicates that the instability associated with fragile states is the production of ineffective . . . governance. Effectiveness refers to the capability of the government to work with society to assure the provision of order and public goods and services.” Three themes emerge from the above defintions: (1) improved security (order), (2) good governance, including improved provision of public goods and services and (3) economic improvement. PEG is not a governance program, and it affects governance only indirectly through its work with public institutions to promote market development. PEG does, however, seek to achieve economic improvement through strengthening market systems and MSEs, and in this way can contribute to improved stability. Finally, there is a theoretical relationship between improved economic conditions and security, as the former is widely believed to reduce the incentive to engage in violent behavior. Thus the answer to this question frames it in the context of whether PEG is contributing to increased security and improved economic conditions. These are discussed in turn below. Increased Security This evaluation failed to find any evidence that PEG contributed to USAID’s aim of improving security through inclusive economic growth. This finding is not tantamount to saying that PEG made no definitive contribution to improved security in the relevant areas of Somaliland and Puntland, but instead that the methods used in the evaluation failed to find any evidence that it did. Evaluators identify at least two reasons for this finding: (1) the small scale of PEG interventions and (2) security concerns were a low priority among PEG beneficiaries. As an experimental pilot program, PEG was designed more to demonstrate the feasibility of the PSD approach through a series of smaller-scale and flexible market (value chain) interventions than it was to achieve a broader￾based impact on the outcomes of interest. As seen in Table 2, the direct outreach for four of PEG’s value chain PAGE 7 – PEG Mid-Term Performance Evaluation interventions in the agriculture and livestock sub-sectors was relatively small, totaling only an estimated 446 persons who were spread across a wide geographic area in Somaliland and Puntland.5 These findings reflect the experimental nature of the program agreed in advance by PEG and USAID/EA/Somalia. The implication of such shallow outreach, however, is that it made it very difficult for the program to achieve any kind of broad-based impact on security or on other higher-level objectives. TABLE 2. OUTREACH OF SELECTED PEG VALUE CHAIN INTERVENTIONS Intervention Outreach Horticulture 81 farmers, 8 extension agents, 25 traders Fodder production 180 farmers Community Animal Health Workers (CAHW) & vet/pharmacist training, Somaliland 94 CAHWs, 12 vets/pharmacists CAHW & vet/pharmacist training, Puntland 40 CAHWs, 6 vets/pharmacists Total 446 Almost none of the key informants interviewed or beneficiaries participating in FGDs felt that security was a substantial problem at the time they were interviewed, nor did they feel that it had either improved or disimproved over the last two years. Below are some examples of their statements on this subject. (See Annex 1 for additional quotes.)  “Security is very good, our people like each other.” (Agriculture Beneficiary, Somaliland)  “I don’t see much improvement with security; it’s as good as it was.” (Youth, Somaliland)  “The security situation is very good. There might be insecure regions, but in general the places we come from are peaceful.” (VetAid Beneficiary, Somaliland)  “Over the last two years, it has been very calm.” (Livestock Spouse, Somaliland) Overall, FGD participants tended to be far more concerned with livelihood issues, particularly related to livestock and the lack of employment or income-generating opportunities than they were with security issues. The FGD findings are supported by the PEG baseline survey, where 82.6 percent of respondents said that the government’s ability to provide security had improved over the past year, while 58.4 percent of respondents reported improvements in the government’s ability to resolve disputes. More generally, 82.2 percent of respondents saw improvements in stability over the past year, while only 5.6 percent perceived worsening stability and 11.1 percent perceived no change. Improved Economic Conditions—Private Sector Entities Strenthened The evaluation found that PEG contributed to strengthening both private and public sector entities benefitting from PEG’s value chain interventions. Key informants consistently said that participation in PEG interventions were effective in improving their knowledge and/or capacity. (See Annex 1 for additional quotes.)  “Technical assistance has proved very useful because farmers didn’t have the knowledge of best farming practices . . . . They used to plant the same crop on the same plot of land over and over. They wrongly used pesticides. I was always in the field with the farmers, that is why I felt it was very useful.” (Amoud University, Somaliland) 5 The total probably undercounts the actual number of direct benecifiaries, including an unknown number of other market actors involved in the fodder intervention. PAGE 8 – PEG Mid-Term Performance Evaluation  “Before, I did not believe it was very useful, because I thought farmers have more than 30-40 years of experience farming. Now I feel it was very useful . . . . I was gaining practical experience farming and taking care of my plot and I have also learned . . . the value of the crops in the market.” (Horiculture farmer/Extension agent, Somaliland)  “We gained more benefits from the training because we initially didn’t know these techniques. The training . . . encouraged us to continue farming and at the same time raise our livestock.” (Fodder farmer, Somaliland)  “I learned more knowledge during the training. I went back to my old farm and [applied] the new techniques I learned from the training. Today my farm is among the best farm in the village.” (Fodder farmer, Somaliland) Among the Partnership Fund sub-grantees, evaluators found that program assistance resulted in substantial improvements in knowledge and capacity. In support of this finding, PEG recorded a 31% improvement in sub￾grantee organizational capacity (compared to a target of 14%) over 12 months as measured by the Business Organization Capacity Assessment Tool (BOCAT). (Evaluators were not able to verify the BOCAT scores.) With regard to public sector authorities actively participating in the interventions and building positive working relationships with private sector entities (IR1), evaluators found that both of these occurred consistently in each of the program sub-activities. Working with and through public sector authorities was a clear hallmark of PEG’s intervention strategy, whether this included working with the Ministry of Agriculture to draft the Seed Testing and Certification Policy in the agriculture sub-activity, working with the Somaliland Ministry of Livestock’s Veterinary Board to develop a standardized CAHW curriculum in the livestock sub-activity, or working with the Ministry of Livestock and Animal Husbandry (MoLAH) in Puntland to strengthen vertical market connections in the livestock sub-activity. Improved Economic Conditions—Priority Value Chains Improved Evaluators found that PEG’s interventions contributed to strengthening the livestock and horticulture value chains. Key actors within these two value chains have improved knowledge and capacity as a result of PEG’s interventions and the weight of evidence (including that cited above) suggests that they are, on balance, performing their roles within the value chain more effectively. Evaluators also found that PEG has contribued to some improvments in the agriculture, livestock and energy market systems, though it will be a while before these market systems function smoothly. One way PEG sought to strengthen the relevant market systems was by forging commercial connections between market actors at different levels in the horticulture, livestock, and fodder value chains. In the horticulture value chain, PEG (via Amoud University) facilitated commercial connections between horticulture producers and horticulture vendors. According to Mohamed Heebaan, Professor of Agriculture at Amoud University, one of the biggest achievements in this intervention was to connect the farmers with the horticulture vendors. Before the intervention, farmers had no idea about existing market demand but now their direct contact with the horticulture vendors has allowed them to respond to the demand and supply of the market, as well as time the harvesting of specific crops to high season. PEG also helped connect Amoud University and the agro-dealer Agrofafa, which sources seeds from international seed and input suppliers from East Africa, including Simlaw and East-West Seeds. In the livestock value chain, PEG connected CAHWs in Somaliland and Puntland to pharmacies and the pharmacies to established, reputable drug wholesalers. In Somaliland, these market connection activities focused on the market for veterinary services, which were supported by a Livestock Awareness Campaign (hosted by the Veterinarian Board and Awke Media) targeting both the supply and demand side of veterinary services by providing drug information to service providers and consumers alike. Also in Somaliland, where no veterinary drug testing lab existed, a partnership between PEG and VetAid provided clear technical criteria for drug importers to guarantee the quality of drugs including: expiration dates, sufficient availability, selection from countries with strong certification systems, and availability of syringes and other veterinary equipment. In Puntland, PEG facilitated connections between the pharmacies and wholesalers with assistance of the MoLAH. PAGE 9 – PEG Mid-Term Performance Evaluation The pharmacies and CAHWs have access to the MoLAH’s veterinary officers and Puntland veterinary board to certify the quality of the drugs and the distributing stores. At this point, the established connections remain active. Finally, in the fodder value chain, PEG brought together farmers and fodder value chain players from various locations—including farmers, livestock herders, transporters, and livestock producers - in workshops and at the six fodder demonstration plots. Evaluators also found that improvements in the knowledge and capacity among private and public sector entities contributed to improvements in on-farm and small enterprise outcomes, such as the adoption of improved practices and increased production, sales, income, and employment. While the evidence for this finding is largely qualitative and cannot, for the most part, be verified quantitatively, it is consistent across all types of market actors interviewed. The following quotes are illustrative of this finding. (See Annex 1 for additional quotes.)  “We were in contact with farmers after some of the new technology was distributed amongst the farmers. Among them was a farmer who has produced twice as much tomato in his field . . . by adopting the recommended practices in his farm. This also applies to the farmers in Baki and Ruqi who produced a significant amount of watermelon and sold it to Djibouti. We saw truckloads of watermelon headed for Djibouti.” (SATG)  “I have my own farm in the village, and I tried my best to apply what I learned from the training. I have realized that preparing the farm in the way they trained us needs investment and more labor force. However, with the help of my son I managed to clear and prepare the farm . . . and I got more fodder than ever before.” (Fodder farmer)  “Farmers who benefited from this training have seen the difference. Today most of them have enough fodder for their livestock for the forthcoming dry seasons. Some of them are planning to sell heaps of fodder produced in their farms to generate income.” (Fodder farmer)  “The program has . . . increased my drugs stock and improved my finances. I am now able to buy more stock whenever I run short of these drugs. The drugs suit the diseases that exist in our country. In quality terms, the drugs are good.” (Pharmacist, Somaliland) The one case where this finding can be verified quantitatively is with horticulture producers where PEG kept careful records of yields for contract farmers. In this case, yields of contract horticulture products increased by 102%, significantly exceeding the target of approximately 20%, and, according to Amoud University, exceeding the yields achieved by average Awdal farmers. The sub-grantees under the Partnership Fund, in addition to Golis Energy, also reported substantial improvements in their business performance as a result of working with PEG, as the following quotes show.  “It will bring job opportunities. We have already hired guards, watchmen and drivers. Technicians will also be recruited when the operation starts.” (Tayo Energy)  “We moved from eight to 24 people working permanently as a result of the project; our trainees became our permanent staff.” (Golis Energy)  “The grant has ... brought continuity to my business. Before receiving the grant, we had 50 camels, and we now have 100. Plus we have doubled our employees from three to six, and we have hired other casual laborers to harvest and load fodder for the dry season. Our profit has doubled.” (Horumar Dairy)  “We placed 40 youths as a result of the training supported by the PEG grant, of which 19 were female and 21 were male. Another 18 youths found jobs at the trade fair, of which eight were female. We did a tracer study to follow up on trade fair participants and found another 125 youths who attended the fair were placed.” (Shaqodoon) Of all PEG’s interventions, the evaluation found that the interventions in the energy and BE sub-activities related to electricity production and the Energy Law have the biggest potential to improve a priority value chain. Evaluators heard from multiple sources that the lack of reliable access to reasonably priced energy is one of the PAGE 10 – PEG Mid-Term Performance Evaluation greatest obstacles to economic growth in Somaliland. The current patchwork of independent power producers (IPPs) constitutes an unsafe, inefficient, costly and unreliable source of electricity. Therefore the widespread adoption of wind-generated electricity has the potential to transform the energy value chain. At the same time, if the Energy Law, to which PEG made a significant contribution, is ultimately approved in a satisfactory form, this too could have a substantial impact on the energy value chain and on the Somaliland economy in general. Unfortunately, at this time, the Energy Law is in limbo owing to adverse changes made to the law by the Ministry of Energy.6 Although no evidence was found that PEG’s interventions have contributed to broad-based improvements in the respective value chains, there remains a chance that they could. Whether they do depends on the extent to which demonstration effects produced by the interventions lead to their replication and expansion. Evaluators found evidence that some demonstration effects are already occurring, particularly in the energy value chain.  The World Bank held a business plan competition in Somaliland and received approximately 12 wind farm proposals, of which it funded five. At least four of these wind farms are due to come online during 2014.  The Somaliland Wind and Solar Power Producers Association (SWSPPA) ordered and paid for wind power generation equipment of utility grade/scale of more than 2.0 megawatts, which amounts to more than 10% of the total electric power production in Somaliland. By 2014, it anticipates generating 15% of Somaliland’s electricity from wind and solar power, and it plans to invest heavily in wind technology at the same rate every year.  Transparency Solutions, an investment advisory firm that was a PEG Partnership Fund grantee, stated its intention to work with several of its committed investors to invest in wind energy.  The Director of the Energy Unit in the Somaliland Ministry of Minerals and Natural Resources has pledged to become a ‘leading pioneer’ in the energy sector by promoting investment in wind energy using, in part, the data generated by the wind monitoring stations installed with PEG support. Demonstration effects have also been observed in the livestock and dairy sectors.  Several development organizations working in the Somali livestock sector participated in developing and validating the standardized CAHW training curriculum—including International Relief, FAO, Terra Nouva, Oxfam, Sheikh Technical Veterinary School and Havayocco. These organizations support the Somaliland Ministry of Livestock’s policy to use the standardized curriculum for future CAHW trainings. The Ministry of Livestock and its Veterinary Board have since confirmed that the German development organization Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) provided funding for a refresher training course throughout the Sahil region using the Standardized CAHW training at the request of the Ministry of Livestock. At total of 50 CAHWs participated in the GIZ training.  In the Puntland livestock value chain, there are signs of independent business expansion on the heels of PEG’s intervention. For example, some CAHWs are investing in new pharmacies in Bayra village, Mudug Region after acquiring licenses from the MoLAH to expand their business, while another investor has established a wholesale operation in Galkacyo to become both wholesaler and retailer in the region. 6 The Energy Law initially included provisions establishing an Independent Energy Commission. The Ministry of Energy, however, changed the wording by eliminating the word ‘Independent,’ which in turn made the law unacceptable to PEG and other energy sector stakeholders. Recently, however, there are indications that the Energy Law may be moving forward. In a meeting with DAI on August 6, 2014, the Minister of Energy said that the Energy Law will be presented and discussed at the next cabinet meeting during the week of August 10. The Minister expressed confidence that the Law would be approved for submission to Parliament within the course of this month. PAGE 11 – PEG Mid-Term Performance Evaluation Moreover, since the project, other veterinarians have established wholesale pharmacies in certain areas of Puntland, while the MoLAH continues to carry out quality control of drugs entering the market.  In late 2013, a €25m project, “Deegaankaagu Waa Noloshaada - Your Environment is Your Life,” was launched in Puntland to improve rangeland conditions and sustainable use of rangeland resources. The four￾year project is funded by the European Union (EU) and managed by the Ministry of Environment and Rural Development (MoERT), African Development Solutions (ADESO) and CARE International. During the PEG livestock value chain and rangeland activities, MoERT was constantly and directly engaged in the intervention planning, implementation, monitoring and evaluation. The Director of the Environment Department at MoERT stated that MoERT and ADESO would use the Kaalo’s gullies rehabilitation approach for the project.  In the dairy value chain, the owner of Horumar Dairy reports that “this project has also attracted other business people and many of them are now interested and even started similar projects.” Finally, other donors and government agencies are taking note of PEG’s operational structures and implementation practices and are looking to follow them.  The World Bank and CARE have each pledged to follow PEG’s example in terms of transparent/fair bidding and procurement practices.  According to Mohamed Suleiman, Director of the Somaliland Department of Planning and Statistics under PEG Phase 1, “We are asking donors to follow PEG’s example . . . because private sector development is sustainable . . . none of the community projects from the past 12 years are functioning.” CONCLUSIONS In drawing conclusions here and elsewhere in this report, the team relied on the largely qualitative evidence presented as findings. Drawing firm conclusions about program results based on qualitative data alone is always tenuous. That said, the responses provided by KII and FGD respondents consistently spoke about the quality and benefits of program assistance in positive terms. Any exceptions to this were few and far between. Improved security: The team concluded that PEG did not make a substantial contribution to improved security in Somaliland or Puntland. The program scale/outreach was too small for it to have a substantial impact on security. On top of this, there is no qualitative evidence that security is a pertinent issue among the PEG beneficiaries; the vast majority of beneficiaries felt safe and secure and in any case had confidence that either village Elders or police were capable of dealing with any conflicts that did arise. Private sector entities strengthened: PEG’s interventions were effective in strengthening the knowledge and capacity of both private and public sector entities. Private and public stakeholers interviewed for this evaluation consistently stated that PEG training/capacity development was effective and useful in increasing their knowledge and improving their ability to carry out their respective roles and responsibilities. This conclusion is corroborated by the large improvement in BOCAT scores achieved by sub-grantee over a 12-month period. Priority value chains improved: PEG was effective in improving the performance of assisted value chain actors. Where specific numbers are available from PEG’s performance monitoring results, they fully support this conclusion. Evaluators also conclude that participation in the program contributed to improved business results among sub-grantees. Furthermore, improvements in the functioning of the horticulture, livestock and fodder value chains can be noted. These improvements, however, are limited to the relatively small number of market actors who participated in program interventions. This conclusion is echoed in PEG’s own report of its intervention in the veterinary vaccine supply chain in Puntland. While first acknowledging that its experience “clearly demonstrates the viability of private vet drug business in Puntland,” it goes on to conclude that it is “perhaps too early to expect the private sector to look at livestock vaccines as a profitable enterprise. Policy changes at donor and at PAGE 12 – PEG Mid-Term Performance Evaluation MoLAH level are necessary and should be vigorously supported before introducing or testing any form of private livestock vaccine supply in Puntland.”7 A similar point is made in the following quotes by Ahmed Ibrahim Awale, Executive Director of Candlelight, and by PEG, in the latter case referring to the CAHW and vet/pharmacy intervention in Somaliland:  “The project aimed to change a centuries-old system, hence those involved didn’t take up the techniques to create added value or businesses. Apart from the connections created during the meetings and workshops, there is no significant value added as a result of the project. It would require a much bigger and more sustained project to yield an added value chain.” (Candlelight)  “As a pilot project covering a huge swath of northern Somalia, we know some of this vision might not necessarily be accomplished in all areas where PEG operated, but we believe it certainly lays the foundation in some areas. In more densely populated regions where there are more educated and accountable regional vets (such as Togdheer) – that can work with both the trained CAHWs and vet pharmacies on a more frequent basis, there is a higher probability of such sustainable commercial relationships. However, in more sparsely populated areas (such as Sanaag), this may not be as easy.” (PEG)  “There are complex embedded community and value chain relationships that must be established to ensure sustainability. DAI believes it laid the foundation of an animal health and veterinary drug value chain activity that was grounded in sound M4P principles and implemented by some of the most experienced partners (such as Vet Aid). However, the activities and relationships that were built can only be considered pilot activities that would need more time, resources, and market facilitation activities to ensure sound sustainability by project end.” (PEG) A potential exception to this conclusion is the CAHW standard curriculum, which could potentially have a more substantial, broader-based impact on the livestock value chain given that it is an intervention that can be scaled￾up quickly or over time with a reasonable amount of investment and effort. This result, in fact, appears to be developing, as described above. A further caveat to the above conclusions is that evaluators are not in a position to assess the long-term sustainability of each intervention. As discussed below, there are reasons to be both optimistic or less optimistic about sustainability, depending on the situation. Finally, there is real potential for substantial demonstration effects to emerge in the energy value chain. PEG’s energy interventions have caught the attention of important market actors; some of whom have already made large investments in the sector and others who are contemplating making investments in the sector. If the Energy Law is ever approved in something like its original form, this too could have a substantial impact on the energy sector. Evidence of demonstration effects is also emerging in the livestock intervention in Puntland, which appear to have the potential to make meaningful improvements to the livestock value chain there, although not nearly as broad-based as in the energy value chain. Improved Stability: Taking all of the above into account, the evaluation concludes that program interventions did not make significant contributions to improved stability in Somaliland or Puntland, whether defined as increased security or improved economic conditions. However, interventions in Puntland appear to have contribued to small-scale improvements in capacity and performance in priority value chains with potential to achieve larger-scale improvements via emerging demonstration effects. RECOMMENDATIONS 7 USAID Partnership for Economic Growth, (2013), “Livestock Vaccines for Puntland: Improvement of the Veterinary Vaccine Supply Chain.” PAGE 13 – PEG Mid-Term Performance Evaluation Based on the above findings and conclusions, evaluators offer the following recommendations:  If PSG programming aims to achieve broad-based change, USAID/EA/Somalia should fund PSD programs with the appropriate scale and over a long enough time period to achieve this objective. PEG successfully demonstrated the feasibility of the PSD approach in Somalia; therefore PSD programs in Somalia need not take such a limited, experimental approach but should instead aim to achieve more broad-based impacts.  To achieve larger scale, and thus broader-based impact, USAID should consider adopting the M4P approach to private sector development.  USAID/EA/Somalia should continue to combine market (value chain) interventions with BE interventions, as the latter can be a particularly effective strategy for achieving scale and broad-based impact.  USAID/EA/Somalia should invest additional resources to expand the CAHW standard curriculum. Along with expanding the curriculum, additional resources would need to be invested in monitoring its implementation to ensure that its application (in addition to its content) is standardized as well. DO THE PROGRAM’S RESULTS SHOW THAT INCLUSIVE ECONOMIC GROWTH HAS BEEN ACHIEVED BY ITS ACTIVITIES? FINDINGS To answer this question, the evaluation defines ‘inclusion’ as the extent to which PEG’s interventions included and benefited members of traditionally excluded groups, with a focus on women, youth, and the poor, in addition to members of different clans and sub-clans. Overall, PEG prioritized inclusion in its intervention design and implementation, employing a variety of inclusion strategies targeted to each of the above four groups. Women: Integrating women was a high priority for PEG, and it made a concerted effort to involve women as both participants and final beneficiaries. PEG’s gender strategy in the agriculture and livestock sub-activities was guided by the agriculture value chain assessment, livestock value chain assessment and ‘Women in Business’ assessment, which were all part of the 2011 Somaliland Private Sector Assessment. In both the agriculture and livestock sub-activities, PEG took a broader market approach to strategically target other value chain actors, including horticulture vendors, which brought together mostly male farmers with mostly female vendors in the agriculture sub-activity, and veterinary pharmacies in the livestock sub-activity, which are often microenterprises run by women. In the Partnership Fund, selection criteria included a component “Impact on women and/or youth.” These criteria were shared with all applicants and distributed widely. In the energy and BE sub-activities, PEG made a strategic effort to incorporate women, especially in the design of the Energy Law public-private dialogues (PPDs) and the Investment Law validation workshops. Here PEG worked closely with Nagaad, the Somaliland national women’s network, during the PPD process. As a result of these targeting activities, PEG managed to include a significant share (greater than one-third) of women in several of its interventions, as the following examples illustrate.8 8 Although approximately one-half of Somaliland and Puntland residents are women, existing cultural and other barriers to women’s market participation, along with existing gender roles existing in certain sectors, make it, in the evaluators’ judgment, unrealistic to expect 50% or better female participation in all interventions. This report thus adopts a one-third female participation rate as evidence of significant female participation. Readers may form their own judgments as to whether a one-third female participation rate is significant. PAGE 14 – PEG Mid-Term Performance Evaluation  Six of the 13 sub-grantees—Togdheer Women’s Livestock Traders Association, Kaaba Microfinance, Shaqodoon, Dheeman Tailoring, Afjireh Gas, and Muliyo Salt—were strategically selected because they were owned and operated by women and/or targeted women and because they had a sound business plan.  Of 500 of youth trained by Shaqodoon, 230 were women. According to Mohamed Hassan Nur, the Executive Director of Shaqodoon, females were ‘clear and major’ targets for selecting participants. He goes on to say that since women are at a greater disadvantage in accessing information on job openings due to their limited mobility and absence from informal information hubs, Shaqodoon developed a website information center, which allows women better access to information than they would get from newspapers or other sources.  Kaaba Microfinance targets women with a female to male client ratio of 80%-20%. With grant funds provided by PEG, Kaaba reopened its Gabile office and provided loans and training to an additional 125 women. With the Gabile office now in place, Kaaba will be able to continue providing loans and training to an increasing number of women over time.  At least half of the participants in the fodder intervention were women. According to Ahmed Ibrahim Awale, Executive Director of Candlelight, “Although women are in the forage business, Candlelight had, to some extent, targeted them for the project for representation purposes.”  Seven of the 22 individuals trained and hired by Golis Energy were women.  Two of six vets/pharmacists assisted by Kaalo to open pharmacies in Puntland were women. In other cases, however, female participation rates were lower than one-third.  Eighteen of 60 CAHWs participating in the VetAid training were women, while six of 27 CAHWs participating in the Kaalo training were women. PEG allowed communities to select training participants, while also specifying that participants should include a more or less proportionate share of women. Nevertheless, the Village Elders favored male participants, resulting in a low share of female participants. As one CAHW described it, “The women participants are very low compared to the men. The selection was made by village and community Elders and you know they always discredit the women.”  In the agriculture sub-activity, approximately 16 of 81 contract farmers and four of eight extension agents were women. In this case, it should be noted that the female extension agents were the top female graduates of Amoud University whom the program strategically selected to break stereotypes regarding the role of educated women in the Somaliland agricultural sector. An additional 25 female horticulture vendors also participated in this sub-activity. Youth: PEG explicitly targeted youths (under 30 years of age) by providing a grant to Shaqodoon to support its youth job placement services, which it used to train an additional 500 youths and to host a job fair. Other interventions targeted or otherwise included a number of youth participants, including the horticulture intervention with Amoud University in which each of the eight extension agents were young graduates of Amoud University and the CAHW training in which a majority of CAHWs receiving training were youths. Nearly all of the 22 individuals trained and hired by Golis Energy were also under 30 years of age. Poor: According to the International Fund for Agriculture (IFAD), about 43% of Somalis live in extreme poverty, or on less than US$1 per day. This figure rises to 53% in rural areas where the poor depend heavily on livestock production or small-scale farming as sources of income. PEG’s interventions in the livestock and horticulture sectors were thus motivated in part to strengthen key markets on which poor households depend for their livelihoods. Finally, providing an accessible and low-cost source of electricity, as per the aim of the energy and BE sub-activities, would likely have a substantial impact on the poor, who are assumed to suffer disproportionately from access to affordable power sources. PAGE 15 – PEG Mid-Term Performance Evaluation Clans/sub-clans: According to PEG, achieving clan/sub-clan diversity is a top priority. Existing norms within Somalia, however, make it difficult to talk openly about ‘clan diversity’ in designing program interventions or in working with stakeholders. In view of this, PEG sought to achieve clan/sub-clan diversity by strategically selecting interventions and sub-grantees where different clans and sub-clans are based. CONCLUSIONS PEG prioritized the inclusion of women, youth, poor and different clans/sub-clans in its intervention strategy and implementation. In the process, it managed to achieve significant female representation in a number of interventions and sub-grants and, in the case of Shoqadoon, reach hundreds of youths with employment assistance. The actual share of different clans and sub-clans or poor among intervention participants and beneficiaries could not be determined, although PEG did consciously seek to achieve clan/sub-clan diversity and poverty inclusion through its geographic targeting strategy and its selection of sectors critical to the livelihoods of poor rural households. As to whether PEG’s interventions produced ‘inclusive economic growth,’ evaluators were not able to answer this question directly. Given the relatively small scale of program interventions, it is doubtful that the program interventions contributed to any broader-based ‘economic growth’, although evaluators did find evidence that interventions contributed to improved performance results for the farmers, CAHWs, pharmacies, enterprises, etc. directly benefitting from program activities. It is fair to assume that women benefited from these improvements according to the extent of their participation in the program. RECOMMENDATIONS Based on the above findings and conclusions, the following recommendations can be made:  PSD programs should include an explicit strategy for achieving inclusivity and creating inclusive impacts for each group targeted. It should ensure that this strategy is closely monitored to ensure adherence and to make adjustments given developments in the field.  As a strategy to promote inclusivity, PSD programs should consider the following approaches: o Target multiple levels in a particular value chain from the farm/firm level to end-consumers where the targeted group is active. Women, for example, often play defined roles within the production and post￾production process and/or within support markets (e.g., vendors), and interventions may seek to strengthen their capacity to play those roles. o Focus on key sectors that cut across social groups and locations as a means to benefit a broad range of social groups at both the farm/firm and household levels.  PSD programs that use participatory methods to select intervention participants should use well-articulated and agreed selection criteria, including criteria for selecting members of targeted social groups. WAS THE DEVELOPMENT HYPOTHESIS AND ACCOMPANYING ASSUMPTIONS THAT SHAPED THE PROGAM DESIGN VALID? AND DOES IT REMAIN VALID FOR THE CURRENT SOMALIA AND SOMALILAND CONTEXT AND USAID OBJECTIVES? FINDINGS Two important assumptions implied in the development hypothesis (please see above) are that a PSD approach will work in the Somali context, and that it is an effective means to generate increased stability as a result of improved production practices and increased returns to economic activities. With regard to the first of these assumptions, the PSD approach works well in the Somali context. As discussed in the previous section and in the following section, PEG successfully implemented its PSD strategy while adhering to high levels of professional standards that produced, by and large, positive results for participants. With regard to the second of these PAGE 16 – PEG Mid-Term Performance Evaluation assumptions, findings generally demonstrate that PEG’s interventions have contributed to improved practices/behaviors and improved performance outcomes. The evaluation team was not able, however, to find evidence to verify the causal links beyond these levels. With regard to the second sentence in the development hypotheses, the evaluation found that PEG did in fact use a flexible implementation and iterative approach that enabled it to adapt its interventions to variations in geography and other environmental factors. Praise for PEG’s flexible approach to intervention design and implementation was a common theme voiced by stakeholders in all sub-activities and by sub-grantees, as the following quotes illustrate. (See Annex 1 for additional quotes.)  “PEG was very flexible. PEG went out of its way to extend technical and operational support. We requested PEG to assist the farmers and vendors with cash and in kind grants, and they obliged—this was revised and included in the project.” (Amoud University)  “DAI was supportive and flexible; for example they modified our contract to include activities during the third planting season.” (Amound University)  “Program staff were very good with technical knowledge in veterinary science, had a good understanding of the livestock industry, and were flexible in working with us on all issues.” (VetAid)  “PEG consulted with us on every step, and it was flexible to minor changes. For example, PEG originally wanted to buy high quality medicine, so it cancelled the Chinese medicine that we had obtained for the project. It later relented and ended up ordering the medicine from another of our Chinese suppliers.” (VetAid) As for the rest of the development hypothesis, its first sentence is correct; all programs are subject to external forces outside their control, and the actual causal mechanisms at play in any case are rarely sequential and unidirectional. Moreover, most programs follow an iterative implemenation process. In its current form, the development hypothesis is too general and fails to articulate an actual hypothesis regarding how economic growth contributed to improved stability. CONCLUSIONS The development hypothesis was accurate when PEG was designed and it remains accurate today—up to a point. It accurately assumed that PSD interventions can successfully be implemented in the Somali context and that they contribute to improved outcomes, particularly improved practice and performance outcomes. Taking this conclusion further, PEG also demonstrated that a PSD approach of working through diverse private, public, NGO and CSO actors is not only possible in the existing Somali context, but that it also contributes to successful program implementation. None of the evidence found suggests that a PSD approach cannot work in Somalia. Evaluators further conclude that the development hypothesis is accurate in assuming that PEG would use a flexible and iterative approach and in stating that this flexible approach would contribute to improved outcomes. Evidence shows that PEG did in fact pursue a flexible implementation strategy that allowed it to adapt to circumstances it encountered in the field and that this flexible approach enabled it to improve its results, in addition to helping create a more collegial and collaborative relationship between PEG and its many sub￾contractors. One cannot, however, conclude that a PSD approach is an effective way to improve stability within Somalia, as evaluators were unable to verify whether PEG’s PSD interventions contributed to improved stability. It remains unclear whether this part of the development hypothesis is accurate. More broadly, as the development does not articulate a hypothesis as to how economic growth contributes to stability, evaluators were unable to assess whether the hypothesized causal mechanisms (intermediate outcomes) are accurate either. PAGE 17 – PEG Mid-Term Performance Evaluation RECOMMENDATIONS Based on the above findings and conclusions, evaluators offer the following recommendations:  In articulating development hypotheses, USAID/EA/Somalia should specify more clearly a theory of change describing how program interventions will contribute to the program Purpose and Goal.  If improved stability is to remain a principal mission goal and an objective of its PSD programming, USAID/EA/Somalia should verify just how important an issue stability is, what its causes are, how it manifests itself, and what its specific connections to economic growth are, and then to incorporate this knowledge into future PSD programming, program design, and work statements.  USAID/EA/Somalia should accept the viability of the PSD approach and focus on methods to replicate and expand it. PEG’s successes—particularly its ability to mobilize multiple stakeholders across the private, public, NGO and CSO sectors to work together on private sector strengthening—lays to rest any question whether a PSD approach is valid in the Somali context.  USAID/EA/Somalia should consider conducting a rigorous impact evaluation of a future PSD program to enable it to measure whether and how PSD programs contribute to improved stability or other high-level Mission objectives.  PSD programs should incorporate a significant degree of flexibility into program design and implementation given the dynamic nature of market systems where they work, given that actual program results, and all of the causal mechanisms leading to those results, are unlikely to be known in advance. HOW HAVE PEG’S OPERATIONAL STRUTURES AND IMPLEMENTATION PRACTICES PERFORMED IN SOMALIA’S CHALLENGING PROGRAMMING ENVIRONMENT? HOW HAVE THEY AFFECTED RESULTS? HOW CAN THEY BE IMPROVED? FINDINGS PEG’s operational structures and implementation practices performed consistently well in the Somali environment. Evaluators found no scenario in which PEG was not able to implement an intervention due to existing environmental barriers, but instead found that PEG was able to implement all of its interventions while adhering at all times to high operational standards and sound implementation practices. Six characteristics of PEG’s operational structures and implementation practices were identified that contribute to this finding: due diligence, transparency, participation, flexibility, sustainability, and other sound operational/implementation practices. Each of these is discussed in greater detail below. Due diligence: PEG consistently implemented a rigorous and in-depth due diligence process before launching an intervention. This practice helped ensure that PEG’s interventions were well-designed and reflected current market and environmental conditions. Examples of PEG’s due diligence activities that illustrate this finding include the following.  Prior to launching the vet lab intervention in Puntland, PEG’s Senior Veterinary Vaccine Cold Chain Specialist conducted an assessment of the existing cold chain and vaccine management practice, the functionality and capacity of existing equipment and the capability of available manpower in the vet drug supply chain. Prior to launching the CAHW and vet/pharmacy intervention in Somaliland, PEG conducted a baseline survey of 30 villages in Togdheer, Sool and Sanaag.  In the Partnership Fund, selection of the sub-grantees involved no fewer than five stages, including (1) wide distribution of request for proposals via radio, short message service (SMS), and newspapers; (2) scoring the 200 proposals received and ranking the top 36 during a two-day retreat including representatives from the PAGE 18 – PEG Mid-Term Performance Evaluation Somaliland Ministries of Commerce and Planning and the Chamber of Commerce; (3) site visits to the top rated 20 proposals; (4) additional analysis of the proposals’ technical criteria; and (5) further due diligence, including a review of the cost share proposal and following up on references.  During the inception period of the project, PEG (Candlelight) conducted a comprehensive baseline survey covering 17 pastoral and agro-pastoral villages in Togdheer to document the current status of fodder production, harvesting, baling and storage systems and select intervention sites. Transparency: PEG adhered to a strict policy of transparency in all of its interventions in each sub-activity and in the Partnership Fund. In doing so, PEG sought to demonstrate that adhering to strict standards of transparency could work in the Somali context where transparency was not common practice.  “Money was managed with transparency.” (Kisima)  “The terms of the grant were clearly stated in the agreement with PEG. PEG’s selection processes were also very transparent.” (Red Sea Fishing)  “The selection process was clean, fair and transparent, and the selection criteria were explained clearly in different fora.” (Tayo Energy)  “The selection process was fair and transparent; grantees were given clear directions, expectations, deliverables and timeframe.” (Shaqodoon) Participation: PEG followed a highly participatory approach with its sub-contractors and sub-grantees to plan and implement interventions. It also worked closely with sub-contractors, sub-grantees and community members during implantation. (See Annex 1 for additional quotes.)  “PEG’s main strength was its broad and comprehensive stakeholder engagement, which contributed to the sustainability of the project.” (Kaalo)  “PEG was always consulting with partners. Their approach minimized drawbacks, mainly as a result of consulting village committees. PEG created a harmonious working environment, increased vet capacity, increased livestock production and improved the environment by creating awareness, motivation and ownership by the community.” (Kaalo)  “The major strength of PEG was its community engagement in the implementation. The whole process of the planning and implementation of the activity was conducted in a participatory manner whereby the community has participated.” (Candlelight)  “There was a strong team effort between us and DAI. The foreigners were at times untrusting of us. However, since most people working for PEG spoke Somali, they were able to help us understand what was going on at all times and were always ready to smooth over relations and misunderstandings.” (Golis Energy). Another sign of PEG’s commitment to participation (and transparency) was the extent to which it brought together diverse actors from the private, public, NGO and CSO sectors to mobilize participation and support, share information, engage in discussion, or validate its work/findings.  Prior to launching its horticulture intervention, PEG held community mobilization workshops in Amoud, Baki and Ruqi attended by farmers, local government officials, and other community members.  PEG (working through Candlelight) carried out sensitization sessions aimed at introducing the benefits and expected outcomes of the fodder intervention to the participating communities, the role of Candlelight in delivering the project inputs, and on the communities’ role in internalizing the project concepts and skills and sharing the results with the other members of the community. PAGE 19 – PEG Mid-Term Performance Evaluation  After completing the draft of the Somaliland Investment Guide, the PEG BE team organized a technical discussion/validation workshop with stakeholders of the Somaliland Investment Guide Technical Advisory Committee, including representatives from the Ministries of Commerce, Planning, Agriculture, Livestock, Industry, Fisheries, and Energy, in addition to the Somaliland Diaspora Agency.  PEG and the Vet Board held a one-day validation workshop on the Standard Training Guidelines for CAHWs attended by more than 60 livestock sector stakeholders, including representatives from the Food and Agriculture Organization (FAO), Ministry of Livestock, Chamber of Commerce, Relief International, Candlelight, VetAid, private veterinarians, CAHWs, NGOs, pastoral associations, and so forth. The Somaliland Investment Guide and Somaliland Energy Law are two additional examples of successful participation. In both cases, PEG brought together multiple and diverse stakeholders, in some cases ones who had never been involved in such proceedings (e.g., IPPs) to take on complex and difficult tasks requiring substantial cooperation. In the case of the Energy Law, PEG brought together no fewer than 150 different stakeholders, including the Presidency, legislators, civil society, private sector, government ministries, etc. Flexibility: As discussed above, PEG also adopted a flexible approach and demonstrated on a number of occasions that it was prepared to adapt its implementation plans when faced with obstacles or in response to conditions on the ground. Sustainability: PEG performed less consistently in operational/implementation practices related to sustainability. In this case, the findings were mixed. In some cases, PEG incorporated, or tried to incorporate, features into its interventions promoting sustainability, while in other cases, PEG was less successful in promoting sustainability.9 One way that PEG helped to promote the sustainability of its interventions was by providing practical and in￾depth training/capacity development to its sub-contractors and sub-grantees that allowed them to carry on carrying out their support roles after PEG assistance ended.  “We are still in contact with trained staff and assisted pharmacies are still operating and requesting medicine. Sixty trained individuals are registered with the Ministry of Livestock and work with the Ministry during campaigns, provide quarantine services and other animal health related work.” (VetAid)  “PEG provided the technical expertise for assembling turbines and encouraged us to expand our workshop, which now builds, among other things, the turbine step ladder. Except for the manufacture of turbines, we can now provide all services required for wind turbine installation and maintenance.” (Golis Energy) PEG also sought to build in sustainability by designing interventions with the capability to generate revenues over time, sufficient to pay operational costs and produce a financial surplus. This was the case for the Hargeisa airport windfarm where the five wind turbines have the capacity to generate in excess of 102 kilowatts of electricity, far exceeding the kilowatts required to run the airport. The Ministry of Airports and Aviation will be able to sell the excess elecriticity and thereby generate income to continue running the windfarm into perpetuity, assuming no other problems, such as the lack of spare parts. The initial plan in the horticulture intervention was for Amoud University to produce and sell seeds and use the income to continue funding agricultural extension and other services to horticulture farmers. For a variety of reasons, PEG has since switched gears and is now working with Amoud University to produce and sell seedings, rather than seeds, to support the ongoing provision of extension and other services. PEG has worked with 9 Information about sustainability (as well as other program results) reported here is as of the time of the evaluation according to the information available to us. The situation may have changed between now and then or may yet change in the future. PAGE 20 – PEG Mid-Term Performance Evaluation Amoud University to develop a business/strategic plan for the seedling sales to become a cost recovery mechanism to pay for the extension workers on a full-time basis by the end of the PEG extension. Notwithstanding the above efforts to promote the sustainability of its value chain interventions, PEG’s operational structures and implementation practices created situations that at times threated the sustainability of its operations. Perhaps the most important of these was the relatively short training period provided to sub￾contractors and farmers. Concerns about the insufficient length of training was a consistent theme in both KIIs and FGDs. (See Annex 1 for additional quotes.)  “The training was of limited duration so that the impact was less than it could have been. The CAHWs need more training. Twenty days is not enough, they need at least six months. There was no follow-up once the activities were completed by Kaalo and reported.” (Kaalo)  “The weakness of the literacy and numeracy training was the limited time. We recommend 90 days of training.” (Amoud University)  “Refresher courses will be necessary in case of changes in medicines.” (VetAid)  “One thing I would have done differently is the training duration, which I see as not being enough to cover all required lessons of the two topics covered in this training. The total days were 10 days for all these lessons, each topic was five days, and as I said, we wanted more time to cover more about such important topics.” (Vet/pharmacist, Somaliland) A second factor that threatens the sustainability of PEG’s value chain interventions is related to the first and is connected to the commercial incentives that support providers — such as sub-contractors, vets/pharmacists, input suppliers — have to continue providing information or technical assistance to other support providers or to beneficiary farmers or enterprises. Behavioral changes are more sustainable if they are rooted in commercial incentives and expectations of greater financial returns. PEG has not established such commercial incentives within its priority value chains, meaning that provision of information and technical assistance is more likely to end once USAID support is withdrawn.10 In other cases, structures PEG had put in place to promote sustainability were not working as planned. In the livestock sector, for example, the commercial connections PEG had facilitated between CAHWs and pharmacists in Somaliland were not always functioning as well as hoped:11  “CAHWs have been attached to us, but I have not seen them since that day. They should have contacted us and bought the drugs from us at a discounted price.” (Pharmacist)  “I don’t see the CAHWs very often. Two of those attached to me came to see me a few times and I lent them the drugs. One of them paid me back but the other has never come back to me. I think that the plan has hardly worked for one reason or another.” (Pharmacist) In the fodder intervention, the sub-contractor identified a problem with PEG’s training approach that threatened the sustainability of a particular aspect of that intervention. 10 For example, in the case of the CAHW training, an M4P approach might involve establishing commercial incentives for vets/pharmacists to train the CAHWs. PEG, however, did not use this approach in this case because of the identified capacity gaps among the vets/pharmacists. Hence, CAHW trainings were conducted by VetAid in Somaliland and Kaalo in Puntland with the ministry staff involved to build the capacity of the vets/pharmacists to take on this role in the future. 11 “In some cases” here refers to each of the three vets/pharmacists interviewed during the MEPS verification exercise. It is unknown whether the remaining nine vets/pharmacists participting in this intervention have had a similar experience. PAGE 21 – PEG Mid-Term Performance Evaluation  “The forage bailing technique that was introduced by the project is in complete contrast to the prevalent practice on the market. The market uses a rope to measure a loosely packed forage, while the new bailing technique tightly compresses the fodder thereby increasing its weight and density. As a result, there was no incentive to use the new bailing technique as it is the size and not weight that matters in the market. Despite the new practice being more effective and efficient, without expanded support, there is not huge potential for sustainability of the new technique on the market.” (Candlelight) Procuring spare parts is a concern that has potentially important implications for the sustainability of the Hargeisa Airport wind-farm. In response to the difficulties encountered sourcing the wind turbines, PEG presumably established a five-year inventory of spare parts and, since future sourcing of spare parts will not be subject to the same USAID procurement rules, PEG does not anticipate substantial service interruptions due to the unavailability of spare parts or delays in sourcing them. According to Golis Energy, however, the quality of the parts and the frequency of their replacement remain concerns, and delays have already occurred due to issues with spare parts.12 Thus whether the availability of spare parts will affect the operation and sustainability of the Hargeisa wind-farm remains a concern, although again this outcome was due more to USAID’s strict procurement rules than it was due to PEG’s operational structures or implementation practices. Another value chain intervention that presents sustainability concerns is the vaccine storage in Puntland. Initially, PEG financed the construction of a large lockable vaccine storage room in Garowe to be equipped with solar powered deep freezers, but this plan was altered by Kisima in response to requests from the MoLAH. As a result of this, the larger room was subdivided into two corridor offices and the location of the deep freezers remained in the entrance hall of the building on the ground floor. Of the nine deep freezers, all had an inappropriate climate class specification, three were switched off, and one had no connection to a power supply. None of the deep freezers’ sockets, moreover, had a voltage regulator or voltage control. On top of this, the vaccines were being stored in a haphazard fashion outside the building and exposed to the elements, access to the deep freezers and vaccines was not secured, and the solar panels powering the freezer were already falling into a state of disrepair.13 Red Sea Fishing was an established, vertically integrated market actor operating at multiple levels in the fish value chain. To power the icemakers used to freeze and preserve fish for transport, Red Sea Fishing relied on the power provided by its sister feronite plant. This feronite plant subsequently shut down, and it was not cost effective to run the plant simply to power Red Sea’s icemakers. At the time of the evaluation, the feronite plant, and thus the ice makers, had not been operating for six months. Another sustainability concern related to the Red Sea sub-grant is the lack of boat mechanics to rehabilitate boat engines. After purchasing the new boats with the PEG grant money, a couple of boats broke down and at the time of the evaluation, Red Sea had the same number of boats operating in the water that it had before the grant. Other sound operating/implementation practices: This last characteristic is a general one that includes other aspects of PEG’s operational structures and implementation practices not covered above. Key informants among the different sub-activities and sub-grantees commented favorably on various other aspects of PEG’s operations and implementation. (See Annex 1 for additional quotes.)  “The PEG project staff were very good and had good technical knowledge in veterinary issues and a good understanding of the dynamics of the livestock industry.” (VetAid) 12 Unfortunately, Golis was not more specific than this. 13 It may be useful to note here that PEG concedes that this intervention has been its most challenging, particularly its relationship with the MoLAH. PAGE 22 – PEG Mid-Term Performance Evaluation  “The unique approach of this project was to provide people with initiatives and people who have been striving to get business going with encouragement, technical assistance and funds. Support to the private sector was the perfect strategy to reach PEG’s objectives.” (Kaalo)  “PEG’s activity plan was sequential, coherent and logical; operational, security and implementation strategies were highly effective, and the process for selecting beneficiaries was all inclusive.” (Candlelight)  “DAI’s approach was very well adapted to the local situation and environment. All activity plans were based on conditions on the ground, the quality of extension workers improved significantly. We observed a good working relationship between DAI and line ministries.” (SATG) PEG also held sub-contractors to high performance standards:  “DAI insisted on things to be implemented perfectly, so this created conflict sometimes whenever DAI felt things were not up to standard.” (VetAid) Finally, PEG enforced strict selection criteria when selecting the sites and participants for different interventions. For example, PEG used 10 selection criteria to select fodder demonstration farms and six selection criteria to select CAHW training participants. There were, however, cases where participants expressed dissatisfaction with different aspects of PEG’s operational structures and implementation practices. One example was the livestock intervention in Burao where some veterinary pharmacists felt that they were not sufficiently consulted in the selection process and were unhappy with the drugs that PEG selected.  “Despite all this, we were not asked our choice of the drugs that are in demand at the time. We were just taken to a wholesaler drug store and provided with packets of drugs. We could have been given the opportunity to choose animal drugs that sell best and are most in demand because we have been in this business for so long, and we know better.” (Pharmacist) VetAid also expressed dissatisfaction with PEG’s approach, noting that the process of selecting CAHWs to participate in the program underestimated the demand for the training and did not consider population and land size and as a result caused “chaos” within the communities. VetAid also stated that PEG did not consult with it ahead of time but asked it to implement a “pre-designed” activity. In addition, the intervention did not include sufficient field training, was not connected to PEG’s promotion activities, and selected medicines that were too expensive for typical livestock owners. Notwithstanding, VetAid acknowledged that the overall quality of the training was “very good.” It is also worth noting here that in all the interviews, and in reviewing the MEPS Verification Reports, there were few complaints about PEG related to its operations or implementation practices, other than those provided above. Several of the MEPS verification reports included a rating scale in which the participants rated PEG along a number of performance categories. Table 3 summarizes the findings. In each case, a higher score is evidence of higher performance. While the sample sizes are again small, the results confirm the overwhelmingly positive feedback from sub-contractors and sub-grantees. PAGE 23 – PEG Mid-Term Performance Evaluation TABLE 3. STAKEHOLDER RATINGS OF PEG PERFORMANCE IN MEPS VERIFICATION REPORTS Performance Dimension Scoring Range Average Score Number Responents Quality 1-4 3.7 6 Usefulness 1-5 4.0 7 Sustainability 1-5 4.2 6 Beneficiaries understand purpose 1-4 3.9 13 Relevance 1-4 4.0 14 Achieved goals 1-4 3.0 12 Beneficiary ownership 1-5 4.2 13 Follow-up steps clearly stated 1-4 3.7 9 CONCLUSIONS PEG’s operational structures and implementation practices performed well in the Somali programming environment. The soundness of these structures and practices played a key role in PEG’s success, both in terms of program implementation and results.  The extensive due diligence work undertaken by PEG prior to launching interventions helped ensure, on one hand, that each of PEG’s market interventions reflected the geographic, economic, social and other factors characterizing specific market systems and, on the other hand, that enterprises offering substantial potential for contributing to enhanced economic outcomes were selected for grant financing.  PEG’s commitment to transparency and participation demonstrated that such an approach works in Somalia. It helped foster an environment of trust and cooperation among the program’s sub-contractors, sub￾grantees and beneficiaries, and it purchased goodwill that helped smooth over any rough patches.  PEG was able to operate successfully in Somalia while adhering to strict procurement rules, holding sub￾contractors to high standards of performance.  PEG’s flexible implementation approach combined with its high quality training/TA assistance was widely appreciated by all stakeholders and contributed substantially to program achievements.  PEG took sustainability seriously and attempted to build it into interventions, although with mixed success. This evaluation also concludes that training/capacity development activities were probably not sufficient and could have benefitted from follow-up reinforcement efforts. VetAid aptly expresses this point within the context of the CAHW training: “When trainees are first trained they usually go under a process of transformation to get acquainted with their new job. This period (the first month or so) is the chronic period when the persons need a lot of help to get the courage to treat animals and sell the veterinary drugs and make the necessary records. The trainees should be visited at least once a week by trainers. In this project the immediate follow-up is always missing due to the lack of staff. It is essential to employ two veterinary assistants with motorbikes to carry out the follow-up and gather the necessary information from the field. VetAid and its supporters should make a long term commitment to remain involved in Somaliland.” (VetAid) Where PEG did encounter implementation challenges, it was able to deal with them and go on to implement its interventions successfully. While there was some dissatisfaction expressed by different stakeholders with PEG’s operating structures and implementation practices, these were the exceptions rather than the norm. On the whole, PEG had sound operational structures and implementation practices, which were instrumental in contributing to the many successful outcomes described by program stakeholders. RECOMMENDATIONS PAGE 24 – PEG Mid-Term Performance Evaluation Based on the above findings and conclusions, recommendations are found below. These recommendations summarize the answers to the question as to how PEG can improve its operational structures and implementation practices.  USAID/EA/Somalia should continue insisting that PSD programs always adhere to accepted best operational/implementation practices. PEG’s success demonstrates that there is no need to sacrifice sound operating/implementation practices in order to implement market interventions successfully. These best operational/implementation practices should include, as a minimum, strict adherence to the principles and practices of due diligence, transparency, participation, flexibility and sustainability.  PSD programs should include an explicit plan to achieve sustainability. Sustainability is more the exception than the norm, and unless PSD programs plan for sustainability and build it into their intervention design and implementation strategies, gains achieved are unlikely to survive much beyond the program.  PSD programs should budget and plan for sufficiently long training/capacity development activities, including after activity follow-up. The training/capacity development assistance provided by PEG was too short in many cases leaving significant capacity gaps remaining. WHAT HAVE BEEN THE MOST IMPORTANT PROGRAM ACCOMPLISHMENTS DURING IMPLEMENTATION? WHAT ARE LESSONS LEARNED FROM THE PROGRAM TO DATE THAT CAN BE APPLIED TO THE PROGRAM’S EXTENSION? HOW SHOULD THESE LESSONS BE APPLIED? This evaluation question represents a summary of the findings and conclusions from the other four evaluation questions. Thus, the team adopts a different format below to answer it, and instead of describing findings, conclusions and lessons learned, answers are given to each question in terms of Important Accomplishments, Lessons Learned, and Applying Lessons Learned. IMPORTANT ACCOMPLISHMENTS The most important program accomplishments are summarized below.  Some of the most important program accomplishments had to do with PEG’s internal operating structures and implementation practices that demonstrated the feasibility of its approach in the Somali context, including the following: o PEG demonstrated that a PSD approach using sound and transparent operational/implementation practices can succeed in Somalia. o PEG successfully brought together diverse market actors from multiple sectors with no previous history of working together to demonstrate that working collaboratively toward mutual benefits is feasible and can be an effective market development strategy. o PEG demonstrated that public sector officials are willing and can be engaged in market development activities. o More generally, PEG demonstrated that donors and other development organizations need not feel bound by ‘traditional’ development approaches and can experiment using other innovative approaches.  PEG fundamentally challenged the prevailing development practice in Somalia, which relied heavily on, among other things, direct or subsidized service/good provision, as reflected in this quote from a technical assistance expert working in PEG’s agricultural sub-activity, “The idea is new. It’s difficult for the farmers to accept the project as it is, because they were expecting some monetary incentives.” On many occasions, PEG met with obstacles created by these traditional development practices, but was able to break through these PAGE 25 – PEG Mid-Term Performance Evaluation barriers in most cases, as noted by the same technical assistance exerted quoted above: While PEG’s approach initially “diminished cooperation and trust . . . we overcame this through awareness.”  One of PEG’s most successful interventions was the energy intervention, including the wind monitoring stations and the wind farm at the Hargeisa airport. Not only did this intervention achieve its immediate objectives, but it has also laid the groundwork for a significant expansion of wind-generated energy in Somaliland.  Interventions in the agriculture and livestock sub-activities were largely successful within their (limited) scope. Producing was shown to improve the practices and performance of different market actors, both public and private. Although relatively small in scale, a number of these interventions have laid the groundwork for building on their success and the market structures and connections they have left in place.  PEG has laid the foundation in the agriculture and livestock markets for additional interventions based on sound M4P principles that can build on PEG’s work to generate large scale and sustainable improvements in the relevant market systems. LESSONS LEARNED The primary lessons learned are summarized below. Some of the lessons learned reflect recommendations made in previous sections.  Development stakeholders in Somalia are open to new development approaches that do not rely on traditional development practices, although it may require persistence, patience, education, and adherence to sound principles and practices to overcome initial skepticism, mistrust, or other obstacles.  Market actors are open to improved ways of doing business; in fact, there exists substantial demand for such things as organizational strengthening; adoption of new practices, technologies or business models; and forming market connections. There is also demand within government industries to improve their role in promoting and strengthening private sector activity.  Building relationships with public and private sector counterparts is critical both to generate support for the program and its interventions and to facilitate implementation.  Transparency in awarding contracts and grants and in implementing interventions builds trust and goodwill, as does using participatory implementation methods.  Successful interventions combine in-depth due diligence with flexibility during implementation. Sub￾contractors, beneficiaries, and other program stakeholders will appreciate the flexibility, which will also help build trust and goodwill.  Building a sound legal and regulatory framework is critical to strengthening the private sector. Public-private dialogues that involve multiple private and public sector stakeholders are a particularly effective method for facilitating legal and regulatory reform.  Both private and public sector entities need to have strong capacity to enable them to play their roles within the relevant market systems. Thus programs should allocate sufficient time and resources to develop local private and public sector capacities, including post assistance follow-up to increase the likelihood that any capacity and performance gains achieved are sustained.  Achieving broad-based improvements in market outcomes and in high-level Mission objectives requires programs of sufficient size and outreach. Identifying and intervening in markets which have the greatest potential for achieving scale (such as the energy sector in Somaliland) is another effective strategy to achieve broad-based impacts. PAGE 26 – PEG Mid-Term Performance Evaluation APPLYING LESSONS LEARNED To apply the lessons learned from PEG’s experience described above, USAID/EA/Somalia should adopt an M4P approach to its PSD programming. It is worth noting that the M4P approach is a specialized application of a general USAID policy recommendation, as expressed in its 2010 US Global Development Policy: “We will also strive to help increase the capacity of our partners by investing in systemic solutions for service delivery, public administration and other government functions where sufficient capacity exists; a focus on sustainability and public sector capacity will be central to how the United States approaches humanitarian assistance and our pursuit of the objectives set out in the Millennium Development Goals.”14 A recent USAID publication expands on this policy recommendation, describing how it might operate in practice: “Each set of interconnected actors whose collective actions produce a particular development outcome is a local system. Improving that development outcome requires a systems approach. Building the capacity of a single actor or strengthening a single relationship is insufficient. Rather, the focus needs to be on the system as a whole—the actors, their interrelationships and the incentives that guide them. Improvements in development outcomes emerge from increasing the performance of individual actors and the effectiveness of their interactions. Similarly, sustaining development outcomes depends on the sustainability of the local system, its built-in durability and a level of adaptability that allows actors and their interrelationships to accommodate shocks and respond to changing circumstances.”15 As described in Annex 10, a distinguishing characteristic of the M4P approach is a systems approach to private sector development, aiming to strengthen the functioning of market systems characterized by large numbers of small-scale farmers, pastoralists and MSEs. Or to put it simply, the M4P approach focuses on the system as a whole—the actors, their interrelationships and the incentives that guide them. PEG implemented many elements of an M4P approach in its agriculture, livestock, energy, and business environment sub-activities, but it was hampered by the limited size and scope of its interventions from achieving any significant systemic changes in the relevant market systems. In the evaluators’ judgment, significantly escalating its PSD interventions using the M4P approach offer USAID/EA/Somalia the best opportunity for achieving its purpose and objectives and the outcomes anticipated under IR1 and IR2. 14 USAID, (2014), “Local Systems: A Framework for Supporting Sustained Development,” p. 3. 15 USAID, (2014), “Local Systems: A Framework for Supporting Sustained Development,” p. 4. PAGE 27 – PEG Mid-Term Performance Evaluation ANNEXES PAGE 28 – PEG Mid-Term Performance Evaluation ANNEX 1: ADDITIONAL QUOTES SUPPORTING FINDINGS AND CONCLUSIONS IMPROVED STABILITY  “I haven’t seen any insecurity for two years for the last two years.” (Livestock Spouse, Puntland)  “The communities I come from like promoting peace and solving their differences.” (VetAid Beneficiary, Somaliland)  “In Puntland, security has been very good over the last two years, and it has been maintained.” (Veterinarian, Puntland)  “The situation is calm, and there are no security problems.” (Veterinarian, Puntland)  “I can say 100% that the security is very good. There has been no change in security for the last two years.” (Microcredit Client, Somaliland)  “There are no conflicts here. We have no need for any government security apparatus.” (Fishing Spouse, Somaliland)  “The security condition is currently very good. Clan Elders are able to settle conflicts.” (Livestock Spouse, Somaliland) PRIVATE SECTOR ENTITIES STRENGTHENED  “Previously . . . I realized people who were selling these drugs did not have the knowledge to prescribe the drugs they were selling. This is now starting to improve, although the change is happening gradually.” (CAHW, Puntland)  “Previously many people did not know how to inject animals even the place for the intravenous muscle, but now they know. Initially, we sometimes used to inject our animals in the wrong places . . . but now we are knowledgeable and have good experience on veterinary services . . . .” (CAHW, Puntland)  “People had the animals, and medicine was available, but people lacked the knowledge of how to use it. Since we received the training, we are now able to help others to understand diseases and inject vaccines, as a result, us and society have benefitted.” (CAHW, Somaliland)  “Skills I received from both trainings took me to another level of knowledge about animal health management, both disease and drugs. This training was very relevant to us because we had some skills in this field and this enhanced our level of understanding about this sector.” (Vet/pharmacist, Puntland)  “PEG has done great in Somaliland and the impact of its contributions to the Somaliland renewable energy sub-sector support and promotion has been and remains commendable. The trainings from the USAID-funded technical assistance have been most useful. The demo wind farm in Hargeisa airport gave us the confidence to go ahead.” (SWSPPA) PRIORITY VALUE CHAINS IMPROVED  “The project created 46 jobs. Forty people are temporarily employed by the Ministry of Livestock and other agencies involved in vet activities. Another six are permanently employed in pharmacies, which are still functioning.” (Kaalo, Puntland) PAGE 29 – PEG Mid-Term Performance Evaluation  “We have not formally followed up, but we know that livestock mortality has fallen since the CAHW training because we are receiving fewer reports of livestock deaths from the training region.” (VetAid, Somaliland)  “Twelve pharmacies have at least one trained full-time employee each.” (VetAid)  “We benefited from the lab equipment in so many ways, for instance we use it now to train our staff in the regions, diagnose and examine the animal diseases and on top of that we saved money that we used to spend on outside diagnosis and medical exams.” (MoLAH, Puntland) PEG’S FLEXIBLE IMPLEMENTATION  “DAI was always consulting with partners and took a very flexible approach; for example, it increased the number of assisted pharmacies from three to six when it saw the need.” (Kaalo)  “PEG was very flexible. It had a participatory planning approach. It employed the right people and technical experts.” (Candlelight)  “The PEG team was very professional, supportive, and flexible.” (Horumar Dairy)  “DAI was flexible, understanding, and very professional.” (Kaaba Microfinance)  “DAI was very cognizant of the local context and environment. We relocated from Lasanod to Bereba because of conflict with an existing IPP in Lasanod, although this was beyond PEG’s control. In this case, PEG was very flexible; it changed the original planned cost because of higher fees in difficult terrain and provided a no-cost extension for delays. DAI was very supportive and flexible when confronted with field obstacles.” (Tayo Energy)  “DAI showed flexibility in amending our contract to add the trade fair.” (Shaqodoon) PEG’S PARTICIPATORY APPROACH TO PLANNING AND IMPLEMENTATION  “The Ministry of Energy, the Berbera Mayor and Tayo were all involved in the process from site selection, land leasing, technological feasibility, and so forth.” (Tayo Energy)  “There was a free flow of information between PEG and partners, for example PEG staff explained the delay in the installation of the turbines.” (Tayo Energy) PEG’S OTHER SOUND OPERATING AND IMPLEMENTATION PRACTICES  “PEG provided very useful and effective assistance.” (Candlelight)  “Frankly, I want to state that the two teachers were qualified and experienced. They taught us such significantly important topics and ensured our understanding, and we appreciate them.” (Vet/Pharmacist, Puntland)  “PEG’s operational structures, security measures, and implementation practices were highly effective.” (Red Sea Fishing)  “The PEG staff were very motivated and provided the needed support. The PEG team follows rules to the letter.” (Horumar Dairy)  “PEG staff were highly motivated, supportive and dedicated, and gave clear instructions. Our grant would have failed without the training and technical assistance from PEG.” (Tayo Energy)  “DAI’s energy expert provided first class technical support and training. PEG introduced innovation, strengthened relevant sector entities in renewable energy and encouraged private sector engagement using a novel and effective approach.” (Golis Energy)  “The PEG team was pushy, motivated and constantly monitored activities; this kept grantees on their toes and made them work harder to deliver results. They were professional and supportive.” (Shaqodoon) PAGE 30 – PEG Mid-Term Performance Evaluation NEED FOR LONGER TRAINING  “The training was just five days. It needs to be longer than this.” (Fodder farmer)  “I am requesting a follow up training in business management areas, because we did not get enough time, we need additional time to understand deeply and feel confident.” (Vet/pharmacist, Puntland) PAGE 31 – PEG Mid-Term Performance Evaluation ANNEX 2: EVALUATION STATEMENT OF WORK Performance Evaluation: Proposed Scope of Work United States Agency for International Development (USAID) Partnership for Economic Growth (PEG) Program I. Evaluation Purpose and Audience The United States Agency for International Development (USAID) is proposing technical assistance to conduct a performance evaluation of the “Partnership for Economic Growth (“The Partnership” or PEG).16 The purpose of the performance evaluation is to provide information and recommendations to USAID East Africa on program achievements, perceived value of activities, and progress towards stated results. The evaluation product will be used to assist USAID to design and shape PEG’s programmatic scope following its recently granted contract extension17. Evaluation findings will be shared with USAID (East Africa and Washington) and relevant sector partners, including the implementing partner, NGOs, donors, and Somali authorities. PEG was originally designed to last from April 15, 2011 to August 31, 2013 with an option to extend to August 13, 2015 (pending approval from USAID Washington, and subject to the needs of the U.S. Government (USG), Contractor’s performance, and availability of funds). The contract was awarded to Development Alternatives, Incorporated (DAI). On August 20, 2013, USAID granted PEG a two-year extension, allowing program activities to run through August 31, 2015. The total obligated budget for 2011 – 2013 was $12.4 million and the anticipated funding for the extension is $8 million. II. Program Background 16 USAID’s Evaluation Policy (http://www.USAID.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf), dated January 2011, states that the two primary purposes for evaluations are accountability to stakeholders and learning to improve effectiveness. Both of these outcomes are central to the objective of The Partnership evaluation. The policy also states that performance evaluations focus on descriptive and normative questions: what a particular program or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. 17 Associated Contract No: AID-623-TO-11-00001 PAGE 32 – PEG Mid-Term Performance Evaluation In the context of the current conflict in Somalia, the Somali private sector can be an agent of change, or a spoiler of peace. Given the absence of a central government for the last two decades, big businesses have come to wield significant power in Somalia. Businesses have continued to thrive despite conflict, and today they bring together entrepreneurs across clan, regional, and ideological differences, and many operate in all parts of Somalia as well as in Kenya, Djibouti and Ethiopia. The private sector has demonstrated its ability to contribute to stability. Historically, Somali women have been among the most negatively affected by conflict and natural disasters, nevertheless considerable opportunity exists to support women to become economically self-reliant as well as to become agents of change in their community through microfinance, commercial credit and business association activities will have a direct impact on stability in Somalia – especially at the community level. Pastoral livestock production and marketing in greater Somalia contribute to 80% of the economy. It brings together the intertwined issues of livestock trade and marketing, delivery of animal health, sustainable natural resource management, capacity building, and conflict management in pastoral livestock systems in the Horn of Africa. Interventions in this area are very likely to foster broad-based economic growth and stability. III. Program Purpose, Development Hypothesis and Description A. Program Purpose: The purpose of the PEG program is to help local authorities and private sector groups improve the enabling environment for investment in order to generate more productive employment and improve other livelihood activities. B. Development Hypothesis and Intermediary Results: Because so many external factors outside of PEG’s control contribute to instability, the sequential and unidirectional progression of achievement from outputs to outcomes to impact does not always occur. The iterative development approach should, to a significant degree, enable PEG to be flexible – to respond to variances in the geography where the program will work, as well as to the changing environment of development in those areas. The Partnership’s program objective is to improve stability through inclusive economic growth. As a result of the “experimental design” nature of the program, PEG has articulated two discrete yet inter-related intermediary results (IRs), which are likewise broad enough to capture the expected outcomes of program activities: IR 1: Private Sector Entities Strengthened To strengthen the environment for private sector development, the public sector authorities in each targeted region of Somalia must not only actively participate in assessing and analyzing technical areas and baseline data, but must also build positive working relationships with private sector entities. As more private sector entities are engaged in activities that build their capacity, increase their access to services and assets and resources, expand employment, and develop a positive relationship with relevant public sector actors to improve policies and regulations, then the resulting stability will encourage further economic growth in the form of more investment and trade. IR 2: Priority Value Chains Improved Assessments and improved data collection on key value chains and the private sector, with the participation of government authorities and other stakeholders in analysis of data and joint prioritization of activities will lay a foundation for the achievement of IR2. As capacities are built for veterinary associations, community animal health workers, fishing communities, extension workers, horticulture producers, or other groups of possible stakeholders, they can increase revenues and expand needed employment opportunities. As PAGE 33 – PEG Mid-Term Performance Evaluation productivity improves, as producers gain access to more buyers and better market information so that knowledge sharing along a value chain maximizes efficiency and improves incomes, and as land and resource management improves, more people and firms will be active participants in their local and regional economies. As a result of inclusive economic growth along targeted value chains, stability should improve among vulnerable populations and geographic areas. Evaluation Questions The key evaluation questions to be answered through this evaluation include: A) Has ‘The Partnership for Economic Growth Program’ achieved the objectives as prescribed in the program’s PMP and in the 2011 contract and subsequent three modifications? B) Do the program’s results show that inclusive economic growth has been achieved by its activities? C) Was the development hypothesis and accompanying assumptions that shaped the Program design valid? And does it remain valid for the current Somalia and Somaliland context and USAID’s objectives? D) How have PEG’S operational structures and implementation practices performed in Somalia’s challenging programming environment? How have they affected results? How can they be improved? E) What have been the most important program accomplishments during implementation? What are the lessons learned from the program to date that can be applied to the program’s extension? How should these lessons be applied? A list of subsidiary questions to the main evaluation questions above is contained in Annex one. The findings of this evaluation will be used to inform USAID decisions regarding future Somalia and Somaliland economic growth programs, including activities that fall under PEG’s contract extension. The final report will include actionable findings, conclusions and recommendations to help inform future USAID economic growth programming in Somalia. The primary audience for this evaluation is the USAID/EA/Somalia mission and officials who will manage PEG through its new completion date. IV. Evaluation Design The evaluation will used a mixed methods approach. A combination of qualitative and quantitative data collection methods will be undertaken, including a review of key program and other documents, key informant interviews, focus groups, baseline surveys and results data. A triangulated analysis of the data derived from these data sources will be presented. The evaluation team will collect and analyze data from both primary and secondary sources. Therefore, fieldwork in Somaliland and Puntland will be critical. This is where most key informant interviews and focus groups with key stakeholders and beneficiaries will take place. It will also include gathering quantitative information related to program implementation and results. The evaluation will include the following three phases: 1. Desk Review (Contract SOW, M&E Plans, PMP, Quarterly Reports) 2. Field research (Key Informant Interviews, Focus Group Discussions) in Somaliland and Puntland. 3. Analysis, Synthesis, Briefing and Reporting In addition to qualitative analysis and performance data, the evaluation will draw upon two important surveys (and the response data therein) that PEG initiated as key components of its overall evidence base:  The Economic Growth baseline perceptions survey  The Business Environment/Enterprise perceptions survey The Economic Growth baseline perceptions survey PAGE 34 – PEG Mid-Term Performance Evaluation The Economic Growth baseline perceptions survey collects demographic and other household level economic and wellbeing information related to beneficiary populations. In the PMP it is stated that the data from ‘The Economic Growth baseline perceptions survey will be used to measure the achievements and (potentially the impact) of the program at the objective level. The survey, among other things, captures data on household income, food consumption and other aspects of household socio-economic wellbeing and will be repeated in FY2015 to determine progress made. The Team will assess the rigor of the survey’s implementation and appropriateness of survey data for measuring program accomplishment. It may also determine whether the survey provides the basis for impact evaluation at a later date. The Business Environment/Enterprise perceptions survey The Somali Business Confidence Questionnaire is designed to collect data for the Somaliland Business Confidence Index (SBCI). The evaluation team will assess the rigor and relevancy of the business Confidence Index as an evaluation tool for the PEG program, and if there is a practical correlative relationship between Partnership activities and changes in the index. The Team will determine if the index is a useful analytical framework to contextualize and/or measure Partnership accomplishments, and if there is a methodologically sound way in which to link index results with Partnership activity. This may be through comparisons of program implementation metrics and information with changes in index scores for the relevant sectors or geographic areas. V. Evaluation Data Collection Methods and Analysis Phase One: Desk Review During the first week after notification to commence, the MEPS evaluation team (“the Team”) will conduct a desk review of the literature available on PEG’s programs, including monitoring and progress reports, contractual documents, related sector assessments and evaluations, conference proceedings/subcontractor events, surveys, Somalia national, ministry, regional district economic development plans, economic growth plans and any other relevant documentation including those suggested by USAID, as well as other literature that might provide a richer perspective into promising practices in economic growth in stabilization environments, and economic development in the region. The information gathered in the literature review phase will provide base knowledge of the implementation process for PEG, as well as a broad perspective on USAID’s experiences with PEG, which will then inform the subsequent field work and analysis phases of the performance evaluation. The Team will use this information to draft and submit the Final Design and Data Collection Methodology and Final Work Plan within one week after notification to commence. While the desk review will commence on January 2, 2014, the Team will request that USAID/EA/Somalia make the appropriate materials (see below) available by December 30, 2013, to enable the Team to have a complete set of data in place when its review begins. Program documents and written materials to be reviewed and analyzed will, at a minimum, include:  The Performance Management Plan.  The PEG Contract dated 1 April 2011.  Modifications to the contract after 2011.  PEG’s quarterly and annual reports.  PEG’s M&EP submitted 5 May 2012 and all associated survey and data collection results  Any lessons learned documents produced by the PEG team as a means of self-assessment and internal evaluation. The desk review will be conducted during the pre-field evaluation period, then continue in Nairobi and Hargeisa for the rest of Phase one. The Team will be based in Hargeisa for the majority of the evaluation. PAGE 35 – PEG Mid-Term Performance Evaluation The period spent in Nairobi will also provide the Team with the opportunity to meet with relevant USAID staff, both during a kick-off meeting, and singularly to gain their input into the evaluation. Phase Two: Field Work The bulk of the evaluation’s fieldwork will be conducted in Hargeisa at the Partnership Headquarters, or else visiting areas in which PEG activities are operating. Fieldwork will include the following data collection approaches. Key Informant Interviews (KIIs). Structured, purposive, KIIs with key stakeholders will be the primary method for eliciting qualitative information about the management and implementation of PEG programming as well as its effectiveness, successes, challenges, and accomplishments. Up to 30 KIIs are likely to be conducted. These structured interviews will be based on predetermined question sets derived from the five key evaluation questions as appropriate to the respondent’s involvement and/or exposure to PEG efforts or specific economic growth activities. Note that the KII questionnaires also will include several open-ended questions, which will be designed to “drill down” and draw out examples of PEG’s effectiveness (or the lack thereof) from interviewees. Core team members will conduct the majority of the KIIs, although its local partner will also carry out some interviews if the Team is unable to do so due to logistical or security factors. Each interview will likely last about 45-60 minutes, and that the KIIs will be conducted, at the very least, with:  USAID COR and Program Office  Partnership Management Team and Staff;  Program Stakeholders;  Local and Regional government officials;  Business leaders and private sector representatives;  Beneficiaries both direct and indirect (where relevant) Structured Focus Group Discussions (FGDs). In consultation with USAID, the Team will arrange focus groups with appropriate categories of grantees and beneficiaries. The focus groups, augmented by interviews with purposively selected individuals in each data collection point, will be a key source of data from beneficiaries. focus groups will comprise between 5 and 9 individuals of the same gender, age group, and community. Focus group sessions will be approximately 2-2½ hours long. The Team will develop the questionnaires for the focus groups; as with its design of the KII questionnaires, the Team will include both closed- and open-ended questions, with the former type enabling the Team to identify community opinions and broader trends, and the latter allowing for the discovery of examples of PEG’s effects on beneficiaries as well as the confirmation of trends in the data. The FGDs will be conducted by our local evaluation partner. The Team also will participate in the in-house piloting of the FGDs prior to their fieldwork to help in the revising and finalization of the FGD questions. Where possible, Team Members also will also observe several of the FGDs to ensure quality and consistency. Due to the nature of Somali local dialects, specialists who speak the particular dialect for a specific area will facilitate the FGD in that area; note-takers assigned to each FGD will also ideally speak this dialect. This effort will ensure that key perspectives are not lost in translation. The Team will comply with the USG “common rule” for the protection of human subjects when submitting summaries of collected data to USAID. The Team will conduct up to 12 focus group discussions. The Team will record its KIIs and FGDs by voice and/or video wherever possible and transcribe each interview. In cases where recordings are not feasible, detailed written notes of each interview will be compiled as Microsoft Word documents. The collected data will be categorized under each key evaluation question and broken down under the appropriate KII/FGD questions. The local partner may further categorize each KII/FGD for the Team according to a set of themes based on the aforementioned questions, using an appropriate qualitative data analysis software (e.g., MaxQDA), but this will ultimately be determined PAGE 36 – PEG Mid-Term Performance Evaluation in consultation with the local partner and USAID/EA/Somalia. The collected data also will be disaggregated by age group, gender, community, and parish, as well as by the role of the respondents in the community. Of the 12 FGDs proposed, the team suggests 5 be held in of Hargeisa and Garowe, and 2 be held in Burao. Should there be significant logistical or security challenges in Garowe and/or Burao, reasons, all of the FGDs can be held in Hargeisa. The core team members will participate in the FGDs in Hargeisa and Garowe. Analysis of Key Informant Interviews and Focus Group Discussions. In order to properly assess its collected interview and focus group data, the Team will employ several methods of analysis appropriate to the type and scope of this evaluation. These will include but not be limited to the following: 1) Time-series analysis – This will enable the examination of data measured during regular periods and over fixed intervals. Specific PEG indicator data reported on a regular basis through USAID/EA/Somalia materials (e.g., quarterly or annual reports, program performance documentation) will provide data to draw on from earlier in the program, which will yield richer comparison studies compared to using only more recent data for a performance evaluation; 2) Trend Analysis – This will enable the Team to examine the collected data over time to identify patterns of convergence [or divergence] of activity results toward the program’s objectives; and 3) Gap Analysis – This will examine which aspects of the PEG’s activities, if any, have fallen short between anticipated and actual performance, and what the likely factors are behind these gaps. These methods will be applied to the responses obtained by the Team to the KII/FGD questions. Based on these questions, team members will analyze the data and seek to identify results that address the following: The effectiveness of the PEG in implementing local economic growth methods; The success of PEG’s activities in increasing participation and economic growth, and in building sustainable economic partnerships; The successes, challenges, and barriers in implementing PEG’s approach; The effectiveness of the operational structure of the PEG program in meeting the initial objectives of the Scope of Work (SOW); and The extent to which economic opportunities for growth increased, decreased, or stayed the same in targeted PEG communities, whether PEG program activities contributed to these outcomes activities, and where economic opportunities did not increase, did this factor act as a barrier to achieving PEG’s overall objectives. Selective Quantitative Analysis of Representative Partnership Programs. The PEG program is premised on the economic benefits of strategic interventions in key value chains to improve productivity and profitability; and building relationships between public and private sectors to strengthen the environment for private sector development in the form of more investment and trade. These two intervention models are aimed at promoting inclusive economic growth. To test this hypothesis, the Team will select key representative programs and initiatives and conduct analysis of their outcomes and economic impacts. This may include (but is not limited to):  Return on investment analysis Cost benefit analysis  Beneficiary outcome surveying and analysis (may utilize SMS to do surveys)  Productivity and profit margin analysis for value chain processing and improvement activities.  Other econometric analysis as deemed relevant by the Team if the required data is available. A key question guiding economic analysis of selected programs and initiatives will be whether the intervention generated income and employment, both of which are critical for inclusive economic growth. PAGE 37 – PEG Mid-Term Performance Evaluation As per ADS 203 and the USAID Evaluation Policy, a discrete gender component will be part of the PEG evaluation, with gender-specific sub-questions part of the Evaluation Design and the analysis.18 The evaluation will review how PEG addressed gender equity and equality, in particular, through support to women’s businesses and how women were supported under value chain activities. Sex disaggregated data will be reviewed for the relevant indicators. Women will also be part of the Key Information Interviews and Focus Group Discussions. Limitations and Constraints  Locating Beneficiaries for KIIs and FGDs – It may be challenging to find sufficient numbers of PEG beneficiaries. The Team will utilize several pragmatic methods to identify and locate such groups, however. Using the ‘snowball’ technique the team will query knowledgeable informants during KIIs about groupings and/or locations of beneficiaries, target community groups located close to and utilizing USAID ‘brick and mortar’ economic growth programs, such as markets, for example, and will select current individuals or organizations that directly benefited from USAID technical assistance. The Team also will communicate extensively with and collect information on program sites and beneficiaries through the Mission’s CORs/AORs, former USAID implementing partner staff and senior USAID or implementing partner staff who knew about or managed these programs before and during the Team’s desk review period to determine as accurately as possible where these beneficiaries are located.  Causality and Attribution – It may prove challenging to establish attribution of impact to PEG’s program activities and the activities themselves, as program beneficiaries may credit positive changes in perception to other events or conditions that were outside of the focus of these programs. This may also prove to be the case for FGD participants located in areas in which activities from other entities were or are also being implemented, as they may attribute any positive changes to a blend of activities or infrastructure programs. The evaluation methodology will mitigate this challenge by utilizing pragmatic techniques for assessing the degree of correlation rather than causality between PEG activities and improved perceptions economic growth. A qualitative “counterfactual” will be established in consultation with USAID/ Somalia to determine what other factors, events, activities or actors besides USAID’s programs could have led to these altered perceptions. Phase Three: Write up of Evaluation The final stage in the evaluation process will be the synthesis and analysis of the data collected from program documents, from qualitative research, program implementation metrics, the achievement against the programs results framework and its wider evidence base (program surveys), and from independent economic analysis by the Team. All of these sources will be used to make determinations about the program’s overall performance and its effect on intended objectives. The synthesizing of the different strands of program and evaluation data will require crafting a valid framework to evaluate the program given that this is a performance evaluation of a program that is working to affect long term economic outcomes. The Team will look in turn at each of PEG’s objectives and activities and use the information collected to determine how the activities were actually carried out, and how implementation modalities affected program outcomes. Evaluators shall seek to identify internal shortcomings or exogenous obstacles that prevented the 18 This activity was designed prior to guidance in USAID's Gender Equality and Female Empowerment Policy, so it did not focus on gathering gender gap data. However, it aimed to address gaps facing women owned businesses, such as access to financing, and supporting women in value chain development, including increasing employment opportunities for women. PAGE 38 – PEG Mid-Term Performance Evaluation PEG Program from achieving its intended objectives, or – if appropriate – what objectives were achieved that were not intended or initially identified. An aspect of the evaluation will be ascertaining which programs and implementation models worked well and produced positive results and which did not, and why this occurred. This part of the evaluation may examine the programs performance data, implementation data (inputs) and the programs geographic cluster monitoring (page 10 of the PMP) to explore how implementation and external factors affected program results and outcomes. This combined with the ‘Economic Growth baseline perceptions survey’ and the ‘Business Environment/Enterprise perceptions survey’, may provide an approach to judge which programs worked better than others. V. Deliverables, Evaluation Report, Personnel, Period of Performance/LOE, Somalia Flight Schedule A. Deliverables Deliverable Description Illustrative Timeline Del 1 - In brief with USAID See timeline Del 2 - Evaluation Design and Work Plan See timeline Del 3 - Out brief with USAID See timeline Del 4 - Draft Report See timeline Del 5 - USAID feedback See timeline Del 6 - Final Report See timeline Del 7 - Final Report Uploaded to DEC See timeline See in Annex the timeline/calendar of activities document for detailed day-by-day description and location of activities. B. Evaluation report content and format The consultant shall submit a draft report within 15 working days of completing the out brief with USAID. This document should explicitly respond to the requirements of the SOW, should answer the evaluation questions, be logically structured, and adhere to the standards and criteria of the USAID Evaluation Policy of January 2011. The evaluation report should include a cover page, an Executive Summary, Table of Contents, List of Acronyms, (possible map of country or program area), Introduction, Background of the local context and the program/s being evaluated, and Actionable Findings, Conclusions and Recommendations to help inform future EG programming in Somalia. The report should not exceed 25 pages, excluding section before the Introduction, e.g. Executive summary, Table of Contents, List of Acronyms, Map, and Annexes. The Summary, Draft and Final reports will substantively follow the three main sections and format established in TIPS 17: Constructing an Evaluation Report. The consultant shall submit a final report within 10 days of receiving USAID feedback on the draft report. The three main sections are: 1. Findings 2. Conclusions 3. Recommendations & Lessons Learned The report should meet the criteria for quality evaluation reports mandated by USAID Evaluation Policy (See Annex A of the Policy). C. Evaluation team Composition PAGE 39 – PEG Mid-Term Performance Evaluation Team Leader and Economic Growth and Evaluation Expert – Al Sanchez19: The Team Leader has over 10 years of USAID or international development evaluation experience, with strong experience with economic evaluation and value chain analysis, and in leading evaluation teams in austere environments. He has strong team management skills and sufficient experience with evaluation standards and practices to ensure a credible, actionable, insightful product, and is able to maneuver through the evaluation research design and planning process. He can deal effectively with senior U.S. and host country officials and other leaders, and has significant experience with USAID and with formative evaluations designed to establish the basis for a future USAID program or the redesign of an existing program. The Team Leader will be responsible for the overall technical execution of the evaluation work, analysis and drafting and completion of the evaluation report, supported by the IBTCI home office team and EG and evaluation expert, as well as the Somali-speaking Economic Growth Expert. Somali-speaking Economic Growth Expert - Sharmarke Farah: has over 5 years of experience in economic growth, quantitative and qualitative evaluations and extensive experience in the Somali context. Note that Mr Farah will provide technical support to both the PEG program evaluation and the Economic Growth assessment. Mr. Farah will however accompany Steve Hadley (Eco Growth Assessment team leader) in Mogadishu when the team splits to allow Al Sanchez to stay longer in Somaliland to focus on PEG related sites and activities outside of Hargeisa, Somaliland. During this time, Mr. Sanchez will be temporarily supported by another Somali-speaking consultant until he rejoins Mr. Hadley and Mr. Farah again in Puntland. The Somali-speaking Economic Growth Expert will be supporting the Team Leader at various stages of the evaluation, including information/data collection and analysis, as well as field-based interviews in the field. The Expert will also provide insight into the Somali context and help facilitate interviews and activities requiring Somali-speaking skills. While keeping in mind that he will also provide support to the EG assessment Team Leader, the Expert contribute to the drafting of the report as directed by the Team Leader, EG and Evaluation Expert – A short term consultants (based in Washington DC) will provide additional technical support to the evaluation team members to help meet tight deadlines, support depth of analysis and ensure compliance with evaluation guidelines. Mr. Gary Woller is a professional evaluator with extensive Economic Growth experience who will provide remote technical support on the design and analysis of the evaluation report (up to 8 days of Level of Effort). At least one member of the team must have experience in gender. Please see Annex two for CVs of the proposed core team. Local Partner: The evaluation will leverage the expertise of a local supplier to facilitate and conduct the FGDs. The local partner will conduct up to 12 FGDs in Somaliland and/or Puntland. The exact location of the FGDs will be determined in consultation with the core team and USAID. As mentioned above, FGDs will be a source of primary qualitative data for the evaluation, and the local partner will be familiar with various techniques to enable local populations, and especially the program beneficiaries, to give their assessments and perspectives on the various evaluation questions. The FGDs should be facilitated by senior field supervisors/facilitators. The partner will work with the core team and IBTCI/MEPS staff to ensure that the FGD methodology agrees with the overall PEG evaluation design. The partner will be local, and have experience designing and carrying out data collection activities, particularly FGDs. The partner will have a 19 Given the nature of the evaluation, the Team Leader's role transitioned over time to being a field lead, with the primary analysis and reporting responsibilities being undertaken by the Senior Technical Adviser. PAGE 40 – PEG Mid-Term Performance Evaluation relevant track record with the donor and implementing partner community. Home Office Support: The IBTCI Home Office will provide quality assurance support by providing a team member with credentials and expertise in assessment design and methods in the economic growth sector. Additionally, IBTCI Home Office will provide publication support in finalizing the evaluation report. PAGE 41 – PEG Mid-Term Performance Evaluation ANNEX 3: BIBLIOGRAPHY Adam Smith International, (2013), “Private Sector Investment and Barriers to Growth: Analysis in South￾Central Somalia and Puntland,” DFID. Candlelight for Health, Education & Environment, (2013), “Narrative Report on Fodder Demonstration Sites Set-up.” Candlelight for Health, Education & Environment, (2013), “Promotion of Community-based Fodder Production, Storage and Handling: Final Report.” DAI and Nathan Group, (2011), “Somalia Economic Growth and Livelihood Initiative: Submitted by DAI/Nathan Group to the U.S. Agency for International Development in Response to RFTOP No. AID-623- 11-006 under the GBTI II IQC.” Food Security and Nutrition Analysis Unit, (2013), “Subsistence Farming in Lower Shabelle Riverine Zone,” FAO. Kaalo Aide and Development, (2013), Livestock Value Chain Project Final Report.” Lovatelli, Alessandro, “EC Rehabilitation Programme for Somalia,” European Union, European Commission Somalia Unit. Partnership for Economic Growth, (2011), “First Quarterly Report: April-June 2011,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2011), “Second Quarterly Report: July-September 2011, USAID/East AfricaLPC Somalia Program. Partnership for Economic Growth, (2011), “Monitoring and Evaluation Plan / Performance Management Plan,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2011), “Somaliland Private Sector Development Assessment, Business Enabling Environment: Chapter One,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2011), “Somaliland Private Sector Development Assessment, Enterprise Development: Chapter Two,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2011), “Somaliland Private Sector Development Assessment, Livestock Value Chain: Chapter Three,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2011), “Somaliland Private Sector Development Assessment, Energy Value Chain: Chapter Five,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2011), “Somaliland Private Sector Development Assessment, Other Productive Value Chains: Chapter Six,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2011), “Somaliland Private Sector Development Assessment, First Stakeholder’s Conference Hargeisa: Chapter Seven,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2011), “Somaliland Private Sector Development Assessment, First Stakeholder’s Conference Borama and Burao: Chapter Eight,” USAID/East Africa/LPC Somalia Program. PAGE 42 – PEG Mid-Term Performance Evaluation Partnership for Economic Growth, (2011), “Third Quarterly Report: October-December 2011,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2012), “Fourth Quarterly Report: January-March 2012,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2012), “Fifth Quarterly Report: April-June 2012,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2012), “Sixth Quarterly Report: July-September 2012,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2012), “Seventh Quarterly Report: October-December 2012,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2012), “Report on Findings Economic Growth Survey (Wave 1: Baseline),’ USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2013), “Eighth Quarterly Report: January-March 2013,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2013), “Ninth Quarterly Report: April-June 2013,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2013), “Tenth Quarterly Report: July-September 2013,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2012), “First Annual Work Plan: May 1, 2011-April 30, 2012,” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2012), “Report of Findings Economic Growth Survey (Wave 1: Baseline),” USAID/East Africa/LPC Somalia Program. Partnership for Economic Growth, (2013), “Linkage workshop between CAHWs and Veterinary Pharmacies Report: Sool Region.” Partnership for Economic Growth, (2013), “Livestock Vaccines for Puntland: Improvement of Veterinary Vaccine Supply Chain.” Peterlini, Edoardo, John Ashley, Ulrich Schmidt, Chris Pint, Mohamed Noor, and Julio Castro, (2012), “10th EDF Supplementary Funding Economic Growth Programme Identification Study: Final Report,” European Union. Somali Agricultural Technical Group, (2013), “Economic Growth Assessment of South-Central Somalia,” USAID/East Africa/LPC Somalia Program. SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Training of Veterinary Pharmacists on Recording Keeping- Protocols and Reporting.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Puntland Livestock Value Chain, Livestock Sector Stakeholders Workshop.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Ruqi Farmers' Field Day.” PAGE 43 – PEG Mid-Term Performance Evaluation SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Bid Committee Review for Office Refurbishment for the Ministry of Commerce.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Al Husseini Farm.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Bid Opening for Afjireh Household Gas Stoves Supplier.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Agricultural Technology Testing and Transfer Stakeholder Workshop.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Kaaba Micro-Finance Loans Disbursement.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Community Based Animal Health Workers Training.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Equipment of Veterinary Laboratory.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Fodder Demo Plot Establishment and Forage Demonstration Site - Beerato Community.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Amoud University.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Farmers Field Day Amoud.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Farmers Field Day Baki.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Training of 20 CBAHWs for 3 Weeks in Lasanod.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Grant for Horumar Camel Dairy.” SORADI, (2013), “MEPS-Field Verification & Monitoring ‘snapshot’ Report: Somaliland Annual Business Conference: Launching Business Confidence Survey.” Springfield Centre, “A Synthesis of the Making Markets Work for the Poor (M4P) Approach,” DFID and Swiss Agency for Development and Cooperation.” Training and Research Group, (2012), “2012 Business Confidence Survey," USAID/East Africa/LPC Somalia Program. USAID, (2009), “A Guide to Economic Growth in Post-conflict Countries,” Office of Economic Growth, Bureau for Economic Growth, Agriculture and Trade. USAID, (2005), “Fragile States Strategy.” USAID, (2014), “Local Systems: A Framework for Supporting Sustained Development.” USAID Afghanistan, “District Stability Framework.” United States Department of State, (2014), “Strategic Plan: 2014-2017.” VetAid, (2013), “Final Report on Community Based Animal Health Workers.” PAGE 44 – PEG Mid-Term Performance Evaluation VetAid, (2013), “Procurement and Distribution of Veterinary Drugs to 12 Pharmacies: Report.” PAGE 45 – PEG Mid-Term Performance Evaluation ANNEX 4: LIST OF KEY INFORMANT INTERVIEWS # Name Organization Position 1 Andrew Read FAO Fisheries Coordinator 2 Aruna F. Gujral FAO Head of Planning and Coordination Unit 3 Cyprien Biaou FAO Livestock Coordinator 4 Ernest Njoroge EU Programme Officer 5 Nicolas Tremblay FAO Programme Coordinator 6 Paolo Toselli EU Programme Manager, Food Security 7 Paul Macharia DFID Senior Programme Officer 8 Richard Walker African Development Bank/Fund Principal Economist 9 Rowan Yamanaka DFID Economic Adviser 10 Daniel Molla FSNAU Chief Technical Advisor 11 Jasper Gosselt RSA Research Director 12 Danny Adams DAI Africa Manager 13 Suleiman Mohamed DAI/PEG Chief of Party 14 Zaki Raheem DAI/PEG Deputy Chief of Party 15 Rich Magnani DAI/PEG Technical Expert Agriculture Sub-Activity 16 Hussein Haji SATG Managing Director 17 Said Hussein Iid Puntland Chamber of Commerce Chairman 18 Hassan Bulbul Kaah Electric Company CEO 19 Saed Mohamed Ahmed Somaliland National Youth Organization Chairman 20 Ahmed M. Omar National Fisheries Cooperation CEO 21 Abdirahman Abdalla Ministry of Livestock-Puntland Minister of Livestock 22 Dr. Abdi Aw-Dahir Ali Ministry of Livestock-Somaliland Minister of Livestock 23 Abdirahman Abdalla Ministry of State for Livestock Minister of State for Livestock 24 Fadumo Alin Kaaba MFI Chairperson 25 Mohamed Hassan Nur Shaqadoon Executive Director 26 Mohamed Habane Amoud University Manager 27 Yussuf Ali N/A Camel Dairy Farmer 28 Dr Ahmed Artan VETAID Executive Director 29 Ibrahim Yacub Tayo Energy Enterprises Company Managing Director 30 Kaltun Sheikh Hassan NAGAAD Women Umbrella Chairwoman 31 Ahmed Jama Sugulle Candlelight Program Manager 32 Osman Hajji Red Sea Fishing Company Proprietor 33 Shafici Mohamed Kisima Administration and Finance Assistant 34 Mohamed Abdullahi Gardaf VETAID Country Director 35 Sayid-Ali Ahmed Abdi Golis Solar Company General Manager 36 Abdi Ali Maal Golis Solar Company Technical Officer 37 Steven Jacobson CNFA Africa Project Manager 37 Mohamud Hamid Mohamed Kaalo Executive Director 38 Abdirahman M. Farah Kaalo Project Coordinator 39 Ahmed Ibrahim Awale Candlelight Executive Director 40 Ahmed Ali Jama Shaqadoon Program Officer 41 Mustafa Othman Shaqadoon Community and Technology Manager 42 Paul Smith EU Programme Manager, Infrastructure PAGE 46 – PEG Mid-Term Performance Evaluation ANNEX 5: LIST OF FOCUS GROUP DISCUSSIONS # Intervention Location Number Participants Gender Participants 1 Fisheries Borama / Darey Macaan, Somaliland 11 Male 2 Fisheries Ceel Shiek, Somaliland 9 Spouses, Female 3 Fisheries Ceel Shiek, Somaliland 8 Male 2 Livestock Garowe, Puntland 8 Spouses, Mixed 3 Livestock Garowe, Puntland 8 Mixed 6 Livestock Burao/Qoyta, Somaliland 8 Mixed 7 Livestock Burao/Qoyta, Somaliland 7 Spouses, Mixed 8 Livestock Burao, Somaliland 8 Mixed 9 Veterinary Garowe, Puntland 10 Mixed 10 Veterinary Burao, Somaliland 8 Mixed 11 Microcredit Gebiley, Somaliland 11 Mixed 12 Agriculture Borama / Darey Macaan, Somaliland 11 Male 13 Youth Employment Hargeisa, Somaliland 9 Mixed PAGE 47 – PEG Mid-Term Performance Evaluation ANNEX 6: KEY INFORMANT INTERVIEW DICUSSION GUIDE 1. Have the PEG strengthened the Private Sector Entities?  Yes / Not-sure / NOT  PEG motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: a. b. c. 2. Are the priority value chains improved with the PEG interventions? Background: (Veterinary associations, Animal health workers, fishing communities, Extension workers, Horticulture producers, other stakeholders increase revenues; Employment opportunities (DO WE HAVE OTHER/BETTER EXAMPLES IN THE TASK ORDER?)  Yes / Not-sure / NOT  PGE motivation and Drive/ Passion to improve value chains: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: a. b. c. 3. In your opinion, has the PEG implemented the expected activities? Background: Training events supported by PEG; increase in skills/knowledge; training in partnerships; support training extension agents or apprentices; site visits by extension agents; services provided by extension services or apprentices; better employment; Partnership support activities; improve their management practices; improvement of women’s organizations/ associations assisted as a result of Partnership interventions (FAF); energy.  Yes / Not-sure / NOT  PGE motivation and Drive/ Passion to implement activities: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: a. b. c. 4. Are you aware whether the PEG objectives and/or activities changed? Background: The Partnership’s program objective is to improve stability through inclusive economic growth  Yes / Not-sure / Not  Yes / Not-sure / Not  PGE motivation and Drive/ Passion to introduce changes: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: How objectives and/or activities changed Why the changes were introduced? a. b. c. 5. Has the PEG successful cases to show? Yes / Not-sure / NOT PGE motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high PAGE 48 – PEG Mid-Term Performance Evaluation Reflections/Perceptions with examples of cases to show: a. b. c. 6. Do you known well the PGE’s success cases? Yes / Not-sure / NOT PGE motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high Reflections/Perceptions with examples of successes: a. b. c. 7. What PGE activities and components achieved the best results? Yes / Not-sure / NOT PGE motivation and Drive/ Passion to achieve results: 1-Low 2-Medium 3-high Reflections/Perceptions with examples: a. b. c. 8. Are you aware whether PGE has taken shortcomings? Yes / Not-sure / NOT PGE motivation and Drive/ Passion to avoid shortcomings: 1-Low 2-Medium 3-high Reflections/Perceptions with examples: a. b. c. 9. Was the development hypothesis that shaped PGE design valid, and does it remain valid for the current Somalia and Somaliland context and objectives? Background: Because so many external factors outside of PEG’s control contribute to instability, the sequential and unidirectional progression of achievement from outputs to outcomes to impact does not always occur. The iterative development approach should, to a significant degree, enable PEG to be flexible – to respond to variances in the geography where the program will work, as well as to the changing environment of development in those areas. Yes / Not-sure / NOT  PGE motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: a. b. c. 10. Are you aware whether the development hypothesis clearly reflected in key program documents including the contract, PMP and work-plan? Background: Because so many external factors outside of PEG’s control contribute to instability, the sequential and unidirectional progression of achievement from outputs to outcomes to impact does not always occur. The iterative development approach should, to a significant degree, enable PEG to be flexible – to respond to variances in the geography where the program will work, as well as to the changing environment of development in those areas Do you know the development hypothesis used for PEG?  Yes / Not-sure / NOT  How do you grade the quality of the hypothesis? 1-Low 2-Medium 3-high PAGE 49 – PEG Mid-Term Performance Evaluation  Reflections/Perceptions with examples: a. b. c. 11. Are you aware whether the program managers, staff, partners understood the PEG hypothesis and did it guide their activities?  Yes / Not-sure / NOT  How do you grade the quality of the contract conditions? 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: a. b. c. 12. Do you know the main issues/objectives in the contract signed among the USAID and DAI?  Yes / Not-sure / NOT  How do you grade the quality of the contract conditions? 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: a. b. c. 13. Have you had access to the PEG Performance Management Plan PMP? Yes / Not-sure / NOT  How do you grade the quality of the PMP? 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: a. b. c. 14. Do you know about the existence of the PEG work Plan? Yes / Not-sure / NOT  How do you grade the quality of the work plan? 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: a. b. c. 15. Has the PGE’s results shown a link between the development IRs and inclusive economic growth, in particular employment? Background: IR 1: Private Sector Entities Strengthened: To strengthen the environment for private sector development, the public sector authorities in each targeted region of Somalia must not only actively participate in assessing and analyzing technical areas and baseline data, but must also build positive working relationships with private sector entities. As more private sector entities are engaged in activities that build their capacity, increase their access to services and assets and resources, expand employment, and develop a positive relationship with relevant public sector actors to improve policies and regulations, then the resulting stability will encourage further economic growth in the form of more investment and trade. IR 2: Priority Value Chains Improved: Assessments and improved data collection on key value chains and the private sector, with the participation of government authorities and other stakeholders in analysis of data and PAGE 50 – PEG Mid-Term Performance Evaluation joint prioritization of activities will lay a foundation for the achievement of IR2. As capacities are built for veterinary associations, animal health workers, fishing communities, extension workers, horticulture producers, or other groups of possible stakeholders, they can increase revenues and expand needed employment opportunities. As productivity improves, as producers gain access to more buyers and better market information so that knowledge sharing along a value chain maximizes efficiency and improves incomes, and as land and resource management improves, more people and firms will be active participants in their local and regional economies. As a result of inclusive economic growth along targeted value chains, stability should improve among vulnerable populations and geographic areas.  Yes / Not-sure / NOT Reflections/Perceptions with examples: a. b. c. 16. Have the new or better employment opportunities created by PEG, impacted both men and women?  For men: Yes / Not-sure / Not ; Level of new/improve jobs: 1-(Low) 2 3 4 5-(high)  For women: Yes / Not-sure / Not ; Level of new/improve: 1-(Low) 2 3 4 5-(high)  PGE motivation and Drive/ Passion to achieve: For men: 1-Low 2-Medium 3-high  PGE motivation and Drive/ Passion to achieve: For women: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: What type and quality of jobs were created? Please estimate numbers. a. b. c. 17. Can any stabilization results be reasonably linked to contributions from Partnerships activities?  Yes / Not-sure / NOT  PGE motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: a. b. c. 18. Are you aware whether the PEG’s operational structures, security measures and implementation practices effective?  Yes / Not-sure / NOT  PGE team motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: a. b. c. 19. Are you aware whether the Partnership faced any notable operational, security or implementation challenges in Somalia or Somaliland?  Yes / Not-sure / NOT  How relevant were these challenges?: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: Where and how did the team overcome these challenges? What impact, if any, did these challenges have on program implementation? a. b. PAGE 51 – PEG Mid-Term Performance Evaluation c. 20. Are you aware whether decisions regarding operational and implementation modalities effected program results?  Yes / Not-sure / NOT  PGE motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: How were the results affected? a. b. c. 21. Did the relationships between/among key stakeholders affected the program?  Yes / Not-sure / NOT  PGE motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: a. b. c. 22. Were key relationships among stakeholder managed well and effectively?  Yes / Not-sure / NOT  PGE motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: a. b. c. 23. Have PEG activities strengthen the business environment for private sector development for both men’s and women’s businesses?  For males: Yes / Not-sure / NOT  For females: Yes / Not-sure / Not  For males: PGE motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  For females: PGE motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples:  How effective have been? a. b. c. 24. Have PEG activities under IR1 directly contributed to inclusive economic growth? Background: IR 1: Private Sector Entities Strengthened. To strengthen the environment for private sector development, the public sector authorities in each targeted region of Somalia must not only actively participate in assessing and analyzing technical areas and baseline data, but must also build positive working relationships with private sector entities. As more private sector entities are engaged in activities that build their capacity, increase their access to services and assets and resources, expand employment, and develop a positive relationship with relevant public sector actors to improve policies and regulations, then the resulting stability will encourage further economic growth in the form of more investment and trade  Yes / Not-sure / NOT  PGE motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: To what extent have Partnership activities under IR1 directly contributed to inclusive economic growth? a. PAGE 52 – PEG Mid-Term Performance Evaluation b. c. 25. Have PGE activities built relationships between the public and private sectors?  Yes / Not-sure / NOT  PGE team motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: What tangible businesses ventures have emerged from this work? How sustainable are these ventures? a. b. c. 26. Has investment and trade been effected by PEG in areas in which it is operating?  Yes / Not-sure / NOT  PGE team motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: How has investment and trade been effected by Partnership in areas in which it is operating? a. b. c. 27. Did the PEG strengthen regulatory effectiveness for key local markets? And if so, how?  Yes / Not-sure / NOT  PGE motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: How the program strengthen regulatory effectiveness for key local markets? a. b. c. 28. Did PGE team has to make decisions (choices) in terms of balancing immediate economic impacts with longer-term economic solutions?  Yes / Not-sure / NOT  PGE motivation and Drive/ Passion to make good choices: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: What decisions did Partnership have to make in terms of balancing immediate economic impacts with longer-term economic solutions? a. b. c. 29. Have targeted values chains been improved by PGE activities for both men and women and what impact has this had on inclusive economic growth? For men: Yes / Not-sure / Not ; Level of perceived improvement: 1-(Low) 2 3 4 5-(high) For women: Yes / Not-sure / Not ; Level of perceived improvement: 1-(Low) 2 3 4 5-(high)  PGE motivation and Drive/ Passion to achieve: For men: 1-Low 2-Medium 3-high  PGE motivation and Drive/ Passion to achieve: For women: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: To what extent have targeted values chains been improved by Partnership activities for both men and women and what impact has this had on inclusive economic growth? a. b. PAGE 53 – PEG Mid-Term Performance Evaluation c. 30. Are you aware whether the collected PGE’s assessments and data on key value chains has been used by the public and private sectors to promote local economic growth  Yes / Not-sure / NOT  PGE motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: To what extent have collected assessments and data collection on key value chains been used by the public and private sectors to promote local economic growth a. b. c. 31. Did the PGE investment in value chain improvement?  Yes / Not-sure / NOT  PGE motivation and Drive/ Passion to achieve: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: What investments in value chain improvement were made by Partnership? What economic pay-offs have these investments produced a. b. c. 32. Did the PGE promote the flow of market information and encourage regional markets for agriculture and livestock?  Yes / Not-sure / NOT  PGE motivation and Drive/ Passion to promote: 1-Low 2-Medium 3-high  Reflections/Perceptions with examples: How the program promoted the flow of market information? How the program encourage regional markets for agriculture and livestock a. b. c. PAGE 54 – PEG Mid-Term Performance Evaluation ANNEX 7: FOCUS GROUP DISCUSSION GUIDES INTRODUCTION  Introduce yourself and the firm you work for  Explain focus group discussion as a method of data collection  Explain the purpose of the discussion/ meeting  Introduce equipments to be used  Stress on the issue of confidentiality GENERAL INFORMATION  Please describe yourself: Name, Age, Occupation, Marital status  How would you describe the following: o Current economic situation? o Current security situation ? o Life in general ?  What are the challenges of the wellbeing of people in your community?  How are these challenges addressed?  What is the role of the government in addressing these challenges?  What is the role of other non-governmental / private sector organizations in addressing these challenges? PARTNERSHIP FOR ECONOMIC GROWTH PROGRAMME AWARENESS  When was the first time you heard about the PEG programme? Probe for year  How information about PEG programme was obtained  Briefly explain the programme activities and its intended benefits  Are you aware of the criteria used in the selection of the beneficiaries for the programme? If yes, please explain the criteria  What aspects of the programme were/ are most relevant to the needs of the participating households?  What aspects of the programme were most relevant to the needs of the community?  In your opinion, what do you think have been the MOST significant impacts to the participating households (beneficiaries)? What do you think have been the MOST significant impacts to the community as a whole? ECONOMIC INFORMATION  What are the economic activities that your community members are involved in? probe for sources of income for the households  For each economic activity mentioned ask for the proportion of the population that relies on the economic activity  Has there been any change in the economic activities that the community members are involved in after the introduction of the PEG programme? Probe for number of jobs created, levels of income, better opportunities for earning additional income, technological advancement, reduction in cost of production, access to information, market stability, access to markets etc.  What types of assets are mostly owned by the community members? Has there been any change in the types of assets owned by beneficiary households after the introduction of the program? Any change on the type of assets owned by the community members? Probe for types of assets owned before and after the PEG programme, number of assets before and after PAGE 55 – PEG Mid-Term Performance Evaluation  What are the major economic challenges that have been addressed by the PEG programme at  A personal / household level  At the community level  What are the economic gaps that are yet to be addressed?  Whose responsibility is it? Why? STABILITY  How safe are you in this community?  What are the causes of insecurity in this area?  Who is responsible for ensuring the security of the community? Of the households?  What are the community priorities to improve security in this area?  Do people meet to discuss security issues in this area?  Who participates in these meetings?  Who leads these discussions?  How effective have these meetings been in improving security in this area?  Do you have community activities that are intended to promote peace in the community?  What are these activities?  Who are the main participants in these activities? Probe for Gender, age  How effective have these activities been in promoting peace? Please explain.  What are the measures undertaken to promote peace and stability in this area? o By the community o By the Government o By non Governmental / private organizations  Any changes on the overall security of the area that came as a result of the PEG programme? Please explain your answer.  ACCESS TO BASIC SERVICES  What types of schools does the community have access to? Probe for private, public, primary, secondary, vocational  What is the distance from the community center for each type of school?  Does any informal education take place in this community?  What sorts of skills are found in the community?  Where do people go to learn these skills?  Describe local levels of literacy and drop outs (indicate gender differences)?  What are some of the efforts by: o The community? o The government? o Non- governmental / Private organizations to improve education?  Any changes in the education system that came as a result of the PEG programme? Please explain.  What types of health facilities are available locally?( probe for government hospital, health clinic/ post, private hospital, private clinic, private pharmacy)  Who has access to these services?  What are the costs of seeking care?  Identify the challenges you face in seeking medical care? Probe for ease of access, affordability, etc  What efforts have been put in place to improve on health care by: o The community? o The government? o Other Non-governmental / Private organizations? PAGE 56 – PEG Mid-Term Performance Evaluation  Any changes in the health system that came as a result of the PEG programme? Please explain  Does the community have access to electricity?  Are there some groups that do not have access to electricity? Who? Why?  What proportion of the population is able to afford electricity?  Does the community have access to fuel for cooking/lighting/heating?  Are there some groups that do not have access to fuel for cooking/lighting/heating?  Does the community have access to piped water?  Are there some groups that do not have access to piped water? Where do they get water from? How is the water stored?  What is the condition of other existing infrastructure( probe for condition of roads, transportation, irrigation, telephone services, post office, sanitation e.t.c)  Any efforts to improve on the infrastructure by: o The community o The government o Non -governmental organizations  Any changes that are evident in the energy sector (sources of fuel for household use as well for industrial use), infrastructure (condition of roads, transportation, irrigation, communication, sanitation/ sewerage systems, water provision) that came as a result of the PEG programme? Please explain. INSTITUTIONAL SUPPORT  Have you received any form of training aimed at improving your work?  When was the last training you attended?  Who organized it? Probe for the institution, groups  What was the training on?  Who were the participants in this training? Probe for gender, age, professions etc  Did you pay for the training?  What are the benefits of such trainings? Probe for benefits to an individual and to the community at large  Do you practice what you were trained on? Why (if not practicising)? Any improvement in your work as a result of the training? Probe for changes in income, increase in production, use of new technology etc.  After training, how have you continued to access information concerning your area of work( probe for market information, access to support services, access to inputs etc)  Does the government provide such kind of trainings?  Are such kinds of trainings provided by the PEG programme? How frequent are the trainings organized by PEG? What are the impacts of such trainings to the beneficiaries? To the community? CROP PRODUCTION  Which major agricultural crops are produced locally? Please list all the crops  Is the produce for subsistence use or for sale? Or both?  If the produce is sold, is the produce sold locally or exported? For export probe for where it is exported to? Mode of transport? How do you access information about market prices? Demand and supply information etc ( for produce)  For how long have you been in Agricultural crop production?  What are the inputs you use?  How do you obtain these inputs?  What are the methods of farming that you use currently?  What are the PEG activities that are aimed at improving Agricultural crop production?  How beneficial are the activities of the programme to your farming activities? PAGE 57 – PEG Mid-Term Performance Evaluation  What are the major challenges in the area of crop production?  How has the government addressed these challenges?  How have the non -governmental/private organizations addressed these challenges?  How has the PEG programme addressed these challenges?  What are the existing gaps in the area of crop production that need to be addressed?  Whose responsibility is it? Why? LIVESTOCK KEEPING/ PASTROLISTS  Major type of livestock  Are the livestock for trade? If for trade where do you sell your livestock? Mode of transport?  For how long have you been keeping livestock/involved in livestock trade?  Any inputs associated with livestock keeping? How do you obtain the inputs?  Do you have access to veterinary services? If yes, do you pay for the veterinary services? Affordability?  Who provides the veterinary services?  How do you access information on markets for livestock and by products, inputs?  Does the government have any role in stabilizing the market prices for livestock markets? Please explain.  What are the PEG activities that are aimed at improving the livestock sector?  How effective have these activities been to the beneficiaries? To the community at large?  What are the existing gaps that need to be addressed in livestock sector?  Whose responsibility is it? Why? FISHERIES  Length of involvement in fishing activities  Involved in commercial fishing or for own consumption? Or both?  If for business, where do you market your products?  Any inputs associated with fisheries? If yes, how do you obtain the inputs?  How do you access market information for fisheries? And any other information on fisheries?  Does the government have a role in the regulation of fisheries sector? Please explain  What are the PEG activities that are aimed at improving/ supporting the fisheries sector?  What are the impacts of these activities to the beneficiaries? To the community?  What are the existing gaps that need to be addressed in the fisheries sector?  Whose responsibility is it? Why? PAGE 58 – PEG Mid-Term Performance Evaluation ANNEX 8: GUIDELINES FOR THE STRUCTURED FOCUS GROUP DISCUSSIONS (FGDS): ENTREPRENEURS’ SPOUSES INTRODUCTION  Introduce yourself and the firm you are working for  Explain focus group discussion as a method of data collection  Introduce equipment to be used  Explain the purpose of the focus group discussion  Information confidentiality- mention that the information will be confidential GENERAL INFORMATION  Please describe yourself- Name, occupation, Family status  How would you describe the current  Economic situation  Security situation  Infrastructure( roads, health facilities, schools, water and sanitation, housing, electricity, public safety)  What are the challenges to the wellbeing of people in this area?  How are these challenges addressed?  Are you aware of the PEG program that is implemented in this area?  What do you know about this program? Probe for activities, beneficiaries- gender and other qualifying factors, benefits ECONOMIC INFORMATION  What do most people do for income in this area?  Were these the same activities that people engaged in before the PEG program? Have there been any changes? Please explain  Are there differences in the types of jobs that people do due to gender? Clan? Political affiliations? Was the situation the same before the introduction of the PEG program?  Are there differences in wages based on gender? By Clan? Related to Political affiliations? Was this the same before the PEG program?  What opportunities exist for earning additional income? Please compare the situation now and before the PEG program.  What are the major expenses for households (food, education, health, transportation etc)? Please do a comparison of the major expenses for households before and after the introduction of the PEG program.  Do people leave to look for work elsewhere?  What type of work do people migrate for?  Where do they go?  Which household members migrate? And why? ( probe for gender, age etc)  Do migrants remit? How important are the remittances to the household income?  How was the situation before the PEG program?  Are you aware of any institutions/ organizations that you can borrow money from? PAGE 59 – PEG Mid-Term Performance Evaluation  If yes please list the various sources of credit  Out of the listed, how many are giving credit specifically to women? Have you ever tried taking credit with the organizations/ institutions? Please explain  Describe the terms of credit and repayment  Has the number of borrowing institutions changed after the introduction of the PEG program? What of the terms of credit and repayment? Please explain.  How is credit used? To meet every day needs? Investment? How was credit used before the introduction of the PEG program? SOCIAL INFORMATION  Are you aware of any organizations that work with the communities in your area?  Type of these organizations( nongovernmental organizations, religious organizations, government organizations, community based organizations)  What are their activities?  Who benefits from these activities?  Describe any positive or negative change that has resulted from these activities?  Has there been any change on the number, activities and beneficiaries of the organizations after the introduction of the PEG program? Please explain your answers  Do social conflicts occur in this community?  What are the causes of the social conflicts?  Has there been any change as a result of the PEG program? Please explain your answer.  Do households rely on other households when they run out of food or income? Please explain  Do some groups have more social support than others? If yes, who? And why? Was this the situation before the PEG program? Please explain. EDUCATION AND HEALTH SERVICES  What sources of education exist in your community (private or public)?  Who has access to education?  What are the literacy and drop out trends (including gender differences)?  Affordability of education in the area  What was the scenario before the introduction of the PEG program?  What types of health facilities are available in this area?  Who has access to these health facilities?  What are the costs to seeking care( consultation, buying drugs, time taken to get treated etc)  What are the most common diseases in this area? How frequent are the diseases in the area?  Are community meetings held to discuss health issues, education issues, hygiene, and sanitation? If yes when were the recent meeting held? What was the meeting about? Any changes that have come as a result of the meeting?  What was the scenario before the introduction of the PEG program FOOD SECURITY  How many months of the year can households meet their consumption needs from the money they earn or the food they grow? Has this been the case before the PEG program? Please explain  Do people plan for eventual food shortages?  How and where is food stored?  Who is responsible for food storage in the home?  Any changes that came up as a result of the PEG program?  What are the main foods that are preferred by households? PAGE 60 – PEG Mid-Term Performance Evaluation  What substitute foods are available when food is in short supply?  Do people have problems accessing markets (time, distance, availability of transport, shop keeper discrimination etc)? please explain  Do people have problems purchasing food or basic necessities?( availability in the stores, affordability)  What are the priorities for your community to improve food security?  Do people get together to discuss food security in this area?  Who participates in these meetings?  Who leads in these discussions?  What was the scenario before the introduction of PEG program in relation to number of meals per day (for children, adults), main preferred foods, available substitutes, accessibility of the markets, and purchasing power for food and / or basic necessities? SECURITY  How safe are you in this community? How safe were you before the introduction of the PEG program?  What are the major causes of insecurity in this area?  Who is responsible for ensuring the security of the community? Households?  What are the community priorities to improve security in this area?  Do people meet to discuss security issues in this area?  Who participates in these meetings?  Who leads these discussions?  How effective have these meetings been in improving security in this area?  Do you have community activities that are intended to promote peace in the community?  What are these activities?  Who are the main participants in these activities? Probe for Gender, age  How effective have these activities been in promoting peace? Please explain.  Any changes on the overall security of the area that came as a result of the PEG programme? Please explain your answer. PAGE 61 – PEG Mid-Term Performance Evaluation ANNEX 9: DISCLOSURE OF CONFLICTS OF INTEREST Name Dr. Gary Woller Title Evaluation Consultant Organization Woller & Associates Evaluation Position? Lead Author Evaluation Award Number (contract or other instrument) USAID Program(s) Evaluated (Include program name(s), implementer name(s) and award number(s), if applicable) Programs Evaluated: PROFIT-Zambia GMED-India SAIBL-South Africa FSD-South Africa NEO-Georgia I have real or potential conflicts of interest to disclose. X Yes If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the program(s) being evaluated or the implementing organization(s) whose program(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose programs are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the program(s) being evaluated, including involvement in the program design or previous iterations of the program. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose program(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose program(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular programs and organizations being evaluated that could bias the evaluation. Dr. Woller has an extensive work history with the PEG Executing Agency DAI having been contracted to work on variety of programs with DAI over time, including the following:  Managing the PSD/IAI initiative under AMAP  Designing an M&E system for FNMD program in Palestine  Designing an M&E system for the PMDP program in Palestine  Designing an M&E system for the MADE program in Nigeria  Designing an M&E system for the PIND Foundation in Nigeria  Advising on M&E system design for the MSME program in Cambodia  Conducting impact evaluation of the PROFIT program in Zambia  Conducting impact evaluation of the GMED program in India PAGE 62 – PEG Mid-Term Performance Evaluation I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 11 2014 PAGE 63 – PEG Mid-Term Performance Evaluation ANNEX 10: THE M4P APPROACH M4P is an approach to developing market systems which seeks to see them function more effectively, sustainably and beneficially for poor people, building their capacities and offering them the opportunity to enhance their lives. Applicable to development and other agencies working in both economic and social fields and who pursue public goals, it is an approach defined by a number of important characteristics, such as those described below. Analysis linked to action: M4P is an approach to development that provides guidance not only about understanding how the poor function in market systems (analysis) but also about how to bring about effective change (action). Analysis identifies the underlying constraints inhibiting market systems and concentrates on addressing these. Market system focus: The M4P approach recognizes that the lives of the poor are inextricably linked to the functioning of the systems around them. Too often, poorly functioning market systems – uncompetitive and unresponsive to producer, worker and consumer needs – have a disproportionately negative impact on the poor, who lack the resources to overcome such dysfunctions. The M4P approach fosters understanding of the functions and players within market systems and how these can be strengthened in order to better serve the needs of the poor. The approach targets interventions at critical weaknesses in the market system, building capacity within the system to enable key players to work more effectively for the benefit of the poor. Sustainability: M4P is about creating the foundation for lasting change where market systems are better equipped to respond to future needs and priorities. It acknowledges that the lives and livelihoods of the poor are continually adapting to the changing environment around them and that the poor need solutions that adapt with them. The M4P approach recognizes that the process through which market system constraints are identified and addressed is as important as the solution itself. If the system, its functions and players, can be equipped to meet future challenges and continue to meet the changing needs of the poor then impact is sustained, rather than short-lived or dependent on further injections of aid. Large-scale impact: The M4P approach explicitly aims to reach large numbers of poor by harnessing the dynamism of the market system to achieve scale and sustainability. It does this by alleviating the constraints that restrict equitable participation and better terms of access to all those who engage with the system. Through a rigorous analysis of the role of the poor within market systems, the M4P approach identifies and targets those markets that are of the greatest importance to the poor and intervenes to trigger improvements, which will have the greatest and most durable impact on reducing levels of poverty. Facilitative role: Within the M4P approach, facilitation is a central element to achieving sustainability and large-scale impact. Lasting, systemic change requires that important market functions are embedded within the system, performed by market players with the capacity and incentives to undertake those roles in the long term. An M4P programme aims to stimulate private and public sector players to take on new (or adapted) functions, while avoiding becoming an active market player itself. The role of the development agent (or agency) in the M4P approach is explicitly catalytic – working towards a future vision of a market, which does not require aid-funded support and ensuring that any intervention is guided by a clearly defined exit strategy. Source: M4P Hub. For more information, see www.M4Phub.org PAGE 64 – PEG Mid-Term Performance Evaluation The above characteristics define M4P and clarify its difference from traditional development approaches, as summarised in the table below. Characteristic M4P Approach Traditional Approach Analysis linked to action Combines coherent rationale, analytical framework and guidance for action applicable in different market systems. Different frameworks and good practice guides. Often disconnect between analysis and actions. Market system focus Priority and focus on changing systems, driven by detailed understanding of underlying constraints (causes). Focus on addressing firm or farm￾level problems rather underlying causes of those problems. Sustainability Priority is to identify and address system constraints that can leverage wider change ‘crowding-in’ other players and activity. Limited scale of impact because of reliance on direct support. More impact needs continual infusion of more aid resources. Large-scale impact Explicit consideration of the alignment of roles and players (and incentives and capacities) in market systems, both now and in the future. Limited view of sustainability–often of direct transactions only rather than the wider market system. Sustainability a secondary concern. Facilitative role Explicit consideration of the alignment of roles and players (and incentives and capacities) in market systems, both now and in the future. Often a very direct provider role in market that, while achieving in short￾term, distorts market systems and increases their reliance on aid. Many of the distinctive features of M4P are manifested in its ‘systemic’ character. Traditional approaches commonly ‘miss’ the means through which growth and access is achieved, e.g., they seek to provide directly to producers/firms. Traditionally, the instinct of donor agencies’ has often been to ask the question “what problems do businesses have and how can I solve them?” and not to ask the more relevant systemic questions: “why isn’t the market environment providing solutions?” to these and “how can I address the constraints that prevent it from effectively doing so?” Similarly, governments often throw money at problems, provide direct solutions or invite development agencies and NGOs to solve the problem for them, rather than identifying and addressing underlying systemic causes of problems. By addressing the market systems within which producers operate, M4P focuses on underlying constraints and gaps and therefore the potential triggers for sustainability and large-scale impact. M4P permits agencies and governments to raise the limit of their ambition. For example, in relation to services and inputs for farmers: while a conventional program might concentrate on providing goods directly or subsidizing them, M4P focuses on improving the systems that provide inputs. Putting M4P into practice also requires different operational capacities and approaches – for example in relation to skills, relationships, intervention design and implementation and program set-up. PAGE 65 – PEG Mid-Term Performance Evaluation U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523