MID-TERM EVALUATION OF THE COMMUNITY LIVELIHOODS PROJECT (CLP) APRIL 2012 April 25, 2012 This publication was produced for review by the United States Agency for International Development. It was prepared under the Yemen Monitoring & Evaluation Project by International Business and Technical Consultants, Inc. (IBTCI) YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) i MID-TERM EVALUATION OF THE COMMUNITY LIVELIHOODS PROJECT (CLP) APRIL 25, 2012 International Business & Technical Consultants, Inc. 8618 Westwood Center Drive Suite 220 Vienna, VA 22182 USA Contracted under RAN-I-00-09-00016, Order Number 1 Monitoring and Evaluation Project DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Authors: Kris Merschrod and Arthur Warman YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) ii ACKNOWLEDGEMENTS The evaluators would like to thank a number of participants in the evaluation. The Yemen Monitoring and Evaluation Project provided full logistical support and various primary and secondary documents. The USAID/Yemen staff was most accommodating on material needs, ad hoc meetings and interviews, and in providing guidance for the evaluation. Finally, the Community Livelihoods Project staff in Yemen and Creative Associates staff in Bethesda were instrumental in opening their doors, facilitating meetings, patiently being interviewed, generously providing candid opinion, and providing the needed information for the evaluation. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) iii Table of Contents Acronyms ................................................................................................................................................................................... iv Executive Summary ................................................................................................................................................................. v 1. Introduction ......................................................................................................................................................................... 1 1.1 The Yemen Context: The Four Dimensions ...................................................................................................... 1 1.2 Development Challenge and USAID’s Approach ............................................................................................. 3 1.3 Development Hypothesis ......................................................................................................................................... 3 2. Background and Historical Narrative........................................................................................................................ 4 2.1 Phase One: Change in Implementation Strategy (July – December 2010) .......................................... 4 2.2 Phase Two: New Management (February – August 2011) ........................................................................ 8 2.3 Phase Three: Building Up (September – December 2011) ...................................................................... 12 3. Scope of Work and Methodology .............................................................................................................................. 15 3.1 Purpose and Key Questions .................................................................................................................................. 15 4. The Development Hypothesis and Geographical Coverage; Findings and Conclusions ...................... 17 4.1 The Development Hypothesis .............................................................................................................................. 17 4.2 Geographical Coverage ........................................................................................................................................... 19 4.2.1 Questions about the Expansion of CLP .................................................................................................... 19 4.2.2 Effectiveness in Regions and Peri‐Urban Areas ................................................................................... 20 4.2.3 A Regional Office for Aden and Recommendations for Other Governorate Offices .............. 22 5. Systemic and Sector Assessment: Findings and Recommendations .......................................................... 23 5.1 Grants and Costs Associated with Grants ........................................................................................................ 23 5.2 Systems and Sectors ................................................................................................................................................. 26 5.3 Systems .......................................................................................................................................................................... 27 5.3.1 Community Mobilizers ................................................................................................................................... 27 5.3.2 Grants Management (GMS) ........................................................................................................................... 30 5.3.3 Monitoring and Evaluation ........................................................................................................................... 32 5.4 Sectors ............................................................................................................................................................................ 35 5.4.1 Agriculture ........................................................................................................................................................... 35 5.4.2 Economic Growth ............................................................................................................................................. 40 5.4.3 Governance and Civil Society ....................................................................................................................... 42 5.4.4 Health ..................................................................................................................................................................... 45 5.4.5 Education ............................................................................................................................................................. 48 6. Overall Evaluation Conclusions ................................................................................................................................. 52 7. Overall Evaluation Recommendations .................................................................................................................... 55 8. Best Practices .................................................................................................................................................................... 58 Annexes ..................................................................................................................................................................................... 59 Annex I – Scope of Work ..................................................................................................................................................... 60 Annex II – Evaluation Work Plan .................................................................................................................................... 71 Annex III – List of Interviews ............................................................................................................................................ 77 Annex IV – References ......................................................................................................................................................... 85 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) iv Acronyms AO Assistance Objectives AOTR Agreement Officer’s Technical Representative BHS Basic Health Services Project CDF Civic Democratic Initiatives Support Foundation CoAg Cooperative Agreement CLP Community Livelihoods Project Creative Creative Associates International, Inc. DEC Development Experience Clearinghouse DG Director General GHO Governorate Health Office IDP Internally Displaced Person IOM International Organization for Migration IRs Intermediate Results MAI Ministry of Agriculture and Irrigation M&E Monitoring and Evaluation MEPI Middle East Partnership Initiative MMT Mobile Medical Team MoE Ministry of Education MoLA Ministry of Local Administration MoPHP Ministry of Public Health and Population MoPIC Ministry of Planning and International Cooperation OTI Office of Transition Initiatives PDC Yemen Partners for Democratic Change PMP Performance Management Plan PPM Private Provider Midwife PRA Participatory Rapid Appraisal RFA Request for Applications RGP Responsive Government Project ROYG Republic of Yemen Government SFD Social Fund for Development SOW Scope of Work TAK Teacher Aid Kit ToT Training of Trainers USAID United States Agency for International Development YASAD Yemeni Association for Sustainable Agriculture Development YEFE Yemen Education for Employment Foundation YFCA Yemen Family Care Association YMA National Yemeni Midwives Association YSI Yemen Stabilization Initiative (OTI Project with IOM as Implementer) YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) v Executive Summary Since June of 2010 Creative Associates International, Inc. (Creative), through Cooperative Agreement (CoAg) 279-A-00-10-00032-00, has been implementing the Community Livelihoods Project (CLP). Three modifications have been made during this 18 month evaluation period, all of which, except Modification No. 02, were for incremental financing. Amendment No. 02 altered the agreement’s scope of work, but did not extend the Award from its original 36 months with two one-year options. Through USAID/Yemen’s Yemen Monitoring and Evaluation Project (YMEP) implemented by International Business & Technical Consultants, Inc. (IBTCI), a mid-term evaluation was requested by the Mission. The field work for this report was carried out in January and February 2012 and the evaluation review period covers July 1, 2010 through December 31, 2011. Per the CLP mid-term evaluation scope of work, “the purpose of the mid-term performance evaluation is to assess the implementation, effectiveness, and progress of CLP in Yemen. The findings of the evaluation are to be used by USAID/Yemen to inform whether the program is on track to achieve programmatic and fiscal goals and targets and whether the initial design of the project addresses the overarching USAID strategy in Yemen.” The CLP mid-term evaluation scope of work posited three specific areas of interest or questions for the evaluation: 1) The primary evaluation questions consisted of:  Has the CLP implementation mechanism of issuing grants to local awardees been an effective strategy for achieving program results?  Have the efforts to include women and youth in the program achieved expected results?  Has the program achieved all of the other expected results at mid-term?  Has the program been cost effective? 2) USAID sought recommendations in the following areas:  Recommendations for moving forward with regard to partnership with international and local sub-grantees;  Recommendations for field teams to enhance/ensure community engagement;  Recommendations regarding enhanced support for field operations in CLP governorates moving forward since CLP expatriate staff are currently located at the main CLP office in Sana’a;  Recommendations as to whether CLP expatriate staff should be located in field offices where security is permissive to enhance grant implementation performance, oversight and effectiveness;  Recommendations to improve CLP grant disbursement rates moving forward;  Recommendations to improve the ratio of grants/project disbursement to CLP overhead and support costs; and  Recommendations to improve CLP’s financial processes and procedures with regard to grant disbursements. 3) Additionally, USAID requested that the evaluators provide an accurate historical narrative of the first 18 months of the program. The methodology used to execute the evaluation scope of work included a complete document review of CLP and other sources; over 100 interviews with CLP personnel, USAID/Yemen personnel, CLP implementing partners, government officials, and counterparts; and email interviews with past CLP and YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) vi USAID/Yemen staff. The overall framework was to reconstruct the evolution of the project and its systems, and provide a sector by sector assessment of CLP implementation based on grants’ records, interviews with personnel executing the sector strategies, and the opinions of CLP implementing partners. FINDINGS 1. The start-up delay of six months can be attributed to what the evaluators term “the protocol dimension.” The Mission Director and CLP Chief of Party (COP) had to adhere to the approval process established with the Ministry of Local Administration (MOLA) in the 2009 Assistance Agreement1 in which a memorandum of understanding with the Mission and the Republic of Yemen Government (ROYG) authorizing field work must be obtained before contact could be made with the governors. The six-month delay in achieving this was beyond the power of the Mission and CLP to overcome.2 2. The protocol dimension had a negative impact on the rate of grant making for the small projects, grassroots approach and precipitated the large project funding ahead of small grants. 3. The overarching USAID strategy to test a “development-in-the-face-of-violence” hypothesis in Yemen changed after the design stage3 . About half way through the implementation stage it evolved into a more classical development model. 4. The community-based development strategy for the identification of projects as outlined in CLP’s CoAg was not implemented, in large part because of the protocol dimension that delayed the hiring of field personnel for six months. In addition, the detailed community-based needs assessments, that the CoAg outlined, had already been done by the Yemeni Social Fund for Development and for this reason the strategy was not needed to reach the goal of generating small, quick impact projects. 5. The original theory of development and accompanying design called for in the Request for Applications (RFA) and CoAg was not appropriate for the changing context of Yemen. The theory was based on a social psychological understanding of drivers of instability and ignored the main instability in the country consisting of social movements or struggles between the central government and regional separatists as well as power struggles between tribal leaders. 4 6. CLP did not meet the majority of its 2010 and 2011 indicator targets and obligations under the CLP CoAg through the first half of the Award. 7. The start-up period issued very few grants and, accordingly, had low disbursement rates. 8. Operating costs outstripped the grant-making and disbursement rates because of the protocol dimension and also because of the instability during the Arab Spring. That is, staffing continued but they could not implement what would have been activity and grant funding causing the accumulated costs of operations and overhead to be disproportionately high. 9. CLP will not meet the financial targets outlined in the CoAg without significantly scaling up grant activities to make up for the delayed start-up. 1 USAID RFA 279-10-006 Community Livelihood Project p.6 2 Community Livelihoods Project Quarterly Report January – March 2010 p.4 3 Consider the design stage to be the period including the USAID Mission Strategy Paper of 2009, the writing of the RFA and through to the CoAg in 2010. 4 One aspect that the evaluators could not weigh, but believe, is that the fear and focus on Al Qaeda of the Arabian Peninsula (AQAP) distracted the theoretical planning from the main drivers of instability, i.e. the factions in the structure of Yemeni society, and, instead, led to the idea that local, individual complaints, would provide subversive external forces with an opportunity to gain a foothold and destabilize the central government. (This belief was confirmed by an anonymous USAID source –April 2012) YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) vii 10. Creative did not assign corporate priority to establishing the basic M&E, Community Mobilizer and Grants Management systems and providing the appropriate staff needed for implementing the CoAg. This in turn contributed to the failure of CLP to achieve the targets and obligations of the first 18 months of the award as outlined in the CoAg. 11. By the time of this evaluation CLP had not set up a system for supporting the Community Mobilizers in the field and to effectively receive, review and respond to their grant proposals. 12. From October to December 2011, Creative assigned greater priority and increased the number and quality of CLP personnel to help make the remaining period of the Award more successful. 13. There is mutual agreement between USAID/Yemen and CLP that communication between them was not satisfactory. Nevertheless, both organizations, over the first eighteen months of the project, have tried to improve communication and a significant but frustrated effort was made just prior to the evaluation. 14. The geographic coverage of activities matches the original eight governorates noted in the CoAg, however, when the 2011 Arab Spring was manifest as an urban phenomenon in the capitals of governorates as well as in the capital city, the USAID policy response was to add emphasis on urban areas. Both the Mission and CLP were not prepared theoretically, as well as with specialized personnel and activities, to respond in urban focused development. CONCLUSIONS 1. Although there were significant mitigating circumstances that impacted CLP implementation, including: the protracted “Arab Spring” uprisings and subsequent instability, forced evacuations, changes in Mission and CLP personnel, among others, these factors do not provide sufficient basis for CLP’s failure from July 2010 through December 2011 to assign corporate priority to CLP in two areas that significantly affected the entire program: assignment of sufficient personnel and developing systems (grants management and grant making) that would drive the program toward success. 2. As of December 2011, the CLP M&E system is now well staffed with the necessary systems (i.e. software, forms and procedures) installed to generate more substantial and accurate reports for USAID and CLP management. Additionally, CLP has provided M&E technical assistance to grantees so that grantees can fulfill their contractual M&E responsibilities. 3. CLP’s Economic Growth sector is only now (February 2012) adequately staffed and prepared for the remainder of the CoAg. Increasing the staff level has enabled a retooling and redirection of the Economic Growth sector. The movement toward larger, international Non-Governmental Organizations is the growth model for this sector as well as the others. The potential growth will depend upon the Annual Program Statement (APS) that was issued by CLP in January 2012. 4. CLP’s Agriculture sector is relevant to the goal of increasing population trust in the ROYG. The third party evaluation found that rural male and female participants in the activities were enthusiastic about the program, and the ROYG Ministry of Agriculture personnel expressed the opinion that this CLP sector effort was the key to their rehabilitation in rural centers . However, the focus on urban agriculture remains to be tested as a means toward meeting urban needs – as an intervention targeting urban unrest this activity is not logically founded. 5. CLP’s Governance and Civil Society component lacks a strategic vision as the emphasis on youth activities is more closely associated with the Economic Growth Component than with community-based civil society development. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) viii 6. All three CLP Health sector activities in place at the time of the evaluation – health and supplies for Internally Displaced Persons (IDPs), Support for Mobile Medical Teams and the development of the Private Provider Midwives – were well targeted and appropriate for this type of program. Moreover, the expansion of the health initiatives is well within the budget of the CLP health sector and the managerial capacity of both CLP and the implementing partners. From the perspective of the Ministry of Health, and also the private provider midwife approach that combines health service with the development of economic opportunities, it is a successful sector, but could be more successful by scaling it up, expanding the type of interventions and moving into new governorates. 7. Involving larger international NGOs will allow for the distribution of larger obligations and absorption of subcontracting the management responsibilities for CLP’s Education sector grant implementation. With the newly arrived expatriate Sector Leader, the sector is undergoing a wholesale revamping in terms of staff, work plans, vision, anticipated activities and targets for the remainder of CLP. 8. The evaluators did not find identified linkages to other activities within the education sector. USAID/Yemen identified the lack of vision, targets, time and geography as key missing variables within CLP’s Education Sector. The evaluators agree with USAID/Yemen that the lack of capacity within the Education Sector was the critical issue facing the sector during the 18-month period being evaluated. 9. USAID/Yemen and CLP are frustrated by a communication breakdown, which is one of the most serious problems facing the implementation of the livelihoods program. If communication cannot be resolved, then the ability of the Mission and CLP to have a coherent and productive strategy going forward during the period of political transition and new USAID strategy is in jeopardy. 10. It would be more efficient for USAID to continue with Creative Associates than to start the program over. RECOMMENDATIONS Based on the findings and conclusions, the recommendations begin with two clear and immediate actions. Depending on those actions, the remaining recommendations can be considered. 1. USAID/Yemen should decide on the feasibility of continuing with the CLP. 2. Creative and USAID/Yemen need to develop and maintain effective communications channels and develop a unified vision of how to implement the program to achieve the stated purposes. If the decision is not to proceed with Creative as the implementer, then the alternative would require a vision and strategy that is being prepared by the Mission at this time for the 2012-2014 “Transition Period” that roughly corresponds with the remaining period of the CLP Award. The evaluators recommend going forward with Creative as the implementer of the CLP because it is the more efficient approach toward immediate action for achieving the goals of the Transition Period than going through a process of a new RFA and decision-making for a new award. Going forward, however, will require that the Mission and CLP put the past complications behind them, focus on accomplishments, build upon the strengthened CLP team, and agree on a new geographic approach and desired results for the remainder of the Award based on the yet-to-be-completed Mission Strategic Plan. The process of going forward requires more than a clearly articulated and structured plan; it requires team building between USAID and CLP. The evaluators made a strong recommendation during the February 13, 2012 in-person out-briefing with USAID/Yemen, and repeat it here because of the gravity of the communication situation, i.e. that an YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) ix independent third-party facilitated review and planning retreat5 be conducted involving the executives and staff of both USAID/Yemen and CLP to try to bring them together so that they can implement the transition program and achieve the goals that will be in the new Mission strategy.6 If the decision is to proceed with Creative as the CLP leader, and if USAID/Yemen and CLP can emerge from the strategic planning and team building process, then the following recommendations should improve the implementation of the CLP CoAg. 1. The most urgent managerial need is that Creative implements the new Grants Management System (GMS). In view of the goal of integrating CLP sectors to increase overall program and geographic impact, it would be useful to reduce the number of sectors. For example, given the nature of their respective grant activities, Economic Growth could be combined with Governance and Civil Society.7 2. Because of the significance of the governorates around Aden, and the importance of the South in the geo-political history and current tension in Yemen, the CLP Aden Office should be elevated to that of a full project office led by an experienced expatriate staff person. 3. Quick action should be taken on the review and approval of the pending projects submitted to the CLP office in Sana’a over the past 15 months. The role of the Community Mobilizers in CLP should be reconsidered. They are knowledgeable about the main cities and large towns of the governorates, and that focus would fit with the urban focus that USAID has emphasized in response to the Arab Spring. 4. Other CLP regional or governorate offices could be built upon empowered and trained Yemeni Mobilizers to facilitate the integration of CLP sectors in those areas and to strengthen links to governors and representatives of the ROYG ministries in those governorates. A low expatriate profile will be necessary to protect the security of those offices and also the expatriates themselves. 5. The system for providing and liquidating travel and expense advances to the mobilizers should be reviewed and corrections made immediately. Material and supply needs for their offices should be assessed and strengthened. 6. Moving forward, more emphasis should be placed on rapidly expanding the scope and size of CLP grant activities that can help the program achieve the results required under the CLP Cooperative Agreement as well help to adequately respond to the needs of the new transition strategy that USAID/Yemen is preparing. This could be accomplished through: the expansion of existing grants and the awarding of new, larger, grants; large subcontracts to proven international implementers capable of supporting CLP’s program success; and CLP direct implementation and proven integration of grant activities. The APS process approved by the Mission in late 2011 and begun in January of 2012 holds promise based on the preliminary responses from NGOs. 7. CLP’s M&E resources should be focused on new grants and grantee ability to do baselines and indicator measurement. M&E and compliance officers should continue to try to reconstruct the 5 Not unlike Missions do with OTI’s third party facilitated bi-annual Strategic Review Sessions when the dynamics of country contexts are similar to that of Yemen and require a rolling planning approach – more frequent than an annual plan and more intensive than the exchange of quarterly reports. 6 After the evaluators wrote the draft of this report, it was learned that such a facilitated review had been proposed by CLP and a contract had been approved by the Mission for the last quarter of 2011, but that it had been postponed. 7 Ear marks or funding lines may complicate the title and purpose of the sector. Nevertheless, as is noted later in this report, it was found that the Agriculture Sector was implementing a Health Sector activity. Is this an indication of integration? YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) x past and enter that information into the database as sector-specific baselines. Community Mobilizers could also be incorporated into the field verification work. 8. The April 2011 Vision Statement for the Economic Growth Sector is outdated. A new vision statement is required. While its focus on youth oriented initiatives is impressive in many ways, it is unclear how these will roll up into a coherent package or whether it should remain a stand￾alone sector. The evaluators believe it should be merged with the Governance Sector because the latter’s portfolio is strongly youth oriented and not particularly governance oriented at the time of this evaluation. 9. Governance and Civil Society should define a vision that responds to a strategic focus. Rationalization of staffing needs against a strategic vision is highly recommended. 10. The CLP health activities are beyond the pilot stage and should be expanded. The implementing partners are well prepared to scale up to cover all of the target governorates. They could also expand and add an urban focus so that the USAID interest in cities such as Sana’a, Taiz, and Ibb could be covered quickly. 11. Going forward with the Education sector must be based on the building up of current staff and projected skill sets now appearing within the sector. Using international NGOs to scale up the sector in terms of grants and activities implemented is warranted and supported if education remains a major thrust under CLP. This sector holds great promise for the urban focus. 12. The Education and Health Sectors should take advantage of the integration opportunity in the Southern Region of Yemen where IDPs have congregated in schools displacing students and causing a crisis in education (triple shifts) and as well as a demand-for-services crisis in the health facilities of Aden. 13. Finally, the social psychological development theory with which the project began should be set aside formally. 8 In its place a sociological perspective of development should be employed that recognizes the tribal and regional separatist movements as the drivers of instability in Yemen. This perspective will require mediation that is beyond the USAID mandate, but concurrently USAID, through CLP, can achieve its strategic objectives of stability and poverty alleviation by providing incentives for strategies such as the decentralization9 of authority and responsibility. This effort would require rekindling of the original idea in the RFA and CoAg that the Responsive Governance Project (RGP) and CLP strategies would be coordinated. BEST PRACTICES The evaluators found that during the implementation of the CLP some best practices were employed and should be highlighted. Some may be suitable for use as success stories. Building on an established Mission portfolio of so-called “legacy projects”10 allowed the project a quicker start given the challenges posed by the protocol dimension. That is, had CLP held to the original plan of emphasis on community-based small projects first and then a long term classical development portfolio, it 8 Interviews with the Mission Director, the AOTR and previous Yemen Mission staff confirmed that the testing of the original theory of community based development in the context of conflict is no longer a goal of the Mission. 9 Decentralization has been a theme of international agencies in Yemen for approximately a decade, and the USAID Mission, when drafting the RFA, had hoped that the concept of “co-responsibility” between the Governorates and the Central government could be facilitated through projects such as CLP and RGP. The goal was to reduce tensions between the regions and tribal leaders. (Source: previous USAID personnel – April 2012) 10 USAID projects funded prior to the start of the current 2010-2012 Yemen Mission Strategy. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) xi would have taken much longer to gain traction in the field. But the best practice is to build upon proven USAID programs. Collaboration between the project and the Mission-funded third party M&E project brought together CLP expertise and YMEP expertise so that the M&E systems of sub-grantees would be appropriate. Gender inclusion based on understanding the roles of women and the expertise of women professionals was found in the hiring practices of CLP at all levels (internal hiring and in sector implementation). In the agriculture sector, training was based on the sound knowledge of the role that women play in that sector. This resulted in those women designing training for other women in animal husbandry and vegetable production to build on their roles in society. The health sector midwife training was, of course, female focused, but the outstanding practice was the private provider midwifery (PPM) focus. For the PPM approach, CLP deserves special mention because they have been promoting an innovative way to improving midwifery by means of the mapping technique and providing equipment so that midwives have enhanced prestige in the community leading to improving not only infant/maternal care, but also improving the economic status of midwives. This holistic approach to a profession is much better and more sustainable than the usual midwife training programs found in developing countries. By focusing on the sustainable clinic aspect as well as the health delivery or technical aspect, CLP integrated the health and economic development sectors. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 1 1. Introduction The Yemen Monitoring and Evaluation Project (YMEP), implemented by International Business & Technical Consultants, Inc. (IBTCI), was requested to conduct this mid-term evaluation. This is a performance evaluation11 in response to the scope of work (SOW) prepared by the United States Agency for International Development (USAID) and submitted to the evaluators on 23 January, 2012 and slightly amended thereafter as explained in Annex I. The subject of the evaluation is the implementation and execution of the Community Livelihoods Project (CLP) by Creative Associates (Creative) from its inception in July 2010 through December 31, 2011. Due to the changing political context, facts on the ground, and corresponding USAID needs, this Cooperative Agreement 279-A-00-10-00032-00 (CoAg) has been through three distinct phases that will be described in the historical narrative. 1.1 The Yemen Context: The Four Dimensions There are four dimensions that must be taken into account when considering the environment in which CLP is being implemented. These serve as background for the historical narrative which follows. The first, general dimension is that Yemenis live in poverty: Living conditions for most of the 22.2 million Yemenis remain difficult with 46.6 % of the population living on less than $2 per day.12 75% of the population is under 30 years of age. Lack of infrastructure leaves much of the rural population living in relative isolation; only about 40 percent of the total population has access to electricity; reliant upon under-developed social services and a difficult governance environment. The situation is particularly dire for women: unemployment is high among young educated women13 and the labor force participation rate for women is among the lowest in the region (21.7% for adult women compared to 65.9% for adult men).14 A second dimension consists of the structure of governance and tensions between sub-national affiliations such as political parties, tribes, classes and regional differences that include, but are not limited to, the aftermath of the 1990 unification and the 1994 civil war between north and south and other historical centripetal forces such as tribal contestation and competition for control within the central government and the Houthi rebellion15, pulling at the concept of a national development project. The third dimension is protocol – that sometimes delicate ritual that Mission Directors and Ambassadors must negotiate in order to maintain cordial intergovernmental relations and come to agreement on the implementation of development projects. At first glance this may appear to be an unusual contextual theme, but, as will be explained, protocol consumed the first six months of CLP and delayed engagement with the governors. This put off the planned field work and implementation of the community-based strategy, and delayed the grants program, causing a very high cost of overhead compared to grants 11 Performance evaluations focus on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual. (USAID 2011. Evaluation Policy, Bureau of Policy, Planning and Learning, Washington, DC. p. 2) 12 Republic of Yemen, Ministry of Planning and International Cooperation/UNDP, Yemen Report, Millennium Development Goals, 2010. 13 Save the Children (2009), Youth Empowerment in Yemen: Baseline Report. 14 Yemen Performance Management Plan (PMP) FY 2011 – FY 2013, p. 4 15 For an excellent summary of the Houthi rebellion from 1964 to their role in the conflict in Sana’a in March of 2011 search “Houthi Rebellion” in Wikipedia. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 2 making and disbursement. This contextual dimension is also partially the reason that the legacy projects took an early role in implementation. The usual chain of events is that USG representatives and national governmental counterparts identify an area of development that can benefit from USG assistance programs. The general idea is scoped out and a bilateral agreement is signed. A Request for Applications (RFA) is drafted and published, contractors submit proposals and are identified, and eventually a cooperative agreement signed. Once the contractor sets up the project in country, the project COP and the Mission meet with the local government officials, in this case the Ministry of Local Administration (MoLA), to establish clearances and the finer details in a memorandum of understanding. Then project personnel interact with the indicated government or private sector institutions to implement. In the case of CLP, according to the RFA16, the following was stated: On September 16, 2009, USAID signed a $31 million bilateral assistance agreement with the ROYG through its Ministry of Planning & International Cooperation (MoPIC), with an anticipated three year assistance total of at least $121 million subject to the availability of funds. The agreement between the two governments reflects agreement on the principles of the 2010-2012 USAID/Yemen Strategy focused on increasing Yemen's stability through community-level interventions and national level governance initiatives. The agreement reflects an explicit understanding and tacit approval by the ROYG that stabilization type efforts are necessary at this time in Yemen's history. Furthermore it was stated: The USAID-ROYG relationship is strong. Ministers are approachable and available to meet with USAID. Ministerial and other senior level officials are generally cognizant of USAID’s efforts in the various sectors and appreciate the interventions implemented to date. During the strategy period, USAID and MoPIC agreed to hold quarterly coordination meetings and “performance reviews”, which will allow the ROYG to maintain a level of oversight of USAID initiatives and to ensure that USAID and its implementers maintain close ties with the ROYG's donor coordination body. According to these citations, it appeared that the start-up of the CLP would proceed in a standard fashion. However, the quick start and easy access did not come to pass. The impact of this dimension on the lagged start-up in the field, grant making and the rate of disbursements will be discussed throughout the report. The fourth dimension is USAID’s perception of the relevant development paradigm17 in which the CLP would be designed. It follows logically from the RFA and is articulated in the 2011 USAID/Yemen Performance Management Plan (PMP). “Yemen has transitioned from a country where classic development assistance was appropriate into a country where stabilization assistance is needed, thus a different analytical framework, like the one proposed in this PMP, is appropriate for the assessment of stability assistance.” 18 This fourth dimension, the stability/development hypothesis, framed the Request for Applications (RFA) that USAID/Yemen developed in 2009 and the model that Creative proposed in response. The CLP Cooperative Agreement was to implement this paradigm.19 16 USAID, 2010 RFA 279-10-006 Community Livelihood Project, p.7 The RFA not only lays out the services expected of the bidders, it lays out the context under which the contractor expects to work. 17 USAID, September 2011, “The Development Response to Violent Extremism and Insurgency” Policy Paper. 18 Yemen Performance Management Plan (PMP) FY 2011 – FY 2013, p. 3 19 In the USAID Evaluation Guidelines of 2011 a project designed to test a paradigm would be called a Theory of Change. The original PMP was “a blueprint of the building blocks needed to achieve long-term goals of a social change initiative.” As will be shown, the community-based model with drivers of stability for targeting was not implemented. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 3 1.2 Development Challenge and USAID’s Approach The challenge that USAID/Yemen was facing by 2010, when the CLP was awarded, was political instability and a weakened state that might provide a foothold for Al Qaeda in the Arabian Peninsula. It is important to point out that before the CLP was designed, USAID/Yemen committed more than a decade of “classical” sector-specific development efforts in primarily the health, agriculture, and education sectors. Accordingly, there was a foundation or legacy upon which new programs could be built as well as elements of successful programs that the Mission hoped would be continued under the CLP award during and after the first phase of quick impact projects for stabilization. Before describing and evaluating the development model implemented by Creative and ascribed to by USAID/Yemen, it is important to describe the development hypothesis that is woven throughout the design. 1.3 Development Hypothesis Seldom is the development hypothesis stated so simply as per the following example from the scope of work that guides this evaluation of CLP: The development hypothesis of the 2010-2012 USAID/Yemen strategy, which guides CLP’s Cooperative Agreement and, therefore, implementation, is that addressing the development needs of Yemen’s underserved communities will help improve social stability by responding to the key drivers of community instability and citizen dissatisfaction. A higher level of livelihood and economic satisfaction will ideally minimize behaviors that create instability, such as extremism, tribal conflict and violence. This hypothesis has not yet been proven but remains an intent of the current USAID strategy in Yemen.20 In the Cooperative Agreement, the development hypothesis is expressed as an assumption:21 The underlying assumption of the CLP is that targeted development interventions can positively affect stability in Yemen. Implementation of this project will test this assumption. The project will directly support USAID/Yemen's 20I 0-2012 Strategy by contributing to the achievement of four of the five strategic intermediate results under two Assistance Objectives which serve as Results for this project: (l) livelihoods in vulnerable communities improved; and (2) governance capacities improved to mitigate drivers of instability. And, in reference to the design of the PMP: In this fashion, it is expected that the PMP will allow stakeholders to test the central hypothesis that improved livelihoods increases political and social stability. 22 As will be described in the historical narrative, the theoretical model and development hypothesis described above was overcome by unanticipated political events on the ground throughout 2011 and was not, therefore, no longer relevant to the context. 20 USAID Yemen SOW p.1; 21 Cooperative Agreement No. 279-A-00-10-00032-00 and subsequent modification #2, p33 22 Cooperative Agreement No. 279-A-00-10-00032-00, Annex II, p.1 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 4 2. Background and Historical Narrative The SOW for this evaluation stated, “Evaluators will first put together an accurate historical narrative of the first period, starting from the signing of the award. This will include all challenges faced, stoppages or blockages of the work and reasons why, how problems were overcome, and what other steps were taken to correct or change the work flow.” This is a key section of the evaluation for putting findings into perspective and understanding the conclusions and recommendations in the following chapters.   The Community Livelihood Project (CLP) was an outcome of the new Stability Framework for USAID/Yemen’s Mission Strategy. It was to be, and is, a companion project to the Responsive Governance Project (RGP) which is the national-level assistance project for improved governance that CLP was to mirror from the community upward. The third project of the Mission’s strategy was the Yemen Monitoring & Evaluation Project (YMEP), responsible for consolidating the overall monitoring and evaluation. All three flagship projects are large multi-year awards and formed the basis of the Mission Strategy for 2010 through 2012. In reviewing the events of the implementation within CLP when compared to the actual events on the ground (the protocol delay, the 2011 “Arab Spring” with resulting evacuations of Mission and CLP expatriate staff, multiple changes in personnel in both USAID/Yemen and CLP, etc.) it became clear to the evaluators that there are three distinctive phases that roughly correspond to the first eighteen months of CLP implementation. By narrative, these are: Change in Implementation Strategy, New Management, and Building Up. Each is a distinct period in the performance of CLP and is presented below to provide a reasoned view of CLP during the first eighteen months of the three-year CLP Cooperative Agreement. However, spanning the first two phases was the delayed start-up. This was the result of events on the ground consisting of the protocol dimension of six months and the Yemen version of Arab Spring, including evacuations for approximately six more months with a one or two month hiatus between the protocol dimension and the Arab Spring dimension (January and February 2011). 2.1 Phase One: Change in Implementation Strategy (July – December 2010) The original strategy for implementation of the CLP was based on the combination of two concepts: identifying the specific geographic areas23 where the drivers of instability24 are strongest and community￾based engagement25 in those identified areas accompanied by small quick-start grants. The initial design of the CLP as articulated in the RFA and CLP’s CoAg Award was based on grass-roots, community-focused, local organization implementation of activities that would mitigate or respond to the identified drivers of community, district, and governorate level instability that could expand into violent activity or resistance against the government in those areas or even at the national level. Accordingly, one of the first systems that should have been established by CLP consisted of hiring personnel experienced in community-based development techniques called Participatory Rural Appraisal (PRA).26 CLP should have hired Community Mobilizers for each of the eight target governorates to 23 C.f., Pp. 16-18 of the CoAg for the four steps to engaging the ROYG and the geographic areas of the stability assessments. 24 This is the essence of what has become “Stabilization theory” extracted from USAID experience, c.f., “The Development Response to Violent Extremism and Insurgency.” USAID Policy Paper September 2011. 25 C.f., Pp. 18-19 of the CoAg. 26 PRA is an approach used by extension agents or mobilizers to engage all strata of community in a self-appraisal of community needs and priorities. An objective beyond decision making on priorities is to establish committees or action groups who will manage future projects based on community development plans. The process is not so rapid because it usually takes months for the process to lead to actual project completion. An important outcome is increased pluralism within the community from the process of opening the decision making processes beyond the traditional leadership structure. Most published material on this came from the Sussex School of development in the YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 5 implement community-based PRA and to launch the small grants component, but Mission approval (because of the protocol issue) for recruiting personnel in five of the governorates was not granted until November of 2010. A complete analysis of this system is described in Chapter 4. However, at the same time that CLP was developing the strategy of using Community Mobilizers to facilitate an impact in rural areas and discussing the availability of mobilizers with the Social Development Fund, it was becoming apparent that engaging the Republic of Yemen Government (ROYG) and corresponding ministerial interests was not as easy as planned and described above in the protocol dimension. In the first quarterly report (July-September 2010), we find: As CLP focuses on community-based programming, the team has at times struggled with balancing the project’s mandate of working with local government and communities, and informing ministries at the national level that could create delays in project implementation.27 By the second quarterly report (October-December 2010) a substantial change in strategy is noted: A switch from the original grass-roots up approach initially designed for this program to closer interaction with the central ministries has been a bit of a challenge. However, CLP is up to the challenge. In cooperation, the program will find solutions that please the central ministries, as well as strive for community-driven projects. 28 This change in emphasis, to work closely with and through the ministries of the ROYG, also came as a response to the political and protocol structure that USAID agreed to with the ROYG I September of 200929. The CLP needed the approval of MOLA30 in order to work with governorates and each governor’s permission was needed to hire31 and work locally. As part of this history it should be noted that both the Mission Director and the CLP COP felt that they had to adhere to the protocol. The COP feared being declared a persona non grata and the Mission Director did not want to go against the afore mentioned September 2009 bilateral agreement in which the protocol had been formalized by “quarterly coordination meetings” and “performance reviews”, which will allow the ROYG to maintain a level of oversight of USAID initiatives and to ensure that USAID and its implementers maintain close ties with the ROYG's donor coordination body.”32 The decision by the Mission Director and the COP to carefully follow protocol was important for the future of CLP and other USG initiatives. Both the mission and the COP tried, monthly, to establish contact to convene meetings with the MoLA from July 2010 onward, but the agreed to coordinating meetings were put off by MoLA. CLP tried to work with the governorates before the end of 2010 and some governors were receptive informally because they had positive impressions and experience with the USAID legacy projects, e.g. health activities in Ma’rib and al-Jawf. Nevertheless, the first general meeting with the governors was thwarted through December of 2010.33 (In late December of 2010, when the COP of CLP was on leave, a meeting was held which the Mission and CLP had hoped would begin the approval process with MoPIC, 1990s, but it was part of the USAID White Paper on Participation in the 1980s. Merschrod published his first paper on it in 1980 based on development work in the 1970s. 27 Community Livelihoods Project Quarterly Report July - September 2010 p.8 28 Community Livelihoods Project Quarterly Report October – December 2010 p.13 29 USAID RFA 279-10-006, Community Livelihood Project. P.6. 30 Community Livelihoods Project Quarterly Report January – March 2010 p.4 31 This is, in part, the reason that mobilizers were not hired during the first six months. 32 USAID RFA 279-10-006 Community Livelihood Project, Pp. 7-8 33 By thwarted, it is meant that the late December meeting that was planned with the governors in Sana’a was cancelled at the last minute. CLP and USAID had worked hard to have that meeting, but it was cancelled for unknown reasons by the ROYG. (Source: Correspondence with previous and current USAID Yemen and CLP staff.) YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 6 but while preliminary ideas were exchanged at that meeting, a formal agreement, such as a memorandum of understanding, was not achieved.) Now anxious to implement after six months of delays, the Mission decided that CLP would have to proceed, cautiously and with a low profile, toward working in the Governorates,34 the idea being that if questioned, the meeting at the end of December would provide sufficient evidence of general agreement for the CLP to proceed. Because the previous legacy projects had been approved by prior governments, the Mission believed that it would be easier for CLP to sign agreements with those ministries, and indeed it was. The agreements with the Ministry of Agriculture and Irrigation (MAI) were signed in September 2010 and with the Ministry of Education (MOE) in January 2011. Meanwhile, the COP of CLP had been in communication with the Social Fund for Development (SFD) with a proposed strategy to fund some of the small quick-start projects through the SFD.35 The SFD, however, was well funded by the World Bank, the European Union and others and did not need the approximately $2 million offered by CLP in late 2010. Because the relationship between CLP and the SFD was cordial, and the SFD constantly trained community mobilizers, those experienced and newly trained mobilizers became a source for personnel needed by CLP to implement the community-based strategy that was to have impacted the drivers of instability in the governorates. In fact, SFD provided the COP of CLP with a list of trained development consultants in all sectors to help CLP staffing. Indeed, virtually all of the mobilizers came from this source The ROYG at all levels wanted36 the CLP to work through the ministries and with their governorate level offices. This may have been an impediment to the intended direct-to-the-grass-roots strategy, but it was not contrary to the goal of building trust between the community and the government as found in the USAID PMP that was finalized in February of 2011.37 Importantly, this approach or model of outreach through the ROYG fit the instruction in the CoAg to support ROYG strategies: In order to make the environment increasingly hostile to the spread of violent extremism, we must help facilitate and improve relationships between Yemeni citizens and their government. 38 In the RFA it was also clear that both models39 or strategies were anticipated. However, the phasing of these two strategies called for a first burst of efforts with many small projects, and that dictated the emphasis and timing of hiring and managerial effort. The small grants focus was the reason for the decision to use the Office of Transition Initiatives (OTI) grant tracking software. The longer-term development projects with implementing partners and the ROYG were to start later, as would staffing up for them. As described above, this first phase of the CLP began with the implementation of USAID activities in health, education, and agriculture in order to continue or expand upon the so-called “legacy projects”40 34 Sources: Both prior USAID and CLP executives provided corroborating narratives. 35 As will be described in the section on mobilizers, SFD-prepared projects were shared with CLP at the governorate level when the mobilizers were in place after January 2011. 36 The hypothesis expressed by CLP and ex-USAID personnel is that the protocol delay was an attempt to force CLP to fund ROYG and not have a program working outside of MoLA. The Evaluators did not have a means to test this through ROYG key informants. 37 Yemen Performance Management Plan (PMP) FY 2011 – FY 2013, Pp. 10-11. 38 Cooperative Agreement 279-A-00-10-00032-00 Pp. 15 & 16 39 The models are: 1) grass-roots, community-based mobilization and 2) Outreach and provision of services by the ministries. 40 USAID has had a Mission in Yemen since 1959 and, as in most countries; new projects or programs find an established base instead of a tabla raza. The advantage is in-country personnel and knowledge plus, in this case, a quick start by incorporating legacy projects - Basic Health Services Project (BHS), Yemen Partners for Health Reform (YPHR), Basic Education Support and Training (BEST), and Ministry of Agriculture and Irrigation Support. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 7 instead of the proposed community-based mobilization for rural impact and then longer-term development projects. These USAID “legacy” projects were well established within the ministries and governorates due to implementation of these projects for the 4-5 years prior to the CLP award, but they were coming to the end of their agreements and USAID/Yemen expected41 CLP to assume these partners via awards as implementing partners. Managing and supervising these types of programs required immediate staffing, managerial capacity, and a different class of grants management system than the community-based model that was being established during this first phase. Although the CoAg and PMP had envisioned the implementation of these longer-term awards, CLP was not staffed, nor did it have the systems to manage them or to capitalize on them during this first phase of the program history. Had it not been for the protocol issue, then this lack of capacity could have been the bottleneck to implementation during the first six months. Indeed, of the first 37 grants42 proposed and approved from July through December of 2010, 26 grants were to Ministries – Health (MoPHP) and Agriculture (MAI) – to strengthen service delivery in established outreach programs. Of the $1.2 million in grants approved in the first semester (July￾September 2010) of CLP, 76% were to ROYG Ministries (16 grants valued at $348,517 were provided in support of the MAI and 10 grants valued at $562,894 were provided in support of the MoPHP). However, as is described in this report under Systems and Sectors, CLP did not have the M&E system and personnel in place to monitor these types of grants; as they had just set up to manage small, short￾term grants using the OTI system.43 The other highlight of this period is the geographic focus. The portfolio of grants was focused on five governorates – Amran, Jawf, Marib, Sa’ada, and Shabwa – because that was where the legacy programs and the Ministries were working. The question of identifying work areas based on stability assessments44 became moot for two reasons: 1) the baseline stability assessments were not carried out by CLP until April-September 2011, and 2) the geographic coverage of the legacy projects determined the targeting. Hence the task of conducting stabilization assessments to guide the geographic approach was not needed nor were they done by the CLP45 as a precondition for project planning. During the first six months and continuing into the third quarter of the project the PMP and work planning process was continually evolving. Creative Associates brought in STTA and the Mission brought in a specialist from the Regional, Cairo office because the CLP staff and the Mission staff could not come to terms and finalize the plan.46 The complicating issue was that the context was changing and the MoLA was not engaging. Nevertheless, with the start of the legacy projects, the Mission began insisting that other sector activities be into the same districts where the Health and Agricultural sectors were working. This geographic concentration and the integration of the sectors theme was started by the first Mission Director and the Agreement Officer’s Technical Representative (AOTR). The second Mission Director and AOTR picked up and tried to reinforce that focus. However, as mentioned above, the CLP was not staffed with that vision; it was still working on the community-based theme and 41 Reference: correspondence with both USAID and CLP officers of that period 42 Data on grants is from the CLP All Activities report through December of 2011. 43 The Office of Transition Initiatives has a established, standard software for tracking and reporting on the small grants that are the hallmark of their strategy. It is used World-wide. A comparison of this with an M&E system appropriate for long term projects with multiple indicators is presented in the chapter on the M&E and Grants Management systems. 44 The strategy of implementation of the paradigm included doing baseline surveys on the drivers of instability in the districts and communities in order to target small grants. The plan was for a sub-grant to Partners for Democratic Change (PDC) to carry out that survey, but the grant was delayed and then instability began in 2011’s Arab Spring. 45 Stabilization assessments for Abyan, Aden, Al-Dhale, Al-Jawf, Amran, Lahj, Marib and Shabwa were not completed until September of 2011 46 Source: Prior Mission AOTR (April 2012) YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 8 indicators of instability that had yet to be measured. Indeed, it was not until the fifth quarter (July￾September 2011) that the model of Thula in Amran governorate was presented. One important philosophical aspect to keep in mind is that although the original approach and conceptualization of the small project, rapid impact program was stabilization theory, i.e., activities would be prioritized based on the drivers of instability and where the drivers are most potent, the approach of the legacy projects and the ROYG Ministries was aimed at overall coverage based on population needs and priorities. Hence, the importance of USAID’s and CLP’s strategic priority of stabilizing key districts based on drivers of instability was not the main rationale for CLP project selection. Because of these considerations, CLP did not evolve from a participatory, community-based model, guided geographically by community or district-level instability studies into a more traditional capacity building, extension oriented, longer term, development model as planned. It was forced into the latter model by the protocol dimension that was beyond its control during the first six months of the project from July to December 2010.47 This transition took place even before the original CLP PMP, which was designed to implement the community-based stabilization model, was approved.48 It also took place before CLP had the personnel and systems to manage the second phase. Furthermore, with the exception of one community mobilizer, the mobilizers for implementing the community-based CLP strategy were not hired until January and February 2011 after CLP and the Mission decided to cautiously go ahead with implementation in the governorates without formal approval of MoPIC. The Mission gave the COP of CLP assurances that they would support him if the ROYG tried to declare him a persona nongrata because of contacts with the governorates. The Mission also facilitated meetings with the governors.49 The main point is that both events on the ground and changing approaches to implementing development activities outpaced the original theory of community-based implementation and the staffing of core personnel. Work plans and staffing, such as the Community Mobilizer system, started six months late and were based on the original community-based strategy that was by passed by the legacy projects. Importantly for the funding flow, the staffing for small projects mobilization (the mobilizers) was delayed by six months and, as will be described, the staffing required for the longer-term and larger development projects was not in place. 2.2 Phase Two: New Management (February – August 2011) The second phase is characterized as a period of changing management within the USAID/Yemen Mission and CLP, precisely as the country began a period of political instability and protests that occurred during what is generally referred to as the “Arab Spring.” It roughly covers the period February 2011 through August 2011. Figure 1 shows these multiple changes in CLP staff with changing colors and also points out the gaps in staffing with blank cells. 47 c.f., Community Livelihoods Project Quarterly Report January – March 2010 p.4 48 The first CLP PMP was presented during the second quarter (Oct-Dec 2010) and finally, formally approved by USAID in April 2011. 49 Triangulated in correspondence with prior USAID and CLP personnel – April 2012. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 9 Figure 1 – Changes in CLP Expatriate Staff What is notable is that there were significant periods when key Team Leader positions were either vacant or only filled on an intermittent, short-term basis. The impact of this is reflected in the analyses of the systems and sectors covered by this evaluation. Yemen’s version50 of the “Arab Spring” began in January with a protest against the government in Sana’a and again in February. In February protests took place in Aden that expressed the former southern Yemeni complaints that existed since the 1990’s civil war. In addition, the Houthi (north-west Yemen) complaints since the 1960s surfaced again with armed rebellion that brought Houthi militia into Sana’a where they occupied sections of the city and attacked opposing tribal neighborhoods. The conflict between protesters (pro-government and anti-government) expanded to conflicts between factions of Yemen’s armed forces, tribal groupings, and also to groups related to the Southern Hirak and Northern Houthi separatist groups. The US Embassy issued initial evacuation orders in March 2011 and another evacuation of the majority of Embassy personnel in May 2011. In spite of the evacuations, both the COP of CLP and the new Mission Director remained in country. Nevertheless, with conflict in the streets it was very difficult for CLP to recruit and place the needed key expatriate personnel. USAID also suffered a similar plight and the USAID/Yemen team remained heavily under-staffed since May 2011. 50 The easiest, most concise reference on the chronology of the Arab Spring in Yemen is found in Wikipedia – key search words are: “Arab Spring Yemen;” “Houthi Rebellion,” and so forth. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 10 On the ROYG side, because of the street violence, sectorization of urban control by factions of the armed forces and the Houthis 51 travel between neighbourhoods in Sana’a was risky and government offices were closed for weeks on end, making communication with ministry officials impossible. In March 2011, the Mission Director in place from 2009 departed and was replaced. When the AOTR for CLP was evacuated in March 2011, it was not until May 2011 that the new USAID Technical Director and AOTR for CLP arrived in Yemen. This left a two month gap in USAID/Yemen CLP oversight and in country AOTR personnel. The first CLP COP was replaced in August, 2011. In brief, the impact of the Arab Spring on USAID, CLP and ROYG was, in a form, to continue the delay that started with the protocol issues and then continued with the changes in strategy regarding the rural urban shift and the reduced rural community-based focus. With these changes the team that had designed the paradigm in the RFA, and had begun to supervise the implementation of the CLP CoAg was gone, and the new team brought in more emphasis for the changes that had already begun under the original Mission team. The previous Mission staff had been moving toward the present model and away from the drivers of instability, community-based model52 when it found that it was not possible to implement the participatory community development approach expressed in the development hypothesis because they did not gain access to the governorates as planned; because the mobilizers had not been hired and deployed; the implementation of survey to identify the communities on a scale of drivers of instability had not come to pass in order to guide the community-based strategy of identifying the community priorities; and, furthermore, the vision for each sector had yet to be developed by CLP at the time of transfer of the Mission from one Director to another. While the vision statements had been requested by the first Mission Director and his team; it became the priority of the new Mission team to put renewed emphasis on the integration theme and strategies to cope with the unrest as manifest in the 2011 Arab Spring. Because of the Arab Spring, the project was no longer focused primarily on a pre-emptive effort to respond to the conditions of instability at the community level; it became a project that was also trying to respond to and defuse the drivers of unrest in the more urbanized areas. The following summary of this period by the second Mission Director indicates both the urban or peri￾urban shift and the change from the proactive identification of the drivers of instability orientation by survey toward identification of the target areas of urban unrest as experienced in the Arab Spring: USAID surmised that relatively little development and “stabilization” would be achieved unless more engagement with local authorities, and more integrated activities were implemented in communities. Since that was not proving possible in the originally targeted communities, emphasis needed to be placed on working in more “permissive” communities, where the grantee could engage local government, communities, and beneficiaries in the identification and resolution of priority needs, and find capable sub￾contractors and sub-grantees to provide technical assistance, training and other inputs in meeting those needs. The implication was that the project needed to shift towards communities in or near the population centers, where engagement was possible and service providers were available. Political instability and significant economic downturn was evident throughout the country. Working in densely populated areas presumably would be easier, while also visible, meaningful, and impactful.53 To avoid any misunderstanding when using the term “community-based” model, the evaluators took the “community-based” description and strategy for mobilizing communities from the RFA and CoAg development hypothesis. Based on that model the community mobilizers would enter target communities 51 The situation was close to civil war with factions of the armed forces, as well as tribal militias, fighting around bases and urban neighborhoods. The “Arab Spring” in Yemen was, and continues to be, much more than youth uprisings. 52 Correspondence with prior USAID Yemen AOTR and Technical Director – April 2012 53 Commentary from Robert J. Wilson, USAID Yemen Mission Director, in response to the draft report of 18 April. Correspondence 22 April 2012. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 11 and engage them in a participatory planning process to identify the types of small projects, and their priority, that could be implemented quickly. CLP was to build upon those projects as a means to integrating as many sectors of CLP and ROYG programs as necessary. This dynamic was different than using the idea of doing projects or providing services aimed at communities without going through the community mobilization model. Although CLP and the Mission continued with the intent to provide services to communities, the programs are not community-based, they are based on strengthening ministries and NGOs to serve communities. Had the program been community-based the evaluators would have found file after file of community assessments and plans generated by the Mobilizers to support a series of projects. What evolved was a more pragmatic vision focused on building a more classical development program based on the integrated development54 of sectors and the engagement of ROYG Ministries. CLP grants were frozen from March until May 2011 until vision statements were written for the five sectors to reflect the new approach to implementation through longer-term awards and integration instead of the community-based development strategy. The Mission also called for the vision papers because the small project proposals that came in did not seem fit a model of development and were scattered. The Mission’s new view, i.e. a more classical development rather than stability approach, did not see the small, rapid impact proposals as adding up to having an impact on specific populations. To be fair to the CLP it has to be noted that the community mobilizers had only started in January and February 2011. Nevertheless, this rough and late start of the mobilizers discredited the small projects’ approach and, as will be described, the mobilizers were marginalized within the CLP administrative structure as well as in the eyes of the Mission, which had moved on toward working with programs and not small community￾based projects. This late start, due to the protocol dimension, is the main reason that 1) the community mobilization model was not implemented, and 2) the rural strategy for attenuating the drivers of instability was not started as planned. As described above, with the 2011 Arab Spring in Yemen came a new perspective on the drivers of instability. That is, the urban youth and urban protests in Sana’a, Taiz, Ibb, Aden, and so forth focused more donor attention toward urban areas. The original focus of CLP was predominantly rural districts and also included the capital cities of the target governorates. The original Mission Director and AOTR urged CLP to reorient activities into the urban and peri-urban areas of Yemen’s major cities. This shift to the urban areas of unrest was continued by the new Mission Director. This took the form of labor￾intensive cash-for-work projects to provide short-term work for unskilled and unemployed youth, while the agriculture sector took on urban gardens and the health sector tackled health specific responsiveness for Internally Displaced Persons (IDPs) in Aden and Lahj. This rural to urban shift was another course change for the CLP team based on the type of activities and also the geographic focus. With the new Mission Director in place, sector vision statements55 and activities that would implement them were required for each sector of CLP. The CLP sector vision statements also included reference to integration, but the strategy for integrating the sectors was neither evident nor implemented until later when the focus on Thula as a model was begun during the fifth quarter of the project.56 The desire of the Mission was that integration would be expressed by having multiple sectors active in specific communities so that CLP activities would have more substantial overall collective impact. This 54 The integration theme was stressed in the RFA and CoAg, but once CLP’s grants portfolio based on the legacy projects was established, the integration at the community level was out of the control of CLP. Furthermore, the targeting by the legacy programs in the ministries was based on ministry needs and not drivers of instability. The vision statements that came later did not stress the integration or a model for integration. 55 The previous Mission Director and AOTR had requested these vision statements, but the period of political instability and the difficulty of fielding CLP staff delayed their preparation until April of 2011. 56 Community Livelihoods Project Quarterly Report July – September 2011, p.21; Community Livelihoods Project Quarterly Report October- December, 2011, p.22.CLP had been urged by both the original and subsequent Mission teams to pick one district for urban and peri-urban focus. Thula became that district. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 12 integration in specific communities was another change in the way that CLP needed to implement. The original community-based, small projects approach was not designed for multiple actions in specific communities, but, instead, for scattered projects. The operationalization of this approach was not immediately accomplished until a model community, Thula, was selected and the sectors coordinated for implementation. It is important to point out that there was not a formal decision or a directive57 to CLP to change from the original model using community mobilizers and measures of instability to the more pragmatic model that would also serve specific urban and peri-urban areas and, in addition, community capacity development would follow rather than precede the short term community work. As the present Mission Director described it: USAID surmised it would be easier and a more effective use of tax-payer assistance dollars to do short term health, education, agriculture, economic growth, and community governance work, to be followed with longer term community capacity development in these sectors, near the populated centers of Aden, Taiz, and Sana’a, for example, than in the farthest, most insecure, and sparsely populated areas of al-Jawf, Marib, and Shabwah.58 Hence, this phase of the CLP ended with new management and new approaches to the development context in Yemen. This renewal became the basis for how the project would be restructured and staffed during the third phase that we have titled “Building up.” 2.3 Phase Three: Building Up (September – December 2011) During the 5th quarter (August 2011) of the project, the first CLP COP departed and a new COP began to invest heavily in team building and increasing both expatriate and local staff to manage the five sectors, the increasing grant load, and to build the M&E system. Between July and December 2011, CLP was changed considerably and Creative demonstrated renewed commitment to its activities. New staff were identified and selected. Program focus and organization took on a far more formalized process with weekly meetings and meetings of sub-groups such as the grants management team and the M&E team. In addition, as evidence of renewed commitment, a new Grants Management System (GMS) was introduced to the staff and USAID/Yemen59 to replace the OTI small grants system.60 A new initiative in effective Monitoring and Evaluation (M&E) was undertaken with linkages to YMEP and correct performance reporting. Figure 1 above shows the turnover of key expatriates and also the build-up during the latter part of the time period. The following Figure 2a shows the local staffing levels during the first 18 month of the CLP. The build-up that took place during the third phase of the narrative is notable. Figure 2b shows the erratic levels of expatriate staffing reinforcing the main point of expatriate turnover as shown in Figure 1. 57 One of the communications issues that was identified was that the Mission and CLP were not formally exchanging correspondence on these changes in emphasis and formally agreeing to plans of operation. 58 Commentary by R. J. Wilson to the 18 April draft CLP report. 22 April 2012. 59 While introduced, the new grants system had not yet been installed and deployed for use within CLP during the evaluation period through December 2011. 60 In the section of this report on the Grants Management System the importance of these changes are discussed. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 13 Figure 2a – CLP National Staffing Local Staff 0 20 40 60 80 100 120 140 JUL 2010 AUG 2010 SEP 2010 OCT 2010 NOV 2010 DEC 2010 JAN 2011 FEB 2011 MAR 2011 APR 2011 MAY 2011 JUN 2011 JUL 2011 AUG 2011 SEP 2011 OCT 2011 NOV 2011 DEC 2011 JAN 2012 Figure 2b – CLP Expatriate Staffing Expatriate Staffing 0 2 4 6 8 10 12 14 AUG 2010 SEP 2010 OCT 2010 NOV 2010 DEC 2010 JAN 2011 FEB 2011 MAR 2011 APR 2011 MAY 2011 JUN 2011 JUL 2011 AUG 2011 SEP 2011 OCT 2011 NOV 2011 DEC 2011 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 14 A total of 130 Yemen national CLP employees and 12 expatriate CLP employees were on staff by January of 2012. The one off-site office, in Aden, is being reviewed for further growth possibilities. In November 2011 the COP called the first whole team retreat for CLP and instituted weekly program meetings, plus weekly grants meetings to review progress and options for new grants. This was a major change in the style of management and is crucial to the success of CLP. At the time of this evaluation, once again, the work plan is being revisited to make it more appropriate to: 1) the integrated and expanded sectors, and 2) reflect the needs of the transition period. A modification (4th) to the Cooperative Agreement is being planned by CLP and USAID that will probably include funding and also programmatic changes.61 The reason that these changes are mentioned, despite that they fall outside of the evaluation period, is that they are yet another indication of the dynamic context in which CLP and the Mission must operate. During the 7th quarter (Oct-Dec 2011) CLP proposed and obtained approval to release an Annual Program Statement (APS) to attract larger national and international organizations to carry out activities more rapidly and with a larger scope and budget. This measure should enable CLP to increase the rate of grant proposals and implementation disbursements.62 Their calculus was that the time and effort to generate new, local grants, and the required effort to manage them and do procurement63 was too great. The APS approach should also increase the geographic coverage and, at the same time, should be a mechanism that CLP can use to respond to the Mission’s concern that the grants budget line be used. As will be shown in a model later in the report, this mechanism can bring the project into line and close to the target of $50 million grants awarded by the end of the first three years. The end of the third phase (December 2011), half way through the 36-month project, finds CLP more prepared and making substantial progress towards the way forward. The errors of the past are recognized, the needs of the future are known, and the people and resources to meet the challenges of implementing CLP are on now board. 61 The changes in the anticipated USAID strategic plan were not sought by the evaluators as it is beyond the evaluation period. 62 In this report we have modeled the use of APS levels of grants to see if it is probable that this mechanism will help CLP achieve its granting goal. 63 Procurement procedures were not found to be a bottleneck in the disbursement process. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 15 3. Scope of Work and Methodology As part of its third party evaluation role for the Mission, YMEP was requested to conduct this mid-term evaluation. A two person team was mobilized to conduct this evaluation: a team leader with evaluation methodology expertise and a stabilization expert. A scope of work (SOW) was drafted prior to departure for Yemen and finalized after discussions between the team and USAID/Yemen staff. Annex I explains why and how these changes were made to the SOW. 3.1 Purpose and Key Questions Purpose The purpose of this mid-term performance evaluation is to assess the implementation, effectiveness, and progress of the Community Livelihoods Project (CLP) in Yemen. The findings of this evaluation will be used to inform USAID/Yemen whether the program is on track to achieving programmatic and fiscal goals and targets and whether the initial design of the project addresses the overarching USAID strategy in Yemen of positively impacting stabilization in key areas.64 Intended users of the evaluation will be the USAID/Yemen Mission, the implementing partner, and YMEP.65 Key Questions This mid-term performance evaluation will measure and analyze the accomplishments and/or progress toward achieving the CLP Integrated Results (IRs), CLP results, and ultimately the program goal. Given the Mission’s increasing concern over the past six months with the low CLP grant disbursement rate, the evaluation will also assess and analyze the reasons (both internal and external) for low disbursements, highlight any aspects of the program that have proved successful, and document why other aspects have not been successful. The primary evaluation questions are as follows:  Has the CLP implementation mechanism of issuing grants to local awardees been an effective strategy for achieving program results?  Have the efforts to include women and youth in the program achieved expected results?  Has the program achieved all of the other expected results at midterm?  Has the program been cost effective?66 Recommendations Sought The report must include recommendations by the evaluators that capture the best practices of the project thus far and also identify any shortcomings so that they can be addressed and a mid-course correction can be implemented as necessary. Specifically, USAID/Yemen is interested in:67  Recommendations for moving forward with regard to partnership with international and local sub-grantees;  Recommendations for field teams to enhance/ensure community engagement;  Recommendations regarding enhanced support for field operations in CLP governorates moving forward since CLP expatriate staff are currently located at the main CLP office in Sana’a; 64 See 2010-2012 USAID Yemen Country Strategy. 65 USAID Yemen SOW, 24 February 2012, p. 1 66 The definition of “cost effective” was discussed. The definition was left to the evaluators with the caveat that expectations of the detail were conditioned by the level of detail. 67 SOW 24 January, 2012, p.9 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 16  Recommendations as to whether CLP expatriate staff should be located in field offices where security is permissive to enhance grant implementation performance, oversight and effectiveness;  Recommendations to improve CLP grant disbursement rates moving forward;  Recommendations to improve the ratio of grants/project disbursement to CLP overhead and support costs; and,  Recommendations to improve CLP’s financial processes and procedures with regard to grant disbursements. Methodology The methodological approach is described in Annex II. In summary:  Document review of over 200 items (Annex IV) – this began prior to departure for Yemen and was expanded as local documents and reports became available in Yemen  Compared the model for CLP implementation expressed in the Cooperative Agreement and subsequent modifications with the actual implementation using additional documentation and extensive interviews  Over 100 different people were interviewed (Annex III) – many multiple times to triangulate data: USAID/Yemen Mission executives, staff and previous personnel; CLP Executives, staff and previous personnel; Implementing partners; YMEP personnel; ROYG executives, personnel, and local officials  Financial and grant data obtained from YMEP, CLP, and Creative Associates were analyzed  Human resource records were used to assess the timing and scale of personnel  Survey data from the Third Party Monitoring of the CLP grants portfolio was used to assess participant and ROYG opinions on the awards  Based on the interviews, project documents, and public domain documents, the historical narrative was prepared.  A mid-term briefing on preliminary findings was discussed with the Mission  A final out-briefing of recommendations was discussed with the Mission  An informal final discussion with CLP management was also provided at the request of CLP and the Mission Mission and CLP feedback in response to the draft of this report was used to correct errors, to clarify historical points as well as weigh the importance of findings. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 17 4. The Development Hypothesis and Geographical Coverage; Findings and Conclusions 4.1 The Development Hypothesis The change in emphasis from small projects and drivers of instability had direct implications for what was to have been a major area of inquiry of this evaluation: Development Hypothesis/Stabilization Questions 1. Does the program design, i.e. results framework and implementation thereof, adequately test the project/development hypothesis? What are the results so far? Do development interventions at the sub-national level positively affect stability in Yemen? 2. Based on CLP’s experience with implementation thus far, does the Yemen Mission’s PMP definition of stabilization hold true? “Stability is the manifestation of belief in and support for decisions and actions that are having an impact on the person’s life and the welfare of his family. This is implied by people’s perception that they enjoy an acceptable quality of life, fair treatment, equality, economic opportunity, societal justice, security, rule of law and the participation of marginalized groups, women and youth in society and that these things are predictable in the daily course of life. Stability is manifest in the trust that citizens show towards their local government.” 68 3. Does the PMP allow stakeholders to test the central hypothesis that improved livelihoods contribute to an increase in stability? Findings The answers to these questions can be summarized by the fact that the community-based program using drivers of instability for targeting was not implemented.69 The stabilization surveys that would have been used for targeting activity interventions were not carried out until the start of the second year of the CoAg; and baseline data for the dependent, independent, and intervening variables needed to test the theory were not gathered. Furthermore, the original Mission PMP, although replete with indicators, did not include a logic model, path diagram or logical framework to diagram the causal links. When the CLP stability surveys were eventually carried out in mid-2011, the unit of analysis was not at the community level but, instead, at the governorate level with sample districts. Although conceived as an experiment, the design was not detailed enough so that indicators and units of analysis could be used for targeting with follow-up to measure impact while having a control or comparison group of similar units of analysis. The question about the definition of stabilization is interesting from a sociological perspective because it focuses on the social psychological aspects of individuals as stated above: “Stability is the manifestation of belief in and support for decisions and actions that are having an impact on the person’s life and the welfare of his family. This is implied by people’s perception that they enjoy an acceptable quality of life, fair treatment, equality, economic opportunity, societal justice, security, rule of law and the participation of marginalized groups, 68 This definition of stabilization was one of the key outputs of the Mission PMP workshop held with key stakeholders, August 25, 2010. 69 The implementation model that was envisioned in the CLP CoAg was stability assessment-based geographic targeting of community-based participatory development. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 18 women and youth in society and that these things are predictable in the daily course of life. Stability is manifest in the trust that citizens show towards their local government.” 70 The social psychological perspective as defined in the Mission PMP, however, ignores the collective unit of analysis and the dynamic of those groups as they compete for power and resources. This collective or sociological perspective was explained above in the section on the Arab Spring as manifest in Yemen. Based on the definition posited, the theory of confronting the drivers of instability goes on to propose that by meeting individual needs through the improvement and provision of government health, education and agricultural services, trust in the government will increase and the populous will be less likely to rise up or join insurgent groups. This could be a worthy theory to test if there were not intervening dynamics of instability such as tribal competition, factions in the armed forces, and regional tensions based on a recent, unresolved civil war as well as historical regional autonomy movements. In brief, the answer to, “Do development interventions at the sub-national level positively affect stability in Yemen?” is that the necessary measurement to test the hypothesis hasn’t occurred, and the model to do so, the community-based methodology, was not implemented. Moreover, the units of analysis do not reference the main destabilizing elements in Yemen – tribal conflict, regional tensions, and factions reflected in the armed forces. Conclusion The original model or theory of development (targeting drivers of instability through community-based projects) proposed for the project was not appropriate to the context of Yemen nor the established base of USAID programming in Yemen, i.e., the legacy projects. The designers assumed that the main cause of instability was at the individual, complaint level about wellbeing, and that those complaints could be used by insurgent forces to mobilize against the government, but, as is described in the narrative above of the Arab Spring in Yemen, the main drivers of instability are in the leadership of polities (tribes, factions of the military, and regional separatist movements) struggling for power, and not the state of individual wellbeing waiting to be mobilized by outsiders. These types of tensions at a higher-than-individual level of societal structure require mediation and are a different type of project. Recommendation Although the poverty suffered by the population (the first dimension indicated in the context of Yemen) is a genuine USAID concern, that concern cannot be met trying to do development unless it accompanies and is coordinated with a strategy that will address the overarching drivers of instability. A different approach is needed that works with the contentious groups. Mediation by a trusted body or individuals based on cultural attributes would be needed to facilitate arbitration between tribes. At the same time as these negotiations, in respect for the regional vs. center tensions (also related to the interests of tribal leaders) a decentralization strategy is needed that delegates authority and responsibility to local leadership so that the concentration of power and resources is not in the capital city71 nor can be monopolized by one group at the expense of others. USAID’s resources through CLP could be used to support decentralization and as part of the incentive for mediation. 70 This definition of stabilization was one of the key outputs of the Mission PMP workshop held with key stakeholders, August 25, 2010. 71 The initial protocol dimension that delayed the implementation of CLP is an example of the problem of concentrated authority. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 19 4.2 Geographical Coverage This section has three sub-sections to answer the specific questions in the SOW with regard to the geography of CLP activities and offices. 4.2.1 Questions about the Expansion of CLP A second set of questions in the SOW notes: Recent political instability and operational security challenges have resulted in an expansion of the areas of CLP’s activity interventions beyond the initial 8 governorates to encompass a more urban and peri-urban focus and a larger geographical reach than initially anticipated.(p.1) And Further, they (Evaluators) will review and analyze the CLP’s geographic strategy for effectiveness including a thorough examination of the CLP geographic areas of implementation. In connection with this analysis, they will determine whether the CLP’s geographic shift from the eight initial target governorates (Shabwa, Marib, al-Jawf, Amran, Aden, Abyan, al-Dhale, and Lahj) to a more urban and peri-urban focus in an expanded number of new and continuing governorates has been an effective strategy looking at the bases for the decision to work in these different areas. Findings From the historical narrative, a synopsis related to this question is appropriate. We found that CLP’s mandate under the original model focused on drivers of instability was to target the most instable communities and districts, particularly in eight governorates (listed above) specified by USAID/Yemen in order to increase stability. Per the CoAg, CLP was to choose two districts in each of those governorates to start the implementation of activities. The renewal of the legacy projects brought with them established district coverage beyond, and not necessarily in, the target districts. Furthermore, the stability studies of the eight governorates were completed after CLP had deployed the Community Mobilizers in their quest for the rapid impact projects in the target governorates. With the upsurge in urban violence in early 2011, geographical emphasis on urban and peri-urban areas was added, but this was a neighborhood level unit of analysis as opposed to a community or village focus, and that would have changed the terms of reference for stability assessments. The neighborhood focus was not mentioned in the CoAg, subsequent strategies, or PMPs. Furthermore, it should be pointed out that the focus on urban areas was a reactive response rather than the original preemptive strategy for identifying drivers of instability. The most graphic, yet aggregated way to answer the question is the following map generated by YMEP. The GIS capability of YMEP is outstanding and will only improve as CLP and its implementing partners improve their M&E systems under the guidance of YMEP. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 20 Conclusions In the aggregate governorate level, CLP has activities in the target governorates and, as explained in the sections on Agriculture and Governance and Civil society, CLP did implement activities in the six urban areas of unrest. 4.2.2 Effectiveness in Regions and Peri-Urban Areas The SOW asked for the following analysis: 1) Analyze the CLP’s geographic strategy for effectiveness including a thorough examination of the CLP geographic areas of implementation. The first observation is that original geographic targeting strategy was not implemented, and aside from the recommendation that will be made in this section about the Aden Regional Office, the evaluators could not go to the governorates outside the national capital region to assess their needs due to security concerns. In addition, considering that 1) CLP focused primarily initially on the locations of the previous USAID legacy projects and then 2) went to the urban areas pointed out by the Mission in late 2011, the original geographical strategy was not implemented beyond the belated start of hiring the mobilizers for each of the eight original governorates. 2) determine whether the CLP’s geographic shift from the eight initial target governorates to a more urban and peri-urban focus in an expanded number of new and continuing governorates YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 21 has been an effective strategy looking at the bases for the decision to work in these different areas. We must start with the “bases for the decision to work in these areas.” As was pointed out in interviews with Mission executives, confirmed by CLP staff and in correspondence with previous Mission and CLP staff, the decision to go to the urban areas was made in response to urban unrest that characterized part of “Arab Spring” in Yemen. This geographical change was made after the fact – it was not a preventative strategy as was the original model. If we assume that the objective for this focus of resources is the same as the original, i.e. increased trust in the government, then the conclusion is that the types of activities are not appropriate for basically the same reasons that the authors have critiqued the original theoretical model, i.e. the drivers of those urban uprisings reflected both regional complaints against the central government – e.g. Aden, Ibb, Hudayda, Dhamar, Taiz – and feuds between pro-government and anti￾government72 polities. As pointed out in the section on the development hypothesis, and will be elaborated on in the overall recommendations, a CLP-type of project might provide incentives to support a program that deals with the main, collective or structural drivers of instability in Yemen, but alone, targeting the individual or community level, CLP-types of activities, in the context of Yemen, will not produce the desired stability. Then, turning to the activities or grants awarded to intervene in the urban areas, we offer a rhetorical question. ““Is there a logical link between the activities and the complaints that may be drivers of the urban unrest in those urban areas?” Have those cities revolted because of unemployment or is it because of the tribal leaders revolting against central authority because the leaders want power?” From this collective perspective we need to assess the suitability of the urban projects funded in response. The CLP Agriculture sector responded to the urban call with six grants to Althuraya and the Yemeni Association for Sustainable Agriculture Development (YASAD) to start urban garden programs in those targeted cities. They are recent grants (September 2011) and have not gone through a complete agricultural cycle so it is not possible yet to measure the impact on the food and incomes of the 1,500 families involved. In the case of the school bags given out to over 100,000 school children (urban and rural), it is hard to visualize a causal link between them and the “youth” in the protests. That Youth category, as used in the documentation both in the CLP and other livelihoods projects,73 included young adults to 24 years of age. The other major urban activities that were implemented included cash-for-work to clean up and beautify urban areas. This is a short term effort begun in December 2012 in Sana’a with 13 grants to SFD totaling approximately $1.3 million which did provide daily labor income for under- and un-employed people who were in the youth category, but as soon as those programs ended, the problem still existed. As a matter of fact, as the funds were ending, the garbage workers went on strike because their rate of pay was not keeping pace with the cost of living. There isn’t evidence to indicate that urban unrest subsided because of these cash-for-work periods. In summary, if we assume that the basis for working in those urban areas was the unrest, then we cannot reach a conclusion yet because firstly, the baseline to measure urban instability is not established, but mostly because the interventions have not been active long enough nor are they large enough to have a comprehensive impact in the city. From a logical perspective these interventions to reduce urban unrest in the cities and towns that have up risen may not be effective for two reasons: 1) the amount of effort was small and 2) although hunger and unemployment problems are extensive, and may have provided able bodies and angry minds, the street sweeping and vegetable gardens do not resolve the complaints of tribal leadership and regional struggles for power or autonomy. 72 In Sana’a, had there only been the Change Square movement galvanizing anti-government sentiments it would have been a simple analysis, However, with the establishment of a pro-government youth movement and related military and party factions also focused in Freedom Square, it is clear that the unrest is not simply “youth.” 73 Merschrod’s experience has included the Pakistan and Colombian Livelihood programs. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 22 Recommendations In keeping with the recommendation of decentralization and incentives made above, the Aden Regional Office is an example that could be replicated, albeit on a smaller scale in other non-regional centers to expand the regional coverage and the intensity of CLP coverage. 4.2.3 A Regional Office for Aden and Recommendations for Other Governorate Offices Initially CLP formed the Aden Regional office with Community Mobilizers, civil engineers, and a general manager. The Governor was very enthusiastic about the idea, but there were tensions between MoPIC and this and other governorates. Due to the Governor’s encouragement and CLP’s strategy, the Aden office was established. However, the office did not reach its potential for coordinating and placing grants and leading the CLP activities in the region. It was overshadowed by the political unrest and the struggle of the CLP’s Sana’a Office to start up the whole process. The interviews with CLP personnel in Aden indicated that they felt that their region was not accorded the level of interest by the Sana’a office that they deserved. The interviews with the ROYG officials in the Aden area also indicated the same belief. Conclusion The southern region has special issues such as the separatist movement and also IDPs which represent a special health and education problem (the IDPs occupied the schools after they fled the attacks on their governorate cities). All of these issues could be attended within the objective of increasing trust in the government. By establishing a Regional CLP Project Office with an expatriate in residence, the planning and coordination of CLP activities in the area would be enhanced and the efficiency of the program could be increased. Recommendation Because of the importance of the governorates around Aden for stability in Yemen and because of the importance of the South in the geo-political history and current tension in Yemen, CLP should elevate the importance of the Aden Office to that of a full project office led by an expatriate staff person with development experience, particularly project managerial experience, and accord this staff person decision making power. An upgraded Aden office also makes logistical sense for transportation and also the liquidation of travel advances for regional personnel. One of the related questions raised by the Mission and CLP was whether it be appropriate and feasible to expand offices in some governorates and have expatriates work from those offices. The security issue is probably the most crucial, because the mobilizers reported that they circulated easily because they were from the area and because they kept their source of employment private. They said that they did not carry identification nor did their materials or projects carry identification in some areas.74 It would not be good for the mobilizers to be seen in the presence of foreigners. It would, however, as will be explained in the section on the Mobilizers, be appropriate to provide better office conditions and service to the mobilizers and to assign coordinating responsibilities to selected mobilizers to further the integration-of-sectors strategies in the regional towns and districts. The mobilizers from those governorates already have established relations with the governors and the SFD program personnel. As conditions permit, these small regional offices could be expanded. 74 When the school bags were produced they had both USAID and ROYG identification. In some areas of the south they did not deliver them because of the identification. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 23 5. Systemic and Sector Assessment: Findings and Recommendations 5.1 Grants and Costs Associated with Grants In addition to the pace and geographic distribution of grant implementation, USAID/Yemen has had a particular concern with respect to the costs of grants when compared with overhead and other costs associated with grant implementation. The CoAg calls for a total of $78.5 million in grant activities over a five-year period.75 Other budgeted items estimated for the same period total $45 million, bringing the total budget to $123.4 million. Not covered in the evaluation review is a final number; $12.4 million in “cost￾share.” In preparing for the analyses, the concern of the evaluators was identifying and obtaining financial data that could be used consistently.76 It was agreed that Creative would prepare monthly financial information for the 18 month evaluation period based on their requests for reimbursement to USAID. This would cover: Overhead (OH), Program Costs (P) and Grants (G). By using these three lines, Creative was able to respond to the needs of the evaluators while not disclosing privileged corporate financial data. This information was prepared by CLP’s Finance Operations Division in full consultation with Creative’s Finance Office in Bethesda, MD. All of the subsequent analyses are based on this information.77 The following analyses begin with the actual figures through December 2011 and then models projections for the remainder of the three-year base-period. The assumptions of the model are explained in the text. If the assumptions are met, the evaluators conclude that CLP could come close to the CoAg grant making targets in terms of dollar amounts. The original small grant targets are no longer applicable because the program transitioned early on to funding programming rather than small grants. Nevertheless, CLP should remain flexible and maintain provisions for making small grants because the dynamic context of Yemen may provide both small grant needs and opportunities. Findings Table 178 Overhead, Program Costs, Grants (18 months) in US$ Category Jun–Sep 2010 Oct–Dec 2010 Jan–Mar 2011 Apr–Jun 2011 Jul–Sep 2011 Oct–Dec 2011 TOTAL Jun 2010- Dec 2011 Overhead 522,000 724,000 1,459,000 2,535,000 2,059,000 4,608,000 11,907,000 Program Costs 133,000 397,000 467,000 685,000 700,000 1,175,000 6,889,000 Grants 0 68,000 667,000 421,000 670,000 1,506,000 3,332,000 75 A three-year base period and two one-year optional extensions. 76 Federal Financial Reports and Creative vouchers to USAID do not present sufficient detail to undertake a comparison analysis. 77 Creative was not asked to detail what elements constituted “Overhead” or “Program Costs”. This would have revealed confidential information. Combined, the two do provide a reasonably accurate comparison of grants to other costs. 78 Numbers are rounded. Actual billings do not reflect when work was carried out, only when it was billed to USAID. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 24 Through the evaluation period,79 the total amount that CLP invoiced was $14.7 million. Disbursements against Grants totalled $3.3 million, Program Costs $3.7 million and Overhead totalled $7.6 million. The following projections are based on the December 2011 invoice80 for all three categories, not allowing for increases in costs or inflation but simply straight lining the data going forward. With no changes from the December 2011 invoice (a generous assumption as more long-term expatriate staff has been added to CLP since that period), the costs of implementing CLP for the twelve months of 201281 will be as follows: Overhead $9.336 million Program Costs $9.228 million Total $18.56 million With projected grants totalling $14.840 million, the overhead and program costs will likely exceed the grants expenditure. It is estimated that CLP will cost $1.25 in program and overhead costs for each $1.00 in grants during the projected 2012 calendar year. If one assumes that Program Costs will be spent on the implementation of the five CLP sectors and not on grants, then the overhead will still be 65% of grants. Visually, the trends in Overhead, Program Costs and Grants for the first eighteen months are presented in Graph 1 below. Graph l Cummulative CLP Expenses $- $1,000,000.00 $2,000,000.00 $3,000,000.00 $4,000,000.00 $5,000,000.00 $6,000,000.00 $7,000,000.00 $8,000,000.00 7/10/10 8/10/10 9/10/10 10/10/10 11/10/10 12/10/10 1/10/11 2/10/11 3/10/11 4/10/11 5/10/11 6/10/11 7/10/11 8/10/11 9/10/11 10/10/11 11/10/11 Grants Programs Operations (Overhead) The figures above show the actual state of grants vs. costs. Note the continuing divergence between non￾grant costs as compared to grant costs. 79 July 2010 through December 2011. 80 Overhead: $778,000; Program $774,000; and Grants $1,237,000. 81 Overhead $778,000 x 12 months; Program $774,000 x 12 months; Grants $1,237,000 x 12 months. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 25 The Scope of Work asked “Has the program been cost effective?” It further asked the evaluators for two recommendations with regard to CLP financial disbursements:  Recommendations to improve CLP grant disbursement rates moving forward; and,  Recommendations to improve the ratio of grants/project disbursement to CLP overhead and support costs. To date, overhead and program costs, when compared to grants, have been very large and not cost effective. This is a significant finding. The obvious recommendation (repeated in the analysis of each sector), is that CLP scale up grant activities in order to achieve the grant making targets. If expenses are held constant and the grants increased, then an acceptable balance may be achieved. This is an unknown area given significant staff increases, but the following section models a possible outcome. In January of 2012 (beyond the time frame of the 18 month evaluation) CLP released a USAID-like Annual Program Statement (APS) to attract larger national and international organizations to implement grant activities. The APS offered that $6 million could be obligated for such projects at this time. Preliminary applications totaled $12 million. Assuming that: 1) CLP can launch and sign a similar level of grants every three months beginning in March of 2012; 2) the obligated funds can be disbursed before the end of June 2013; and 3) present program and overhead costs are constant, then CLP will realize a better balance of grants to non-grant costs by the end of the FY 2013. Graph 2 below shows a projection of cumulated funds based on these assumptions: Graph 2 Projection from January 2012 to June 2013 0 5,000,000 10,000,000 15,000,000 20,000,000 25,000,000 30,000,000 35,000,000 40,000,000 45,000,000 7/10/2010 9/10/2010 11/10/2010 1/10/2011 3/10/2011 5/10/2011 7/10/2011 9/10/2011 11/10/2011 1/10/2012 3/10/2012 5/10/2012 7/10/2012 9/10/2012 11/10/2012 1/10/2013 3/10/2013 5/10/2013 Grants Programs Operations (Overhead) With these assumptions, CLP will reach an intersection of overhead costs to grants in October of 2012. Of note is that this model projects a total of $41 million in grants compared to the $51.8 million called for in the CoAg by the end of the third year. If this model can be carried out, then, in contrast with the 65% YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 26 overhead shown in reference to Graph 1, the overhead, compared to grants, will be 37% in June of 2013. If CLP were to reach the target of $51.8 million in grants, then the overhead would approach 30%. Conclusions This is an expensive grants program to implement, even when allowing for no increase in non-grant costs going forward. The pace of grant invoices is an indication of the slow growth of the grants portfolio and also an indication of the cited problem that some of the implementing partners could not do effectively conduct their own procurement and had to work through CLP. The start-up period demonstrated very few grants and, accordingly, low disbursement rates. The slow start-up is attributable to the protocol issues occurring precisely when the quick start grants program was to have been in place and therefore field personnel could not be placed in the governorates. The second most important factor was the delay in staffing caused by the Arab Spring that slowed the staffing necessary for more and larger grants development grants. Nevertheless, the final invoice for the 18 months (December of 2011) had more than $l.2 million in grant costs. CLP will not meet the financial targets outlined in the CoAg without significantly scaling up grant activities. The current $6.0 million APS is not sufficient unless replicated as per the model every quarter.82 The purpose of this financial analysis is not to explain what happened within the grants portfolio of CLP, but rather to point out the finding and conclusion that the costs of implementing the program to achieve the objectives to date, and that the projected costs going forward are expensive unless changes are made as explained in the model above. In retrospect, the combined protocol delay and the turmoil in management at both CLP and USAID during the Arab Spring did frustrate the original grant making strategy, but the overhead and staffing costs continued to accumulate. In part, these accumulated costs are a burden to the grants to costs ratio. Recommendations It is recommended that the management of CLP undertake a major effort to bring about a significant scaling up of activities through direct implementation, expansion, or extension of existing grantee activities, and use large, international NGOs so that CLP may achieve the grant-level targets outlined in the CoAg. 5.2 Systems and Sectors In order to answer many of the questions posed in the SOW and to have a sound basis for making the recommendations requested, the evaluators divided the structure of the CLP into five sectors and three systems. The objective of this division is to be able to analyze the staffing, implementing partners, level and number of grants, the geographic coverage, accomplishments compared with targets, and to describe ROYG and participant satisfaction.83 Each of the sections includes: a description of the purpose and evolution, findings, analysis, conclusions, and recommendations.84 The evaluators describe the following systems: 1. Community Mobilizers – the system for promoting community development and identifying the grants needed 2. Grants Management System - the $78 million center of the CLP strategy for implementing the program 3. Monitoring and Evaluation - the system for tracking the inputs, outputs and results 82 $6.0 million was all that Creative had available at the time of the APS to commit. 83 In the YMEP Third Party Activity Monitoring Review of CLP, in October, 2011, CLP grants were found to be timely and more than adequate for assessing the satisfaction of the ROYG and beneficiaries. 84 The guidance for the report comes from “EVALUATION Learning from Experience,” January 2011, USAID Bureau for Policy, Planning, and Learning, YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 27 The CLP was structured around five sectors that would contribute to the intermediate results. These sectors employ most of the employees and are directly linked to the ROYG, implementing partners, and awardees: 1. Agriculture 2. Economic Growth 3. Governance and Civil Society 4. Health 5. Education Each of the systems and sectors is presented using the standard USAID evaluation format. In this way these sections can be used as stand-alone items for discussion by mission specialists and CLP sector and systems leaders or managers. 5.3 Systems 5.3.1 Community Mobilizers Background CLP’s strategy for community-based development required Community Mobilizers who could go to the communities to begin a process generally called participatory rural appraisal (PRA). The idea is that the mobilizers meet with communities to glean the input of all social strata, male and female, and the leadership to facilitate a process of discussion resulting in the articulation of the community’s priorities and, in some cases, establish a plan for development. This process of facilitated discussion can have a dual output of capacity building for sustainability and increased pluralism within the community. The other important output from the process would be projects that can be quick solutions to urgent needs, e.g., a bridge, water project, school repair, etc. According to the CLP model, those immediate and short term activities can be the basis for longer term projects for social and economic development. Hence this system of Community Mobilizers was crucial to the model proposed by CLP and clearly stated in the awarded CoAg. One of the proposed implementing partners, the Social Fund for Development (SFD), has for many years trained and had a training program based on the PRA methodology under various international donors such as the World Bank and USAID. Although the original intent of CLP was to work with the SFD, that relationship did not materialize.85 However, because the CLP COP had established communication with the SFD, CLP was able to recruit both experienced SFD Community Mobilizers as well as some who were PRA trainers. CLP, as part of this relationship, built a team of trained college graduates, many of whom have also been involved in at least a year of PRA practice. This provided an opportunity for PRA graduates to work in the rural areas of Yemen for SFD as well as the skills to work for other development agencies. This training was also an opportunity for CLP to recruit Community Mobilizers from diverse academic backgrounds and real-life experience in the districts where they would work. Staffing Staffing up was not timely for the purpose of a quick start. As described in the historical narrative, the protocol delay set the hiring of field staff back by two quarters. The staff hired as mobilizers, however, was appropriate. The manager and first Community Mobilizer were hired in late 2010, but the remainder of the mobilization team could not be hired until January 2011. By March 2011 there were two Community Mobilizers (one male and one female) in all eight target governorates. They all came from 85 Correspondence with previous CLP and USAID personnel indicated that the SFD had ample funding from other international donors and the amount of funding ($2 million) offered by CLP was not enticing enough. Nevertheless, CLP and SFD planned to collaborate on the identification of projects. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 28 appropriate local cultural backgrounds that enabled them to circulate among their governorates and districts without being involved or threatened during periods of instability. Indeed, after a time,86 they were provided offices in the same buildings as the governorate staffs in their respective governorates which helped CLP’s coordination with the governors. They were uniformly college graduates and trained in the PRA methodology necessary to implement the community-based model. All of the Community Mobilizers interviewed (twelve from seven governorates) stated that they were able to carry out the CLP process of project verification and identification without harassment. The Community-Based Process CLP, however, did not implement the community-based process described in the CoAg for the following reasons:  The SFD had systematically identified community needs in most districts and governorates using PRA methodology prior to CLP being funded  The Governors and District Councils had already identified priority projects  It was urgent to identify and verify quick start projects that CLP could fund. Thus, the CLP model became one of verification of identified projects in the communities. The role of the Community Mobilizers, as was explained by them uniformly in interviews, was to go to the governorate and district offices to look at the existing development plans. They picked projects from those plans that fit the areas that CLP was mandated to support, and then they went to the communities to verify the need and interest. Based on the verification, the Community Mobilizers wrote up the grant ideas using the CLP format and sent them to the small projects Manager. Again, this is evidence that the original model, the community-based development model, was not implemented. This short cut in the process boded well for the quick generation of the small impact projects desired for the start-up.87 That is, all of the usual community development efforts, the formation of Community Action Groups (CAGs), and the time involved were saved. However, three factors intervened to slow down the quick grant process:  The grant processing system within CLP  The USAID Mission hold on grant making in early to mid-201188  Non-utilization of 89 the Community Mobilizers by CLP when the legacy focus began Data Management Based on the CLP instructions and training, the Community Mobilizers began the project identification and verification process. The Mobilizers sent their proposals to the CLP Manager for mobilization in Sana’a. The proposals had to be translated from Arabic to English for the expatriate sector supervisors to review. However, to the general disappointment of the Community Mobilizers, their proposals were neither accepted nor rejected.90 Many of the proposals submitted by the Mobilizers to the CLP office in 86 In some cases they worked from their homes for a few months until CLP made arrangements with the governors. 87 As described in the Community Mobilizer System analysis, less then 5% of the grant ideas (approximately 240) generated final grants (about 12 of 240 mobilizer-generated ideas). 88 The hold came from the Mission’s perception that the initial grants proposed did not represent a strategy for having an impact and also that the quality of the proposals was poor. The Mission was frustrated by consistently receiving poorly prepared initial ideas from CLP and then by inadequately prepared final proposals. At the same time (1st and 2nd quarter of 2011) the Mission and CLP were working on a revised PMP to reflect the Mission’s new approach. 89 The evaluators concluded this because: 1) proposals were not processed, 2) travel advances were not given 3) reimbursement was slow; 4) after a year fewer than a dozen of over 200 grants were funded; and 5) grant proposals that were not acceptable were not explained to the Community mobilizers. 90 The interview with the woman who had handled the translations at the beginning revealed that this chore was not her main role (Public relations was her title) and that she also handled some finance paper processing. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 29 Sana’a were, for the most part, filed away without having been translated into English and were, therefore, not processed or utilized. Although the Community Mobilizers kept lists of their grant proposals, the central office and Manager of the Community Mobilizers in Sana’a did not have a data base for tracking the status of proposals received and decisions made. The lists were kept in MS Word files instead of entering them into the OTI data base or creating an alternative tracking system. In the interviews with the Community Mobilizers in the CLP Aden regional office (representatives from Abyan, Lahj and Aden), of 70 projects submitted none were accepted and, as far as they know, they are still pending. In the case of the four Community Mobilizers interviewed who represented Amran, Shabwa, Marib and Al-Jawf,91 of 105 proposals made during the first 9 months of 2011, only four were accepted. The stream of potential projects flowed one way, that is, upward from the Community Mobilizers, but feedback or approved grants seldom flowed back to the field. Community Mobilizers were seldom informed of why their submitted grant proposals were not approved or not reviewed. Feedback would have helped to build the Mobilizers capacity and their utility to CLP. The early part of this period coincided also with the two month hold put on projects by the Mission until CLP completed vision statements for each sector in April 2011. In all fairness, the startup period for the Mobilizers (the first two quarters of 2011) coincided with the Arab Spring, but when interviewed about this period of their work in their home districts, the Mobilizers said that they moved about freely and continued writing and submitting small grants proposals. Hence, the Arab Spring as an urban set of events, was not an impediment for some of the mobilizers in their rural districts. For some mobilizers the danger was in their urban centers because of bombing. From the perspective of the CLP office in Sana’a and Aden, the problem was receiving the proposals and the general perception of being closed down at the urban CLP offices. The backlog of small project ideas and proposals became a bone of contention in September 2011 when the new COP conducted a retreat and review of systems. The Manager of the Agriculture sector reported92 to the evaluators that of 30 water ideas that the Community Mobilizer department claimed to have sent to the Agriculture sector for review since May of 2011:  Only seven had actually been received  Three had been approved and signed  Two had been returned for more information and  Two were rejected Another negative aspect of the CLP mobilizer system was the slow start for providing advances for travel expenses. Some mobilizers did not receive their advances until the third month of employment; then liquidation of those advances was not timely and most Community Mobilizers said that the liquidation of the advances did not come until they were at least a couple of months out of pocket and pressed the CLP for reimbursement of their expenses. That is, they had to use their personal salary money for project related expenses. The other universal complaint from CLP Community Mobilizers was the lack of office supplies and fax and email connections. Some complained that transportation was not adequate in their areas and that larger advances and authorization to rent cars and drivers were needed. It was found that the Governance and Civil Society sector had to hire specialized Community Mobilizers because the present Community Mobilizers did not have the appropriate background to promote governance activities. 91 The Community Mobilizers from al-Dhale were not omitted for any reason other than that the evaluation team asked CLP to provide a mobilizer from four of the five northern governorates. 92 Memo from Mohammed Ilyas, Agriculture sector Manager, of 28 SEP 2011 Re: CMS Workshop to Abdulsalam Al-Zubayri, Community Mobilizer Manager. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 30 Conclusions As of the time of this evaluation, CLP had not set up a system for supporting the Community Mobilizers in the field or to effectively receive, review and respond to their suggested grant proposals. During the field component of this evaluation various CLP and Creative grants and program managers sat in on the evaluation interviews with some of the Community Mobilizers; they took careful notes and told the Community Mobilizers that they would resolve these issues. The incomplete interaction between the Community Mobilizers and the central CLP office in Sana’a generated frustration and a loss of face and credibility in the field for the Community Mobilizers. Considering the original intended use of the OTI data base93 by CLP for tracking quick impact projects, the lack of a data base of project ideas was inexcusable. Moreover, the lack of response to submitted project ideas from the Mobilizers meant that the initially desired quick impact and show of interest in the field was lost. The desired good will between CLP and the governors and district councils has also been lost because ideas submitted to CLP Mobilizers from governor and district council offices were, in large part, not processed into grant activities by CLP. At the IR level94, the contribution that these types of projects could have made toward generating popular trust and confidence in the government was also lost. In spite of the findings above, the renewed interest in quick impact projects makes this backlog or bank of projects relevant and could, if utilized effectively, become a source for increasing grant action if the projects were approved and moved out of the central CLP office. Recommendations Quick action should be taken on the review and approval of the pending projects submitted to the CLP office in Sana’a over the past 15 months. The role of the Community Mobilizers in CLP should be reconsidered. At the present time they are not being used to generate new projects, but they could be well utilized if they were oriented toward specific sector or system needs. For example, some Community Mobilizers have become integrated into the M&E effort to verify compliance of grantees. They are knowledgeable about the main cities and large towns of the governorates and that focus would fit with the urban focus that USAID has emphasized in response to the Arab Spring. Considering the diverse background of these Community Mobilizers, their training, long standing involvement in CLP, and their cultural relevance to each governorate, they should be incorporated into the program as a management strategy for integrating the development efforts across components in their specific governorates. The evolving structure of having Community Mobilizers under one supervisor, additional Community Mobilizers under each sector manager, and compliance officers under M&E only increases the complexity of the current CLP structure and contributes to stove piping instead of the project integration desired. Their role as compliance or M&E verification officers has been a creative move by the M&E manager, but this role needs to become part of a more comprehensive job description modification. 5.3.2 Grants Management (GMS) Introduction The Community Livelihood Project (CLP) is by definition a grants program. Grants are to be the means for improving the livelihoods of men, women, and children in unstable communities and also for increasing trust in the ROYG. A total of $72 million in grant activities was planned. 93 The use of the OTI grants tracking software is explained under the Grants Management System that follows. The OTI data base will track grants from the idea stage through completion, but the ideas generated by the Community Mobilizers were not entered and tracked. 94 CLP IR 2.2: Local governance capacity strengthened (supports USAID Yemen IR 2.2 - Local governance and capacity for basic service provision improved) YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 31 Findings Systems From the Award in June 2010 through August 2011, the OTI database was used for grant monitoring and reporting. OTI’s database does not provide the breadth of reporting that a more sophisticated data base allows, nor was it designed to do so. Moreover, this database does not allow for grant modification and significant changes in implementation. With the departure of the first CLP Chief of Party twelve months into the eighteen month evaluation period, and the change toward longer-term projects requiring tracking against indicators over time, the need was identified to report more fully on grant activities. Given Creative’s extensive experience with grants management and monitoring and evaluation systems, Creative should have launched a comprehensive grants management and M&E tracking system at that time. Ideally, such a system would have been started in the second quarter of the CoAg. Instead, an Excel spread sheet approach was developed and this system was in use through the last quarters (September –December 2011) of the evaluation period. The Excel-based system was not any more appropriate for the M&E needs of CLP or reporting to USAID than was the OTI-type system. Finally, in October 2011, USAID/Yemen was introduced to a more formal grant management system operated by Creative worldwide with commitments to implement a robust grant system based on this model. As of February 2012, the more formal system had not been installed, and the grant monitoring and reporting has been through the informal Excel-based grants tracking system. While not ideal, the Excel-based system did provide more tracking information than previously available. Several issues arise when using an informal system. First, information reconciliation between multiple, informal and formal systems – i.e. finance, monitoring, procurement, etc. – usually are not in sync. The evaluators found this to be true during the evaluation. Second, there are more “write” privileges under an informal system than there is in a formalized grants management system. Third, the data entered is only as relevant and correct as the information inputted. In other words, bad data can populate grant reporting fields. Finally, a formalized system allows integration of information across systems and across the scope of CLP. It was found that M&E, along with finance and other systems, were not joined with Grants Management to provide consistent and timely data. This consolidation and integration of various CLP data systems has not been realized thus far under CLP (up until early February 2012), and each request for information from the Mission generates a significant amount of work by CLP staff to ensure a correct response, whereas a formalized grant system could generate information on a faster and more efficient basis. All references in this report to numbers of grants and amounts obligated and disbursed are from the grants data base closed on 31 December 2011 that had been generated by Creative home office staff on assignment to CLP in Yemen. Staffing Grants management has not had staffing qualified to establish and manage the needed grants management system throughout the evaluation period. Moreover, the evaluators did not find grant management as an assigned priority anywhere in the documentation reviewed or by actions on the part of the implementer. The first Grants Manager left prior to the departure of the first Chief of Party and was not replaced. His qualifications were not reviewed by the evaluators. The second Grants Manager is a former Foreign Service National employee from USAID/Somalia whose grant management expertise was learned during the last twelve months at CLP. He is a qualified “second in command” but he has not had the training necessary to assume the responsibility as a Grants Manager or to develop or implement a full grants system. Expatriate technical assistance was provided to the Grants Manager on a very ad hoc and short￾term basis, but a formal system was not introduced. Usage of the OTI data base and an informal tracking sheet led to staffing up related to grants management in all the sectors as well as in the grants management office. Redundancy in responsibilities is seen as YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 32 there are many individuals involved in the process. While the evaluation was taking place, but after the end of the 18 month evaluation period, Creative’s newly assigned expatriate staff was helping to formalize procedures and processes. These included how grants were processed and who has authority in approving or moving grant applications forward. Previously, this was more ad hoc in nature with varying degrees of participation. A case in point is grants generated by the field Community Mobilizers, the largest source of grant activities under CLP. Concepts came into Sana’a in Arabic, and depending upon the availability for translation, were either lost or languished for long periods. This was an oft mentioned complaint by Community Mobilizers as reported in the section of this report called Community Mobilizers. Conclusions Grants management was not accorded the responsibility it needed throughout the evaluation period. The failure to install a real grants management system immediately following the first quarter of the Cooperative Agreement is a project failure, one which reverberated through CLP during the first eighteen months of the Award. It is unclear why Creative, a firm well experienced in grants management, has limited the priority assigned to this important element of the Award. Given that this system should have been developed during the first two quarters, and knowing that Creative Associates has a global project management information system for grants management that it uses widely in many of its programs, its delay cannot be attributed to the protocol issue or the Arab Spring that began eight months after signing. The evaluators concluded that many of the informational frustrations expressed by USAID/Yemen and internally within CLP could have been avoided if an effective grants management system were in place. While it was not anticipated, implementation of grants by CLP saw that more than 75% of grants were directly implemented by CLP due to a lack of counterpart abilities. This was further impacted by not having formalized grant procedures understood and in place arising from a formalized grant management system. Recommendations Creative should ensure immediately that a formalized grant management system is in place, fully integrated, and relevant personnel are assigned to implement the grants management system. The failure to do this previously is why many of the issues in implementation have arisen, and there is confusion on who has responsibility and what happens to grant ideas once they are generated. Also, a formal grants management system will reduce redundancy and facilitate accuracy between the various data bases in place under CLP. Finally, it is only with a formalized grants management system that Creative has the opportunity to recapture most of the grant targets in terms of dollar amounts prior to the end of the CoAg. 5.3.3 Monitoring and Evaluation Background Monitoring and Evaluation (M&E) was planned to play a role beyond the usual tracking of indictors and the provision of information for management decision making. According to the CoAg,95 the M&E system was to be designed so that the development hypothesis could be tested: The development hypothesis for CLP is that targeted development interventions at the sub￾national level can positively affect stability in Yemen. Implementation of this project will test this assumption.96 95 Cooperative Agreement No. 279-A-00-10-00032-00 and subsequent modification #2 P.33 96 Creative Associates, April 2011. Community Livelihoods Project Performance Management Plan. p.2 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 33 The development hypothesis was more than a simple hypothesis, it was a theory or paradigm97 of action targeting identified drivers of instability in Yemen and a methodology proposed for increasing stability of governance in a context, perceived at the time, of increasing regional instability. In addition to the usual M&E process indicators and outputs, such as changing maternal/infant mortality, agricultural production, training and so forth, the PMP contained a plethora of indicators98 for establishing a baseline of instability and drivers of instability that were to be used to guide the geographic targeting of project grants and activities. The dependent variable was the level of stability and the independent variables predicting changes in stability would be the project inputs and outputs. These specialized M&E functions were in keeping with testing the hypotheses, but these M&E needs only complicated the start-up period when the urgency was to have been oriented toward small, quick impact projects based on local-level indicators of the drivers of instability. 99 Findings Data System Design Because of the initial strategy of doing OTI-like small, quick impact projects, Creative’s experience in other countries with the OTI grant managing software, and the first COP’s experience with that same software, the initial grant tracking software was the OTI system. That system is ideal for tracking disbursements and participants in multiple, small projects, but the grant or project is the primary subject or unit of analysis and not the state of the participants or communities. The OTI database system is not designed for effectively tracking baseline and subsequent, time series indicators,100 especially when the unit of analysis is the community as per the work plan.101 Creative intended102 to scale up the M&E system to manage what had been planned as the phase of longer term, development projects, but the decision and M&E system were delayed until the end of the third quarter of 2011 because the original M&E manager left and because the period of instability made recruitment of an expatriate M&E Manager difficult.103 This quarter CLP submitted and obtained USAID approval for the Program Management Plan. With approved indicators in place to accurately measure program progress and achievements, both the program department and M&E unit better track the process of grant activities and measure their progress to stated goals and objectives of the CLP program. (January – March 2011, p. 5) Creative did place an M&E Manager in the field from February to May of 2011 following up on that quarterly report, but that person did not develop a system. Accordingly, CLP did not start the design of the M&E system until the seventh month (January 2011) of the implementation process. It depended on inappropriate software that was inadequate to effectively track comprehensive M&E data from the program. Indeed, it was not until the end of the fifth quarter of the project (September 2011) that Creative placed a qualified M&E Specialist on the CLP staff. 97 USAID, 2011 “The Development Response to Violent Extremism and Insurgency.” Policy Paper September 98 Ibid. p.4 and Annex II 99 It should be pointed out that CLP did not have the experimental design expertise necessary to carryout such a methodological feat under M&E. YMEP may have had that expertise in its first COP, but, when comparing the failure of the design of the drivers of instability survey that YMEP contracted, it is clear that YMEP did not have that methodological capability to test the development hypothesis. 100 Merschrod reviewed the OTI data-base from this perspective in 2009 for a similar project in Colombia 101 CoAg Modification #2 Pp. 10-12 102 CLP Quarterly Report JAN-MAR 2011.p.5 103 The original AOTR observed that the CLP was not able to place appropriate M&E staff and the responsibility went back and forth between the COP and DCOP. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 34 During the evaluation (January and February of 2012) the evaluators found that the M&E Manager had established an appropriate software system that could track and report on the indicators by multiple units of analysis (project, geographic, funding lines, etc) and the M&E local staff were gathering data from the grant files, the OTI database, and the excel sheets to load the information into the data base. The present system includes appropriate staffing so that field verification and follow-up data is being gathered on training participation and impact at the individual level. Unfortunately, baseline data (both community and public opinion) on the AO-level and stability survey data to test the impact of CLP activities on community stability had not been gathered through December 2011. CLP plans to be able to generate reports with the system by the end of year two (June of 2012). The CLP M&E team, that will be collecting data throughout grant implementation activities and trainings, will clean, analyze, and report on this data by the end of Year 2.104 Staffing The current M&E Manager is an experienced, international M&E expert who has managed M&E systems for larger and more complex projects. Her local staff now includes a leader for each of the five sectors and compliance officers for field verification and follow-up. The M&E system is now poised to track inputs and outputs, plus it will generate periodic reports for CLP. The interaction between the CLP M&E expert and YMEP during the last two quarters has provided crucial technical assistance for CLP so that appropriate questionnaires were prepared for each sector and, importantly, so that the Yemen Clearinghouse can receive the CLP data for Mission purposes and also for the GIS services that YMEP can provide to CLP in the form of maps to show the geographic coverage of the CLP activities. One of the mandated areas for CLP M&E is that its sub-grantees and collaborating partners must track their own data and forward it to CLP. Interviews with CHF (a July 2011 grantee in the field of education) verified that, indeed, they had received technical assistance from CLP on the design of the CHF indicators and data system. Conclusions  The CLP M&E system has, finally, been well staffed. During the last quarter (October-December 2011) they have installed the necessary systems, that is, software, forms, and procedures, to generate more substantial and accurate reports for USAID and CLP management.  CLP has provided M&E technical assistance to grantees so that grantees can fulfill their contractual M&E responsibilities  Now that CLP has a capable M&E staff, YMEP has been an appropriate and timely source of M&E technical assistance and can fulfill its role.  The training of all 42 Community Mobilizers and compliance personnel in the use of the “Routine Monitoring and Final Evaluation report formats, and Grant Completion Questionnaires” should enhance and inform the GMS.  The baselines for testing the development hypothesis have not been done and testing the development theory is a moot question as far as USAID/Yemen is now concerned.105 104 CLP “Annual Workplan July 2011 – June 2012, p.15 105 Although testing the theory was an interest of the USAID Yemen team at the time that the RFA was written and the CoAg was awarded, by the end of 2011 when the new USAID Yemen team changed the focus to more classical development, the strategy/theory was no longer of interest. Interviews and correspondence with past and present mission staff indicate different assessments of the context and the appropriate way to approach the challenge of YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 35 Recommendations  Continue the staffing of M&E as it is.  Focus M&E resources on new grants and grantee ability to do baselines and indicator measurement.  CLP M&E and compliance officers should continue to try to reconstruct the past and enter that information into the data base as sector-specific baselines.  Formally incorporate the Community Mobilizers into the field verification work with the M&E Compliance officers.  Because of the change in direction noted in the first phase of the narrative, testing the hypotheses is no longer appropriate, the multiple public opinion surveys called for in the RFA and proposed in the CoAg should be dropped.106  Focus on very specific input/output indicators to measure clear CLP deliverables.  Continue the integration of information gathering between Grants Management and M&E. 5.4 Sectors 5.4.1 Agriculture Introduction The CLP Agriculture sector, with a grants portfolio of $1.76 million, is the third largest of the five CLP sectors. Disbursements through 31 December 2011 total $ 683,379. The purpose of the sector is to increase food production and trust in the government by the farmers. To date a total of 26 grants have been issued to four implementing partners in all target governorates and six cities of focus. The activities funded by the grants range from teaching women to manage livestock and food processing to training urban gardeners on how to produce vegetables in small spaces. The importance of the strategy used by this sector to date also includes working through and closely with the ministry of Agriculture and Irrigation so that, through training of trainers and direct work with MAI personnel and farmers, there is renewed trust in the MAI. This sector started early in the program based on the continuation of previous USAID agriculture programs and the interest in the governors in the activities. For the latter reason, the grants with MAI were signed during the protocol period when the CLP and Mission were stymied when trying to establish the program. This sector, however, needs to be scaled up to achieve the goals set for CLP. Findings Staffing By January 2011, CLP had hired 15 experienced agriculture coordinators or livestock specialists and an overall sector coordinator for the Agriculture sector. It should be noted that this sector was staffed even before the small project Community Mobilizers, when CLP assumed support for the legacy program working with the Ministry of Agriculture and Irrigation. Clearly, from the start, with the assumption of the MAI legacy projects, CLP was moving ahead with the longer-term development approach in parallel with the short-term quick impact grants program that required Community Mobilizers. By March 2011, Creative had identified and added an Agriculture sector Leader with international experience. It should be pointed out that although this was a key position, the project design and CoAg107 did not recognize this development and the unforeseen transition in government of Yemen. (Source: past and present CLP and USAID personnel – April 2012.) 106 If the Mission wishes to do public opinion surveys to gauge the changing attitudes, then a specialized consulting group should be contracted because these surveys are beyond the expertise of the CLP M&E unit. 107 In the CoAg, prior to Modification #2 of June 2011, an Agricultural specialist was not included among key personnel. With modification # 2 the Sector Leader became a key person. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 36 as a “Key Position” because the original conception, for the first two quarters, was that of a small grant, immediate impact program. It is an example of how on the one hand the management was flexible and quick moving, and, on the other hand, how project implementation constantly needed, and still needs, to move outside of the formal, written bounds of the CoAg. Implementing Partners The key implementing partner has been the MAI. During the latter part of 2011, in response to the Mission request that CLP add an urban focus to the program, CLP added two NGO implementing partners to the agriculture sector in order to implement a program focused on urban gardens. The two new partners are Yemeni Association for Sustainable Agricultural Development (YASAD, in Aden, Sana’a and Taiz) and Althuraya (in Dhamar, Ibb and Hudaydah). The Childhood and Youth Development Association was granted funds for water filters to improve potable water. This is more appropriately a Health Sector subject; nevertheless, it could be considered an indication of the integration of the sectors even though one would question the accounting line charged with the grant. As of 31 December, 2011, 26 grants had been obligated for a total of $1,759,325, and disbursements through 31 December 2011 total $ 683,379. Among the earliest grants by CLP, the agriculture sector signed ten grants with the MAI totaling $348,516 during September and October of 2010 and in May and June of 2011 seven additional grants for $648,122 were signed. In September 2011 six grants were signed with YASAD and Althuraya totaling $590,726. Geographic The initial grants with MAI covered the eight rural target governorates. The targeting of these CLP activities was not the result of community assessments of drivers of instability but rather it was based on the location of the existing MAI and the USAID legacy agriculture project. In contrast, the grants that were signed in late 2011 for YASAD and Althuraya to work in urban areas were a rapid response108 to the need for an increased urban focus in six governorates with urban unrest during the “Arab Spring” that lasted most of 2011109. Accomplishments vs. Targets – Urban Households The urban target of 1,500 households (250 each in Aden City, Dhamar, Ibb and Hudayda) has been reached. The households are well documented by Althuraya and YASAD, but baselines and targets such as increased area, production and diversity of crops are not established. That being said, both organizations keep data bases with some pertinent information about the participants and the extent of the diversity of their gardening and small stock base. But they do not record baseline data on productivity and production, either consumed or sold, or income derived. In the follow-on studies that they will do, it should be possible to compare the baseline diversity and area with changes in the number, type, and extent of production. However, comparing productivity and production with a baseline, either consumed or sold or income derived, is not possible. In their initial surveys to identify interested urban gardeners, they have data on families that were not interested in participating nor were appropriate for inclusion. Although not designed as a control group, these neighboring families may provide a comparison group. Unlike the more recent urban-targeted grantees with well-developed data bases, the other implementing partners and CLP did not do baseline surveys and they have not been meticulous when it comes to accomplishments. As indicated in the M&E section, only during the last quarter of 2011 has there been a 108 The response was to the USAID Yemen request for an added urban focus YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 37 system in place and information gathering to reconstruct the database on past activities. The following explains a frustrated attempt to use CLP and other records to compare targets with achievements. Accomplishments vs. Other Targets This is a difficult assessment to make in the case of the CLP for two reasons: 1) the state of the M&E system during the period evaluated, and 2) the initial work plan did not have complete sets of targets by sector. The first work plan (18 November 2010) aligns activities with the Intermediate Results (IRs) as components, and then within each component the sector activities that correspond to the component are listed. From this document come targets for the first year through June 2011. For the second year (July 2011 to June 2012) there is also a matrix, and, as an improvement, it is prepared by sector. There is also a narrative in which goals against targets are presented by quarter by sector. However, in some cases they are in references to bases without the base, e.g., “During the first quarter, 50% of the training will be completed.”110 One has to refer to the matrix version and then sum the appropriate training to compare. Then too, when looking at the quarterly or annual reports, there are narratives by sector with some accomplishments, but not the targets. The appendix in the latest quarterly report111 is a performance data table that refers to indicators, but those indicators, in some cases, are aggregates across sectors and it is not designed to compare results by sectors. One of the easiest ways to research the accomplishments was by means of the narrative draft “Fact Sheets” 112 that were generated to cover each sector from July 2010 through November 2011. These figures were used to assemble the following table and then the targets were gleaned from the work plans. It has to be noted that the fact sheets and the categories used in quarterly reports did not always agree and the targets for the work plans did not line up by category. The data on targets and accomplishments is not accurate enough to arrive at a satisfactory conclusion. Thinking through the data below, if one assumes that nothing was accomplished until after July 2011, then the percentages in the “Percentage of Goal” column does not look too bad for the 50% mark of the year two plan. However, that is probably not a sound assumption since some agricultural activities were carried out from 2010, especially in the training of extension agents, but we do not have the time-specific and baseline data. ACTIVITY Agriculture sector Target 2nd Year Work plan Reported113 from June 2010 through November 2011 % of 2nd year Goal (1) Improve capacity of Extension Agents (EAs) in improved crop management Trained 300 EAs 180 60.0% (2) Improve capacity of MAI Veterinarians (Vets) in animal health and livestock management Trained 175 Vets 75 42.9% (3) Improve capacity of farmers in good agricultural practices Trained 4000 farmers 2689 67.2% (4) Develop skills in females on food processing Trained 550 females 239 43.5% 110 CLP Annual Workplan July 2011 - June 2012 p. 4 111 CLP Quarterly Report October-December 2011.Appendix. 112 Public Information Office Draft Fact Sheets, November 2011 113 Public Information Office Draft Fact Sheets, November 2011 and CLP Quarterly Report October-December 2011, p. 7 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 38 ACTIVITY Agriculture sector Target 2nd Year Work plan Reported113 from June 2010 through November 2011 % of 2nd year Goal (5) Improve skills of females in animal nutrition and livestock management Trained 550 females 174 31.6% (6) Support vulnerable livestock farmers by providing them livestock inputs supported 550 families 1,982 360.4% Farmers improved farming livestock management practices. not in work plan 3,641 (7) Develop household capacity in home gardening for income generation and health improvement through nutrition management Training and material support for 5,000 households 1,500 30.0% (8) Employ unskilled and skilled workers for 2 months under cash for work grants employed 160 workers not reported (9) Improve capacity of local beekeepers in improved beekeeping practices for rural business development Training and material support for 4,664 beekeepers not reported Family Water Filters no target 2,500 For the other sectors, it is an equally frustrating chore to try to quantify and compare targets with accomplishments. Hence the recommendation in the M&E System section that the data bases be reconstructed and that the new accomplishments be carefully entered into the new M&E system. ROYG satisfaction Interviews were conducted with the Director General (DG) of Agricultural Extension and Training in Sana’a and the Director General of Agriculture and his extension and veterinary officials in Aden. The main point that the DG of Agricultural Extension and Training made for the MAI was that if it had not been for the CLP, then they would not have had the nation-wide stimulus in the form of rehabilitated centers to work from and training for veterinarians and extension workers. He believed that CLP was the only reason that they were able to reach the farmers and women livestock owners to hold up their image. As with the other public officials interviewed, their wish was for more training and more equipment. One of the characteristics of the Aden Agricultural office is that it is a predominantly urban governorate. Nevertheless, it was one of the first offices in 2010 (June and July) to enter CLP and that began with the first training of veterinarians, women in food processing, and extension personnel the same year. Most of the discussion was about the YASAD urban gardens and small livestock program because of the urban character of Aden. That program was praised, as was the CLP agricultural effort to train extension agents and also to teach women how to process vegetables during the peak production times. Two of the MAI extension women were hired by YASAD to conduct the training funded by CLP and they were happy to be able to be part of CLP in this way. The other theme that the DG raised was the need for food testing so YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 39 that the quality of food being imported would be controlled. Aden is a major port and the Ministry of Agriculture does not have the laboratory capacity or training to conduct such testing. It was explained that this type of program was a USDA and not necessarily USAID program. Another source of feedback on CLP is the assessment of ROYG satisfaction or criticism that comes from the third party monitoring report performed by YMEP.114 Of the nine MAI projects sampled, the consensus of the ROYG interviewees was that the CLP agricultural grants were appropriate and well received and, importantly, that the program needed to be scaled up so that more farmers and districts would be covered. Participant satisfaction This evaluation was not designed115 to interview a sample of beneficiaries; the focus was on the development of the project itself. Nevertheless, the third party monitoring of CLP grants116 reported on participants and opinions on a sample of agricultural grants. The interviewees were generally satisfied with the training that they received. Judging from the commentary, however, it was clear that the trainees had varied backgrounds, and while some were very satisfied and found the training very useful, others complained that the training should have been more in-depth, longer, and that more sessions are expected. The commentaries of the individual participants echoed the commentaries of the ROYG interviewees – satisfaction with the program, but more activities are needed. Conclusions The Agriculture sector is relevant to the goal of increasing population trust in the ROYG as shown by the Third Party Monitoring survey, and it is important for the longer-term goal of improving rural incomes and the supply of food in Yemen. Both the urban and rural grants have been designed to include women in relevant areas of household, farm, and farm-to-market production because both MAI and CLP know the role of women in livestock and vegetable production. That is, gender balance has not been a haphazard outcome because it was designed into the programs. The inclusion of women extension agents has also enhanced the status of women in MAI. Finally, a grants portfolio of $ 1.76 million, for an agriculture sector occupying 70% of the population in a country of poverty, and a country with an established cadre of extension agents and centers is a small proportion for a program with a potential total grant capacity of $70 million. The MAI clearly has the need, interest, and ability to expand geographically and also to concentrate more personnel in target districts. Recommendations This first recommendation comes from the MAI interviews. Increase the trainings for livestock management, vegetable production and water management (both urban and rural) by funding an expansion of activities in all districts with the Ministry of Agriculture and also the other, urban oriented implementing partners. A second area that should be considered for more attention is the vegetable production chain from producer and handlers to the market place. The role of women in the production and handling of vegetables can be enhanced if value is added in the production chain. It was noted by the CLP Agriculture sector leader that during the period of most urban disturbance the prices and the supply of vegetables into Sana’a held steady. That is, in spite of the unrest and road blocks, that particular production chain was not targeted. Fresh produce markets in urban areas usually offer development projects an opportunity to provide new, neighborhood sanitary marketplace projects that are useful for the producer/truckers and appreciated by the urban population. These types of infrastructure projects would contribute to the goal of increasing trust in the government. They would also provide the platform for the 114 YMEP. October 2011. Third Party Activity Monitoring of the Community Livelihoods Project.Tables 8-11 115 See Annex I 116 Ibid Table 12 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 40 integration of the health, agricultural, and economic development sectors in all of the market towns and cities. Usually mayors can gain credit for these marketplace improvements, but if skillfully coordinated then the government health and agricultural offices in the governorate centers can also gain credit. Since Yemen imports 75% of its food, a third recommendation is to study opportunities under the concept of import substitution. On the other hand, one must be aware of inexpensive imports that could undermine local production, especially in the processed food category, e.g. the idea of processing tomatoes or other seasonal vegetables into concentrates, paste or juices when inexpensive production or dumping can ruin such local markets. 5.4.2 Economic Growth Introduction The second sector in size, Economic Growth has a total of $2.439 million obligated in thirty two grants through the evaluation period. Disbursements through 31 December 2011 total $818,000. The focus of the sector is on three priority technical sub-sectors: micro and small enterprise development, value chain and marketing systems, and workforce development. The goals are to increase income, create new jobs, and improve economic development at local, district, and regional levels. Economic Growth realized the first three grants under CLP. Although grant activities started very slowly, the focus has been on youth interventions in job training, entrepreneurship, and vocational training. To date, a total of eleven Implementing Partners have worked with the sector, the largest grantee being Yemen Education for Employment (YEFE). For its part, YEFE is implementing a $1.9 million sub-grant from Creative to Education for Employment (EFE) focusing on vocational capacity development, training, and apprenticeship programs. As with other sectors, Economic Growth has ramped up activities since the fall 20ll, and has broadened its focus to include urban areas and tighter integration with the other sectors. Findings Staffing Staffing for the component has been intermittent at best through the evaluation period. The one expatriate staff member left in April 2011, leaving one experienced Yemeni staff member to manage the activities. This professional has been with CLP since December 2010 and had prior work experience with the Ministry of Industry and Trade, donor-financed projects, and with the Yemen Embassy in Washington after obtaining a degree in international business. Further, he did a good job of holding the sector together and moving the limited number of grants forward until a renewed focus on the sector followed the replacement of CLP’s original Chief of Party in August 2011. A strong replacement expatriate advisor arrived in August 2011 to take over as Team Leader. Since then, the staffing for the sector has grown significantly. Economic Growth anticipates being fully staffed by March 2012 with a complement of up to six professionals. In the ramped up state, the staffing is more than adequate to take the sector forward during the remainder of CLP. Implementing Partners There are eleven implementing partners for the thirty two grants under implementation. Similar to the other sectors, it was found that capacity of implementing partners was limited. This resulted in CLP having to take on most implementation activities, most notably procurement. This has changed somewhat with three implementing partners taking on greater responsibilities: YEFE, For All Foundation, and Together Stronger. Each is focusing on youth employment and cash-for-work activities. It should be noted that YEFE is an original member of Creative’s consortium for the Cooperative Agreement, and it has been active in Yemen since 2008 through the Department of State’s Middle East Partnership Initiative (MEPI) funding. The evaluators found YEFE to be implementing well, increasing the size of its portfolio of activities as it expands its reach, and a solid partner for the Economic Growth portfolio. The main issue YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 41 for YEFE is its “absorptive capacity” with respect to funding. Noting this issue, attempts were made to undertake larger grant activities with Mercy Corps and UNDP. The latter did not go forward because the intent of the proposed activity was budgetary support. Negotiations with Mercy Corps for a $2.0 million grant to target training and financial assistance to micro and small entrepreneur youths in Aden, Lahj and Taiz was in final review during the evaluation period. Granting Level – As of 31 Dec 2011, 32 grants had been obligated for a total of $2.439 million. $818,000 of these grants funds have been disbursed to date. Among these grants, all have consistently focused on youth related activities. Grants have ranged from approximately $12,500 for Goal Foundation to $1.9 million for YEFE. Geographic The thirty-three grants undertaken during the evaluation period covered the eight of the governorates identified in the Cooperative Agreement. This sector responded to the Mission desire for an urban shift by funding, through Goal Foundation for Growth, and YEFE, urban cash-for-work in Taiz, Sana’a, and Aden. Targeting of grants has not been the result of baseline or other assessments; rather it has been at the initiative of implementing partners or USAID’s emerging urban focus. Accomplishments vs. Targets Although the April 2011 PMP does outline targets to be accomplished, baseline data was not developed. As recently as the bi-weekly CLP Report through January 31, 2012, final development of targets and tools to measure progress remained under development for all sectors. The Year One Work Plan does lay out illustrative targets and deliverables. With the Stability Assessments the opportunity arose to develop baseline information, but this was not realized. Although lacking in baseline data, the sector does report accomplishments during the evaluation period. Currently, the Year Two Work Plan is under revision by CLP, and this will change the targets within the Economic Growth sector. Among the accomplishments of the Economic Growth Sector are:  Training youth about business development and management in incubators (54 people)  Training students about business management, microfinance and film production (60 people)  Training associations about business development, management, funding and management (7 associations)  Training youths about small enterprise development, funding and management (125 youth)  Training youths in English, accounting, life skills development, and vocational skills (80 youth)  Provide training, tools, and equipment to targeted farmers for use in bee keeping (125 farmers) Participant satisfaction117 This evaluation was not designed to interview a sample of beneficiaries; the focus was on the development and implementation of the project itself. Nevertheless YMEP’s third party assessment reported on participants’ opinions on a sample of economic growth grants. Most participants surveyed rated their training as “medium satisfied.” Beyond the normal complaints about lack of allowances or other benefits, most found the training useful and wanted more training opportunities. Participants also felt that job opportunities would arise from the activity. Given the private sector focus of the activities reviewed, interviews with ROYG were not recorded for economic growth sector activities in 3rd party monitoring and were not covered by the evaluators. What was noted in the interviews was the lack of sufficient scaling up of these activities as viewed by counterparts. 117 YMEP, October 2011. Third Party Activity Monitoring of the Community Livelihood Project (CLP) YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 42 Conclusions The Economic Growth sector is only now (January/February 2012) adequately staffed and prepared for the remainder of CLP. During the evaluation period, the sector undertook ad hoc and opportunistic activities often generated by Implementing Partners or Community Mobilizers and not in line with a coherently articulated strategy. For most of the evaluation period, the sector was implemented under the direction of one Yemeni staff member. June 2011 saw the assignment of a strongly qualified expatriate advisor as Team Leader. This has been followed by planning for up to seven staff within the Economic Growth Sector for the remainder of the second year. In conjunction with the other changes occurring within CLP, increasing the staff level has meant a retooling and direction of the Economic Growth sector and developing more initiatives as part of sector activities. Late in the evaluation period, the activities were often in response to directions from USAID/Yemen and are reflected in the move to more urban￾focused activities within Economic Growth. The movement toward larger, international Non-Governmental Organizations is the growth model for this sector. The potential will depend upon the APS that was issued by CLP in January of 2012. Building upon notable successes by YEFE, this model of using international organizations is counter-balancing the heavy work load generated by using less than qualified local counterparts who do not have implementing experience. Two exceptions noted are For All Foundation and Together Stronger. Both organizations continue to grow but do have noted absorption capacity issues as they relate to abilities to implement. They will require capacity building in order to scale up. Recommendations As a non-legacy sector, Economic Growth should define a vision and determine how it fits within CLP. For the emerging approaches, the April 2011 Vision Statement is outdated at this juncture. While its focus on youth oriented initiatives is impressive in many ways, it is unclear how these will roll up into a coherent package. It is also unclear whether it should remain a stand-alone sector or should be merged with Governance to further focus youth related activities and build a volume of grants supportive of a separate operating unit. The evaluators believe it should be merged with Governance as the latter’s portfolio is strongly youth oriented and not particularly governance oriented, and neither Governance nor Economic Development, as currently implemented, are significantly different in terms of grant activities The full complement of Economic Growth staffing will exceed the needs at the current volume of grants. Without a strategy, the sector will continue to identify and implement ad hoc initiatives. Ramping up international NGOs and building upon successful implementing partners will serve to balance staffing and the overall portfolio. There should be a rationalization of staff needs to an articulated, coherent strategy that fits within the rubric of CLP. Without a ramping up and focus on Economic Growth, it will remain in many ways an outlier program. 5.4.3 Governance and Civil Society The fourth sector in size, Governance and Civil society has a total of $1.187 million in eighteen grants through the evaluation period. Disbursements through 31 December 2011 total $534,000. The goal of the sector is for civil society organizations and independent media to increase their capacity to a level of competence to effectively engage local government on community needs and to monitor and hold local government accountable. By design, it was to be the “mirror” image of the CLP companion Responsive Government Project (RGP) whereby CLP implemented from the bottom up and RGP top down. The technical priority areas of the sector are local government capacity development, community and civil society development, and youth organization development. Grants do not mirror this focus. To date a total of eleven Implementing Partners have worked with the sector, the largest in terms of number of activities YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 43 has been Apex Consulting.118 The largest single recipient of grant assistance was the Ministry of Health, which had five grants. The dearth of counterpart expertise outside Apex and MoPHP led to an emphasis on direct implementation within the sector for the majority of activities. As with other sectors, Governance and Civil Society has ramped up activities through fall 2011, almost all by direct CLP implementation. The sector has refocused its efforts on youth and broadening its reach to include urban initiatives. Findings Staffing Staffing for the sector has been intermittent at best through the evaluation period. No expatriate Team Leader has led the division. While an expatriate advisor was provided to develop the sector’s strategic vision, this individual did not work out and left CLP shortly after submission of the vision in April 2011. Since September 2010, the Governance and Civil Society Sector has been led by a Yemeni professional whose prior experience was as a Foreign Service National (FSN) within USAID/Yemen. While not a governance professional, this individual has grown significantly in the position and has been supported by expatriate staff drawn from Creative Associates or elsewhere within CLP. Plans are underway to have by March 2012 up to nine local-hire professionals on staff. They are to be assigned as Governance Officers, Field Coordinators, and Governance Specialists. It is envisioned that all will write and manage grant activities. The managers of this sector conclude that the original community mobilizers are not prepared for governance and youth activities, and for this reason they believe that specially trained governance mobilizers are needed in Aden to replace the existing mobilizers..119 Implementing Partners Through the first year of CLP, most of the activities were implemented through seven implementing partners. Apex Consulting received eight awards to undertake eight stability assessments of governorates for CLP that were not used by either USAID/Yemen or CLP in developing baseline data or targeting community activities.120 Both parties felt the reports were less than required due to the lack of quality on the part of the sector assessments and significant deviation from what was agreed upon in terms of questions to be answered by Apex. Partners for Democratic Change (PDC), an original member of the CLP consortium, were the first grantee in December 2010, and were used for community mobilization training. The three awards to the MoPHP for furniture in three governorates had little to do with governance function. To date, no international Non-Governmental Organization has been used within the Governance and Civil Society sector. Aside from the APEX and MoPHP awards, most of the grants have focused on youth related initiatives in summer camping, leadership, civic training, sports, arts and vocational training. Very few of the activities implemented have a relationship to the April 2011 Strategic Vision and the elements outlined therein. Most activities have been identified by CLP staff and/or implementing partners as the Community Mobilizers were found to be lacking in skills related to governance.121 For its part, USAID/Yemen sees the activities of this sector as lacking a strategic vision through which a focus of activities can be developed. The governance staff found a complete lack of counterpart capability within the sector. This led to direct design and implementation by CLP late in the evaluation period and the lack of counterpart capability 118 APEX Consulting was used primarily for sector assessments and other analysis needs and not for grant activities related to the sector’s focus. 119 The failure was in the selection of the mobilizers for the small projects such as infrastructure sorts and case for work, but not for governance. 120 One should question if these studies were really appropriately funded under this sector. After all, the baseline studies were really an M&E function or a general program cost for planning all sectors. 121 This was according to the Governance Team Leader. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 44 became the basis for staffing as it relates to grant activities going forward. Specifically, the plan for Year Two envisions each professional staff member writing and managing grant activities. Geographic The 18 grants undertaken during the evaluation period covered five122 of the eight governorates identified in the cooperative agreement. The nine Apex Consulting activities are not included in this allocation because Apex was, for the most part, related to sector or other assessment needs and not actual Governance and Civil Society development. Targeting of grants has not been the result of baseline or other assessments, but rather at the initiative of CLP directly. Accomplishments vs. Targets While the April 2011 PMP does outline targets to be accomplished, baseline data was not developed. As recently as the bi-weekly CLP Report through 31 January 2012, final development of targets and tools to measure progress remain under development for all sectors. The lack of a unified vision and application of the Governance and Civil Society strategic vision designed by CLP in March 2011 has meant that almost all sector activities, outside the original grant to PDC and assessments by Apex, have been opportunistic in nature and implemented directly by the sector staff. These activities have ranged in size from $9,453 to $601,100 with most in the middle range of $40,000. All have been youth focused initiatives. Among the accomplishments of the Governance and Civil Society components are:  Sector assessments undertaken by Apex Consulting  Youth related activities  Thula focused initiatives (integration attempts in an urban setting)  Crisis management training  Furnishings for three Government Health Offices Participant Satisfaction The evaluation was not designed to interview a sample of beneficiaries; the focus was on the development and implementation of the project itself. Nevertheless, third party monitoring reported on participants’ opinions on a sample of Governance and Civil Society grants. Grants under Governance were for specific activities, often not for governance. Provision of health equipment was well received by the ROYG, but was not related to civil society or governance. Participants in youth related activities found the activities supportive, and most responses asked for additional training. Youth camp activities were found to be less well developed, and both CLP and its implementer, Youth Empowerment for Development Association (YEDA) felt that the activities did not meet the expectations of the participants. The eight grants for stability assessments, undertaken by Apex Consulting, do not have satisfaction surveys, although USAID/Yemen and CLP found them generally lacking. Conclusions By definition, Governance and Civil Society fits within the design of the RFA that governed the award of the Community Livelihoods Project. However, it is a component lacking a strategic vision and coherent fit within the Project. If it is to be a governance program, the emphasis on youth activities is far more associated with the Economic Growth Component than with community-based civil society development. Stated another way, it is a CLP component looking for definition and a home within CLP. Its portfolio of 122 The stability assessments are not used in this count, only places where activities that involved support to governance or the goals of the sector are counted. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 45 activities to date raises the question as to whether it should be a stand-alone component or merged with another component within CLP. The lack of expatriate direction does not mean that the component is inadequately led. Rather, it is the lack of a clearly stated and accepted vision. The promise of the sector lies in the second half of CLP, not in performance to date. The disruptions that impacted CLP generally were exacerbated within this sector due to accompanying lack of implementing partners, grant approval refusals, and a general lack of expertise in identifying and moving grants forward. Recommendations As a non-legacy sector, Governance and Civil Society should define a vision that fits within CLP. Its focus to date varies widely from the Strategic Vision developed by CLP and is defined more by opportunistic activities rather than a response to a strategic focus. Ramping up to a full complement of staffing requires a corresponding understanding of how the sector contributes to the overall success of CLP. This is not in evidence as the staffing is being hired. Rationalization of staffing needs against a strategic vision is highly recommended. Equally, the use of local institutions and counterparts has proven problematic for CLP. This is evident by the increased emphasis on direct implementation by sector staff. Consideration should be given to using international Non-Governmental Organizations in ramping up the volume of activities under the sector. Continuation of CLP staff-generated leads, ideas, grants and opportunities is not encouraged based on the degree to which such activities have not had USAID/Yemen approval in the past nor have they fit an agreed upon strategy. Rather, a coherent, mutually agreed upon strategy for the sector should be the emphasis going forward for the remainder of CLP. This is not in evidence and does not respond to the needs of USAID/Yemen for the sector at this time. Continuation without an agreed upon strategy, vision, and work plan will continue to see ad hoc approvals and a lack of coherence within the sector. 5.4.4 Health Introduction The Health sector has a grants portfolio of $1.072 million and disbursements through 31 December 2011 total $326,145. It is the smallest of the five CLP sectors. The purpose of the sector is to increase trust in the ROYG health services. The goal of improved maternal infant and population health also contributes to one of USAID’s worldwide goals and receives “ear mark” funding. To date a total of 22 health grants have been issued to four implementing partners in the target governorates. The CLP health sector was built from the existing base of USAID/Yemen’s legacy $25 million “Basic Health Services Project” (BHS) ending in December 2010 with the MoPHP. During that time over 5,000 MoPHP midwives were trained to meet new standards.123 BHS also started the Private Provider Midwives (PPM) program that covered four of the eight target governorates. These types of activities were the basis of the health sector awards implemented by CLP as described below. The health vision statement124 with three separate sub-sector strategies for Maternal and Child Health, Family Planning, and Nutrition was prepared in March of 2011. It was well written, appropriate for the context, and realistic regarding implementing partners. 123 The MoPH had added new procedures that midwives could practice that had been limited to MDs and the BSH program. The YMA helped the MoPH train up to these procedures. 124 CLP. 2011. Sector Visions and Strategies – Health. April. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 46 Findings Staffing CLP began staffing this sector in September and October 2010 by hiring two local health specialists and an excellent expatriate health professional who had extensive multi-decade experience in Yemen. In addition, from January to March 2011, health coordinators were employed to cover four of the target governorates. The CLP health sector staff through year end 2011 consisted of nine health professionals. When the original expatriate Health Sector Team Leader left, she was replaced by another expatriate who also has extensive international public health experience and is a native Arabic speaker. Implementing Partners There are three partners playing distinct roles in the health sector. The Ministry of Public Health and Population is supported with stipends, maintenance, and supplies to expand the Mobile Medical Teams reaching rural districts of Marib, Shabwa, Amran, and Lahj. Yemen Family Care (YFC), in the occupied schools of Aden and Lahj, provides basic health and hygiene for IDPs. These displaced people from the internal conflict areas of Lahj and Abyan have found refuge in over 30 schools in Aden alone. The National Yemen Midwives Association (YMA) trains midwives in the MoPHP system and implements their signature program of Private Provider Midwives in Marib, Al-Jawf, Amran and Shabwa governorates. Training for midwives and helping unemployed midwives to set up private practice in their communities is also an economic development input. Interviews125 were conducted with two of the Governorate Health officers and with the directors of the Yemen Midwives Association. Both CLP counterparts indicated that they had the staffing, links with regional affiliates in the case of the YMA, and the ability to expand their services geographically if the CLP would provide the grants. As of 31 December 2011, 22 grants had been obligated for a total of $1.072 million with $326,145 disbursed through December 2011. The breakdown of this total is as follows: Ministry of Health for the Mobile Medical Teams – $636,976 Yemen Midwives Association – $269,296 Yemen Family Association – $165,614 Geographic The Health Sector of CLP is covering six of the eight target governorates. The three implementing partners could scale up to cover all of the remaining target governorates and more, plus they could expand and add an urban focus so that the USAID interest in cities such as Sana’a, Taiz, Ibb etc. could be covered quickly. Accomplishments vs. Targets As with the other sectors, CLP did not develop baseline data for the health indicators and targets for inputs were not clear from the beginning. However, the appendix of the October to December 2011 Quarterly Report, although lacking in Life-of-Project targets, has some numbers to date for many indicators, but unfortunately most are incidental, such as the number of births attended, and are not focused on input and output targets of the funded activities. They do not report on the 15 Mobile Medical units. From the interviews with the YMA it was learned that the first 29 PPMs have been trained and set up in clinics, and that the second group of 30 PPMs was about to be completed. The training of only 59 PPMs over the first 18 months of the CLP is a very small program when compared with the many midwives available and distributed in the neediest areas of the country. 125 See Annex III Interviews for the complete list of interviewees. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 47 ROYG satisfaction Interviews with the Government Health officers from both Abyan and Aden set the tone also seen in the third party monitoring126; i.e. the MoPHP would like to see more mobile medical and more midwife training, plus they would like to see the PPMs trained and established in all districts. They do not question the methods or approach; they just would like more extensive coverage of the CLP health activities in order to target the extensive health needs of Yemenis. Two critiques – that the courses were “too short” or “some women selected were not the best” – do not condemn the program, but point to the desire for more training in the former and more care in selection of participants in the latter. Participant satisfaction Due to security concerns, the only opportunity for a field visit to a community served by the mobile medical team in Lahj was cancelled. However, in USAID’s third party monitoring127 there were interviews with beneficiaries that revealed a mix of opinions. One set of opinions consisted of complaints about the lack of supplies, that the private provider midwives had to dig into their own pockets to build the clinic room in her house, and that women in the village were poor and could not pay for supplies. Another set of opinions concerned the training received – that it was positive and they expressed appreciation on the follow-up visits to provide technical assistance. The third set of opinions concerned the women who received the services from the midwives and mobile medical teams. They were all positive to enthusiastic, especially for the more remotely located or isolated families. In the case of the IDPs, the people were very happy to receive the supplies and the clothes. The contrast between IDPs and the ROYG offices was interesting; in the former they were most appreciative of the services and the clothes, in the latter their interpretation of the needs of the people was that they needed food and not clothes. Conclusion All three health sector activities – health and supplies for IDPs, Support for Medical Mobile Teams, and the development of the Private Provider Midwives – were well targeted and appropriate for this type of program. Moreover, the expansion of this program is well within the budget of the CLP health sector and the managerial capacity of both CLP and the implementing partners. In comparison with the prior USAID scale of funding, this program was a letdown for the health sector especially in the field of child and maternal health in Yemen. Recommendations Considering the USAID worldwide emphasis on maternal infant health and family planning, and the dire maternal infant conditions in Yemen, this sector should be a priority for immediate expansion. It is a successful sector, but could be much more successful by scaling up. Also, in view of the MoPHP governorate health office’s five-year history with the previous USAID￾funded BHS program, the CLP health activities are beyond the pilot stage and should be expanded. The implementing partners are well prepared to scale up to cover all of the target governorates, plus they could expand and add an urban focus so that the USAID interest in cities such as Sana’a, Taiz, Ibb, etc. could be covered quickly. The Ministry of Public Health and Population has centers in most districts and the mobile medical units could increase the geographic coverage. YMA has affiliates in all governorates, a census of all midwives both in and outside the MoPHP, and they have the training experience and staff to expand. In addition, the MoPHP would like to use the BHS model of more extensive follow-up and training. That model could provide excellent data for the monitoring and evaluation function of YMA and CLP. 126 YMEP. 2011 Oct. Third Party Activity Monitoring of The Community Livelihoods Project (CLP) .Tables 8-11 127 Ibid Table 12 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 48 Health related support activities for Yemen’s IDP population are important for the reputation of the ROYG in attending to its people. After all, some of those IDPs are refugees as a result of fighting between ROYG forces and insurgents.128 Because of the dislocation of the school population to accommodate the IDPs, resolving this problem should be integrated with the Education sector.129 Both of these approaches could be part of an incentive strategy when mediating these conflicts as recommended in the section on the development hypothesis. 5.4.5 Education Introduction The largest sector in size with regard to the CLP grants approved to date, the Education Sector has a total of $5.4 million obligated in twelve grants with $1.29 million disbursed through December 2011. As a legacy sector, the dollar amounts of the sector mask the number of impacting activities within the sector. Rather, the Education Sector has a large number of small activities, with one very large grant subcontracted to CHF. The technical areas of focus are basic education, accelerated learning, and scholarships. Basic education is focused on access to and quality of community schools. Accelerated learning focuses on access to education and increasing the quality of educators within the education system. Finally, scholarships are an emerging program yet to be fully realized within CLP. Four implementing partners have been used during the 18 month period covered by the evaluation. It should be noted that the Ministry of Education was not a primary counterpart for the sector until late in 2011. Also, four of twelve grants were for school bags, a program initiated late in the evaluation period and not envisioned in the education strategy. Through these programs, the implementer delivered bags in eight governorates. Similar to the other sectors, education is ramping up activities, staffing, and plans for the remainder of the CLP award period. A new sector strategy is under development, as are modified work plans for the remainder of the CLP grant. Findings Staffing The Education Sector has had two intermittent, short-term expatriate leaders throughout the entire evaluation period. A permanent expatriate Sector Leader arrived in January 2012 during the course of the evaluation field work. Accordingly, work plans are being revised for the remainder of the CLP award period. Equally important, although staffing was historically low throughout the evaluation period, new staffing is called for that will total up to twelve local staff covering school rehabilitation, teacher training and literacy, mother/father associations, scholarships and sector management. This staff will be complemented by up to thirteen field coordinators, up from eight education coordinators in the past. The new expatriate Sector Leader plans on working with schools as “cluster” activities, emphasizing integration with other sectors in CLP, and she wants a far more active dialogue with the Ministry of Education, one which was initiated in September 20ll.130 Implementing Partners While officially there are four implementing partners, outside of school bag delivery, there are two active implementing partners: Cooperative Housing Foundation (CHF) and NAMMA. CHF is an international 128 Some of these insurgencies are from long standing tribal conflicts and the struggle of the south after unification and should not be confused with Al-Qaeda-type opposition to the central government. 129 In interviews with the Government Health Officer from Abyan and Aden, they indicated that over 30 schools in Aden were occupied by IDPs, causing the school system to go on triple shifts in the school structures that were not occupied. 130 CLP initiated a formal dialogue with MOE in fall 2011 and is now proposing now proposing schools for assistance, and five grants are under review by CLP. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 49 non-governmental organization with education enhancement activities in ten governorates. Moreover, the COP of CHF has been working in Yemen for over 30 years and brings substantial talent to CLP’s effort. NAMMA is a local organization focusing on literacy, adult education, and strengthening teacher training in eight governorates. NAMMA staff is formally from the MOE. With the exception of CHF, all activities under the education sector have required direct implementation support by CLP given the paucity of local counterpart expertise. Granting Level – As of 31 December 20ll, twelve grants had been obligated for a total of $5.4 million, with $1.29 million having been disbursed to date. Dominating this is the single grant to CHF at $4.8 million. Among the grants, the majority were for the purchase of schoolbags and their subsequent distribution (four grants). Two active grants with NAMMA are raising the skill levels of teachers through teacher training in literacy and grade level reading in Aden and Lahj. The largest grant to CHF has three active programs in enhancing educational access and quality and one completed grant in Aden for youth vocational kits. Geographic The grants, including the schoolbags, covered all eight governorates identified in the Cooperative Agreement. Expansion to urban areas occurred late in 2011 principally through school bag distribution. Targeting of grants or activities with implementing partners has not been the result of baseline data or other assessments. Most activities were identified by either CLP generalist Mobilizers, education sector Mobilizers, and in-house staff, and not through the consultative process envisioned in the CLP design. Accomplishments vs. Targets There is no baseline data for the sector. Some activities were identified by the predecessor Basic Education Support and Training (BEST), a $20.3 million cooperative agreement implemented by the Academy for Educational Development (AED). NAMMA did do needs assessments related to its activities in Aden and Lahj. As recently as the bi-weekly CLP report through January 31, 2012, final development of targets and tools to measure progress remain under development for all sectors. It should be noted that the newly arrived Education Team Leader has identified baseline and other data requirements that are a major need within the education sector going forward. Formalized targets were not used prior to the Strategic Vision, March 20ll, and targets identified in the vision statement for this sector were not applied to the grants. Rather, each grant has specific targets identified, i.e. number of teachers trained, school bags delivered, etc. The CHF project was signed in September 2011. Nonetheless, modest progress has been made in the education sector. This includes:  USG-supported primary schools or equivalent non-school-based setting education reaching 126,300 students directly  Numbers of schools repaired: 12  Teacher training: 9,095  Associations or similar "school" governance structures supported: 190  School bags distributed: 150,000  School rehabilitation: 12 It should be noted that although there are many grants within the Education Sector, CHF’s grant totals $4.8 million of the total $5.3 million for the sector, and this grant started very late in the evaluation period. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 50 Participant satisfaction While the April 2011 PMP does outline targets to be accomplished, baseline data was not developed. As recently as the bi-weekly CLP Report through 31 January 2012, final development of targets and tools to measure progress remain under development for all sectors. For school bag distribution, there was almost universal praise for the one-off initiative. The one exception was a governorate that received bags with the USAID/Yemen logo significantly larger than deemed appropriate and they were not distributed. In addition, the bags with logos were not distributed where it represented a security risk for the children and Ministry of Education.131 On a large scale, the training of trainers and follow-on training of teachers, under NAMMA, has gotten significant praise as indicated by the Acting Director General for Education in Aden. With initiation in late 2011 of an ongoing dialogue with the Ministry of Education, a greater working relationship is evolving. Conclusions The Education Sector is only now (January 2012) adequately staffed and prepared for the remainder of CLP’s award period. During the evaluation period through December 2011, although some activities were based on prior analyses and assessments, (e.g. NAMMA), most activities (counted by the number of grants and not the funding level) arose from outside sources. The failure to have an active dialogue with the Ministry of Education until the final quarter of the evaluation period meant in large part that the sector did not build upon the needs known to the ministry. With a lack of baseline data and without engaging this wealth of ministerial knowledge, most efforts were seen by the evaluators as simply ad hoc or opportunistic. The use of a large, international NGO such as the Cooperative Housing Foundation is a good move for the sector. Using larger institutions will allow for larger obligations and, at the same time, subcontract the management responsibilities for grant implementation. In the past all grants were directly implemented by Education Sector staff. With the newly arrived Expatriate Sector Leader, the sector is undergoing a systematic revamping in terms of staff, work plans, vision, anticipated activities and targets for the remainder of CLP. For all but the last portion of the evaluation period, other than CHF and NAMMA related activities, there has not been much to show in the way of tangible results for the first eighteen months of the education sector. The work with both NGOs has been well thought through, and is achieving results in their respective activities. The evaluators did not find identified linkages to other activities within the education sector. For its part, USAID/Yemen identified the lack of vision, targets, time, and geography as key missing variables within CLP’s Education Sector. Also, the evaluators agree with USAID/Yemen that the lack of capacity within the Education Sector was the critical issue facing the sector during the 18-month evaluation period. Recommendations Activities going forward with the education sector must be based on the building of current staff and projected skills sets now appearing within the sector. Going forward cannot be based on results to date. Given the “ear mark” funding for the sector, CLP is woefully behind in making a known contribution in the education sector. The newly arrived Sector Leader is imminently qualified to lead the team going forward by increasing levels of in-house staff and working more closely with the Ministry of Education. These will be positive steps going forward in the second half of CLP. All indications are that the newly revised component is 131 This issue, i.e., low profile where insurgency is a problem, should be used to establish a waiver when it comes to USAID branding. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 51 well on its way to being far more effective than it was previously. The strong leadership, accompanied by the right staffing model, bodes well for this component going forward. The use of international NGOs (such as CHF) for larger grants and direct implementation may permit the education sector to catch up in terms of its share of the projected portfolio disbursements. Continual reliance upon local organizations will not achieve the critical mass, even though contributions through organizations like NAMMA are notable. Scaling up the sector in terms of grants and activities implemented is warranted and supported as education remains a major thrust under CLP. This, the use of experienced international NGOS, may prove to be a “best practice,” however; implementation is too recent to measure, but coordination with the Ministry of Education is a must if the goal of increasing trust in the ROYG is to be achieved. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 52 6. Overall Evaluation Conclusions Specific conclusions are included in the evaluation of each sector and system so that those sections could be stand-alone discussion pieces and provide actionable feedback to the Mission and the management of CLP. This section starts off with two overall conclusions that are entwined; the changing context and CLP/Mission communications. The other conclusions are more typical of project evaluations. First, from the historical narrative, an important conclusion is that one of the most difficult operational aspects has been to cope with the changing context and needed reorientation of management, staff, and vision for selecting and guiding projects in the field. Intervening variables described in the narrative made it difficult to adjust from a focus on quick impact grants to a program that could manage more substantial development-focused activities. When an urban-area focus was added following the 2011 Arab Spring, after starting up as a predominantly rural community program, it challenged the management and initial technical and staffing resources. Given the dynamic and fluid context of Yemen, it should be expected that these kinds of changes will continue to require very agile management and staff. At the same time it will require that the Mission work closely with CLP staff so that strategic changes can be planned. Second, accompanying and exacerbating the first conclusion is the conclusion that the relationship between the Mission and CLP has been and is strained. We reach this conclusion from very straightforward statements by virtually all key members132 of the Mission and CLP interviewed during this evaluation. Both partners in this CoAg say that they are trying to be responsive and are trying to make their ideas clear, but they express frustration that they are not able to satisfy each other. Both sides can cite reports or correspondence, conversations and meetings, where they thought that they were understood only to later conclude that the signals were mixed. The issue of communications is complex and has changed since the inception of the program.133 The Mission (Director, Technical Officer and AOTR, with the backing of the Ambassador) and the CLP COP had a unified approach to grappling with the protocol issue. They also were unified in the beginning of the legacy projects as-needed before they could implement the community-based strategy. The decision to move ahead cautiously and the timing to hire the mobilizers was also a concerted meeting of the minds. The first misunderstanding and frustration on the part of USAID began when the first grant proposals of questionable quality came in (late 2010 and early 2011) because they did not seem to add up to a strategy that would have impact.134 Recall that in late 2010 and early 2011, the CLP PMP was still being negotiated to adapt to change, and the mission requests for vision statements, plus the Mission requests for a strategy on the peri-urban and urban shift in focus, were not met with a satisfactory response from CLP. This was the beginning of the communication problems prior to the change in Mission staff. The Mission was frustrated by the apparent inability of CLP to quickly revise the program at a time when CLP was just putting the field teams in place to do a rural, community-based strategy that was to have been the first phase before the legacy-type projects. The requested vision statements were beyond the capacity of the existing CLP staff and they had to be postponed until Creative could place sector leaders. With vision statements pending, and also the strategy for an urban focus pending, the Mission staff changed. The new 132 Bob Wilson current Mission Director; Chuck Swagman current Mission Technical Director and AOTR; Ed Scott current CLP COP; Robert Jacobi, Creative Associates; Craig Davis previous CLP COP, Dana Stinson previous Mission AOTR and Sean Jones, previous Mission Technical Director. 133 This conclusion is based on interviews with previous USAID/Yemen Mission staff. 134 Triangulation based on correspondence in April 2012 with previous Mission AOTR and previous Mission Technical Director. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 53 Mission Director tried to pick up on those themes, but, for example, CLP had not prepared a response to the request for an integration strategy or the urban peri-urban shift. The style of the new Mission Director was informal (in a positive sense) when discussing these pending needs, but in retrospect, he or the USAID Technical Team should have should have written a position paper and both parties should have met to discuss it. However, part of the circumstance was that the original COP was on his way out and, as can be expected under the circumstances, was no longer focused on future planning that a new COP would have to implement. Shortly after the new COP arrived he began strategic planning within CLP and he was aware of the communications issues between CLP and the Mission. One of his initiatives was to propose a facilitated workshop retreat for CLP and USAID personnel.135 The Mission signed off on a purchase order for the first week in December 2011, but, because of high level diplomatic visitors from DC it was postponed for the second week. Then the Mission canceled the workshop. Both the Mission and CLP were under tremendous pressure to plan the transition initiative and under pressure to ramp up program activities. As evaluators, in view of the continuing and dire need for improved communication and coordination, we conclude that the constant effort to write, plan and implement urgent activities seems to get in the way of planning as well as communicating. This is counterproductive when the two parties need to have a unified vision in this precarious moment. These facts reinforce the recommendation that was presented to the Mission in the out briefing i.e. to do a combined USAID and CLP strategic planning retreat. This state of miscommunication has been and continues to be an impediment to the implementation of the project and will put the Mission strategy for adjusting to the transition period in jeopardy unless resolved. The frustration of the Mission with Creative’s slow start is understandable (see third conclusion below), and Creative’s complaint of the Mission’s requests for ever changing emphasis is also understandable. The evaluators conclude that these frustrations and complaints have their basis in poor communication, that is, the style of interaction. The main point is not that directions and communications have or have not been exchanged; it is that the teams have not interacted as a whole. In this dynamic context, the CoAg partnership should be a flexible vehicle for implementation. A few times it was mentioned that a contract with set targets, activities and deliverables would have made the whole project more manageable for the Mission, but under the dynamic circumstances contracting would not be so useful because the Mission needs the flexibility to change or add emphasis from quarter to quarter. The third overarching conclusion is that Creative Associates did not meet its Home Office responsibility of immediately setting up systems (GMS, M&E) that are needed in all projects for efficient management. Moreover, evidence of corporate commitment to this CoAg seemed to be lacking until the final quarter (September to December 2011) of the evaluation period. Creative did not place adequate emphasis and support towards this Award until after four quarters of implementation. Finally, however, the project is capable in the M&E area, but it is still struggling to implement the Grants Management and Community Mobilizers systems. The fourth conclusion is that the project is salvageable if the Mission and CLP can be brought together behind a mutually shared implementation plan where the level of effort and expected outcomes are understood. Then each quarter that purpose and vision needs to be reviewed and honed to meet the ever￾changing Yemen context while holding a steady course toward the basic goal of strengthening the sectors, enhancing citizen’s confidence in its government, and achieving the Intermediate Results stated in the Award. Had it not been for the prior foundation of the former USAID “legacy” projects upon which CLP built many of its initial and ongoing grants, little would have been accomplished toward the goal of increasing the trust of the people in the government. 135 The evaluators did not hear of this plan from CLP or USAID staff in spite of discussing the need. This fact was revealed to us after the evaluation – Source correspondence with Creative Associates. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 54 The fifth conclusion is based in the purposes of the CoAg. Following extensive interviews of CLP staff, implementers, and counterparts in Aden, the evaluators conclude that the Aden Regional Office needs to have a higher profile and more decentralized decision making authority in order to accelerate implementation and to serve the region. Finally, based on interviews with 75% of the Community Mobilizers, the conclusion is that as a system established for the original purpose of community-based assessment of prioritized needs and project ideas, the Community Mobilizers have been largely abandoned and not well utilized within the CLP structure. They are a valuable CLP human resource, can operate in their respective tribal contexts, and can be reincorporated into the project as governorate managers, compliance personnel, and M&E staff. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 55 7. Overall Evaluation Recommendations Specific recommendations were included in the evaluation of each sector and system so that those sections could be stand-alone discussion pieces and provide actionable feedback to the Mission and the management of CLP. Rather than repeat them here, out of context, the reader is referred to those sections. The exit briefing for the USAID/Yemen Mission of 13 February, 2012 was dedicated to recommendations gleaned from the various sections of the report. Importantly, the main, overarching recommendation that the evaluators made came from our observations, review of documents, and extensive interviews. The theme of this overarching recommendation was “Close the door on the first eighteen months of CLP and go forward with oversight, strategic direction, integrative programming, and strengthen the collective partnership between Creative and USAID.” Per the conclusion that there were unfortunate and overburdening communication challenges in the history of the project, that first recommendation was followed by a second, articulating practicable and actionable ways ahead with communications. That is, a third-party, facilitated, off-site strategic planning workshop136 should be held prior to and in preparation for the fourth modification of the CoAg. The recommendation is that for such a workshop CLP should demonstrate its accomplishments to date – especially the substantial build up in staffing ability during the last few months of 2011 and the initial months of 2012. USAID/Yemen should share its goals for the new Mission strategic plan (2012-2014) once it is developed and finalized, its’ clear expectations for CLP going forward, and a clear recognition of the mutual importance for success in the Yemen “Transition Period.” The goal of the strategic planning retreat should be to achieve documented consensus for the next eighteen months of CLP activities and targets. It was recommended that the workshop include all management and technical personnel from CLP and the USAID/Yemen mission. In keeping with the “oversight” aspect, it is also recommended that along with quarterly reports, periodically there should be joint USAID/Yemen and CLP review/strategy sessions137 involving the USAID Mission Director, AOTR, USAID sector technical staff, and similar CLP counterparts. In the quickly changing context of Yemen, the exchange of written quarterly reports and 30 day deadlines is not always the most appropriate means of communication. Face-to-face decisions for the coming quarter’s work must be made in a timely manner and should not lose clarity via email. This should help open up lines of communication between the mission and CLP. It is recommended that “intra-netting” be the style of management and communication between the Mission and CLP. Intra-netting refers to multiple paths of communication without regard for hierarchy or gatekeepers. The purpose is to decentralize decision making, speed the flow of information, avoid stove piping, and generally avoid the slow pace of centralized decision making where communications between organizations or departments are restricted to specific leaders. Both organizations are too complex and the need for decisions based on current, local, on-the-ground information too urgent to use old-style vertical, hierarchical communications. Technical staff need to be empowered, but that only works if the strategic plan is understood and the quarterly strategy sessions are forward-thinking and results-oriented. This kind of intensive effort in the form of strategic planning and organizational development is justified when one considers that CLP is the flagship program of the Yemen Mission and that the USAID strategy for transition in Yemen depends upon it. 136 We feel obliged to repeat, that at the time of the writing of the original draft, neither the CLP nor the USAID staff told us that they had tried to do a workshop. During the out briefing the idea was met with hostility on the part of the Mission Director. We conclude that the communication problem is a very serious problem that merits a facilitator, 137 OTI has facilitated periodic reviews instead of quarterly reports. Given the dynamics of the Yemeni context, this format and style could be very productive for CLP and USAID during the transition period. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 56 It is recommended that the Aden Regional Office be given the full status of a decentralized decision making project office with an expatriate named as director.138 This is based on the conclusion that the head of the office during the first 18 months was by-passed in that communication, such as the Mobilizer￾generated projects, tended to be lost in the Sana’a office and did not have an advocate to force action. The seven specific recommendations requested in the SOW are as follows: 1) Recommendations for moving forward with regard to partnership with international and local sub￾grantees; In each sector of the report it was indeed recommended that the sector be scaled up by: a) expanding and extending existing agreements to cover more districts and train more ministry and private sector citizens and b) by attracting new but proven subcontractors via the APS mechanism. This recommendation was based on the conclusions in the sector sections that some existing grantees are capable of doing more, and on the apparent success of the APS to generate response at twice the funding level available. 2) Recommendations for field teams to enhance/ensure community engagement; The conclusion is that the community-based model has been by-passed by work through the ministries and also by the district and governorate offices with their plans and bank of projects. That being said, in the section on Community Mobilizers it was recommended that the back log of submitted but not yet approved projects be reviewed immediately so that the relationship between the Community Mobilizers and the local officials could be restored. 3) Recommendations regarding enhanced support for field operations in CLP governorates moving forward since CLP expatriate staff are currently located at the main CLP office in Sana’a; After the field trip to Aden, the evaluators concluded that an expatriate could be stationed there permanently to upgrade the stature of that office to a full project office. The observation on the Community Mobilizers is that they are capable of more responsibility and could manage and facilitate CLP activities in each governorate. The evaluators were not able to travel in other governorates to properly assess the idea of enhancing other field operation by placing expatriate personnel or mobilizing then to the other governorates. However, the security issue is probably the most crucial. In the interviews with mobilizers it was indicated that they could circulate because they were from the area and because they kept their source of employment private. It would be inappropriate to expose mobilizers in their home district to the presence of foreigners. To enhance offices in governorates it would be appropriate to provide better office services to the mobilizers, train them, and assign coordinating responsibilities to selected mobilizers. This would enhance the relationship with the governors, the SFD, and facilitate integrating sectors in the regional towns and districts. 4) Recommendations as to whether CLP expatriate staff should be located in field offices where security is permissive to enhance grant implementation performance, oversight and effectiveness; Recommendation #2 above touches on this question. The attitude that expatriate staff is needed is misplaced if CLP will empower and train Community Mobilizers (all have university degrees, some engineering). Expatriates in governorates are not needed, would be expensive and, importantly, expatriates generally do not have the cultural background to operate with a low profile. This is without taking into account the security concerns and additional security costs that would be raised by placing expatriate staffing in the governorates. 5) Recommendations to improve CLP grant disbursement rates moving. 138 The stature should be equivalent to a Deputy COP so that CLP’s southern regional needs can be given equal priority. After all, Aden, Abyan and Lahj governorates have priority urban and civil unrest. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 57 Unexpected in the design was the dearth of capacity in counterparts, resulting in CLP needing to directly implement 75% of all grants thus far, i.e. CLP had to do the procurement because the local implementers did not have the clerical ability to meet compliant USAID standards. Disbursement rates can be enhanced in two critical areas. First, a functioning grants management system, formalized and implemented, will cut down many of the delays that have existed in an informal grants management system. Second, heavier reliance on international organizations that can absorb and quickly move larger grants is required. Without this, CLP will be hard pressed to meet the targets outlined in the CoAg. 6) Recommendations to improve the ratio of grants/project disbursement to CLP overhead and support costs. In the Grants to Overhead analysis it was pointed out that the only approach to changing the ratio is to increase the grants in addition to possibly reducing unnecessary staffing numbers. As CLP starts to subcontract large grants to international or local partners, the need for large numbers of CLP personnel correspondingly decreases. It was also pointed out that the cost of grant making and supervision in this context is more expensive than planned. Projections through the end of 2012 indicate that the ratio will be improved overall, but that throughout the first 18 months of the project the ratio was very high. The conclusion is that this is an expensive grant management model. 7) Recommendations to improve CLP’s financial processes and procedures with regard to grant disbursements. The financial operations of CLP were reviewed and found to be more than adequate. The evaluation team did not evaluate ancillary functions like procurement because implementing partners did not indicate that procurement done by CLP was an issue or bottleneck. Overhauling the CLP grants system, following the evaluation, should see economies of scale and notable increases in efficiencies, both of which will impact grant disbursement rates. No other recommendations are noted at this time. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 58 8. Best Practices The incorporation of Best Practices identified during an evaluation means that a project has implemented or used strategies that exemplify world class management, technical, theoretical, or otherwise notable action that should be called to the attention of both the project as praise or to the Mission so that they might be used in papers about the program. In the case of CLP a few notable practices are listed below. Building Upon Existing Mission Program Base The best practice was to build upon the USAID legacy portfolio for a quicker start than would have been possible in establishing relations with the ROYG and enabling some service delivery in target areas had CLP only carried out the community-based strategy. Collaboration with YMEP Collaboration between YMEP M&E expertise, CLP M&E staff, and grantee M&E systems development is definitely a best practice.139 Although this did not happen in 2010, by late 2011, YMEP and CLP M&E leadership had combined expertise and measurement approaches so that when CHF began its sub-grant as an implementing partner in Education, CLP and YMEP worked with CHF to develop their M&E system so that appropriate baseline and tracking indicators were defined and a system designed for tracking and exchanging that information. This integration is important for the role of each level of management and will make for the delivery of timely, consistent, data and reports that the USAID Mission can trust. Gender Inclusion The best practices when it comes to gender inclusion, i.e., the role of women and the inclusion of women, are found in the hiring practices of CLP at all levels (internal hiring and in sector implementation). Community Mobilizer teams for each governorate were purposely male and female. Agriculture sector training was based on the sound knowledge of the role that women play in that sector with the result that they designed training for women in animal husbandry and vegetable production to build on their roles in society. The health sector midwife training was of course female focused, but the stellar practice was the private provider midwifery focus. For this CLP deserves special mention for promoting an innovative approach to improving midwifery by means of the mapping technique and providing equipment so that midwives have enhanced prestige in the community, leading to improving not only infant/maternal care, but also improving the economic status of midwives. This holistic approach to a profession is much better and more sustainable than the usual midwife training programs found in developing countries. By focusing on the sustainable clinic aspect as well as the health delivery or technical aspect, CLP integrated the health and economic development sectors. 139 During the last 30 years the observed general practice by contractors is to leave M&E decisions and designs to the COP. Most COPs do not have the expertise, time, and, in some cases, the interest to develop appropriate M&E systems much less incorporate research designs to test hypotheses. The M&E person should be a key person in all USAID contracts and the M&E database should be a deliverable simultaneous with the workplan and PMP. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 59 Annexes Annex I - Scope of Work Annex II - Work plan Annex III - List of Interviews and Contacts Annex IV - References YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 60 Annex I – Scope of Work Explanatory note: Background: The original scope of work was prepared by Yemen Monitoring and Evaluation Project (YMEP) at the request of the USAID/Y Mission with the original target date for the evaluation (January – March 2012). The Mission provided feedback on that original draft, and based on that draft an evaluation team was configured consisting of two expatriates. The evaluation team prepared a field portion work plan (Annex II of this report) after conducting the document review but before leaving for Yemen. The afternoon prior to the in-briefing and discussion of the proposed work plan the Mission sent YMEP a highly re-written version of the original scope of work and added a very specific section II Evaluation Questions and Under VII Evaluation Deliverables seven specific areas of interest for recommendations. The questions and deliverables made specific areas of inquiry that had been proposed. Reason for this Note: However, text was added in this final SOW that greatly expanded the original intent and evaluation objectives (III. Data collection and analysis methods). The emphasis of the SOW changed from a process-oriented mid-term evaluation toward an impact assessment via surveys for various levels of beneficiaries. The desired surveys were not possible to carry out in the time allotted. Accordingly, during the in-briefing the evaluation team explained that they could not carry out the described survey, but that they could carry out, if warranted, the survey proposed in the work plan based on the sample used previously during the Third Party Grants Evaluation.140 It was agreed that the survey would be conditional upon there being evidence that the area of inquiry (community mobilization) had, indeed been carried out. It was evident after a series of interviews during the first week that the community mobilization, participatory planning, and the formation of community oversight committees’ methodology were not part of the CLP implementation strategy and, therefore, the proposed area of inquiry did not warrant the survey. Based on the above determination by the evaluation team, the following was recommended141 to the Mission Director and the AOTR: It is recommended that beneficiary and other level survey(s) related to the mid-term evaluation not be undertaken as part of the mid-term evaluation. Rather, while some beneficiary information will be forthcoming, due to the information below, it is recommended that a survey of beneficiaries be undertaken separately and its outcome be incorporated in the implementation guidance forthcoming under CLP as it completes the next work plan. The Mission Director approved142 the recommendation on February 1, 2012. 140 YMEP, October 2011 Third Party Activity Monitoring of the Community Livelihoods Project, Draft Final Report, 10 DEC 2011 141 See attached Memorandum dated 30 JAN 2012. 142 See attached email traffic dated 1 FEB 2012. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 61 International Business & Technical Consultants, Inc. 8618 Westwood Center Drive Suite 220 Vienna, VA 22182 USA January 30, 2012 MEMORANDUM To: Charles Swagman, COTR for CLP Cc: Robert J. Wilson, Mission Director From: Kris Merschrod, CLP Evaluation team leader Ref: Surveys for measuring impact and results for the Mid-term CLP evaluation Recommendation It is recommended that beneficiary and other level survey(s) related to the mid-term evaluation not be undertaken as part of the mid-term evaluation. Rather, while some beneficiary information will be forthcoming, due to the information below, it is recommended that a survey of beneficiaries be undertaken separately and its outcome be incorporated in the implementation guidance forthcoming under CLP as it completes the next work plan. Background The SOW for the evaluation received on 22 January differs substantially from the draft that we had to work from to prepare the draft work plan submitted a week ago. Although we reviewed and compared the questions in the original nine areas of inquiry with the questions in the later SOW, I did not realize implications of the methodological approaches. The work plan submitted by the CLP Evaluation team on 23 JAN had one survey that was conditional upon the findings from the interviews during the first week (through 26 Jan). Now that we have reviewed the documentation and conducted interviews across the five sectors and two systems at CLP, we are in a position to consider surveys. Originally we proposed a very specific survey that was described as follows: The final level of analysis for reconstructing the model based on the participatory community approach and for assessing the grants-making system is the community. That being said, the survey approach for the community level has been informed by the Cooperative Agreement Modification #2, and the October 2011 third party activity monitoring of the Community Livelihoods Project (CLP) draft final report dated 10 DEC 2011. It is proposed that the same sample be used and that researchers provided with focus group guides would gather the information so that we can ascertain just how the CLP community strategy was applied and answer questions such as: degree to which community decision making is strengthened, to what level local oversight was productive, to what level of governance were efforts linked, and the overall enhancement of community perceptions on the support provided by the ROYG or local governing bodies.proposed to assess the community-based planning and mobilization. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 62 What we have found in the interviews to date is that community-based planning and mobilization was not used by CLP. The emphasis from the grants and the scaling up of the staff has been on working through line ministries and implementing partners. The proposed survey is not relevant because the intervention was not carried out, baseline and other assessment data does not exist, and, at best, such a survey would generate historical information not completely relevant to the future directions of CLP. SURVEY(S): The Future Although, the surveys implied or specified in the revised SOW are beyond what would be considered in a mid-term process evaluation, we can propose a separate evaluation to meet those needs. We are confident that we can answer the majority of the process oriented assessment questions from the interviews that are proposed in the work plan in the time frame of the next two weeks. However, the baseline and opinion surveys that are implied in the revised scope of work (See annex I for the parsed methodology paragraphs.) need a careful survey design, interview schedules, sampling to cover the governorates, training for teams who have access to the local areas, field testing, supervision, data entry, and data analysis. The process for such surveys would take approximately eight weeks from design to reporting. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 63 Mission Concurrence with the Recommendation of 30 JAN 2012 Wilson, Robert J (Sanaa) Feb 1 (9 days ago) to me (Kris Merschrod) Approved. SBU This email is UNCLASSIFIED. From: Kris Merschrod [mailto:kmerschrod@ibtci.com] Sent: Wednesday, February 01, 2012 7:18 AM To: Wilson, Robert J (Sanaa) Subject: Memo on removing Surveys from the CLP SOW Dear Bob,' Attached is the memo that I mentioned yesterday on removing the surveys from the CLP SOW. Chuck said that he had passed it on to you, but now I have modified the attached to CC you. I would sincerely appreciate your approval of this change, that is signed as approved. Sincerely, Kris -- Kris Merschrod Team Leader CLP Evaluation IBTCI M&E Practice Skype kris.ibtci 712 543-509 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 64 USAID/Yemen, CLP Mid-Term Evaluation Scope of Work 143 I. Evaluation Use and Purpose The purpose of this mid-term performance evaluation is to assess the implementation, effectiveness, and progress of the Community Livelihoods Project (CLP) in Yemen. The findings of this evaluation will be used to inform USAID whether the program is on track to achieving programmatic and fiscal goals and targets and whether the initial design of the project addresses the overarching USAID strategy in Yemen of positively impacting stabilization in key areas.144 Intended users of the evaluation will be the USAID/Yemen Mission, the implementing partner and YMEP. The initial CLP project approach and design was based on the 2010-2012 USAID/Yemen Country Strategy “to increase Yemen’s stability through targeted interventions in vulnerable areas.” As the strategy describes, this goal mandates a localized, integrated development programming to address drivers of instability in specific areas of the country and to respond directly to the articulated community needs and address key drivers of instability. Recent political instability and operational security challenges have resulted in an expansion of the areas of CLP’s activity interventions beyond the initial 8 governorates to encompass a more urban and peri-urban focus and a larger geographical reach than initially anticipated. The development hypothesis of the 2010-2012 USAID/Yemen strategy, which guides CLP’s Cooperative Agreement and, therefore, implementation, is that addressing the development needs of Yemen’s underserved communities will help improve social stability by responding to the key drivers of community instability and citizen dissatisfaction. A higher level of livelihood and economic satisfaction will ideally minimize behaviors that create instability, such as extremism, tribal conflict and violence. This hypothesis has not yet been proven but remains an intent of the current USAID strategy in Yemen. USAID’s assistance, including CLP, is specifically designed to provide support in areas where the Yemeni government does not have sufficient resources or expertise to execute, organize or manage such activities. To successfully implement the mission strategy, close collaboration with the Republic of Yemen was envisioned in order to improve livelihoods of citizens in disadvantaged communities and improving governance capacities to mitigate the drivers of instability. Development progress in Yemen is a major foreign policy priority for the US Government. Over the past few years and particularly as a result of the protracted Arab Spring uprisings in 2011, Yemen has suffered from a struggling economy, limited opportunities for a large youth population, a rapidly growing population, unequal development, declining government revenues, growing natural resource scarcity, tribal and regional conflict, and violent extremism. It is well recognized that many of the factors contributing to the current destabilized state of Yemen are beyond the capacity of USAID and the Implementing Partners to wholly or even partially control. The severe operational constraints imposed by the recent changes in Yemen’s context have impacted CLP’s effectiveness. Following the sudden onset of widespread uprisings in May 2011 (resulting in the temporary evacuation of Mission and Implementing Partner staff), increased insurgency, economic decline, the assassination attempt on President Saleh in June 2011 with his eventual return to Yemen and the ensuing November 2011 GCC agreement for a political transition, CLP has faced unexpected 143 This version of the SOW was received on the afternoon of 22 January, 2012 and was discussed during the 23 JAN 2012 in-briefing with the Mission Director, CLP AOTR, and Shaif al-Hamdany. 144 See 2010-2012 USAID Yemen Country Strategy. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 65 challenges throughout 2011 with implementation of activities and disbursement of allocated grant funding. Description of program to be evaluated: CLP is a US$123 million three-year program (with two option years) implemented by Creative Associates International (Creative) and designed to work closely with the Republic of Yemen Government (ROYG) to help increase employment opportunities, improve citizen access to quality public services, promote community participation and empowerment and strengthen local governance capacity. Initially focused on primarily rural communities in eight governorates: Shabwa, Marib, al-Jawf, Amran, Aden, Abyan, al-Dhale, and Lahj, CLP’s geographic focus has recently been expanded to encompass more urban and peri-urban areas (Taiz, Ibb, Sana’a, Hodeidah among others) to respond to the large populations and the growing civic unrest in these urban centers of Yemen. Creative’s program approach, to address the drivers of instability, as outlined in their Cooperative Agreement, entails conducting needs assessments at the governorate, district and community levels that lead to community-level interventions provided through grant instruments that fund activities focusing on community needs such as in education, health, water, agriculture, among other sectors. Operationally, CLP’s utilizes grants that are intended to be relatively simple, low-cost but high-impact to fill immediate gaps in community development that can be completed within a few months. CLP also undertakes longer￾term interventions that link short-term interventions with development approaches to ensure sustainability. CLP is a flagship project for the USAID Mission in Yemen’s 2010-2012 strategy. This program is also intended to be technically complementary to the Responsive Governance Project (RGP) implemented by Counterpart International. USAID/Yemen views a close working relationship as absolutely key to the success of both CLP and RGP as their goals, objectives and approaches are strategically integrated. Further, CLP is responsible for monitoring outputs and measuring project impact, with results and accomplishment verification conducted by USAID’s Yemen Monitoring and Evaluation Project. The Assistance Objectives (AOs) of the strategy are: 1) livelihoods in vulnerable communities improved; and 2) governance capacities to mitigate drivers of instability improved. The Intermediate Results (IRs) of these objectives that the CLP is responsible to achieve include the following: IR 1.1 Employment opportunities increased IR 1.2 Access to and delivery of quality services improved IR 2.2145 Local governance and capacity for basic service provision improved IR 2.3 Community-based institutions and mechanisms to ensure active participation in governance and locally driven solutions strengthened The overall development hypothesis for CLP is that targeted development interventions at the sub￾national level can positively affect stability in Yemen. Implementation of this project has been testing this assumption, which directly relates back to the overall goal of the USAID/Yemen strategy. Following is CLP’s results framework: CLP Goal: Yemen’s stability increased through targeted interventions in highly vulnerable areas (= USAID/Yemen Assistance Goal) CLP Result 1: Livelihoods in targeted vulnerable communities improved (=USAID/Yemen strategy Assistance Objective 1) 145 IR 2.1 of the USAID Yemen Results Framework is not part of the CLP results framework but is public policies and institutions facilitate more equitable socio-economic development. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 66 CLP IR 1.1: Employment Opportunities increased (=USAID/Yemen strategy IR1.1) CLP IR 1.2: Access to and delivery of quality services improved (= USAID/Yemen IR 1.2) CLP Result 2: Governance capacities to mitigate drivers of instability improved (=USAID strategy Assistance Objective 2) CLP IR 2.1: Community participation and empowerment promoted (supports USAID/Yemen IR 2.3) CLP IR 2.2: Local governance capacity strengthened (supports USAID/Yemen IR 2.2) II. Evaluation Questions This mid-term performance evaluation will measure and analyze the accomplishments and/or progress toward achieving the CLP IRs, CLP results and ultimately the program goal. Given Mission concern over the past 6 months with the low CLP grant disbursement rate, the evaluation will also assess and analyze the reasons (both internal and external) for these challenges, highlight any aspects of the program that have proved successful and document why other aspects have not been successful. The primary evaluation questions are as follows:  Has the CLP implementation mechanism of issuing grants to local awardees been an effective strategy for achieving program results?  Have the efforts to include women and youth in the program achieved expected results?  Has the program achieved all of the other expected results at midterm?  Has the program been cost effective? III. Data collection and analysis methods Through a review of monitoring data collected thus far with respect to the program, interviews with participants at all levels of the program (implementers, grantees, sub-grantees and beneficiaries), surveys and/or focus groups composed of beneficiaries of the program and formal socio-economic analysis tools (such as Social Return on Investment--SROI). Use and presentation of geospatial data on program results that has been collected thus far is essential. Of particular concern are the Cooperative Agreement targets for Year 1 (480 rapid response projects, 31 medium-long term stability projects and 12 cluster projects-total 523 projects) and for the first half of Year 2 ( 360 rapid response projects, 90 medium-long term stability projects and 38 stability projects total 488 projects). Based on the implementation for the first 18 months of the 3 year award timeline, the evaluation will determine whether the program has reached approximately 50% of its Cooperative Agreement 3 year target for grant budget disbursement of $52,875,000 and evaluators will also analyze reasons contributing to CLP challenges with grant disbursement in the first 18 months of contract award. The evaluation team will determine whether the CLP’s Cooperative Agreement describes key partners with which it works including the Yemen Government. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 67 Review of Data Evaluators will first put together an accurate historical narrative of the first period, starting from the signing of the award. This will include all challenges faced, stoppages or blockages of the work and reasons why, how problems were overcome, and what other steps were taken to correct or change the work flow. Also evaluators will summarize in the narrative all the successful achievements and what factors made them successful, review and summarize progress vis-à-vis implementation and any results or impact achieved by the project at this stage and answer the following evaluation questions—including a detailed explanation of the reasons why the project has or has not achieved expected results. The evaluation team will analyze the program design, i.e. results framework and implementation thereof, to determine how successful the program has been thus far. Taking into account quality and consistency, evaluators will determine whether program reporting has met USAID standards and whether sufficient baseline data was collected at the beginning of program implementation. Further, they will review and analyze the CLP’s geographic strategy for effectiveness including a thorough examination of the CLP geographic areas of implementation. In connection with this analysis, they will determine whether the CLP’s geographic shift from the eight initial target governorates (Shabwa, Marib, al-Jawf, Amran, Aden, Abyan, al-Dhale, and Lahj) to a more urban and peri-urban focus in an expanded number of new and continuing governorates has been an effective strategy looking at the bases for the decision to work in these different areas. The team will compare outcomes/outputs to date against the work plan and the PMP and determine whether the PMP and work plan are effectively linked and whether the data they include is detailed enough to establish causal links to the IRs and targets by number, quarter, year, sector, across sectors (i.e. integrated activities), geographical locations, and governorate/district. Key Informant Interviews Evaluators will examine the roles of CLP partners/sub-grantees including Civic Democratic Initiatives Support Foundation (CDF), Yemen Partners for Democratic Change (PDC), and Yemen Education for Employment (YEFE) to determine whether CLP’s sub-grantees have helped CLP to accomplish its mid￾point programmatic objectives. Interviews will also explore: how security considerations have impacted implementation of the project and what mitigation measures has CLP taken to minimize security constraints on program implementation; whether CLP is using an integrated, participatory and inclusive community-led approach to its interventions at the sub-national level; whether CLP conducted adequate needs assessments and viability assessments (conflict analysis) of each of the regions where work is taking place; a whether CLP programming been informed by and developed in response to these assessments; and whether CLP has been able to successfully target beneficiaries for the interventions. Since a large component of the program is to build capacity and organizational development to increase local capacity to identify and address their needs, interviews will also focus on how effective this has been, how effective the community education and mobilization and awareness raising/outreach activities/efforts have been; whether or not CLP has leveraged non-USG resources; how and from whom is a portion of each project funded in cash, commodities or in-kind contributions from the private sector, the ROYG, other donors, NGOs, communities, etc. : and whether or not CLP has been able to start looking at longer term, process-based, high impact, community-driven activities. Surveys/Focus Groups Surveys and/or focus groups will explore the questions of how youth and gender has been considered in beneficiary targeting; how involved local Yemeni individuals and organizations have been in project implementation; how the project is perceived and valued by the stakeholders (i.e. officials, beneficiaries, YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 68 civil society, and other donors): whether the local communities are fully engaged; whether they think that the project is fulfilling their needs; what their level of ownership over the programs is; whether the Project Oversight Committees discussed in the CLP proposal are being utilized and how effective they have been; and whether the program incorporates an understanding of the local context in addressing local grievances and drivers of instability elicited from the community. Socio-Economic Analyses A review of the effectiveness and efficiency of the grant mechanism/processes for providing grants, i.e. Rapid Response Projects, Stability Projects, and Cluster Community Projects, to determine whether the various activities are considered and approved from the perspective of how they will resolve or address the drivers of conflict/instability; whether they are designed with attention to the social, political, and economic forces at play within and among the local communities; whether the proposed sequencing is effective with the RRPs first, then SPs and the CCPs; and overall, what the major CLP achievements of the program to date have been. These analyses will help to determine if program activities have been sufficient to make a real impact, even at this mid-term stage, whether there any lessons learned from the first half of the program that can be applied to the remaining time and whether the program activities/components link and reinforce each other. In addition, economic analyses will take into account the management structure of CLP and whether it is appropriate to the need to develop rapid, locally-responsive stabilization, high impact grants. In comparing the costs and benefits of the program the team will analyze the CLP staff and management changes over the first 18 months of the contract award and comment on the impact or significance of such changes to CLP implementation effectiveness; compare and analyze the initial (July 2010) CLP management and team structure with the new team structure (following the arrival of the new COP in September 2011) and the major increase in expatriate staff along with the current resources necessary to support CLP’s effectiveness moving forward; identify the major bottlenecks in the program (e.g. procurement, finance, grants, operations); assess the efficiency of the CLP field teams; compare monthly and annual grant disbursement rates to date with CLP’s Cooperative Agreement annual grants budget targets; assess and analyze the reasons for CLP’s grant disbursement shortfall; analyze CLP’s proposed grant disbursement pipeline for the next 6 months and determine feasibility to achieve these targets; analyze the ratio of grants/project disbursement to CLP overhead and support costs; and analyze CLP’s financial processes and procedures with regard to grant disbursements and make recommendations, if needed, to improve expediency and efficiency. IV. The Evaluation Team 1. One senior-level evaluation methodologist with extensive designing and conducting evaluations in fragile states and the ability to conduct high level socio-economic analyses. Expertise in SROI is particularly useful in this respect. The senior level evaluation methodologist will serve as team leader and be responsible for the document review, field portion, interviews, analysis, the draft and final evaluation reports, and the debriefs in the field. 2. One senior-level evaluator with expertise in stabilization. The senior level evaluator will work closely with the team leader and in all areas of document review, field portion, interviews, analysis, the draft and final evaluation reports, and the debriefs in the field. 3. Two local YMEP research assistants/evaluators. V. Stakeholder Participation and Local Capacity The written design of the evaluation will be shared with country-level stakeholders as well as with implementing partners for comment before being finalized per the USAID Evaluation Policy to the extent YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 69 possible. USAID expects that its staff; partners; customer/beneficiaries, local evaluation experts, or other stakeholders will participate fully in the conduct of the evaluation through interviews and participation in surveys and focus groups. In addition, the evaluation will utilize local research assistants/evaluators who will either be based in, or travel to selected regions/cities where CLP activities have been implemented to conduct focus group discussions, interviews or other means of project evaluation with program interlocutors and beneficiaries. Given the security situation, the expatriate evaluation team members will be restricted to evaluation review in Sana’a and Aden. VI. Evaluation Timeline and Logistics Team Leader: Total ~34 days (based on 6 day work week) – 4 days prep, 18 days in field, 3 to 4 travel days, 5 days for first draft report writing, 3 days for revisions/final report preparation after comments received. Team Member: ~34 days (based on 6 day work week) – 4 days prep, 18 days in field, 3 to 4 travel days, 5 days for first draft report writing, 3 days for revisions/final report preparation after comments received. Final Evaluation Tasks 1. Desk Review a. Documents USAID will provide for desk review include: i. CLP RFP ii.CLP Cooperative Agreement and modifications iii. CLP PMP iv. Yemen Mission PMP v.CLP Year 1 and Year 2 work plans vi. CLP Quarterly Program reports vii. CLP All Activities Grants Report with data current as of Jan 2 2012 viii. USAID/Yemen country strategy ix. Stabilization reports, assessments x.YMEP’s Third Party Monitoring Report draft of CLP activities, Dec 2011 xi. Other relevant CLP documents (success stories, operating/grant/financial/procurement/M&E procedures and protocols xii. CLP staffing organigram from Dec 2010 and Dec 2011 (noting changes that have been made to the staff and staffing/team structure 2. Develop an appropriate methodology for the evaluation including data collection tools. 3. Prepare a field and HQ work plan, including interview plan (both current and former CLP and USAID staff responsible for CLP). 4. Field work 5. Document, in a draft evaluation report with findings, conclusions, and recommendations VII. Evaluation Deliverables The contractor shall provide the following deliverables: 1. Brief outline of methodological approach for evaluation before departure for Yemen. 2. A proposed itinerary, schedule for interviews, and list of all logistical support needs for the field visit based on desk review of documents and grants database, interview lists, and initial conversation with implementing partner staff regarding CLP. This deliverable shall be submitted to the YMEP COTR and CLP AOTR prior to departure to Yemen; however, it can be adjusted within the first few days in on-ground in Yemen during the in-brief with USAID/Yemen. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 70 3. Mid-field visit briefing to inform USAID of progress and any major issues encountered (date TBD with COTR) 4. Debrief with USAID/Yemen prior to departure 5. Draft of the evaluation report submitted to YMEP COP and IBTCI HQ seven working days after departure from Yemen 6. Final evaluation report in English, deliverable no later than two weeks after receipt of all comments from USAID on first draft. The report must include recommendations by the evaluators that capture the best practices of the project thus far but also identify any shortcomings so that they can be address and a mid-course correction can be implemented as necessary. Specifically, USAID is interested in: 8) Recommendations for moving forward with regard to partnership with international and local sub-grantees; 9) Recommendations for field teams to enhance/ensure community engagement; 10) Recommendations regarding enhanced support for field operations in CLP governorates moving forward since CLP expatriate staff are currently located at the main CLP office in Sana’a; 11) Recommendations as to whether CLP expatriate staff should be located in field offices where security is permissive to enhance grant implementation performance, oversight and effectiveness; 12) Recommendations to improve CLP grant disbursement rates moving forward; 13) Recommendations to improve the ratio of grants/project disbursement to CLP overhead and support costs; and 14) Recommendations to improve CLP’s financial processes and procedures with regard to grant disbursements. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 71 Annex II – Evaluation Work Plan Implemented by IBTCI Evaluation Team Kris Merschrod, Team Leader and Art Warman, Stabilization Specialist January 23, 2012 PROJECT BACKGROUND AND PURPOSE The initial Community Livelihood Project (CLP) approach and design, as stated in Cooperative Agreement 279-A-00-10-00032-00, was based on the 2010-2012 USAID/ Yemen Country Strategy “to increase Yemen’s stability through targeted interventions in vulnerable areas.” As the strategy describes, this goal mandates a localized, integrated development programming to address drivers of instability in specific areas of the country, to respond directly to the articulated community needs, and to address key drivers of instability. More recent political instability and operational security challenges have resulted in an expansion of the areas of CLP’s interventions beyond the initial eight governorates to encompass a more urban and peri-urban focus and a larger geographical reach than initially anticipated. The development hypothesis of the 2010-2012 USAID/Yemen strategy, which guides CLP’s Cooperative Agreement146 and, therefore, implementation, is that addressing the development needs of Yemen’s underserved communities will help improve social stability by responding to the key drivers of community instability and citizen dissatisfaction. It is argued that a higher level of livelihood and economic satisfaction will ideally minimize behaviors that create instability, such as extremism, tribal conflict and violence. It would further serve to enhance community perceptions of support to them by the Government of Yemen. USAID’s assistance, including CLP, is specifically designed to provide support in areas where the Yemeni government does not have sufficient resources or expertise to execute, organize or manage such activities. To successfully implement the mission strategy, close collaboration with the Republic of Yemen was envisioned and collaboration with local organizations was part of the design. EVALUATION PURPOSE The purpose of this mid-term performance evaluation is to assess the implementation, effectiveness, and progress of the Community Livelihoods Project (CLP) in Yemen. The findings of this evaluation will be used to inform USAID whether the program is on track to achieving programmatic and fiscal goals and targets and whether the initial design of the project addresses the overarching USAID strategy in Yemen of positively impacting stabilization in key areas.147 Intended users of the evaluation will be the USAID/Yemen Mission, Creative Associates (Implementing Partner) and the Yemen Monitoring and Evaluation Project (YMEP). EVALUATION TEAM MEMBERS IBTCI has fielded two, senior- level specialists who have managed, designed, and or evaluated programs in conflict and post-conflict areas of countries in diverse culture areas. They will apply this diversity of experience to both the analysis of the process of implementation of the CLP and also to the constructive recommendations called for in the SOW. In addition to YMEP-provided logistical support, two local 146 And also is reflected in Modification #2 of July 2011. 147 See 2010-2012 USAID Yemen Country Strategy. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 72 YMEP research assistants/evaluators will participate in the evaluation. The local research assistants/evaluators will either be based in, or travel to selected regions/cities where CLP activities have been implemented to conduct focus group discussions, interviews or other means of project evaluation with program interlocutors and beneficiaries. Given the security situation, the expatriate evaluation team members will be restricted to evaluation review in Sana’a and Aden. The Team Leader, Kris Merschrod, is a Development Sociologist with 35 years of evaluation experience in designing and conducting projects and evaluations in developing societies. Of direct relevance to the evaluation of the CLP, Mr. Merschrod has designed and evaluated post conflict programs in Guatemala, Peru, Kosovo, Colombia and Pakistan. Specifically, in the case of Pakistan (2008) Mr. Merschrod designed and managed the opinion survey of the seven FATA agencies and produced the analysis that informed the OTI and follow-on Livelihoods Project in 2008. In the latter, Livelihoods Project, his responsibilities included the PMP, M&E approach, and the community engagement strategy. In Colombia, from the initial years of Alternative Development Project through to the 2005 transition to the next generation of Colombian USAID projects, he designed the M&E approach, trained the personnel, identified and mentored the M&E director, designed the data management system, and oversaw the programming of what became a web-based M&E system. He also designed, oversaw the internal survey and evaluation of the effectiveness of the communities’ agreements to eradicate coca. He wrote quarterly reports and also the papers explaining how the community engagement methodology of participatory planning, small project design, and local oversight, via local NGOs and governments at the municipal and district level, generated: trust in government, linkages between levels of government, and community solidarity. These three dimensions of social structure are generally known as social capital. Both the Pakistan and the Colombian experiences of Creative Associates were cited in their original response to USAID’s RFP and are embodied in the cooperative agreement for the CLP to be evaluated.148 As team leader he will be responsible for the design of the evaluation work plan and, with the second team member, Art Warman, will conduct the document review, field work design, interviews, analysis, the draft and final evaluation reports, and multiple briefings of the USAID and CLP personnel in the field. The second Team Evaluator, Art Warman, is a Senior Development Consultant with 33 years of experience in stabilization, post conflict capacity development, grants management, marketing, enterprise development, assessments, policy and regulatory reform, change management, and project design and implementation. His earliest work in post-conflict development was with the Government of Nicaragua following the change of government in 1978 where he was responsible for all private sector growth programs for USAID. More recently, he was responsible for standing up the reconstruction agency for Banda Aceh, Indonesia, following the 2004 earth quake. This included full design of the agency, training of staff members and outreach to affected communities. Of direct relevance to the evaluation of the CLP, Mr. Warman has recently completed eight assignments in Afghanistan, including design and implementation of cash-for-work programs at the local level, design of a change management engagement for the Ministry of Agriculture, facilitating an inter-governmental conference for USG agencies operating under Regional Command East, third-party monitoring of two large capacity development programs in Eastern and Southern Afghanistan, and serving as Team Leader for a USAID Mission capacity development evaluation covering twenty-two programs. 148 In reference to “USAID Evaluation Policy.” Jan. 2011, p.5 on stating prior experience and potential conflict of interest, Creative Associates cited their Pakistan and Colombian experience in their response to the RFA. Merschrod has been a consultant to Creative Associates, and he did do the design and analysis of the Pakistani survey for Creative Associates under OTI. In the Colombian case his work was for Chemonics and then ARD from 2002 to 2005; Creative’s work on post-conflict demobilization for OTI came later (2007). Merschrod does not have current or pending agreements or understandings with Creative Associates or any of the CLP partners. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 73 METHODOLOGY The SOW work is comprehensive, and an initial examination of the Cooperative Agreement and modification #2 reveal that the development hypotheses, assumptions and strategies for implementing the five components of the CLP are also comprehensive, complex and changing. The initial design of the CLP was based on a grass-roots, community-focused, local organization implementation strategy or model to resist and dilute the drivers of instability identified on previous studies. However, in the early reporting there was a significant expansion or change in approach: A switch from the original grass-roots up approach initially designed for this program to closer interaction with the central ministries has been a bit of a challenge. However, CLP is up to the challenge. In cooperation, the program will find solutions that please the central ministries, as well as strive for community-driven projects. 149 This change in implementation strategy to work closely with and through the ministries of the Republic of Yemen Government (ROYG) had implications for the grant-making process; the rate of grant making is a specific area of inquiry for this process evaluation. From the preliminary review of the documentation, it is clear that two project implementation models have been posited. The first approach to the evaluation will be to reconstruct these models by tracing the actual implementation strategies and activities by component, and then compare those actions with the agreed upon original, proposed strategy and development hypotheses. The project is structured into five sectors or components: 1) Government and Civil Society, 2) Health, 3) Education, 4) Economic Livelihoods, and 3) Agriculture. They are aligned with the four Intermediate Results (IRs): 1) Employment opportunities increased; 2) Access to quality services increased; 3) Community participation and empowerment promoted; and 4) Local governance capacity strengthened. In turn, the IRs conform to the results frame work under the two Mission Assistance Objectives: 1) livelihoods in vulnerable communities improved; and 2) governance capacities to mitigate drivers of instability improved. In addition to the five sectors, there are two key, project systems that have been incorporated into the evaluation scope of work: 1) the Monitoring & Evaluation and 2) Grants Management. Based on the evidence gathered by the methods described below, the evaluation team will be able to arrive at conclusions, and make recommendations to the Mission as per the SOW. In terms of the two models, the evaluators will assess them and the potential synergies that may have evolved. Areas of Inquiry A. Historical Narrative Evaluators will first put together an accurate historical narrative of the first period, starting from the signing of the award. This will include all challenges faced, stoppages or blockages of the work and reasons why, how problems were overcome, and what other steps were taken to correct or change the work flow. Also evaluators will summarize in the narrative all the successful achievements and what factors made them successful, review and summarize progress vis-à-vis implementation and any results or impact achieved by the project at this stage and answer the following evaluation questions—including a detailed explanation of the reasons why the project has or has not achieved expected results. The 149 Community Livelihoods Project Quarterly Report October – December 2010 p.13 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 74 evaluation team will analyze the program design, i.e. results framework and implementation thereof, to determine how successful the program has been thus far. B. Key Questions This mid-term performance evaluation will measure and analyze the accomplishments and/or progress toward achieving the CLP IRs, CLP results and ultimately the program goal. Given Mission concern over the past 6 months with the low CLP grant disbursement rate, the evaluation will also assess and analyze the reasons (both internal and external) for these challenges, highlight any aspects of the program that have proved successful and document why other aspects have not been successful. The primary evaluation questions are as follows:  Has the CLP implementation mechanism of issuing grants to local awardees been an effective strategy for achieving program results?  Have the efforts to include women and youth in the program achieved expected results?  Has the program achieved all of the other expected results at midterm?  Has the program been cost effective?150 C. Recommendations Sought The report must include recommendations by the evaluators that capture the best practices of the project thus far but also identify any shortcomings so that they can be address and a mid-course correction can be implemented as necessary. Specifically, USAID is interested in:151  Recommendations for moving forward with regard to partnership with international and local sub-grantees;  Recommendations for field teams to enhance/ensure community engagement;  Recommendations regarding enhanced support for field operations in CLP governorates moving forward since CLP expatriate staff are currently located at the main CLP office in Sana’a;  Recommendations as to whether CLP expatriate staff should be located in field offices where security is permissive to enhance grant implementation performance, oversight and effectiveness;  Recommendations to improve CLP grant disbursement rates moving forward;  Recommendations to improve the ratio of grants/project disbursement to CLP overhead and support costs; and  Recommendations to improve CLP’s financial processes and procedures with regard to grant disbursements. Document Review At the time of this writing, the following key documents have been identified for the evaluators. Many of them have been studied as part of draft work plan development. CLP RFP CLP Cooperative Agreement and modifications (especially Mod#3) CLP PMPs (Original and the 2011 Revised) Yemen Mission PMP CLP Year 1 and Year 2 work plans CLP Quarterly Program reports 150 The definition of “cost effective” was discussed. The definition was left to the evaluators with the caveat that expectations of the detail were conditioned by the level of detail 151 SOW 24 January, 2012, p.9 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 75 CLP “All Activities Grants Reports” with data current as of Jan 2 2012 USAID/Yemen country strategy Stabilization reports, assessments YMEP’s Third Party Monitoring Report draft of CLP activities, Dec 2011 All TMEP reports Other relevant CLP documents (success stories, operating/grant/financial/procurement/M&E procedures and protocols) CLP staffing organigram from Dec 2010 and Dec 2011 (noting changes that have been made to the staff and staffing/team structure) Visions Statements for the components Component Approach The methodological approach behind “reconstructing the model” is to do a component by component analysis in order to compare how and what the CLP team planned to do with how the CLP team did implement, and then describe how integrated the components were at all levels. In the case of the CLP, the grant-making and tracking, and the M&E systems are common to all components. The implementation of the five components depends on the grants system for its execution and the management of those components depends on the M&E system. The evaluators will divide responsibility for the assessment of the components152 between them. In addition to serving as Team Leader, Mr. Merschrod will assume: 1) Agriculture and 2) Health components. He will also the review of the M&E system and interview the mobilizers. Mr. Warman will assume: 1) Education, 2) Economic Livelihoods, and 3) Governance and Civil Society components and, he will review the Grants Management System (GMS). Interviews After the initial in-briefing and agreement on a final evaluation scope with USAID/Y and the YMEP staff, interviews will be requisite personnel aligned with the five components, as elaborated upon in Annex 4. These will include, inter alia, implementing partners, CLP staff, USAID officials, ROYG counterparts, etc., as agreed upon with USAID/Yemen. As needed, field work will be undertaken, including the use of a survey instrument, to ascertain local level participation, achievement of the objectives outlined in the Project. The final level of analysis for reconstructing the model based on the participatory community approach and for assessing the grants￾making system is the community. That being said, the survey approach for the community level has been informed by the Cooperative Agreement Modification #2, and the October 2011 third party activity monitoring of the Community Livelihoods Project (CLP) draft final report dated 10 DEC 2011. It is proposed that the same sample be used and that researchers provided with focus group guides would gather the information so that we can ascertain just how the CLP community strategy was applied and answer questions such as: degree to which community decision making is strengthened, to what level local oversight was productive, to what level of governance were efforts linked, and the overall enhancement of community perceptions on the support provided by the ROYG or local governing bodies. 152 The background for these five components is informed by the five Sector and Vision Strategy papers prepared in the second quarter of FY 2011. YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 76 Assessment of the project M&E system In the quarterly reports it was noted that the PMP had not been approved until second quarter of the first year and, until that time, data collection on indicators had not begun. The evaluation will review the process of gathering the data, baseline surveys, the system for in-putting the information, how the data can be accessed and used for generating special and periodic (quarterly) reports, and how the M&E system is used by management. Information from this system will be used to assess progress toward goals. Assessment of the Grants Management System In the SOW153 and in the Third Party Review of the grants system a comprehensive review of the CLP grant design and approval processes is called for. Each step in the granting process will be examined: community and NGO decision and submission, CLP review process, USAID vetting process, disbursement and review of advances, advance amount in relation in to CLP turnaround time. The ample grants data base will be used to analyze the time process, disbursement and utilization rate and the adequacy of the data base for grants management and decision making. The data and the system will also be assessed from the perspective of having and maintaining baseline data for future impact assessments. Geographic coverage The evaluation team is limited to Sana’a and Aden, but they anticipate interviewing CLP personnel and field staff from the totality of the areas where activities and grants have been implemented when those interviewees are available in Sana’a and Aden. In the case of the to-be-determined survey, the interviews will be conducted by the local researchers. WORK PLAN/GANTT CHART AND DELIVERABLES Annex 2 provides a fully developed work plan, time line and set of deliverables for the evaluation scope of work. Along with the areas of inquiry this work plan will be subject to full review and approval by USAID/Yemen upon initiation of field work. Upon final approval, the areas of inquiry, work plan and interview structure will be refined, codified and incorporated into the work. Annex l Evaluation Scope of Work Annex 2 Work Plan/Gantt Chart Annex 3 Interview Structure 153 P.8 “Financial” YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 77 Annex III – List of Interviews USAID Date Name Position Organization Telephone Email 1/18/2012 Julie Fossler Former AID/W Yemen Desk Officer AID/W 1/18/2012 Lillian Wasvary AID/W Yemen Desk Officer AID/W 1/23/2012 Charles F. Swagman AOTR CLP USAID/Y 733-442-233 SwagmanCF@state.gov 1/23/2012 Neal P. Cohen Development Economist PSC to the Mission USAID/Y 1-352-271-4792 1- 352-328-1818 nealpcohen@yahoo.com 1/23/2012 Robert J. Wilson Mission Director USAID/Y Wilsonrj@state.gov 1/23/2012 Shaif, Al￾Hamdany Technical officer for CLP USAID/Y 967-1-755-2197 alhamdanysh@state.gov 1/30/2012 Abdulhamid Alajami Sr. Education Advisor USAID/Yemen 967-1-755-2087 alajamian@state.gov 1/30/2012 Alfra Alzouba USAID/Yemen Governance Advisor USAID/Yemen 967-1-755-2087 Aflzouba@state.gov 1/30/2012 Jeff Noel Vanness Democracy and Governance Field Advisor DCHA/DG 202-216-3231 jvanness@usaid.gov 1/30/2012 Mayad Yafi Economic Growth Specialist USAID/Yemen 967-1-755-2229 yafims@state.gov 1/30/2012 Nagwa Saleh Economic Growth Specialist USAID/Yemen 967-1-755-2243 salehns@state.gov 28/1/2012 Charles F. Swagman COTR CLP USAID/Y 733-442-233 SwagmanCF@state.gov 28/1/2012 Mayad S. Yafai Econ Growth Specialist USAID/Y 967-1-755-2229 yafaims@state.gov YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 78 28/1/2012 Nagwa S. Saleh Econ Growth Specialist USAID/Y 967-1-755-2243 salehns@state.gov 29/1/2012 Ashraf Ali Zabara Health and Pop specialist (new) USAID/Y 967-1-755-2187 zabaraaa@state.gov 29/1/2012 Nawal Al. Baabbad Health and Pop specialist USAID/Y 967-1-755-2187 Baabaadna@state.gov YMEP Date Name Position Organization Telephone Email Various JAN-FEB Jane Thomson COP YMEP 711-003-346 jthomson@ibtci.com Various JAN-FEB Rich Mason Senior M&E Specialist YMEP/Yemen 737-882-940 rmason@ibtci.com Various JAN-FEB Mohammed Ibrahim M&E Officer YMEP 777-291-970 MIbrahim@ibtci.com PRIOR USAID AND CLP Date Name Position Organization Telephone Email Various JAN-APR Craig Davis COP June 2010- Aug 2011 CLP Various JAN-FEB Caryle Cammisa COTR for YMEP and Program Office Director 2009 - March 2011 USAID/Yemen Caryle.Cammisa@socom.mil Various MAR-APR Dana Stinson AOTR (Nov 2010- August 2011) USAID/Yemen DStinson@usaid.gov Sean Jones Technical Director (2009-May 2011) USAID/Yemen sjones@usaid.gov Various Attempts to contact but no response Jeff Ashley Mission Director (2009 - March 2011) USAID/Yemen AshleyJW@state.gov CLP Date Name Position Organization Telephone Email 1/1/2012 Prasanna Vernenkar Technical Manager Creative, DC 202-572-1331 prasannav@creativedc.com 1/1/2012 Robert Jacobi Senior Associate, Creative, DC 202-772-2108 robertj@creativedc.com 1/17/2012 Prasanna Vernenkar Technical Manager Creative, DC 202-572-1331 prasannav@creativedc.com YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 79 1/17/2012 Robert Jacobi Senior Associate Creative, DC 202-772-2108 robertj@creativedc.com 1/17/2012 Tom Wheelock VP, Communities in Transition Creative, DC 202-966-5804 thomasw@creativedc.com 1/23/2012 and 1/25/2012 Ed Scott COP CLP/Yemen 967-1-413-947 M 967-736-010-075 Edwards@clp-yemen.com 1/25/2012 May Nasr Team Leader, Economic Growth CLP 735-872-956 mayn@clp-yemen.com 1/25/2012 Sami Sofan Entrepreneurship Manager CLP 736-779-971 samis@clp-yemen.com 1/25/2012 Sandra (Sam) Loli Health Sector Team Leader (through Jan 2012) CLP/Yemen 1/26/2012 Carmen Garriga Senior Monitoring & Evaluation Specialist CLP/Yemen 967- 736-308-869 carmeng@clp-yemen.com 1/28/2012 and 1/29/2012 Mohammed Ilyas Agriculture Team Leader CLP/Yemen 967-735 200 195 ilyasm@clp-yemen.com 1/28/2012 Prasanna Vernenkar Technical Manager, (interviewed in Sana’a Creative, DC 202-572-1331 prasannav@creativedc.com 1/29/2012 Doa’a Al Tahami Education Coordinator CLP 736-333-285 Dosst@clp-ymen.com 1/29/2012 Geraldine Schafer Deputy Chief of Party CLP 737208300 Geraldines@clp￾yemen.com 1/29/2012 Rida Baldas Interim Team Leader, Education CLP 735471152 ridab@creativedc.com 1/30/2012 Murad Barak Team Leader, Governance CLP 736779974 muradb@cly-yemen.com 1/31/2012 Brian Block Senior Program Manager CLP 734-886-071 Brianb@clp-yemen.com 1/31/2012 Issac Mwiti Grants STTA CLP 734418457 issacn@cyp-yemen.com 1/31/2012 Tom Nasongo Interim Team Leader, Grants CLP 733469162 tomn@cyp-yemen.com YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 80 2/5/2012 Abdulfattah Al Kishi Economic Growth Advisor CLP New hire 2/6/2012 Amani Fattah Governance Consultant CLP 711-099-015 amanif@clp-yemen.org 2/6/2012 Asja al Fadhli Education Mobilizer CLP-Aden 2/6/2012 Mohammed Al￾Sanbani Governance Consultant CLP 733-554-812 malsanbani@clp￾yemen.org 2/6/2012 Mohammed Ismeli Interim Office Director CLP-Aden 2/9/2012 Adam Pearce Director of Finance CLP 733-781-622 adamp@clp-yemen.org 2/11/2012 Amal Saleh Acting Community Mobilizer Supervisor CLP, Sana'a amals@clp-yemen.com 2/11/2012 Essam Assad School Reconstruction CLP 736-467-121 essama@clp-yemen.com 2/11/2012 Joy du Plessis Team Leader, Education CLP 736333172 joyd@clp-yemen.com 2/11/2012 Kawther Mhoshin Juzailan Mobilizer Al-Jawf CLP, Sana'a 736799984 2/11/2012 Mohammed Al￾Sawka Economic Dev Assistant CLP New hire 2/11/2012 Mohammed Salam Lahgard Mobilizer Shabwa CLP, Sana'a 777321514 Mohammedsalam131@yah oo.com 2/11/2012 Najeeb Ahmed Alshehari(Fema le) Mobilizer Amran CLP, Sana'a 2/11/2012 Nora Attem Baebe Mobilizer Marib CLP, Sana'a 771742622 2/11/2012 Raiden Al￾Maqrami Admin. Assist Mobilizers CLP, Sana'a 1/29/2012 Mohsen Al￾Hubaishi Deputy Agriculture Specialist CLP/Yemen mohsenh@clp-yemen.com 1/31/2012 Abdul Salam Al Kohlani Compliance Director, was first Manager of Mobilizers CLP, Sana’a 967-736-799-921 abdulsalaa@clp￾yemen.com YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 81 1/31/2012 Abdul Salam Zubayri Senior Livelihood Manager, Now in charge of Mobilizers CLP, Sana’a 736779961 / 777033086 abdulsalamz@clp￾yemen.com Nagwa Samir Winget Health Sector, team Leader CLP/Yemen nagwaw@clp-yemen.org CLP - ADEN Date Name Position Organization Telephone Email 2/6/2012 Amal Mohammed Ahmed CM- Lahj CLP 737501268 2/6/2012 Ameen Ismael Naji Regional Engineer￾Aden CLP 733333682 2/6/2012 Ashjan Mohammed Education Coordinator CLP 736800472 2/6/2012 Fatom Abdo Ali CM- Aden CLP 736800471 2/6/2012 Fawaz Hassan Ahmed CM- Lahj CLP 737501268 2/6/2012 Ghaleb Ali Thabet Agriculture Coordinator CLP 734347907 2/6/2012 Mohammed Ismail Coordinator of the Mobilizers, Aden CLP office for the south CLP 2/6/2012 Radwan Haidra Civil Engineer￾Abyan CLP 737501264 2/6/2012 Vet. Ismail Animal Health Coordinator, Aden CLP 2/6/2012 Yaser Shaher Yafouz CM- Aden CLP 736800327 2/7/2012 Sami Mohammed (male) CM- Abyan CLP 737501289 2/7/2012 Wjood Ali Abubaker (female) CM- Abyan CLP 737501384 2/8/2012 Dr. Asma Ali Dureab (female) Health Coordinator￾Abyan sits in Governorate Health Office (GHO) building CLP 777583748 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 82 2/8/2012 Dr. Nawshad Ahmed Agriculture Coordinator- Abyan CLP 737700310 2/8/2012 Dr. Osama Badeeb Health Coordinator￾Aden sits in GHO building CLP 777241991 2/8/2012 Ing. (Woman) Rabea Abdulwali Mohsen Moghalis (Woman) Civil Engineer for CLP in Lahj CLP 2/8/2012 Ing. Ameen Ismael Naji Al￾Dubaai Civil Engineer for CLP in Aden CLP 2/8/2012 Ing. Redhwan Mohammed Nasser Haidarah Civil Engineer for CLP in Abyan CLP 2/8/2012 Rabea Abdulwali Moghlis Civil Engineer- Lahj CLP 737700338 ROYG Date Name Position Organization Telephone Email 1/292012 Mansour M. Al-Agil Director General Ex. & Training MAI 967-1- 561245 2/7/2012 Adel Al-Khader Abdullah Rep. On Mualla Dist. Council Mualla Dist. Council, Aden 2/7/2012 Dr. ??? D.G. Agriculture, Aden MAI 2/7/2012 Dr. Al-Khader D.G. Health for Aden GHO Aden 2/7/2012 Dr. Mushtaq M. A. Sattar Veterinarian - Aden MAI 738802303 2/7/2012 Habib Hasan Abbas Tech. Director Irrigation- Aden MAI 2/7/2012 Eng. Wafa Al Sayed Abubaker (female) Engineer- Aden, Also trainer at YASAD MAI 733836908 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 83 2/7/2012 Mayta Osstiss Acting Director General, Education Min Educ, 2/7/2012 Nabeela Abdulla (female) Engineer- Aden, Also trainer at YASAD MAI 733836808 2/8/2012 Dr. Ali Mansour Dep. Manager Gov. Health Office GHO Abyan IMPLEMENTING PARTNERS Date Name Position Organization Telephone Email 1/29/2012 Stephanie Baric Director of Yemen Programs Governance and Civil Society, former CLP Senior Program Manager STTA (Sept-Dec 2010) Counterpart International, VA 703-237-4334 sbaric@counterpar t.org 1/30/2012 Sabri al Hakim Training Director NAMMA 777-732-228 2/5/2012 Abdulfattahn Al-Kibsi Managing Director Qube 777-425-486 abdukibsi@qubece ntre.org 2/5/2012 Ameen Al￾Qadri Training Manager YEFE 737-789-199 Aalqadri@efefoun dation.org 2/5/2012 Yasser Al￾Saidi CEO YEFE 726-8350777 yalsaidi@efefound ation.org 2/7/2012 Adel Aziz Shamsan Consultant YASAD 770146654 2/7/2012 Amin Al Hakimi Head of YASAD YASAD 711658330 2/7/2012 Dr. Ahmed Mohammed Ahmed Sallam Member of the Association YASAD 777844320 2/7/2012 Dr. Mohammed Abdulla Kusseim Consultant YASAD 733235569 2/7/2012 Eng. Ahmed Abdulmalek Head of Auditing Committee YASAD 777701708 2/7/2012 Eng. Saeed Al Shargabi Engineer YASAD 777933191 2/7/2012 Mohammed Muthna Consultant YASAD 711744331 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 84 2/7/2012 Sameera Abdulla Member YASAD 700470217 2/11/2012 Abdul Gabbar Al-Kirshi Manager Al-Thuraya, Sana’a 733272637 alkirshi@yahoo.co m www.athuraya￾acts.com 2/11/2012 Roberta Contin Program Director CHF 735540251 rcontin@chfyemen .org 2/12/2012 Dr. Suad Quasem President of the National Yemeni Midwives Association Yemeni Mid-wife Association, Sana'a 45430 2/12/2012 Fatoom Noor Aden Founder and Board Member Yemeni Mid-wife Association, Sana'a 2/12/2012 Samir Taher Coodinator of CLP grant in Sabwah Yemeni Mid-wife Association YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 85 Annex IV – References 1. CLP Activity Report with GPS grids, Sept 28 2011 2. CLP Activity Status sent 2012-01-22 3. CLP All Activities Report 2, Dec 19 2011 4. CLP All Activities Report with GPS, Jan 14 2012 5. CLP All Activities Report, Dec 5 2011 6. CLP All Activities Report, Dec 19 2011 7. CLP All Activities Report, Jan 14 2012 8. CLP All Activities Report, January 1 2012 9. CLP All Activities Report, Oct 16 2011 10. CLP All Activity Report October 16, 2011 11. CLP Detailed Activities Report, July 26 2011 12. CLP Detailed Activities Report, May 17 2011 13. CLP Detailed Activities Report, Sept 4 2011 14. CLP Detailed Activities Report, Sept 19 2011 15. CLP Grant Process Forms Required, Nov 2011 16. FORM 3.9 Environ Doc Form 17. CLP M&E Training Objectives & Agenda, Dec 17-19 2011 18. CLP Org Chart-Program (Sana’a) May 2011 19. CLP Annual Report 2010-2011_8 30 11 20. CLP Quarterly Report, April - June 2011 21. CLP Quarterly Report, Jan - March 2011 22. CLP Quarterly Report, July - Sept 2011 23. CLP Quarterly Report, July - Sept 2010 24. CLP Quarterly Report, Oct - Dec 2010 25. CLP Sector Strategy Papers, April 2011 26. CLP Agriculture and Water Vision Paper_4 10 11_FINAL 27. CLP Economic Development Vision Paper_4 10 11_FINAL 28. CLP Education Vision Paper_4 10 11_FINAL 29. CLP Governance and Civil Society Vision Paper_4 10 11_FINAL 30. CLP Health Vision Paper_4 10 11_FINAL 31. CLP National Assessment (Abyan Final Report) 12-September-2011 32. CLP National Assessment (Aden Final Report) 12-September-2011 33. CLP National Assessment (Al-Dahle Final Report) 12-September-2011 34. CLP National Assessment (Al-Jawf Final Report) 12-September-2011 35. CLP National Assessment (Amran Final Report) 12-September-2011 36. CLP National Assessment (Lahj Final Report) 12-September-2011 37. CLP National Assessment (Marib Final Report) 12-September-2011 38. CLP National Assessment (Shabwa Final Report) 12-September-2011 39. 1_Yemen_Success_Story_Vocational_Training_Public_Version_12_Jan_2011 40. 1_Yemen_Success_Story2_Vocational_ma’rib history_13_august_2011 41. 1_Yemen_Success_Story3_Vocational_yemen’s farmer_13_august_2011 42. CLP Agriculture FACTSHEET M&E comments 43. CLP Economic Development FACTSHEET M&E comments 44. CLP Education FACTSHEET M&E comments YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 86 45. CLP Governance FACTSHEET M&E comments 46. CLP Health FACTSHEET M&E comments 47. USAID/Yemen Community Livelihoods Project-Program Fact Sheet-DRAFT2 48. CLP Weekly Report – 1 January 2012 – FINAL (Dec 24-28, 2011) 49. CLP Weekly Report – 4 Dec 2011 (Nov 26-30 2011) 50. CLP Weekly Report – 4 December 2011 – FINAL (Nov 26-30, 2011) 51. CLP Weekly Report – 11 December 2011 – (Dec 3-7, 2011) 52. CLP Weekly Report – 18 December 2011 – (Dec 10-14, 2011) 53. CLP Weekly Report – 25 December 2011 – FINAL (Dec 17-21, 2011) 54. CLP Weekly Report – 27 November 2011 – FINAL (Nov 27, 2011) 55. CLP Weekly Report, Aug 1 2011 (weekly update 31 July 2011) 56. CLP Weekly Report, Aug 9 2011 (CLP weekly update August 7, 2011) 57. CLP Weekly Report, July 4 2011 (CLP weekly update 3 July 2011) 58. CLP Weekly Report, June 5 2011 (CLP weekly update June 6 2011) 59. CLP Weekly Report, June 19 2011 (CLP weekly update 19 June 2011) 60. CLP Weekly Report, June 27 2011 (CLP weekly update 26 June 2011) 61. CLP Weekly Report, March 9, 2011 (CLP weekly update March 11, 2011) 62. CLP Weekly Update 2 Oct 2011 (CLP weekly update 2 of October 2011) 63. CLP Weekly Update 9 Oct 2011 (CLP weekly update 9 of October 2011) 64. CLP Weekly Update 14 Aug 2011 (CLP weekly update 14 August 2011) 65. CLP Weekly Update 16 October 2011 (CLP Mobile Medical Team) 66. CLP Weekly Update 18 September 2011 (CLP weekly update 18 of September 2011) 67. CLP Weekly Update 20 November 2011 (CLP weekly update November 2011) 68. CLP Weekly Update 28 August 2011 (CLP weekly update 27 August 2011) 69. CLP-Year_1_Workplan_DRAFT_by component 70. Annex I CLP Year 2 Annual Workplan 71. CLP Narrative Workplan Yr 2 (July 2011 - June 2012) 72. CLP Quarterly Workplan Aug – Oct 2011 73. CLP Y1 Workplan by Sector, Nov 2010 FINAL 74. CLP Y2 Workplan Annex – timeline, Oct 2011 75. Mission Director Comments on CLP Workplan, Oct 15 2011 76. CLP M&E Guidelines 2011-11-12 77. 1Yemen Background – other USAID 78. USAID_PolicyFramework_FINAL_PL 79. USAIDEvaluationPolicy 80. Risk Assessment 6-279-11-001-s 81. USAIDYemen2010Strategy 82. Yemen’s Economy – Challenges, Risks, and Reform 10.01.12 83. Cohen Adam accounting breakdown desires 84. Copy of Yemen – SF425 85. Disbursement as at 31 Dec 2012_Adams 86. Survey result – tentative chickens 87. Survey result – tentative gardens 88. APs CLP Yemen 89. Attachment 2 – CHF Partner Quarterly 90. BudgetCLP Grant YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 87 91. CHF CLP Sub Award SOW 92. CHF Map 93. CHF_CLP_Invoice2 94. Attachment 3 – PDC Partner Quarterly 95. PDC SOW for CLP FINAL 96. PDC 97. 1 CLP All Activities Report, January 1 2012* 98. 1 Data Bases on Activities 99. CDF Jacobi’s explanation 100. CLP All Activity Report October 16, 2011* 101. CLP Detailed Activities Report, May 17, 2011* 102. CLP Grant Process Forms Required, Nov 2011* 103. CLP Org Chart – Program (Sana’a) May 2011* 104. EmOC_TPM_RGPYEM_EmOC_Sanaa_draft_Jan23_2012 105. MMT_TPM_CLPYem_MMT_Lahj_draft_Jan29_2012_Final 106. CLP Stability Assessments 107. CHUCK SWAGMAN FEEDBACK 108. CLP National Assessment (Abyan Final Report) 12-September-2011* 109. CLP National Assessment (Aden Final Report) 12-September-2011* 110. CLP National Assessment (Al-Dahle Final Report) 12-September-2011* 111. CLP National Assessment (Al-Jawf Final Report) 12-September-2011* 112. CLP National Assessment (Amran Final Report) 12-September-2011* 113. CLP National Assessment (Lahj Final Report) 12-September-2011* 114. CLP National Assessment (Marib Final Report) 12-September-2011* 115. CLP National Assessment (Shabwa Final Report) 12-September-2011* 116. Districts per Governorate sampled 117. ECON List of Implementation Partners 118. USAID/Yemen Quarterly Stabilization Review, Draft Final, August 24, 2011 (3) 119. CLP Staffing 120. CLP Local Staff List History 121. CLP Org Chart – Program (Sana’a) May 2011 122. CLP_Chart – OPS – ADEN (1) 123. CLP_Chart – OPS – ADEN 124. CLP_Chart – OPS – SANA 125. CLP_Chart – PROG – SANA 126. CLP work plan year 2 127. CLP Community Integration Plan 128. CLP Detailed Activities Report, May 17, 2011* 129. CLP Weekly Report – 4 December 2011* 130. DRAFT (3) PP Portfolio Review Oct 4 131. Thula Overview 132. Annex I CLP Annual Workplan Timeline Yr 2 133. CLP Community Integration Plan 134. CLP Narrative Workplan Yr2 km 135. CLP Narrative Workplan Yr2 136. CLP_Year_One_Workplan_18_Nov_2010_by component_Final YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 88 137. ORIGINAL CLP PMP Notes 138. CoAg and Mods 139. ANNEX II orig PMP 140. CA 279-A-00-10-00032-00.Art.1.13.2012 (1) 141. CA 279-A-00-10-00032-00 142. CLP – Final RFA – 19 Jan 10 km 143. CLP – Final RFA – 19 Jan 10 144. Mod #2 (1) 145. Mod #2 146. Program Description from CA 147. PROGRAM DESCRIPTION mod 148. 1 CLP All Activities Report, Jan 14 2012 149. 1 CLP Grants Program Ops Breakdown 150. 1 Disbursement as at 31 Dec 2012_Adam* 151. CLP All Activities Report, January 1 2012* 152. CLP grants ops break down notes 153. CLP Local Staff List History (1) 154. Report by Region no duplicates 2012-01-14_1 155. Report by Region with GPS 2012-01-14_1 156. CLP All Activities Report 2, Dec 19 2011 graphing 157. CLP All Activities Report 2, Dec 19 2011 158. CLP All Activities Report, Dec 5 2011 159. CLP All Activities Report, Oct 16 2011 160. CLP M&E Guidelines 2011-11-12 161. CLP PERFORMANCE DATA TABLE 2012-01-31 162. Clp1 (Baseline Update Requirements) 163. RGP1 (RGP Baseline Update Requirements) 164. Maps 165. CLP – Grants Disbursements Dec 20 2011 166. CLP – Grants Approval Dec 20 2011 167. Governorates word map 168. Governorates 169. OTI – Grants Approved Dec 20 2011 170. OTI – Grants Disbursed Dec 20 2011 171. Yemen_admin_2002 172. DRIVERS OF CONFLICT 173. 120121 Daily Report (UN) 174. DPS 1.19.12 175. DPS 1.20.12 176. CLP COP and USAID comments on OTI Assessment, Aug 3, 2011 177. Yemen Stabilization Initiative, OTI Rapid Lessons Learned Assessment, July 31 2011 178. PMP 179. CLP Final PMP, April 11 2011 180. CLP_Year_One_Workplan_18_Nov_2010_by component_Final (1) 181. 1 CLP Quarterly Report – Jul-Sep 2010 YMEP: Mid-Term Evaluation of the Community Livelihoods Project (CLP) 89 182. 1 CLP Quarterly Report, Oct-Dec 2010 183. 2 CLP Quarterly Report, Jan-March 2011 km 184. 3 CLP Quarterly Report, April-June 2011 185. 4 CLP Quarterly Report, July-Sept 2011 186. 5 CLP Quarterly Report, October-December 2011 187. CLP All Activities Report, January 1 2012 188. CLP All Activities Report October 16, 2011 189. CLP M&E Bi-weekly Report 2012-01-24 190. CLP PERFORMANCE DATA TABLE 2012-01-31 191. Final YMEP Table 3 192. Letter Final YMEP Jan 15 193. October 2011 Monthly Third Party Monitoring Draft Report, Dec 18 2011 194. FATA SURVEY FINDINGS_4_May_08 195. CLP Interviews.1.30.2012 196. RIG Risk Assessment of USAID/Yemen, March 2011 FINAL REPORT 197. SOW 198. 1 USAID approved CLP mid-term evaluation SOW 22 JAN 199. THIRD PARTY 200. Final CLP Response to TPM report, Jan 15 2012 201. October 2011 Monthly Third Party Monitoring Cohen comments 202. USAID/Yemen Quarterly Stabilization Review, Draft Final, August 24, 2011 (2) 203. 1 USAID/Yemen PMP, FINAL, Feb 2011 v1 204. USAID Policy on Development in Response to Violent Extremism and Insurgency, Sept 2011 205. USAID-CLP-Report-EN-May 31, 2011-Ver.5 206. USAID Geographical Prioritization 207. CLP Geographic Focus, May 5 2011 FINAL 208. CLP Geographic Focus, May 14 2011 FINAL 209. CLP Geographic Focus 2011 210. CLP Strategy for Geographic Prioritization – Nov 2010 211. Criteria selection slides, Nov 2010 212. Geographic Prioritization - USAID/Yemen 213. OTI District Selection Criteria, Yemen 214. CLP Agriculture and Water Vision Paper_4 10 11_Final* 215. CLP Economic Development Vision Paper_4 10 11_Final* 216. CLP Education Vision Paper_4 10 11_Final* 217. CLP Governance and Civil Society Vision Paper_4 10 11_Final* 218. CLP Health Vision Paper_4 10 11_Final* 219. clp-youth-vision-dd3pm1_geuad-032_090524_729505 220. Youth focus and Gender Assessment Annexes – Final 221. Youth focus and Gender Assessment – Final Sept 19 2011 222. Youth focus on Rpt of SEP 2011 223. Summary on Econ’s Recommended Partners