PARCERIA CÍVICA PARA BOA GOVERNAÇÃO (PCBG) ACTIVITY Midterm Performance Evaluation Report FEBRUARY 2019 This publication was produced for review by the United States Agency for International Development. It was prepared by Kristie Evenson and Edson Correia for Management Systems International (MSI), a Tetra Tech Company. (THIS PAGE INTENTIONALLY LEFT BLANK) PARCERIA CÍVICA PARA BOA GOVERNAÇÃO (PCBG) ACTIVITY Midterm Performance Evaluation Report Contracted under Contract Number AID-656-C-17-00002 Mozambique Monitoring and Evaluation Mechanism and Services (MMEMS) DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. (THIS PAGE INTENTIONALLY LEFT BLANK) USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | i CONTENTS ACRONYMS......................................................................................................................................................II EXECUTIVE SUMMARY.................................................................................................................................1 INTRODUCTION AND BACKGROUND...............................................................................................7 EVALUATION PURPOSE AND EVALUATION QUESTIONS ............................................................7 EVALUATION METHODS AND LIMITATIONS.....................................................................................8 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS.............................................................10 PCBG OVERVIEW.................................................................................................................................................10 EVALUATION QUESTION 1.............................................................................................................................10 EVALUATION QUESTION 2.............................................................................................................................17 EVALUATION QUESTION 3.............................................................................................................................21 EVALUATION QUESTION 4.............................................................................................................................25 ANNEX A: STATEMENT OF WORK ......................................................................................................27 ANNEX B: SUBGRANT SURVEY FINDINGS.........................................................................................33 ANNEX C: INFORMANT GUIDES...........................................................................................................39 ANNEX D: LIST OF PERSONS INTERVIEWED....................................................................................46 ANNEX E: LIST OF DOCUMENTS REVIEWED....................................................................................48 ANNEX F: FINDINGS, CONCLUSIONS AND RECOMMENDATIONS MATRIX......................49 ANNEX G: PCBG ADVOCACY SUBGRANTS .....................................................................................57 ii | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV ACRONYMS APS Annual Program Statement CESC Centro de Aprendizagem e Capacitação da Sociedade Civil COP Chief of Party CPI Counterpart International CSO Civil Society Organization DG Democracy and Governance ET Evaluation Team FCR Findings, Conclusions and Recommendations FY Fiscal Year IP Implementing Partner KII Key Informant Interview LWA Leader with Associate (Award) M&E Monitoring and Evaluation MEL Monitoring, Evaluation and Learning MMEMS Mozambique Monitoring and Evaluation Mechanism and Services NNL No Net Loss OD Organizational Development ODI Overseas Development Institute PCBG Parceria Cívica para Boa Governação RFA Request for Applications RFP Request for Proposals SOW Statement of Work TA Technical Assistance USG U.S. Government USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 1 EXECUTIVE SUMMARY EVALUATION PURPOSE The Parceria Cívica para Boa Governação (PCBG) midterm performance evaluation is intended to inform USAID/Mozambique on PCBG’s progress to date in achieving envisioned advocacy and organizational development activity results and provide recommendations for reaching optimal results until the end of the activity. The evaluation was conducted under the Mozambique Monitoring and Evaluation Mechanism and Services (MMEMS) contract, implemented by Management Systems International. ACTIVITY DESCRIPTION PCBG was designed to strengthen and widen the range of civil society organizations’ (CSOs’) policy advocacy capacities and effectiveness in Mozambique. USAID awarded Counterpart International (CPI) a cooperative agreement in the amount of $10.5 million to implement the four-year activity. Additional funding of $1.325 million was issued July 31, 2017, to support peaceful and credible elections. The cooperative agreement runs from March 25, 2016, to March 24, 2020. CPI is partnering with 11 Mozambican CSOs to advocate in the target sectors of biodiversity conservation, climate change, education, extractive industry, health, transparent and accountable governance, and peaceful elections. EVALUATION METHODS AND LIMITATIONS The evaluation team (ET) used a mixed-methods iterative process to address the evaluation questions. This included a review of PCBG documents and secondary activity data, a subgrantee survey with all 11 advocacy subgrantees, two additional unsuccessful applicants and 24 key informant interviews (KIIs) with four groups of stakeholders, including subgrantees, PCBG and CPI, USAID and external CSO and donor experts. The ET was aware of the limitations in assessing still-early advocacy and organizational development (OD) results, the potential of (subgrantee) respondent bias and potential subgrantee fatigue with multiple PCBG-related consultations. The team mitigated these issues as much as possible. OVERALL ASSESSMENT PCBG’s design reflects a vision for assisting CSOs to do better advocacy in Mozambique. However, the complex design process, funding constraints, and selection of an implementing partner (IP) without a country track record provided a weak foundation for the activity. The IP had a slow start and has had a rocky implementation experience, including the major distraction of adding an election component slightly more than one year into the activity. Advocacy efforts are in process. PCBG subgrantees are engaging in advocacy efforts in issues as diverse as extractive industry oversight to education budgeting and are achieving some success, although description of most results are still not captured in PCBG’s inadequate monitoring, evaluation and learning (MEL) system. The potential level of results is unclear, given that many subgrantee projects are ambitious for the CSOs’ capacities and likely will bear fruit beyond PBGC’s duration. CPI advocacy 2 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV technical assistance (TA) for subgrantees has been spotty; until now, scant -evidence has shown that PCBG has substantially increased subgrantees’ learning about how to conduct advocacy. PCBG OD support to subgrantees has a similar trajectory. It started quite late and is still very much in process without clear monitoring of progress to date. Here, though, there are signs that subgrantees are utilizing OD support in a manner, which could result in greater organizational capacity by the activity’s end. The first priority area for course correction during the remainder of PCBG is a greater focus on capturing and tracking advocacy and OD progress. Secondly, PCBG needs to enhance its advocacy￾focused technical assistance efforts to subgrantees to support the subgrant efforts and to impart advocacy strategic learning. It also needs to ensure quality OD technical expertise offerings for subgrantees, particularly in high demand areas, such as financial sustainability planning and gender inclusion strategies. Lessons learned for future advocacy and OD activity design and implementation considerations include ensuring that the design is ‘fit for purpose,’ ensuring correct sequencing of activity inputs and technical assistance, matching advocacy results expectations with capacities and environment, and endeavoring to make local partner voices an integral part of the process. CONCLUSIONS AND RECOMMENDATIONS EVALUATION QUESTION 1: To what extent has PCBG achieved anticipated advocacy results with the organizations to date? Why and why not? CONCLUSIONS PCBG has not engaged in a systematic MEL process that clearly shows the actual advocacy results achieved to date. The MEL system is partial and lacks the relevant industry standards for tracking advocacy policy results. Therefore, the ET cannot report on progress against expectations. High-level results in conservation and extractive industries can be linked to the activity even as (at least for the conservation advocacy achievements) it seems the activity played only a partial role, given the CSO’s earlier and ongoing efforts. Anecdotally, other advocacy results appear to be in the process of achievement and subgrantees appear committed to their advocacy work. However, most results are not being captured or recognized by the current monitoring system. Modest advocacy results to date are partially due to subgrantee implementation challenges. Most need more time due to ambitious subgrants and the realities of doing such advocacy work on both local and national levels in Mozambique. It is also not clear whether research component use has helped or hindered results to date. The low level of PCBG technical advocacy assistance also has likely limited subgrantee results to date. PCBG provided insufficient guidance to address these challenges. It also did not sufficiently frontload support efforts, such as advocacy mapping, monitoring and evaluation (M&E) plan finalization or advocacy training and support that could have assisted subgrantees. USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 3 Nor has PCBG provided much outreach and facilitation support. Probably due to CPI’s inexperience and low visibility in the country, PCBG has not provided much value added in building alliances with other policy actors, beyond what the CSOs already do individually. Finally, PCBG’s general management of the process has affected results to date. PCBG’s slow start, compounded by the long grant proposal and review process left CSOs frustrated and with subgrants that, in some cases, they needed to adapt to make workable. CSOs were further frustrated by grant management reporting, which required a steep learning curve for some, and took significant time away from subgrant efforts. RECOMMENDATIONS (Re)focus effort on capturing what is happening in advocacy subgrants. 1. PCBG should engage with subgrantees to comprehensively map out actual activities, results and timelines to obtain a clear understanding of subgrantee efforts to date 2. PCBG should use the suggested mapping to bring together existing data on subgrantee activities and advocacy results in a unified and comprehensive advocacy results tracking system.. 3. PCBG should consider adding process-level advocacy indicators to MEL plans for more recent subgrants. Bolster advocacy support and expertise. 4. PCBG should prioritize advocacy training and advisory assistance to subgrantees to introduce and reinforce this advocacy approach, including how to use/design evidence and research effectively. Consider an advocacy workshop to start things off, focusing on such issues as: − Challenges of working locally and nationally at the same time; − How objectives, strategy and MEL relate and can be synergistic; and − Demand side of advocacy. 5. PCBG should consider increasing its advocacy TA capacities by having its incoming advocacy specialist collaborate with an external MEL advocacy expert Continue streamlining management processes. 6. PCBG should further streamline subgrantee oversight practices, ensuring adequate staffing and systems in place to support subgrantees. Identify select ways that PCBG can assist subgrantees’ outreach efforts. 7. PCBG should review and prioritize where it can assist subgrantees in alliance and networking efforts. 4 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV EVALUATION QUESTION 2: To what extent has PCBG achieved anticipated organizational capacity advances of the organizations to date? Why and why not? CONCLUSIONS The level of PCBG OD progress is unclear, given the lack of systematic data to date and the early stage of implementation efforts. Therefore, the ET cannot comment on actual advances in OD among CSOs assisted by PCBG. However, the data suggest that subgrantees are making use of the OD opportunities afforded, which is a promising sign. Nevertheless, more TA targeted to specialized areas, such as financial sustainability, M&E assistance and gender inclusion, would benefit the CSOs greatly. PCBG implementation of the OD component has been a constraining factor in OD progress. PCBG appears to have captured the interest of subgrantees at the beginning of the OD process, but implementation since this start has been slow and somewhat bumpy due to internal CPI management and budgeting for the OD component. Identifying appropriate advisory consultants for specific organizations has also raised challenges. This is likely to continue, especially for the more specialized areas of financial sustainability and gender inclusion. RECOMMENDATIONS Ensure a practical focus on capturing what is happening in OD action plans and support. 1. PCBG should endeavor to harmonize an internal subgrantee OD action plan tracking system that includes the six areas of OD focus covered in the plans. 2. PCBG should consider unpacking activity-level Indicator 12 into the six OD pillar areas of focus for a clearer activity-level tracking option. Bolster OD expertise and support. 3. PCBG should prioritize identifying a strong pool of experts available to work with PCBG grantees. Financial sustainability and gender specialists are especially high priorities. 4. PCBG should consider the extent to which it can shift funding allocations to allow for increased options so that PCBG can offer OD directly and not through the subgrantee budgets. . EVALUATION QUESTION 3: To what extent did the design and structure (as amended during implementation) of the activity help and hinder achievement of results to date? CONCLUSIONS Important aspects of the PCBG design and structure hindered the PCBG activity, which was designed to work intensively on advocacy with a broad set of CSOs. Yet the funding levels secured and the IP mechanism used limited USAID’s ability to procure an IP with the functional and country-specific experience that could realize this vision. USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 5 The multi-sector structure of the activity presented significant budgetary, technical and management challenges that burdened the IP, may have distorted local CSO ownership of the advocacy efforts and provided little clear benefit for USAID technical office program efforts. The introduction of the election component further distorted implementation efforts and primarily appears to have had a negative effect in the past year. The IP was not able to adapt and take on the extra programming without compromising its already challenging core advocacy and OD implementation efforts. RECOMMENDATIONS For USAID’s considerations in further design: 1. Critically align funding mechanism, funding levels and focus. 2. If multi-office funding, ensure shared activity objectives and understanding of technical offices’ engagement in process. 3. Increase local consultation in design to ensure shared priorities. For current PCBG implementation: 4. PCBG should build internal “firewalls” for sufficient management of both elections and ongoing program work. EVALUATION QUESTION 4: What lessons can be learned from the PCBG experience to date for crafting future USAID civil society programming that promotes self-reliance and government transparency/accountability? The evaluation exercise revealed a number of larger lessons for consideration when designing future civil society programming. Fit for purpose: 1. Combining OD and advocacy purposes requires an IP that is familiar with country context and able and ready to carry out intensive work. 2. Advocacy projects need to be designed and managed to capture advocacy influence processes. Realistic design assumptions: 3. CSO advocacy capacity must be aligned with advocacy influence expectations. 4. Invest the necessary inputs and strategy to conduct both national and local-level advocacy. 5. Advocacy programming requires mapping out advocacy demand at the design stage. Sequencing matters: 6. Apply phasing to first provide OD, then advocacy technical assistance inputs. 7. Advocacy strategy and MEL need to be included in the design process. Maximizing partner relevancy and ownership: 6 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV 8. OD benefits across all CSO experience levels are possible if balanced guidance and flexibility are part of designing individually tailored OD plans. 9. Supporting partner-driven interventions while fulfilling Mission priorities requires a nuanced balance of issue priorities and a degree of flexibility. USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 7 INTRODUCTION AND BACKGROUND The PCBG activity was designed to strengthen and widen a range of civil society organizations’ (CSOs’) policy advocacy capacities and effectiveness in Mozambique. Focused on six issue-area sectors (education, health, extractive industries, biodiversity, climate change and governance) aligned with Mission programming, the activity anticipated working with a diverse set of CSOs across the country to promote transparent, accountable government practices in these sectors. USAID awarded Counterpart International (CPI) a cooperative agreement in the amount of $10.5 million to implement the four-year PCBG activity. Additional funding of $1.325 million was issued July 31, 2017, to support peaceful and credible elections. The cooperative agreement runs from March 25, 2016, to March 24, 2020. PCBG identified and awarded subgrants to 11 CSOs from across the six sectors to advocate in the target sectors with two inter-related objectives: 1) more effective advocacy by partner CSOs in target sectors and 2) strengthened organizational performance by partner CSOs. FIGURE 1: CSOs BY SECTOR The activity theory of change proposes that by improving the advocacy and organizational capacities of motivated, committed Mozambican partners, these partner CSOs will implement more effective advocacy strategies, resulting in specific changes in government policies, procedures, services and enforcement. Those reforms are expected to contribute to longer-term strengthening of accountable, effective democratic governance in Mozambique in target sectors. EVALUATION PURPOSE AND EVALUATION QUESTIONS This PCBG midterm performance evaluation is intended to inform the Mission on PCBG’s progress to date in achieving envisioned advocacy and capacity development activity results (Objectives 1 and 2)1 and to identify factors affecting program performance that might need review and course correction during the remainder of the PCBG implementation period. As part of this process, the evaluation provides USAID with lessons learned and insights on developing similar activities in the future. The evaluation was conducted under the Mozambique Monitoring and Evaluation Mechanism and Services 1 PCBG appended a third objective to the activity with the addition of the election component in July 2017. This evaluation does not focus on the election objective, but covers the elections in relation to how they affected overall PCBG programming. 8 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV (MMEMS) contract, implemented by Management Systems International. The evaluation focuses on the following questions: EQ1: To what extent has PCBG achieved anticipated advocacy results with the organizations to date? Why and why not? EQ2: To what extent has PCBG achieved anticipated organizational capacity advances of the organizations to date? Why and why not? EQ3: To what extent did the design and structure (as amended during implementation) of the activity help and hinder achievement of results to date? EQ4: What lessons can be learned from the PCBG experience to date for crafting future USAID civil society programming that promotes self-reliance and government transparency/accountability? EVALUATION METHODS AND LIMITATIONS The evaluation team (ET) used a mixed methods iterative process to answer each question. This included a review of PCBG documents and secondary subgrantee data, a subgrantee survey and key informant interviews (KIIs). The evaluation was undertaken in three phases. First, the ET conducted a desk review to gain a descriptive understanding of activity development and key phases and issues within the PCBG. Based on this, the ET drafted a semi-closed-ended question survey for the 11 subgrantees,2 which focused on getting their perceptions on the PCBG implementation process, from the call for proposals to PCBG provision of advocacy and OD technical assistance. The survey findings informed the refinement of the KIIs, particularly subgrantee KIIs. For the third phase, the ET undertook KIIs with four sets of stakeholders, which included 1) subgrantees, 3 2) PCBG and CPI staff, 3) USAID staff and 4) external CSOs and donors. In total, the ET held 24 KIIs encompassing 30 people between November 29 and December 10, 2018. For a list of all KIIs, please see Annex D. Throughout the fieldwork, the ET took an iterative approach to interpreting and synthesizing data. This included daily team meetings to share impressions and key points from interviews. It also included development of a findings, conclusions and recommendations (FCR) matrix to guide the refinement of analysis and to focus on practical recommendations. Please see Annex F for the FCR matrix. LIMITATIONS Timing of evaluation: Gaining a clear understanding of advocacy and OD results to date was partially limited by the fact that subgrants for these areas are mostly still midway completed (and in several cases 2 The survey included two unsuccessful CSO applicants for their impressions of the proposal application and development process. 3 Subgrantees furthest along in their subgrants were selected for KIIs. However, one replacement (Kuwuka) was made for the CSO AENA, which was unavailable to meet with the ET. USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 9 only recently started). To address this, the ET interviewed CSOs with 12 or more months of PCBG implementation. Even then, the ET found that implementation for many was slower than planned. Respondent bias: All subgrantees still have active projects with PCBG. This has likely affected their willingness to be critical of PCBG’s performance, at least in formal settings. The ET endeavored to mitigate this by gaining the confidence of subgrantees through discussing their views with them in more detail. The ET noted, for example, that subgrantees gave relatively good numerical ratings to PCBG in the survey, but when asked to explain these ratings during the survey, subgrantee comments were often more critical. A similar dynamic was noted during KIIs, where sometimes the CSOs felt comfortable to provide more critical feedback only at the end of the interview. Subgrantee consultation fatigue: PCBG, along with CPI colleagues, engaged in a series of “listening meetings” with subgrantees during the past several months to better understand their needs. Along with the evaluation survey and KII visit, these required subgrantees to be available multiple times during what many described as a busy end-of-the-year period. The ET worked to be available as subgrantees’ schedules permitted, but in two cases, meetings that were tentatively scheduled were later canceled. 10 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV FINDINGS, CONCLUSIONS AND RECOMMENDATIONS PCBG OVERVIEW PCBG first issued an Annual Program Statement (APS) for proposals at the end of 2016 and early 2017 across the six issue areas of education, health, extractive industries, biodiversity, climate change and general governance. A second APS followed in April 2017 with a request for proposals (RFP) focusing only on climate change and biodiversity in December 2017. During the first half of 2017, PCBG held two proposal boot camps for potential subgrantees and worked directly with applicants on developing and editing proposals. Each subgrant contains two advocacy and one organizational development (OD) objective. Please see Annex G for details on each subgrant and objectives. Subgrants last from 18 to 24 months, with awards ranging from $127,000 to $242,000. EVALUATION QUESTION 1 To what extent has PCBG achieved anticipated advocacy results with the organizations to date? Why and why not? FINDINGS PCBG MEL FOR ADVOCACY RESULTS ET review of PCBG documents found that subgrant MEL matrices do not fully align with the PCBG performance indicator table advocacy indicators, despite what both subgrantees and PCBG describe as an intense and long MEL development process.4 PCBG describes using the monitoring information from subgrantees that “fits” into the table. Until now, it did not have documentation that systematically tracks all subgrantee MEL plan indicators. The most recent quarterly report, fiscal year (FY) 2018 quarter 4, is the first report that provides a narrative description for the matrix numbers. PCBG recently began developing and populating a subgrantee advocacy tracking matrix, but this was still incomplete at the time of evaluation review. The PCBG performance indicator table documents that PCBG is meeting or surpassing seven of 10 advocacy indicator targets. However, a closer review indicated that many of these indicators tell little about subgrant advocacy progress, and PCBG recognized this during interviews. The indicators primarily cover subgrantee outputs, are descriptive indicators tracking U.S. Government (USG) support, have some duplication and highlight some sectors (health and climate change) more than others. For example, the matrix includes descriptive indicators like: “Number of CSOs receiving USG assistance engaged in advocacy interventions.” 5 This might apply to all subgrantees, which primarily denotes the number of subgrants rather than specific information or progress of the projects. An slightly more illuminating output indicator is: “Number of activities that contribute to government engagement with citizens.” Yet 4 Please see PCBG performance indicator table and subgrantee M&E plans (BIOFUND, CESC, KUWUKA, TV Surdo, AMDEC and MEPT). 5 Please see Indicator 5 on the PCBG performance indicator table. USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 11 this still provides only the information that subgrantees are working to engage with the government, not potential short-term or interim results or outcomes of this engagement process.6 FIGURE 2: PCBG PERFORMANCE INDICATOR TABLE # PCBG Advocacy Indicator Type of Indicator 1 Proportion of CSOs reporting that they impacted government public policies as a result of USG assistance in target sectors Self-reporting indicator 2 # of public policies introduced, adopted, repealed, changed or implemented consistent with citizen and/or civil society input High-level outcome indicator 3 # of activities that contribute to government engagement with citizens Output indicator 4 # of mechanisms for external oversight of public resources use supported by USG assistance Descriptive indicator tracking USG support 5 # of CSOs receiving USG assistance engaged in advocacy interventions Descriptive indicator tracking USG support 6 # of evidence-based position statements on police presented for public dialogue Output indicator 7 # of organizations conducting HIV and AIDS advocacy with USG/PEPFAR* support Health sector indicator 8 # of advocacy materials and documents produced with USG/PEPFAR* support Health sector indicator 9 # people trained in climate change adaptation supported by USG assistance Climate change indicator 10 # of organizations engaged with networks Descriptive indicator tracking USG support *PEPFAR is the U.S. President’s Emergency Program for AIDS Relief PCBG’s only outcome indicator focuses on capturing policy content change: “Number of public policies introduced, adopted, repealed, changed or implemented consistent with citizen and/or civil society input.” MEL industry standards for capturing policy influence generally consider such indicators as high-level outcomes and part of a longer advocacy process than the short subgrant timeframe allows to track. 7 ADVOCACY RESULTS Three policy content changes per the outcome indicator have been registered to date for two subgrants in biodiversity and the extractive industries, which were among the first to start their subgrants. These included a subgrantee’s efforts to pass the new conservation law at the end of 2017 and to delineate content of some of its regulations during 2018, as well as one subgrantee’s efforts to revise legislation to include local communities affected by the extractive industries, also during 2018.8 6 Please see Indicator 3 on the PCBG performance indicator table. 7 For an overview of a leading industry approach on monitoring and evaluating advocacy influence, please see the Overseas Development Institute (ODI) publications, including 2014’s “Monitoring and Evaluation of Policy Influence and Advocacy.” Page 38 references a framework for capturing advocacy outcomes from incremental to more substantial outcomes. https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/8928.pdf [Accessed December 17, 2018] 8 Per PCBG reporting, this encompasses AENA’s extractive industry work on revision of Circular 01-MPD-2013, which includes creation of a mechanism for community participation in use of public resources and a community fund to manage 12 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV Beyond these, subgrantee interviews provided insights into advocacy results progress that PCBG has not tracked. Five of the six subgrantees interviewed revealed that the indicators do not capture, and PCBG does not track, much of what they are accomplishing that could be classified as an interim or short-term advocacy process outcome or result. WHAT INTERIM ADVOCACY OUTCOMES ARE NOT BEING CAPTURED? PCBG’s reporting showcased CESC’s (Centro de Aprendizagem e Capacitação da Sociedade Civil) citizen manifesto-related work and its use in local elections, but neither PCBG’s reporting nor CESC’s MEL captured what CESC described as interim outcome effects of this effort. This included increases in voter registration, voter participation and percentage of women’s participation. Three of the six subgrantees described how their subgrant has, until now, led to specific mindset changes among targeted key government actors. For example, one subgrantee said that through their work, they “capacitated key parliamentarians to understand the issues around school council management responsibilities.” Three of the six subgrantees also described gaining recognition as a policy actor on an issue or reinforcing their role through PCBG support. One specifically described “getting a seat at the table” after a series of meetings among ministry officials engaged in HIV policy efforts. Overall, five of the six subgrantees described having already reached part of at least one of their subgrants’ two advocacy objectives.9 They said this was possible because they have been able to adapt their activities from what the approved proposals outlined. “CSOs are becoming more proactive. They are connecting with decision-makers at different levels. They are sharing documents.” — PCBG staff member assessing the subgrantee advocacy progress SUBGRANTEE IMPLEMENTATION CHALLENGES Despite subgrantees’ advocacy progress results, four of the six anticipated needing more time than the current subgrant duration to achieve their second objective. They offered two main reasons: • Including both national and local-level components has required some staggering of activities and engagement. One subgrantee, for example, described how they focused mostly on high-level policy engagement to get buy-in for inclusion of the school council president in school management and oversight. When they reach an agreement, they plan to work more closely with the local communities to educate them on how to make this work in practice. This example, like most subgrants the ET reviewed before meeting with the six subgrantees, highlighted the often complementary but not necessarily coherent nature of the local and national levels of subgrantees’ objectives. resources channeled to the communities. The second effort is Biofund’s work on the Conservation Law Regulation (Decree No. 87/2017), which the government approved in 2017, and then its efforts to include the “no net loss” (NNL) principles into the law regulations. See PCBG FY17 and FY18 annual reports. 9 Each project has two advocacy objectives and one organizational development objective. USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 13 • Subgrantees pointed to the reality of accommodating legislative calendars, and that work with government entities takes more time. In the case of Biofund’s policy results with the conservation law, they explained that PCBG funding came at the right time for them to fund the specific segment of legislative focus as part of their larger effort on affecting the content and implementation of the conservation law. However, as they also described in the “lessons learned” section of their most recent PCBG quarterly report, further work on putting in place the financial mechanisms for the “no net loss” (NNL) provisions of the law would take more time than anticipated.10 Also unclear is the level to which the research component envisioned in each project has been or will be completed. Survey data showed that subgrantees gave a “poor” rating11 to PCBG in assisting them to both “use research to inform advocacy” and “produce policy products.” During interviews, subgrantees provided limited specific detail about how they used research, and in several cases they acknowledged that the research was not yet complete, even though they had proceeded with advocacy on the issue. In at least two cases, this was due to the poor quality of the research product (done by consultants outside the subgrantee organizations), but it was also due to how PCBG assisted in this process. For example, in two cases the ET heard about, the subgrantees hired external experts to conduct the research; in both cases, PCBG considered the product poor, asking the subgrantee not to use it until PCBG found someone else. However, in both cases, the subgrantees waited a substantial amount of time (over a year, in one case) for PCBG to identify an expert, leaving the CSOs to continue their advocacy without benefit of the anticipated evidence. PCBG TECHNICAL ASSISTANCE SUPPORT Figure 3 provides an overview of subgrantee feedback on advocacy assistance. Its scale of 1 to 4 denotes: 1) strongly disagree, 2) disagree, 3) agree and 4) strongly agree. 10 Please see the Biofund PCBG July-September 2018 Quarterly Report (p.9). 11 “How to use research” rating was 2 and “how to produce policy products” was 2.17 out of 4 on a scale from “very poor” to “very good.” 14 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV FIGURE 3: PCBG SUBGRANTEE RATINGS FOR ADVOCACY ASSISTANCE LEARNING ADVOCACY TACTICS RATHER THAN STRATEGY … PCBG described assisting subgrantees to prepare questions for a commission meeting, but not how to use that meeting as part of their strategy to achieve a specific advocacy objective. Survey and subgrantee interviews point to the fact that most subgrantees have received only basic advocacy and M&E assistance, and they need more. PCBG reports note examples where PCBG has provided tactical guidance on drafting talking points or reviewing policy products. 12 Yet subgrantee feedback during interviews gives the impression that they have been primarily on their own in considering advocacy strategies. One subgrantee noted, “We feel like a guinea pig with PCBG. PCBG established themselves while having partners that they needed to support.” Others shared similar feelings, suggesting that they often felt that they were the ones leading PCBG, rather than PCBG guiding them. Overall, subgrantees identified a lack of PCBG staff with either advocacy or sector expertise as a key constraint.13 PCBG acknowledged its difficulty identifying and retaining an advocacy expert who speaks Portuguese. Since it began, PCBG had at least two temporary advocacy experts and a permanent staffer (without Portuguese language skills), until the December 2018 hire of a new PCBG advocacy specialist. Sequencing of TA also appears to have been problematic. Two areas particularly stand out: MEL plans and sector advocacy mapping. Program documents and subgrantee interviews highlight the slow process of completing MEL plans for the individual subgrants, taking six months on average to complete after the start of subgrants. The MEL process, in theory, sought to enhance MEL indicators and subgrantees’ 12 See PCBG FY18 annual report for details. 13 Survey data found that six interviewees rated the level of sector expertise support “poor” and two rated it “very poor.” USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 15 understanding of how to use MEL. But subgrantee interviewees gave the impression that most were not familiar enough with their MEL plans, seeing them as a reporting tool but not necessarily a management tool. One subgrantee did reflect when speaking with the ET the need to perhaps also take the MEL plans more seriously. The subgrantee acknowledged that its MEL plan has a number of redundant indicators and that it had only recently assigned an assistant to deal with the plan. Review of PCBG reports and subgrantee interviews revealed that depending on the sector,14 advocacy sector mapping efforts were undertaken up to six months after subgrant awards. Subgrantees did not provide clear feedback on the utility of the mapping that they completed, but several explained that most of their strategic shifts and thinking about their subgrants was done through experience and “learning by doing” rather than as part of a more guided PCBG process. Survey results supported this feedback, with six of the 11 giving low ratings for TA on designing advocacy strategies and advocacy campaigns. OF THE SUBGRANTEES VISITED, MEL PLAN COMPLETION TOOK: 5 months = 1 subgrantee 6 months = 2 subgrantees 7 months = 3 subgrantees 14 Mapping was initiated for extractive industries and health in FY18 Q1, but other sectors started later. Per PCBG reporting, mapping included identification of advocacy initiatives and stakeholders. 14 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV PCBG OUTREACH AND FACILITATION SUPPORT As part of PCBG’s statement of work (SOW), CPI anticipated supporting subgrantees’ efforts to further alliances and networks. While the FY18 Q4 report highlights current subgrantee networks, suggesting some type of PCBG contribution to this, eight of the 11 subgrantees surveyed in their narrative feedback found that PCBG has done little to reinforce coalitions and alliances even as they stressed the potential for action. Almost all subgrantees met described being already part of some networks in their issue areas and mostly able to do this themselves, given previous support and projects. One subgrant specifically includes forming an alliance in its objectives, and PCBG has helped support this effort.15 But overall, interviewed subgrantees noted few facilitation efforts. This is particularly true in relation to engaging with government and other policy actors, where subgrantees specifically noted in the survey and interviews that more support would be helpful.16 PCBG itself acknowledged that it has had a low public profile due to trying to get established and what it described as difficulties in dealing with the government. PCBG IMPLEMENTATION AND MANAGEMENT Both USAID and CPI described the slow start-up efforts to get the office open and staffed, noting the legal registration difficulties as well as identifying qualified staff with Portuguese language skills. PCBG and CPI also noted the challenge of synching field and HQ management processes, partially due to internal CPI issues, but also due to the general unfamiliarity of working in Mozambique. CPI noted low staffing levels as well as high staff turnover throughout implementation (particularly for advocacy specialists) as key constraints. Management practice effects are particularly visible when looking at the proposal selection and development process. While CSOs gave PCBG marks for a transparent concept and proposal process,17 starting from the initial APS in December 2016/January 2017, CPI and subgrantees point to a drawn-out proposal development process as the reason that only five proposals had been approved at the end of FY18 Q4. A majority of surveyed subgrantees referred 18 to PCBG’s inconsistent feedback during proposal development, which they described as giving them unclear guidance from multiple PCBG staff on how to develop their proposals. PCBG acknowledged that during this time they were all doing everything to help develop the subgrantees’ advocacy proposals and this resulted in some “projects by committee.” PCBG’s proposal editing process also has been a factor. Subgrantees and CPI described the editing process as taking between three and six months due to the need for review by PCBG and CPI headquarters. During this time, both sets of informants acknowledged that the PCBG editing process changed some subgrants, sometimes due to translation 19 and sometimes more substantially to “pass muster with [headquarters].” PCBG staff reflected that this process resulted in producing some subgrant activities that may have lacked internal coherence, increased subgrantee frustration and also might have lessened subgrantee 15 PCBG could not financially support the subgrantee to provide funding to other network members for their involvement. The subgrantee resolved this by reallocating funds through the IP of their other USAID-supported grant for this need. 16 Six of 11 subgrantees in the survey rated PCBG’s support in this area as “poor” (four ratings) or “very poor” (two ratings). 17 Four CSOs rated the process “good” and six rated it “very good.” 18 Eight of 11 subgrantees noted this. During subgrantee KIIs, three re-emphasized this point. 19 At least three subgrantees noted translation changes in the subgrantee survey. USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 15 ownership. The ET noticed during interviews that subgrantees were more comfortable discussing the general parameters of their work than specific activities and details. PCBG posited that the CSOs do have ownership of their interventions, but it has come with time. Subgrantees also pointed to general grant management reporting as a key concern and cause of confusion. For some, PCBG noted in their interviews and quarterly reports that learning to have a USAID-compliant financial grant reporting system took some time.20 However, a clear message out of survey data from nine of 11 subgrantees21 and three subgrantee interviews is the view that the CPI reporting requirements were burdensome, beyond requirements in place for other USAID-funded work that a number had, and generally a distraction from project efforts. Several subgrantees even noted that the reputation PCBG had acquired in managing subgrantees might have been a disincentive to potential applicants. CONCLUSIONS PCBG has not engaged in a systematic monitoring, evaluation and learning process that clearly shows the actual advocacy results to date. The MEL system is partial and lacks the relevant industry standards for tracking advocacy policy results. Therefore, the ET cannot report on progress against expectations. High-level results in conservation and extractive industries can be linked to the activity even as, for at least the conservation advocacy achievements, it seems that the activity played only a partial role, given the CSO’s earlier and ongoing efforts. Anecdotally, other advocacy results appear to be in the process of achievement and subgrantees appear committed to their advocacy work. However, most results are not being captured or recognized by the current monitoring system. Modest advocacy results to date are partially due to subgrantee implementation challenges. Most need more time due to ambitious projects and the realities of doing such advocacy work on both local and national levels in Mozambique. It is also unclear if use of the research component has helped or hindered results. The low level of PCBG technical advocacy assistance also has likely limited subgrantee results to date. PCBG provided insufficient guidance to address these challenges. It also did not sufficiently frontload support efforts, such as advocacy mapping, M&E (Monitoring & Evaluation) plan finalization or advocacy training and support that could have assisted subgrantees. Nor has PCBG provided much outreach and facilitation support. Probably due to CPI’s inexperience and low visibility in the country, PCBG has not provided much value added in building alliances with other policy actors beyond that which the CSOs are individually already doing. Finally, PCBG’s general management of the process has affected results to date. PCBG’s slow start, compounded by the long grant proposal and review process left CSOs frustrated and with subgrants that, in some cases, they needed to adapt to make workable. Subgrantee efforts were further frustrated 20 One example was in-kind contribution financing and reporting where nine CSOs in the survey expressed concerns over the lack of clarity on how to make these contributions. 21 Survey numerical feedback gave the reporting requirement process a 2.61 rating on a scale of 1 to 4, where 1 = strongly disagree, 2 = disagree, 3 = agree and 4 = strongly agree. 16 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV by grant management reporting, which required a steep learning curve for some and for all took significant time away from subgrant efforts. RECOMMENDATIONS (Re)focus effort on capturing what is happening in advocacy subgrants. 1. PCBG should engage with subgrantees to map actual activities, results and timelines to understand how subgrantees may have shifted and adapted from proposal design, and to obtain a clear understanding of subgrantee efforts to date and the challenges that subgrantees still face, including how they undertake and use research in their advocacy efforts. 2. PCBG should use the suggested mapping to bring together existing data on subgrantee activities and advocacy results in a unified and comprehensive system. PCBG could code it to address process￾based advocacy indicators, then add them to the PCBG activity-level system. This would provide more complete information on progress. 3. PCBG should consider a more comprehensive revision of some subgrantee MEL plans aligned with process-based indicators for subgrantees in the early stages of implementation (potentially three recipients) that run through December 2019. This is a way to begin to track advocacy processes more comprehensively. Bolster advocacy support and expertise. 4. PCBG should prioritize advocacy training and advisory assistance to subgrantees to introduce and reinforce this advocacy approach, including how to use/design evidence and research effectively. All subgrantees could benefit from receiving guidance in this area, even those with grants that end in the next three months. This could start with a workshop with CSOs to talk about key advocacy lessons to date, including issues such as: • Challenges of working locally and nationally at the same time; • How objectives, strategy and MEL relate and can be synergistic; and • Demand side of advocacy. 5. PCBG should consider increasing its advocacy TA capacities by having its incoming advocacy specialist collaborate with an advocacy process expert who could provide guidance on crafting advocacy process strategies and MEL approaches. Alternatively, or additionally, this expert could be directly available to subgrantees to work with them. Continue streamlining management processes. 6. PCBG should continue to review its grant management practices and support mechanisms to subgrantees and ensure that adequate staffing and systems are in place for the support needed. Identify select ways that PCBG can assist subgrantees’ outreach efforts. 7. PCBG should prioritize where and how it can add value to subgrantees in networking efforts and engagement with the government and focus on these areas during the remaining life of PCBG. USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 17 EVALUATION QUESTION 2 To what extent has PCBG achieved anticipated organizational capacity advances of the organizations to date? Why and why not? FINDINGS PCBG MEL FOR OD FIGURE 4: PCBG PERFORMANCE TABLE OD INDICATORS # PCBG OD Indicator Type of Indicator 11 Proportion of CSOs that received funding from donors other than Counterpart Financial sustainability indicator 12 # of CSOS with improved organizational capacity Composite indicator (No FY17/18 data) 13 # of CSOs effectively integrating gender into organizational structures, procedures and programming Gender indicator 14 # of institutions with improved capacity to assess or address climate change risks supported by USG assistance Climate change sector indictor (No FY17/18 data) 15 # of CSO staff who report using new skills and knowledge received in USG-funded training and provide an example Application of skills indicator 16 # of new funding sources identified, submitted, contracted or received by each CSO Financial sustainability indicator ET review of the PCBG performance indicators table found that the data provide only a partial picture of OD subgrantee progress. According to the table, two of six OD-focused indicators do not have reporting targets for FY17 or FY18. Indicator 12 covers all six areas of the PCBG OD action plan, with no reporting to date. Indicator 14 covers only subgrantees that are focusing on climate change, (two of 11 subgrantees). The four indicators that have reporting data provide a partial understanding of subgrantees’ OD progress. Two indicators specifically have FY18 Q4 reporting, but duplicate an area of a composite indicator: Indicators 11 and 16 cover aspects of financial sustainability, covered in the Indicator 12. Indicator 13 focuses on gender inclusion, which is a crosscutting area of PCBG focus, and is not currently captured in the OD action plan. Reporting through FY18 indicates that PCBG is behind in its targets (zero of six). Indicator 15 (CSOs self-reporting on using new skills through PCBG training) perhaps provides the most insights into progress to date, suggesting that PCBG reached FY17 targets and was just short of FY18 targets.22 PCBG does have OD assessment baseline data for all 11 subgrantees that covers the six areas of the OD action plan. This is presented in PCBG quarterly reports that were shared with the ET. However, PCBG acknowledged that they do not have a unified system in place that tracks each subgrantee’s 22 FY17 targets were 32 staff trained and self-reporting; PCBG reached 36. FY18 denoted additional staff trained, and PCBG reached 17. Please see the PCBG performance indicators table for more details. 18 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV individual OD action plan progress. According to PCBG, the activity is trying to put this into place after receiving a CPI OD action plan template and guidance on how to do this. Subgrantee OD action plan progress is in early to mid-level stages of their 12-month cycles. PCBG confirmed that nine of 11 subgrantees have approved OD action plans, but the earliest start of these action plans was in FY18 Q2, which means no subgrantees have completed their action plans. Survey data reinforced the early stage nature of OD efforts, suggesting that 10 subgrantees identified a medium to high need for further OD support. FIGURE 5: SUBGRANTEE CHART ON ONGOING/FURTHER SUPPORT OD AREAS AND FOCUS Survey data suggest that OD learning is occurring across the six areas. 23 Subgrantee survey and interview data point to high interest across CSO experience levels to complete what they consider realistic plans to further their organization’s development. This included some tailoring of the standard offerings to increase their relevance. For example, one of the more experienced subgrantees used OD funds to design a program-level M&E framework for one of the organization’s three programming areas. FIGURE 6: PCBG ORGANIZATIONAL DEVELOPMENT EFFORTS 23 These include external relation and partnerships; financial management and accounting; financial sustainability; human resources and materials; leadership and strategic management; and program management and quality control. USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 19 Subgrantees gave specific attention in the survey, and later in interviews, to areas that require more support. Three standout areas are M&E assistance, gender inclusion assistance and financial sustainability.24 Regarding gender and social inclusion strategies, PCBG reporting describes holding one training and working with subgrantees to develop gender action plans during FY18 Q2 and afterward. 25 Yet at least one subgrantee the ET met noted that practical application of these concepts required additional mentoring. This follows survey feedback, where seven indicated having developed a gender integration plan, but seven also indicated having no plan for gender mainstreaming. PCBG feedback also suggested having taken some time for subgrantees to “understand what it means to have a gender lens.” Several other subgrantees interviewed pointed out the need for additional financial sustainability mentoring or advisory assistance. 26 Two described hiring a consultant to assist in designing a financial sustainability strategy, but receiving poor-quality products. They reported little progress in revising or using these plans. The overall quality of financial plans developed during PCBG is unclear. Half of those surveyed said the plans were tailored to their organization and the other half suggested that they were not.27 Regardless, several noted that they need further assistance in implementing the plan. LEARNING AND VISIBILITY WISH LIST ● (More) Opportunities to share, compare and strategize among current subgrantees ● Opportunities to learn from peer CSOs in other countries ● More PCBG visibility on subgrantee work and advocacy efforts (Source: Subgrantee survey and KII feedback) PCBG IMPLEMENTATION Subgrantees give the initial OD assessment efforts strong marks. All subgrantees from the survey highlighted PCBG’s interactive OD assessment process and PCBG staff’s availability to answer OD￾related questions. Within this process, two subgrantees noted in the survey and later in interviews that PCBG was unwilling to use capacity assessments similar to those they had already done, and instead had the organizations redo the process. Approval of the OD action plans, however, is identified as a key constraint. Both PCBG and subgrantees pointed to slow OD action plan approval as a key delay in their OD progress. PCBG staff described having the plans go through two levels of CPI review—field office and headquarters—which took three 24 Survey findings suggest that 11 subgrantees indicated a need for more M&E assistance; seven indicated a need for more assistance on gender; and nine noted a need for assistance on financial sustainability planning. 25 Please see the PCBG FY18 annual report. 26 Subgrantee KII feedback tracks with survey feedback, where four subgrantees noted that one workshop on financial sustainability is not enough and suggested that a need for more support exists. 27 See survey findings: Financial sustainability planning process. 20 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV to six months.28 Several subgrantees complained about delays in the approval process, and others linked the lack of sufficient advocacy TA to the slow process of OD action plan approval.29 Implementation of some actions plan areas also appears to be problematic due to the way that PCBG allocated budgeting for OD support. PCBG staff noted that the PCBG operating budget for provision of OD was insufficient and did not allow for some planned OD activities. One example was a recent November 2018 advocacy training that had to be canceled due to lack of budget. The explanation provided was that PCBG OD funding must primarily come from the individual subgrantee OD budgets, 30 as general PCBG OD funding does not exist to cover these needs. CONCLUSIONS The level of PCBG OD progress is unclear, given the lack of systematic data to date and the early stage implementation efforts. Therefore, the ET cannot comment on actual advances in OD among CSOs assisted by PCBG. However, the data suggest that subgrantees are making use of the OD opportunities afforded, a promising sign. Nevertheless, more TA targeted to specialized areas, such as financial sustainability, M&E assistance, and gender inclusion would benefit the CSOs greatly. PCBG implementation of the OD component has been a constraining factor in OD progress to date. PCBG appears to have captured the interest of subgrantees at the beginning of the OD process, but implementation since this start has been slow and somewhat bumpy due to internal CPI management and budgeting for the OD component. Identifying appropriate advisory consultants for specific organizations has also been a challenge. This is likely to continue, especially for the more specialized areas of financial sustainability and gender inclusion. RECOMMENDATIONS Ensure a practical focus on capturing what is happening in OD action plans and support. 1. PCBG should endeavor to harmonize an internal subgrantee OD action plan tracking system that includes the six areas of OD focus covered in the plans. 2. PCBG should consider unpacking activity-level Indicator 12 into the six areas of focus for a clearer activity-level tracking option. Bolster OD expertise and support. 3. PCBG should prioritize identifying a strong pool of experts available to work with PCBG grantees. Financial sustainability and gender specialists are especially high priorities. 4. PCBG should consider the extent to which it can shift funding allocations to allow for increased options for PCBG to offer OD directly and not through the subgrantee budgets. 28 One subgrantee surveyed, whose approval time was even longer, indicated that they have already waited a year for approval of their OD action plan. 29 See survey sections III, “Advocacy Assistance” and IV, “Ongoing Developmental Efforts.” 30 Each subgrantee project has up to 10 percent of its budget to use toward OD. USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 21 EVALUATION QUESTION 3 To what extent did the design and structure (as amended during implementation) of the activity help and hinder achievement of results to date? FINDINGS ACTIVITY DESIGN VISION USAID undertook the PCBG activity design process to address what one USAID informant described as a need for “growing the B team of CSOs" in Mozambique to begin to do advocacy, or do better advocacy, in sectors where governance issues touch people’s lives. Those within USAID involved in design before the Request for Applications (RFA) thought it would be a success “if we could end up with a series of groups systematically more capable and able to follow up with government at various levels with a few concrete wins.” However, USAID describes this vision as becoming less crisp during the design process due to the realities of getting the activity funded and approved. Due to lack of Democracy and Governance (DG) funding, the activity design sought funding from different USAID technical offices’ funding streams.31 The multi-sector funding created a situation where multiple activity conceptual ideas affected the coherence of the design; as described by USAID, “The program description became a consensus document.” It also created a situation where, as USAID described, the activity was underfunded.” USAID claims that it selected the pre-competed Leader with Associate Award (LWA) procurement mechanism because this was the only option remaining to the Mission, given timing and USAID internal contracting office availability. This resulted in a single IP, CPI, being eligible to implement the activity. USAID acknowledged that concerns existed about the proposed IP, but USAID considered the proposal “good enough” to move forward. USAID TECHNICAL OFFICE INPUTS PCBG did not have technical expertise on staff, and the technical offices’ inputs into PCBG Annual Program Statements (APS) provided PCBG with the technical expertise to assist in designing the APS across the six thematic sectors. Similarly, PCBG noted that the technical offices’ expert presence during the proposal review process assured that those subgrantees selected had a certain level of sector specific competency. At the same time, several subgrantees referred to the APS design in their specific sector as limiting, suggesting that they would have prioritized different regions of the country or emphasis within their issue area if they could have. USAID described sector guidelines as being designed to complement other ongoing USAID programs. PCBG also suggested that working with the technical offices on the proposal review process extended the time necessary to complete the process. PCBG felt little support or presence from technical offices later in implementation. PCBG and USAID both noted that the activity performance indicators included several that were sector-specific (also 31 USAID described getting DG money later for FY 17 and FY18, which changed some of the anticipated pledges from the various offices, but for practical purposes, the Activity still had multiple streams of funding. 22 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV noted in Evaluation Question 1) without clear articulation of how these would affect PCBG indicator coherence. The ET received few details on how and whether these indicators helped the technical offices achieve the USAID development objectives linked to these mandatory indicators. USAID suggested that these challenges reflect general tensions around the technical offices having control over the programming and supporting the programming and the importance of the IP being able to develop its own coherent program. The IP was most vocal about the challenge of managing the multiple funding streams supporting the activity. CPI described spending significant time on financial reporting, while acknowledging that USAID was flexible on reporting requirements. PCBG also noted the difficulties in planning and allocating funding to the sectors (reflecting the relative amounts of contributions by each technical office) and subgrantee funding decisions. They described having insufficient funds in one sector and excess in another, given funding streams and the specific successful grantee applicants. For example, early on, PCBG described extractive industries as fully funded while health (specifically HIV) was not. Similarly, PCBG had funding in climate change but couldn’t identify subgrantees to which it could grant the funds. Juggling funding streams also affected subgrantee proposal designs. The subgrantee survey and interviews point to at least three examples where subgrantees needed to go back and change their proposals and budgets due to changes in funding ceilings (given available funds from the different streams) from $400,000 to $200,000 limits. Both USAID and PCBG described working to address these problems by combining funding streams (in the case of extractive industries, DG, climate change and biodiversity). But both expressed frustrations over resultant slow spending and delayed financial pipeline spend-downs. ADDITION OF THE ELECTION COMPONENT FIGURE 7: Election Subgrants USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 23 Elections subgrantees # Elections subgrantee Objective 1 MASC (representing Votar Moçambique) · Support monitoring and reporting efforts on electoral and political violence in the Nampula municipality by-election. · Secretariat of the Civil Society Monitoring Network (CSMN) 2 CEDES - Comité Ecuménico para o Desenvolvimento Social · Lead monitoring and reporting efforts in the southern parts of the country (Maputo, Matola and Chókwè municipalities) 3 ORPHAD - Organização Para a Promoção da Paz e Desenvolvimento Humano · Lead monitoring and reporting efforts in the northern region (Angoche, Monapo, Nacala, Nampula, Montepuez and Mocímboa da Praia) 4 CEURBE - Center for Urban Studies · Manage gender in elections component 5 ANDA - Associação Nacional para o Desenvolvimento Auto￾Sustentado · Coordinate the monitoring exercise in Manica province (Municipalities of Chimoio, Manica, Gondola and Catandica) 6 PNDH - Pressão Nacional dos Direitos Humanos · Lead monitoring and reporting efforts in Sofala province (Municipalities of Beira, Nhamatanda, Gorongosa and Marromeu) 7 CESC - Centro de Aprendizagem e Capacitação da Sociedade Civil · Lead monitoring and reporting efforts in Zambézia province (Municipalities of Quelimane, Gurué and Mocuba.) Adding the election component had several unexpected positive effects. PCBG’s intense work with its election subgrantee CSOs increased its visibility among donors and other CSOs in Mozambique. While PCBG described getting the CSO elections network up and running as being challenging and labor intensive, its efforts made it more visible. For example, the ET met another civil society donor that was not familiar with the PCBG advocacy efforts but was familiar with the activity’s election work due to its donor coordination efforts and work with the election CSOs. Another pleasant surprise was that existing PCBG advocacy subgrantee CESC, which had been working with citizen manifestos on the local level in Nampula and Maputo as part of its advocacy subgrant, was asked to step in to assist on elections through a second subgrant on elections. This process helped create synergy between the two parts of PCBG, as CESC used lessons learned from its efforts to work with PCBG on engaging communities similarly in Zambézia as part of PCBG’s election component. However, even as both USAID and PCBG indicated plans to work together again on elections in 2019, the component took PCBG staff time away from core activities. The component, which was designed to have an additional set of subgrantees focus on 2018 local elections, was added just as PCBG´s core advocacy efforts had finally gained traction. After more than a year of start-up, proposal calls and review, the first five subgrantee awards were made during FY17Q4. This was also the period when OD assessment efforts for these subgrantees began in earnest. PCBG hired two additional election officers 24 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV to work only on elections, but PCBG noted that other activity staff who were engaged in OD, advocacy, grant management, etc., also assisted the election subgrantees (see Figure for reference).32 FIGURE 8: PARTIAL PCBG ACTIVITY TIMELINE PCBG and USAID also noted that PCBG leadership was primarily engaged on election issues rather than advocacy and OD efforts. Both PCBG and USAID estimated that the PCBG chief of party (COP) at the time, who had election expertise, spent most of her time (between 60 percent and 70 percent) addressing the challenges of implementing the election component. PCBG noted that leadership for the advocacy and OD efforts shifted to the PCBG deputy (who also had partial advocacy duties) during this time. PCBG staff noted this compromised coordination and overall management of the many diverse aspects of the activity. CONCLUSIONS Important aspects of the PCBG design and structure hindered the activity. It was designed to work intensively on advocacy with a broad set of CSOs. Yet the funding levels secured and the IP mechanism used limited the ability of USAID to procure an IP with the functional and country-specific experience that could realize this vision. The multi-sector structure of the activity presented significant budgetary, technical and management challenges that burdened the IP, may have distorted local CSO ownership of the advocacy efforts and provided little clear benefit for USAID technical offices’ program efforts. The introduction of the election component further distorted implementation efforts and appears to have had a primarily negative effect in the past year. The IP was not able to adapt and take on the extra programming without compromising its already challenging core advocacy and OD implementation efforts. 32 ET review of PCBG FY18 quarterly reports noted the set of phases that election subgrantees passed through, including a proposal stage, modified OD stage and advocacy stage. USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 25 RECOMMENDATIONS For USAID’s considerations in further design: 1. Critically examine whether funding mechanism, funding level, and activity specifications are aligned in order to provide a basis for optimal outcomes. This includes ensuring sufficient procurement process time to be able to use the proper IP mechanism. 2. In the case of multi-office funding, then clear and shared activity objectives need to be designed between the offices that align with the respective development objectives and that include a clear understanding of design and implementation engagement and responsibilities. 3. USAID should consider how to allow design of sector-specific funding to be more consultative with local partners in identifying needs and priorities to ensure feasibility and ownership. For current PCBG implementation: 4. PCBG should consider carefully how to manage new election work which may include building “firewalls” into the PCBG activity (sufficient staffing and management plan) to ensure that anticipated election efforts do not overwhelm its core advocacy efforts. EVALUATION QUESTION 4 What lessons can be learned from the PCBG experience to date for crafting future USAID civil society programming that promotes self-reliance and government transparency / accountability? The evaluation exercise reveals lessons for consideration when designing future civil society programming. Fit for purpose: 1. Combining OD and advocacy purposes requires an implementing partner familiar with country context and able and ready to carry out intensive work The PCBG experience underscores the challenge of crafting subgrants with dual objectives in countries like Mozambique where CSOs likely require both advocacy and OD support in a challenging environment. If the IP does not have this familiarity, subcontractors may be able to fill in the necessary gaps. What is crucial whichever the IP composition, is the ability to start quickly and strongly lay out a base to guide subgrantee learning and efforts. 2. Advocacy activities need to be designed and managed to capture advocacy influence process results. Advocacy is a process and advocacy-focused activities need MEL that can capture interim results in this advocacy process. This requires Tan IP with advocacy and MEL expertise to guide and work with subgrantees to ensure a strong foundation in advocacy training, MEL tools and advocacy technical assistance. Realistic design assumptions: 3. CSO advocacy capacity must be aligned with advocacy influence expectations. CSOs with lower levels of advocacy capacity require longer-term engagement and realistic expectation on what can be 26 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV achieved. The USAID-envisioned design sought to work with a “B” team, yet the IP guidance to subgrantees in designing advocacy objectives was ambitious and the timing was relatively short, suggesting a misalignment in expectations between USAID and CPI. 4. Invest the necessary inputs and strategy to do both national and local-level advocacy, if both levels are desired. The advocacy effort required for doing synergized national and local advocacy is greater than simply including local and national components in a subgrant that may or may not be complementary. It requires a deliberate design process with subgrantees to strategize and map out actors and options. It also requires clear mechanisms to enhance alliances of these actors, which may suggest flexible funding mechanisms for subgrantee alliance or partner work, as well as larger funding levels and longer subgrant duration. 5. Advocacy programming requires mapping out advocacy demand at the design stage of the overall activity and by subgrant issue area. Advocacy programming requires a critical review of the “demand” side for the advocacy. This suggests a need for sector mapping and government receptivity (capacity and interest) as part of design to craft and implement realistic short-term and medium-term programming parameters and objectives. Sequencing matters: 6. First provide OD and then advocacy technical assistance inputs. If OD and advocacy are together in a grant mechanism, the duration of the grant needs to be longer and OD needs to be frontloaded to ensure that CSOs have the necessary capacity and skills to carry out the advocacy programming. 7. Advocacy strategy and MEL needs to be part of the subgrant design process. For the most effective learning- and result-oriented advocacy programming, early implementation emphasis is needed on coordinating the design of advocacy objectives, strategy and MEL to have focused and realistic subgrants that can adapt. Maximizing partner relevancy and ownership: 8. OD benefits across all experience levels are possible if balanced guidance and flexibility are available in designing OD-tailored plans. This includes assurance that both more and less experienced organizations can prioritize their OD needs, coupled with IP ability to provide quality advisory service options to subgrantees. 9. Achieving a local partner-driven activity requires a nuanced balancing of donor and local partner issue priorities and a degree of implementation flexibility. Local CSO and expertise consultation should be part of activity design, and the proposal and implementation process should be designed to provide space for CSOs to lead and adapt while the IP supports a coherent implementation and learning process. USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 27 ANNEX A: STATEMENT OF WORK MIDTERM PERFORMANCE EVALUATION (MTE) OF THE PARCERIA CÍVICA PARA BOA GOVERNAÇÃO (PCBG) ACTIVITY 1. Overview & Program information 1.1 Overview Despite nearly four decades of international donor assistance, the overall capacity of Mozambique’s formalized civil society is considered among the weakest within the southern African region. The overwhelming majority of CSOs suffer from the lack of qualified human resources, weak internal governance and financial management capacity, and inability to influence government policies, especially at the national level. Advocacy is a relatively new concept for most Mozambican CSOs, associated with a narrow set of one￾off activities, such as dissemination of publications and holding meetings rather than well-planned strategies sustained over a period of time. The number of organizations capable of conducting evidence￾based research and conducting issue-based advocacy is limited to roughly one dozen national-level CSOs based primarily in Maputo. However, these CSOs often lack linkages to grassroots constituencies and less organized community-based organizations (CBOs) that represent them. Despite the challenges, civil society in Mozambique has recently accomplished important gains in holding itself and the Government of the Republic of Mozambique (GRM) more accountable. The concept of social accountability also appears to be taking hold as more and more CSOs at all levels are using performance monitoring tools, including citizen report cards (CRCs), community scorecards, budget hearings and social audits, to track government’s service delivery performance. Various groups are learning to leverage the power of collective bargaining, creating networks, thematic forums and cross￾sector platforms at all levels. 1.2 Activity Objectives The goal of the Parceria Cívica para Boa Governação (PCBG) activity is to improve accountable and effective democratic governance in Mozambique. To this end, Counterpart International will partner with up to 12-18 Mozambican civil society organizations (CSOs) to advocate in the target sectors of biodiversity conservation, climate change, education, extractive industry, health, transparent and accountable governance, and peaceful elections. This goal is supported by two interrelated objectives: 1) more effective advocacy by partner CSOs in target sectors and 2) strengthened organizational performance by partner CSOs. The Activity theory of change proposes that by improving the advocacy and organizational capacities of motivated, committed Mozambican partners, these partner CSOs will implement more effective advocacy strategies, resulting in specific changes in government policies, procedures, services and/or enforcement. Those reforms will contribute to longer term strengthening of accountable, effective democratic governance in Mozambique in target sectors. The program intends to contribute to the development objectives (DOs) laid out in USAID/Mozambique Country Development Cooperation Strategy. 28 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV The Activity aligns with three objectives of USAID/Mozambique’s Project Appraisal Document (PAD): Objective 1: More effective advocacy by partner CSOs by supporting the technical ability of partner CSOs to work on sectors of biodiversity conservation, climate change, education, extractive industry, health, and transparent and accountable governance. Objective 2: Strengthened organizational performance by partner CSOs through intensive and customized technical assistance, coaching/mentoring, and training for partner CSOs using a customized participatory organizational development assessment model. Objective 3: Facilitation of a credible, participatory and peaceful electoral process through long-term domestic electoral violence monitoring, conflict mitigation and advocacy activities throughout the election period of what may become the most contested election in Mozambique’s democratic history. 1.3 Activity Implementation USAID awarded Counterpart International a Cooperative Agreement in the amount of $10,500,000 to implement the four-year PCBG activity and an additional component of $1,325,000 to support peaceful and credible elections, which was issued on July 31, 2017. The Cooperative Agreement runs from 25 March 2016 to 24 March 2020. 2. Statement of work 2.1 Evaluation Purpose The evaluation will provide information to help USAID and the implementing partner make adaptations to improve results and to guide design of future interventions. This will also ensure that the MTE provides a comprehensive picture of implementation experience of the Counterpart International and subgrantees to date. 2.2 Evaluation Objective & Questions To achieve that purpose, the evaluation team will pursue the evaluation objectives and related evaluation questions presented in Figure 1. FIGURE 1: PCBG MIDTERM EVALUATION OBJECTIVES AND QUESTIONS Evaluation objectives Evaluation questions A. Determine the progress to date in achieving advocacy and capacity development activity results and implementation factors affecting this. 1. To what extent has the PCBG activity achieved anticipated advocacy results with the organizations to date? Why and why not? 2. To what extent has the PCBG activity achieved anticipated organizational capacity advances of the organizations to date? Why and why not? B. Understand how activity design and structure affected results. 3. To what extent did the design and structure (as amended during implementation) of the Activity help and hinder achievement of results to date? C. Consider best practices for developing similar advocacy focused activities. 4. What lessons can be learned from the PCBG experience to date for crafting future USAID civil society programming that promotes self-reliance and government transparency/accountability? USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 29 3. Methodological Approach 3.1 Methods The evaluation will use a three-step mixed method approach to examine and unpack implementation performance of the PCBG in achieving specific activity objectives and in working towards the goal of more accountable and effective governance, as detailed below.33 I. Secondary Data of PCBG Program Documents First, the Team will examine secondary data through desk review of major PCBG programming documents provided by USAID and Counterpart International. This will provide the Team with a descriptive understanding of major PCBG operational and programming implementation experience (and milestones). Major trends and issues will be documented.34 This will include a review of planned vs. actual results to date and identification of factors that explain the divergence in achieved programming results. II. Subgrantee Survey Informed by the document review, a short survey35 will be designed for subgrantees that will focus on getting feedback on implementation and programming efforts to date. The survey mechanism will serve as an initial data gathering technique for understanding grantee experience with PCBG grant development and implementation. This will provide subgrantee perspective on activity design, operational performance and implementation experience. In addition, any particular implementation challenges subgrantees note as part of PCBG engagement during the different phases of pre subgrant, grant, and TA efforts will be captured. Not all grantees will have had experience in all areas of PCBG programming given the implementation schedule,36 but the survey will provide a broad mapping of subgrantee experience with PCBG and the key advocacy and capacity results to date per organization. The survey will include closed and semi-open questions, which will provide data points for qualitative analysis. Preliminary analysis will help finetune interview instruments for subsequent KIIs. The survey will focus on: − Feedback on design of grant application process; − Feedback on grant implementation process; − Feedback on advocacy TA related to the grant and advocacy learning gains; − Feedback on capacity building efforts and capacity gains; and − Feedback on what else is further needed. 33 Election CSOs will be considered in relation to the activity’s capacity building efforts and overall implementation efforts. Election programming and outcomes are outside the focus of this evaluation. 34 The evaluation team will continue to collect relevant activity data and documents throughout the evaluation. This includes such documents as the subgrantee advocacy proposals and grants, the OD Action Plans, Gender Integration Plans, and the Advocacy MEL plans as well as the most recent quarterly report (FY18Q4) as will be possible before and during field work. 35 This survey will primarily be administered in person if subgrantees are present in Maputo. Those not present will be interviewed by phone. 36 PCBG quarterly reporting through FY18Q3 and its anticipated progress through FY18Q4 suggests that there are 12 advocacy subgrantees and 3 election subgrantees. For the advocacy subgrantees, only a portion (approximately 6-8) will be able to answer all question related to PCBG technical assistance on advocacy projects and organizational development, given some awards were only recently signed. The three election subgrantees will be asked all questions except those focused on TA for advocacy projects. 30 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV III. Key Informant Interviews Based on analysis of survey findings and the implementation trends noted in document review, the evaluation team will design a set of key informant interview (KII) guides for three groups of stakeholders and beneficiaries. The first set of the semi-structured instruments will seek USAID and Counterpart International perspectives on PCBG experience, challenges and progress to date. More specifically, the MTE will look to identify underlying factors that may have facilitated or hindered activity implementation. This will include key Mozambique and HQ Counterpart International personnel involved in PCBG as well as Mission personnel overseeing the PCBG activity.37 This may include group meetings with either PCBG or USAID staff. The second set of informants will be a select set of the subgrantees.38 Structured questions for this group will focus on assessing in greater detail subgrantee capacity and advocacy programming outputs of the activity to date. This might include a case study approach which examines several of the grants that are most ‘advanced’ to better understand the dynamics and factors affecting progress to date. The third set of informants will primarily be external stakeholders and experts.39 This group may include select policy actors from the GRM, 40 other donors, local CSOs, and policy advocacy experts in Mozambique. Semi-structured questions for this group will focus on assessing their perceptions of PCBG performance, the policy advocacy context in some sectors, and advocacy support programming in Mozambique. This will likely include approximately 8 informants. Survey data, descriptive statistics, and transcripts from three types of KIIs will provide main data points for evaluation analysis. This will be supplemented as is relevant by any field notes from observations of any relevant advocacy or capacity focused activities or events seen by the Evaluation Team. 3.2 Sampling All subgrantees engaged in PCBG will be interviewed for the survey, and a set of these will be further engaged for KIIs. Subgrantees engaged with the project are primarily based in Maputo.41 The Team will conduct in-person surveys with all subgrantees based in Maputo, and subgrantees outside of the capital will be interviewed for the survey by phone and Skype. For KII selection, the Team will adopt a purposive sampling approach to identify in consultation with Counterpart and USAID an initial list of 37 Anticipated interview numbers include between 6-8 PCBG staff and approximately 5 USAID staff. Informant numbers and selection will be decided in consultation with USAID and Counterpart. 38 Subgrantees will be selected partially on survey feedback and in consultation with PCBG and USAID. 39 The one exception may be inclusion of the organization Maraxis BVT who did the civil society assessment for Counterpart. 40 Specifically if doing a case study approach, relevant government policy actors from the sectors examined will be identified. 41 Per PCBG data, 3 of the 12 advocacy subgrantees are not in Maputo; 2 of the 3 election CSOs are not in Maputo. KII groups Approx. No. Group 1 PCBG USAID 12 Group 2 Subgrantees 8 Group 3 External Stakeholders and Experts 8 Total: 28 USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 31 informants in each of the informant groups. The team will add informants throughout the evaluation, as appropriate, based on information received during the field research.42 3.3. Data Analysis Methods and Data Disaggregation The Evaluation Team will take a systematic approach to analyzing the data. Data collected from each of the 3 data sources for a given question will be analyzed separately, then considered with the data analysis and findings from other sources used to answer the question. This will allow the team to consider both points of convergence and divergence and reasons behind these differences to come up with its findings. As part of this process, descriptive statistics from the survey may be disaggregated by sector or other delineation as may be useful. 4. Team Composition, Responsibilities, and Roles The evaluation team will be comprised of two short-term consultants and MMEMS full-time staff. MMEMS will also provide the required logistics. MMEMS Senior Evaluation Officer, or his designate, will provide evaluation oversight throughout the evaluation. He/she will be responsible for the overall client management, provide quality assurance and technical oversight. The core team includes Dr. Kristie Evenson (Team Leader) and Edson Correia (Civil Society Specialist). The Team Leader will oversee the design, implementation, and analysis of the evaluation and writing of the report. The Civil Society Specialist will participate in data collection, analysis of data and also serve as a translator for translating the survey and interview guides into Portuguese and for carrying out key informant interviews as needed. It is envisioned that the Civil Society Specialist will carry out the initial survey with subgrantees in close cooperation with the Team Leader. The Evaluation Team will carry out Key Informant Interviews (KII) together when the Team Leader arrives in Mozambique. 5. Period of performance and expected level of effort This SOW is designed to cover a three-month period from October 2018 through submission and finalization dates in January 2019. Total LOE for Team Lead is up to 40 days, and Civil Society Specialist is 33 days 6. Deliverables The first deliverable will be the evaluation workplan, to be approved by USAID. The work plan will include Subgrantee Survey instrument and describe the overall evaluation approach. Initial survey data will be compiled and analyzed prior to the arrival of the Team Leader. In addition, the Team will prepare draft KII guides and an interview schedule before fieldwork. As possible, interviews will be arranged prior to Team Leader arrival in Maputo. With the arrival in-country of the Team Leader, the team will conduct a Team Planning Meeting (TPM) to finalize details around KII guides, scheduling interviews which will result in work plan finalization and a shared understanding of implementation and write up and analysis efforts. Other PCBG evaluation deliverables will follow: 1. Weekly progress reports 2. Findings, Conclusions, and Recommendations table (FCR matrix); 42 Disaggregation by informant group, sector, gender, region, etc. will be part of the informant summary descriptive statistics. 32 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV 3. Draft report; 4. PowerPoint and oral presentations, including methodology, data analysis methods and data disaggregation, findings, conclusions, and recommendations; and 5. Final report. Deliverables table Evaluation phase Deliverables Dates Preparation − Subgrantee Survey Development November 2018 Fieldwork Planning − Survey Initial Findings − KII Draft Guides − Work plan Finalization November 2018 By November 27, 2018 By November 30, 2018 Fieldwork − Team Planning Meeting − KII interviews and write-ups − Data analysis and development of Initial Findings, Conclusions, & Recommendations November 28, 2018 December 2018 December 2018 Reporting − Out-brief presentation to Mission − Draft report − Comments to report − Final report December 2018 January 2019 January 2019 January 2019 USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 33 ANNEX B: SUBGRANT SURVEY FINDINGS SUMMARY SURVEY FINDINGS INTRODUCTION As part of the evaluation, a subgrantee survey was conducted between November 12 and 19 which targeted 11 advocacy subgrantees in order to do an initial mapping of their implementation and technical assistance experience as well as an indication of advocacy and OD capacity gains to date. The survey included semi-closed ended questions, which provided additional data points for qualitative analysis, which were explored in more detail through KIIs. Those CSOs who received grants for the elections component were not included. The survey informed KIIs in a way that: • Further highlighted key concerns already flagged from the document review. For example, reminded team of the need to specifically ask about in-kind contributions from IP and subgrantees. • Added additional questions on details of design and implementation to probe with all three sets of stakeholders: For example, raised issue of language problems in proposal development process. • Provided general trends perceptions for use in querying IP and USAID • Provided CSO specific feedback, which will be tailored for each of the 7 subgrantee KII guides to solicit additional details and feedback. In all, 12 interviews43 using the survey template were conducted involving 11 CSOs. These include: TV SURDO, LIVANINGO, MAHLAHLE, CESC, FRUTICAD, KUWUKA, MEPT, BIOFUND, NTUMBULUKO, AENA, and AMDEC. 2 additional surveys were conducted in November 27 with 2 CSOs, OPHAD and UATAF, who submitted complete proposal but did not receive the grants. For these 2, the survey specifically focused on the grant application process. FIGURE 7: CSOS BY SECTOR The results are presented in 5 sections as follows: I. Grant application process: includes the APS call for proposal, support for proposal development 43 There were 2 interviews with Livaningo 34 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV II. Grant implementation process: includes the grant disbursements, reporting requirements, in￾kind contributions and M&E design process. III. Advocacy assistance: includes sector specific expertise inputs, advocacy design and strategy inputs and linkages with policy makers and other CSOs IV. Organizational development efforts: includes the OD assessment, OD action plan, gender assessment and integration and financial sustainability planning. V. Ongoing/further needs: focus on identifying areas of further support. Each section includes a set of sub questions for which a 1 – 4 scale44 was used for ratings. Respondents’ reasons for the ratings are coded according to the key themes for each sub question. The chart below shows the average rating for each key question. These will be further detailed in sub questions and sub items. FIGURE 8: AVERAGE RATING – ALL SECTORS I. GRANT APPLICATION PROCESS • (3.43): Application process: while gave good ratings on clarity, most had comments about confusion in the process: too long; differing PCBG guidance; translation issues; different ideas of advocacy, geographical scope and budgets. • (3.55): Bootcamp: generally considered as useful • (3.25): Mentoring: PCBG general availability good; individual feedback mixed due to PCBG staff knowledge and language levels and consistency of advice. • (3.23): Award process considered clear but generally taking a long time. II. GRANT IMPLEMENTATION PROCESS • (3.54): Grant disbursement process: generally, consider okay with only a few mentioning delays • (2,61): Reporting requirements process: considered to be unclear with inconsistent PCBG feedback to assist, considered too bureaucratic and too much effort (initially weekly reports and now monthly). 44 1- very poor, 4 - very good USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 35 • (2.88): In kind project financing project: almost all were confused by information presented for how to make in-kind contribution and many did not think should apply to CSOs. • (3.23): M&E design process: most perceived as useful; several considered late for their project III. ADVOCACY ASSISTANCE • (2.31): Sector level expertise inputs for additional technical knowledge on your sector issues: most said sectoral expertise was poor and a number also said it was late due to OD plans. • (2.55): Advocacy design and strategy inputs for your projects: had a training but many feel that have not received enough support and that there is not enough PCBG capacity to do this. • (2.36): Facilitation with policy makers / private and public sector actors: some have benefitted but more needs to be done and earlier. • (2.55): Facilitation with other CSOs involved in the sector: some effort to match organizations individually, but not a lot of group meeting efforts or networking yet. IV. ORGANIZATIONAL DEVELOPMENT EFFORTS • (3.4.2): OD assessment process: considered interactive and participatory by most; several noted that duplicative of earlier OD assessment efforts with different methodology which PCBG would not consider. • (3.55): Design of OD action plan: most consider the plan realistic and doable and see PCBG willing to help. • (2.89): Gender assessment and integration process planning: training was considered okay and most have some type of plan, but some confusion about how to do actually implement it. • (3.10): Financial sustainability planning process: trainer considered not knowledge of local context even as training useful; many feel the need for further training / assistance to make plans realistic for their organization. V. ONGOING/FURTHER NEEDS FIGURE 9: LEVEL OF SUPPORT NEEDED • Advocacy project technical management: Still need for attention: new CSO staff and need to reinforce knowledge • Advocacy project strategic planning: Sense that follow up trainings are needed. 36 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV • OD Action Plan: Need follow up support on most of 6 pillars with financial sustainability mentioned most often. • Financial Sustainability: See above. • Other: Sharing of experiences among CSOs from other countries with similar experiences; Consider how to have more visibility for PCBG and its subgrantees results. SOME COMMON IMPLEMENTATION THEMES • PCBG is available and willing to assist • Timeliness of PCBG interventions/assistance is mixed • PCBG assistance/feedback not consistent • PCBG capacities to provide advice required mixed RATINGS BY SECTOR FIGURE 10: AVERAGE RATING BY SECTOR FIGURE 11: GRANT APPLICATION PROCESS RATING BY SECTOR USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 37 FIGURE 12: GRANT IMPLEMENTATION PROCESS RATING BY SECTOR FIGURE 13: ADVOCACY ASSISTANCE RATING BY SECTOR 38 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV FIGURE 14: ORGANIZATIONAL DEVELOPMENT EFFORTS RATING BY SECTOR USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 39 ANNEX C: INFORMANT GUIDES USAID and CPI Guide Date Position and Organization Introduction and purpose of the interview as part of the evaluation Introduction 1. Tell us about your role in relation to the PCBG Follow up: When did you begin working on/with PCBG? (As is relevant for informants) 2. Can you provide us some background on the design of the activity? Follow up: What experience until 2016 prompted the focus on these sectors? Follow up: What experience until 2016 suggested the need for this capacity design? Follow up: How does OD approach build on/duplicate/other of previous USAID other donors? Performance implementation (EQ1, 2, 3) 3. In general, how do you think PCBG implementation is going? Why? 4. Which operational issues have been most challenging and/or need attention? (if any) Follow up: Effects of COP and frequent staff changes? Follow up: Effects of slow official opening of the office? Follow up: Effects of long proposal process? 5. To what extent have other factors external to PCBG affected the project? Follow up: How have the funding stream restrictions affected PCBG (FY18Q1)? Follow up: Effects of the addition of the election component? Follow up: Political happenings/situation in Mozambique? 6. Which program issues have been most challenging and/or need attention? (if any) Follow up: Attitude of CSOs: did they see the need for both advocacy and OD development? Follow up: Management of subgrantees and their grant management capacities Follow up: Sequencing/synchronizing the many advocacy and OD TA components Advocacy Results (EQ1, EQ4) 7. How would you assess subgrantee advocacy results until now? Follow up: Sector specific results vs. overall results Follow up: What is better than expected and why? Follow up: What is slower than expected and why? 40 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV 8. Which advocacy skills do you think CSOs have particularly gained through PCBG? Follow up: How would you rate Networking and alliance efforts? Follow up: How would you rate advocacy strategy design? Follow up: How would you rate use or creation of policy products in advocacy? Follow up: How would you rate engagement with government and other policy actors? 9. To what extent do you see the CSOs either seizing the opportunity or dealing with the current policy context in their sector? Follow up: Which sectors have the most policy opportunities for advocacy? Follow up: Which sectors have the most need to for advocacy? Organizational Development (EQ2, EQ4) 10. How would you assess OD results until now? Follow up: Which of the six program areas appears to be the most challenging? Follow up: How is financial sustainability being emphasized and what is the CSO reaction? Follow up: What about gender integration and M&E skills/interest? Follow up: What level of OD skills do you anticipate CSO will gain by the end of the project? 11. From your experience with PCBG, which types of technical assistance appear to be the most appropriate for different OD and advocacy capability building efforts? Follow up: Training/workshop setting vs. STTA expertise Follow up: PCBG mentoring vs. local CSO mentors (?) Lessons Learned and Looking Ahead (EQ1-4) 12. Looking towards 2020, what needs to happen in the PCBG activity to make you feel satisfied with your efforts? Follow up: What are you biggest lessons learned and how have you addressed these? Follow up: What are the biggest challenges you yet anticipate and how will you address these? 13. What (additional) changes if any need to be made to the programming/management for the remainder of the PCBG? 14. What advice would you give to designers of a follow on program to PCBG? Follow up: Which areas of design worked/did not? Follow up: Need and contextual appropriateness 15. Any other information/feedback/recommendations you would like to share with us that we did not discuss? Additional Specific CPI Questions Current status issues: 1. What is the status of the KMP and use of SharePoint? Can we have access? USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 41 Conceptual/design issues: 2. Tell us about your approach to tracking advocacy influence. Can you share with us your advocacy￾training curriculum and can we walk through your thinking in the design of the Indicators Performance Table? Management issues: 3. Walk us through the different COP changes and how CPI managed this. 4. In relation to elections, how much staff time and energy did this take since adding this component? How much was planned vs. actual? 5. Can you explain the management structure in relation to working with subgrantees in the different sectors? 6. Advocacy vision from within CPI: how has changeover in advocacy specialist position (new from Jan 2018) affected advocacy planning and support? Sequencing/timing issues: 7. Walk us through how you scheduled out the different trainings which had advocacy and OD training components: 8. For example, why was training for advocacy mapping done in FY18Q3 for those awarded in FY17Q4? Similarly, why was M&E tracking systems for the grant done in FY18Q2 for those awarded the previous year? 9. Branding and outreach strategies in FY18Q2 and how this affected ongoing subgrants? 10. How has the management process of onboarding changed during the life of the project – Ways to sequence differently, combine – i.e. frontload some things like M&E? PCBG role in supporting advocacy efforts: 11. Can you explain rationale for mapping advocacy initiatives and stakeholders in health and extractive industries in FY18Q1 rather than during proposal process or at project start? 12. And also, for the sequencing of the desk review of specific advocacy networks sectors? (completed FY18Q2 rather than at beginning of grants, why?) 13. Advocacy role of PCBG: how does it see itself vis-à-vis the gov’t -- as a donor/as a policy actor? 42 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV Additional Questions to Consider from the Survey for KIIs APS CALL FOR CONCEPT AND PROPOSAL PROCESS USAID PCBG • Understand how the budget was derived for the proposals. Who was the decision maker? • What was USAID expectation of what was feasible or limited the budget. • What was expectation of PCBG coverage in Mozambique regarding CSO selection? • Understand how the budget was derived for the proposals. Who was the decision maker? • What was USAID expectation of what was feasible or limited the budget. • How were CSO targeted? Survey indicated that Maputo was national and excluded other regions. Why? • Staff turnover issue. How did CPI try to manage review process? MENTORING FOR PROPOSAL DEVELOPMENT USAID PCBG • • Language issue • How budget ceilings were defined? GRANT DISBURSEMENT PROCESS USAID PCBG • • What caused delays with disbursement? • What PCBG did in terms of clarifying the timing for disbursement process. Why? What could be improved? REPORTING REQUIREMENT PROCESS USAID PCBG • Was USAID involved in reviewing the fund requests from Subgrantees? • How was this designed? • How did PCBG manage orienting subgrantees? There was availability, but inconsistent feedback happened. Staff turnover? • What support was provided continuously? How? • What could have been done during boot camp or prior to help CSOs manage time better and reporting? IN KIND PROJECT FINANCING PROCESS USAID PCBG • • How requests for in kind contribution was made / explained to subgrantees? SECTORE LEVEL EXPERTISE INPUTS USAID PCBG • • What was done in terms of providing specific sector level expertise to subgrantees? FACILITATION WITH POLICY MAKERS / PRIVATE SECTOR / OTHER CSOs USAID PCBG • • What was done in this regard? DESIGN OD ACTION PLANS USAID PCBG • • Why are there delays in approving OD plans? USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 43 Subgrantee Guide Date Position and Organization Introduction and purpose of the interview as part of the evaluation Thank you for speaking with us again about the PCBG. The last time we spoke you provided rich feedback on your implementation experience. Today we would like to focus mostly on your advocacy project to hear from you how you think your advocacy project is going and what you are accomplishing. As part of this we would also like to talk a bit more about aspects of PCBG technical assistance to better understand how these have affected your work and organization. Introduction 1. To give us a better understanding of your work in ___sector, can you give us some more background on your organization’s work in this area? 2. What did you hope the PCBG grant would allow you to do given your previous work in this sector? Follow up: Was interest more on advocacy or capacity or both? Follow up: Was it based on any previous advocacy work in this area? Advocacy Results (EQ1) 3. Walk me through the main objectives/outcomes of your project. Why did you think these objectives were possible at this time? 4. Until now, we understand45 that you have had some level of success/challenge in ____ what have you learned from this process? 5. What you think you will be able to achieve by the end of your project? Follow up: How does this differ from what you initially anticipated? PCBG Advocacy Technical Assistance (EQ1) (As relevant based on survey findings per organization) 6. In your survey you suggested that TA on advocacy ___ was ____, can you give us an example of how you have been able to apply these skills in your project? and/or 7. You suggested that more guidance on advocacy ___ would be useful. Can you explain how you would use this in your advocacy efforts? 8. Overall where do you see your strengths in doing advocacy and where does your organization still have something to learn? 9. Which is the best method in your view for PCBG to impart these skills? (trainings, mentoring one on one, STTA visit, other) 45 This is based on either survey and or subgrantee reports. 44 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV How let’s talk a bit about the third objective of the subgrantee in relation to your organizational development. PCBG OD Technical Assistance (EQ2) 10. What is your experience in having the subgrant include both advocacy and organizational development objectives? Follow up: how does this effect ability to do MEL/ conceptual clarity/track progress? 11. Tell us (more) how you have used the capacity building funds Follow up: Were there concerns in how to operationalize these? Follow up: Were there priorities you identified that exceeded budget levels? (As relevant based on survey findings per organization) 12. In your survey you suggested that TA on OD ___ was ____, can you give us an example of how you have been able to apply these skills in your organization and as part of your projects? and/or 13. You suggested that more guidance on ___ would be useful. Can you explain how you would see this affecting your organization’s ability to do more effective advocacy? 14. Financial sustainability appears to be a challenge for most; how do you view the prospects for diversification, if not sustainability after this grant? 15. Overall how you think you have improved as an organization due to the PCBG? Follow up: What have you learned? Lessons Learned and Looking Ahead (EQ1-4) 16. How does your organization see its role and future efforts in __ sector after this experience? Follow up: What skills can you apply from PCBG to address this? Follow up: anticipate doing more advocacy-focused work vs. service delivery/other? 17. What is your vision for the possibilities of advocacy work in your sector? Follow up: What is possible to accomplish in relation to political and policy context? Follow up: What needs to be done next by CSOs? By donors? 18. What advice would you give to USAID and PCBG as they consider how to best use the remaining time of the activity? 19. What advice would you give to USAID as the look beyond this project to support similar future efforts to support CSOs in making the government accountable? USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 45 External Stakeholders Guide Date Position and Organization Introduction and purpose of the interview as part of the evaluation Introduction 1. Tell us about how you know/cooperation with the PCBG project? Performance Implementation (EQ1,2) 2. What are your general impressions of how the project is going? Follow up: Does the project have visibility? Follow up: Is there any particular advocacy action that stands out? 3. To what extent do you see the CSOs either seizing the opportunity or dealing with the current policy context in their sector? Follow up: Which sectors have the most policy opportunities for advocacy? Follow up: Which sectors have the most need to for advocacy? Advocacy and Organizational Development (EQ 1-3) 4. What is your impression of the capacity building needs of the CSOs cooperating with PCBG? Follow up: Do you notice any improvements in their advocacy efforts? Follow up: How does this fit in relation to other CSOs working in their respective sectors? 5. To what extent do you see this programming complementing current or early efforts in advocacy and OD assistance? Follow up: To what extent is there a set of ‘mid-level’ CSOs present in MZ? Follow up: Are donor trainings tailored to this segment or more general in focus? 6. A key OD need identified is a focus on financial sustainability. From your experience what is realistic to expect in relation to CSOs being able to diversify funding in the current environment? Lessons Learned and Looking Ahead (EQ1-4) 7. How would you assess the overall advocacy and policy environment today in Mozambique? Follow up: Where and what is possible to realistically achieve? Follow up: What is your experience/impression of how to approach government actors in the different sectors? 8. How would you assess the donor environment for this type of support? Follow up: Impression of level of donor coordination Follow up: Impression of donor future priorities 9. What advice would you give to PCBG and USAID as they consider the remaining two years of the project? 10. What further advice would you give to USAID when considering future follow-on programming? 46 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX D: LIST OF PERSONS INTERVIEWED PCBG SURVEY # Name Organization Position Group 1 Sousa Camanguira TV Surdo Moçambique Executive Director PCBG Subgrantee 2 Sheila Rafi LIVANINGO Executive Director PCBG Subgrantee 3 Avelina Pascoal LIVANINGO Project Assistant PCBG Subgrantee 4 Pascoal N. Mapilele MAHLAHLE Project Manager PCBG Subgrantee 5 Emma Cardoso CESC Program Manager PCBG Subgrantee 6 Luís Palmeirim FRUTICAD Former Project Manager PCBG Subgrantee 7 Camilo Nhancale KUWUKA JDA President / Chair PCBG Subgrantee 8 Pedro Mazivila MEPT Project Officer PCBG Subgrantee 9 Joaquim Uate NTUMBULUKO Project Manager PCBG Subgrantee 10 Denise Nicolau BIOFUND Project Coordinator PCBG Subgrantee 11 Hermenegildo Manuel CESC Project Manager PCBG Subgrantee 12 Jordão M. Junior AENA Director PCBG Subgrantee 13 Isabel Bufalo AMDEC Project Manager PCBG Subgrantee 14 Marcelino Lampião ORPHAD Program Director Unsuccessful applicant 15 Isidro Alberto Joaquim UATAF Director Unsuccessful applicant PCBG KIIS # Name Organization Position Group 1 Abdul Sacoor EUROSIS Consultant External Stakeholders 2 Fernanda Farinha Independent Consultant Consultant External Stakeholders 3 Adriano Nuvunga Associação Desenvolvimento e Sociedade (ADS) President External Stakeholders 4 Nilla Ingstrop Embassy of Sweden (AGIR) Senior Program Manager External Stakeholders 5 De Jesus Alfredo Embassy of Sweden (AGIR) Program Officer External Stakeholders 6 Charlotte Cerf CPI /PCBG Current COP CPI /PCBG 7 Ana Paula Bande CPI /PCBG OD Specialist CPI /PCBG 8 Enoque Neri Muabsa CPI /PCBG M&E Specialist CPI /PCBG 9 Franco Wandabwa CPI /PCBG Advocacy Specialist CPI /PCBG USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT | 47 # Name Organization Position Group 10 Lucília Rosina Ferreira CPI /PCBG HR Manager CPI /PCBG 11 Silja Paasilima CPI /PCBG Past COP CPI /PCBG 12 Mary Kagunyi-Levasseur CPI /PCBG Project Coordinator CPI /PCBG 13 Kelly Jones CPI /PCBG Interim COP/Senior Advisor CPI /PCBG 14 Isabel Francisco da Silva CPI /PCBG Grants Officer CPI /PCBG 15 Utungo Mário Bila CPI /PCBG Accountant CPI /PCBG 16 Hermenegildo CESC Project Manager Subgrantee 17 Emma Cardoso CESC Program Manager Subgrantee 18 Camilo Nhacale KUWUKA Director Subgrantee 19 Gilda Jossias AMDEC Director Subgrantee 20 Sérgio L. Albuquerque TV SURDO Project Coordinator Subgrantee 21 Abel das Neves MEPT Project Coordinator Subgrantee 22 Isabel Francisco da Silva MEPT Executive Secretary Subgrantee 23 Pedro Mazivila MEPT Project Officer Subgrantee 24 Denise Nicolau BIOFUND Project Coordinator Subgrantee 25 Sean Nazerali BIOFUND Technical Advisor Subgrantee 26 Jason Smith USAID AOR for PCBG USAID 27 Dana Beegun USAID Former AOR for PCBG USAID 28 Susan Jay USAID Led PCBG design USAID 29 Olivia Gilmore USAID Environment Officer USAID 30 Maria Branquinho USAID Health Officer USAID 48 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX E: LIST OF DOCUMENTS REVIEWED PCBG PROGRAM DOCUMENTS • PCBG Basic Award and SOW • Mozambique Workplan Narrative final - year 1 • Revised PCBG FY18 Workplan Final • MELP March 2016-September 2017 • Work Plan IP PCBG Mozambique final • PCBG RF.pptx • Grantee Advocacy Tracking Matrix Final • PCBG FY17 Q2 Quarterly Report • PCBG FY17 Q3 Quarterly Report • PCBGFY17 Q4 and Annual Report • PCBG FY18 Q1 Quarterly Report • PCBG FY18 Q3 Quarterly Report • PCBG FY18 Q3 Quarterly Report • PCBG FY18 Q4 and Annual Report • 2016 Assessment of Civil Society Organizations in Mozambique SUBGRANTEE DOCS • CSO Subgrantee advocacy project proposals • CSO OD Action Plans • CSO Subgrantee MEL Plans • Subgrantee project reports USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION DRAFT REPORT | 49 ANNEX F: FINDINGS, CONCLUSIONS AND RECOMMENDATIONS MATRIX Focus Area/ Question Findings Conclusions Recommendations Question 1: To what extent has the PCBG Activity achieved anticipated advocacy results with the organizations to date? Why and why not? What was Achieved? What PCBG secondary data say… F1.1. ET review of the PCBG Indicators Performance Table documents and PCBG reports note that there have been three policy-level change results documented by PCBG attributed to two of the subgrantees (Biofund and AENA). F1.2. ET review of the PCBG Indicators Performance Table indicates that PCBG is meeting or surpassing 7 of 10 advocacy indicator targets for the Activity. However, PCBG KIIs reveal that the table primarily covers outputs rather than outcomes, that it has some duplicative indicators, and that it highlights some sectors more than others (health, climate). F1.3. ET review of a set of subgrantee M&E matrices and subgrantee and PCBG KIIs finds that subgrantee M&E plans are not aligned with PCBG indicators. F1.4. Five of the six subgrantee KIIs reveal that subgrantee advocacy results already achieved are not captured in their M&E plans. (For example: shift in policy actor mindset). F1.5. PCBG KIIs and document review reveal that PCBG has not had a system in place to systematically and comprehensively track subgrantee advocacy progress. What evaluation primary data say... F1.6. Subgrantee KIIs describe reaching part or most of at least one of each subgrantee’s two advocacy objectives and describe adapting subgrants as necessary to achieve this. (five of six describe this) F1.7. Three of six subgrantee KIIs point to a number of subgrants that have resulted in a changed mindset of key policy actors on issues. C.1 PCBG has not engaged in a systematic monitoring, evaluation and learning process that clearly shows the actual advocacy results achieved to date. The MEL system is partial and lacks the relevant industry standards for tracking advocacy policy results. (F1.2-F1.5) C.2 High-level results in conservation and extractive industries can be linked to the Activity even as for at least the conservation advocacy achievements it seems like that the Activity only played a partial role given the CSO’s earlier and ongoing efforts. Anecdotally other advocacy results appear to be in the process of achievement and subgrantees appear committed to their advocacy work. However, most results are not being captured or recognized by the current monitoring system. (F1.1; F1.4-F1.8; F1.13) C3. Modest advocacy results to date are partially due to subgrantee implementation challenges. Most need more time due to ambitious projects and the realities of doing such advocacy work on both local and national levels in Mozambique. Use of the research component is also not What can be done until the end of the PCBG? 1. PCBG should engage with subgrantees to map out actual project activities, results and timelines to understand how projects may have shifted and adapted from proposal design, and to obtain a clear understanding of subgrantee efforts to date and the challenges subgrantee projects still face (including research challenges) (C1 and C2) 2. PCBG should use the suggested mapping to bring together existing data on subgrantee projects and advocacy results in a unified and comprehensive system. It could also be coded by PCBG to address a number of process-based advocacy indicators, which could be added on to the PCBG Activity level system. This would provide more complete information on progress. (C1 and C2) 3. PCBG should consider a more comprehensive ‘redo’ of some subgrantee M&E plans aligned with process-based indicators 50 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION DRAFT REPORT USAID.GOV Focus Area/ Question Findings Conclusions Recommendations F1.8. Three of six subgrantee KIIs describe gaining recognition as a policy actor on an issue or reinforcing their role through their project efforts. F1.9. Subgrantee KIIs show that, in many cases, it is not clear how the research component has been used in advocacy (not done, part of earlier research, etc.). F1.10. Four of the six CSOs interviewed anticipate needing more time to achieve the second advocacy objective than the current timeframe of their subgrants. Different reasons were given (staggered engagement between local/national legislative calendar; things take more time) F1.11. All subgrantees claim to have plans or intentions to continue working on the advocacy issues after the PCBG subgrant. Why F1.12. KIIs from all informant groups link the late start of subgrantee awards (FY17Q4) to CPI unfamiliarity with local context and CPI slow adaptability. F1.13. ET review documents that the two subgrantees with policy level change results (Biofund 2, AENA 1) were the first grants signed. Biofund KII describes how PCBG funding support came at the right time to fund a portion of the inputs into the conservation law of their larger conservation programming efforts. F1.14. While CSOs give PCBG marks for a transparent concept and proposal process, subgrantee survey (at least 8 CSOs) and KIIs (3 CSOs emphasized) indicate that PCBG was inconsistent in its feedback and guidance to subgrantees during the proposal development and refinement process. F1.15. PCBG and subgrantee KIIs acknowledge that some subgrantee project proposals were edited by PCBG and translated in a manner that changed important parts of the projects. clear in whether it has helped or hindered results to date. (F1.9- F1.10, F1.13, F1.16) C.4. The low level of PCBG technical advocacy assistance also has likely limited subgrantee results to date. PCBG provided insufficient guidance to address these challenges. It also did not sufficiently frontload support efforts, such as advocacy mapping, M&E (Monitoring & Evaluation) plan finalization or advocacy training and support that could have provided assistance. F1.18 – F1.21, F1.24) C.5 Nor has PCBG provided much outreach and facilitation support. Probably due to CPI’s inexperience and low visibility in the country, PCBG has not provided much value added in building alliances with other policy actors beyond that which the CSOs are individually already doing. (F1.22- F1.24) C.6 PCBG’s general management of the process has affected results to date. PCBG’s slow start, compounded by the long grant proposal and review process left subgrantees frustrated and with projects that, in some cases, they needed to adapt to make workable. Subgrantee efforts were further frustrated by grant for subgrantees that are in early stages of implementation (potentially three of them) and which run through December 2019 as a way to begin to track more comprehensively advocacy processes. (C1 and C2) 4. PCBG should prioritize advocacy training and advisory assistance to subgrantees to introduce and reinforce this advocacy approach, including how to use/design evidence and research effectively. All subgrantees could benefit from receiving guidance in this area, even those with grants ending in the next three months. (C3 and 4) 5. PCBG should consider increasing its advocacy TA capacities by having its incoming advocacy specialist collaborate with an advocacy process expert who could provide guidance on crafting advocacy process strategies and MEL. Alternatively, or additionally, this expert could be directly available to subgrantees. (C3 and 4) 6. As is already in progress, PCBG should review its grant management practices and support mechanisms to USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION DRAFT REPORT | 51 Focus Area/ Question Findings Conclusions Recommendations F1.16. KIIs with subgrantees highlight the fact that subgrantee project implementation targeted at both local and national levels were taking longer than anticipated, and with different sequencing than anticipated. F1.17. PCBG and Subgrantee KIIs reinforce the impression reported in the subgrantee survey that subgrantees have spent a more than normal amount of time on grant management due to rigid reporting requirements, confusion on in-kind contributions as well as lack of subgrantee experience, F1.18. ET desk review and subgrantee KIIs reveal that Advocacy Strategy Mapping by sector was undertaken as late as six months after subgrantee awards. Revised M&E plans were completed approximately six months after the projects began. F1.19. Subgrantee survey and KIIs indicate that most subgrantees have received only basic strategic advocacy and M&E assistance and that this remains a need identified by subgrantees. F1.20. Subgrantee KIIs identify too few PCBG staff and lack of sector and advocacy expertise levels as key constraints to the Activity being able to provide adequate advocacy support. F1.21. USAID and PCBG KIIs point to the difficulties of hiring PCBG staff in a timely manner. High staff turnover throughout PCBG was also reported. F1.22. Subgrantee KIIs and survey data reveal that PCBG has done little to reinforce subgrantee coalition and alliance building to date. In one case, rather the opposite occurred by not allowing subgrantees to use projects to fund CSO alliances. F1.23. Subgrantee KIIs describe varying levels of Mozambique government receptivity and capacities across the six sectors in relation to their advocacy work. F1.24. Subgrantees describe supporting PCBG staff to better management reporting, which required a steep learning curve for some and for all took significant time away from subgrant efforts. (F1.12; F1.14; F1.15; F1.17; F1.21) subgrantees and ensure adequate staffing and systems in place for the support needed. (C6) 7. PCBG should prioritize where and how it can add value to subgrantees in networking efforts and focus on these areas during the Activity remainder. (C5) 52 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION DRAFT REPORT USAID.GOV Focus Area/ Question Findings Conclusions Recommendations understand sector specificities and advocacy approaches rather than have this support from PCBG. Question 2: To what extent has the PCBG Activity achieved anticipated organizational capacity advances of the organizations to date? Why and why not? What the PCBG secondary data say F2.1. ET review of the PCBG Indicators Performance Table progress in achieving OD results notes that most of seven OD-focused indicators do not have reporting or targets for FY17 or FY18. One indicator (#12) covers all six areas of the PCBG Action Plan, with no reporting to date. The other six indicators cover specific sector areas or duplicate parts of the #12 composite indicator. F2.2. PCBG KIIs acknowledge that there is currently no comprehensive system in place to track subgrantee OD Action Plans and results. What the evaluation primary data says F2.3. PCBG documentation indicates that 9 of 11 subgrantees had OD Action Plans approved. Subgrantee KIIs confirm that they have started working on their plans. However, none began prior to FY18Q2. F2.4. Subgrantee survey data reveal that 10 CSOs overall identifying a medium to high need for further OD support. F2.5. Subgrantee survey and KIIs identify financial sustainability, gender inclusion issues, and M&E assistance as areas that specifically require follow-on support after initial training or introductions. F2.6. Subgrantee surveys and KIIs point to high interest across all CSO experience levels in completing what they consider as realistic OD Action Plans and to tailor them to their needs. (ex: one org has taken the opportunity to use OD to advance their program-level M&E). Why? Considered useful and needed: C.1 The level of PCBG OD progress is unclear, given the lack of systematic data to date and the early stage implementation efforts. However, the data suggest that subgrantees are making use of the OD opportunities afforded, a promising sign. Nevertheless, more TA targeted to specialized areas, such as financial sustainability, M&E assistance, and gender inclusion would benefit the CSOs greatly. F2.1-F2.6) C.2 PCBG implementation of the OD component has been a constraining factor in OD progress to date. PCBG appears to have captured the interest of subgrantees at the beginning, but implementation since this start has been slow and somewhat bumpy due to internal CPI management and budgeting for the OD component. Challenges in identifying appropriate advisory consultants for specific organizations have also been a factor. This is likely to continue to be challenging, especially for the more specialized areas of financial sustainability and gender inclusion F2.3-F2.7, F2.8-F2.11) 1. PCBG should endeavor to harmonize an internal subgrantee OD Action Plan tracking system that includes the six areas of OD focus covered in the plans. (C1) 2. PCBG should consider unpacking Activity-level indicator #12 into the six areas of focus for a clearer Activity￾level tracking option. (C1) 3. PCBG should prioritize identifying a strong pool of experts available to work with PCBG grantees. Financial sustainability and gender specialists are especially high priorities. (C21and C2) 4. PCBG should consider the extent to which it can shift funding allocations to allow for increased options for PCBG to offer OD directly and not through the subgrantee budgets. (C2) USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION DRAFT REPORT | 53 Focus Area/ Question Findings Conclusions Recommendations F2.7. Subgrantee survey and KIIs highlight that PCBG conducted the OD assessment process in an interactive manner and actively made themselves available to subgrantees on OD issues. Slow in implementation and progress F2.8. PCBG KIIs identify that OD Action Plan approval required two levels of CPI review and approval, which often took 3-6 months. F2.9. Subgrantee survey and KIIs indicate that PCBG was unwilling to use results of similar OD assessments already done for some subgrantees. F2.10. PCBG KIIs note that the PCBG operating budget for OD was insufficient to allow some planned OD activities. F2.11. PCBG and subgrantee KIIs indicate a number of disappointing subgrantee experiences with external consultant quality/outputs. Question 3: To what extent did the design and structure (as amended during implementati on) of the Activity help and hinder achievement of results to date? Design and Funding F3.1. USAID KIIs describe the Activity design process as being led by multiple actors at different phases of design conceptualization and implementation. F3.2. USAID KIIs note that the pre-competed LWA funding mechanism selected to fund PCBG did not allow an additional competitive process to choose the implementing partner. F3.3. USAID KIIs describe that, during the design process, advocacy objectives were intentionally broad-based in order to incorporate sector-specific advocacy interests and AID’s impressions of advocacy needs. F3.4. USAID KIIs indicate that the level of funding for the Activity was modest, given the scope and combined advocacy/OD focus. F3.5. ET review of the PCBG Performance Indicators Table finds that there are two indicators specific for climate C1. Important aspects of the PCBG design and structure hindered the PCBG Activity. The Activity was designed to work intensively on advocacy with a broad set of CSOs. Yet the funding levels secured and the IP mechanism used limited the ability of USAID to procure an IP with the functional and country-specific experience that could realize this vision. (F3.1-F.3.3) C2. The multi-sector structure of the Activity presented significant, budgetary, technical and management challenges that burdened the IP, may have distorted local CSO ownership of USAID future efforts 1. Critically examine whether funding mechanism, funding level, and Activity specifications are aligned in order to provide a basis for optimal outcomes. This includes ensuring sufficient procurement process time to be able to use the proper IP mechanism. (C1) 2. If multi-office funding, then clear and shared Activity objectives need to be designed between the respective offices which align with respective Development Objectives and which include a clear understanding of design and 54 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION DRAFT REPORT USAID.GOV Focus Area/ Question Findings Conclusions Recommendations and two for health (so 4 of 16 in total) to align with sector specific development objectives, but. USAID was not clear not whether these indicators have helped it reach higher development objectives. F3.6. PCBG KIIs note the useful USAID technical inputs into the call for proposals SOW and the proposal review grant committees, although note that the later engagement lengthened the proposal review process. F3.7. PCGB and USAID KIIs indicate that USAID technical inputs and engagement have been minimal during subgrantee implementation. F3.8. PCBG KIIs relates that despite USAID flexibility in financial reporting, the multi-sector reporting took significant time and effort on both HQ and field office levels. F3.9. USAID and PCBG KIIs detail early problems with being able to sequence funding to subgrantees by sector and to pace the PCBG burn rate in relation to the committed funds. F3.10. Subgrantee survey and KIIs find that at least three subgrantees needed to go back and change their proposals and budgets due to changes in the allowed funding ceilings. F3.11. Several subgrantee KIIs and subgrantee survey feedback point out that proposal requirements caused them to work in different regions than they would prefer, while USAID described provided these guidelines to complement their other sector programming. Added Election Component (FY17Q4) F3.12. PCBG and USAID KIIs point out that the COP at the time (who had election experience) focused most of the time (from late summer 2017 - summer 2018) on the election’s component (60-70%) with core Activity competencies related to getting the subgrants started the advocacy efforts, and provided little clear benefit for USAID Technical Offices program efforts, (F3.5-F3.11) C3. The addition of the elections component resulted in two positive unexpected byproducts. But its overwhelming effect was negative, The introduction of the election component further distorted implementation efforts. The IP was not able to adapt and take on the extra programming without slighting its already challenging core advocacy and OD implementation efforts. (F3.12- F3.16). implementation engagement and responsibilities. (C2) 3. USAID should consider how to allow design of sector￾specific funding to be more consultative with local partners in identifying needs and priorities to ensure feasibility and ownership. (C2) 4. PCBG should consider carefully how to manage new election work which may include building ‘firewalls’ into the PCBG Activity (sufficient staffing and management plan) to ensure that anticipated election efforts do not overwhelm its core advocacy efforts. (C3) USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION DRAFT REPORT | 55 Focus Area/ Question Findings Conclusions Recommendations covered by the rest of staff. F3.13. PCBG KIIs further indicate that regardless of hiring two election-focused staff, other core project staff time was spent on OD, M&E, advocacy, and grant management of the election subgrantees. F3.14. PCBG staff described the management of the internal technical team and advocacy subgrantees during much of this time as uncoordinated and lacking skills to oversee the many ongoing efforts. F3.15. PCBG and one subgrantee describe how the CSO’s work on citizen manifestos was used also in election work. F3.16. External and PCBG KIIs highlight the visibility PCBG gained in the donor community and civil society from coordinating elections. F3.17. PCBG and USAID KIIs indicate that they plan to work together on elections in the coming year. Focus Area/ Question Lessons Learned 56 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION DRAFT REPORT USAID.GOV Focus Area/ Question Lessons Learned Question 4: What lessons can be learned from the PCBG experience to date for crafting future USAID civil society programming that promotes self-reliance and government transparency / accountability? Fit for purpose 1. Combining OD and advocacy purposes requires an implementing partner familiar with country context and able and ready to carry out intensive work. (EQ1 and EQ2) 2. Advocacy is a process. Advocacy Activities need to be designed and managed to capture advocacy influence process. (EQ1) Realistic design assumptions 3. CSO advocacy capacity must be aligned with advocacy influence expectations. CSOs with lower levels of advocacy capacity require longer-term engagement, and realistic expectation to what can be achieved. (EQ1 and EQ3) 4. Invest the necessary inputs and strategy to do both national and local-level advocacy. The advocacy effort required for doing national and local synergized advocacy is greater than simply including local and national components in a project. (EQ1 and 3) 5. Advocacy programming requires mapping out advocacy demand at the design stage. Advocacy programming requires a critical review of sector mapping and government receptivity to the specific advocacy focus in order to craft and implement realistic programming. (EQ1 and 3) Sequencing matters 6. First provide OD and then advocacy technical assistance inputs. If OD and advocacy are together in a grant mechanism, duration of the grant needs to be longer and OD needs to be frontloaded. (EQ1 and EQ2) 7. Advocacy strategy and MEL needs to be a design process: Early implementation emphasis is needed on coordinating the design of advocacy objective, strategy and MEL in order to have focused and realistic projects that can adapt/course/correct). (EQ1) Maximizing partner relevancy and ownership 8. OD benefits across all experience levels are possible if there is balanced guidance and flexibility in designing OD tailored plans. (EQ2 and EQ3) 9. Achieving a partner-driven Activity requires a nuanced balance of issue priorities and a degree of flexibility. (EQ 1 and EQ3) USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION DRAFT REPORT | 57 ANNEX G: PCBG ADVOCACY SUBGRANTS # Sector Subgrantee Subgrant Objectives 1 Extractive Industry AENA • Objective 1: By the end of 2018, the regulatory ruling number 1/MPD-MF/2013 is reformulated by the government of Mozambique to ensure citizens effectively benefit from the projects resulting from mineral and petroleum resource exploitation revenues. • Objective 2: By 2019, citizens from the provinces of Cabo Delgado and Zambézia are engaged in the decision-making processes and have share recommendations on improving transparency in the implementation of projects resulting from extractive sector revenues. • Objective 3: By 2019, the technical and institutional capacity of AENA to lobby for the reformulation of the regulatory ruling regulating the process of channeling revenues resulting from the extractive sector is strengthened. 2 Education AMDEC • Objective 1: To identify policy gaps in the General Regulation of Basic Education (REGEB) guidelines on the role of School Councils and School President in promoting inclusion and accountability in management of primary schools • Objective 2: To advocate for the revision of the REGEB guidelines to clearly define the profile of the School Council President and increase participation of the School Councils in decision-making and management of primary schools by 2019 • Objective 3: To strengthen the capacity of AMDEC to effectively manage and implement education advocacy interventions. 3 Biodiversity conservation BIOFUND • Objective 1: Biofund’ s technical and institutional capacity is strengthened to effectively conduct advocacy and engage with government and private sector. • Objective 2: By the end of 2018, BIOFUND strengthened the participation of civil society and local communities in the process of mechanisms for biodiversity offsets at national and local level. • Objective 3: By the end of 2018, BIOFUND ensured the correct incorporation of the concepts of biodiversity offsets into legal provisions. 58 | PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION DRAFT REPORT USAID.GOV # Sector Subgrantee Subgrant Objectives 4 Transparent and accountable government CESC • Objective 1: Improve the participation of citizens and CSOs in the promotion of accountability and the provision of quality public services in the municipalities of Nampula and Maputo. • Objective 2: Increase the participation of citizens in the defense of the decentralization of key functions that promote the best provision of services by the municipalities. • Objective 3: Strengthen the organizational capacity of CESC to implement policies and advocacy. 5 Climate change FRUTICAD • Objective 1: Build the capacity of local and community leaders to address effects of climate change deforestation in accordance with government legislation in Metuge district. • Objective 2: Advocate for the implementation of the Forest and Wildlife Law that promotes climate change adaptation in Metuge district. • Objective 3: Reinforce the organizational capacity to carry out climate change advocacy. 6 Extractive industry KUWUKA JDA • Objective 1: Advocate for the implementation of EITI Standard Requirements that promote transparency and delivery in governance of the mining sector in Mozambique. • Objective 2: Advocate for broader inclusion of key stakeholders to include the 5th Commission Parliamentarians, Regulatory Institutions (INP and INAM) and Public Companies of the Mining and Hydrocarbons Sector (ENH and ENEM) in the ITIEM CC. • Objective 3: Improved Institutional Capacity and Guiding Instruments for Operation and Internal Governance of KUWUKA JDA. 7 Climate change LIVANINGO • Objective 1: Government influenced for inclusion and recognition of climate refugees in the National Adaptation Plan (NAPs) and Local Adaptation Plans on Climate Change (PLAs). • Objective 2: Analyzed the investment in programs of climate change adaptation and mitigation in Mozambique. • Objective 3: Improved organizational capacity of Livaningo. USAID.GOV PCBG ACTIVITY MIDTERM PERFORMANCE EVALUATION DRAFT REPORT | 59 # Sector Subgrantee Subgrant Objectives 8 Transparent and accountable goveranance MAHLAHLE • Objective 1: Advocate for District Governments of Massinga, Funhalouro and Inhassoro to adopt accountability mechanisms to efficiently and effectively inform communities about the plan and budget implementation through District civil society platforms. • Objective 2: Support the citizens’ involvement in monitoring the implementation of the economic and social plan and budget in the districts of Massinga, Funhalouro and Inhassoro of the Inhambane province. • Objective 3: Strengthen MAHLAHLE’s institutional capacity to conduct advocacy initiatives. 9 Education MEPT • Objective 1. To conduct policy engagement with policy makers on the equitable student investment between provinces that promotes quality basic education in Mozambique. • Objective 2. To increase public awareness and engagement on participatory budgeting for basic education. • Objective 3. To strengthen the technical capacities of MEPT Secretariat and its affiliate members to conduct policy advocacy and ensure inclusive, transparent and accountable governance in the education sector.