Evaluation of the Zimbabwe American Development Foundation Project November 15, 2001 (updated March 4, 2002) Prepared by: Tonya Himelfarb Purchase Order No. 613-O-00-02-00001-00 Unrestricted Version TABLE OF CONTENTS I. Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 II. Evaluation Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 III. Evaluation Process and Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 IV. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 V. USAID’s Cooperative Agreement with Pact . . . . . . . . . . . . . . . . . . . . . . . 4 A. The Foundation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 B. The Modified Program Description . . . . . . . . . . . . . . . . . . . . . . . . 8 VI. Principal Findings – Pact’s Performance . . . . . . . . . . . . . . . . . . . . . . . . . . 9 A. Establishment of a ZADF/Pact Office . . . . . . . . . . . . . . . . . . . . . . 9 B. Development/Implementation of a Grants Management System . .11 C. Provision of Technical Assistance and Mentoring . . . . . . . . . . . . . 13 D. Workshops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 E. Networking/Coalition Building. . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 F. Management of the Advocacy Index Process . . . . . . . . . . . . . . . . .17 G. Workplans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 H. Results Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 I. Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 VII. Principal Findings – CSOs' Performance . . . . . . . . . . . . . . . . . . . . . . . . . 20 A. Performance on the Advocacy Index. . . . . . . . . . . . . . . . . . . . . . . 20 B. Contribution of ZADF/Pact to CSO Progress . . . . . . . . . . . . . . . . 20 C. CSOs' Perception of Parliament . . . . . . . . . . . . . . . . . . . . . . . . . . 22 D. ZADF/Pact Timeframe. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 VIII. Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 IX. Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 A. For Phase I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 B. For Phase II. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Annex A: Evaluator’s Scope of Work Annex B: People/Organizations Interviewed Annex C: Questionnaires Annex D: USAID/Zimbabwe’s Results Framework, 2000-2005 Annex E: Indicators ACRONYMS CA Cooperative Agreement CSO Civil Society Organization CSP Country Strategic Plan CTO Cognizant Technical Officer D/G Democracy and Governance FY Fiscal Year IQC Indefinite Quantity Contract IR Intermediate Result MP Member of Parliament NPA Non-Project Assistance OTI (USAID’s) Office of Transition Initiatives PD21 Policy Determination 21: Guidelines for Endowments Financed with Appropriated Funds REI Request for Expressions of Interest SO Strategic Objective SpO Special Objective SUNY State University of New York UI Urban Institute USAID United States Agency for International Development USAID/W United States Agency for International Development in Washington ZADF Zimbabwe American Development Foundation Project ZIMDES Zimbabwe Democracy Strengthening Project ACKNOWLEDGEMENTS I would like to thank the many people that took time out from their busy days to be interviewed, to answer questions, to provide me with reports and documents, and to otherwise make certain that I had the information required to complete this evaluation. I sincerely hope that I have adequately reflected the thoughts and sentiments that were conveyed to me and that the information and recommendations contained in this report will contribute to a stronger and more successful ZADF program. ZADF/Pact Evaluation 1 I. EXECUTIVE SUMMARY The Zimbabwe American Development Foundation (ZADF) project is USAID/Zimbabwe’s primary means of achieving the first intermediate result under its participation strategic objective: improved civil society organizations’ representation of citizen’s interests at the national level.” Phase I of ZADF is being implemented under a cooperative agreement with Pact which ends on September 30, 2002. This evaluation was requested by USAID to assess impact to date and to make recommendations for the planned Phase II of the ZADF program, from October 2002 through September 2005. The evaluation concluded that Phase I of the ZADF/Pact program has strengthened the capacity of targeted civil society organizations to advocate with Parliament, as measured against the advocacy index. Pact developed a sound grants management system which not only efficiently awards sub-grants, but which also contributes to capacity building and institutional strengthening within the sub-grantees. Similarly, Pact’s management of the advocacy index process, which was designed and introduced by USAID, has been excellent, resulting in the advocacy index being used as a multi-purpose tool by (sub) grantees. The workshops, training and technical assistance provided by Pact were also given high marks by the (sub) grantees. While there were some serious issues with early workshops not adequately building advocacy skills, Pact continues to take steps to address these issues and subsequent workshops have been more effective than earlier ones. The evaluation contains several recommendations which can be easily implemented during the last year of Pact’s cooperative agreement to improve performance, including increased focus on networking, increased emphasis on reaching the decision-makers within (sub) grantees and greater use of one-on-one technical assistance, rather than workshops and group training sessions. The evaluation confirms the need for USAID to proceed with implementation of Phase II of the ZADF program. Not only are both capacity building and advocacy long-term activities which require additional support under Phase II if interventions are to be sustainable, but also the political and economic crisis in Zimbabwe dramatically illustrate the urgent need for civil society to be able to make its voice heard in a constructive manner. The evaluation recommends that Phase II continue to focus on increasing participation and understanding of the advocacy process, rather than on changing specific legislation. It further recommends that USAID continue to work with the same group of CSOs supported under Phase I, to further deepen their ability to advocate with Parliament. At the same time, criteria need to be developed for determining when an organization should be phased out of the ZADF program. As CSOs leave the program, new ones could be added. During Phase II, ZADF should adopt a more holistic approach to working with the CSOs. There is a strong need for general organizational development assistance to ensure that advocacy becomes institutionalized within the (sub) grantees’ organizations. ZADF/Pact Evaluation 2 II. EVALUATION OBJECTIVE This evaluation was requested by USAID/Zimbabwe to assess the performance of one component of its democracy and governance program: the Zimbabwe American Development Foundation (ZADF) project which is being implemented by Pact under a cooperative agreement, as amended, for the period from September 29, 1998, to September 30, 2002. Specifically, the evaluation was requested to provide USAID with information and feedback on Pact’s performance, specific and broad achievements, the timeliness of rendered services, and the overall impact of the activity. The evaluator’s terms of reference are included as Annex A. The results of the evaluation will be used to guide USAID/Zimbabwe in any necessary changes to the current program as well as to inform the direction and content of the planned Phase II of USAID’s advocacy program, which will be carried out between October 2002 and September 2005. III. EVALUATION PROCESS AND METHODOLOGY This evaluation was carried out between October 18 and November 15, 2001, by Tonya Himelfarb, an independent consultant, at the request of USAID/Zimbabwe, under Purchase Order No. 613-O-00-02-00001-00. The evaluator arrived in country on October 17, 2001, and spent her first few days in Zimbabwe reviewing project documents and interviewing USAID staff. On Oct. 24, the evaluator met with the Pact Director and one of the Pact program officers. She then initiated a series of interviews with a random sample of civil society organizations (CSOs)1 receiving sub-grants and/or technical assistance under ZADF, Pact field staff, Pact headquarters staff, core partners2 , members of the advocacy index panel3 and ZADF resource people4 . In all, the evaluator interviewed seven CSOs, all six of the professional Pact field staff, the Pact president and Director of Program Operations, five USAID employees, six resource people, two core partners and one member of the advocacy index panel. Annex B contains a complete list of people and organizations 1 Pact is providing services to both grantees (under the ZIMDES project) and sub-grantees (under ZADF). For ease, the evaluator will use the term “the CSOs” or (sub) grantees to include both. Unless otherwise indicated, this reference is only to those of the 16 CSOs receiving USAID funding through the two programs that were actually interviewed by the evaluator. 2 During the development of its democracy and governance strategy, USAID established an informal advisory group of ten eminent, knowledgeable Zimbabweans to provide input and guidance. This was so successful that the group was later formalized as the “core partners” group by USAID and civil society. Core partners meet frequently with USAID to provide guidance to the USAID mission, to ensure that USAID remains informed of civil society’s views on the political and economic climate, to ground truth USAID’s assumptions, and to serve as a point of interface between Zimbabweans and USAID. 3 The advocacy index panel is made up of a group of six Zimbabweans that assist USAID and ZADF/Pact in reviewing CSO progress in advocacy as measured on the advocacy index. 4 Resource people are those who have provided services to Pact, usually as consultants, trainers or by presenting topics at workshops. ZADF/Pact Evaluation 3 consulted. Interviews were conducted using the questionnaires found in Annex C of this report. Questions were open-ended and interviewees were encouraged to expand on the topics and provide as much information as was deemed relevant to the evaluation. On November 8, 2001, the evaluator presented initial findings to USAID for comment. A November 14 debriefing with the Pact Director focused on the evaluation’s recommendations. This document incorporates feedback received in those sessions. IV. BACKGROUND The ZADF program was originally conceived of in 1996/97, when Zimbabwe had been identified by USAID as a country ready to graduate from USAID assistance. As envisioned, a foundation was to be established which would serve as a long term, sustainable mechanism for providing grants to civil society organizations after USAID closed its offices in 2003. In September 1998, Pact was competitively awarded a $2.5 million Cooperative Agreement (CA) to implement Phase I of this program. Under the initial terms of the cooperative agreement, the development of the methodology and means to establish the Foundation were left to be worked out during the term of Pact’s program, but before the Mission closed. During this time Pact was expected to recruit and train a Zimbabwean staff and Board of Directors and establish the Foundation as a Zimbabwean organization, in compliance with Zimbabwean law. It was anticipated that ZADF/Pact would initially make grants to CSOs with USAID funds, and later with funds from an external endowment that would include a $10 million dollar investment from USAID. (See Section V.A. of this evaluation for a more complete discussion of the Foundation.) Soon after the cooperative agreement with Pact was signed, Zimbabwe entered a period of political and economic turmoil. The plans for USAID close out were no longer realistic. Concurrently with the arrival of the Pact Director in Zimbabwe in January 1999, the Mission initiated the process of developing a crisis mitigation strategy to respond to the changing situation. The Pact Director was quick to understand the changing environment and to adjust flexibly to the new realities. Following the submission and approval of USAID/Zimbabwe’s new Country Strategic Plan for 1999- 2005, Pact submitted an amendment to the cooperative agreement to realign ZADF activities to better fit within USAID's new strategy. As set forth in the new Country Strategic Plan, the ZADF Project supported the “participation” strategic objective (SO), “Enhanced citizen participation in economic and political decision making.” This SO had been developed in close cooperation with the core partners group and was based on the premise that the lack of effective political and economic dialog between civil society organizations and the government was a key problem to achieving the mission goal of “supporting Zimbabwean’s access to greater and more equitable benefits from their nation’s social, political and economic development.” To build upon the democracy and governance Special Objective (SpO), which was approved in June 1998, and to encourage enhanced public/civil society dialog, a two-pronged, demand and supply, “participation” strategy focused on two fronts: first, ZADF/Pact Evaluation 4 building demand by strengthening the ability of CSOs to advocate in an effective manner on behalf of citizens, and second, strengthening the “supply” of democratic governance by building the capacity of selected government institutions to respond to this public input. The three intermediate results (IRs) constituted an integrated approach to strengthen citizen participation in the decision-making processes that affect their lives. IR1: Improved civil society organizations’ representation of citizens’ interests at the national and local levels IR2: More effective and accessible selected national government institutions IR3: Local authorities more capable and open to local citizen input The results framework in Annex D graphically displays the participation SO and how it fits within the mission’s overall strategy. Under its amended cooperative agreement, ZADF/Pact was to focus on achievement of IR1, with a focus on citizen representation at the national level. Separate agreements were entered into with the State University of New York (SUNY) to work with the Parliament (IR2) and the Urban Institute to focus on local government (IR3). The program to be implemented by Pact under the amended program description was viewed by both USAID and Pact as a modification and extension of the originally planned Phase I of USAID’s advocacy program. There was always an assumption that Phase II would be implemented to cover the period from October 2002 through September 2005, to coincide with the end date of USAID’s new country strategic plan. One question of this evaluation is whether Phase II should continue to be implemented by Pact or whether alternate implementing arrangements would be preferable. V. USAID’S COOPERATIVE AGREEMENT WITH PACT USAID’s involvement in ZADF can be divided into two overlapping periods. The first, between 1997 and 1999, was based on the establishment of a Zimbabwean foundation or endowment. The second, from 1999 to the present, has been guided by the terms of the modified program description in Pact’s cooperative agreement. These two periods are discussed in detail below. A. The Foundation In 1996, Zimbabwe was identified by USAID as one of several countries poised to “graduate” from USAID assistance. For Zimbabwe, graduation meant that bilateral programs would be phased out by 2003 and that any subsequent U.S. government foreign assistance would occur as part of regional or global efforts with a multinational focus. To prepare for graduation, USAID/Zimbabwe was requested to submit a close out plan for the period 1997-2003. During the development of the close out plan, a close out activity entitled, the Zimbabwe American Development Foundation (ZADF), was conceived of as a long-term, ZADF/Pact Evaluation 5 sustainable institutional mechanism that would enable a development relationship to continue between the United States and Zimbabwe after 2003. As envisioned, the Foundation would support, in perpetuity, non-governmental organizations actively engaged in increasing opportunities for participation in the private sector and political processes. The formation of the Foundation was expected to take four years, during which time an implementing organization would recruit and train a Zimbabwean staff and board of directors and establish the Foundation as a Zimbabwean organization, in compliance with Zimbabwean law. The April 1, 1997, “Proposal to Establish a Foundation and Endowment with Appropriated Funds” clearly set forth the rationale, strategy, planning and management considerations, and expected results of the Foundation. In April 1997, USAID/Zimbabwe presented its Country Strategic Plan for 1997-2003 to USAID/Washington. The plan included a new Special Objective (SpO), “Increased opportunities for participation in the private sector and political processes,” as a key element of the strategy and proposed ZADF as the key strategic partnership for achieving this SpO. The inclusion of the Foundation generated considerable discussion during the USAID/Washington review of the strategy. It was clear during these discussions that people had differing views on what the Foundation would be and do. Discussions were couched in terms of the political economy (which was never well defined) and the Foundation’s client was never clearly specified. In spite of reservations, there were strong political pressures for the Foundation to proceed and in May 1997, USAID/Washington approved the strategy, including the Foundation. Initially, it was expected that USAID/Zimbabwe would obligate to the Foundation the $5 million it had available in deobligated non-project assistance (NPA) funds in fiscal year (FY) 1997. This turned out to be unrealistic given the requirement for a Policy Determination (PD) 21 (“Guidelines: Endowments Financed with Appropriated Funds”) review, and USAID/Zimbabwe lost these funds. It was subsequently agreed that the ZADF program would be initiated in FY98, but that funding levels and timing for the proposed endowment would be subject to the PD21 review, which was expected to take place in FY99. On August 18, 1998, the USAID/Zimbabwe Mission Director signed an Action Memorandum approving the ZADF activity. Early in 1997, when USAID/Zimbabwe was discussing the close out strategy and securing approval for the Foundation within its own organization, externally it initiated a process for identifying potential partners. On May 10, 1997, a Request for Expressions of Interest (REI) for the Zimbabwe American Development Foundation was issued and several organizations responded before the closing date of April 10, 1997. In June, letters went out to selected organizations, inviting them to submit formal applications for ZADF. These were received in August 1997. During USAID discussions of the proposed foundation, democracy experts challenged plans to invest scarce democracy and governance (D/G) resources in an endowment when Zimbabwe had more immediate needs that would not be addressed by the medium- and long-term goals of the proposed endowment. USAID/Zimbabwe was advised that additional D/G resources could be forthcoming if the mission, jointly with the entire ZADF/Pact Evaluation 6 country team, were to develop a comprehensive D/G strategy. USAID/Zimbabwe, with Embassy input, therefore undertook a D/G sector assessment, which was presented and accepted in Washington in November 1997. Based on this assessment and other input, USAID/Zimbabwe staff then developed a D/G strategy, which, among other things, refined the original Special Objective to “increased opportunities for citizens’ participation in economic and political decision making.” On June 11, 1998, USAID/Washington approved the D/G strategy for the SpO. By this time, USAID/Zimbabwe was already aware of several significant issues, specific to ZADF, which made the establishment of this foundation even more complex than usual. These included: · Under Policy Determination 21, an organization must meet certain criteria before U.S. government funds can be used to endow. Given that there was still no organization in Zimbabwe, it was impossible to meet those criteria. Therefore, in order to endow the funds, USAID had the choice of either making a special request to Congress or first establishing an organization that was able to meet the PD21 criteria. · While the REI had required that applicants match USAID’s anticipated $10 million contribution to the endowment, no applicants met this requirement. Pact, with $100,000 in core funding, came closest. Potential partners felt that it was unrealistic to expect them to engage in fundraising for a matching contribution to the endowment when the foundation had not yet been established and USAID’s ability to commit funds to the endowment was still uncertain. · Unless the endowment was assured, establishment of a Zimbabwean organization was premature. Without the endowment, the organization would not be able to provide the grant-making services for which it was being created. · The impetus to continue with the creation of the endowment was mainly political. To democracy experts, it was clear that there were enormous opportunity costs to establishing an endowment versus actively using funds for D/G activities deemed critical as Zimbabwe entered a period of economic and political crisis in late 1997, which escalated in 1998. In spite of these concerns, the contracting process continued. In September 1997, the Regional Contracting Officer notified applicants that the procurement process would be delayed until new staff arrived at post and a democracy sector assessment and strategy were completed. This correspondence was followed by an October 1997 letter which explained that an “informational hold” on the Congressional Notification for ZADF had prevented USAID/Zimbabwe from making any financial commitments to ZADF in FY1998. However, late in 1997, revised proposals for ZADF were requested for submission in March 1998. In this submission, applicants were requested to divide their proposal into two sections, representing two phases of programming: Phase I for the creation and development of a grant-making institution and Phase II for the creation and development of an endowment. Negotiations with the prime applicant concluded in September 1998 and Cooperative Agreement No. 690-A-00-98-00252-00 was signed with Pact for implementation of Phase I of the ZADF activity. The cooperative ZADF/Pact Evaluation 7 agreement awarded Pact with $2.5 million through March 31, 20015 , during which time, Pact was to establish a Pact/ZADF office in Zimbabwe, establish a Zimbabwe-based organizing committee, recruit and train staff, including a Zimbabwean deputy team leader, implement a series of workshops and develop and test a grants management system. It was anticipated that the grant unit would be converted to a foundation under Phase II of the activity. Under the terms of the cooperative agreement, Pact was also to move forward on the documentation and conditions precedent to seeking PD 21 approval for the endowment fund. In recognition of difficulties anticipated in establishing the endowment, the CA included a statement that “If PD 21 approval or USAID’s decision to fund the endowment are not forthcoming, Pact will continue to manage and institutionalize the grant-making unit in furtherance of the purposes for which it was created, converting it into ZADF.” Concurrently with the conclusion of negotiations and the signing of the cooperative agreement, the political and economic deterioration in Zimbabwe began to accelerate. The USAID Country Strategic Plan for 1997-2003, that envisioned close out of the USAID program and development of the Foundation, was no longer realistic. In November 1998, USAID/Zimbabwe was asked to draft a crisis mitigation strategy to respond to the critical situation in the country. The new realities prompted USAID to take a harder look at what it could do to help consolidate democracy in Zimbabwe. After intensive consultations, the Country Strategic Plan 2000-2005 elevated democracy initiatives from a SpO, which focused on increased opportunities for citizens’ participation in economic and political decision making to a Strategic Objective (SO) that expected to enhance citizen participation in economic and political decision-making. This change of emphasis, coupled with a longer-term outlook, led to the conclusion that the Foundation was no longer the most effective means of achieving USAID’s objectives. Rather, the strategy expanded the depth and scope of the demand and supply equation by increasing the number of advocacy grantees at both the national and local levels and by building the capacity of numerous selected government institutions (Parliament, Local Government Authorities, and Land Reform institutions) to respond to this public input. In March 1999, USAID/W approved the new country strategy, effectively abrogating the 1997 closeout strategy, and in May 1999 formally reversed the closeout decision and increased funding for an enhanced USAID program in Zimbabwe. Pact responded by submitting to USAID in August 1999, a draft proposed revised program description that focused on (1) establishing a grants management system which would make sub-grants to Zimbabwean CSOs in support of “improved civil society organizations’ representation of citizens’ interests at the national level” and (2) providing technical assistance to sub￾grantees. Establishment of an endowment was not included in the revised program description. Following receipt of Pact’s proposed revised program description, USAID held numerous discussions with the Regional Contracting Officer and decided to 5 While the initial text of the cooperative agreement stated that the termination date was March 31, 2000, this was determined to be a typographical error and was corrected in Modification 1 to the CA to the intended date of March 31, 2001. ZADF/Pact Evaluation 8 incorporate two other revisions into the planned modification of the cooperative agreement. Namely, it decided to increase the program budget to $4,605,737 and extend the life of project to September 30, 2002. Following further discussions between USAID and Pact, final revisions to both the program description and the budget were completed by Pact in February, 2000 and USAID sent a non-funded Maard to the Regional Contracting Officer in March, 2000. Given that funds were not immediately available; the amendment formalizing these changes was not signed until June 1, 2000. The officially signed amendment specifically stated that “establishment of an endowment is not being pursued since the premise for considering an endowment (i.e. close out of USAID) no longer obtains.” With this amendment, all activities in support of the endowment formally ceased. At the time of this evaluation, USAID/Zimbabwe continues to believe that an endowment fund is neither the best use of USAID resources nor the most effective means of promoting democratic participation in Zimbabwe given the current political and economic crisis. There is no interest on the part of either USAID/Zimbabwe or Pact in reviving the endowment concept and USAID has no plans to pursue the idea under its current strategy. B. The Modified Program Description When the Pact Director arrived in Zimbabwe in January 1999, the political, economic and social conditions in the country were deteriorating and USAID was discussing the development of a new country strategic plan that might render the endowment/foundation he was meant to establish irrelevant. The Pact Director was responsive to the new internal and external environment and was ready to cooperate with USAID to revise ZADF goals and activities so that they better supported USAID’s revised strategy. As described above, in consultation with USAID, he developed and submitted a revised program description which serves as Modification No. 2 to the cooperative agreement. During this period, USAID made a decision to initially focus its “participation” strategic objective on demand and supply at the national level. For the supply side, the Parliament of Zimbabwe, which was undergoing a reform process, was identified as the government institution with the greatest potential for being able to respond to civil society. Under USAID’s new “Zimbabwe Democracy Strengthening” (ZIMDES) project, the State University of New York (SUNY) was selected to provide support to Parliament, to strengthen its effectiveness and accessibility to the public. Under Modification No. 2, ZADF activities were to complement ZIMDES activities by building the capacity of civil society organizations to advocate at the level of Parliament as an institution. In order to accelerate ZADF implementation during the interim period between the signing of the cooperative agreement with Pact and the arrival of the Pact Director in country and establishment of a ZADF/Pact office, in February 1999, USAID/Zimbabwe took advantage of the bilateral ZIMDES project to issue an RFA for the first round of advocacy sub-grantees. ZADF took this reality into consideration when modifying the ZADF/Pact Evaluation 9 program description and generously agreed to provide the same technical assistance to the ZIMDES grantees that was to be provided to the ZADF sub-grantees. Under its revised program description, Pact was therefore to: · Establish a ZADF/Pact office (this activity was essentially completed before the modification was signed) · Develop a grants management system and make sub-grants to Zimbabwean NGOs and CSOs in support of the “participation” strategic objective · Provide technical assistance and mentoring to both ZADF/Pact sub-grantees and ZIMDES grantees, specifically in the areas of financial management, monitoring and evaluation, advocacy and gender · Organize and manage workshops · Manage the advocacy index process · Submit annual workplans · Provide input into USAID’s results reporting. Pact’s impact/success was to be measured by three qualitative indicators, which were also the basis for USAID’s annual results reporting. These indicators and targets were6 : Indicator Baseline 1999 Target/ Actual 2000 Target/ Actual 2001 # of targeted CSOs that work at the national level that show improvement on the Advocacy Index 0 7/9 10/9 # of targeted CSOs that satisfy the test of advocacy improvement that represent women and marginalized communities/populations in their issues 0 4/5 6/6 CSO perceptions of valid engagement with Parliament on issues relevant to their area of concern 3 8/16 14/26 To facilitate communications between USAID and Pact and to ensure that Pact activities remained integrated into the mission strategy, Pact participated as a member of the mission’s expanded D/G team and become a member of USAID’s core partners group. VI. Principal Findings - Pact’s Performance During CY1999, while USAID was developing its new strategy and USAID and Pact were working to redefine ZADF’s role, Pact focused on setting up a field office in Harare, recruiting staff and developing a grants management system. The CY 1999 workplan also included one “outreach” activity in relation to the establishment of the Foundation. During the first half of CY 2000, while the CY2000 workplan and the revised program description for the cooperative agreement were being discussed and negotiated, Pact operated under a “gentleman’s agreement” with USAID. While Pact 6 Annex E contains a complete description of these indicators. ZADF/Pact Evaluation 10 carried out the CY 2000 activities agreed upon with USAID, these activities did not completely correspond to the program description in the original cooperative agreement, specifically with respect to the endowment. They were, however, considered appropriate under the Cooperative Agreement and coincided with the intent of Pact’s proposed revised program description. As of CY 2000, by mutual agreement between USAID and Pact, activities in support of the endowment were no longer pursued. A. Establishment of a ZADF/Pact Office During 1999, Pact established a field office led by a Director and assisted by a Finance Officer. Except for the Deputy Director, required staff were also recruited, including two program officers, one executive assistant, one finance director, one accountant, a grants management specialist (on a contract basis), and logistical support personnel. Position descriptions for each position were developed and, for the most part, the incumbents believe that their terms of reference adequately reflect the work they actually perform. Only one staff person mentioned that his responsibilities had expanded, but that his additional responsibilities had not yet been incorporated into his terms of reference. The Pact Director has relied on teamwork for program implementation as well as a means of mentoring and building capacity among the staff. While the staff members initially found the practice of teamwork difficult to implement, with time they have grown to appreciate its advantages. Several Pact staff members mentioned that teaming contributes to good working relationships and communications and enables one team member to effectively fill in for another in times of high workloads or absences. USAID appreciates Pact’s use of teams, but at times is concerned that team members are asked to assume responsibilities beyond their capacity. For example, the Director of Finance recently filled in for a program officer at a CSO workshop and gave a presentation on advocacy. USAID questions whether the Director of Finance has the skills in advocacy to make such a presentation and whether this type of arrangement is the best use of staff skills and potential. While there was universal agreement within the Pact office that the staffing pattern and people occupying the staff positions are adequate to the requirements of the cooperative agreement, USAID expressed concern with Pact’s lack of technical expertise and judgement in the specific areas in which it is expected to provide training and technical assistance. Of particular concern is the lack of advocacy expertise within the existing staff. The two program officers, who have assumed responsibility for advocacy training, had strong backgrounds in working within CSOs when hired, but had very little experience with advocacy. Most of what they now know about advocacy has been learned on the job—mainly through the workshops sponsored by ZADF/Pact. Hence, their level of knowledge about advocacy is currently roughly the same as the level of knowledge within the CSOs they are meant to assist. USAID believes that problems related to early workshops were due to the lack of advocacy expertise among Pact staff and is disappointed that the Pact Director has never taken steps to address this concern. ZADF/Pact Evaluation 11 Resource people interviewed concurred that Pact is lacking capacity in advocacy and the ability to consult with a wide variety of people with knowledge in constituent building, empowerment, training, etc. in order to ensure that their approach to workshops and technical assistance is appropriate. As one resource person said, “they need a senior person with local knowledge to get involved and give practical advice.” It is Pact’s position that their organization was never expected to have technical expertise in areas such as advocacy. Rather, ZADF/Pact expected to contract such expertise. By approving the position descriptions prepared by Pact and concurring with the persons selected to fill these positions, USAID implicitly agreed with this position. As the focus of the ZADF project shifted from establishing an endowment to building capacity within CSOs, however, Pact recognized the need for advocacy expertise and included a new position for an advocacy officer in its proposed revised program description in 1999. USAID, however, felt the position should remain a deputy director position, as in the original cooperative agreement. At the time the modification was signed, it was envisioned that the deputy would work half time for Pact and half time for the USAID Office of Transition Initiatives (OTI) project, which was also being implemented by Pact. USAID viewed the deputy position as key for several reasons: (1) a highly qualified deputy would provide ZADF/Pact with an intimate knowledge of the CSO community and Zimbabwean political context; (2) the deputy would provide in-house advocacy expertise; and (3) the deputy would contribute to long-term sustainability. Unfortunately, the person selected for the position in the first recruitment attempt in July 2000 did not accept the position. A second recruitment attempt resulted in no suitable candidates. A third attempt is now underway, with the position having been advertised in October 2001. The fact that Pact has not filled the deputy position, as required in both the original cooperative agreement and the modified program description, has been a long-standing, major source of friction between USAID and Pact. Pact questions the value added of a deputy, particularly as there is now less than one year left under the current cooperative agreement. Given that the Pact staff members are confident of their ability to fill the requirements of the cooperative agreement, there is concern that a deputy would add another layer of bureaucracy to the office and that a politically active deputy would make Pact appear to be political organization. USAID, however, sees the lack of a deputy as a significant failure on Pact’s part. USAID contends that the deputy is needed for advocacy expertise/political knowledge as well as for sustainability--and points out that the hiring of a top-notch deputy would do much to alleviate its concerns about the lack of advocacy skills within the office. Pact has made some efforts to build the capacity of its staff and increase in-house understanding of relevant subjects. For example, realizing that the staff lacked knowledge of Parliament, the Director requested that one staff member undertake research on how Parliament works. A research paper was prepared and presented to Pact staff. It was Pact’s perception, however, that this was not encouraged by USAID and it was not carried any further. It was also Pact’s perception that USAID did not encourage cooperation between Pact and SUNY, which was an alternative means of strengthening Pact’s knowledge of Parliament. USAID contends, however, that it has never ZADF/Pact Evaluation 12 discouraged informal contact between SUNY and Pact. Formally, however, USAID has taken a cautious approach for a variety of reasons. Initially USAID was concerned that both Pact and SUNY needed to consolidate their own programs and develop their own relationships with their respective clients before embarking on cooperative endeavors. This feeling was reinforced by the recognition that SUNY deals with a ZANU-PF￾dominated Parliament, while ZADF/Pact clients are frequently viewed as “the opposition.” SUNY’s credibility could be compromised if it is seen to be too intimate with the CSO community. Additionally, USAID believed that, until recently, it was too early for the CSOs to benefit from contact with SUNY. Instead, it was felt that the CSOs needed to focus on defining their advocacy issues, carrying out the necessary research, and otherwise preparing for advocating to Parliament. Now that the CSO programs have matured, strategic linkages between ZADF/Pact and SUNY, as in the second networking dinner and a recent luncheon, are beginning to emerge. Under the terms of the original cooperative agreement, funding was provided for training which Pact viewed as staff development. ZADF/Pact encouraged staff to take advantage of this opportunity and several enrolled in relevant courses. Under Modification No. 2 to the cooperative agreement, all funds for staff development were removed, based on the rationale that those hired should already have the qualifications for their position and should not require further training. USAID had never viewed the “training” line item as including staff development and only became aware of the distinction when the modification was being developed. However, Pact, as an institution, has a policy of supporting capacity building. As a result, Pact now uses the field staff “fringe benefits” line item of the cooperative agreement to fund limited staff development. The Pact Director demonstrated strong loyalty to his staff and enormous confidence in their abilities. This was echoed by Pact headquarters, which stated that the ZADF/Pact finance team was “crack” and mentioned that a Pact institutional priority worldwide is to protect their local staff. B. Development and implementation of a grants management system With the hiring of the grants management specialist, Pact initiated establishment of a grants management system. A document entitled “Grants Manual and Appendices” was developed to explain the system and provide easy reference for all involved in the system. The manual was reviewed by the USAID Regional Contracting Officer to ensure that it was in compliance with USAID rules and regulations. Pact uses the manual to guide its work not only with ZADF sub-grantees, but also with ZIMDES grantees. This is an excellent process adopted by Pact that helps to standardize the way in which CSOs are strengthened and monitored across USAID’s D/G portfolio. To date, two rounds of sub-grants have been successfully awarded using Pact’s grants management system. Six sub-grants were awarded in Round I and five in Round II. In addition, extensions have been granted to the six Round I sub-grantees. As envisioned in the cooperative agreement and set forth in the grants management system, USAID and the core partners were actively involved in the selection, approval and extension of ZADF/Pact Evaluation 13 ZADF sub-grantees. The advice of the core partner group was viewed as a critical input in this process. Under the grants management system, sub-grantees go through a two-stage application process. This two-stage application/selection process was originally developed by USAID/Zimbabwe for the award of ZIMDES grants. Given its success, ZADF adopted this technique. Successful grantees reported that the process was beneficial in helping them to focus their programs and identify their priorities. While somewhat lengthy, the two stages were, more importantly, viewed by the CSOs as an educational process which contributes to CSO strengthening. The pre-award surveys carried out by Pact for both ZADF and ZIMDES (sub) grantees were comprehensive and effective in identifying areas in which potential (sub) grantees required improvement. CSOs viewed these surveys positively, mentioning that they helped minimize the chances of problems later. They also helped the CSOs realize the importance of setting forth policies in various financial and administrative areas. The CSOs noted that there is high staff turnover in most CSOs and that having written policies resulted in easing the staff transitions. Financial/administrative monitoring of (sub) grants has taken place through compliance visits, which were carried out regularly by Pact staff. During these visits, Pact not only ensured financial and administrative compliance with the terms of the (sub) grants, but also provided technical assistance and training to the finance and administrative officers within the CSOs. ZIMDES grantees reported that although Pact conducted the compliance visits, they have a tendency to contact the USAID Controller’s Office directly for questions on financial matters. The USAID Controller’s Office concurred that this is true, but added that the Pact staff is fully capable of responding to ZIMDES grantees’ inquiries. ZIMDES grantees also mentioned that USAID usually attends the compliance visits with Pact. ZADF sub-grantees, on the other hand, reported less frequent attendance by USAID on compliance visits and said that they contact Pact on financial and administrative issues. USAID contends that it is not given adequate advance notice of the compliance visits to be able to attend more frequently. One grantee mentioned that the CSOs also do not receive adequate advance notice of the visits. While initially USAID did not always receive compliance visit reports from Pact, during the last year reports from the visits have been submitted more regularly. One CSO reported that inadequate assistance in financial management had been received. The person interviewed felt that Pact’s financial team was not proactive enough. (Conversely, it is possible that the CSO may not have been proactive enough in requesting assistance.) Two CSOs reported that the requirement to liquidate the previous month’s advances was sometimes hard to meet. There was a feeling that the system was not adequately flexible to respond to the realities of an advocacy program in which meetings change, are spontaneously set up, etc. One grantee also reported that the requirement for monthly reporting was onerous. ZADF/Pact Evaluation 14 One sub-grantee expressed dissatisfaction with the size of grants available, saying that the funding to grantees should be commensurate with the scope of the grantee’s program. Specifically, that those with national programs needed more funds to be able to reach their constituents. To date, the final stage of Pact’s grants management system, the closeout of sub-grants, has not yet been required. However, no problems are anticipated. Pact headquarters reported that the grants management system set up for the ZADF program has been so successful that it is now being used as a model for other Pact field offices. The ZADF manual has been shared with other offices and is used for training. C. Provision of technical assistance and mentoring to ZADF/Pact sub-grantees and ZIMDES grantees Under the terms of its cooperative agreement, ZADF/Pact was to provide technical assistance and mentoring to both ZIMDES grantees and ZADF/Pact sub-grantees. At the time of this evaluation, there were five ZIMDES grantees and eleven ZADF sub-grantees. Although all of these organizations met the criteria for (sub) grant selection, there is a great variety of interests and abilities among them. While some of the (sub) grantees are organizations that engage primarily in advocacy, for others, advocacy is only a part of the organization’s program. Some of the CSOs are membership organizations (i.e. have other CSOs as members), while others have constituents. Some are relatively new organizations; others are more mature. Some have a relatively strong capacity; others are weaker. Because of these differences, some organizations have received significant amounts of technical assistance and mentoring through ZADF/Pact, while others have received less. All of the Round I CSOs interviewed reported that they had received technical assistance in each of the four areas included in the cooperative agreement: advocacy, monitoring and evaluation, financial management and gender analysis. One-on-one technical assistance and mentoring is provided mainly by the Pact staff, although at times Pact has assisted grantees in accessing outside assistance. Assistance in financial management is provided principally during compliance visits (see Section VI.B., above). Assistance in monitoring and evaluation is provided mainly in conjunction with the advocacy index process (see Section VI.F., below). One-on-one technical assistance for advocacy and gender takes place both during programmatic visits and in response to specific requests from the CSOs. Pact’s “open door policy” is highly valued by grantees, which feel completely free to telephone Pact with any technical and /or administrative questions and to request Pact assistance when needed. As one person said, “we know we have friends there.” Another stated that ZADF/Pact was “the most cooperative and supportive program of any I’ve dealt with in the past fifteen years.” Communications between the Pact program officers and the CSOs is frequent and regular. Two CSOs (both ZIMDES grantees), however, ZADF/Pact Evaluation 15 mentioned that at first they didn’t know whether to contact Pact or USAID with their questions. One said that finally they figured out that technical/administrative questions should be directed to Pact, while policy issues should be directed to USAID. Another ZIMDES grantee said that they tend to contact USAID for all their questions. All of the CSOs interviewed agreed that the technical assistance and mentoring provided by Pact was useful and had had a positive impact on their programs. They had trouble, however, differentiating between what they had learned in workshops and what they had learned from one-on-one technical assistance. They also had trouble trying to define the extent to which participation in ZADF/Pact had improved the performance of their organization. The CSOs agreed that Pact provides guidance and options, but does not impose decisions. As one person said, “Pact gives you the resources and teaches you how to do it. Its capacity building puts it a cut above other programs. It doesn’t run your program, but helps you run it more effectively.” D. Workshops To date, ZADF/Pact has hosted ten workshops covering topics such as the advocacy index, understanding advocacy, advocacy planning and implementation, advocacy research, grant management, and advocacy communications tactics. CSOs involved in the ZADF/Pact program consistently attend all these group training sessions. Those attending normally include advocacy officers, program officers and financial officers (when relevant). For the most part, grantees were pleased with workshop topics. Several CSOs mentioned that the topics had to be appropriate and relevant, as the CSOs themselves provided input into the workshop topics. CSOs recognize that, in many cases, the workshops built upon previous workshops. CSOs were mainly pleased with the resource people presenting sessions at workshops. Only a couple presenters received fairly negative remarks. Several CSOs mentioned, however, that some of the presentations were too theoretical and that they (the CSOs) had difficulty in applying the concepts to their programs—that theory needed to be followed by discussions of practical applications. The communications/media workshop was praised for involving the CSOs, rather than having them attend as passive recipients. Early workshops were not as effective as they could have been and some topics had to be repeated due to practices such as: · Over-reliance on theoretical presentations. Trainers interviewed stated that adults do not learn well from presentations. Rather, they need hands-on, practical applications to learn. · Lack of contact and coordination between the presenters before the workshops. · Lack of follow-up after workshops. ZADF/Pact Evaluation 16 Pact has, however, learned from past experience and continues to take measures to improve the workshops. Specific improvements include: · Greater preparation and cooperation among resource people before the workshops. · Consultations with CSOs before workshops to identify needs and thus tailor the workshops more specifically to those needs. · Pairing of lecturers with trainers (i.e. theory and practice). · Narrowing of the focus of workshops (rather than making them too general). · Making certain that politically acceptable resource people are selected (including increased consultation with USAID on the selection of presenters). Gender was the only area consistently mentioned by CSOs in which they felt that the workshops had fallen short. Principal reasons cited for this included: · This is a new area for most CSOs and is seen as being outside of traditional Zimbabwean culture. · Expectations were unrealistic, given the time allocated to gender. · The sequence in which gender was introduced was not logical. For example, the integration of gender into planning was introduced before awareness had been adequately raised. By not following the proper sequence, little impact could be expected. · Gender was addressed as a separate step or an adjunct to developing an advocacy program rather than as an integral part of any advocacy effort. Gender has to be part of problem identification, research, targeting, etc. · There was a lack of continuity between presenters. Three early workshops included gender topics, but since the resource people didn’t know each other nor know what had been presented previously, they were unable to build upon and reinforce each other’s presentations. Pact recognizes that gender continues to be a concern and has a gender workshop planned for December 2001. Two gender consultants recently concluded pre-workshop needs assessments with the (sub) grantees. While the approach being taken for this workshop incorporates lessons learned and is a significant improvement upon earlier attempts to provide gender training, there is still a certain disconnect. The needs assessments are expected to feed into a workshop. However, it is possible that the results of the needs assessments will show that a workshop is not the correct venue to address the needs of the CSOs. The evaluator’s conversations with (sub) grantees indicate that individual technical assistance may be required. Attendees at the workshop are expected to produce, however, a gender action plan for their organization which will then be utilized to guide follow-on training and technical assistance. In general, workshop attendees prefer local and/or regional experts to ones from the United States, unless the topic is highly specialized and the consultant brings something to the table that cannot be found locally (David Cohen of the Advocacy Institute was offered as a positive example of outside expertise that was highly valued). It was felt that non-Zimbabwean experts need to have more exposure to Zimbabwe so that they can make their topics relevant to the needs of the CSOs. ZADF/Pact Evaluation 17 One person interviewed mentioned that workshops could be strengthened through the attendance of local professionals related to the workshop topic. For example, the communications/media workshop would have benefited from the inclusion of members of local media or media consultants. Some CSOs felt that there had been enough workshops—that basic, but important, information was provided at the workshops, but that now one-on-one technical assistance was more important for helping the CSOs apply the lessons of the workshops to their organizations’ programs. There were mixed feelings among the CSOs and resource people interviewed regarding the advisability of holding workshops outside of Harare. Some felt that this allowed them to focus on the workshop and helped ensure that all participants attended all sessions. Others felt that leaving town was not convenient (particularly if the venue was quite far from Harare) and expressed concern that Pact could gain a reputation for putting on workshops at plush resorts. Resource people expressed concern that there continues to be inadequate follow up after workshops to ensure that the training covered at the workshops is applied in the workplace. It was recommended that, after the workshops, Pact work individually with the CSOs to overcome specific personal and/or institutional problems that prevent the training from being implemented. Because of inadequate follow-up, training has been put into practice in varying degrees in the different CSOs. Given that several of the workshops were planned as a series, building on the information provided in the previous workshop, the information became progressively less relevant to those who could not implement the early training. Following each workshop, Pact produces a report which either summarizes presentations and/or contains the overheads used by the presenters. These reports are disseminated to all those who attended the workshops as well as to USAID. One resource person mentioned that, while these reports are probably useful to those attending the workshops (as a resource and refresher on what was covered), they are not useful as resources for those not attending the workshops. At the end of most workshops, Pact usually distributes an evaluation form. One person mentioned that these evaluations are meaningless, except for finding out who was the most popular presenter. Another mentioned that forms had not been distributed at the last two gatherings (one workshop and one networking dinner). Instead, comments were solicited by email or verbally (i.e. “Please send Pact any comments.”) This person felt that this was not an effective means of soliciting feedback and did not set a professional example for the CSOs. USAID expressed strong disappointment that Pact has not been able to identify and recruit “cutting edge” technical assistance in the area of advocacy for use in both workshops and one on-one-technical assistance. It feels that Pact has not made strategic ZADF/Pact Evaluation 18 use of consultants, local resources (including those at USAID), and other regional Pact programs. USAID expressed concern that Pact might not be doing more in this area because the Pact field staff is not aware of what else is possible and/or available—and because the expertise available through Pact headquarters is not being adequately utilized. The fact that Pact has not developed a database of excellent consultants, in spite of USAID's encouragement to do so, highlights USAID's sense that this is not a priority for Pact. Pact, however, feels that it has tried to respond to USAID's desire for innovation, but also believes that "there is nothing wrong with meat and potatoes." E. Networking/Coalition Building All CSOs interviewed said that they engage in networking/coalition building and that this activity is important to them. For the most part, networking is informal and is based on friendships. While participation in the ZADF/Pact program has contributed to increased networking between program participants, all CSOs interviewed stated that they engaged in networking before ZADF/Pact and that they continue to network with CSOs outside of the ZADF/Pact program. As one person stated, “ZADF/Pact itself cannot be considered a coalition, as the only thing the grantees have in common is participation in the program.” At the same time, several (sub) grantees mentioned that participation in ZADF had raised their awareness of the importance of networking and coalition building. Workshops and the newly implemented networking dinners were viewed as the two ways in which ZADF/Pact has facilitated networking. To most of the CSOs, networking is an important aspect of workshops. The networking dinners have also been successful, although some feel that there is not adequate time at the dinners for real networking to take place. Others do not agree. Some also feel that the dinners are still too formal, as many come straight from work and are still in their formal work clothes. Two persons described the dinners as “mini-workshops.” F. Management of the “Advocacy Index” process The advocacy index is a USAID tool presented in the USAID Center for Democracy and Governance Handbook as a means of monitoring progress in advocacy without measuring quantitatively the impact on policy-making. In January 1999, USAID/Zimbabwe chose to use the advocacy index as the primary indicator for the impact of its advocacy interventions. USAID/Zimbabwe procured technical assistance to help adapt the model suggested in the Handbook to the realities of Zimbabwe and to introduce the index to Pact and core partners. Pact then assumed primary responsibility for implementing the index. At USAID’s suggestion, Pact continues to recruit the same consultant that helped modify the index for ongoing technical assistance. Pact has built a good working relationship with this consultant and his input has contributed greatly to the success of the index. Under the process established, each (sub) grantee uses the index to complete a self￾assessment immediately upon receipt of (sub) grant funding. USAID and Pact also each assess the (sub) grantee. The (sub) grantee then presents its assessment orally before the ZADF/Pact Evaluation 19 advocacy index panel7 , which also has access to the USAID and Pact written assessments. Following the oral presentation, the panel independently rates the CSO, thus establishing an independent baseline rating for the CSO. This process is repeated annually, enabling all involved to monitor progress against the eleven elements in the index. Until USAID introduced the advocacy index in 1999, the Pact staff in Zimbabwe had no previous experience in its use. However, Pact staff members, particularly the program officers, were quickly trained by the consultant in how to use the tool. They are now fully conversant in its use and are able to provide quality training and technical assistance to the CSOs. When the (sub) grantees were initially introduced to the advocacy index, most found it to be confusing and difficult. After going through the process of completing the index together with Pact staff the first time, however, the CSOs interviewed agreed that it was easy to complete by themselves the next time. The CSOs interviewed are now highly enthusiastic about the advocacy index and feel comfortable using it. Virtually all organizations interviewed reported that they find the index to be a useful, multi-purpose tool that is used for planning, monitoring and self￾evaluation. Several reported that they have used the index for activities within their organization other than those supported by ZADF/USAID. All felt that that they would continue to use the index, or at least some form of it, even if they received no further funding through USAID/ZADF. (Sub) grantees appreciate the opportunity to present their programs before the advocacy panel and receive feedback. They feel that this is a learning experience during which the panel members point out things about themselves that they have not seen (both positive and negative). One person stated that the panel gives lots of encouragement. Its independence and objectiveness give the CSOs confidence in it. The member of the panel that was interviewed mentioned that initially she had serious doubts about the advocacy index. She was concerned that it would place an unfair burden on already stretched project officers, forcing them to invest their time in administrative matters rather than on their program. These doubts have now been dispelled. She reported that the panel consistently receives positive feedback on the process, that those CSOs which are stronger report that the process requires little time, yet is helpful, while those that are organizationally weaker find that it helps them to conceptualize and focus. According to the interviewee, all panel members feel that by serving on the panel, they are making a contribution to civil society by helping the CSOs to upgrade themselves and become more professional. Because USAID/Zimbabwe sees the value in the advocacy index and has had success in using it under ZADF, it is adapting the index for use in other mission activities. 7 The Advocacy Index Panel consists of six highly qualified Zimbabweans active in civil society. The Panel was established to provide an independent assessment of (sub) grantee progress against the index. ZADF/Pact Evaluation 20 USAID/Zimbabwe’s use of the index is now used by USAID’s Center for Democracy and Governance as an example for other USAID missions. G. Workplans Pact submits annual workplans to USAID based on the calendar year. Under terms of the cooperative agreement, draft annual workplans are to be submitted to USAID not later than 60 days prior to the start of the next workplan year. USAID then has 30 days for comment and Pact has 15 days to finalize the workplan. Development of the annual workplan submitted to USAID is carried out in a highly participatory manner within Pact, involving all Pact staff members. All members are aware of the contents of the workplans and their responsibilities under them. For a variety of reasons, the workplan process has consistently taken place later than it should and has taken longer than is necessary. Because the cooperative agreement was being modified, the draft CY 2000 workplan was submitted in February 2000, rather than November 1999. Correspondence on the workplan continued between USAID and Pact until May, when the Regional Contracting Officer sat in to discuss and resolve various issues. In 2001, Pact again submitted a draft annual workplan in February, rather than November 2000. USAID provided extensive comments and a revised/final workplan was submitted in April. For CY2002, at the time of this evaluation, Pact had initiated discussions of the workplan. However, ideally the recommendations of this evaluation should be incorporated into the planning process, meaning that the workplan will again be delayed. H. Results Reporting Initially Pact submitted quarterly performance reports to USAID. Modification No. 2 to the cooperative agreement included a provision that Pact should submit semi-annual, rather than quarterly, reports. All required performance reports have been submitted by Pact. However, the reports do little more than summarize actions and activities—many of which have been previously reported upon. They contain no critical analysis of trends within the CSO community; no indication of looking at the “big picture” and seeing how ZADF fits into civil society. As such, the reports are read and filed by USAID, but have not been used for any further purpose, such as helping USAID in its programming decisions. A few times, reports were submitted late. Pact has provided input for USAID’s annual results reporting for both 1999 and 2000. In both cases and at the suggestion of USAID, Pact hired the consultant that helped create the advocacy index to prepare the reports. The reports were useful to USAID and adequately provided the information required for USAID to meet its annual reporting requirements, including a discussion of results achieved against each of the three indicators and a summary of Pact activities that had contributed to attainment of the positive results for the year. ZADF/Pact Evaluation 21 Silveira House Recognizing that lack of access to education in their own language was a principal contributing factor to the high illiteracy rate in Binga, Silveira House decided that something had to be done. Working hand in hand with those affected, Silveira House is now helping minorities to lobby for a new language policy that will promote the teaching of minority languages up to university. To jump start the process and to help give minorities pride in their own language, Silveira House recently helped publish a collection of Tonga proverbs. I. Cost ZADF expenditures are currently under budget. At current spending rates, it is anticipated that there will be over $650,000 in unexpended funds at the end of the cooperative agreement in Sept. 2002, including $283,000 for sub-grants (38% of the total for this line item). This represents slightly less than nine months of operation. The main reason for this large surplus is the devaluation of the Zimbabwe dollar. All sub-grant budgets are required to be submitted in Zimbabwe dollars. Because of devaluation, sub-grantees receive fewer U.S. dollars than anticipated. While the evaluator was unable to confirm that rapid inflation (officially placed at 86% for September 2001) has been a problem for CSOs thus far, certainly the potential exists for the Zimbabwe dollar budgets to be inadequate for carrying out planned programs. While some inflation factor may have been built into budgets, the rapidly escalating rate of inflation could not have been foreseen by (sub) grantees (or Pact or USAID) when CSOs prepared their grant budgets. Because of the rapid devaluation of the Zimbabwe dollar and rising inflation, in June 2001, Pact tied local salaries to the U.S. dollar. Pact has no institutional policy on such matters. Rather, it tries to coordinate with USAID actions in the respective countries in which it operates. USAID/Zimbabwe tied local salaries to the U.S. dollar a few months after Pact. VII. PRINCIPAL FINDINGS – CSO’S PERFORMANCE A. Performance on the Advocacy Index The six Round I ZADF sub-grantees and four of the five ZIMDES grantees have completed baseline, one-year and two-year assessments using the advocacy index. For both years one and two, nine of the ten organizations made substantial progress; each year one organization made only marginal progress. The six Round II sub-grantees and one of the ZIMDES grantees have completed only their baseline assessments. Their first year performance assessments will not take place until August/September 2002. B. Contribution of ZADF/Pact to CSO Progress Among the CSOs interviewed, there was a distinct feeling that you can see the difference between CSOs that have participated in ZADF/Pact and those which have not. All expressed the belief that participation in ZADF/Pact has helped them to realize ZADF/Pact Evaluation 22 National Association of Societies for the Care of the Handicapped (NASCOH) Believing that the time had come for people with disabilities to demand participation in the formulation of laws that affect them, NASCOH set up five regional advocacy committees, each headed by a Chairman with a disability. These committees are now meeting with local and regional officials to advocate on behalf of the handicapped on issues ranging from housing for the disabled to improving educational facilities for disabled children to including the disabled as beneficiaries in land redistribution programs. themselves and find new potential. CSOs reported ZADF/Pact participation as particularly important in: · Improving their ability to carry out an advocacy program · Helping them be able to monitor their progress · Helping them to train their members and thus garner more support · Helping them to raise their profile · Allowing them to hire better staff (frequently an advocacy officer) and acquire some equipment. The CSOs interviewed specifically felt that ZADF/Pact deserved credit for helping them to succeed in their advocacy efforts. They said that training in advocacy had been useful because it provided a structure and a model to follow. Even those who had previously engaged in advocacy didn’t know the steps and sequence required to be effective. Areas of assistance that they found particularly useful included helping them to focus/more narrowly define advocacy programs, helping them to understanding the importance of advocacy research, and helping them to monitor their advocacy programs through the use of the advocacy index. Several groups mentioned that they already had advocacy programs before becoming part of ZADF/Pact, but that participation in the program had nudged them to expand their program and begin to advocate to Parliament. Silveira House expressed this sentiment well when it stated that ZADF was “the right thing at the right time” for them. Silveira House had already been involved in civic education, with changing the attitudes of the marginalized, and with local governments, but needed to take its activities to the next step. Participation in ZIMDES/ZADF encouraged the organization to broaden its program and approach its members of parliament (MPs). One CSO reported that because of its participation in ZADF/Pact, it was now able to provide advocacy consulting services to others in the region. In doing this, it made use of many of the materials provided at ZADF/Pact workshops. CSOs also pointed out, however, that building the capacity of the advocacy officers was insufficient to ensure the sustainability of advocacy within an organization. Several mentioned that there is significant movement of personnel among CSOs and that if the trained officer leaves, there is still no growth in the organization. Others pointed out that trained employees were not always allowed to put their training into practice if their superiors were not convinced of the need for advocacy or did not understand what ZADF/Pact Evaluation 23 Confederation of Zimbabwe Industries (CZI) A CZI commissioned study on the utility of the Zimbabwe Manpower Development Fund concluded that industry’s contribution to the Fund (1% of their annual wage bill) far outweighs the benefits. After presenting these findings to MPs and stakeholders, industry proposed several changes to make the Fund more transparent and responsive to the needs of industry. CZI then prepared redrafted legislation, which will soon be presented to the Attorney General’s Office for refinement, pending presentation to Parliament. advocacy was about. They pointed to the need for a holistic change within an organization before advocacy could be institutionalized. C. CSO’s Perception of Parliament Participation in ZADF/Pact has had a significant impact on (sub) grantee’s perception of Parliament. Several CSOs mentioned that whereas they used to think that Parliament was unapproachable, they now realized that it was made up of people like them and now had no fear in approaching Parliament. As one person said, “Pact’s assistance helped break the barriers.” ZADF/Pact’s assistance has also given the CSOs a greater appreciation of how Parliament works and where the entry points for advocacy are. While some CSOs stated that they have found the MPs and committees to be receptive, others complained that MPs do not consistently respond to CSOs’ invitations to workshops and meetings. These CSOs feel that the MPs need to be educated to realize that CSOs can help them to become more knowledgeable and present ideas effectively. Conversely, it is possible that the CSOs need to find alternative advocacy mechanisms. Given how few MPs there are and the demands being placed on them, CSOs must be realistic with their demands. One person interviewed stated that working with Parliament is a challenge that requires patience and perseverance as well as being polite and accommodating. Another stated that “you just have to be well prepared.” Several mentioned that not appearing to be politicized was a constant challenge. While the CSOs credit ZADF/Pact with helping them approach Parliament, they feel they are now ready to benefit from SUNY’s expertise in this area. D. ZADF/Pact Timeframe ZADF/Pact is scheduled to end its four year program in September 2002. Under the program, sub-grants were only given for one year, with some renewed for a second year. Advocacy, however, is a long-term process and making inroads to Parliament takes time. One to two years is not enough time to interface with Parliament and become effective. SUNY mentioned that it has taken them two years just to build relationships with Parliament, understand the system and overcome suspicion. CSOs will need a similar time frame, under one mechanism or another. Several (sub) grantees mentioned that they will just be beginning their advocacy program when their current grant ends. ZADF/Pact Evaluation 24 VIII. CONCLUSIONS Phase II is needed There is a clear need for USAID to proceed with implementation of Phase II of the advocacy program and continue to provide support for civil society organizations advocating to Parliament. Indeed, those interviewed expressed a strong belief that the potential for program impact would only expand. Zimbabwean civil society, which has long been quiet, is poised to make its voice heard. By continuing to build the capacity of a small number of strategic CSOs, USAID can help ensure that voices are raised in a constructive manner that contributes to the furtherance of democracy in this country. The strategy adopted by USAID in 2000 continues to be appropriate for Phase II. The focus on increasing participation and understanding the advocacy process, rather than on changing specific legislation, will have a long-term developmental impact. This approach gives Zimbabweans the skills to tackle their own concerns, resulting in more ownership of the process, allowing them to proceed at their own pace and enabling them to address the issues that are of highest priority to them. Phase I has demonstrated successes Pact should be commended for the flexibility and cooperation it demonstrated when the direction of the ZADF program was undergoing revisions in 1998 and 1999. Pact was understanding of the changing environment and quick to respond to USAID’s new priorities and challenges. Under these conditions, Pact was willing to work with USAID and began to pursue a new direction even before the project description was officially modified. ZADF is currently funding and supporting a good cross-section of CSOs which represent critical sectors in democracy and governance. Round II sub-grantees were strategically selected to fill sector gaps in media and gender. By building advocacy capacity within these sixteen organizations, the ZADF program will help empower broad sections of civil society, so that they are able to express their views to the government and advocate for positive change. As measured against the three SO1/IR1 indicators, Phase I of ZADF has successfully assisted a significant group of CSOs to move to a higher level of advocacy and begin to address Parliament. In 2000, targets for all three indicators were exceeded. At the SO level, from a baseline of three, ZADF activities were expected to contribute to eight incidences of “valid engagement” between civil society and parliament in 2000. With a score of 16, the program was clearly ahead of target. At the IR level, nine CSOs made substantial progress in advocacy capacity and performance, as measured by the advocacy index, against the target of seven. Among these, five were organizations that represent women and marginalized communities. This was from a baseline of zero and exceeds the target of four. Data for 2001 indicate similar positive results. At the SO level, 26 valid ZADF/Pact Evaluation 25 engagements with Parliament were reported, against a target of only 14. At the IR level, reorganization within one CSO kept it from attaining the anticipated increase on the Advocacy Index, although all other CSOs advanced as planned. The target for CSOs that represent women and marginalized communities was met. To confirm on a qualitative basis what the indicators quantitatively demonstrate, the evaluator found that interviewees agreed that the ZADF/Pact program helped contribute to the growth of a community of advocacy experts (both individual and institutional) that are now conversant in the concepts of advocacy and participation and that are moving ahead in the implementation of advocacy programs that will ensure that the voices of civil society are heard. The grants management system and advocacy index process are models USAID, Pact, ZIMDES grantees and ZADF sub-grantees all concur that Pact has established an excellent grants management system. The system should be commended for not only providing an effective and efficient process for awarding sub-grants, but for also incorporating capacity building/institution building. Sub-grantees agreed that going through the selection process and pre-award survey helped them to grow as an organization. Similarly, the advocacy index process that is managed by Pact is working extremely well and is providing (sub) grantees with not only the ability to monitor their advocacy programs, but also with a tool that is used for program planning and management. Additionally, the advocacy index provides timely input for USAID’s results reporting requirements. There is still room for improvement Differing visions: As highlighted by their differing views on the need for advocacy skills within Pact, the need for “cutting edge” technologies, and the role of a Deputy, USAID and Pact have different visions of what ZADF should be. Their inability to effectively share their visions has led to an erosion of confidence in each other, which undermines ZADF’s potential. Sustainability: Because of its “open door policy” and the close relationship between Pact program officers and (sub) grantees, it is easy for grantees to become overly dependent/reliant on Pact. While Pact is expected to provide technical assistance and mentoring to the (sub) grantees, Pact staff should not be relied upon to attend every grantee workshop nor act as resource people for grantee workshops (or should play only a minimal role in these workshops). Such reliance does not promote sustainability or growth within the CSOs. With ZIMDES and ZADF/Pact funds, CSOs have been able to hire advocacy officers and initiate advocacy programs. This has increased the recurrent expenses of the organization. It is not clear that steps are being taken to ensure that the organizations can cover these costs when no further funding is available from USAID. ZADF/Pact Evaluation 26 Need to reach CSO decision-makers: Those attending ZADF workshops and receiving one-on-one technical assistance are principally advocacy officers, program officers and financial managers. In many CSOs, these are lower level staff members, with little room to maneuver and implement new ideas. Pact has not recognized the need to ensure that decision-makers within the (sub) grantee organizations are brought on board and become supportive of the advocacy programs. Without their support, it is unlikely that those trained will be able to fully put their training into practice or that advocacy can become institutionalized within the CSO. One CSO leader mentioned that Pact does not have the skills to engage these decision-makers. Importance of organizational development: This issue is related to the above. Under the terms of the cooperative agreement, Pact is not responsible for providing systematic, broad, organizational development technical assistance to (sub) grantees. Instead, the scope of Pact's involvement in organizational development is defined by the advocacy index. However, (sub) grantees require more organizational development assistance than they are receiving in order to ensure that advocacy becomes integrated into their organizational structures. Without the provision of this type of assistance, advocacy will remain an “add on,” or "stand alone" program that is never institutionalized. Failure to incorporate organizational development assistance into the ZADF program threatens the sustainability of USAID’s investment. Requirement for follow-up: Workshops need to be followed by intensive one-on-one technical assistance to the CSOs to ensure that training can be put into practice. However, Pact, as currently staffed, does not have the capacity to offer this type of intensive technical assistance. More extensive use of local consultants and experts is required. Need for networking: Networking dinners and workshops are still not fulfilling the networking needs of the CSOs. More structure is needed. This should include informal venues, which need not include a didactic presentation. Cost-effectiveness: Because of inadequacies at early workshops, certain workshop topics have had to be repeated, resulting in a poor use of resources. Value added in CSOs' work & programs The evaluator was asked to assess the value added of ZADF/Pact’s training and technical assistance to the CSOs' work and programs. It is clear that (sub) grantees have made significant progress since the inception of ZADF. The CSOs, themselves, give significant credit to ZADF for this progress. It was impossible, however, for the evaluator to determine what proportion of the CSOs' progress was attributable to ZADF/Pact or to what degree other factors, including the CSO's own initiative and contact with USAID, influenced progress. Some CSOs clearly have stronger leadership, better-qualified employees and a more supportive organizational structure. These CSOs have been able to make more rapid progress because of that advantage. It is also clear ZADF/Pact Evaluation 27 that some CSOs have stronger relationships with USAID than others, which could be an advantage. The sample interviewed was too limited, however, to make definite conclusions about value added. In trying to assess value added, the evaluator noted that some of the CSOs see little difference between USAID and ZADF/Pact. One person stated that they were two sides of the same coin and said that they almost didn’t know who belonged to which. Another included the name of the ZADF/Pact CTO when discussing Pact staff. Another said that ZADF/Pact is a duplication of USAID. It appears that (sub) grantees do not understand why some CSOs are funded directly by USAID and others through ZADF/Pact and that the roles of each institution are not clear. While this does not seem to affect project implementation, it could merit a repeat explanation at the next CSO gathering. IX. RECOMMENDATIONS A. For Phase I There are several relatively simple actions that both USAID and Pact can take between now and the end of Pact’s current cooperative agreement on September 30, 2002, to improve the way in which the ZADF/Pact program is being managed and implemented. Joint actions · USAID and Pact should discuss expectations for the following year before Pact produces a first draft workplan. This will allow issues to be resolved in advance, resulting in a first draft workplan that is more acceptable to USAID. In addition, Pact should consider including representatives from the (sub) grantees in the workplanning exercise. Consultations with Pact’s clients at this point will help ensure that planned programs are demand-driven and responsive to the needs of the CSOs. USAID actions · USAID needs to implement a system for ensuring that all ZIMDES grant agreements, financial reports, progress reports, etc. are forwarded to Pact in a timely manner. · Following the second progress assessment by Round I (sub) grantees, USAID should carry out an evaluation of the Advocacy Index to (1) determine whether any adjustments/modifications are needed to ensure that it is appropriate to the Zimbabwe environment; (2) enable USAID to expand use of the index to other programs; and (3) allow USAID to share this success with others. Dr. David Hirschmann, who has provided extensive training and technical assistance on the advocacy index for both Pact and USAID would be an appropriate person for this exercise. · USAID should encourage Pact attendance at USAID training related to grants management issues, including the ADS, compliance regulations, etc. These rules and regulations are constantly being updated and Pact cannot remain current without such training. · Increased cooperation between ZADF and SUNY through more regular meetings, information sharing, etc. should be encouraged (although Pact and SUNY must ultimately be responsible). Specific areas in which SUNY could help ZADF include: ZADF/Pact Evaluation 28 understanding how Parliament works, understanding how and when to approach Parliament, understand what kind of information is useful to MPs, building relationships, etc. · Similarly, contact between Pact and the Urban Institute should be fostered as this new program gears up. · USAID should reinstitute the practice of holding regular expanded team meetings. · USAID should hold semi-annual area reviews which involve USAID senior management, contractor/cooperative agreement senior management and the appropriate USAID technical officers. Regular contact between management would help ensure that all contractors have a common understanding of USAID's priorities and current thinking, would allow contractors to inform USAID senior management of relevant issues, and would otherwise contribute to better communication and understanding between all those involved in implementing USAID's D/G program. Pact actions · While initially there was a need for broad workshops to ensure that all CSOs had the same knowledge base, the focus now, particularly for the Round I (sub) grantees, needs to shift toward more one-on-one technical assistance to help the CSOs put their knowledge into practice within the constraints/organizational structures of their own CSO. This will require taking a broader look at organizational development. USAID’s AIDSPact project, which is also being implemented by Pact, is using the Zimbabwe Organizational Capacity Assessment Tool (ZOCAT) to carry out a similar exercise. ZADF/Pact should consider adapting this or a similar tool to systematically look at organizational development issues within the (sub) grantees, particularly as they affect the CSO’s ability to institutionalize their advocacy program. · Senior Pact management (Deputy) should embark on an "advocacy campaign" with the Chief Executive Officers, Boards of Directors and Executive Directors of (sub) grantees, explaining the program to them and recruiting their support for advocacy within their organization. The Pact staff does not have the stature to engage in this effort. High-level, personal relationships must be established with these people. The new Deputy must be proactive and take the initiative. Unless the senior leadership/decision-makers of the CSOs are on board, the advocacy officers and program officers receiving Pact training will have trouble putting their training into practice (this is particularly true in CSOs which do not have advocacy as their main purpose). · Pact should ensure that workshops are followed by intensive one-on-one technical assistance to help (sub) grantees apply their training in the workplace. This technical assistance could be provided by either Pact staff or by consultants. · Pact should make greater use of the (sub)grantees for the hands-on, case studies during workshops as well as for mentoring other ZADF/Pact (sub) grantees. Having to make presentations forces the CSOs to think deeply about what they are doing and helps to build capacity within the organization. Pact might even consider devoting an entire workshop to CSO presentations, giving each of the 16 (sub) grantees an opportunity to present a self-evaluation (successes and problem areas). Afterwards, groups could be formed, mixing strong teams in area x with weaker teams in that same area. This would foster mentoring, sharing, etc. among the grantees. ZADF/Pact Evaluation 29 · Pact should place increased focus on fostering networks and facilitating coalition building. This could be done through informal meetings, distribution of reports, bringing together groups with similar interests, etc.—and would be a way of integrating groups that are not receiving grants (e.g. for the future, a way of widening the pool of CSOs without actually funding them). Possibilities for regional networks should also be explored, including the potential offered through USAID’s regional advocacy program. · To facilitate the identification of appropriate expertise, Pact should develop a database of potential international, regional and local experts in the areas in which it is providing technical assistance. There is a need for new blood and fresh ideas within the program. The database should include lawyers, as two CSOs mentioned the need for legal advice for drafting bills and legislation. USAID has previously provided input for such a database and remains willing to assist in this endeavor. Resource people interviewed also mentioned that they would be pleased to provide names. Pact’s headquarters should also be enlisted to help identify and recruit appropriate international advocacy experts, including those from other Pact programs in the region. · Pact needs to develop guidelines for weaning overly dependent CSOs. It should be clear to all involved the extent to which Pact will provide assistance to CSOs in the implementation of their program. · Pact staff (specifically the Director and program officers) should make a greater effort to attend meetings and workshops outside of those sponsored by their (sub) grantees. This would be a way of augmenting their knowledge of the CSO community, building their reputation, etc. · Pact should develop quarterly schedules of site visits, compliance visits and feedback visits and share these with both USAID and the CSOs to ensure that both have adequate lead time to plan for and attend the visits. When planning compliance visits for ZIMDES grantees, Pact should be aware of the fact that the USAID Controller’s office is normally very busy at month end/month beginning. Scheduling the visits between the 6th and 25th of the month would facilitate USAID Controller participation. · When taking resource people to a CSO, the CSO would appreciate receiving adequate notice and should receive, in writing, a short description of who the visitor is and his/her purpose for visiting. · When inviting a CSO to participate in a workshop as a presenter/case study, the CSO must receive adequate advance notice, including a complete workshop agenda so that they know when and where they fit into the program. · Program evaluations should be distributed at the event (workshop or networking dinner), not solicited by email afterwards. B. For Phase II For Phase II (2002-2005), USAID will need to make some critical decisions. Unfortunately, there is pressure to make these decisions quickly, as the grant extensions provided to Round I grantees will end in/around March 2003 and Pact will be unable to continue to fund them—or fund other sub-grantees—unless the future is clear. ZADF/Pact Evaluation 30 As the program moves into Phase II, USAID should consider renaming the ZADF project to eliminate the word “Foundation” from the title. Given that there is now no plan to establish an endowment, having this word in the project name is confusing. At the same time, all those involved with the project are familiar with the ZADF acronym. Another name with the same acronym might be appropriate, such as: Zimbabwe Advocates for a Democratic Future. Program/Project Recommendations 1. Deepen the current pool of CSOs Three possible ways that USAID could continue to support CSO advocacy during Phase II include: deepening the knowledge base of the 16 CSOs currently being assisted, widening the group to include new CSOs, or supporting some combination of the two. Each direction has different implications for the program. Deepening the capacity of the current group of CSOs calls for increased focus on issues and content, thus requiring more specific expertise and technical assistance. For example, training in subjects such as conflict resolution and managing political diversity would be appropriate given the current political environment. Additionally, as CSO advocacy programs mature, assistance in addressing later stages of advocacy, such as how to follow up once a policy is changed (i.e., enforcement, budgeting, internalization, etc.) would be required. Broadening the group of CSOs with which ZADF works requires vigilance in judging the absorptive capacity of MPs and committees to respond to CSOs. Given how few MPs there are, they could easily be overwhelmed by too many CSO advocacy programs. Pursuing a combination of deepening and widening requires developing criteria for “graduating” some CSOs from assistance and/or eliminating poor performers while continuing to recruit new partners. It is the evaluator’s recommendation that USAID focus on deepening the skills of the current pool of (sub) grantees, while also slightly widening it. Following two years of monitoring on the advocacy index, USAID, Pact and the Advocacy Panel should review (sub) grantee performance to determine which, if any, (sub) grantees should (a) be eliminated from the program due to poor performance or (b) “graduate” from the program due to excellent performance. As (sub) grantees are dropped, new ones should be added. At no time, however, should the number of (sub) grantees receiving ZADF assistance exceed twenty. So that CSOs which graduate are not completely cut off from the ZADF program, they should continue to be used as resources/partners/mentors for new (sub) grantees. To complement the deepening of a relatively limited pool of CSOs, USAID should explore mechanisms for reaching out and strengthening other CSOs, without providing them grants or direct technical assistance. This could be through the production and dissemination of literature, through holding public meetings, thorough expanding networking/coalition building efforts, or by encouraging each (sub) grantee to “adopt” a CSO to mentor. ZADF/Pact Evaluation 31 2. Adopt a holistic approach During Phase II, USAID should place increased focus on ensuring that advocacy is institutionalized within (sub) grantee organizations and that the organizations are sustainable. This requires taking a more holistic view of the CSOs and their overall organizational structure. Embarking on an advocacy program is a completely new initiative for several of the (sub) grantees (particularly those involved in welfare-type issues). For advocacy to become integrated into their organizational structure requires more than just advocacy training. It requires that the structure of the organization and the budget be reexamined and, frequently, reorganized to include advocacy. It is essential that the CSO executive director and at least one member of the Board be included in the process, that participatory methods be used, and that adequate follow-up be offered. To accompany the organizational development focus, CSOs also need additional assistance with strategic planning. This is particularly true for the ZIMDES grantees, which have longer grant periods. Strategic planning efforts for advocacy programs must reflect knowledge of Parliament (i.e. activities need to be tied to happenings in Parliament). 3. Build a cadre of advocacy consultants To help foster sustainability, USAID should consider investing in building a cadre of local consultants that could serve as resource people for all of USAID’s DG programs. This might include sending candidates to short-term training programs (in either Zimbabwe or the region), arranging lectures by professionals brought in as consultants under ZADF, or supporting other training venues. The personnel from “graduate” CSOs and other (sub) grantee members who demonstrate potential should form the core of this cadre of consultants. To complement the cadre of specialists and to provide needed material resources, USAID should consider supporting the development of an advocacy resource center. This center could be located at the ZADF office and could include audiotapes and videos from project-funded workshops, information on the advocacy index, etc. 4. Provide longer grants USAID should lengthen the period of grants under ZADF, particularly for those sub￾grantees which have successfully implemented a short-term sub-grant. Given that the focus of the grants is on building advocacy capacity, which is a long-term process, it is unrealistic to expect significant gains under a one-year grant. Longer grant periods would allow the CSOs to receive support for the duration of an entire advocacy program (i.e. from planning and research to approaching MPs to drafting revised legislation). This would not only help CSOs in planning and implementing longer-term advocacy programs, but would also help ensure that efforts to achieve sustainability were included in the program. ZADF/Pact Evaluation 32 5. Develop program synergies USAID’s strategy provides a solid, integrated rationale for why supply and demand must be addressed and why national and local governance must both be targeted. On the ground, however, more could be done to integrate the various programs addressing these elements. USAID needs to examine how linkages between the programs can be fostered. ZADF should be able to provide lessons learned to the local governance program. Also, several ZADF (sub) grantees are starting regional programs that could possibly link with the local government project. Ways in which ZADF/Pact and SUNY could cooperate were discussed above. Similarly, there are opportunities for the participation strategic objective to link with other mission objectives. Both the AIDS program and the micro￾enterprise program work extensively with CSOs. There may be opportunities for linking the CSOs supported under the three programs, for building networks and coalitions. As the AIDS SO initiates its advocacy program, ZADF (sub) grantees could possibly be called upon to mentor or advise AIDS-related CSOs. 6. Develop project-level information mechanisms For USAID, the existence of the core partner group has been critical in ensuring that USAID remains informed and in touch. In the future, the ZADF implementing organization should develop similar mechanisms for ensuring that it has access to the current thinking and salient issues in the CSO community. This group could include a set of “virtual core partners” located within the region or abroad, to ensure that the organization had access to the latest thoughts, techniques and technologies. Implementation Decisions USAID has several options for how it might implement Phase II of the ZADF program. Options and some of the “pros” and “cons” for each are listed below, followed by the evaluator’s recommendations. Option 1: Pact No Cost Extension With funding in the current cooperative agreement, Pact could be given a no cost extension for approximately eight to nine months. During that period, Pact could be required to take several measures related to improving its training and technical assistance capacity. Should USAID be satisfied with Pact’s performance at the end of the extension period, a funded extension through the end of Phase II could be awarded to Pact. Key steps that Pact should take during the no cost extension should include: · Agreeing on the purpose/role of the Deputy and filling the position · Ensuring that the Deputy is adequately trained and ready to fully assume management responsibilities by the end of the extension period ZADF/Pact Evaluation 33 · Developing and implementing a plan for providing either on-going or intermittent assistance to the Deputy for the period between Sept. 2002 (when the current Director departs) and the end of the no cost extension · Reassessing the qualifications of the current staff and ensuring that the staffing pattern adequately reflect the expertise needed to provide and/or supervise increased one-on-one technical assistance in the areas needed, including advocacy, organizational development and strategic planning. · Developing and implementing a comprehensive technical assistance plan for the extension period. This could include a subcontract with an organization that specializes in advocacy and D/G issues. · Developing and implementing a plan for providing organizational development assistance to (sub) grantees. · Developing a database of potential international, regional and local experts for the provision of TA. · Exploring the economies of combining offices with other Pact programs in Zimbabwe. Pros · This option avoids another lengthy procurement process. · Pact is on the ground and can begin work under the extension immediately. It already has a functioning field office in Harare and knows the program well. · This arrangement would allow an extension of current sub-grantees’ grants, assuming they meet the requirements, without a break in services. · Pact has already developed close working relationships with the (sub) grantees. · The Deputy would be poised to assume management of Phase II of the program at the end of the extension period. · Pact would be able to provide capacity building in organizational development, an area in which it is a recognized leader. Cons · The “baggage” of the past USAID/Pact relationships could be hard to shed. · Even if the Deputy has advocacy expertise, if (s)he is to ultimately assume program management responsibilities, it is unlikely that (s)he would be able to also provide technical assistance and oversight in advocacy. Additional staff may be required. Option 2: Pact becomes a grants manager only Under this option, Pact would continue as the grants manager, but another organization would be contracted to provide technical assistance to (sub) grantees. As the grants manager, Pact could continue to make grants and provide administrative and financial management assistance to both ZIMDES grantees and ZADF sub-grantees. Given its success with the advocacy index process, it would be advisable for Pact to continue this function also. ZADF/Pact Evaluation 34 USAID would procure services for the provision of technical assistance through either a separate contract/cooperative agreement or through an Indefinite Quantity Contract (IQC). USAID's Bureau for Humanitarian Relief is currently bidding such an IQC. Pros · This option allows Pact to continue with what it has done well—grants management and the advocacy index process—while shifting responsibility for technical assistance—an area in which Pact’s performance has not been as strong—to another entity. Cons · The line between grants management and technical assistance is not always clear. Coordination and cooperation could become an issue. · Whereas grants management could continue unbroken, it is possible that USAID would not be able to procure the services of a technical assistance provider in a timely manner, leaving (sub) grantees without access to training and technical assistance. · Managing two implementing partners is a greater burden for USAID. · Pact may not be pleased with this arrangement, which could make future relationships uncomfortable. 1. USAID competitively awards a completely new contract or cooperative agreement Pros · This option offers a fresh start for all parties. · USAID could choose to have a contract, rather than a cooperative agreement, which would give USAID more control over the program. Cons · This is a time-consuming, lengthy process. It would have to be initiated immediately to ensure that there is no gap in ZADF implementation. · Even if the new partner were ready to assume duties on October 1, 2002, current sub￾grantees would experience a gap in services. Pact would need to close out current sub-grants before departure and it would take the new partner some time to develop and implement a new grants management system. · Development of a new grants management system when there is already an excellent one in place is not an effective use of U.S. government funds. The Evaluator’s Recommendation: The evaluator recommends that USAID adopt Option 1, above, and grant Pact a no cost extension through approximately July 2003. The no cost extension should be made with the understanding that the extension period is an opportunity to clear misunderstandings between USAID and Pact and that Phase II will be awarded to Pact based on Pact’s performance during that period. ZADF/Pact Evaluation 35 The first task should be for USAID and Pact to work together to develop a common vision of how the program should evolve between now and July 2003. This vision should form the basis of the revised program description for the extension period and should flow into a comprehensive workplan for the entire period. It is recommended that a senior person from Pact headquarters participate in this exercise to ensure that Pact headquarters is in agreement with the vision for the future. As part of the “common vision” exercise, USAID and Pact should agree on not only the role of the deputy during the extension period, but also on this person's potential role during Phase II. Once this common understanding is reached, Pact should be required to fill the deputy position as soon as possible. Ideally the position would be filled by January 1, 2002, so that the deputy would have nine months to work with the current Director before his departure (and so that, with the extension, the deputy could be offered an eighteen-month contract). This is, however, unlikely and a March target might be more realistic. Following the departure of the current Director, Pact should assign a co-director to work with the deputy (who would then become the other co-director) through the nine-month extension period—and possibly longer if Pact is awarded Phase II. Co-directorship allows the Deputy to play the active role envisioned, without being overburdened by administrative duties. Additionally, it allows Pact to provide a senior person with the organizational development expertise required to implement the recommendations of this evaluation. This person should have long-term experience with USAID and Pact and thus be able to continue to mentor the Zimbabwean co-director in these areas. Following the “common vision” exercise, Pact should be required to conduct an honest assessment of the skills of the current staff vis a vis the future needs of the program. A move to more intensive one-on-one technical assistance, an increased focus on organizational development and strategic planning, and the need for assistance in specialized topics (conflict resolution, etc) may demand a different staffing pattern. This is the time for USAID and Pact to clear the air on what degree of technical expertise is needed within the ZADF/Pact office. Following the staff review, Pact should be required to develop a comprehensive plan for the provision of technical assistance during the extension period. Pact may determine that a sub-agreement is the best mechanism for ensuring responsiveness to the cooperative agreement. At least six months before the end of the extension period, Pact should be required to submit a proposal for Phase II of ZADF. The proposal should clearly set forth how Pact proposes to deepen the knowledge base of the original sub-grantees while also addressing the needs of new sub-grantees. It should also address issues such as weaning “older” grantees from the program, developing a cadre of consultants, and other Phase II recommendations of this evaluation. ZADF/Pact Evaluation 36 Annex B Persons/Organizations Consulted USAID/HARARE Rose Marie Depp, Mission Director Michael Foster, Program Officer Stephanie Funk, Democracy Team Leader Deprose Muchena, Senior D/G Advisor and ZADF CTO Mollyn Saurombe, Controller’s Office PACT HEADQUARTERS Sarah Newhall, President and CEO Dan Craun-Selka, Director of Program Operations and ZADF Backstop Officer PACT ZIMBABWE John Rigby, Director Natalie Barefoot, Financial Specialist Shobna Chakravarti, Grants Manager Synodia Chikanza, Program Officer Killron Dembe, Program Officer Madzivanyika Moto, Director, Finance and Administration RESOURCE PEOPLE David Hirschmann Brian Kagoro John Makamure John Makumbe Petronella Maramba Everjoice Win ZADF GRANTEES Confederation of Zimbabwe Industries (CZI), Dakarai Matanga, Melvern Rusike National Association of Societies for the Care of the Handicapped (NASCOH), Farai Mukuta, Flora Shiringo Urban Councils Association (UCAZ), Francis Duri, Joel Zowa ZIMDES GRANTEES Combined Harare Residents Association (CHRA), Barnabas Mangodza Silveira House, Ignacius Musona, Sister Janice Transparency International, Zimbabwe (TIZ), Andrew Nongogo Zimbabwe National Chamber of Commerce (ZNCC), James Jowa ADVOCACY INDEX PANEL Eileen Sawyer CORE PARTNERS Eileen Sawyer John Makumbe Annex C QUESTIONNAIRE FOR PACT STAFF 1. Please tell me about your position in the ZADF/Pact office. Are your job description and responsibilities clear? (ask for TOR) Is the work that you actually do what is found in your term of reference? What could be done to allow you to perform your work more effectively? 2. Please describe how the Pact office is staffed and run. Do you feel that the staffing pattern is appropriate for the work that needs to be done? 3. How would you characterize relationships within the Pact office? Are communications good? Is there a team spirit? 4. There has been ongoing discussion about the need for a deputy in your office. The expectation is that this person would be a Zimbabwean that is politically astute and familiar with the CSO community. Do you have any comments about this idea? 5. To what extent are you consulted in the development of annual workplans? Are you familiar with the annual workplan and knowledgeable of what portions of the workplan depend on your input? 6. What is your view of Pact’s overall performance in carrying out the ZADF project? 7. In which areas do you feel Pact has been most successful? Less successful? 8. Originally, Pact was expected to establish a Foundation or Trust to carry on once USAID departed Zimbabwe. What is your understanding of why the program design was altered? How do you feel that this decision has affected implementation of the ZADF project? 9. What is your understanding of the relationship between ZADF/Pact and OTI? 10. How would you characterize your relationship with USAID staff? ZADF/Pact’s overall relationship with USAID? Who is responsible for that relationship? 11. In your view, what could be done to make ZADF/Pact a more successful program? QUESTIONNAIRE FOR CSOS Re: Grants Management System (for ZADF/Pact grantees only) 1. What was your impression of the two-stage system that was used for selecting grantees under ZADF? 2. Was Pact’s pre-award assessment useful in helping you understand what needed to be done to receive and manage a ZADF grant? 3. Are requirements for disbursement of funds clear? Are funds disbursed in a timely manner? Re: Programmatic Monitoring 1. Do you receive quarterly site visits from Pact? Compliance visits for financial/administrative management? 2. Do you receive feedback from Pact on your reports and activities? Re: Technical Assistance/Workshops/Training 1. Under its agreement with USAID, Pact is expected to provide technical assistance in the areas of financial management, monitoring and evaluation, skills in management and performance of advocacy and gender analysis. It does this through both one-on-one technical assistance and group training/workshops. a. Thinking only of one-on-one technical assistance: · To what extent has your organization received one-on-one TA? · How would you assess the overall quality of this TA? · Has the TA been of the type and depth needed? · To what extent do you believe that the technical assistance provided by Pact has improved the performance of your organization? b. Now thinking only of workshops and group training sessions: · To what extent has your organization participated in the workshops? · Who attended? · How would you assess the overall quality of workshops and group training? Have trainers/presenters been appropriate? Experts in their area? Good communicators? · To what extent to you believe that the training provided by Pact in workshops has improved the performance of your organization? 2. Focusing only on advocacy, to what extent has Pact’s assistance improved your organization’s understanding of advocacy and ability to plan and implement an advocacy program? 3. How does your organization feel about advocating with Parliament? Have you planned and/or implemented any type of advocacy program with Parliament? If so, how important was Pact’s assistance to your being able to implement this program? Re: Networking 1. This year ZADF/Pact initiated a series of networking dinners centered on a theme of common interest. Have these events been useful to your organization? Have the themes been relevant? 2. For your organization, is networking an important aspect of attending ZADF workshops? 3. Does your organization have regular contact with other CSOs on topics of interest? If so, how much of this can be attributed to being involved in the ZADF project? Re: the Advocacy Index 1. What are your feelings about the Advocacy Index? Do you feel that it is a useful tool? Do you feel that the presentations of your self-assessment are useful and allow you to adequately present your organization to USAID, Pact and the Advocacy Panel? 2. Has your organization used the Advocacy Index for any programs other than the ZIMDES or ZADF/Pact grant? 3. Do you think that your organization would use the Index even if it did not continue to receive USAID/ZADF/Pact funds? Re: the overall ZADF project 1. How do you feel that the ZADF project is viewed within the CSO community? What about USAID? Pact? 2. Do you feel that ZADF/Pact has a good understanding of the CSO community and the political environment in Zimbabwe? Does USAID? Other 1. What do you see as the “success stories” of your organization? What role has ZADF/Pact played in these successes? What role has USAID played? 2. Are there any changes that you would recommend for the ZADF/Pact program for the short term? For the long term? QUESTIONNAIRE FOR RESOURCE PEOPLE 1. Please tell me about your role as a resource person for ZADF/Pact. 2. Were Pact’s expectations of you clear (i.e. were Pact staff able to articulate what they expected of you)? Were you given adequate background information and lead-time to prepare for your role? Were the overall objectives of the workshop/training clear? Did you know who the other presenters were and what they would be presenting? 3. Outside of your presentation, did you participate in the workshop/training (i.e. attend other presentations)? How would you assess the quality and appropriateness of the other presentations? 4. What recommendations would you make for improving the quality and effectiveness of technical assistance and training provided under ZADF?