1 Mid-term Evaluation Health Decentralization & Community Participation with Centro de Informacion y Recursos para el Desarrollo (CIRD) (Cooperative Agreement No. 526 A 00 92 00019 00) (SO 526 006 – Use of Voluntary Reproductive Health Services Increased) Asuncion, Paraguay November 2003 Barbara O’Hanlon Independent Consultant 2 Table of Contents Section Page # Executive Summary………………………………………………………………………………………..4 1.0 Introduction………………………………………………………………………………………..9 2.0 Methodology………………………………………………………………………………………9 2.1 Evaluation objectives……………………………………………………………………..9 2.2 Evaluation questions……………………………………………………………………...9 2.3 Evaluation methods……………………………………………………………...............10 2.4 Evaluation approach……………………………………………………………………..11 3.0 Evaluation Areas………………………………………………………………………………….12 3.1 Technical progress toward achieving results………………………………….................12 3.1.1 Relationship between CIRD technical activities and indicators………………...12 3.1.2 Recommendations to strengthen CIRD indicators and measurements…….........15 3.1.3 Recommendations to strengthen monitoring of CIRD’s results...………..……..18 3.1.4 Progress in achieving results……………………………………………….........18 3.2 Management support toward achieving results…………………………………………..28 3.2.1 Key findings on and recommendations to improve CIRD managing for results..28 3.2.2 Recommendations to improve CIRD management……………………………..29 3.3 Quality of CIRD technical assistance…………………………………………………....32 3.4 CIRD relations with Alianza partners…………………………………………………....33 4.0 Recommendations to strengthen CIRD program…………………………………………………35 4.1 Recommendations strengthen overall technical approach……………………………….35 4.1.1 Consolidate CIRD’s technical model…………………………………………...35 4.1.2 Replicate and/or scale -up the model…………………………………………….36 4.1.3 Sustain CIRD’s technical model………………………………………………...38 4.2 Recommendations to strengthen specific technical areas………………………………..39 4.2.1 Health promoters………………………………………………………………..39 4.2.2 Referral and Counter-Referral System………………………………………….39 4.2.3 IEC………………………………………………………………………………40 4.2.4 Dissemination…………………………………………………………………...44 Diagrams Diagram 1: Mapping USAID Strategic Framework to CIRD Results Framework ………………………13 Diagram 2: Project Overview for CIRD Health Decentralization and Community Participation Project..14 Diagram 3: CIRD Health Decentralization & Community Participation Project Strategy and Activities..16 Diagram 4: CIRD Health Decentralization and Community Participation USAID Indicators and Measurements……………………………………………………………………………17 Diagram 5: Illustrative Communication Strategy for Alianza Members………………………………….40 Diagram 6: Roles and Responsibilities of Alianza Members in Joint Communication Strategy…………41 3 Section Page # Figures Figure 1: CIRD organizational chart in practise………………..…………………………………………30 Figure 2: Option #1 for restructuring CIRD Health Project………………………………………………30 Figure 3: Option #2 for restructuring CIRD Health Project………………………………………………31 Figure 4: Illustration of CIRD’s technical assistance approach…………………………………………...37 Figure 5: Illustration of alternative technical assistance approach………………………………………..37 Tables Table 1: Data for Indicators #1 and #2……………………………………………………………………20 Table 2: Data for Indicator #3 …………………………………………………………………………….22 Table 3: Data for Indicator #8 under IR 1.0……………………………………………………………….23 Table 4: Cumulative Record of IEC events and materials for Indicator #6 ………………………………25 Table 5: Data for Indicator #8 under IR 4.0……………………………………………………………….27 Table 6: Data for Indicator #9……...………………………………………………………………...……28 Annexes Annex 1: Evaluation plan Annex 2: Final agenda and interview schedule Annex 3: Bibliography of documents reviewed Annex 4: CIRD sites Annex 5: CIRD technical organization chart Annex 6: Illustrative M&E Plan Annex 7: Draft job description for Project Coordinator Annex 8: Illustrative agenda and talking points for USAID/CIRD management meetings Acronyms Acronyms Name CETEC Centro de Educacion, Capacitacion y Tecnologia Campesina CEPEP Centro Paraguayo de Estudios de Poblacion CIRD Centro de Informacion y Recursos para el Desarrollo CLS Consejo Local de Salud FP/RH Family Planning/Reproductive Health IEC Information, Education and Communication IR Intermediate Result M&E Monitoring and Evaluation PREVER PRIME II PROMESA Promocion y Mejoramiento de la Salud R/CR Referral/counter-referral system SIG Sistema Informatico de Gestion SO Strategic Objective SCLS Sub-consejo local de salud TA Technical assistance USAID United States Agency for International Development 4 Executive Summary Introduction USAID signed an agreement (No 526-A-00-01-00100-00) with the Centro de Informacion y Recursos para el Desarrollo (CIRD) to promote decentralization and community participation in health services. CIRD is entering its third year under the Cooperative Agreement and like the other USAID’s Alianza projects, is subject to a mid-term evaluation. USAID contacted an external consultant to conduct the mid￾term evaluation during October, 2003. This document presents the evaluation findings and recommendations. Methodology An independent consultant traveled to Paraguay in October to carry out the evaluation. The objectives of the mid-term technical evaluation of CIRD are: · Review CIRD’s the progress-to-date in achieving the stated results. · Review the Project’s technical management to determine if the technical staff, budget and operations support and are on-track to achieve desired technical results · Interview CIRD “clients” to ensure CIRD is providing effective, timely and quality technical assistance responsive to USAID’s and counterpart’s needs · Provide recommendations to USAID/Paraguay and CIRD management on how to modify and/or focus technical strategies to ensure CIRD activities will achieve results The Evaluator applied a variety of methods for the mid-term evaluation. The Evaluator conducted one￾on-one semi-structured interviews that lasted from 60 to 90 minutes with CIRD staff, Alianza members, and CIRD counterparts including government authorities at the Regional and municipal levels, health workers and members of the local health councils. The Evaluator conducted approximately 20 interviews in the field at 9 of 12 communities located in all four regions where CIRD is active – Itapua, Misiones, Centrales and Cordillera. In addition to the interviews, the Evaluator visited with and informally observed CIRD staff in the field and the counterparts who have received technical assistance and/or training from CIRD to see if and how the counterparts apply the new skills and knowledge received. Finally, the Evaluator reviewed all relevant documents while on TDY. Key Findings: Technical progress toward achieving results CIRD is off on start and is well on its way to achieve the its strategic objective (SO), as measured by Indicator #1. By the end of Year Two, seven of the twelve Alianza Consejos Local de Salud (CLS) are functioning at or above medium levelproficiency. An fully operational CLS is defined as able to articulate the CLS’s functions, state what are the members’ roles and responsibilities, describe a couple of the CLS community health priorities and describe a few of the Local Health Plan’s activities. Given their progress at the end of Year Two, CIRD will more than likely exceed the expected result. CIRD has achieved the expected results in three out of five key program areas. In the area of creating and strengthening CLS capacity to oversee health services in their communities, CIRD is on track to achieve Indicator #2 and will more than likely exceed it by the end of the project. In the second area of health finance, CIRD has achieved indicator #3 -three (3) finance models implemented- by the end of Year Two and will clearly exceed it by the end of the project. CIRD is implementing three different types of finance models: 1) Fondos Rotorios to finance Farmacias Sociales, 2) Social Insurance and 3) Cost Recovery. Fondos Rotorios to finance the Farmacias Sociales is by far, one of the most successful components of the CIRD project and the most widely implement finance model. The Farmacias Sociales is a huge success, spreading like wildfire, and demonstrating this activity can go to scale . Itapua has taken the activity to scale in 86 departamentos and Missiones is working in 5 and will add another 5 shortly. The challenge for 5 CIRD will be how to manage the growth of this activity while maintaining their technical momentum in the 12 Alianza communities. CIRD is also working with FRAM on a social insurance scheme and should explore opportunities to replicate this successful model to other areas. The third successful area is dissemination. CIRD is actively implementing networks and exchanges and has achieved the expected result in Year Two. The forums and guided visits for CLS members have contributed to the increased demand for CIRD activities to be replicated in other communities. In addition to network and exchanges, CIRD has published five documents and one video. They are also in the processing of implementing three innovative initiatives. And CIRD has posted five experiences on their webpage in addition to the bimonthly news updates on CLS activities. CIRD has recently started a new activity – Referral/Counter referral (R/CR) system and has therefore limited progress to date in the fourth technical component area. CIRD is still in the design phase of this activity. The current plan proposes CIRD implement a R/CR system at the department level in Itapua. It is still unclear if CIRD will also implement the same activity in one, two, three or all four departments where Alianza is present. CIRD and USAID need to clarify the geographic scope of this activity and performance period. It is important to note that PRIME II will provide the international expertise required by CIRD to define this component and to assist in the implementation as well. CIRD is experiencing difficulties in two important technical areas: health promoters and information, education and communication (IEC). For a variety of reasons, some of which are out of CIRD’s control, the health promoter activity has experienced the least progress to date. First, this activity experienced a slow start-up. Second, there is a “disconnect” between the original design of community health worker with its current implementation. Third, the two institutions responsible for this activity have different visions and approaches to community health promotion. Finally, CIRD has created some confusion among community members by training “volunteer educators” or “promoters” under its IEC component. The Evaluator’s findings show that this strategy is not working due to the fundamental institutional differences. USAID will need to intervene now to determine which organization should assume sole responsibility because two years experience have demonstrated co-sharing implementation no longer works. Despite the problems associated with this activity, the concept of health promoters involving communities in their own health decision making is a sound one, is producing modest results, and should be continued under the Alianza program. If one only counts the number of IEC activities, then CIRD is on track with their IE&C activities. However, closer examination of the IEC activities reveals that although CIRD has implemented a wide range of IEC activities, they have produced limited impact. CIRD recognizes this problem and has taken several steps to address this problem, including: a) changing staff, b) designing an IEC strategy, and c) participating in an IEC working group with other Alianza members. Recommendations to strengthen monitoring of CIRD’s progress In addition to the specific recommendations on how to “tighten” several measurement definitions for certain indicators, the Evaluator recommends that CIRD management take the following steps to ensure consistent tracking, analysis and reporting of the Project’s SO and IRs. They include: 1) designing a project-wide M&E Plan; 2) informing staff of their M&E responsibilities; 3) assigning indicators for staff to monitor and report related to their technical activities; and, 4) reviewing all project documentation to better link USAID’s and CIRD’s and indicators to information presented in the reports. Key Findings: Management support toward achieving results CIRD is doing an excellent job in managing – both technical and operationally - the Health Decentralization Project. A unique organization characteristic of the CIRD Health and Decentralization 6 Project is the clarity of mission and share values. All staff interviewed, starting from the top at the management level down to the administrative level, could articulate, even at times passionately, the project’s overall mission. Also, all the staff shares the same core values which influences positively their technical assistance approach. Moreover, CIRD is implementing the activities efficiently and effectively. As to be expected at the beginning of Year 3, CIRD is in full implementation mode. There are several factors that contribute to CIRD’s successful implementation: 1) the project enjoys strong technical leadership from the Project Coordinator; 2) the Project Coordinator encourages an open and “safe” environment where staff feel free to discuss openly and propose new ideas to address technical and operational issues; 3) the Project Coordinator communicates his technical expectation and standards to staff; and 4) the staff uses technology creatively to manage implementation. Recommendations to improve management The following are two minor suggestions to strengthen implementation: 1) improve coordination of transportation of staff to the field 2) allocate more time for staff to reflect and think strategically. The more substantial recommendations to improve management are: 3) Re-design organizational structure. The current organizational structure can no longer support level of technical activity. During the last year, CIRD experienced dramatic growth -doubling in staff and opening four field offices- yet the organizational structure did not change to match this growth. Also, the Project Coordinator has become a bottleneck because all technical and operational decisions, technical review and oversight, and lines of communication depend on him. The Evaluator recommends that the CIRD Executive Director and CIRD Project Coordinator review the technical and management recommendations and decide how best to restructure the staff, supervision relationships and lines of decision-making and communication. This report provides two models for their consideration. 4) Rethink UOLs roles and responsibilities and assign UOL staff to a technical team: Originally the UOL were established to coordinate all of the Alianza members’ activities in a given community. In practice, some UOLs have become a logistical adjunct to the CLS while others are provide technical support to CLS. As a result, the original objective of integrated coordination of the Alianza members is not happening. It is important to reinforce the original purpose of the UOLs so as to minimize CLS depending on the UOLs to function. Key Findings: Quality of CIRD technical assistance and relations The CIRD technical team receives high marks from their counterparts. CIRD staff demonstrate a common approach and TA style that resonates with the CLS and communities. CIRD staff regard themselves as facilitators who “never impose but instead guide and orient” CLS members. Moreover, CIRD staff have earned their counterpart’s respect and trust based on: a) the high quality and level of technical expertise among CIRD staff; b) CIRD staff commitment and dedication; and c) CIRD staff responsiveness and availability when needed. Moreover, the CIRD staff are a high performing, creative technical team. In reviewing the technical portfolio of activities, CIRD staff apply the latest technologies and methodologies, apply relevant tools and methodologies the counterparts can understand and use and produce high quality technical products. In discussing how the partnership is working between the Alianza members, the Evaluator received the following comments. First, Alianza partners recognize and value the high quality and technical expertise of CIRD staff. The members also indicated that CIRD, as well as other partners, are collaborating better. However, the members also expressed frustration, stating that the current mechanism for technical collaboration among the partners – collaboration meetings – is not working well. Second, technical competition and territoriality still exist among the Alianza members. The members state CIRD technical 7 staff are open and receptive to technical collaboration and that institutional rivalry has improved, but they also indicated that senior-level managers at CIRD still remain competitive and possessive of certain technical domains. Third, the Alianza is working in the field. Some of the CLS understand who the different Alianza members are, what their respective roles are, and are beginning (and only beginning) to coordinate locally. Now the Alianza members need to nurture the synergy between projects, using the technical working groups, UOLs and CLS-oriented programming as coordination mechanisms. Recommendations to strengthen overall technical approach As the CIRD project begins its third year, it is an opportune moment to address directly and plan for sustainability of their technical approach. To address these sustainability issues, CIRD will need to identify now strategies and implement activities that will consolidate, replicate and sustain their technical model. First, CIRD should consolidate their work into a technical model so that others can assume and replicate elsewhere. Second, CIRD should seek out opportunities to experiment in replicating and/or scaling up their technical models, studying how and under what conditions. Third, CIRD should create institutional capacity among others to sustain the technical model. · Consolidate CIRD’s technical model: Steps to consolidate CIRD’s technical approach include: 1) Simplifying the technical model to reduce counterpart’s dependency on CIRD staff. Many of the technical activities are dependent on staff and technical skills external to the CLS to complete the activities. Moreover, the technical model continues to grow in different technical directions. Moreover, if CIRD is going to replicate its approach to several other municipalities/communities, then CIRD needs to reduce the technical approach to its essence so that others can eventually assume the responsibility of forming and training new CLS beyond the 12 Alianza communities. In simplifying the technical model, theCIRD team should remember that FRAM is the exception and that Peguajho and Curupicayty are the norm. 2) Creating mechanisms to better integrate different components of model. The four technical components are implemented as vertical programs, sometimes creating confusion but more often, straining the counterpart’s ability to receive and absorb technical assistance in some many diverse technical areas. There are some simple mechanisms that can facilitate better technical integration, including programming activities by CLS/communities to creates an workplan specific that integrates all technical areas at the CLS level. 3) Remove policy barriers to technical model. Many local, state and national level government officials are taking note of CIRD’s success with the CLS and are requesting CIRD’s to replicate their activities. Even though there exists strong political will, there are some policy barriers preventing expansion of a comprehensive model that include financing. During the last two years, CIRD has increasingly become closer to the MOH through its work with the local and regional MOH officials. Coupled with recent changes in political environment, including a new Ministry of Health expressing interesting in decentralizing, the moment is right to influence national policy within the MOH. CIRD should seize the moment and redirect technical activities under dissemination and program sufficient staff LOE to advocate for policy reforms that will facilitate expansion of locally led health management. · Replicate and/or scale -up the simplified model. Steps to replicate a simplified model include: 1) Focus on task at hand and resist temptation to respond to requests. To achieve focus, the Evaluator recommends CIRD focus the technical approach and stop adding new technical “bells and whistles”. Also, CIRD staff need focus the geographic areas and resist geographic “crawl”. CIRD is under extreme pressure from government officials to work with their communities and should work with USAID manage counterparts’ expectations and disappointment. 8 2) Re-orient TA approach from “implementer” to “mentor”. CIRD staff are working at maximum capacity to meet the needs of the 12 Alianza communities. If CIRD is going to replicate its model and can not add more staff due to budget constraints, then CIRD will have to change the way they provide technical assistance. One suggestion is to change CIRD TA approach from “doing” all the technical interventions to “mentoring” a group who will carry out the activities. Once CIRD identifies an intermediary group to assume the direct TA role, then CIRD staff can assume a mentor role . In this capacity, CIRD can transfer the tools and methodologies to this group and build their capacity to provide training and assistance to the CLS/communities. This underscores the need to simplify the technical model as well as the tools and methodologies so a diverse group of people with varying skills can implement it. 3) Document the model and standardize tools and methodologies. As part of the replication and scale up strategy, CIRD staff will need to take the time to document the simplified, integrated technical model. In addition, CIRD staff should also standardize their approaches in the different technical areas by developing “how to” manuals with companion training guides. · Sustain CIRD’s technical model. Suggestions to ensure sustainability of the technical model are: 1) Experiment with replication and scale up to determine what is required to sustain the technical model. One of the ideas discussed with CIRD staff is to take the technical model to scale in one department. Misiones is a strong candidate for scale -up. Once CIRD decides to scale-up in one department, then they will have to: a) implement their plans to begin graduating some of the 12 Alianza CLS/communities, b) identify local counterparts within “gobernacion” to become the intermediary group, and c) strengthen the local UOL coordinator to assume more of a technical role. In addition, CIRD should document the scale-up process and plan to evaluate it toward the end of the project to identify what worked and didn’t work, and what conditions favor or impede scale -up, etc. 2) Strengthen CIRD/USAID relationship to create a partnership that directs and monitors CIRD’s path toward sustainability. Currently, CIRD project management and USAID enjoy an open and positive relationship. They are in constant communication and discuss a wide variety of topics. The discussions, however, tend to focus the day-to-day tasks and details, with limited opportunity to reflect or discuss new technical directions, challenges confronted during implementation, and changes in the political environment that influence CIRD’s work. Moreover, the communication is informal and frequent, but not regular and consistent. The Evaluator therefore recommends that USAID and CIRD formalize channels of communication, establish a meeting schedule and implement a uniform agenda that covers consistent information. Next Steps This report, although a final step to the mid-term evaluation, should initiate a process between CIRD and USAID to reach agreement on future directions for the years remaining on this contract. This report provides information on a wide range of areas and concrete recommendation to modify and strengthen the Project. In addition, the report offers USAID and CIRD a framework to discuss and agree upon what changes maybe required in the technical scope and cooperative agreement in order to leave a technical program that is sustained by counterparts after the project ends. The CIRD Health and Decentralization Project is off to a strong start, and with some adjustments outlined in this report, is well positioned to end on an even stronger note that will yield impact beyond the original project design, leave a technical legacy of successful examples of locally led health management, and provide best practices for other countries struggling with decentralizing health services. 9 1.0 Introduction USAID signed an agreement (No 526-A-00-01-00100-00) with the Centro de Informacion y Recursos para el Desarrollo (CIRD) to promote decentralization and community participation in health services. In order to achieve this goal successfully, the project focuses on four main areas: a. providing support for the strengthening of selected local governments and local health councils to design, plan, implement and manage health services, b. promoting community participation in health related activities, including a health promoters system and a network of health services (with a referral system), c. designing and implementing reproductive health IEC related activities, and d. implementing dissemination activities. This Health and Decentralization project contributes directly to the USAID/Paraguay Strategic Objective #3 (SO) as described in USAID/Paraguay’s Strategic Plan FY2001-2005: “Use of Reproductive Health Services Increased.” CIRD’s technical activities focus on Intermediate Result (IR) 3.1: decentralized community-based health care improved. This project also supports and complements decentralization activities conducted under the USAID/Paraguay Democracy SO by identifying and training key departmental and municipal government leaders in locally led health management. CIRD is entering its third year under the Cooperative Agreement and like the other USAID’s Alianza projects, is subject to a mid-term evaluation. USAID contacted an external consultant to conduct the mid￾term evaluation during October, 2003. This document presents the evaluation findings and recommendations. Although this report is the final step to the mid-term evaluation, this report should initiate a process between CIRD and USAID to reach agreement on future directions. To facilitate the steps after the mid-term evaluation, the Evaluation Report hopes to provide sufficient information for CIRD staff discuss and decide: a) how to modify their current technical strategies, b) how to organize staff to manage project implementation, and c) how to redirect staff, level of effort and activities to strengthen an already strong technical program with the view toward phasing out over project’s last year. Moreover, this report offers USAID and CIRD a framework to discuss and agree upon what changes maybe required in the technical scope and cooperative agreement in order to leave a technical program that is sustained by counterparts after the project ends. 2.0 Methodology 2.1 Evaluation objectives The objectives of the mid-term technical evaluation of CIRD are multiple: 1. Review the overall project design, the workplans and progress of technical activities from October 2001 to September 2003 to verify if the CIRD program activities are on track to produce the results stated in USAID strategy. 2. Review CIRD’s the progress-to-date in achieving the stated results. 3. Review the Project’s technical management to determine if the technical staff, budget and operations support and are on-track to achieve desired technical results. 4. Interview CIRD “clients” to ensure CIRD is providing effective, timely and quality technical assistance responsive to USAID’s and counterpart’s needs. 5. Provide recommendations to USAID/Paraguay and CIRD management on how to modify and/or focus technical strategies to ensure CIRD activities will achieve results. 2.2 Evaluation questions Below are a set of general questions organized by categories that support the evaluation goals and objectives. The general questions served as the basis for the interview guides. 10 Technical progress toward achieving results · Is the CIRD technical strategy clearly linked to the USAID results? · Are the workplan andactivities clearly linked to the USAID results? · What measurable progress has been made toward the expected results? · Does the monitoring and evaluation plan, process and tools adequately measure the desired results? · If there have been problems achieving the stated results, what have they been and why? · How can the strategy and/or workplan be modified to address these challenges? Management support toward achieving results · Has CIRD hired the staff and consultants with the appropriate skills and experience? Are there enough staff to carry out the workplan within budget? · Do CIRD staff and consultants’ scopes of work job match the technical program? Are the staff fully utilized to their potential? Are the staff producing the desired technical results? · Does the workplan program the right type and mix of technical activities that will lead to the desired technical results? · Does the budget support these activities? · Does CIRD management review workplan activities and budgets on a regular basis to ensure that activities are on-time and on-track to deliver results? Do they review it with the staff? · Do CIRD staff and consultants receive adequate and timely guidance and feedback on their work from Project Director? From USAID? Quality of CIRD technical assistance · Does CIRD provide timely and quality technical assistance that responds to the counterparts needs? · How well do CIRD staff work with the counterparts? With USAID? With other partner organizations? · Do CIRD staff use the latest methodologies and technical approaches in their work? · What is positive and you want CIRD to continue and/or expand? · How can CIRD improve the quality of their technical assistance? 2.3 Evaluation methods The Evaluator applied a variety of methods for the mid-term evaluation. They included: · Interviews: The Evaluator conducted one-on-one semi-structured interviews that lasted from 60 to 90 minutes. Within CIRD, the Evaluator conducted multiple interviews of the CIRD President and Health Project Coordinator. In addition, the Evaluator met with CIRD staff on several occasions to first, review in depth their technical strategies, tools and methodologies and activities realized in the field. The second and third interviews with CIRD staff focused on technical management of activities and discussion of the feasibility of implementing proposed recommendations. Using the same methodology, the Evaluator interviewed many stakeholders of the CIRD Health Project. In Asuncion, the Evaluator interviewed representatives from USAID and Alianza members such as CEPEP and PRIME II. In the field, the Evaluator interviewed CIRD Health project counterparts including government authorities at the Regional and municipal levels, health workers and members of the local health councils. · Site Visits: The Evaluator conducted approximately 20 interviews in the field at 9 out of 12 communities located in all four regions where CIRD is active – Itapua, Misiones, Centrales and Cordillera (Refer to Annex 2 and 4 for a list of the sites). In communities with health clinics, the Evaluator conducted individual, semi-structured interviews with representatives from the 11 municipalities, health establishments and local health councils. At the health post and Farmacias Sociales, the Evaluator grouped together at least one representative from each of the main stakeholder groups in the CLS and conducted the interviews jointly. · Observations: In addition to the interviews, the Evaluator visited with and informally observed CIRD staff in the field as well as the counterparts who have received technical assistance and/or training from CIRD to see if and how the counterparts apply the new skills and knowledge received. · Document Review: Finally, the Evaluator reviewed all relevant documents. A list of the documents is provided in Annex 3. 2.4 Evaluation approach The mid-term evaluation was a three-step process. First, the Evaluator prepared for the trip to Paraguay by finalizing the evaluation methodology, reviewing documents and organizing the trip’s logistics. During the second step, the Evaluator traveled to Paraguay to conduct a series of interviews, meetings and site visits as well as analyze data, statistics, reports and documents. During this phase, the Evaluator finalized the evaluation methodology and site visits, design and elaboration of interview instruments, and collection of necessary data. In the third step, the Evaluator debriefed key stakeholders, solicited their input on and documented the findings and conclusions. 12 3.0 Evaluation Areas Section Three provides the analytical content of the evaluation and covers three evaluation areas: · Technical progress toward achieving results · Management support toward achieving results · Quality of CIRD technical assistance The discussion of these three evaluation areas also contains observations and suggestions on how to improve each of these areas. However, the main recommendations on how to strengthen the technical strategy and activities are located in Section Four. 3.1 Technical progress toward achieving results Section 3.1 reviews CIRD’s progress in achieving the results. The results described in this section come from the USAID Results Framework for the SO indicator and the CIRD Monitoring and Evaluation (M&E) Plan. It is important to note that the CIRD M&E Plan exceeds USAID’s requirements. The evaluation focuses only on a core number of indicators (9) that USAID needs to report to it’s SO and monitor the Cooperative Agreement. The evaluation also attempts to identify any methodological issues associated with the core indicators to help CIRD meet it’s reporting obligations. First, this section presents several diagrams that map the key relationships between USAID Strategic Framework and CIRD’s technical strategies and activities. The relationships include: · relation between USAID Strategic Framework and CIRD’s project · relationship between CIRD’s Strategic Objective (SO) and four intermediate results (IRs) · relationship between CIRD’s SO, IRs and technical activities · relationship between CIRD’s four IRs and core indicators. The diagrams serve multiple purposes. First, they help clarify for staff the connection between USAID’s goals and objectives and CIRD’s technical activities. Second, the diagrams clearly link the nine core indicators to their technical scope, underscoring each staff’s responsibility to collect and analyze his/her indicator. Third, the diagrams highlight the issues related to the nine core indicators and their measurement definitions that need to be resolved in collaboration with USAID. The second part of this section reviews each indicator and presents evidence supporting CIRD’s progress￾to-date. The section concludes with recommendations on how to finalize the CIRD results framework, clarify some of the measurement definitions, monitor and report the nine indicators. 3.1.1 Relationship between CIRD technical activities and indicators Diagram 1 provides a visual overview of the relationship between USAID’s Strategic Framework and CIRD’s technical strategies. CIRD directly supports USAID SO #3 and is the Alianza member responsible for IR 3.1. “Decentralized community based health care provided”. It is important to note that CIRD also indirectly supports USAID’s SO #1. Given this dual supporting role, CIRD management has expressed a desire to not only work with the SO#3 team but to also be included as part of the SO #1 strategy and activities. Diagram 2, in turn, becomes CIRD’s strategic framework. In Diagram 2, USAID’s IR 3.1 is CIRD’s overall SO. Diagram 2 maps out CIRD’s four technical components and translates them into IRs: · IR 1.0 Strengthened local capacity to manage and oversee decentralized community based health services · IR 2.0 Increased community participation in own health care and decision-making · IR 3.0 Increased demand for decentralized community based health services through IEC activities · IR 4.0 Increased support for decentralized community based health services through lessons learned & dissemination of successful examples 13 Diagram 1: Mapping USAID Strategic Framework to CIRD Results Framework USAID SO #1: Institute key democratic practices USAID SO #2: IR 3.1: Decentralized community based health care provided Strengthened local capacity to manage and oversee decentralized community based health services Increased community participation in own health care & decision-making Increased demand for decentralized community based health services through IEC activities Increased support for decentralized community based health services through lessons learned & dissemination of successful examples USAID SO #3: Use of voluntary FP/RH services increased 14 Diagram 2: Project Overview for CIRD Health Decentralization and Community Participation Project Inform key stakeholders on successful examples of decentralized basic health services Build support for decentralized basic health services Demonstrate examples of replication and/or scale of successful model(s) of decentralized basic health care Create promoters network organize & mobilize communities on health Increase public’s knowledge of rights and responsibilities in health care Increase community knowledge in select # of preventive health behaviors, with emphasis on FP/RH Increase community acceptance and practice in select # of health preventions, with emphasis on FP/RH Transfer skills to promoters to resolve & monitor community’s health in select # of preventive health behaviors Help promoters to link community & CLS partners Establish a Referral System to ? Need to define objective of activity SO: Decentralized community based health care provided Assist CLS to become more transparent & accountable to communities Assist CLS to mobilize and manage financial resources Assist CLS to prioritize & redirect health resources Assist CLS to function and become operational 3.0 Increased demand for decentralized community based health services through IEC activities 4.0 Increased support for decentralized community based health services through lessons learned & dissemination of successful examples 1.0 Strengthened local capacity to manage and oversee decentralized community based health services 2.0 Increased community participation in own health care & decision-making I R s S t r a t e g i e s 15 In addition, Diagram 2 lists the different strategies to achieve the four IRs. The Evaluator worked with the respective technical teams for each of the components to draft the IRs and strategies. As Diagram 2 demonstrates, IRs 1.0 and 2.0 did not experience many changes from the original proposal. On the other hand, the technical teams modified the strategies and activities for IRs 3.0 and 4.0 based on field experience and the recommendations stemming from this evaluation. Diagram 2 quickly illustrates design issues associated with IR 2.0. In the area of community health promoter, the first three strategies (create promoter network, organize & mobilize communities on health, and transfer skills to promoters to resolve & monitor community’s health in select # of preventive health behaviors) depend on CEPEP to carry out these activities in order for CIRD to achieve its results. This type of design - assigning co-responsibility between two different institutions - usually creates problems, even under the best of circumstances. Please refer to Section 4.1 for recommendations on how to resolve the problems associated with joint implementation of the Health Promoters activity. In addition, the new activity of establishing referral and counter-referral systems (R/CR) has been placed under IR 2.0. This activity does not “neatly” fit under this IR 2.0. As a result, the CIRD team need to define it’s objective as it relates to the expected result for IR 2.0. Currently, the R/CR model is under discussion with USAID and hopefully the discussion will clarify what is the R/CR objective. Once defined, the objective should be reflected in this diagram as well as other project reports. Diagram 3 organizes CIRD’s SO and IRs, strategies and activities, providing a road map for CIRD staff on how and where their activities and technical scopes fit into the CIRD results framework. Also, Diagrams 3 helps staff focus their technical activities, presenting criteria (expected results) by which to select or eliminate new activities. For example, if the new or proposed technical activities does not directly support the strategy or help achieve the IR, then CIRD staff should not consider adding it to their already ambitious portfolio of technical interventions. This diagram will become a helpful tool for CIRD staff as they move towards focusing and simplifying the technical model, as discussed later in Section 4.1. 3.1.2 Recommendations to strengthen CIRD indicators and/or measurements Diagram 4 links CIRD’s SO and IRs to the core nine indicators CIRD needs to track for USAID. The results framework assumes that if CIRD staff achieve and successfully track Indicators #2 through #9, then CIRD will meet the requirements for Indicator #1. Below is a discussion of the indicators and their measurements by IRs. The discussions also contain recommendations for modifying the indicator or clarifying the measurement. SO: As Diagram 4 demonstrates, CIRD is responsible for 15 communities, including the 12 Alianza communities, Filadelfia and Fram IR 1.0: Indicators #2 and #3 support IR 1.0. Also, Indicator #8 is counted under IR 1.0 and IR 4.0. IR 2.0: Aside from the design issues raised in the discussion for Diagram 1, CIRD needs to define what does “in coordination with CEPEP” mean and how to measure it. Examples of possible measurements include: number of coordinating meetings, # of joint activities implemented, etc. Furthermore, CIRD needs to propose how to define Indicator #5 in terms of number of departments and over what time period. 16 Diagram 3: CIRD Health Decentralization and Community Participation Project Strategy and Activities Activities: · Farmacia Social · Seguro Social · SIG · SIAF · 2.1: Create promoter network to organize &and mobilize communities on health 2.3: Help promoters to link community & CLS partners 2.2: Transfer skills to promoters to resolve & monitor community’s health in select # of preventive health behaviors CEPEP Activity CEPEP Activity Activities: · Unclear 4.1: Inform key stakeholders on successful examples of decentralized basic health services 4.2: Build support for decentralized basic health services Activities: · Functionalidad · Institutionalidad Activities: · Ntl. level policy, advocacy @ state level, guided visits · Institutionalize ? Need to define 4.3: Demonstrate examples of replication and/or scale of successful model(s) of decentralized basic health care Activities: · Plan Local de Salud · Paquete Basico Activities: · Webpage, Informes, Publications · Institutionalidad 3.1: Increase public’s knowledge of rights and responsibilities in decentralized basic health 3.2: Increase community knowledge in select #, including FP/RH, preventive health behaviors 3.3: Increase community health prevention behaviors, including FP/RH Activities TBD Activities: SO: Decentralized community based health care provided 1.0 Strengthened local capacity to manage and oversee decentralized community based health services 2.0 Increased community participation in own health care & decision-making 3.0 Increased demand for decentralized community based health services through IEC activities 4.0 Increased support for decentralized community health services through lessons learned & dissemination of successful decentralized basic health services 1.1: Assist CLS to function and become operational 1.3: Assist CLS to mobilize and manage, resources, including finances 1.2: Assist CLS to prioritize & redirect health 1.4: Assist CLS to become more transparent & accountable to communities Activities: · Parte Interna · Memorias, etc. · 2.4 Establish a Referral System to ? Need to define objective of activity Activities: · Doc. T & M · Pilot 1 – 2 cases & document · Presentations, papers, seminars 17 Diagram 4: CIRD Health Decentralization and Community Participation USAID Indicators & Measurements 1# : number of communities offering health services which include local health plans, a data collection process, a defined basic health care package, a system of promoters & a functioning financial/management information systems #8: 12 activities to promote networks and exchanges of information between local actors implemented · Need to define how many activities? over what time period? #6: # of IEC & Promotion activities implemented · Need to better define measurement: How many? How often? Where? #9: At lease 15 dissemination activities designed & implemented · # of publications · # of innovative initiatives · # of experiences available on webpage #7: Demand for FP/RH services increased · Need to determine if this is a CIRD or a PRIME indicator · # community census · # communities w/info on production, costs, etc. · # communities in PLS · # of PLS designed · # health surveys · # basic health package defined · local health insurance · rotating funds · cost recovery #5: Referral/counter referral system implemented in select # of departments · Need to define # of departments over what time period Decentralized community based health care provided 1.0 Strengthened local capacity to manage and oversee decentralized community based health services 2.0 Increased community participation in own health care & decision-making 3.0 Increased demand for decentralized community based health services through IEC activities 4.0 Increased support for decentralized community based health services through lessons learned & dissemination of successful examples #2: CLS, authorities & health facilities technically strengthened to oversee health services #3: 3 finance models implemented #4: Health promoter network implemented in coordination with CEPEP · Need to define measurement: ie, what does “in coordination” mean? #8: 12 activities to promote networks and exchanges of information between local actors implemented 18 IR 3.0: Indicators #6 and #7 support the IEC intermediate result. CIRD needs to better define the measurement for Indicator #6 in terms of: How many activities? How often? Where? Indicator #7 is more complicated because it is unclear which Alianza member is responsible for collecting data to measure demand -CIRD or PRIME? Either way, CIRD needs to clarify with USAID who is responsible for this indicator. Based on USAID’s decision, CIRD will either start collecting the data or turn to PRIME for that piece of data to complete CIRD’s reports. IR 4.0: Indicators #8 and #9 support the Dissemination IR. Given the technical recommendations to expand the scope of these activities, the Evaluator proposes renaming this IR to better reflect the policy and advocacy work CIRD will be performing in the next 2 ½ years. Moreover, CIRD will also have to better define the measurement definitions for both these indicators to reflect these changes. 3.1.3 Recommendations to strengthen monitoring of CIRD indicators In addition to the specific recommendations on how to “tighten” several measurement definitions for certain indicators, the Evaluator recommends that CIRD management take the following steps to ensure consistent tracking, analysis and reporting of the SO and IRs: · Inform staff of project-wide M&E responsibilities. This can be easily achieved by 1) distributing copies of the USAID Strategic Framework and Diagrams 1 – 4 to all staff, and 2) reviewing the information and discussing its implication during one of CIRD’s staff meeting. · Design a project-wide M&E Plan. The project needs to start collecting in a systematic way the data to report the indicators. To this end, the M&E Specialist can develop a simple M& Plan. Annex 5 provides an illustrative M&E Plan based on several USAID globally funded project’s M&E Plans. The key is to keep the Plan simple so it can inform staff how, when and where they collect data. The Plan also services to standardize the collection, analytical and reporting processes so the client receives consistent and reliable information on a regular basis. · Assign M&E responsibilities. Although there is a M&E Specialist charged with the responsibility to implement all of the M&E activities, all technical staff should assist the M&E Specialist to fulfill this function by taking individual responsibility to collect, analyze and report out on their respective technical areas’ indicators. To help the staff, the M&E Specialist should meet one-on-one with each technical team to: 1) review the M&E Plan, their technical indicator(s) and the measurement definition(s), 2) discuss data sources and collection methodology, 3) program when data should be collected, analyzed and reported and 4) agree upon how, in what format and when the staff will report on their indicator. · Review all project documentation to see if they can better link USAID’s and CIRD’s, indicators to the information presented in the reports. The key project documents are the annual workplan and quarterly report. CIRD management and the M&E Specialist can review them to determine how to make clearer connections, identify result “goals” for the year, and discuss how and when to report these results. 3.1.4 Progress in achieving results This sections reviews the data to determine how well CIRD is on track to achieve the results as measured by the core set of indicators. The data for this section comes from CIRD project documents including the Quarterly Reports, CIRD project documents, staff interviews and site visits. The text is organized by indicator. 19 Indicator #1: Twelve (12)communities offering health services which include local health plans, a data collection process, a defined basic health care package, a system of promoters & a functioning financial and management information systems. In reviewing Table 1, CIRD is well on its way to achieve the SO, as measured by Indicator #1. By the end of Year Two, seven of the twelve Alianza Consejo Local de Salud (CLS) are functioning1 at or above medium level proficiency. A fully functioning CLS is defined as able to articulate the CLS’ functions; CLS members’ roles and responsibilities; identify a couple of the community health priorities; and describe a few of the Local Health Plan’s activities. Given their progress at the end of Year Two, CIRD will more than likely exceed the expected result. During the site visits, CLS members were very eloquent in describing how CIRD’s technical assistance has helped the CLS as a group and as individuals. Comments included: “CIRD opened our eyes”. “CIRD helped us know what to do and how to do it”. “They taught us how to work together”. As one CLS member said, “Before CIRD, I was confused and did not know what to do. Now I am confident”. Indeed, some CLS are so confident in their newly acquired roles and responsibilities that they are in the process of replicating their experience to form sub-consejos (for example, Atyra and Arazape). In general, the CLS stated they acquired the following skills: · How to develop CLS norms and statues · How to form a CLS or SCLS · How to operate as a team · How to develop a workplan · How to work with the community Moreover, the CLS members feel better informed on: · Community health needs · Mayor’s planning and the budget process · Health clinic/post service constraints · Community composition Concrete results demonstrating CIRD’s success in forming and ensuring thate CLSs are functioning include: · Half of the CLS stated that their community is aware of the CLS role and functions. · Half stated that they now have better relations and more influence with mayor’s office. · Most CLS stated that they have greater credibility with communities. · Most CLS members stated they are better informed about their community and health needs. · Most CLS stated that their communities are better informed about their health needs. · Most stated that the community want greater participation in the CLS, as evidenced by the increased number of individuals and groups attending CLS sponsored events and asking to become CLS members. The most tangible evidence of CIRD’s success is the continuous demand from mayors and governors to replicate CIRD’s activities in their respective city and/or state. CIRD has become, in a positive sense, a victim of their own success and is under pressure to respond to the demand. This has created an inner tension for management and staff since they are extremely responsive, a hallmark characteristic of their relations with their counterparts, but they do not have the capacity to reply. The Evaluator discussed with CIRD the need to enlist USAID’s help to balance the new requests for technical assistance. 1 Functioning is measured by proficiency levels that range from low, low-medium, medium, medium-high, high. 20 Indicator #2: CLS, authorities & health facilities technically strengthened to oversee health services CIRD is on track to achieve Indicator #2 and will more than likely exceed it by the end of the project. As Table 1 demonstrates, ten out of the twelve Alianza communities have completed the majority of technical interventions: Censo Sanitario, Plan Local de Salud, diverse community participation, Phase #1 of Basic Health Package, and Health Facility Survey. There are three exceptions. Ita is listed as dysfunctional due to political infighting and unwillingness to implement CLS as envisioned by the law (for example, they are unwilling to open up the CLS to community participation). Ita requires a decision, more than likely to terminate TA CLS because of they will not change no matter the level of assistance. Cristo Rey and Curuani, are not functioning because CIRD has just recently started activities with these two communities. But they should up and running by the end of the project. Table 1: Data for Indicators #1 and #2 Municipalities CLS level of functioning Censo Sanitario Plan Local de Salud # of communities involved in PLS Basic health package defined Communities with management data Health Facility Survey San Miguel Med ¸ Implementing 4-5: Schools, political party, Artisan Coops ¸ Phase 1 SIG ¸ Arazape Med-High ¸ Implementing 8: Schools, Church, Coops, Agriculture Committees ¸ Phase 1 FS ¸ Ita Yuru Med ¸ Implementing 8: Plan Lechero, Church, Schools, Junto de Sanimiento, ¸ Phase 1 FS ¸ Cl. Bogado Med ¸ Implementing 6-8: Reps from Orgs representing civil society ¸ Phase 2 SIG/SIAF ¸ Cristo Rey In formation ¸ In formation N/A N/A ¸ Curunai In formation ¸ In formation N/A N/A ¸ Ita Disfunctional ¸ Implementing 2: Political parties only ¸ Phase 1 SIG in process ¸ Peguajho Med ¸ Implementing 5: Women & Youth Groups, Church, Ag Committee ¸ Phase 2 ¸ Curupicayty Med ¸ Implementing 5: Women & Youth Groups, Church ¸ Phase 2 ¸ Aytra Med-High ¸ Implementing 5: Women Groups, Junta de Sanimiento, School ¸ Phase 1 SIG in process ¸ Candia Low ¸ Implementing 5: Women & Youth Groups, Church, Ag Committee ¸ Phase 1 ¸ Bernadino Caballero Low ¸ Implementing 5: Women & Youth Groups, Church, Ag Committee ¸ Phase 1 ¸ The CIRD technical team for IR1 indicated that the most labor intensive activity has been increasing community participation in the CLS. For a variety of reasons, increasing community participation has not been easy to implement. Principal among them have been: 1) CLS member resistance to the concept 21 and/or 2) lack of skills on how to include community representation. Despite this challenge, Table 1 indicates that CIRD has achieved its goal of diversification of community representation. Communities are either directly represented by individuals, like in Atyra or Arazape, where they opened participation through elections at a general assembly or indirectly, like Coronel Bogado and FRAM, where the CLS invited leaders who represent different community organizations. One finds the full gambit of community groups participating in the CLS, ranging from different political parties, to woman’s group, to trade associations (agricultural and artisan cooperatives), to other public sectors such as education and water sanitation, to religious groups. Sustaining all the 12 CLS at above medium functioning level will continue to challenge CIRD. The CLS capacity to operate varies widely. At one end of the spectrum, there is FRAM which represents the gold standard on how a CLS should operate. At the other end, there is Candia , who with technical assistance still struggles. There are several factors explaining why some CLS function better than others. Interestingly, education and experience are not essential conditions for a CLS to succeed. For example, FRAM CLS members are well educated, have more experience in governance and management and are more sophisticated as a group. Arazape CLS members, on the other hand, come from a more modest community and do not have the same level of sophistication and experience, yet they are functioning at a high proficiency level also. Then there is ITA where the CLS members are more educated, professionals with more governance experience yet they are completely nonfunctional. FRAM and Arazape CLS members offered the following insights as to why their CLS function well: · A CLS requires strong political support at the municipal or state levelto succeed · All CLS members have to leave their political party and religious affiliation “at the door” · CLS members share a common vision of “stewardship” and “responsibility” for their community’s health · There exists a strong tradition, either within the community or local government, of leadership · CLS members demonstrate mutual respect for each other and for each other’s differences The challenges going into the future will be: a) defining a level of operation at which point, CIRD staff can slowly withdraw their technical support, b) designing a strategy to slowly graduate functioning CLS, c) determining if the foundation developed by CIRD staff will endure after turn-over and new members take office in the CLS, and d) developing a TA package that others can apply to quickly form new CLS that function. Indicator #3: three (3) finance models implemented As Table 2 demonstrates, CIRD has achieved indicator #3 by the end of Year Two. And depending on how you define three finance models, CIRD may have exceeded it. CIRD is working in all 12 Alianza communities to implement one of the three different types of finance models: 1) Fondos Rotorios to finance Farmacias Sociales, 2) Social Insurance and 3) Cost Recovery. However, if one counts coverage and impact, then CIRD has exceeded the expected results with its Fondos Rotorios/Farmacias Sociales. Fondos Rotoraios to finance the Farmacias Sociales is a huge success, demonstrating this activity can go to scale. CIRD developed a simple program that can be implemented in a low resource setting by staff with limited education and no professional experience in the subject area. As a result, the program is spreading like wildfire: Itapua has taken the activity to scale and developed a department level infrastructure (parque sanitario) that supports 86 Farmacias Sociales. Misiones has five Farmacias Sociales operational and is currently adding another five. The project is going so well in Misiones that the mayors, unprompted by CIRD, grouped together to appeal to the governor for funding, staff and infrastructure to take this activity state-wide. The mayors interviewed in Misiones expressed confidence the governor will approve their request in ’04 and are preparing for this eventuality. In addition, the 22 newly appointed health advisor for the newly elected governor of Central (who is a doctor) has been lobbying CIRD to help them implement Farmacias Sociales for their state. Table 2: Data for Indicator #3 Municipalities Farmacias Sociales Rotating Funds Extended Farmacias Sociales Rotating Funds Social Insurance Others San Miguel* 3 Sta Maria Arazape* 3 San Patricio Ita Yuru* 3 Coratei Ayolas Cl. Bogado 4 Itapua – 86 operating Cristo Rey In process Farmacias Curunai In process Sociales Ita** 3 Peguajho** 3 Generacion de fondos Curupicayty** 3 Generacion de fondos Aytra ? Candia 2 Sta Rosa San Ignacio Bernadino Caballero 2 San Ramon Santiago Villa Florida FRAM 4 ¸ Social Pharmacies are ranked on a scale of 1 – 4, 1 representing in formation and 4 autonomous. * Will be level 4 in 2 years. **Will be level 4 in 1 year. Farmacias Sociales are popular because they are a “win-win” for all participants. The Farmacias Sociales provide legitimacy for mayors and other politicians with their constituency. Mayors, in a short time period, can demonstrate to the electorate a concrete program that responds to an immediate and perceived need. The Farmacias Sociales provide credibility for the CLS, permitting the CLS to show that they can deliver and manage a program that is popular with the community. And the Farmacias Sociales are busy with clients because they provide affordable and geographical accessible medication. The challenge for CIRD will be how to manage the growth of this activity while trying to maintain their technical momentum in the 12 Alianza communities. CIRD is also working successfully with FRAM on a social insurance scheme. FRAM is not part of the 12 Alianza communities but CIRD/USAID agreed to include FRAM because they offer an important opportunity to explore a promising new financing scheme. Social insurance, however, has limited options because there is a policy constraint restricting health establishments from collecting and retaining funds at the facility. FRAM secured special permission through a contract signed with the state, allowing it to collect and spend funds. Despite the policy limitations, social insurance merits further exploration and will require addressing the policy constraint if CID is going to replicate this activity to other communities. Given the success of the two finance models, CIRD and USAID need to discuss what will be the future direction for health financing activities under this cooperative agreement and to modify the activities in the next two years to achieve the new goals. Should CIRD focus only on scaling up Farmacias Sociales to other states? Or should CIRD replicate the FRAM model in another state to demonstrate to central 23 level MOH’ officials the success of social insurance schemes? Based on the decision, CIRD and USAID will need to clarify the indicator measurement. Indicator #8: 12 activities to promote networks and exchanges of information between local actors implemented Indicator #8 is the only indicator counted in two places: IR 1 and IR 4. Under IR 1, CIRD is on its way to achieving the indicator. Table 3 illustrates that CIRD has conducted three types of activities to date: regional forums, study tours and meetings. Table 3: Data for Indicator #8 under IR 1.0 Event Description Date Regional Forums · CIRD, in collaboration with USAID, GTZ, OPACI, AJUMPA, conducted seven Regional Forums to train newly elected mayors and municipal boards on CLS. April and May 2002 Study tours · CIRD conducted three guided visits for Mayors and CLS members to see Farmacias Sociales in Misiones and Itapua and social insurance programs in FRAM. Meetings · 3 Meetings of SIG operators encouraging a network of information and experience exchange · 2 Meeting of CLS of Misiones (6 municipalities) · 20/12/01,16/04/02, 09/05/03 · Jun,2002 and Jul, 2002 These activities have been very successful. Many CLS members interviewed expressed how influential the tours and forums were in: a) motivating them to either form a CLS and/or implement a Farmacia Social, b) improving their understanding of their roles and responsibilities, and c) observing how a functioning – FRAM – CLS operates. These forums and guided visits have also contributed to the increased demand for CIRD activities to be replicated in other communities. CIRD anticipates implementing more of these exchanges during the next two years and will surpass this result. Indicator #4: Health promoter network implemented in coordination with CEPEP The Health Promoter activity is one of CIRD’s weakest technical areas. For a variety of reasons, some of which are out of CIRD’s control, this activity experienced a slow start-up. In the first year, CEPEP and CIRD worked together to design the program, develop the training modules, and design the supervision model and registration system. In the first two years, CEPEP -the lead on this activity- experienced constant turn-over in staff, contributing to the slow start. As a result, there are health promoters working in the Misiones region only. In Misiones, there are 27 promoters, 5 supervisors and 7 new trainees in San Miguel-urbano; 7 promoters, 2 supervisors and 5 new trainees in Arazape; and 4 promoters, 2 supervisors, and 5 new trainees in Ita Yuru. CEPEP is actively training health promoters in Itapua and Cordillera. They have recently completed training in November 2003 of 22 new health promoters in Coronel Bogado. And CEPEP will start training in Atyra centro, Candia and Bernardino Caballero at the end of this year. The second reason for underperformance in this area is the “disconnect” between the original design of community health worker with its current implementation. According to the Program Design, the goal of the community health promoter is to “contribute to improving the population’s health in the 12 Alianza communities through a system of community health agents through promotion and prevention”. The concept paper goes on to list several tasks that comprise the community health workers tasks. In essence, the community health promoter works with the families in their community to identify health problems, promote key healthy behaviors and refer health problems/risks to the closest health facility. What has emerged, however, is something different. The health promoter has become a hybrid between a health 24 educator, IEC volunteer, and provider of basic health services and products. Consequently, there is no consistent standard for health promoters’ roles and functions. A third factor contributing to the underperformance in this technical area is the different institutional visions and approaches to community health promotion. Although both organizations had work together in defining the profile of the promoters, two different perspectives still remain among the system generating a “complicated and confusing approach”. Finally, CIRD has created some confusion among community members by training “volunteer educators” or “promoters” under its IEC component. The Evaluator findings show that this strategy is not working due to the fundamental institutional differences. USAID will need to intervene now to determine which organization should assume sole responsibility of implementing this activity because two years experience have demonstrated co-sharing implementation no longer works. See Section 4.2.1 for more discussions on next steps. Despite the problems associated with this activity, the concept of health promoters involving communities in their own health decision making is a sound one and producing modest results. In Misiones, the CLS members: 1) “perceive2 ” there has been an increase number of clients seeking health services at their health post and/or clinic because promoters are referring them (in most cases, the promoters go with client); 2) state that the community health promoters have helped promote CLS and explain CLS role and activities to the community; and 3) understand their role in programming and supervising the health promoters’ activities. Linking CLS and health promoters is also working in two out of the three Misiones’ communities: Both CLS members in Arazape and Ita Yuru work very closely with the health promoters and incorporate the health promoter activity into their annual workplans. Indicator #5: Referral/counter-referral (R/CR) system implemented in select # of departments CIRD is still in the design phase of this activity. The current plan proposes CIRD implement a R/CR system at the department level in Itapua. It is still unclear if there is discussion or an expectation that CIRD will also implement the same activity in one, two, three or all four departments where Alianza is present. CIRD and USAID need to clarify the geographic scope of this activity and performance period. Once clarified, then they also need change the indicator measurement accordingly. Until the measurement is clarified, the Evaluator can not conclude if CIRD can or will achieve it during the life of the project. It is important to note that PRIME II will provide the international expertise required by CIRD to define this component and to assist in the implementation as well. Indicator #6: # of IEC & Promotion activities implemented If one only counts the number of activities, then CIRD is on track with their IE&C activities. Table 4 presents all the IEC activities conducted in the last two years. But closer examination of the activities reveals that although CIRD has implemented a wide range of IEC activities, they have produced limited impact. CIRD recognizes this problem and has already taken steps to address this problem, including: a) changing staff, b) designing an IEC strategy, and c) participating in an IEC working group with other Alianza members. Section 4.2.3 provides a more in-depth discussion on how to strengthen the IEC component and also reviews CIRD’s staff capabilities to carry out the IEC activities. Notwithstanding the actions taken to strengthen this component and recommendations to redirect CIRD’s IEC activities, USAID and CIRD need to better define the indicator measurement. Counting the number of IEC activities implemented will not measure whether the activities produce any results. Strategies to better define the indicator and its measurement include: a) clarifying How many activities? What type of activities? How often? 2 The evaluator did not have time to substantiate the increase in clients but it would merit closer examination by CIRD staff by reviewing clinic records and/or counting promoters’ referral slips. 25 Table 4: Cumulative Record of IEC events and materials (2002, 2003) for Indicator #6 Municipalities Implementing Agencies # of IEC events Others Grupo de Apoyo San Miguel* PREVER · 10 events w/youth leaders ¸ (-) CECTEC · 2 events w/CLS · 30 events w/youth · 10 events w/couples · 5 events training grupo de apoyo · 20 events w/women 15-45 yrs. Arazape* PREVER · 10 events w/youth leaders · ¸ (+) CECTEC · 2 events w/CLS · 30 events w/youth · 10 events w/couples · 6 events training grupo de apoyo · 25 events w/women 15-45 yrs. · Support for Vaccine & Dengue campaign · Inauguration of Farmacia Social & UOL office · Special events: Fiesta Patronales · Media presence · Materials: 5000 posters, 6200 volantes, 2550 remeras, 8000 tripticos, 3000 calcomanias, etc. Ita Yuru* PREVER · 10 events w/youth leaders ¸ (++) CECTEC · 2 events w/CLS · 20 events w/youth · 10 events w/couples · 6 events training grupo de apoyo · 10 events w/women 15-45 yrs. Cl. Bogado PREVER · 15 events w/youth leaders ¸ (+) CECTEC · 10 events w/youth · 10 events w/couples · 3 events training grupo de apoyo · Inauguration of UOL office · Support for Dengue campaign · Support to Escuela para Padres · Materials:2200 volantes, 2150 posters, 1000 dipticos Cristo Rey CECTEC · 10 events w/youth In formation Curunai CECTEC · 10 events w/youth See above (-) In formation * Check mark indicates presences of volunteer group; (-) indicates weak volunteer group, (+) medium strength volunteer group, (++) strong volunteer group 26 Table 4: Continued Municipalities Implementing Agencies # of IEC events Others Grupo de Apoyo Ita PREVER · 10 events w/youth leaders (-) In process PROMESA · 4 events w/CLS · 5 events w/youth leaders · Materials: 250 remeras, 500 posters, 2000 calcomanias, 5000 dipticos Peguajho PREVER · 10 events w/youth leaders · 5 events w/adults ¸ (+) PROMESA · 5 events w/CLS · 6 events w/adults · 12 events w/youth · Support for Desparasitacion & fluorizacion dental in 2 schools · 4 focus groups · Special events: Sports events, inauguration of UOL, · Media presence · Materials: 600 remeras, 4 carterle s, 100 posters, 1500 calcomanias, 1500 dipticos Curupicayty PREVER · 10 events w/youth leaders · 5 events w/adults ¸ (+) PROMESA · 5 events w/CLS · 6 events w/adults · 12 events w/youth · Support for Desparasitacion & fluorizacion dental in 2 schools · 4 focus groups · Special events: Sports events, inauguration of UOL, · Media presence · Materials: 600 remeras, 4 carterles, 100 posters, 1500 calcomanias, 1500 dipticos Aytra PROMESA · 10 events w/adults · 5 events w/youth · Special events: Sports events, inauguration of UOL, · Media presence · Materials: 9000 dipticos, 3000 calcomanias, 50 guias, 3 spots, 100 garbage cans (-) In process Candia PROMESA · 5 events w/adults · 5 events w/youth (-) In process Bernadino Caballero (-) In process * Check mark indicates presences of volunteer group; (-) indicates weak volunteer group, (+) medium strength volunteer group, (++) strong volunteer group 27 b) identifying indicators that measure the common results expected from IEC activities, such as increased awareness, increased knowledge, change in certain behaviors USAID and CIRD can modify this indicator after CIRD has designed its IEC strategy, as proposed in Section 4.2.3, thereby linking the proposed activities to indicators. Indicator #7: demand for FP/RH services increased There is some confusion surrounding this indicator. First, most of CIRD staff are unaware that they are responsible for this indicator and as a result, are not collecting data. Second, PRIME II collects data on demand for their project. CIRD and USAID need to clarify soon whether increased demand is one of CIRD’s core indicators, and if it is, does CIRD need to collect the data for this indicator when PRIME II already collects it. The Evaluator recommends that this indicator remain in CIRD’s core set of indicators but CIRD rely on PRIME for the data. Indicator #8: 12 activities to promote networks and exchanges of information between local actors implemented As mentioned before, Indicator #8 is located under both IRs 1.0 and 4.0. Under IR 4.0, CIRD is on track and will exceed it the expected result. As Table 5 illustrates, CIRD has already implemented at least 6 advocacy activities. It is important to note that the technical staff and Evaluator reshaped the technical interventions based on new objectives for this technical area (see below). Table 5: Data for Indicator #8 under IR 4.0 Interventions Activities Inform key stakeholders on successful examples of decentralized basic health services · 2 national workshops on national experience with community promoters · Joint workshop with UNFPA for communicators · Presentations at the request of different mayors and municipalities Build support for decentralized basic health services · Jornadas Departmentales de Informacion y Dialogo on Health Reform and Decentralization · Graduate courses in different health reform and decentralization issues Demonstrate examples of replication and/or scale of successful model(s) of decentralized basic health care · FRAM Presentation, Chile Indicator #9: At least 15 dissemination activities designed & implemented Indicator #9 supports and reinforces the policy communication focus of Indicator #8 by providing dissemination materials. Table 6 demonstrates that CIRD is well on its way to exceeding the expected result. They have published 5 documents and one video. They are in the processing of implementing three innovative initiatives. And CIRD has posted 5 experiences on their webpage in addition to the bimonthly news updates on CLS activities. The types of activities are well defined, as reflected in Table 6. 28 Table 6: Data for Indicator #9 Interventions Activities · # of publications · Informes (3 to date): “Informativo de Salud” · Publications: a) Paquete Basico, b) Farmacias Sociales · Video on Farmacias Sociales · # of innovative initiatives · Scale up of Farmacias Sociales in Itapua · Scale up of Farmacias Sociales in Missiones · Integrated model in FRAM · # of experiences available on webpage · Have posted 5 experiences · Farmacias Sociales experience in Itapua · Application of SIG · Reglamientos for Missiones and Ita · Paquete Basicos · PLA for 12 municipalities · Trimester news updates on CLS activities (24 to date) 3.2 Management support toward achieving results This section reviews any potential issues associated with management systems and organizational structure that would help or impede their ability to achieve results. The Evaluator conducted a series of individual meetings with all the technical staff to review: a) organization structure, b) decision-making processes, c) lines of communication, and d) roles and responsibilities. The Evaluator did not review administrative and financial policies since they will be evaluated by an outsider auditor. 3.2.1. Key findings on CIRD management Listed below are several key findings on CIRD’s management and organization. This section concludes with recommendations to strengthen CIRD’s organizational structure. Cleary defined and shared mission and values A unique organization characteristic of the CIRD Project is the clarity of mission. All staff interviewed, starting from the top at the management level down to the administrative level, could articulate, even at times passionately, the project’s overall mission. This translates into a unified perspective on the project’s technical approach, greatly assisting the staff to work as a team toward a common goal or in this case, common results. Also, all the staff shares the same core values which influences positively their technical assistance approach. Core values include: community focus, field driven, responsiveness to the community, team work, trust and transparency. These values also shape how CIRD staff interact and treat each other as well as the quality of the technical assistance. Efficient implementation of activities As to be expected at the beginning of Year 3, CIRD is in full implementation mode. They are implementing the technical tasks efficiently and effectively. There are several factors that contribute to CIRD’s successful implementation: · The project enjoys strong technical leadership from the Project Coordinator. The Project Coordinator continuously reinforces the Project’s mission, values and technical approach through a variety of mechanisms (weekly meetings, one-on-one meetings, periodic retreats). · The Project Coordinator encourages an open and “safe” environment where staff feel free to discuss openly and propose new ideas to address technical and operational issues. This kind of environment fosters creativity, innovation and experimentation. 29 · The Project Coordinator communicates his technical expectation and standards to staff. This is done through the weekly meetings and the occasional one-on-one meetings. But most staff expressed a desire to have more regular and formal technical reviews. As one stated, the Project Coordinator “provides guidance if you ask for it”. · The staff uses technology creatively to manage implementation. Managing implementation of activities in remote sites is a challenge and CIRD has created several innovative mechanisms to facilitate implementation. The Mariscal is a great tool to help program and monitor tasks. Mariscal is widely used, including by the driver. Moreover, the Mariscal software is available to external users (ie, USAID and Alianza) as a coordination and monitoring tool. Staff use cell phones and “text messengering” to coordinate location and logistics. Also, CIRD has established an internal network for all staff, including the UOLS using WIFI, thereby enabling everyone to: a) communicate through email, b) access and share documents, reports and forms, and c) monitor implementation status through the Mariscal Software. 3.2.2 Recommendations to improve management 1) Improve coordination of transportation of staff to the field. Staff rely too heavily on their own transportation to travel to the field sites. Transportation may be difficult to address since most of the technical visits occur in the evenings and weekends when the CLS members are available. The Evaluator recommends CIRD explore improving coordination between staff to visit sites or improving scheduling (i.e., organize transport based on one to two month activity projections) of the transport/driver. 2) Allocate more time for staff to reflect and think strategically by formalizing program review processes. Currently CIRD staff meet every Monday to coordinate and plan the week’s activities. In addition, staff present updates on implementation to the Project Coordinator for the Quarterly Report. Staff feel that they focus too much on the day-to-day and do not have sufficient opportunity to reflect and learn from their field experience. The only exception was the Internal Review meeting last month, which was a big success. Building on that meeting, the Evaluator suggests staff meet on a quarterly basis for ½ day to review implementation progress to plan and budget, discuss challenges and lessons learned, and plan for the next quarter. 3) Re-design organizational structure. The current organizational structure can no longer support level of technical activity. During the last year, CIRD experienced dramatic growth in staff. The CIRD Health and Decentralization Project doubled in staff while at the same time, opened and staffed four field offices. Figure 1 demonstrates the organizational structure in place through practice. All major technical and operational decisions, technical review and oversight, and lines of communication are highly centralized in the Project Coordinator. This model was adequate when the Project only had 4 -5 project staff, as was the case during project start-up. As Figure 1 illustrates, the Project Coordinator is overburdened with reporting responsibilities. It is important to note that this Figure does not take into account all the external relations the Project Coordinator maintains with other key organizations, such as USAID, CEPEP, PRIME II, MOH, etc. There are several consequences to the highly centralize model: · The Project Coordinator has become a bottle neck for decisions and review, slowing down implementation momentum. · The Project Director does not have time to maintain and expand his important function as representative of the CIRD Health and Decentralization Project to outside stakeholders. · The Project Coordinator does not have time to focus on critical management and leadership tasks that are being crowded out my day-to-day operational issues. · The Project Coordinator will not be able to sustain this type of organizational structure and if changes are not made soon, he will burn out shortly. 30 Figure 1: Actual Organizational Chart The Evaluator recommends CIRD Executive Director and CIRD Project Coordinator review the technical and management recommendations and decide how best to restructure the staff, supervision relationships and lines of decision-making and communication. First, CIRD management team should design an organizational chart that illustrates roles and functions. The Evaluator recognizes that CIRD’s organizational culture prefers a flat org chart, but the Project has now reached sufficient size and complexity to merit an extra level of management to share in the decision making and supervisory responsibilities. Below in Figures 2 and 3 are two different models for CIRD Management’s consideration. The first one adds a Deputy Director who can resume most of the day-to-day operations, freeing up the Project Coordinator to focus on external relations, project representation and technical leadership. To work smoothly, the Project Coordinator and Deputy need to clarify their roles and responsibilities and who makes what decisions in which areas. The Project Coordinator then needs to delegate tasks and functions to the Deputy Director and reinforce this division of labor with the staff and others. This model also assumes there is sufficient budget to hire another senior staff person. Figure 2: Option #1 Project Coordinator Inst. Team IT support FS UOLs Health Promter R/CR IEC M&E UOLs UOLs UOLs UOLs Driver Deputy Coordinator Project Coordinator CIRD Executive Director Inst. Team IT support FS UOLs Health Promter R/CR IEC M&E UOLs UOLs UOLs UOLs Driver F/A F/A 31 A second option is the management team approach. This structure creates two teams: 1) management team comprised of the Project Coordinator and leaders from each of the technical teams and 2) five technical and F/A teams. The management team would be responsible for: a) establishing the overall project strategic direction, b) providing technical leadership and oversight, and c) monitoring project implementation. In turn, the Team leaders would be responsible for their individual technical component and: a) review technical activities, b) monitor and track implementation against plan and budget, and c) manage day-to-day operations. Figure 3: Option #2 The most likely team structure would follow the current four technical components but with modifications as described in Section 4.2. The technical areas that remain constant with some modifications are Institutional Strengthening and IE&C. The Evaluator recommends that Dissemination be elevated and organized as a technical component on the par of others, requiring a dedicated technical staff person to organize and lead this technical area. Other issues to consider are: a) does the project elevate the management and financial systems activity to a technical area as suggested by some staff? b) if health promoters goes away, where does CIRD put R/CR? And where does CIRD management reassign the current health promoter staff person to capture and better utilize her technical skills? c) who should the UOLs report to? In addition to redesigning the org chart, CIRD management will need to lead a process to transition to the new roles and responsibilities. The process will entail: · Sharing the new org chart with the entire staff, allowing time and space to comment and reflect on the changes. · Rewriting the Project Coordinator’s job description (see Annex 6). The Project Coordinator’s job will change based on the new org chart. Rewriting the job description, sharing it with staff to communicate new roles and functions, as well reinforcing the new job through practice are critical actions to facilitate the transition to the new Org Chart. Annex 6 provides a draft job description for the Project Coordinator. · Rewriting and sharing the new job description for the Team Leaders with staff. 4) Rethink UOLs roles and responsibilities and assign UOL staff to a technical team. Originally the UOL were established to coordinate all of the Alianza members activities in a given community. In practice, some UOLs have become a logistical adjunct to the CLS while others have become a technical support to CLS. What is not happening, however, is the original objective of integrated coordination of the Alianza members. It is important to reinforce the original purpose of the UOLs so as to minimize the potential problem of CLS depending on the UOLs in order to function and carry out it roles and responsibilities. Team #1 Institutional Strengthening Team Leader Team members Team #2 ?? Team Leader Team members Team #3 IE&C Team Leader Team members Team #4 Dissemination Team Leader Team members Team #5 F/A Team Team Leader Team members Project Coordinator 32 The UOLs can play an important role in the recommendation to reorient CIRD’s TA approach from “doer” to “mentor” (See Section 4.1). Some of the UOLs coordinators can be trained to provide TA locally to the CLS in their region, serving as an intermediate step to institutionalize counterpart capacity to form and operationalize CLS. CIRD has already developed this approach with the UOL in Ita. In essence, Magdalena has become a full fledged member of the Institutional Strengthening Team by participating in weekly meetings, offering input on her region workplan and activities, and providing TA to the three CLS in her region with backup from the Headquarter Team. This “local” TA approach has produced results: Peguajho and Curupicayty are medium to medium-well functioning CLS. And moving the locus of implementation from Headquarter to the UOL staff has helped transform HQ staff from implementers to mentors in this region, freeing up HQ staff time to focus on other TA matters. CIRD has several highly qualified staff that are being under-utilized as coordinators and who can provide TA. Staff with potential include: Wilfreda in Misiones, Ramona in Encarnacion and Martin. HQ can incorporate Wilfreda and Ramona into the Institutional Strengthening team, involving them in planning, programming while at the same time, training them in the technical approach and tools. Therefore, UOL staff should report to the Institutional Strengthening Team. Issues to be addressed will be: will Ramona be interested in this expanded role given her sole focus on the Farmacias Sociales? Should CIRD hire a more technical person in Coronel Bogado so all UOLs can assume a TA role? The case of Lic. Martin Gonzalez is different. He provides TA in an unique area - educating the CLS on how to understand and relate to the mayor’s office - that is required by all 12 Alianza communities. CIRD management should officially and programmatically recognize this TA need by integrating his activities in to the Institutional Strengthening TA package and systematizing his approach. Currently, his workplan is reactive based on immediate needs and not proactive. Designing an approach that could be implemented in all communities would increase the impact of his TA. In addition, Martin should also be assigned to the Institutional Strengthening Team, where his TA corresponds best. 3.3 Quality of CIRD technical assistance The CIRD technical team receives high marks from their counterparts in the field. When asked about the quality and relevance of the CIRD technical assistance, the CLS members enthusiastically express their support for CIRD staff, stating CIRD staff are: “lo maximo”; “lo mejor que hay”; and “se nota que son gente apasionada por su trabajo”. The CLS in Arazape specifically asked that the Evaluator include a statement thanking USAID for allowing them to receive CIRD’s technical assistance stating the CLS is “proud and satisfied” of the results they achieved for their community thanks to CIRD’s work with them. The feedback on CIRD can be organized into three categories: First, the shared vision translates into a common approach and TA style responsive to the CLS and communities. Almost all CLS interviewed express similar comments about CIRD’s TA style, reflecting that CIRD staff “never impose but instead guide and orient” CLS members; or that CIRD staff “gently push and cajole us to change our ways”. One CLS member observed: “We have received assistance from other international organizations and their approach is ‘we are here to show you the truth’. CIRD is completely different. They do not come with a set recipe on what is right or wrong.” Another theme is CIRD staffs’ openness and flexibility. Over half the CLS stated that CIRD staff “listen and encourage open dialogue”. A third theme is CIRD’s problem solving orientation in their TA approach. One third CLS stated that CIRD staff helped them solve problems and identify simple solutions: “We learned from CIRD that we can’t wait for someone to help us. They showed us that the majority of the problems can be solved right here in our community”. CIRD staff have earned respect from and trust of their counterparts. From the CLS interviews, several factors emerge contributing to the high level of respect and trust. First, almost all CLS 33 interviewed recognize the high quality and level of technical expertise among CIRD staff. Second, all CLS interviewed indicate they are impressed on how dedicated CIRD staff are and acknowledge that CIRD staff spend evenings and weekends away from their families to work with the CLS. Third, a couple of CLS stated CIRD staff are responsive, always available and come when needed. Arazape CLS provided the clearest example. The CLS learned there would be a meeting in two days at the mayor’s office to review proposals for next year’s budget. The Arazpe CLS called and CIRD staff mobilized in less than one day to help them develop their proposals. Based on CIRD’s TA, the CLS presented their projects at the budget meeting and the Mayor approved all of Arazpe’s projects. Fourth, almost all CLS said that CIRD staff are true to their word. The most common example of how CIRD staff follow through on their word is: CIRD staff “always arrive in the community when they say they will come, even though it is raining or road conditions are bad”. Finally, all CLS mentioned in some form or another that CIRD staff are “gente de calidad y calidez” The CIRD staff are a high performing, creative technical team. In reviewing the technical portfolio of activities, CIRD staff apply the latest technologies and methodologies. For example, the Institutional Strengthening team apply best practices found in other Latin American countries to create local capacity to identify, plan and monitor health priorities. The Farmacias Sociales program is similar to a nation￾wide program implemented in Peru. CIRD’s advocacy approach builds on international experience, as reflected in their dissemination activities and Advocacy Manual. Moreover, CIRD staff apply relevant tools and methodologies that the counterparts can understand and use. All CLS stated the TA is relevant to their needs. As one CLS member said during the interview, “the assistance is 100% what we needed”. The CLS members also recognize that CIRD modify the tools, making them more accessible and user friendly, increasing the likelihood that the counterparts will understand and use them in their CLS work. A good example is the tools CIRD adapted to help the CLS design and implement their Local Health Plans. Finally, CIRD produces high quality technical products. Of note are: a) the Informativo de salud newspaper series, b) the CIRD Website, c) the CIRD Manual Series, Manual 1#: La Definicion de un Conjunto Basico de Salud Esenciales, and Document #2: El Cabildeo: una estrategica para incidir las politicas publicas, d) workshop summary reports, such as Memoria y Conclusion del Seminario – Taller sobre experiencias Comunitarias en Salud, d) the Monitoring and Evaluation Plan and e) the Alianza logo and branding. 3.4 CIRD relations with Alianza partners In discussing how the partnership is working between the Alianza members, the Evaluator received the following comments: · Alianza partners recognize and value the high quality and technical expertise of CIRD staff. They also indicated that CIRD, as well as other partners, are collaborating better. Indeed, almost all technical level staff interviewed at CIRD and PRIME indicated a desire to meet more often and work together more. Also, they expressed frustration, stating that the current mechanism for technical collaboration among the partners – collaboration meetings – is not working well. Instead, they suggested forming technical working groups in specific activity areas that involve the different partners, such as Referral/Counter-referral, IEC to name a few. Interestingly enough, none of the Alianza members perceived the Mariscal or the UOLs as additional mechanisms for collaboration and coordination. · Technical competition and territoriality still exist among the Alianza members but at the Senior Management level. CIRD and PRIME technical staff are open and receptive to technical collaboration 34 and sharing. Moreover, they are clear on what are their technical areas of expertise, their strengths and weaknesses and their respective scopes. All the Alianza members, including CIRD, stated the last senior management meeting greatly helped the different institutions carve out their respective roles and responsibilities but follow up actions and decisions are required to concretize the agreements reached at the meting. Although Alianza members state that institutional rivalry has improved, they also indicated that senior-level managers at CIRD still remain competitive and possessive of certain technical domains. · The Alianza is working in the field. For example, discussions in Missiones and Ita revealed that the CLS understand who the different Alianza members are, what their respective roles are, and are beginning (and only beginning) to coordinate locally. It was encouraging to see the CLS meet at the health facilities in Peguajho and Curupicayty and have the CLS members discuss how CIRD is helping strengthen their roles as CLS members while at the same time, discuss how PRIME helped strengthen the skills of the health facility staff and improve the quality of care. Now the Alianza members need to nurture this synergy between projects, using the technical working groups, UOLs and CLS-oriented programming as coordination mechanisms. 35 4.0 Recommendations to strengthen CIRD program 4.1 Recommendations to strengthen overall technical approach As the CIRD project begins its third year, it is an opportune moment to address directly and plan for sustainability of their technical approach. To address these sustainability issues, CIRD will need to identify now strategies and implement activities that will consolidate, replicate and sustain their technical model. First, CIRD should consolidate their work into a technical model so that others can assume and replicate it elsewhere. Second, CIRD should seek out opportunities to experiment in replicating and/or scaling up their technical models, studying how and under what conditions. Third, CIRD should create institutional capacity among others to sustain the technical model. Below is a presentation of the different recommendations to strengthen CIRD’s technical approach. The recommendations are organized by the three actions required to facilitate sustainability. 4.1.1 Consolidate CIRD’s technical model Simplify technical model to reduce counterpart’s dependency on CIRD staff. In reviewing the CIRD technical model, the Evaluator noticed that many of the technical activities are dependent on staff and technical skills external to the CLS to complete the activities. For example, to complete all three phases of the basic health package design, the CLS sends the draft package description to CIRD’s health economist to conduct the financial analysis, determine the cost of the package and estimate the budget required to finance the package. To complete the “senso sanitarios” requires sending the data to CEPEP who compile, analyze and present the information. To develop the GIS maps requires computer equipment and special software as well as CIRD staff time. And the list continues. Moreover, the technical model continues to grow in different technical directions. It is only natural that a creative group of technical professionals would be excited about applying new and promising methodologies and/or technologies to their activities. What has happened, however, is the technical approach has expanded to add complicated tools and methodologies that will potentially yield limited results. A clear example is the SIG. One of the CLS’ primary responsibilities is to oversee and monitor health services to ensure the services respond to community’s needs. Given this task, CLS members require same basic information to monitor services. CIRD’s response to provide the CLS with this data has been to implement the SIG software, a complicated tool that is only appropriate for health clinics when the data could be found in service delivery statistics and other data sources. Other examples of technically exciting but questionable impact are GIS maps and its adaptation to vaccination campaigns. A technical model that depends on outside expertise can not be sustained by the counterparts. Moreover, if CIRD is going to replicate its approach to several other municipalities/communities, then CIRD needs to reduce the technical approach to its bare essence so that others can eventually assume the responsibility of forming and training new CLS beyond the 12 Alianza communities. In simplifying the technical model, the CIRD team should remember that FRAM is the exception and that Peguajho and Curupicayty are the norm. Create mechanisms to better integrate different components of model. The four technical components are implemented as vertical programs, sometimes creating confusion but more often, straining the counterpart’s ability to receive and absorb technical assistance in some many diverse technical areas. There are some simple mechanisms that can facilitate better technical integration, including: · using the team management approach to plan, program and monitor activities, · conducting monthly meetings with team leaders to discuss implementation issues across technical areas, and 36 · promoting quarterly internal coordination meetings of all staff to improve programming across all technical components. In addition, CIRD could “flip” the annual workplan process upside down. Currently, CIRD programs, monitors and reports on activities by the four technical areas. Instead, CIRD could program activities by CLS/communities, creating a workplan for each CLS/community. This bottom-up approach to planning requires representatives from the different technical areas sit together to design an annual workplan for each CLS/community and to track implementation as a team. This planning approach will CIRD: a) ensure all components, if appropriate, are sufficiently represented, b) better plan events and TA so as to not “bunch” up and overload counterparts, and c) improve coordination, logistics and application of staff. Once CIRD staff complete this bottom up approach, they can sum up the activities by technical areas to report to USAID for their quarterly report and to track their indicators (also organized by technical areas). Remove policy barriers to technical model. Many local, state and national level government officials are taking note of CIRD’s success with the CLS and are requesting CIRD replicate their activities. Even though there exists strong political will, there are some policy barriers preventing expansion of a comprehensive model that includes financing. During the last two years, CIRD has increasingly become closer to the MOH through its work with the local and regional MOH officials. Coupled with recent changes in political environment, including a new Ministry of Health expressing interesting in decentralizing, the moment is right to influence national policy within the MOH. Consequently, the Evaluator recommends CIRD redirect technical activities under dissemination and program sufficient staff LOE to advocate for policy reforms that will facilitate expansion of locally led health management. Technical suggestions are listed in the section 4.2. 4.1.2 Replicate and/or scale -up the simplified model Focus on task at hand and resist temptation to respond to requests. A complementary suggestion to simplify is to focus. To achieve focus, the Evaluator recommends CIRD limit the technical approach and stop adding new technical “bells and whistles”. Diagram 4 provides selection criteria for CIRD staff to follow in reviewing the current and new technical activities. Also, CIRD staff need to concentrate the geographic areas and resist geographic “crawl”. CIRD is under extreme pressure from government officials to work with their communities. In response to the constant demand, CIRD tends to say “yes” to communities located close to the 12 Alianza communities or in a community who has agreed to pilot one of CIRD’s new and interesting tools and methodologies. As a result, CIRD is overextended and located in too many geographic areas outside its agreement. The Evaluator suggests CIRD works with USAID to help them manage counterparts’ expectations and disappointment when CIRD has to say “no”. Re-orient TA approach from “implementer” to “mentor”. CIRD staff are working at maximum capacity to meet the needs of the 12 Alianza communities and the additional communities that have been added along the way. Clearly, the staff can not assume more CLS/communities or technical activities. So if CIRD is going to replicate its model and can not add more staff due to budget constraints, then CIRD will have to change the way they provide technical assistance. One suggestion is to change CIRD TA approach from “doing” all the technical interventions to “mentoring” a group who will carry out the tasks. Figure 4 illustrates the current TA model where CIRD staff directly implements all technical activities with the 12 + CLS/Communities. 37 Figure #4: Illustration of how CIRD provides TA to CLS/Communities Now is an opportune time to put into place strategies that will transfer CIRD’s technical approach and skills to an intermediary group. Figure #5 demonstrates how CIRD can achieve greater geographic coverage by transferring skills to an intermediary group. To prepare for this change in role, CIRD needs to take several steps. First, communicate to the counterparts CIRD’s expectation that some day the CLS/community will become more independent. Indeed, many of the CLS are aware of this and have Figure #5: Illustration of alternative TA approach CIRD = CLS/Community 38 CIRD = CLS/Community = intermediary group 39 expressed their intent to start other CLS (Ita, Arazape). Second, develop a plan to “graduate” CLS and help them become more independent. This entails not only simplifying the technical model, but also providing sufficient TA and training with graduation in mind. This underscores the need to simplify the technical model as well as the tools and methodologies so a diverse group of people with varying skills can implement them. Also, CIRD needs to continue monitoring CLS after graduation to ensure continued operations and results. Third, develop a new TA model to build counterpart capacity to train and provide assistance to the CLS/communities. A critical element to the new TA model is the process to identify the intermediary group and locate an institutional “home” among the counterpart for this TA activity. Document the model and standardize tools and methodologies. As part of the replication and scale up strategy, CIRD staff need to take the time to document the simplified, integrated technical model. In addition, CIRD staff should standardize their approaches in the different technical areas by developing “how to” manuals with companion training guides. They can also standardize many of the Powerpoint presentations on different topic areas. 4.1.3 Sustain CIRD’s technical model Experiment with replication and scale up to determine what is required to sustain the technical model. One of the ideas discussed with CIRD staff is to take the model to scale in one department. To date, CIRD has had great success in taking Farmacias Sociales to scale in varying degrees in both Itapua and Misiones. Misiones is a strong candidate for scale -up. Misiones has demonstrated political will at the governors and municipal levels and strong support from the Regional Director of Health and health facility managers. Also, two of the three CLS are ready to graduate and are taking initiative to replicate their experience by forming new SCLS. There is a highly skilled local CIRD coordinator who, with training and mentoring, can take on more technical responsibilities. Finally, Misiones is only two hours from Asuncion. Once CIRD decides to scale up in one department, then they have to: a) implement their plans to begin graduating some of the 12 Alianza CLS/communities, b) identify local counterparts within “gobernacion” to become the intermediary group, and c) strengthen the local UOL coordinator to assume more a technical role (see Section 3.2). In addition, CIRD should document the scale-up process and plan to evaluate it toward the end of the project to identify what worked and didn’t work, what conditions favor or impede scale-up, etc. Strengthen CIRD/USAID relationship to create a partnership that directs and monitors CIRD’s path toward sustainability. Currently, CIRD project management and USAID enjoy an open and positive relationship. They are in constant communication and discuss a wide variety of topics. The discussions, however, tend to focus the day-to-day tasks and details with limited opportunity to reflect or discuss new technical directions, challenges confronted during implementation, and changes in the political environment that influence CIRD’s work. Moreover, the communication is informal and frequent, but not regular and consistent. The Evaluator therefore recommends that USAID and CIRD to formalize their channel of communication by establishing a meeting schedule and implementing a uniform agenda that covers consistent information (See Annex 7 for illustrative agenda.) In addition, CIRD can invite USAID staff to occasionally attend key technical meetings, such quarterly and/or annual workplan discussion, review of new technical product, and provide feedback on new technical design 40 4.2 Recommendations to strengthen specific technical areas in CIRD’s model The prior section offered suggestions on how to strengthen CIRD’s overall technical approach. This section focuses on recommendations to strengthen specific technical areas. 4.2.1 Health promoters Section 3.1.4 presented many of the technical and organization problems challenging the Health Promoter activity. Interviews and documents revealed fundamental problems with the shared responsibility for implementing the community Health Promoter strategy. Given the impasse, the Evaluator recommends that USAID intervenes now to decide, in collaboration with the two Alianza members, which organization should assume sole responsibility for this activity. Based on the decision, then USAID will need to modify the organization’s indicators and possibly, cooperative agreement. 4.2.2 Referral and Counter-Referral (R/CR) System CIRD and PRIME II need to work together to standardize the activities in R/CR. Given the fact that both organizations will implement a department-level R/CR system, they need to present a uniform approach to the MOH or it will only create confusion and problems during the implementation phase. Therefore, the Evaluator recommends the following actions to help synchronize efforts as well as strengthen CIRD’s institutional capacity to design and implement a R/CR model. 1. Form an Alianza working group on R/CR comprised of CIRD and PRIME II staff. 2. CIRD and PRIME II projects work together to design the R/CR activity, including a shared vision, identical design and a common approach. Moreover, both projects should use the same tools and methodologies to reinforce and present a consistent approach on R/CR. This will require that CIRD delay implementation of their activity to coordinate with the PRIME II activity. 3. CIRD hire the same international consultant to finalize the CIRD R/CR model and develop a workplan to operationalize the model. The international consultant has applied experience in rolling out a R/CR model nationwide and can help CIRD develop a realist workplan for the two years remaining in the project. 4. Leverage the PRIME II technical resources to help implement CIRD R/CR activity. For example, the PRIME II international consultant will travel to Paraguay in November to help PRIME II diagnose the current R/CR infrastructure in Cacauzu. The CIRD staff member can join the PRIME II team to learn how to conduct the diagnosis as he prepares to launch his activity in Itapua. Also, both projects will be conducting similar training for their respective counterparts in Cacuazu and Itapua. The projects can, together, develop training materials and conduct jointly training for both counterparts together to save resources. 4.2.3 IEC CIRD is in the process of re-designing it technical strategy for IEC activities and has set into motion several activities to focus CIRD’s activities so they yield greater impact. Some of steps include: change in leadership, new procurement process and draft global communication strategy. The Evaluator spent considerable time with the IE&C team to develop this list of recommendations: 1. Develop immediately a joint communication strategy for all Alianza members. The three Alianza members acknowledge the needs to better coordinate IEC activities. CIRD took an initial step by proposing some preliminary ideas for a joint IEC strategy, but it is not sufficient for a global IEC strategy. To accelerate the process by which to design a global IEC strategy, the Evaluator suggests: 41 Diagram 5: Illustrative Communication Strategy for Alianza Members Increased demand for and use of decentralized community based health services through IEC activities Increase community knowledge in select # of preventive health behaviors and practices, with emphasis on FP/RH Increase community/population’s awareness of rights and responsibilities in decentralized community based health services Increase community/population’s awareness of rights and responsibilities in auto-cuidado Increase community/population’s knowledge in priority health prevention areas Encourage client behavior change through improved counseling & quality services Increase public’s knowledge of rights and responsibilities in health Increase client’s knowledge in priority health prevention areas through improved counseling Increase community acceptance and practice in select # of health prevention areas, with emphasis on FP/RH Encourage client behavior change through promoters detecting and referring clients to appropriate services Encourage community behavior change through CLS promotion activities 42 Diagram 6: Roles and Responsibilities of Alianza Members in Joint Communication Strategy Increased demand for and use of decentralized community based health services through IEC activities 1.0 Increase public’s knowledge of rights and responsibilities in health (Joint responsibility between CIRD & PRIME; CIRD lead) 2.0 Increase community knowledge in select # of preventive health behaviors and practices, with emphasis on FP/RH (Prime lead) 3.0 Increase community acceptance and practice in select # of health prevention areas, with emphasis on FP/RH (Joint responsibility between CIRD, CEPEP & PRIME; Lead to be determined)) Increase community/population’s awareness of rights and responsibilities in decentralized community based health services (CIRD lead) Increase community/population’s awareness of rights and responsibilities in auto-cuidado (PRIME lead & CEPEP Promoters reinforce messages) Increase community/population’s knowledge in priority health prevention areas (PRIME lead, CIRD promotional activities reinforce messages) Increase client’s knowledge in priority health prevention areas through improved counseling (PRIME lead ) Encourage client behavior change through improved counseling & quality services (PRIME lead) Encourage community behavior change through CLS promotion activities (CIRD lead ) Encourage client behavior change through promoters detecting and referring clients to appropriate services (CEPEP lead ) Increase client’s knowledge of quality health services in community (CIRD lead ) CIRD lead CIRD support role 45 i) Develop joint technical working group with Alianza staff responsible for implementing IEC activities. ii) Leverage technical assistance currently provided by the PRIME II international expert in IEC to assist Alianza members to design a global IEC strategy. Modify the international consultant’s current terms of reference to include design activity in two upcoming November trips. iii) Design a global IEC strategy for Alianza members before developing new materials and/or contracting groups to implement IEC activities. Diagrams 5 and 6 present some initial ideas the Evaluator and CIRD team drafted for the IEC global strategy and roles and responsibilities, respectively. They are proposals only and need to be vetted with USAID and Alianza members. Any changes may result in modification to the cooperative agreements. 2. Delay CIRD Procurement until the global IEC strategy is finalized. CIRD is currently in the first stage of a competitive process. CIRD can continue with this step to qualify the first round of eligible companies to compete. However, the CIRD IEC staff recognize they can not develop the terms of reference for the technical scope without the global IEC strategy in place. This presents some timing and logistical issues for CIRD since the current IEC activities are scheduled to end in November, requiring the following actions. i) Negotiate with USAID a delay in schedule for IEC activities to better synchronize with Alianza Joint Communication Strategy design. ii) Provide a cost-extension to current (some? all?) CIRD subcontractors to enable the IE&C activities to continue at a reduced level during the procurement process. iii) Make changes in CIRD workplan to reflect delay in procurement and therefore, postponement in IE&C activities so CIRD will not be penalized. iv) Request technical assistance from international IEC consultant to translate CIRD roles and responsibilities in the global IEC strategy into technical terms of reference for the procurements. During the international consultant’s November trips, CIRD IEC staff can set a date with the international consultant, develop her scope of work, and execute a consulting agreement. 3. Review IEC staffing. If CIRD is going to lead this important activity, they need to carefully review their staffing in this component to ensure they have the right technical staff to lead and implement it. As mentioned before, CIRD has taken some actions to strengthen their institutional capabilities in IEC through staffing changes. The two existing staff acknowledge they do not have the skills to design the Alianza global IEC strategy and develop the CIRD IEC strategy. Moreover, given the breadth and scope of the proposed IEC activities for both Alianza and CIRD, it is not realistic to assume that they should have all the skills required given the complexity and breadth of IEC activities anticipated in the global IEC strategy. Hiring the international IEC expert to design the global IEC strategy and CIRD IEC strategy and workplan, will help address this shortcoming. In addition, CIRD should be able to access international experts in specific IEC areas if the technical skill does not exist locally. The question remains if the CIRD IEC staff have the skills to lead and oversee the Alianza global IEC strategy and implementation of CIRD’s IEC activities. In the first area, it is too soon to tell if the new CIRD IEC director has the skills to lead the Alianza global IEC strategy given the fact that she has only been with the project for two months and the strategy has yet to be designed. With respect to the second area, the long￾term CIRD staff person has excellent field experience implementing the kind of IEC activities CIRD will carry out during the remaining life of the project. She will need, however, a clear road map with benchmarks to help her oversee implementation. To ensure CIRD has the right staff, the CIRD Coordinator will need to monitor frequently the IEC team’s performance, and support them through professional training, international consultants, etc. 4. Modify the procurement approach. To date, CIRD has visualized multiple subcontracts assigned by geographic areas. This approach, however, does not address some of the problems identified by the IE&C staff and Evaluator: a) lack of consistent messages for core health themes; b) lack of economy of scales in 46 IE&C material development and production, c) lack of economy of scale and therefore savings in media use. The Evaluator has proposed that CIRD consider the following approach. Draft two technical scopes of work. The first scope focuses on developing generic health messages, producing generic IEC materials to be used in all regions, and using mass media and other channels to reach a larger audience beyond the 12 Alianza communities. This scope of work should coordinate with the other CIRD recommendations to scale-up in one state. Moreover, the design and development of these materials should be done in close coordination with the other Alianza members. The second scope of work focuses on implementing different promotional activities in the 12 Alianza communities. The second competition would not preclude a single contractor from either: a) competing for the 1st and 2nd procurement or b) competing for all the department level procurements. To modify the procurement process, CIRD should take the following steps: 1) discuss new approach with Alianza members during the global IEC strategy design, 2) discuss new approach with USAID, and 3) coordinate international consultant SOW with new procurement approach so she can assist CIRD to draft the technical scope for the multiple procurements. 4.2.4 Dissemination There are two recommendations to expand the scope of and strengthen the dissemination activities. 1. Assign a staff person to coordinate and lead the dissemination activities. Due to the Project Coordinator’s heavy workload, the dissemination activities, although successful, have been opportunistic and do not add up to a measureable impact. CIRD management need to assign a staff person to be in charge of this technical component who will: i) Develop a strategy with clear goals and objectives linked to CIRD’s indicators. Possible goals include: Goal: Create favorable environment for locally led management of health services Objectives: Inform key stakeholders on successful examples of decentralized basic health services; Build support for decentralized basic health services; and Demonstrate examples of replication and/or scale of successful model(s) of decentralized basic health care. ii) Systematize many of the routine activities required to fulfill CIRD indicators (ie, # of publications, # of innovative initiatives, # experiences available on the webpage). iii) Identify new activities that will support CIRD’s expanded advocacy role (see item #2 below). 2. Expand disseminations technical focus to address the policy constraints. To seize the opportunity presented by the change in political leadership and growing support by regional government officials for CIRD’s technical approach, the Evaluator suggests that CIRD develop an advocacy strategy targeted to national and key regional level policymakers within the MOH to remove some of the policy barriers and motivate the MOH to pilot some of success stories, such as functioning CLS, FRAM integrated model, Farmacias Sociales in other regions. 47 5.0 Next steps Although this report is the final step to the mid-term evaluation, this report should also initiate a new process between CIRD and USAID to reach agreement on future directions for the years remaining on this contract. This report provides information on a wide range of areas as well as concrete recommendation on how to modify and strengthen the Project. This report provides sufficient information for CIRD staff discuss and decide: a. how to modify their current technical strategies, b. how to organize staff to manage project implementation, and c. how to redirect staff, level of effort and activities to strengthen an already strong technical program with the view toward phasing out over project’s last year. In addition, the report offers USAID and CIRD a framework to discuss and agree upon what changes maybe required in the technical scope and cooperative agreement in order to leave a technical program that is sustained by counterparts after the project ends. The CIRD Health and Decentralization Project is off to a strong start, and with some adjustments outlined in this report, is well positioned to end on an even stronger note that will yield impact beyond the original project design, leave a technical legacy of successful examples of locally led health management, and provide best practices for other countries confronted with decentralizing health services. 48 Annex 1: Mid-term evaluation plan for Health Decentralization & Community Participation with Centro de Informacion y Recursos para el Desarrollo (CIRD) Introduction The purpose of this paper is to propose a methodology and process by which to evaluate the USAID’s Health Decentralization and Community Participation contract with CIRD (Herein referred to as CIRD Mid-term Evaluation). The paper is divided into three components: 1) Methodology and Approach, 2) Logistics and Organization of Evaluation, and 3) Sample of Instruments. The methodology and approached outlined below are in draft form and will be modified to reflect the comments of both CIRD Project and USAID/Asuncion to ensure the evaluation will meet both group’s needs. 1.0 Evaluation Methodology and Approach 1.1. Evaluation Objectives The objectives of the mid-term technical evaluation of CIRD are multiple: 1. Review the overall project design, the workplans and progress of technical activities from October 2001 to September 2003 to verify if the CIRD program of activities are on track to produce the results stated in USAID strategy 2. Review CIRD’s the progress-to-date in achieving the stated results. 3. Review the Project’s technical management to determine if the technical staff, budget and operations support and are on-track to achieve desired technical results 4. Interview CIRD “clients” to ensure CIRD is providing effective, timely and quality technical assistance responsive to USAID’s and counterpart’s needs 5. Provide recommendations to USAID/Asuncion and CIRD management on how to modify and/or focus technical strategies to ensure CIRD activities will achieve results 1.2. Evaluation Questions Below are a set of general questions organized by categories that support the evaluation goals and objectives. The purpose of listing these general questions is to facilitate dialogue between the evaluation team, CIRD management and USAID to determine if the evaluation is meeting their needs and expectations. Technical progress toward achieving results · Is the CIRD technical strategy clearly linked to the USAID results? · Is the workplan and activities clearly linked to the USAID results? · What measurable progress has been made toward each of the objectives? · Does the monitoring and evaluation plan, process and tools adequately measure the desired results? · If there have been problems achieving the stated results, what have they been and why? · How can the strategy and/or workplan be modified to address these challenges? Management support toward achieving results · Has CIRD hired the staff and consultants with the appropriate skills and experience? Is there enough staff to carry out the workplan within budget? · Do CIRD staff and consultants’ scopes of work job match the technical program? Is the staff fully utilized to their potential? Is the staff producing the desired technical results? · Does the workplan program the right type and mix of technical activities that will lead to the desired technical results? · Does the budget support these activities? 49 · Does CIRD management review workplan activities and budgets on a regular basis to ensure that activities are on-time and on-track to deliver results? Do they review it with the staff? · Do CIRD staff and consultants receive adequate and timely guidance and feedback on their work from Project Director? From USAID? Quality of CIRD technical assistance · Does CIRD provide timely and quality technical assistance that responds to the counterparts needs? · How well do CIRD staff work with the counterparts? With USAID? With other partner organizations? · Do CIRD staff use the latest methodologies and technical approaches in their work? · What is positive and you want CIRD to continue and/or expand? · How can CIRD improve the quality of their technical assistance? 1.3. Evaluation Methods The Team will apply a variety of methods for the mid-term evaluation. They include: · Interviews: The Evaluator will conduct one-on-one semi-structured interviews that will last from 60 to 90 minutes. Within CIRD, the Evaluator will conduct multiple interviews of CIRD President and Health Project Director. In addition, the Evaluator will meet with CIRD staff on two different occasions to first, review in depth their technical strategies, tools and methodologies and activities realized in the field. The second interview will focus on technical management of CIRD activities. Using the same methodology, the Evaluator will also interview many stakeholders of the CIRD Health Project. In Asuncion, the Evaluator will interview representatives from USAID, Alianza members such ad CEPEP and CIRD, and a select number of CIRD Board members. In the field, the Evaluator will interview CIRD Health project counterparts including government authorities at the Regional and municipal levels, health workers and members of the local health councils. · Site Visits: The Evaluator will visit approximately 20 interviews in the field at 12 sites in 4 out of 4 regions where CIRD is active – Itapua, Misiones and Centrales (Refer to Section 2.2 for a list of the sites). In the regions with CIRD activities centered on a health clinic, the Evaluator will conduct individual, semi￾structured interviews with representatives from the municipalities, health establishments and local health councils. In the regions with health posts, the Evaluator will group together at least one representative from each of the main stakeholder groups and conduct the interviews jointly. · Observations: In addition to the interviews, the Evaluator will have a chance to visit and informally observe CIRD staff in the field as well as the counterparts who have received technical assistance and/or training from CIRD to see if they are applying the new skills and knowledge received. · Document Review: Finally, the Evaluator will review all relevant documents. A preliminary list of the documents is provided in Section 2.1. 1.4. Evaluation Approach The mid-term evaluation will be a three-step process. The first, the Evaluator will prepare for her trip to Paraguay by finalizing the evaluation methodology, reviewing documents and organizing the trip’s logistics. During the second step, the Evaluator will travel to Paraguay to conduct a series of interviews, meetings and site visits as well as analyze data, statistics, reports and documents. During this phase, the Evaluator will be joined by other team members and will together, finalize the evaluation methodology and organization of site visits, design and elaboration of interview instruments, and collection of necessary data. In the third step, the Evaluation Team will debrief key stakeholders, solicit their input on and document the findings and conclusions. Below is a more in-depth description of the evaluation process. 50 Step One: Preparation for the Paraguay Mid-term Evaluation · Conduct telephone interviews with CIRD staff to formulate the scope of the mid-term evaluation, including confirmation of evaluation’s objectives, list of questions to guide interview, schedule of interviews, meetings and site visits. · Review relevant Paraguay program documents such as work plans, trip reports, presentations, evaluations, etc. · Coordinate with the CIRD Project Director to organize logistics of Paraguay visit. Step Two: Field Trip to Paraguay Evaluator will travel to Asuncion and · Finalize evaluation methodology with CIRD management and USAID. · Debrief CIRD staff of goals and objectives of mid-term evaluation to provide information and solicit their input on evaluation methodology. · Conduct interviews of CIRD staff located in Asuncion. · Collect and analyze additional local technical reports, documents, statistics, data, etc. · Assess overall M&E plan, process and tools for CIRD activities · Meet with major stakeholders of CIRD program, including USAID, Alianza members, members of CIRD Board of Director. · Visit site(s) of CIRD activities in the field. Also interview CIRD field staff in the regions. · Document findings and preliminary conclusions to be included in mid-term evaluation report. · Conduct intermittent meetings with CIRD President and Project Director to review and solicit feedback on initial findings. · Conduct exit meeting with CIRD staff to present preliminary findings and to solicit their feedback on findings. · Conduct debriefing with USAID to share preliminary findings of mid-term evaluation. Step Three: Document findings in a mid-term evaluation report To finalize the report, the Evaluation Team: · Write up and circulate the mid-term evaluation report to CIRD and USAID for comments. · Finalize the mid-term evaluation report based on input received. · Below is a table with the updated timeline for the evaluation. Table 1: Updated Timeline Activities Step 1 Step 2 Step 3 Step 1: Confirm team members · Confirm dates, trip logistics, etc. · Draft evaluation plan including methodology and timeline · Design and develop interview guides, Prepare for trip 10/10 10/10 10/18 10/18 Step 2: Evaluator travel to Paraguay · Introductory meeting with CIRD management · Preliminary meeting with USAID · Introductory meeting with CIRD staff · Conduct site visits and Asuncion interviews · Debrief CIRD staff, Debrief USAID · Depart Paraguay 10/19 10/20 10/21 10/21 10/22-10/28 10/29 10/29 10/30 Step 3: Develop first draft of mid-term evaluation report · Solicit comments from CIRD and USAID · Finalize document, send to CIRD and USAID via email 11/7 11/7-11/14 11/14 – 11/24; 11/30 51 2.0 Organization and Logistics of Evaluation 2.1. Evaluation Team and Roles and Responsibilities Barbara O’Hanlon (CIRD consultant) – Management Expert, FP/RH Policy Specialist – will serve as the main Evaluator. The Evaluator will rely on Agustin Carrizosa (CIRD President), Rueben Gaeta (CIRD Project Director) and Graciela Avila (USAID/HPN Officer) as local experts and resources for the evlaution. Below is the description of roles and responsibilities: Table 2: Roles and Responsibilities Barbara O’Hanlon · Design approach and methodology for mid-term evaluation · Design and set-up agenda of activities for Paraguay trip · Review documents, conduct interviews by phone in preparation for field visit · Conduct evaluation in Paraguay, including conducting interviews, visiting site visits, analyzing findings, leading debriefings · Conduct briefings and debriefings with CIRD staff and USAID · Draft and finalize evaluation report Rueben Gaeta Agustin Carrizosa CIRD · Review and provide comments on evaluation methodology · Assist with logistics and agenda · Verify Evaluator’s information and observations when asked · Provide information and feedback on preliminary findings · Review and comment on draft report Graciela Avila USAID · Review and provide comments on evaluation methodology · Verify Evaluator’s information and observations when asked · Provide information and feedback on preliminary findings · Review and comment on draft report 2.2 List of Documents to be Reviewed Prior (in alpha order) The Evaluator has downloaded the following documents from the CIRD website for review. · Attachment I: Program Description for CIRD. · CIRD Workplan for 2002 – 2003. · Fondos rotatorios. MSPyBS. · Fondos Rotatorios de Medicamentos Básicos para Farmacias Sociales. La experiencia en el departamento de Itapúa. · Manejo de reuniones como herramienta para mejorar la efectividad de la organización. · Memoria y conclusión del seminario - taller sobre experiencias Comunitarias en Salud · Organización y Funcionamiento de los Consejos Locales de Salud y Subconsejos Comunitarios. · Plan de monitoreo y evaluación. · Producción, Eficiencia y Costos de Servicios de Salud en el Paraguay · Principios y Fundamentos de los Fondos Rotatorios de Medicamentos Básicos para Dispensarios y Farmacias. · Reglamento de Funcionamiento de los Subconsejos de Salud de Itaguá. · Reglamentos del fondo rotatorio del Parque Sanitario Distrital de Itá. · USAID/Asuncion Strategic Plan and Results Framework. Once in Asuncion, the Evaluator will ask to see workplans and budgets, quarterly reports, monitoring and evaluation progress reports, other technical documents. Also, she will review staff scopes of work and or job descriptions and CVs. 52 2.3 Draft List Individuals and Organizations to be Interviewed In collaboration with the CIRD Project Director, the Evaluator has identified the individuals listed below to represent their organizations and/or facilities in the mid-term evaluation. The Evaluator selected this group based on the subsequent objectives: · sufficient number of “clients” (ie, USAID, MOH, municipalities) who can comment on the quality and effectiveness of CIRD’s technical assistance · sufficient number of local representing the CIRD’s different technical areas to determine if activities are on track to achieve desired results · sufficient number of staff to determine if determine if the technical staff, budget and operations support and are on-track to achieve desired technical results · all partners to determine how well CIRD is collaborating, coordinating and participating in Alianza partnership Table 3: Draft list of scheduled interviews Organization Individual Title USAID Josceline Betancourt, Graciela Avila · HPN Officer · HPN Officer CIRD Staff Dr. Agustin Carrizosa Lic. Ruben Gaete Dr. Carols Guato Lic. Diana Rodriguez Lic Sandra Villalba Lic. Julio Fernandez Lic. Ramona Barboza Lic. Livia Sanchez Dra. Mirtha Mongelos Dra. Esperanza Martinez Dr. Carlos Rodriguez Lic. Horacio Spinzi Lic. Martin Gonzalez Wilfreta Fleitas Magdalena Villagra Dahina Diaz Noelia Torales · President · Coordinator of Health Project · Director, IEC · IEC · Director, SIG & Alternative Financing · Rotating Funds · Social Pharmacies · Local Health Plans · Local Health Plans · Health Promoters · Health Systems · Information Systems · Coordinator of Department · Coordinator of Department · Coordinator of Department · Coordinator of Department CEPEP Lic. Cynthia Prieto · Executive Director, IPPF Affiliate PRIME Dr. Gregorio Soriano · Executive Director Comunidad de Peguajhó (Itá) Itá-urbano: Curupicayty: · Sub-Health Committee members · Director of Health Post · Intendente Municipal · Member of Local Health Committee · Director of Health Clinic · Sub-Health Committee members · Director of Health Post. Atyrá Comunidad de Bernardino Caballero Comunidad de Candia · Intendente Municipal · Members of Health Committee · Director of Health Post. · Members of Sub-Health Committe · Director of Health Post. · Members of Sub-Health Committee · Director of Health Post Encarnación: · Social Pharmacy Network 53 Fram: · Local Health Committee members. Coronel Bogado: · Intendente Municipal · Member of Health Committee · Director de Sub Regional Hospital. Arazapé (San Miguel, Misiones) Itá Yurú (San Miguel, Misiones) San Miguel (Misiones): · Mmeber of Local Health Committee · Director of Health Post. · Mmbers of Local Health Committee · Director of Health Post. · Intendente Municipal · Membersof Local Health Committee · Director of Health Post · Director of Health Clinic 2.4. Draft Interview Schedule and Interview Objectives (Please refer to Annex 2 for a finalized version of the agenda and interview schedule) Below is an overview of the interview and activity schedule for the Evaluator. The Evaluation Team is still discussing the number of site visits needed to achieve the evaluation objectives with finalize this with USAID and CIRD Management in Paraguay. 2.5. Products The Evaluator team will provide the PRIME II project with the following products in electronic form: · Interview questions · Interview schedule · Bibliography of documents reviewed · Final report with an Executive Summary, Table of Content, Analysis, Conclusion, and Annexes 54 Annex 2: Final agenda and interview schedule Time/ Location Meeting Schedule/Organization Individuals Monday 10/20 Asuncion Afternoon · Mtg w/CIRD management · Rueben Gaeta, · Sandra Villalba Tuesday 10/21 Asuncion Morning · Mtg w/ USAID Team · Mtg w/IEC Team · Mtg w/M&E Team Afternoon · Introductory Mtg w/CIRD staff · Mtg w/CLS Team · Mtg w/Health Systems Team · Mtg w/Health Promoter Coordinator · Graciela Avila, Steve Marma · Diana Rodriguez, Veronica Balanza · Sandra Villalba, Victor Gauto) · All technical staff · Livia Sanchez, Mirtha Mongelos, Magdalena Villagra, Martin Gonzalez · Sandra Villalba, Horacio Spinzi, Manuel Vargas · Esperanza Martinez Wednesday 10/22 Asuncion Morning /Afternoon · Mtg w/USAID · Mtg w/PRIME II · Mtg w/CEPEP · Mtg w/Finance Specialist · Graciela Avila · Gregorio Soriano · Cynthia Prieto · Julio Fernandez Thursday 1/23 Site visits near Asuncion Asuncion Morning · Atyrá: Individual interviews with CLS members: 1) Intendente Municipal, 2) members of Health Committee and 3) Director of Health Post. · Comunidad de Candia (Atyrá): Mtg w/ members of Sub-Heatlh Committee and Director of Health Post. Afternoon · Mtg w/CIRD PD · Dr. Felix Carossini, Director de Centro de Salud · Juan Francisco Ferreira, Mayor · Isabelina Ferreira, CLS member · Maria Crisnilda Romero, CLS member · Lic. Olga Paez, CLS member · Elias Torales, CLS Pres · Celina de Solis, CLS member · Fermina Fernandez, CLS Tresurer · Prof Esteban Jara, CLS member · Victor Rojas, Health Post · Rueben Gaeta Friday 10/24 Site visits near Asuncion Comunidad de Peguajhó (Itá): Mtg w/ Sub￾Health Committee members in Peguajhó and Director of Health Post. Curupicayty: Mtg w/ Sub-Health Committee members in Curupicayty and Director of Health Post. · Soturino Cardozo, CLS Pres · Federico Enciso, CLS VP · Pablo Paez, CLS Secretary · Guillermina Solis, CLS Treasurer, · Ruth Cardozo, CLS Member · Miriam Solis, Director of Health Post · Sergio Enciso, Health Post · Sonia Portillo, Director of Health Post · Miriam Dominguez, Director of Health Post · Gustavo Caputo, CLS Pres · Simeona Coronel, CLS member · Victoria de Sotelo, CLS Treasurer · Hugo Jimenz, Health Service 55 Time/ Location Meeting Schedule/Organization Individuals Friday 10/24 Return to Asuncion Itá-urbano: Separate interviews with 1) Intendente Municipal, 2) member of Local Health Committee, and 3) Director of Health Clinic Lunch with Secretariate of Health of the Central Department Mtg w/CIRD staff · Lic. Mariana Gonzalez de Lopez, Nurse of Health Clinic · Mario Villalba, CLS VP · Lic. Valdez, CLS Secretary · Dra. Riquelme, Health Advisor for new Governor for Central One-on-one meetings to discuss org. structure Sunday 10/26 Afternoon 15:00 hs: Depart Asunción to Encarnación. (Travel time: 5 hours) 20:00 hs: Stay at Encarnación Resort Hotel. Monday 10/27 Viaje a San Miguel. Stay at “Cabaña san Francisco” Morning Encarnación: Visit Parque Sanitorio for Farmacias Sociales de Itapúa. Visit to Farmacia Social de Capitán Miranda. Fram: Mtg w/ Local Health Committee members. Afternoon Coronel Bogado: Individual interviews with: 1) Intendente Municipal, 2) member of Health Committee, 3) Director de Sub Regional Hospital Gral Delado: Programa Vacuna y Farmacia Social · Ramona Barboza, CIRD Coordinator · Dr. Ricardo Pont, Secretario de Salud, Gobernacion de Itapua · Victoria Baez de Espinola, Director of Parque Sanitorio · Dr. Jose Rolon, Director of Health Post · Nicolas Trociuk, CLS Pres · Barbara de Jara, CLS member · Dahiana Diaz, UOL Cnl Bogado · Dr. Marcelin Rodriguez, Dir of Hospital · David Szostak, CLS member · Fatima de Fretes, CLS member · Carlos Acosta · Arnaldo Galeano Tuesday 10/28 Missiones Llegada al Hotel en Asunción. Morning Arazapé (San Miguel, Misiones): Mtg with members of Local Health Committee in Arazapé and Director of Health Post. Itá Yurú (San Miguel, Misiones): Mtg with members of Local Health Committee in Itá Yurú and Director of Health Post. San Juan: Farmacia Social del Hospital Regional Afternoon San Miguel (Misiones): Individual interviews with: 1) Intendente Municipal, 2) members of Local Health Committee, 3) Director of Health Post and 4) Director of Health Clinic · Wilfrida Fleitas, UOL Missiones · Antonia Rojas de Bernal, Director of Health Post · Daniel Medina, CLS Pres · Alberto Corvalan, CLS VP · Dionicia Fernandez, CLS Secretary · Alberta Rodas, Director of Health Post · Claudia Cantero, CLS VP · Secundina Caballero, CLS Pres · Rosarina Reyes, Health Promoter · Prof. Efren Gonzales, CLS Pres and Mayor · Dra Carmen Zorilla de Ramirez, Director of Health Clinic, CLS member and Secretariate of Health for Misiones Department · Don Silvio Lizza, CLS member and Farmacias Social 56 Time/ Location Meeting Schedule/Organization Individuals Wednesday 10/29 Asuncion Morning · Final interviews with CIRD staff · Mtg with CIRD management Afternoon · Debrief CIRD staff · All technical staff Thursday 10/29 Asuncion Morning · Debrief USAID Depart for Airport · Wayne Nilsestuen, Mission Director · Sergio Guzman, Deputy Director · Graciela Avila, Health Project Officer 57 Annex 3: Bibliography of documents reviewed 1. General Project Documents Contract Documents · Attachment 1 Program Description for Cooperative Agreement (CA) 526-A00-01-00100-00 RFA 526-2003/29 for modification No.3 Draft · Recent Modification ? Dissemination Materials · Website description of CIRD Project Internal Documents and Reports · “ Taller Interno del Equipo de Salud: Octubre 2, 2003”. · Progress Report: October 2003 Powerpoint Presentations · Overview of Proyecto de Salud, CIRD/USAID · Overview of “Decentralization of Health Services” for Taller de los Equipos Tecnicos USAID Reports · Workplan Year 1: October 2001 – September 2002 · Workplan Year 2: October 2002 – September 2003 · Workplan Year 3: October 2003 – September 2004 DRAFT · Quarterly Report #1: October – December 2001 · Quarterly Report #2: April – June 2002 · Quarterly Report #3: July – September 2002 · Quarterly Report #4: October – December 2002 · Quarterly Report #5: January – March 2003 · Quarterly Report #6: April – June 2003 2. Technical Areas Component #1: Strengthening CLS Powerpoint Presentations · Definicion del Paquete Basico de Salud. Sept. 2003. Technical Documents and Tools · Censo Sociosanitario: Conceptos Basicos y Herramiento para el Censo · Manual #2. de CIRD: El Cabildeo: una estrategia para incide en las politicas publicas. · Manual de CIRD: Manuel para la Definicion de un Conjunto de Servicios de Salud Esenciales. 2nda Edicion. · Manuel de CIRD. Manejo de reunions como Herramienta para mejorar la efectividad de la organizacion. Sept. 2003. · Planes Locales de Salud. Avances en Paquestes Basicos de Salud Financiamienot Comunitario. Agosto 2003. · Roles y Funciones de los Consejos Locales de Salud. September 2003. · Indicadores de participacion social en salud en componente de fortalecimiento de los CLS. 2003. Work Products · Memoria de la Gestion de CLS en Atyra, Cordillera. September 2003. · Memoria de la Gestion de CLS en Ita Yuru, San Miguel. October 2003. 58 · Memoria de la Gestion de CLS en San Miguel. March 2003. · Organizacion y funcionamiento de los Consejos Locales de Salud. Ita. October 2003. · Seminario taller para Elaborar un Plan Operativo del Plan Local de Salud. Ita. Agosto. 2003. · Seminario taller para Fortalecimiento institutional y vinculacion con el CLS. Cnl. Bogado. Mayo 2003. Component #1: Financing and Management Powerpoint presentations · Sistemas Nacional de Salud de Paraguay: Organizacion y Funcionamiento de los CLS · Sistema de Informacion Administrativa y Financiera (SIAF). Hospital Materno Infantial Villa Choferes del Chaco – Boqueron. Technical Documents and Tools · Experiencias 1. Fondos Rotatorios de Medicamentos Basicos para Farmacias Sociales. La Experiencia del Depto Itapua. 3ra Edicion. · Fondos Rotatorios MSP y BS. Informe Preliminar para comentarios. Agosto. 2002 · Principios y funadamentos de los Fondo Rotatorios de dedicamentos basicos para pispensarios y farmacias socials. June 2003. Component #2: Promotores Technical Documents · Manuel de Apoyo. Modulo III. Programa de Capacitacion a Agentes Locales de Salud. Encarnacion – Itapua. May/Junio 2001. · Diseno del programa de salud comunitaria: Promotores comunitarios de salud. Propuesta 2002. · Guia para capacitadores. Modulo 1: Salud y desarrollo Work Products · Memoria y Conclusion del seminario-taller sobre experiencias Comunitarias en salud. November 2002. Component #3: IEC Management Documents · Convocatora para inscirpcionen el reigstro para implementer estrategias de IEC. October. 2003. Powerpoint Presentation · Plan Estrategico IEC. Technical Documents · Marco Teorico: IEC Que es? · Plan General para IEC · Datos de Diagnosticos realizados para el diseno de indicadores en el marco del plan de IEC. · Resumen de las actividades realizadas en el marco de IEC. 2002 – 2003. Component #4: Dissemination Work Products · El Infomatico de al salud. Ano 1 #1 · El Infomatico de al salud. Ano 1 #2 · El Infomatico de al salud. Ano 1 #3 Monitor and Evaluation Technical Documents · Plan de Monitoreo y Evaluacion: Proyector Health Decentralization and Community Participation. March 2002. · Resumen Ejecutivo. Lineal Basal. · Resumen Ejecutivo. Estudio Evaluativo de la Cuidad de Ita. 59 Annex 4: CIRD Sites Department Municipality Community Health Facility Central Ita Ita-urbano Curupicayty Peguaho Health Center Health Post - State Health Post- State Cordillera Atyra Atrya – urbano Coronel Duarte (Candia) Bernanrdino Caballero Clinic Health Post Health Post Itapua Cnel. Bogado Cnel Bogado – urbano Colonia Cristo Rey Colonia Kurunay Hospital Sub Regional Health Post Health Post Misiones San Miguel San Miguel – urbano Ita Yuru Arazape Clinic Health Post Health Post 60 Annex 5a: Illustrative M&E Plan MONITORING PLAN DATA COLLECTION EVALUATION (Effect/Impact Indicators) PROJECT OBJECTIVES (Inputs, Processes, Outputs) Indicator Threshold Methods/ Instruments Goal: (Describe Project SO) Decentralized community based health care provided List Project IRs in this column Objective One: Strengthened local capacity to manage and oversee decentralized community based health services In this column, describe data collection process. · What data will be collected? Where will be the data be collected? · When will the data be collected? · What resources will be required to collect data? · Who will collect the data? Describe indicators and measurement definitions. For example. #2: CLS, authorities & health facilities technically strengthened to oversee health services · # community census · # communities w/info on production, costs, etc. · # communities in PLS · # of PLS designed · # health surveys · # basic health package defined #3: 3 finance models implemented · local health insurance · rotating funds · cost recovery #8: 12 activities to promote networks and exchanges of information between local actors implemented · Need to define how many activities? over what time period? Describe the achievement level in workplan year toward meeting indicator. For example: · 12 of 12 communities with census in Yr 3 · 10 of 12 communities w/info on costs · etc. (Describe the methodology to be used in collecting data and analyze data. · What method/ instrument will be used to collect the data? What method will be used to analyze the data? · Where will the data be analyzed? · Resources required to analyze the data? · What will analyze the data? · Who will report the data? Objective 2: Increased community participation in own health care & decision-making #4: Health promoter network implemented in coordination with CEPEP · Need to define measurement: ie, what does “in coordination” mean? #5: Referral/counter referral system implemented in select # of departments · Need to define # of departments over what time period 61 MONITORING PLAN DATA COLLECTION EVALUATION (Effect/Impact Indicators) PROJECT OBJECTIVES (Inputs, Processes, Outputs) Indicator Threshold Methods/ Instruments Objective 3: Increased demand for decentralized community based health services through IEC activities #6: # of IEC & Promotion activities implemented · Need to better define measurement: How many? How often? Where? #7: Demand for FP/RH services increased · Need to determine if this is a CIRD or a PRIME indicator GNC DHMT Rose M. Peggy August 2002 Objective 4 Increased support for decentralized community based health services through lessons learned & dissemination of successful examples #8: 12 activities to promote networks and exchanges of information between local actors implemented · Need to define how many activities? over what time period? #9: At lease 15 dissemination activities designed & implemented · # of publications · # of innovative initiatives · # of experiences available on webpage Rose M Peggy Ann Lion/Linda Ippolito GNC November /December, 2002 62 Annex 5b: Illustrative Evaluation Schedule ACTIVITY RESPONSIBILITY PURPOSE PRODUCT TIMING Bring activities from monitoring and data collection column List individuals responsible for leading activity and/or event. Also include participants. Describe purpose of activity Describe output and/or products from activity. Indicate who will receive product. Indicate timeframe 63 Annex 6: Draft Job Descriptions Project Coordinator · Ensure technical quality of CIRD products and technical assistance - Establish CIRD technical direction and focus. Communicates to staff. - Establish technical standards and guidelines. - Establish technical review process and oversee Technical Team Leaders implementation of review process. Occassionly reviews key technical products. - Review and approve/reject new technical activities. · Manage technical activities to produce results - Establish and implement standards approach to program management and implementation, including developing tools, materials, standard report formats and calendar to help staff apply program standard management approach and training all staff in program management approach and in tools and materials. - Implement program management approach through bi-weekly coordination meetings with Technical Team Leaders and weekly coordination meetings with all staff. · Oversee implementation of all project activities - Work with Technical Team Leaders to develop annual workplan. - Monitor progress of activity and country programs according to workplan through bi-weekly meetings and quarterly review meetings. - Manage program budget with F/A Team Leader on monthly basis. F/A Team Leader submits financial reports to Technical Team Leaders for their review. Also submits financial report for Quarterly Reports. - Ensure staff provide information, reports and feedback, including quarterly report, M&E annual reports, annual workplans. - Ensure staff, with assistance from F/A Team Leader, comply with USAID regulations. · Represent CIRD Project to outside stakeholders - Meet with USAID CTO on weekly basis to discuss technical direction, progress toward achieving results, implementation issues and operations. - Serve as Senior Management representative of CIRD project to Alianza partners. - Represent CIRD Project to key stakeholders, such as MOH, World Bank, IDB, CLS, etc. - Provide leadership in policy and advocacy activities supporting decentralization of health services. 64 Technical Team Leaders · Serve as main spokesperson for technical area to both clients and other external audiences. · Serve as technical expert in area for CIRD Decentralization Health Project. · Develop and/or contribute to CIRD Decentralization Health Project long-range planning document and technical area workplan. · Lead and monitor implementation of technical areas workplan once approved by Project Coordinator - Review SOWs for team members. - Review and approve technical directions for all activities in area. - Identify staffing needs, including consultant staff. - Review and assure quality of staff work. - Chair and provide technical guidance to team members at weekly coordination meetings and technical review meetings. - Review and approve documents prepared by team members. - Raise controversial issues with Project Coordinator as they arise. · Monitor operations. - Track monthly budget expenditures to annual budgets. - Work with local F/A Manager to monitor program operations and administration. - Update Mariscal and participate in bi-weekly meetings with Project Coordinator. - Write Annual Workplan and develop budget. Submit to Project Coordinator for review and approval. - Write Quarterly Report, M&E report required by the client. - Assure compliances with USAID rules and regulations as CIRD policies and procedures. · Manage the Team’s work - Make assignments to team members and manage Team’s work. - Assure activity budgets are developed and monitored effectively. - Assure that activities are implemented on time, on budget. - Inform Deputy Director of controversial technical and management issues as necessary. 65 Annex 7: Illustrative Agenda and Talking Points for USAID/CIRD Management Meetings Below is a list of items cover more strategic and forward thinking issues that need to be covered on a periodic basis. Under each item are examples of topics to be considered. The items are not presented in order or priority. As the project ebbs and flows, certain items will become more important than others. Nonetheless, it is important to review them consistently to ensure both Project Management and USAID review them as a team. · Pipeline analysis, other budget related and contractual issues - Review pipeline - Review financial disbursements - Discuss changes or modification to budget - Discuss upcoming procurements (scope, activities, resources needed) - Inform client of any compliance issues · Counterpart and stakeholder relations - Discuss status and changes in relationships with CLS - Inform USAID of any new relationships supporting Project activities - Discuss changes in political environment and opportunities and obstacles present to Project - Discuss need to cultivate new partners, if needed, to support Project activities · Staffing issues - Inform USAID of any major staffing changes (ie, departure, addition) - Discuss staffing needs as they arise (new skills, different skills, etc) - Inform client of needs for international expertise - Inform client of any problems associated with staffing (ie, too little, too many, not right technical skills, etc.) · Technical performance - Discuss technical direction and priorities - Discuss new and emerging technical trends in field work - Present new tools and methodologies - Discuss new technical strategies and/or approaches - Inform USAID of important technical events - Give new technical reports, products, etc to USAID - Present key findings, results and lessons learned as they occur - Discuss any problems associated with technical performance · Implementation issues - Review status of activities according to workplan schedule and budget - Inform USAID of upcoming activities in near future - Discuss any problems associated with operations (logistics, staffing, right skill mix, etc) - Discuss any problems associated with implementation (turn-over in CLS members, changes in political environment, CLS not responding to TA, etc)