PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 1 Improving RH Quality PRIME II Mid-Term Assessment January, 2003 SO 526 006 “Use of Voluntary Reproductive Health Services Increased “ CA HRN-A-00-99-00022-00 Prepared by Josceline Betacourt Ann Lion-Coleman Barbara O’Hanlon PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 2 Table of Contents 1.0 Introduction 1.1 Purpose of Mid-term Evaluation 1.2 Scope of Mid-term Evaluation 2.0 Background 2.1 Evolution of PRIME II’s scope of work in Paraguay 2.2 Evolution of the sites 2.3 Evolution of the partnership with Alianza 2.4 Change in Program Directors 3.0 Progress in Implementing Program Activities 3.1 Implementation 3.2 Technical Program Access Quality National Reproductive Health Policy Monitoring and Evaluation 3.3 Management 4.0 Progress in Meetings Results 4.1 PRIME II Program linkages to USAID/P Results 4.2 Measurable progress toward Project Objectives and USAID/P results (Findings) 4.3 Measurement Tools and Methodologies 5.0 Recommendations to Strengthen Implementation of the Technical Program 5.1 To Strengthen Implementation 5.2 To Strengthen the Technical Program Access Activities Quality Activities Policy Activities M&E Activities 5.3 To Strengthen Management 5.4 To Strengthen Alianza Partnership PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 3 1.0 Introduction 1.1 Purpose of Mid-term Evaluation During January 21 through 31, 2003 USAID/Paraguay and the PRIME II project conducted a joint evaluation of the PRIME II Quality Project in Paraguay. The objectives of the midterm evaluation were: · to analyze the implementation of the Agreement between PRIME II and USAID/P, · to review the progress in the implementation of the workplans and toward expected results, and · to provide recommendations to PRIME II and USAID/P to modify or change the activities for the remainder of the life of the Quality Project. 1.2 Scope of Mid-term Evaluation The Evaluation Team critically reviewed the following aspects of the PRIME II Quality Project, seeking to identify strengths and weaknesses and to recommend corrective actions in the following three areas: Technical progress toward achieving results · Do Program documents present expected results in clear, unequivocal and measurable terms? Are the expected results realistic and commensurate with human and financia l resources available? · Is the PRIME/P strategy and workplan clearly linked to the USAID/P results and does it contain the right type and mix of technical interventions that will lead to the desired results? · What measurable progress has been made toward each of the Program objectives? Does this progress translate into visible improvement at the health facilities? · Do PRIME/P monitoring and evaluation tools and processes adequately measure the desired results? What changes and improvements are recommended? · If there have been problems achieving the stated results, what have they been and why? How do the realities of the Paraguay health sector impinge on the achievement of Program results? · How can the strategy and/or workplan be modified to address these challenges? Management support toward achieving results · Do PRIME/P staff and consultants have the appropriate competencies required to effective assist the health provider clients? How is PRIME/P staff perceived by their MOH staff at the health facilities? Are there enough staff and consultants to carry out the work plan? · Does the budget adequately support Program activities in the field? Are funds being invested appropriately, i.e. are they directed to the most critical program needs? · Does Program management review work plan activities and budgets on a regular basis to ensure that activities are on-time and on-track to deliver results? · Do PRIME/P staff and consultants receive adequate and timely guidance and feedback on their work from Program Director? From LAC Regional Office (RO)? From Headquarters? From USAID/P? · Does PRIME/P fulfill its reporting obligations to the in a timely and adequate manner? Is Mission management satisfied with the quality of PRIME/P reports? Any recommendations for improvements? Quality of PRIME/P training and technical assistance · Does PRIME/P provide timely and quality training and technical assistance that responds to client needs? Do MOH health providers perceive value in PRIME/P interventions? · Is the content and methodology of PRIME/P training and technical assistance appropriate to the needs of the MOH health care providers at the Program health facilities? · How well do PRIME/P staff work with the counterparts? With USAID/P? With Alianza partners? · What is positive and you want PRIME/P to continue and/or expand? PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 4 · How can PRIME/P improve the quality of their technical assistance? 2.0 Background Below is a description on how various aspects of the PRIME II Quality Program, such as scope, program sites, partners and implementation strategy, have evolved over the last 18 months of the program. 2.1 Evolution of PRIME/P’s scope of work in Paraguay PRIME II submitted a proposal to USAID/P in April 30, 2001 that provided the conceptual framework and overall design for the Quality Program. The design outlined in the original proposal remains valid and continues to guide and inform the workplans proposed in Year 1 and Year 2. The overall strategy areas – increase access, improve quality and strengthen RH related policies – continue as the three primary PRIME/P strategies in assisting USAID to achieve its Strategic Objective. Table One demonstrates that although the strategies remain constant, the scope of activities has became more focused and defined over time. The April 2001 proposal was ambitious. Under the first strategy – improving access – PRIME/P planned activities to address several factors preventing access to RH services, including geographical, medical and knowledge barriers. The second strategy – improving quality – proposed a comprehensive yet ambitious set of interventions requiring diverse staff with different skills. The activities proposed for the third strategy – strengthening RH related policies – appeared to be an appropriate level of effort given its relative (less) priority to other strategies. The Year 1 Workplan underwent a reduction in activities, reflecting a tighter focus and more realistic approach to implementing the PRIME/P program. PRIME/P acquired some experience during the initial months in Paraguay and USAID/P wisely insisted that PRIME/P reduce the scope and number of sites to create a more feasible Workplan. As Table One denotes, the technical focus, as represented by the three strategies, remained the same in the Year 1 Workplan. Under Strategy One, diffuse activities like mobile clinics and improving national logistical systems were omitted. Instead, access related activities focused on community awareness through IEC, working through Local Health Councils, closer coordination with health stakeholders at the community level. Quality related activities under Access, like guidelines and norms, referral/counter referral systems and supportive supervision for rural providers, were relocated under the Quality objective and integrated into the Quality activities. The Year Two workplan further defined and focused the PRIME/P activities. Access related activities continue to concentrate on strengthening providers’ IEC skills and community awareness. Interestingly, some of the quality related activities that were integrated into Quality in Year 1 strategy re-appear under Access in Year 2. These activities are referral/counter referral systems, supportive supervision and COPE (Gestion de Calidad). Under Quality, training interventions were defined according to needs. They are: a) contraceptive technology, b) infection prevention, c) post-partum/post-abortion FP methods, and d) counseling. Three technical activities originally proposed remain undefined and merit further discussion between USAID/P and PRIME/P. They are: · Certification and promotion of quality services: All three of the Alianza partners – CEPEP, CIRD, and PRIME/P – are interested in this activity but have very distinct ideas on how to carry this out. The proposals range from a very complicated US-based hospital accreditation model to a more simple promotion model based on the USAID sponsored Gold Star program in Indonesia. Further direction on the approach and respective organizational roles in this activity from USAID/P will greatly assist Alianza on how to proceed in a unified manner. PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 5 Table 1: Overview of 2001 Proposal and Year One and Year Two Workplans April 2001 Proposal Year 1 Workplan 7/01 – 6/02 Year 2 Workplan 7/02 – 6/03 EXPANDING ACCESS TO RH SERVICES 1.1. Ensure local capacity to deliver services by · Develop FP/RH performance standards · Develop service delivery guidelines for rural providers · Train & certify providers · Implement supportive supervision system · Use mobile clinics to increase access 1.2. Establish effective referral systems · Develop easy to use system · Establish referral/counter referral system · Train in its use 1.2. Decrease medical & policy barriers · Disseminate FP/RH norms & protocols · Increase awareness · Disseminate Ntl RH Strategy · Improve contraceptive logistics · Work w/MOH to increase client awareness of services EXPANDING ACCESS AMONG RURAL WOMAN TO RH SERVICES IN 4 AREAS 1.1 Increase awareness of RH Services though marketing & IEC activities 1.2 Develop RH network of services 1.3 Create strategic alliances with Local Health Councils 1.4 Facilitate dialogue & alliances at the local level between MOH & NGOs, TBAs & local leaders EXPANDING ACCESS TO RH SERVICES 1.1 Increase knowledge & skills of clients in appropriate use of RH services thru evaluation of materials, IEC activities, providers’ counseling skills in collaboration with Alianza 1.2 Increase coverage by strengthening network of RH services 1.3 Strengthen Regional Supervision System using facilitative supervision 1.4 Strengthen select # of facilities apply COPE tool to improve quality IMPROVING QUALITY OF SERVICES 2.1 Improve providers’ clinical skills thru tailored and OTJ training on FP/RH concepts, CTU, Infection prevention 2.0 Improve provider interpersonal skills to provide effective counseling, informed consent, client respect 2.1 Apply COPE as methodology for providers to assess & take actions 2.2 Establish supportive supervision to ensure continuous support & training of providers 2.3 Coordinate with FPLM Project on contraceptive supply and logistic 2.4 Certify quality FP/RH services IMPROVING QUALITY OF RH SERVICES 2.1 Improve maternal health outcomes thru TA to update norms & integrate into 4 areas; conduct PNAs to better target TA to improve provider performance; improve providers’ clinical skills 2.1 Improve quality of FP services thru contraceptive updates; improved supportive supervision; and improved referral systems within networks 2.1 Improve quality of adolescent RH programs thru improving provider skills to work effectively with adolescents, improving access & quality; focus areas include imp roving provider interpersonal skills, implementing referral system for adolescent care & establishing peer-counseling clubs 2.1 Strengthen MIS in 4 priority areas, building on recommendations from Deliver Project IMPROVING QUALITY OF RH SERVICES 2.1 Strengthen providers ability to provide quality services 2.2 Train providers in quality FP services using different training methodologies 2.3 Train providers in infectious disease control 2.4 Train providers in use in maternal health norms and protocols 2.5 Train providers in select hospitals in Post-partum and post-abortion FP services and use of MEU 2.6 Integrate and support providers counseling skills through facilitative supervision 2.7 Monitor provider performance thru TA 2.8 Conduct M&E activities, establishing a baseline, implementing a diagnostic of facilities quality and monitoring progress in achieving quality IMPROVING RH NATIONAL POLICIES 3.1 Evaluate current RH Strategy 3.2 Assist MOH to design & implement 2002 RH National Plan IMPROVING RH NATIONAL POLICIES 3.1. Evaluate current RH Strategy 3.2. Assist MOH to design & implement 2002 RH National Plan IMPROVING RH NATIONAL POLICIES 3.1 Evaluate 2001 National RH Strategy 3.2 Assist National RH Council to formulate 2001 RH National Plan 3.3 Provide TA to disseminate new plan PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 6 · Quality providers: The original proposal indicated PRIME/P would work with a variety of providers including but not exclusively public sector providers. USAID/P would like PRIME/P to revisit this assumption in light of the fact that PRIME/P is only working with MOH providers and explore the feasibility of including private and NGO sector providers, including physicians, pharmacists, etc. · Contraceptive logistics: Contraceptives are a constant theme throughout the PRIME/P initial proposal and two workplans. PRIME/P carried out an assessment of the logistics situation in November 2001 however follow-up activities have not taken place. The site visits demonstrate that contraceptive stock-outs and shortages present a significant barrier to PRIME/P quality sites if not addressed. 2.2 Evolution of sites In January 2002, USAID/P approved the PRIME/P workplan to work in 32 sites. The 32 sites were selected through a participatory process involving PRIME/P staff, Ministry of Health staff from both the national and regional levels. PRIME/P held 4 meetings, inviting the MOH as well as representatives from local government and communities in the 4 Departments identified for activities. Table Two lists the evolution of PRIME/P sites. Originally, PRIME/P proposed a large number of sites but USAID/P encouraged PRIME/P to reduce the number, stating that SO required fewer sites to achieve its objective and that USAID/P wanted PRIME/P to balance improving quality at a select number of sites with developing a model to replicate quality at other sites in Paraguay. By December of 2001, PRIME/P and USAID/P agreed on 32 sites and PRIME/P commenced activities in 5 of the 32 facilities. In March 2002, the sites were modified, changing the location and composition of the health facilities. The reason for the change in site selection is described below in Section 2.3. Table Two compares the original 32 sites with the current 23 sites: 14 health facilities were taken out of the original list of 32 and 7 new facilities were added. The revision also added a new region - Cordillera - to the Program. The selection criteria for the new sites were: · the sites would correspond to a CEPEP collaborating community and/or CIRD Local Health Council, and · the sites would represent each level (primary, secondary and tertiary levels) required in a referral and counter-referral system. The change in sites was disruptive and created a break in implementation momentum. PRIME/P had established relations with all 32 sites and begun training and technical assistance in COPE, adult le arning theory as well as completing a full baseline assessment in all 32 sites. In April, they had to inform 14 facilities they would no longer continue working with them. Moreover, PRIME/P staff had to established contact and build relations with 7 new facilities and reprogram all their activities to adjust to the changes in site selection. To accommodate for this significant change, PRIME/P adopted a new implementation model, initiating intensive training and technical assistance in Misiones in June, 2002. PRIME/P staff rolled out this intensive implementation approach in Itapua in September and Central in November, 2002. The PRIME/P staff maintained this concentrated level of effort until the end of the 2002, reaching all 11 sites. PRIME/P is scheduled to begin activities in the remaining 13 Quality sites in February and March of 2003. PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 7 Table 2: Evolution of PRIME II Sites !st List of Sites List of Sites (Current sites 2002) List of Sites (To be launched 2003) Asuncion · Hospital Barrio Obrero · H.M.I. Trinidad · C.S. Santa Ana · C.S. Perpetuo Socorro Asuncion Asuncion · Hospital Barrio Obrero · H.M.I. Trinidad · C.S. Santa Ana · P.S. Botanico Centrales · Hospital Nacional Itaugua · Hospital San Lorenzo · H.D. de Aregua · H.D. Capiata · H.D. Fernando de la Mora · P.S. Posta Ybycua · P.S. San Miguel · P.S. de S.P.S.- Con. de Consejo · P.S. de S.P.S. - Potrerito · P.S. de S.P.S. - Villa Ofelia Centrales · C.S. de Aregua · C.S de Ita · P.S.de Peguajho · P.S. de Curupicayty Centrales · Hospital Nacional Itaugua · Hospital San Lorenzo Cordillera Cordillera Cordillera · Hospital Regional Caacupe · C.S. Atyra · P.S. Coronel Duarte · P.S. B. Caballero Itapua · H.R.Encarnacion · C.S. Coronel Bogado · C.S. Fram · P.S. Cristo Rey · P.S. Curunai · P.S. Santo Domingo · P.S. San Antonio · P.S. Uru Sapucai · P.S. San Pedro · P.S. Mboi Ca’e · P.S. Itaindy Itapua · H.R. Encarnacion · C.S. Coronel Bogado · C.S. Fram Itapua · P.S. Cristo Rey · P.S. Curunai Misiones · H.R C.S. San Miguel · P.S. Ibanez Rojas · P.S. Arazape · P.S. Itayuru Misiones · H.R. San Juan Bautista · C.S. San Miguel · P.S. Arazape · P.S. Itayuru Total – 32 Total - 11 Total - 12 2.3 Evolution of the Alianza partnership While USAID/P reached an agreement with PRIME/P in May 2001 on the program design, there was a time lag in finalizing the scopes of work with CEPEP (IPPF affilia te) and CIRD. The scopes of work for CEPEP and CIRD under the Alianza initiative were finalized in early 2002. Prior to this time, PRIME/P participated in coordination meetings, but without clear outcomes. By March 2002 it became clear that PRIME/P would need to work in a more coordinated fashion with Alianza. However PRIME/P’s geographic areas did not overlap or correspond with those of the Alianza partners. PRIME/P therefore changed both technical and geographic direction in order to have a more coherent overall program with Alianza. At that time, PRIME/P further focused efforts on the providers at the site level. Alianza’s focused on mobilizing the community and promoters. Clarification of each of the organization’s roles and PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 8 responsibilities has helped improve working relations among three organizations and facilitated greater coordination. All the partners recognize more needs be done to operationalize this coordination at the regional and site levels. 2.4 Change in PRIME Management The PRIME/P started under the direction of Denis Zaenger. After 10 months, both the PRIME/P LAC/RO and USAID/P agreed to replace Mr. Zaenger for a more effective Project Director. Lic. Leonel Valdivia arrived in May, 2002. 3.0 Progress in implementing program activities Below are several findings regarding progress-to-date in implementing the Paraguay Quality program. The conclusions are organized into the following sections: 3.1) implementation, 3.2) access, 3.3) quality at the site level, 3.4) policy at the national level, 3.5) monitoring and evaluation, and 3.6) management. 3.1. Implementation Key Findings on Implementation: · The PRIME/P program got off to a slow start in Paraguay. The first Project Director proposed an overly ambitious and complicated workplan that underestimated the technical, staffing and financial requirements to implement it. USAID/P spent an enormous amount of time negotiating with the Project Director, trying to inject some realism into the workplan. As a result, it took six months to agree upon a more feasible scope of work, articulate activities and approve them. · There is a misalignment between Access and Quality Strategies. During the last 18 months, PRIME/P has become more integrated into USAID/P’s other health programs under the Alianza initiative. As the roles and responsibilities for each of the Alianza partners became clearer, many of the access related activities originally designated for the PRIME/P program were reassigned to other collaborators because of their linkages to community and local governments and/or their expertise in demand generation activities. As a result, PRIME/P has very few activities left under the Access strategy in its Year Two workplan, creating an imbalance among the activities between the Access and Quality strategies. The remaining activities under Access focus primarily on the strengthening provider’s ability to interact with the community. · There has been much discussion among staff about the two different implementation models but what has resulted is a hybrid between the two approaches. The first was a gradual implementation model in which the training and technical assistance were rolled out over time and in several regions simultaneously. Once the initial interventions were implemented, PRIME/P staff would continue monitoring the sites through follow-up visits. The second model – a concentrated model – was initiated in June, 2002. All staff were mobilized to work together in one region to implement all training and technical assistance in a condensed timeframe. Once achieved, the regional coordinator would follow-up at the sites in this new region with regular visits. The team then moves on to the next Region and starts the process again. In reality, what has happened is in-between both approaches. Some activities are being rolled out over time in multiple sites (like the referral system) while others have been implemented in an intense and short timeframe (such as the multiple training modules). The Evaluation Team could not find documentation describing the either implementation approach. The hybrid approach has effectively jump started activities and helped compensate for missed time due to turn-over in Program Directors and change in sites: staff are implementing a lot of activitie s successfully, effectively and quickly. PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 9 But the lack of documentation has created some confusion on exactly what is the new implementation model and how is it implemented, resulting in some inconsistencies. For example, in Misiones, the facilities had not received a supervision visit in two months whereas in Central, facilities received a follow-up visit every week in the last month. · Clearly the PRIME/P Program has developed momentum with its recent activities at the 11 sites. The new Project Director hit the ground running and in record time, was able to draft a work plan and get it approved by USAID/P. He also hired staff to get the technical program jump started in all the new sites. He has also established good working relationships with other Alianza organizations. The Program is well poised to maintain its momentum with a few technical and management adjustments outlined in the Recommendation Section. Areas to Strengthen in Implementation: · The conceptual framework for the PRIME/P program design is not reinforced and shared consistently with the staff. As a result, the PRIME/P staff members are implementing, albeit effectively, a series of activities that are not related or well integrated with each other. In particular, the activities within a strategy area – access, quality or policy – do not connect with those in the other strategies. Also, staff have limited understanding on how their work contributes to the overall program and its achievements. · The vision of a quality reproductive health services focuses almost exclusively on improving provider performance, overlooking organizational factors that impede quality and access. The majority of PRIME/P technical assistance and intervention help providers improve their technical competence, acquire new skills and perform to standards and norms. Staff are cognizant of the imbalance and are looking for ways to also include technical assistance that address some of the service delivery issues - such as hours, staffing patterns, pricing, supply, etc – that present access barriers to users · To date, PRIME/P approaches and strategies are not well documented, therefore not meeting USAID/P expectation of building models to replicate at other sites and departments. The Evaluators were able to find many of the critical pieces that would be included in a concept piece on PRIME II’s approach, including a technical strategy, workplans, goals, objectives and benchmarks, calendar of activities by regions, training curriculum with supporting training materials, monitoring and evaluation reports, etc. All of these important pieces are not are not pulled together into one or multiple document(s) that clearly document the PRIME/P quality model. · It will be difficult to sustain the current implementation model - a hybrid between a gradual and concentrated approach. First, the current staff and assignments can barely keep up this rhythm of work and they have yet to launch the remaining 12 sites scheduled for this Spring. Of the 12 sites, 5 are major hospitals that will require more level of effort and follow-up than the health clinics and health posts. Second, the implementation model does not build counterpart’s capacity to oversee and assure ongoing implementation of quality model. 3.2. Technical Program The PRIME/P has developed a mix of technical interventions that are yielding results at the facility level. The basic package – or toolkit if you will – is outlined below in the table. Under Access, the interventions include: a) strengthening providers ability to interact with the community through IEC activities, b) establishing referral/counter referral system, c) reinforcing provider’s new skills and technical competencies through facilitative supervision, and d) addressing access and quality issues at the facility through COPE. Under Quality, PRIME/P is applying innovative training approaches to transfer core skills and competencies in FP contraceptives technology, infection prevention, post-partum and post- PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 10 abortion FP methods. They are also distributing and helping providers apply FP/RH norms. Finally, PRIME/P staff are improving provider interpersonal skills through training in counseling and providing necessary IEC materials. PRIME/P staff are using primarily training – either in workshop or on-the-job format – and follow-up visits as the means to transfer and build provider capacity in these technical areas at the 11 sites. PRIME II Toolkit Access IEC Access Referral System Access Facilitative Supervision Access COPE Quality FP Methods Training Quality Infection Prevention Training Quality Post Partum Post Abortion Methods Training* Quality Norms Training, Distribution & Use Quality Counseling Training & IEC Materials * only at appropriate sites The PRIME/P staff receive excellent marks from the counterparts on the quality and relevance of the training and technical assistance received. Many of the providers interviewed at the sites stated that the training workshops “were excellent”, “participatory” and “necessary”. The majority of providers who were interviewed and observed referred to, used and applied the information and skills learned in a PRIME/P training and/or technical assistance visit, indicating the topics and themes selected are relevant and timely. The providers hold the PRIME/P staff in high regard and consider PRIME/P staff to have the appropriate skills and experience. The providers interviewed also remarked on the excellent working relationship they have with PRIME/P staff, commenting that PRIME/P staff are our “partners” and “work with us”. The providers also offered constructive advise to PRIME/P on how to improve their training, program areas they need to strengthen, and what important skills/issues to include in the training: · Providing a more continuous learning approach instead being supervised · Providing more training and help in applying norms and standards · Providing training in STD diagnosis, treatment and counseling and support materials on STDs · Providing training in and support materials on pre-natal visits and emergency obstetric management · Addressing contraceptive supply issues · Assisting providers to better connect with community (desire for more connection with CIRD) and to work on issues like maternal mortality, etc. Below is a discussion of the findings and areas to strengthen by technical strategy area – Access, Quality and Policy. To inform the discussion, Table Three presents the status of different activities implemented by sites. In addition, this section contains key findings on Monitoring and Evaluation. Key Findings on Access Activities: · Several of the technical areas proposed in the Year 2 Workplan do not directly support the Access outcome but instead, support quality. For example, referral and counter-referral systems, COPE (Gestion de Calidad) and Facilitative Supervision, are all activities that have traditionally been used to improve and/or strengthen quality. Some of the confusion on what would be appropriate Access activities for PRIME/P stem from the shifting scopes of work between the Alianza partners. PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 11 Table 3: Overview of Activities Implemented by Sites Sites Access IEC Access Referral System Access Facilitative Supervision Access COPE Quality FP Methods Training Quality Infection Prevention Training Quality Post Partum Post Abortion Methods Training* Quality Norms Training, Distribution & Use Quality Counseling Training DEPARTAMENTO CENTRAL C.S. Itá ÷ ÷ ÷ ÷ ÷ C.S. Areguá ÷ ÷ ÷ ÷ ÷ P.S. Peguajhó ÷ ÷ ÷ ÷ ÷ P.S. Curupicayty ÷ ÷ ÷ ÷ ÷ DEPARTAMENTO DE ITAPUA H.R. Encarnación ÷ ÷ ÷ ÷ ÷ ÷ ÷ ÷ H.Sub R. Cnel. Bogado ÷ ÷ ÷ ÷ ÷ ÷ ÷ ÷ C.S. Fram ÷ ÷ ÷ ÷ ÷ ÷ ÷ ÷ DEPARTAMENTO DE MISIONES H.R. San Juan Bautista ÷ ÷ ÷ ÷ ÷ C.S. San Miguel ÷ ÷ ÷ ÷ ÷ ÷ P. S. Arazapé ÷ ÷ ÷ ÷ ÷ P.S. Itayurú ÷ ÷ ÷ ÷ ÷ PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 12 · All sites have IEC posters and brochures produced by PRIME/P that providers use with their clients. PRIME/P has focused their IEC efforts, limiting them to IEC activities that take place within the clinic or facility walls (intra-mura) while Alianza will undertake IEC outside the facility and in the community. This division of labor on IEC has not translated into greater coordination at the site level among partners. The Evaluators determined through interviews of site staff and Alianza technical staff that IEC activities are not promoting health services where PRIME/P works. Moreover, health promoters or Local Health Councils do not call upon the PRIME/P providers to participate in community activities or Council activities. Areas to Strengthen in Access: The Access related activities have not received as much level of effort as the quality activities. Table Three summarizes the activities that have been implemented at the 11 sites. · As Table Three demonstrates, there have been minimal activities under Networks. There have been some initial discussions in Itapua with the providers on the referral and counter-referral systems and how this activity will be implemented. It is interesting to note that where these meetings have been held, they have begun to produce results. One Hospital Director stated that relations and coordination with other sites have improved as a result of the Networks workshops. The Director sited the example of one Health Center with four emergency cesarean sections at the same time but only had instrumentation and sterile clothing for two. With the new trust level and established communication, instead of simply referring the two cesarean sections to the Hospital, the Health Center Director called the Hospital and negotiated: “either send me the instrumentation and sterile clothing or we will refer two C-sections.” The Hospital quickly sent over the instrumentation and sterile clothing to the Health Center, resulting in better service for the patient. Other sites have bartered for equipment: one Hospital had an extra aspirator and donated it to another Hospital; a second Hospital had an extra autoclave and traded it to another Hospital that did not have one. Despite some progress, this activity is behind schedule and has not started in all 4 regions. The Evaluation Team read a concept piece that: a) proposed an approach that exceeded the scope of establishing a referral/counter-referral system, b) did not clearly articulate a strategy and c) did not describe how the activities were to be implemented. · The 11 sites are at different levels of proficiencies in applying COPE/Gestion de Calidad to strengthen the quality of their services. All 11 sites have received training in COPE. In addition, the sites have conducted some form of analysis of barriers to access and quality, and developed an action plan to address them. There has not been, however, reinforcement with the providers at the site role of Gestion de Calidad as a tool that can be used continuously to strengthen quality and/or follow-up on progress with their action plans. PRIME II expert Miriam Parra conducted a comprehensive assessment of the status of Gestion de Calidad activities in all of the original sites. Of these, 11 had already been working (inconsistently) with COPE prior to PRIME/P’s arrival. PRIME II’s initial, efforts therefore, focused on following up with these sites in addition to introducing COPE to the new sites. Quality Improvement Plans were developed by virtually all the sites (27). Those missing were small health posts. Although the assessment made several recommendations that were site specific, two overall recommendations emerged which guide the Gestion de Calidad activities to-date: 1) the importance of monitoring and supervising the implementation of the quality improvement plans and 2) the need to involve the site directors in the plans development and implementation. However, these recommendations have not been fully followed in the Year 2 Workplan. · There have been insufficient activities under facilitative supervision. PRIME II LAC supervision expert Consuelo Juarez conducted an assessment in all five regions. This work included a situational PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 13 assessment of the on-going supervision status through interviews and observations. She worked with the MOH Directors of RH at the regional level in all of the five PRIME regions. She also interviewed the seven hospital directors (except Region 8) where PRIME/P was working. As a result of the assessment, she developed a regional training plan in facilitated supervision. However, with the Year 2 workplan, the follow-up activities were not included. There has been some confusion regarding this technical intervention. The PRIME/P staff conduct frequent follow-up visits with providers at the sites after the training that has been referred to as “facilitative supervision”. This confusion may persist in the absence of a defined methodology as well as programmed activities for facilitative supervision. Key Findings on Quality at 11 Sites Visited: The Evaluation Team visited all 11 sites, using the instrument to guide their interviews and observations. Please refer to Appendix A for a list of the sites and persons interviewed and Appendices B and C for the instruments. Below is a description of the key findings on quality from the site visits. Section 4 will outline the quality findings from the monitoring report. · All sites have separate FP exam rooms with full-time dedicated staff attending to FP clients. In many cases, there is clear signage to direct people to the FP counseling area and/or exam room. The exam rooms are clean and offer privacy for the clients. Some exam rooms even have amenities, such as privacy curtains and wooden hangers for clothing offered by local Lions Club or the Local Health Councils. The exam rooms are equipped with educational materials, such as FP posters and flipcharts. · All FP staff are enthusiastic, motivated and with a sense of empowerment. All the providers who attended a PRIME II workshop or on-the-job (OTJ) training were eager about the new skills and knowledge they acquired. One nurse midwife stated, “For me, PRIME is very important for my work – I am very happy and more secure…” A Hospital Director shared with the Team that.. “What I liked the most was the workshop on Quality of Care….. it awakened us to the necessary changes”. In addition, the providers felt empowered to apply these new skills in their work, as observed by the Evaluation Team at the sites. · All FP staff is conversant in FP methods and has necessary support materials. It is interesting to note that the IEC materials are not consistent and produced by a variety of USAID sponsored projects (JHPIEGO, EngenderHealth, CCP/JHU). When asked, staff could properly describe the pros and cons about each method. All sites properly stored and tracked FP methods. · All sites are implementing consistent standards of infection control for FP activities. Infection control is one of the success stories, with demonstrable changes in behavior as well as infrastructure. All staff discussed how they now wash hands before and after a client visit. At one health post, a community health worker informed the Evaluation Team that she never realized how many germs she carried on her hands after being out in the community and that now she is very conscious about washing her hands. Many sites have created a separate, closed fresh water container with a spigot, with special soap, towel and bucket for hand washing. All sites have separate containers with a narrow opening for disposal of sharp materials. Hydro chlorate solution and instrument buckets are now available in the examining room and the provider is clear on how to mix the solution. Where there is a sterilization stove for instrument, providers know how to use it. · There have been dramatic changes in interpersonal relations between providers and their clients. Many stated how they treat their clients differently, using phrases like “we take the time to talk to them and get to know them” or “we put ourselves in their shoes” or “we greet the clients in the hallway when we see them” or “when we send our clients to other departments on-site, we walk with PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 14 them.” The Evaluation Team was unable to confirm this change in attitude and behavior from the consumer perspective. Virtually all staff, including those not working the FP, is wearing nametags. Counseling skills and techniques have improved. Some sites have re-organized staff to allow for a full-time FP counselor. All sites indicate they provide all clients individualized counseling on available methods, presenting the pros and cons of each method before the client chooses the method. All staff use available IEC materials in counseling. · All birth sites are providing post-partum/post abortion FP methods on-site. Many providers stated they were appreciative of the training and opportunity to learn how to insert IUDs. One site indicated that demand for IUDs has doubled due to better counseling and ability to insert them on-site. Areas to Strengthen in Quality: · Different and innovative approaches in training have yielded more results at the site. PRIME/P staff have used a variety of training methods to strengthen provider knowledge and skills in the four core areas (refer to Toolkit). The recent monitoring and evaluation report demonstrated that the OTJ used in Central has produced dramatic changes in behavior performance in a short-time period. Moreover, providers have indicated that they prefer training that is conducted on-site and more closely linked to their actually jobs. · Follow-up is critical to increased and sustained use of new skills and knowledge. All sites and all providers indicated that follow-up visits helped reinforced and to remeber what they learned. Also, the problem-solving approach and discussions that take place during the follow-up visit encouraged the providers to apply what they learned. The follow-up visits motivated providers: many stated that they appreciated the fact that PRIME II “do not abadon us after the training” and that “PRIME/P work along with us”. The Evaluation Team, however, observed that the follow-up visits were inconsistent in terms of the number of times the staff visited the sites, the time when the staff visited the sites and the content of the site visits. It is important to develop a follow-up “protocol” the would systematize and structure the follow-up visits while ensuring regular visits. The providers indicated they want mothly, follow-up visits that are jointly planned. · All staff have an updated copy of the MOH norms and standards for maternal and reproductive health. Indeed, all staff had the norms prominently displayed in the exam room or counseling room. Moreover, the staff were cognizant of the fact that it is important to provide services the follow the norms and standards. What was less evident was if the staff opened the norms to consult them. PRIME II staff need to find “creative” approaches to motivate MOH staff to refer to and apply norms. · Most sites experience stock-outs and/or shortages in critical supplies like contraceptives and gloves. PRIME/P staff need to evaluate the source of this problem and to work with Alianza partners to identify creative solutions to address supply issues. Key Findings on Policy Activities: · PRIME/P was charged with developing the methodology for evaluation of the National Reproductive Health Plan and carrying it out. As a result of a number of factors, the Evaluation was six months late. The main reasons for delay included: difficulty designing the methodology locally which required assistance from Chapel Hill in March 2002, identification of a coordinator who proved incapable of following the design, which then required the brand new Project Director to essentially oversee the day-to-day implementation of the evaluation. As a result of the delay, the PRIME/P staff formed a small working group comprised of members from the Grupo Conductor to finalize the analysis and draft the Evaluation document. The formation of this working group produced positive and PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 15 unanticipated consequences: a) the Grupo Conductor acquired in-depth understanding of the current National RH Policy, with its strengthen and weaknesses, b) the working group discussions shaped the Grupo Conductor’s clear expectations for the next National RH Policy, and c) informed their thinking to make recommendations on a new process to develop the next National RH Policy. The Grupo Conductor is satisfied with the Evaluation Report, including the MOH even though the Report identifies some of the Policy’s shortcomings. The Grupo Conductor is also satisfied with PRIME/P’s role, receiving high marks on its leadership and technical conduct in developing the Evaluation Report. Areas to Strengthen in Policy1 : · Currently, the PRIME/P Program does not have a plan in place to assist the Consejo Nacional to formulate next National RH Plan. Moreover, there is no one among the PRIME/P staff assigned with the responsibility for this activity. The absence of a plan and staff person will limit PRIME/P’s ability to help the MOH meet its August deadline of producing a draft of the next National RH Plan before Dr. Bataglia leaves his post. · PRIME/P Program does not have a staff member on board with the appropriate skills to lead this activity. The Evaluation Report recommends a participatory process that involves many of the key groups, both in- and outside of the health sector, to develop a draft Plan. The PRIME/P staff are primarily clinicians with extensive MOH background. This activity requires a generalist with strategic planning and facilitation skills. Key Findings on Monitoring &Evaluation Activities: The Monitoring and Evaluation Baseline was late by six months. Main reasons for this delay was the change in sites in March 2002 which resulted in carrying out new baseline efforts in 15 new sites, lack of consensus on the presentation of the baseline, and overall delay in presenting the final report due to the many other priorities. The Program now has a baseline for all 23 sites and has also collected date for the first monitoring report. This activity is now on track to produce the necessary reports and analysis to demonstrate that PRIME/P has or has not established quality services in 23 sites. The baseline information collected includes the following: ÿ Infrastructure inventory ÿ Provider interviews ÿ Observations of providers ÿ Client exit interviews ÿ Service statistic review PRIME/P has an excellent staff member in Paraguay who is not only skilled in M&E methodologies but is also a physician. He is now freed up from the National RH Policy Evaluation Policy activity to dedicate full-time to realizing all the M&E activities. In addition, he is receiving support from the PRIME II Regional M&E Specialist. The Regional M&E Specialist has: a) reviewed the quality of the baseline database, b) strengthened the M&E instruments, c) refined the monitoring database, d) developed survey instruments to measure quality at hospital-level. The Specialist has also helped the local M&E staff member define a detailed M& E workplan. Continued technical assistance from LAC/RO is required to help the local M&E specialist realize his scope of work. 3.3 Management Areas to Strengthen under Management: 1 For a list of the members from the Grupo Conductor interviewed, please refer to Appendix A. PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 16 · There are limited technical oversight processes and tools in place. Observed behavior as well as staff interviews reveal that technical supervision is informal and responds to the task at hand. There is no discernable technical oversight process in place other than approval of monthly calendar of activities and occasional review of documents. The monthly workplan process involves one-on-one staff meetings with the Technical Coordinator; once the plans are finalized the Project Director approves them. But this process is not consistently followed. There is not formal process in place to review key documents such M&E reports, technical reports, curriculum designs, training materials, etc. · PRIME/P has good workplans that clearly link PRIME/P strategies and supporting activities to USAID/P expected result. Moreover, the activities described in the workplans are coherent and consistent, and their logic will likely achieve USAID/P desired results. But the Program Director or Technical Coordinator do not use the workplan as a technical management tool. They do not conduct regular meetings with the entire staff or with individual staff to annually reassess the program design to determine the program is on track to produce the desired results, to monitor progress of activities, and to trouble-shoot implementation or technical issues encountered in the field. · Technical information is not uniformly shared with all staff. At one time, there were semi-regular staff meetings to discuss technical strategies, activities and plan, but these meeting no longer happen Communication and exchange of technical ideas of made more difficult by the staff’s travel schedule to the field: on average they in the field 3 out 5 days a week. As a result, best practices and cross￾fertilization of ideas between regions and strategy areas (Access and Quality) rarely occurs. · There are no clear linkages between budget and program activities. The Evaluation Team reviewed the overall budget for the PRIME II activities in Paraguay. Staff interviews revealed that the Paraguay offices does not develop budgets by activity nor do they technical staff work with budgets to monitor the cost of their activities, so there is no way for the technical staff to determine if they are producing the products and results proposed in the workplan according to plan and budget. Also, it is not clear that the Program Director uses the budget to inform program decisions. 4.0 Progress in Meetings Results 4.1 PRIME Program linkages to USAID/P Results The overall program design proposed in 2001 is still valid. Both the site visits and recent six-month Monitoring Report demonstrate that the mix of technical interventions – or PRIME II toolkit - is producing results at the facility level. The Mid-Term Evaluation demonstrates the PRIME II will need to make some adjustments in the implementation approach and re-activate certain technical activities if the program is going to produce maximum results at the facility level (Please refer to Section 5). These modifications, however, are due primarily to change in operating environment as the PRIME/P program integrated into the Alianza initiative and roles and responsibilities have shifted among the partners. But the program design and technical activities are fundamentally sound. Most of the program documents clearly link the PRIME II activities to the USAID/P expected results. This is most evident in the Year 2 Workplan and the recent Monitoring Report (Ann – Need to confirm since I have not seen it). The only document that is not consistent in linking with the USAID/P results and workplan milestones is the Quarterly reports. The original format followed (July - Sept 2001, and Oct – Dec 2001, Jan – March 2002) was an excellent reporting format that described activities in relations to each strategy objective and milestones and allowed for comments regarding the status of the activity. In addition, there was a one-page summary that provided context for the activities, helping the reader understand the challenges in achieving the milestones, objectives and results. The last Quarterly Report PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 17 reviewed did not follow this format and instead, presented just a list of activities without any analysis related to milestones and results. Table Four: USAID/P SO and IR Related to PRIME II SO/Result Indicator Indicator Description Target by Year Means Verification SO: Use of voluntary RH services increased CYP # of couples protected from pregnancy by FP services during 1 year period, based on volume of all contraceptives sold or distributed 2002 - 200,000 2003 - 220,000 2004 - 240,000 2005 - 260,000 The MOHs, local NGOs, Pharmaceutical Association’s annual reports IR 1.2: Access to quality RH services expanded Delivery points providing quality RH care Quality RH care is based on the following basic elements: · # of modern FP methods, · counseling given to clients, · technical competence, · interpersonal relations · explanation informed consent 2002 – 10 2003 – 15 2004 – 20 2005 – 25 MOH and Quality Project reports 4.2 Measurable progress toward Project Objectives and USAID/P results As Table Four indicated, PRIME/P is expected to deliver 10 quality sites by 2002. The site visits demonstrated effective change in provider practice and behavior at all 11 facilities. The 1st Monitoring Report also confirms that Quality (defined below) has been achieved at all facilities). Moreover, PRIME/P is on track to deliver 5 more sites by this year with the proposed modifications to the technical interventions and implementation approach. At this rhythm of implementation, PRIME/P will exceed the expected result of quality at 20 sites by May, 2004 by 3 sites. PRIME/P developed an indicator to measure quality based on the five priority areas defined by USAID/P: Counseling, Informed Consent, Technical Competence, CPI, and availability of methods. To assess quality at the facility, PRIME/P implemented a survey of a 130 questions with multiple indicators for each category. If the definition of quality is fully met, then the site receives a “yes”. If the site meets the requirements partially or not at all, it receives a “no”. Each sites receives a total number, comprised of a composite number of the scores for each of the categories. The monitoring instrument, to be carried out every six months, is a shorter version of the baseline and only looks at performance of all RH providers. It is 22 questions, 4 which relate to counseling, 6 which relate to informed consent, 4 which relate to technical competence, and 6 relate to CPI and 2 to availability of methods. Again, the monitoring reporting is calculated the same way as the baseline report, leaving no room for graduations. The monitoring visit from December demonstrated the changes at the site level (of the sites we have been working with) and the summarized increase in overall quality for these sites. As the Graph #1 demonstrates, the average score for the 11 sites was 32.2 out of 100 at the time of the baseline. To understand the score, 0 represents the absolutely no quality while 100 represents a perfect score and excellent quality. USAID/P and PRIME/P agreed that a score of 80 would be the target score to measure quality. Six months later, these same clinics realized a two-fold increase in their score, from 32.2 to 73. (Insert Graph Comportamiento del desempeno de los proveedores de los servicios de SR, Paraguay 2002) PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 18 There are differences in quality levels by region, as illustrated by Graph 2. Itapua and Misiones increased to the same level, from 35.8 to 69 and 28.3 to 69, respectively. The sites in Central realized an even more dramatic increase, rising from 31.1 to 80. The difference in quality levels between Itapua and Misiones compared to Central can be attributed to different learning approaches used in Central. ( Insert graph Comportamiento del desempeno de los proveedores pro Region de Servicios de SR) Table 3 demonstrates the difference in provider performance according to training methodology. The Regional Coordinator in Central introduced and relied almost exclusively on OTJ as the learning methodology whereas the other Regions used more traditional methods such as workshops. Providers who received OTJ are out performing their counterparts, receiving a score of 80 compared to 69. This has major implications for PRIME/P training approaches with the new sites and follow-up visits with current 11 sites. (Insert graph Comportamiento del desempeno de los proveedores segun modalidad de capacitacion) 4.3 Measurement Tools and Methodologies · PRIME II has adequate systems and process in place to monitor and evaluate the program’s progress in achieving results. The system is designed and the processes are in place, but the local evaluation person has not been able to dedicate sufficient time to the full monitoring effort. These recommendations are in his recent report. The overall structure begins with a baseline for provider performance, user satisfaction and site adequacy. The monitoring should be done at the provider and user level at a six-month interval. The plan is that the results are then shared with the project implementers to analyze and include in corrective actions for the next six-month period as well as in the design of the next workplan. 5.0 Recommendations In order to meet USAID/P’ result of 23 quality RH sites and expectation of developing technical models that can be replicated at other sites/regions, the Evaluation Team recommends the following: 5.1 Recommendations to Strengthen Implementation · Articulate and reinforce strategic framework to inform and guide program design and activities. The Evaluation Team has determined that the original design and technical approach for PRIME/P Quality Program in Paraguay are still valid. During the last 9 months, the PRIME/P staff have been very busy establishing relationships with new sites, consolidating the technical approach, launching activities, and building a team. Now is an excellent opportunity for the PRIME/P team to step back and revisit the conceptual framework given their recent field experience and upcoming launch in 13 new sites. Also, this is an opportunity to further refine some of the strategies and technical areas based on the suggestions emerging from the Mid-term Evaluation and first monitoring report. This exercise will help reinforce a common vision of what is a quality RH program among the staff and will serve multiple purposes. First, it will help guide and inform the staff’s technical activities going forward as they launch the new sites and continue following-up with existing sites. Second, the common vision will serve as the starting point for the write up and description of the Quality Model requested by USAID/P. Finally, a clear understanding of the quality RH program will facilitate greater coordination and integration of PRIME activities with Alianza’s activities. · Develop, document and disseminate the Quality Model and implementation approaches so that other donors and agencies can replicate in other regions. As mentioned before, all the pieces for defining and documenting the Quality Model and the methodological approaches for each of the technical interventions are located in different documents and require synthesizing them into a framework and PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 19 single or multiple consistent documents so the readers can understand how each of the different technical areas fit into the larger picture. Appendix C provides an outline for this document. A useful exercise will be documenting both the implementation approach and defining the activities for each of the technical interventions. This exercise should focus on describing in detail how staff carry out the activities, almost in “cook book” style, thereby ensuring a consistent approach in implementing the interventions across regions and sites. · Develop tailor-made technical assistance plans for each site that roll up into workplans by region. Originally the PRIME/P approach assumed staff would implement all interventions in all sites. Yet no two sites are alike: the providers have different skills and expertise in FP/RH; each facility faces different challenges in overcoming access and providing quality services. Moreover, launching activities in a hospital setting is more complex and challenging compared to a health. Therefore the Evaluation Team recommends that PRIME/P develop six month TA plans that address the individual needs of the sites. PRIME/P can access a wealth of information to assist in the design: a) the results each of the facility COPE exercises can help identify TA needs in the area of organization of services that present access constraints; b) results from the CIRD community assessments can also help identify access constraints from the community perspective and can help design Provider/Community Outreach activities; c) site assessments determine what core technical areas the providers require training in. In the case of the hospitals, the counterparts can participate in the workplan design, increasing the counterpart ownership in the process and activities. · Design and integrate strategies and activities that will ensure sustained quality model in other regions. To ensure the Quality Model is continued after the life of the Program, PRIME/P needs to find ways to transfer the model and capacity locally. Documenting the model and methodologies is one strategy. But another critical one is to change the implementation approach from an operational mode to a more strategic institutional building one. PRIME/P can use one of its proposed technical interventions - facilitative supervision – as a means by which to build local capacity and transfer the Quality Model to Paraguayan health system. The Evaluation Team recommends that PRIME/P explore further with the MOH working with the Regional MOH Supervision staff to strengthen their skills and capacity to carry out supervision, continuing education, and technical assistance in select areas. The Evaluation Team recognizes this will not be an easy task given the weak MOH supervision infrastructure. 5.2 Recommendations to Strengthen the Technical Program Access Activities · Adjust workplan activities and budget to reflect the fact that PRIME/P is doing less work in access. With PRIME/P ‘s integration into the Alianza initiative, PRIME/P is reducing its scope in this area and focusing Access related activities to provide IEC materials for providers so they better interact with and the community. PRIME/P and CIRD have taken steps to collaborate “sharing” a staff person who will bridge CIRD’s community IEC and also develop PRIME/P IEC activities that will help get the providers out into the community. All program documentation – budget, workplan, and monitoring reports – should reflect this decreased level of effort in this area. Also, the M&E activities and reports need to acknowledge that PRIME/P access objective depends on Alianza activities since they have assumed most of the responsibilities for all the activities related to demand generation and the community. PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 20 · The Evaluation Team also recommends moving activities such as Networks (referrals/counter￾referrals), COPE/Gestion de Calidad and facilitative supervision – to Quality strategy. Under the quality objective, these activities will be better able to support provider quality of care. · PRIME/P has ensured the facilities where they work have sufficient IEC materials, as a result, there is no longer a need to produce additional IEC materials for providers. PRIME’s comparative advantage in future IEC activities will be providing technical assistance to the Alianza partners on the medical content of the IEC materials to be used with the community to ensure accuracy as well as consistency. PRIME/P should focus their future IEC activities in assisting providers to better relate to the community, providing health education to the community and promoting their quality services among the community. This focus provider/community focus will serve as the link between the CEPEP and CIRD activities. Quality Activities · Strengthen PRIME/P training approaches. The recent monitoring report and site visits clearly demonstrate that innovative approaches to transferring knowledge and skills such as OTJ produce dramatic changes in provider practices and behavior. The Evaluation Team recommends that PRIME/P adopt OTJ and other proven learning approaches developed by PRIME II internationally to promote continued learning among the providers. PRIME/CH can assist PRIME/P through: a) training in different learning methodologies and transfer of learning strategies, and b) updating the training modules, curriculum and materials to reflect the new approaches. In addition, all the training activities should be linked to job descriptions and performance measures (quality indicators in this case). Finally, PRIME/P staff should develop a follow-up protocol to ensure that all sites receive regular and consistent follow-up visits that maximize the learning opportunity. · PRIME/P staff need to revisit the recommendations, strategies and workplans developed by the PRIME consultants and staff for COPE and Facilitative Supervision to re-initiative these activities. A lot of good work has been conducted in assessing the Paraguayan environment, designing a strategy, and developing and negotiating workplan both for COPE and Facilitative Supervision activities. Given the groundwork completed, it will require little time and effort to update these assessments and workplans and to get these activities started in the eleven 2002 sites and new 2003 sites. · As the PRIME/P staff move to re-vitalize Gestion de Calidad activities, they can expand the application of COPE to address some of the program needs identified in the mid-term evaluation. For example, the self-assessments conducted at the site can be an important input to tailor design PRIME/P technical assistance at the facility level. COPE is an effective tool to identify access barriers and help PRIME/P and facility staff overcome these constraints through the Action Plan. Under facilitation supervision, COPE can become one of the many tools and methodologies the supervisors use to encourage and motivate providers to improve performance. · PRIME/P staff will also need to begin activities under Network. However, this area will require more preparatory work and level of effort. CIRD and PRIME/P have developed a preliminary strategy to for this activity but it is inadequate. The Redes activity would greatly benefit from outside technical assistance from PRIME/CH or LAC staff that have extensive experience in establishing referral and counter-referral systems. The Evaluation Team recommends bringing the Project Director from El Salvador or Nicaragua to work with the PRIME/P team to design this activity. · PRIME/P has done an excellent job of distributing the norms and standards to all sites and fostering a culture of performing to norms. Through OTJ training, PRIME/P need to create innovative and creative ways to encourage providers to use the norms. PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 21 · Improvement in provider counseling skills is one of the success stories of PRIME/P and now the staff need to document their methodologies and lessons learned so they can continue building on this success as they launch similar activities in the 13 new sites. In addition, the staff will need to continue monitoring providers in the original sites to provide refresher training and updates in contraceptive methods and informed consent. Also, staff will need to introduce written informed consent forms in both Guarani and Spanish. · PRIME/P staff can work with Regional MOH staff and Social Pharmacies to identify strategies to ensure consistent supplies of necessary items such as contraceptives, gloves, towels, etc. Policy Activities · PRIME/P need to actively disseminate the Evaluation of National RH Plan. There are some activities programmed but PRIME/P can develop a systematic dissemination plan to ensure the Evaluation Report is widely disseminated to all key stakeholders both inside and outside the health sector at the national and regional levels. · The Evaluation Team recommends eliminating technical assistance to MOH to develop the next National RH Plan. The Evaluation revealed there is limited political will to develop and implement a National RH Plan and that this activity is driven by a few, strategic individuals who consider it important. Although having a comprehensive National RH Plan is important to provide the legal and policy support for FP/RH services, it is not essential. Indeed, many FP/RH activities are happening at the regional level given the decentralization initiative, not requiring a centralized national policy framework perse. The resources allocated to carry out this activity would be better served in mobilizing some of the delayed activities in the quality sites or in documenting the Quality Model. M&E · M&E activities are well defined and in place. To remain on track, the local M&E specialist will require continued support and technical supervision from the LAC/RO M&E Expert. · Currently, M&E activities are separate from the technical activities. The M&E specialist reports directly to the Program Director, contributing to the lack on integration and synergies between technical strategies and M&E. The M&E specialist collaborative work-style has helped bring M&E and the technical areas closer but the two areas can be further integrated through restructuring. First, the M&E Specialist should report to the Technical Leader like all the other technical staff. Second, the M&E Specialist should be involved in all strategy design, workplan development, and program management meetings so he can integrate the M&E perspective as well as better understand the fieldwork. Third, the M&E Specialist should participate in the technical coordination meetings (see below) as part of the technical team. · Add to the M&E Specialist scope of work the role of disseminating lessons learned and best practices from the Quality Model. The M&E Specialist is well positioned to take on this responsibility since he will be monitoring the overall progress of the workplan as well as measuring the program’s progress in achieving USAID/P’s desired results. Through this frequent and periodic analysis, he and other technical staff, will be able to identify best practices. The Evaluation Team recommends developing a series of “technical pages” similar to PRIME Pages on the Paraguay successes. The first technical page can be on the success of the O-T-J approach in Central. In addition, the M&E Specialist should be responsible for disseminating the technical pages as well as the Quality Technical approach and PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 22 methodologies. The Evaluation Team suggests that the PRIME Communication Department assist him in the production and dissemination of all these technical products. 5.3 Recommendations to Strengthen Management of the Technical Program · Establish and maintain mechanisms to improve communication and exchange of information with all relevant stakeholders. Clear and transparent communication is critical to fostering a positive and collaborative working environment. The Program Director needs to dedicate more time to ensuring communication and exchange of information takes place. First, the PRIME/P should fulfill their commitment to meet with MOH on weekly basis and share workplans, training agenda and other requested information. Second, establish a regular meeting schedule – as discussed below under technical management – with LAC/RO to discuss a set agenda of technical and management topics. Third, establish a similar structured meeting with the USAID/P but on a more frequent basis. The Evaluation Team recognizes that PRIME/P and USAID/P speak almost daily but a more structured meeting to discuss would ensure communications cover key management and technical issues, avoiding getting mired in the day-to-day implementation. Fourth, re-open and maintain open communication between LAC, PRIME/P and USAID/P on the same management and technical topics so all three parties are sharing the same information. Fifth, continue the coordination meetings with the Alianza partners at the leadership level but also encourage more frequent, and operational meetings at the technical levels. · Establish structure, processes and tools to oversee and manage technical activities. The following is a list of specific recommendations to put a structure and process in place. - Structure needs to start at the top whereby the LAC/RO is in constant consultation with the Program Director about overall technical direction, technical skills and staff required, and timeframe and budget to complete them. The LAC/RO and the Program Director should schedule a monthly call where they agree upon an agenda that discuss the progress in achieving expected results. Topics for this call include: a) relationship with counterpart, b) relationship with USAID, c) technical activities against workplan, d) budget expenditures and burn rate, d) staffing requirements and other staffing related issues, and e) subcontracting issues. - Program Director and/or Technical Leader need to re-institute the bi-weekly technical meetings so the technical staff can discuss: activity design and strategies, methodologies, workplans, implementation issues, design and strategies. Also, staff can start an informal lunch series where they can present different topics, reports, findings, etc. as a means to not only share technical information but also develop staff’s technical skills. - The Technical Leader should conduct quarterly review meetings with the entire technical team to discuss how the program is doing against plan and budget as well as how is the program tracking to the desired results. The M&E specialist can assist the Technical Leader by providing the information from the monitoring report. In addition, the Technical Leaders should sit down with each of the technical staff and the budget manager to review individual workplan and budget; to review activities to date; discuss any implementation issues (both technical and operational), and; analyze the budget. - Establish a set of uniform program documents that are linked to Program objectives and USAID/P results, and a process to review them that will help the technical staff manage their technical activities. The documents include: Activity design and methodology; workplan, timeline and budget by activity; quarterly report; monitoring report, and; annual M&E report. - Identify technical staff in PRIME/CH or LAC to partner with staff in key technical areas. Another strategy to strengthen technical management and grow staff’s skills is to create stronger links between technical resources located both in CH and LAC. The PRIME/P staff identified the following areas they would like refresher training and/or they need assistance. PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 23 Table Five: Learning Partners Technical Area Staff Partner Proposed Activity Facilitative supervision Elodia Vysokolan Felicita Aranda Consuelo Juarez · Review and update assessment · Design strategy to launch in all sites · Train PRIME/P in methodology · Launch FS activities Responsive Learning and Training Jorge Sosa Felix Brizuela Elodia Vysokolan Felicita Aranda TBD · Provide training in different RTL · Review current workplan and training activities to integrate RTL and identify opportunities to implement innovative training · Link training to job performance and quality · Provide OTJ with PRIME/P on how to develop learning strategies for all trainings Redes Felix Brizuela Douglas Jarquin · Assist PRIME/P staff to design Redes model and strategy · Assist PRIME/P draft workplan and timeframe to launch · Conduct initial meetings with counterpart · Exchange info with staff on Redes activities in other LAC countries M&E Jorge Galeano Gregorio Soriano · Monitor M&E activities · Provide technical direction to M&E activities and review products · Develop activity-based budgets and use for program decision-making. The prior recommendations suggested developing workplans for each facility, then rolling up the facility workplan to a regional level workplan for planning and management purposes. The compliment to this activity is developing budgets for each activity so the Regional Coordinator can track progress in implementing activities, but also the costs. The unit would be all the technical activities required to support a facility. The budget exercise should include not only local expense, but also international costs. PRIME/LAC can assist the local F/A to develop this budget to support the technical staff’s six-month workplan. · Organize and mobilize staff more efficiently. The PRIME/P has increased the number of technical staff working for the program from 2 full-time to 5 ¾ full-time equivalents. The current staff level can barely keep up with the existing workload and they have yet to fully implement several key technical activities already programmed for the original 11 sites (facilitative supervision, COPE, Redes) and launched the 12 new sites. To address this challenge, the Evaluation Team suggests the following: - Evaluate staff workload and assignment to redistribute sites and activities according to skills/experience/talent: Currently, there is a Regional Coordinator for each Region but each Region has a different number and profile of sites. For example, Misiones has 3 health posts and 1 Hospital compared to Asuncion with 2 Hospital, 1 clinic and 1 Health Post. Although they both have the same # of sites, Asuncion mix of sites is more complicated, requiring a more level of PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 24 effort. Moreover, the Hospital staff are dominated by Physicians and will require a Physician spear head that activity. Therefore, the Team suggests the Program Director redistribute the sites by level of effort and task, not by geography. This redistribution should be done now before as the staff develop the workplan to launch activities in the 12 new sites. - Redistribute the roles and responsibilities between the current Technical Coordinator and Program Officer. The Team proposes redesigning both their job descriptions to benefit from each of their respective talents as well as acknowledge actual practice. The Technical Coordinator’s scope will focus on technical leadership, management, oversight as well as implementing some of the technical activities like Redes and working with Hospitals in Asuncion. The current responsibility of developing monthly calendar of activities and coordinating workplans will be transferred to the Program Officer, who will now become the Program Coordinator. The Technical Coordinator will still play a role in reviewing this workplans (now quarterly) to ensure activities proposed follow the workplan but the Program Coordinator will do all the logistics, coordination and planning. - Redo all the job descriptions according to the geographic reassignments and new implementation focus (See Appendix E for illustrative SOWs). - Consult the budget to determine if program can increase LOE and/or hire one more technical staff person to help - Determine if administrative staff workload can be redistributed to better support workshop and site visit logistics. 5.4Recommendations to Strengthen Alianza Coordination The Evaluation Team had the opportunity to interview the Alianza partners and USAID/P on the status of the Alianza partnership. Although this was not part of the terms of reference of the PRIME/P mid-term evaluation, some important recommendations emerged from the interviews related to the Alianza partnership that impact PRIME/P. They are: · USAID/P maintains a consistent position on Alianza goals and objectives with all partners. During the initial formation of the Alianza, the goals and objectives were unclear and therefore, created confusion and some turmoil between the partners. Last Spring, USAID/P clarified its vision for Alianza and the partners acknowledge that they now have a better understanding of what their respective roles and responsibilities are under Alianza partnership. It is important that USAID/P present a uniform front on their vision for the project and be consistent in this message with all partners. · USAID/P communicates consistent expectations through a variety of mechanisms, venues and channels with the Alianza partners. Although USAID/P may think they have been perfectly clear on what is there vision and what are their expectations, it still merits repeating them as often as possible and with all the partners. It is only human nature for organizations to “hear” what they what to hear, particularly if it involves difficult change. So if UAID/P continuously repeats it message in different settings, eventually the message will sink it. · Demonstrates leadership in Alianza coordination by establishing mechanisms for and creating opportunities for partners to further integrate vision, scopes and activities. It is difficult to get three organizations to naturally come together and expect them coordinate activities. Therefore it is incumbent upon USAID/P to gently “force” situations whereby the organizations will need to work together. A first activity may involve a one-day session facilitated by an outsider to help the senior leadership and technical advisers of USAID/P, PRIME/P, CIRD and CEPEP develop a common vision that focuses on the Alianza initiative and not on their respective organizational agendas and PRIME II Quality Program Mid-term Assessment Final Report 9/9/2004 25 activities. After enough focusing events like the proposed one and other routine mechanisms like monthly coordinating meetings, the partners will eventually start working together. · Alianza partners coordinate between organizations and assure that program and activities integrate at the site level. The interviews revealed that the Leaders of the different organizations are meeting regularly and coordinating at the strategic level. What is not happening are: 1) the Leaders of these organizations are not sharing the information and direction received at these coordinating meetings with their own staff, and 2) the technical staff from each of these organizations are not coordinating programs and activities yet. Therefore coordination efforts now need to focus on filtering down into each of the organizations so that staff members understand the Alianza vision, goals and objectives and the role their respective organization contributes to this effort. In addition, more horizontal coordination needs to occur among different technical staff. To facilitate this, the partners can form technical teams around regions and/or sites to discuss all the activities happening in that area as well as how to coordinate technic al resources and time activities. Immediate Next Steps: Sara Espada, Latin America and Caribbean Program Manager, PRIME II, will travel to Paraguay on March 17, 2003 to work with the Quality Team on the findings of this report. As top priority she will work with the staff on a revised workplan, the development of an activity based budget and the reorganization of staff to best meet the needs of the project.