MEASURE Evaluation 
ME Final.doc i 12/30/2005 
EVALUATING SHORT-TERM TRAINING IN HEALTH 
PROGRAM EVALUATION 
AN ASSESSMENT OF CAPACITY BUILDING AND UTILIZATION IN 
THREE REGIONAL M&E PROJECTS
by 
Jack Reynolds, Ph.D. 
Submitted to: 
USAID/Washington 
December 2005 
MEASURE Evaluation 
ME Final.doc ii 12/30/2005 
ACKNOWLEDGEMENTS
This assessment could not have been conducted without the cooperation of the workshop 
participants, instructors and organizers who contributed their time, energy and ideas. It also 
could not have been completed without the collaboration of the other members of the Evaluation 
Team: Randolph Augustin, Laverne Webb, Melanie Kindsfather-López, Charles Katende, Lisa 
Maniscalco and Virginia Lamb. Special thanks go to MEASURE Evaluation staff, in particular 
Nina Frankel, Hemali Kulatilaka, Anne LaFond (M/E-JSI), and Gustavo Angeles for their help 
in making arrangements and facilitating access to key informants, documents and data. 
MEASURE Evaluation 
ME Final.doc iii 12/30/2005 
EXECUTIVE SUMMARY
The Partnerships. This is one of four reports on the evaluation of workshops on monitoring 
and evaluation (M&E) supported by MEASURE Evaluation (M/E).1
 The other three are case 
studies of the partnerships between M/E and three regional training institutions: The African 
Center for Advanced Management Studies (CESAG) in Senegal; the Institute for Population and 
Social Research (IPSR) in Thailand; and the National Institute for Public Health (INSP) in 
Mexico. M/E signed Memoranda of Understanding (MOU) with each partner to design and 
implement annual, three-week regional workshops on the monitoring and evaluation (M&E) of 
population, health and nutrition (PHN) programs. These workshops are intended to strengthen 
the M&E capacity of professionals working in PHN programs in Francophone Africa, Asia and 
the Near East (ANE), and Latin America and the Caribbean (LAC). 
The Objectives. The evaluation has two specific objectives. The first is to assess progress in 
developing the capacity of the partners to carry out these workshops independently, without the 
technical or financial support of M/E. The second is to determine the degree to which workshop 
participants have utilized new knowledge and skills gained in the workshops. 
The Methodology. To do this, the Evaluation Team reviewed documents and data on the five 
workshops carried out by the partners since early 2004,2
F visited M/E offices in Chapel Hill, and 
visited partner offices in Dakar, Bangkok and Cuernavaca. The Team also carried out two 
surveys. The first was key informant interviews of a limited number of partner staff (directors, 
coordinator and faculty) that were particularly knowledgeable about the workshop and the 
institutes’ training programs. The second was an email and telephone interview survey with as 
many participants as possible that had attended the workshops in 2004 and 2005. The samples 
were small, but the results seem to be consistent as well as informative. 
The Interventions. M/E and its partners have 
followed a comprehensive strategy to build 
capacity and it seems to be working well (see 
box). It helps that the partners already had 
established training institutions, solid reputations 
in their regions, and links with the Ministries of 
Health, strong leadership and faculty, and 
commitments to building their M&E capacity. 
The workshops. Five workshops had been 
carried out between July 2004 and November 
2005. Demand well exceeds supply and all three 
institutions plan to hold workshops in 2006. 
CESAG and IPSR held “standard” or basic M&E 
workshops. INSP conducted “impact” evaluation 
workshops. M/E staff collaborated with their 
partners in the design and implementation of all 
of the workshops. The agendas were similarly 
structured for all of the workshops (see Annex A, 
Table 23-24). The first week was devoted to “fundam
Capacity Interventions 
• Assessment of M&E capacity and 
support to strengthen it 
• Engaging leadership 
• Training of key faculty 
• Technical update workshops on M&E 
• Engaging key faculty in TA and 
evaluation research 
• Training of trainer workshops 
• Fellowships for selected participants 
• Development/adaptation of curricula 
and training materials 
• Provision of instructors, materials and 
logistic support 
• Administrative and financial planning 
and procedures 
• Work on sustainability 
entals” and most of the second and third 
1
 MEASURE Evaluation is supported by USAID through a Cooperative Agreement with the Carolina Population 
Center in Chapel Hill, North Carolina. 
2
 A sixth workshop was carried out by IPSR in November 2005 after the Team had completed its field work. 
H
 
MEASURE Evaluation 
ME Final.doc iv 12/30/2005 
weeks to M&E for specific health programs or statistical models for measuring program impact. 
At the same time, small groups were formed to develop evaluation plans or impact designs. 
The participants. A total of 132 individuals from 38 countries were trained during this period. 
None of the institutions had trouble recruiting participants. CESAG received 49 applications its 
first year, about 30 had their own funding. IPSR received over 70 applications, 32 of which had 
secured funding. INSP received 44 applications, but only 7 with funding. USAID funding is 
quite small, usually limited to two to four fellowships per workshop. CESAG and IPSR could 
probably continue attracting enough funded applicants to carry on without USAID fellowships. 
INSP probably cannot. This is a significant issue for INSP, as it does not have funds for 
fellowships, especially for non-Mexicans. 
Participant evaluations have been very positive. On a scale of 1-10, overall satisfaction has 
been in the 9-10 range. The participants have been equally positive about most of the workshop 
sessions. Criticisms centered around presentation: too much lecturing, not enough time, need 
for more practical sessions, more group work, more real life examples of applications. M/E and 
the partners have been very responsive to this feedback and have revised the curricula 
accordingly. However, the participant evaluations do not tell us anything about utilization. We 
will discuss that later on. 
Achievements in capacity. Capacity ratings were divided into two types: institutional capacity
and substantive training capacity. Table 6 (in the body of the report) shows that INSP is well on 
its way to taking over the program. On a scale of 1-10, a score of 8 and above indicates that the 
institution has that capacity. INSP and IPSR both need to improve their technical capacity in 
M&E, but they are both doing very well on the other three components. CESAG has lower 
overall ratings but no specific weaknesses. The Team believes that the CESAG stakeholders 
rated their program too low and that CESAG is also on its way to gaining the institutional 
capacity needed to run the course independently. The text provides individual ratings for each 
of the 17 elements that make up the capacity assessment tool. 
Substantive training capacity is progressing. A score of 3.0 or better on a 4-point scale 
indicates capacity to deliver all the modules without external assistance. None of the partners 
has an overall rating that high. CESAG is the leader in this area (2.7), but it still needs to 
improve its capacity to deliver the fundamental models, in particular. IPSR (2.3) also needs to 
improve its capacity in this area, and in the program modules. INSP (2.4), on the other hand, is 
strong in both the fundamental and practicum modules but weak in its capacity to deliver the 
impact modules. The text provides individual ratings on each of the 16-25 training modules 
used in the different workshops. 
Achievements in utilization. The participants generally agreed that they learned a lot from the 
workshops but most did not have the opportunity to apply what they learned. Some did, 
however, and stated that they used some of the modules often. Surprisingly, the conceptual 
modules (the importance of evaluation, evaluation concepts, frameworks, indicators, evaluation 
plans) were the most used of the modules. The fundamentals of research were already well 
known (experimental designs, observations, qualitative methods, information systems, data 
sources). The program and impact modules were used less than the fundamental and practicum 
modules, largely because there were few opportunities to apply them. One quote from a 
respondent sums up what many concluded: 
“This [Fundamental concepts; frameworks] was one of the most important modules for me. It 
was the clearest, most precise module for my present work. I learned about indicators and 
qualitative studies. I’m now helping design a new study and this gives me the ability to do it.” 
H
MEASURE Evaluation 
ME Final.doc v 12/30/2005 
Facilitating factors and constraints. A number of external factors affect project performance. 
On the facilitating side are: the reputation and experience of the institution; demand for M&E 
training; linkages with national governmental and international agencies; and leadership. On the 
constraint side are: limited donor funding for PHN; lack of personnel with experience in M&E; 
language and culture (positive for CESAG and INSP, negative for IPSR). 
Conclusions. The M/E partnerships have been productive and both M/E and its parties would 
like to continue them. M/E and its partners share a common vision and work well together. 
Capacity is being developed, but more TA will be needed over the next two years, especially in 
finding fellowships, developing human resources, developing and implementing effective 
follow-up procedures, and building capacity in delivering the training modules. Most 
participants learned something new and valuable in the workshops, but many have not had the 
opportunity to apply this knowledge. 
Methodology. The methodology was not perfect, but it has its merits. The universe was small 
and so, therefore, were the samples, although the participant response rates for CESAG and 
INSP were respectable (49 and 58 percent). The scales used were not exactly consistent, which 
affects interpretation. The capacity assessment tools have not been validated. Despite these 
weaknesses, the approach used is an interesting, and simple way to get standardized, in-depth 
information from a variety of respondents in a small amount of time that actually produce 
quantitative data. This may be a useful model for future “qualitative” evaluations that USAID 
relies on heavily. The instruments themselves appear to be valuable, easy to administer and 
useful for both snapshots of current capacity and trends over time. 
Recommendations for MEASURE Evaluation. M/E should continue its TA to its partners but 
with greater focus on those capacities that are essential to independence. Targeted TA can speed 
up capacity and graduation. The following table identifies areas of high (A), medium (B) and 
low (C) capacity. The Team believes that the institutions will not need further TA in the 57 
capacities listed in the A category. The organizations are already strong in these areas. Where 
each institution will need more TA to build capacity is in the 35 capacities listed in the B 
category (see Table 28 in the Annex) and especially in the 17 capacities in the C category (see 
Table 29). Although this is a large number of capacity elements and training modules, M/E and 
the three training institutions are already addressing many of them. The emphasis now should 
be on targeting those needs that are critical to independence. 
H
H
Table 1: Summary of Institutional and Substantive Needs 
 Institutional Substantive Total
Rating CESAG IPSR INSP CESAG IPSR INSP No. Percent
A. High capacity 11 8 12 13 5 8 57 52 
B. Medium capacity 5 6 4 6 4 10 35 32 
C. Low capacity 1 2 1 6 7 17 16 
Total elements/modules 17 16 17 19 15 25 109 100 
M/E should develop an exit strategy for each program. Some options should be developed in 
case all of the TA needs cannot be filled over the next two years. Examples are: introduce team 
teaching, focus on the most critical modules, phase modules in as experience is gained, and drop 
those modules for which capacity cannot be developed within the next two years. Provide 
assistance to INSP, in particular, on identifying financial support for fellowships. Assist the 
partners in undertaking capacity assessments and a feasibility analysis of an expanded vision. 
Help the partners find and test effective and affordable follow-up procedures. 
MEASURE Evaluation 
ME Final.doc vi 12/30/2005 
Recommendations for the Partners. Each partner should take responsibility for developing a 
realistic “post M/E” vision. This should include national as well as regional initiatives. Given 
the findings on utilization, each partner should take another look at its curriculum to see if it 
should be revised to fit demand. A strategy needs to be developed to build the human resources 
needed to deliver each module. This should include the identification and recruitment of local 
outside experts. Each partner should undertake the comprehensive capacity assessment exercise 
that is underway in Senegal. The results should then be used, along with the feasibility analysis 
mentioned above, to identify priorities for development of institutional and substantive capacity. 
All three institutions need to conduct a financial analysis of how they will cover all workshop 
costs now supported by M/E. 
Recommendations for USAID. INSP and IPSR will continue to need financial assistance for 
fellowships, especially for regional applicants. Will USAID provide that support? All three 
institutions need to know what assistance USAID will be willing and able to provide after M/E 
ends. Does USAID plan to continue to support the expansion of the M&E course? Related to 
that is the issue of expansion to create M&E centers of excellence that would provide M&E 
services (studies, training and TA) throughout the region. It would help to know what USAID’s 
position is on this as well as its M&E priorities. 
MEASURE Evaluation 
ME Final.doc vii 12/30/2005 
TABLE OF CONTENTS 
ACKNOWLEDGEMENTS..................................................................................................................................... II
EXECUTIVE SUMMARY.....................................................................................................................................III
ABBREVIATIONS AND ACRONYMS.......................................................................................................................... IX
I. INTRODUCTION ................................................................................................................................................. 1
A. BACKGROUND .................................................................................................................................................... 1
B. SCOPE OF WORK AND METHODOLOGY ............................................................................................................... 1
II. FINDINGS: PHASE II......................................................................................................................................... 2
A. ASSESSMENT AND SELECTION OF PARTNERS ...................................................................................................... 2
B. VISION, OBJECTIVES AND STRATEGIES ............................................................................................................... 2
C. AGREEMENT AND INTERVENTIONS ..................................................................................................................... 3
D. WORKSHOPS....................................................................................................................................................... 5
E. PARTICIPANTS..................................................................................................................................................... 8
F. PARTICIPANT EVALUATION................................................................................................................................. 9
1. Satisfaction .................................................................................................................................................... 9
2. Content ........................................................................................................................................................ 10
III. ACHIEVEMENTS ........................................................................................................................................... 10
A. CAPACITY BUILDING: MOVING UP THE CONTINUUM........................................................................................ 10
B. INSTITUTIONAL CAPACITY BUILDING ............................................................................................................... 12
1. Organizational Capacity ............................................................................................................................. 12
2. Technical Capacity in M&E ........................................................................................................................ 12
3. Training Capacity........................................................................................................................................ 13
4. Organizational Motivation .......................................................................................................................... 13
C. SUBSTANTIVE CAPACITY BUILDING ................................................................................................................. 13
1. Capacity in Fundamental Modules.............................................................................................................. 14
2. Capacity in Modules for M&E in Health Programs.................................................................................... 15
2. Capacity in Modules for Impact Evaluation................................................................................................ 16
3. Capacity in Practical Application Modules................................................................................................. 16
.D. UTILIZATION OF NEW KNOWLEDGE AND SKILLS............................................................................................. 17
1. Follow up Survey Findings......................................................................................................................... 17
2. Participant Survey Findings ....................................................................................................................... 20
E. FACILITATING FACTORS AND CONSTRAINTS.................................................................................................... 24
1. Facilitating Factors..................................................................................................................................... 24
2. Constraints................................................................................................................................................... 24
IV. CONCLUSIONS ............................................................................................................................................... 25
A. THE PARTNERSHIPS .......................................................................................................................................... 25
B. THE CAPACITY BUILDING OBJECTIVE............................................................................................................... 26
C. THE UTILIZATION OBJECTIVE........................................................................................................................... 26
D. THE INTANGIBLE BENEFITS .............................................................................................................................. 27
E. THE METHODOLOGY......................................................................................................................................... 27
V. RECOMMENDATIONS ................................................................................................................................... 27
A. RECOMMENDATIONS FOR MEASURE EVALUATION........................................................................................ 27
B. RECOMMENDATIONS FOR INSP ........................................................................................................................ 28
C. RECOMMENDATIONS FOR USAID..................................................................................................................... 29
ANNEXES................................................................................................................................................................ 31
A. TABLES............................................................................................................................................................. 31
B. METHODOLOGY ISSUES .................................................................................................................................... 37
1. Documents and Secondary Data.................................................................................................................. 37
2. Institutional Capacity Assessment ............................................................................................................... 37
3. Substantive Capacity Assessment. ............................................................................................................... 38
4. Participant Utilization Assessment.............................................................................................................. 38
5. Instrument scales ......................................................................................................................................... 39
6. Lessons Learned .......................................................................................................................................... 39
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
MEASURE Evaluation 
ME Final.doc viii 12/30/2005 
C. SELECTION OF PARTNER INSTITUTIONS, BY CHARLES KENADE........................................................................ 44
1. Introduction ................................................................................................................................................. 44
2. Why partnerships with Makerere and Costa Rica universities were terminated ......................................... 47
D. STATEMENT OF WORK...................................................................................................................................... 48
E. CONTACTS ........................................................................................................................................................ 59
F. DOCUMENTS REVIEWED.................................................................................................................................... 60
TABLES 
TABLE 1: SUMMARY OF INSTITUTIONAL AND SUBSTANTIVE NEEDS ............................................................................ V
TABLE 2: INTERVENTIONS TO BUILD CAPACITY TO DELIVERY M&E WORKSHOPS ..................................................... 4
TABLE 3: WORKSHOPS CONDUCTED BY PARTNER INSTITUTES .................................................................................... 5
TABLE 4: TRAINING MODULES AND ASSIGNMENTS (M/E ASSIGNMENTS SHADED) ...................................................... 7
TABLE 5: WORKSHOP PARTICIPANTS BY COUNTRY AND REGION ................................................................................ 9
TABLE 6: OVERALL INSTITUTIONAL CAPACITY RATINGS .......................................................................................... 11
TABLE 7: OVERALL SUBSTANTIVE CAPACITY RATINGS............................................................................................. 11
TABLE 8: ORGANIZATIONAL CAPACITY RATINGS (LOWEST RATINGS SHADED)......................................................... 12
TABLE 9: TECHNICAL CAPACITY IN M&E.................................................................................................................. 12
TABLE 10: TRAINING CAPACITY RATINGS (LOW RATINGS SHADED) .......................................................................... 13
TABLE 11: ORGANIZATIONAL MOTIVATION CAPACITY ............................................................................................. 13
TABLE 12: CAPACITY IN FUNDAMENTAL MODULES (LOW RATINGS SHADED) ........................................................... 14
TABLE 13: CAPACITY IN HEALTH PROGRAM MODULES (CESAG AND IPSR) ........................................................... 15
TABLE 14: INSP CAPACITY IN IMPACT MODULES (IN RANK ORDER) ........................................................................ 16
TABLE 15: CAPACITY IN M&E PRACTICAL APPLICATIONS ........................................................................................ 16
TABLE 16: MODULES LISTED AS MOST USEFUL IN M/E FOLLOW-UP SURVEY (HIGHEST RATINGS SHADED)............. 17
TABLE 17: PARTICIPANT UTILIZATION OF TRAINING MODULES ................................................................................ 21
TABLE 18: PARTICIPANT UTILIZATION OF FUNDAMENTAL MODULES (HIGHEST RATINGS SHADED) ......................... 21
TABLE 19: PARTICIPANT UTILIZATION OF CESAG AND IPSR PROGRAM MODULES (HIGHEST RATINGS SHADED).... 22
TABLE 20: PARTICIPANT UTILIZATION OF IMPACT MODULES, 2005 .......................................................................... 23
TABLE 21: PARTICIPANT UTILIZATION OF PRACTICUM MODULES (HIGHEST RATINGS SHADED) ............................... 24
TABLE 22: SUMMARY OF INSTITUTIONAL AND SUBSTANTIVE NEEDS (PRIORITY NEEDS SHADED) ............................. 26
TABLE 23: AGENDA FOR THE CESAG 2005 PHN REGIONAL WORKSHOP ON MONITORING AND EVALUATION ........ 32
TABLE 24: AGENDA FOR THE IPSR 2004 PHN REGIONAL WORKSHOP ON MONITORING AND EVALUATION............. 33
TABLE 25: AGENDA FOR THE INSP 2005 PHN REGIONAL WORKSHOP ON IMPACT EVALUATION ............................. 34
TABLE 26: HIGH ORGANIZATIONAL CAPACITY COMPONENTS, NO TA NEEDED........................................................ 35
TABLE 27: HIGH SUBSTANTIVE CAPACITY COMPONENTS, NO TA NEEDED............................................................... 35
TABLE 28: TECHNICAL ASSISTANCE NEEDS IN INSTITUTIONAL CAPACITY BUILDING ............................................... 36
TABLE 29: TECHNICAL ASSISTANCE NEEDS IN SUBSTANTIVE CAPACITY BUILDING.................................................. 36
TABLE 30: INSTITUTIONAL CAPACITY ASSESSMENT TOOL FOR TRAINING INSTITUTIONS.......................................... 40
TABLE 31: PARTNER ASSESSMENT OF SUBSTANTIVE CAPACITY................................................................................ 42
TABLE 32: PARTICIPANT UTILIZATION OF M&E MODULES ....................................................................................... 43
TABLE 33: CRITERIA FOR SELECTION OF M/E LOCAL PARTNER INSTITUTIONS. .......................................................... 44
TABLE 34: PARTNER INSTITUTIONS ACCEPTED/REJECTED FOR M/E II, BY MAJOR REASONS M/E GAVE FOR THE 
SELECTION/REJECTION ...................................................................................................................................... 45
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H H
H
H
MEASURE Evaluation 
ME Final.doc ix 12/30/2005 
Abbreviations and Acronyms 
AIDS Acquired Immuno-deficiency syndrome 
CB Capacity building 
CESAG Centre Africain D’Etudes Superieures en Gestion 
CEPH Council on Education in Public Health 
CPC Carolina Population Center 
DK Don’t know 
HIV Human immunodeficiency virus 
HMIS Health management information system 
IPSR Institute for Population and Social Research 
INSP National Institute of Public Health 
K/S Knowledge and skills 
M&E Monitoring & evaluation 
M/E MEASURE Evaluation 
MIS Management information system 
MOPH Ministry of Public Health 
MOH Ministry of Health 
MOU Memorandum of Understanding 
NA Not applicable 
NGO Non-governmental organization 
PHN Population, Health, and Nutrition 
PMP Project monitoring plan 
SARA Support for Analysis and Research in Africa Project 
SOW Scope/statement of work 
STD Sexually-transmitted diseases 
STI Sexually-transmitted infections 
TA Technical assistance 
TOT Training of trainers 
UNC University of North Carolina 
USAID United States Agency for International Development 
WHO World Health Organization 
MEASURE Evaluation 
ME Final.doc 1 12/30/2005 
I. INTRODUCTION
This is one of four reports on the evaluation of monitoring and evaluation (M&E) workshops 
supported by MEASURE Evaluation (M/E). Besides this summary report, there are three case 
studies. These case studies provide in-depth assessments of recent M&E workshops in 
Thailand, Senegal and Mexico. The workshops were collaborative efforts between M/E and 
its sub-contracted partners. In Thailand the partner is the Institute for Population and Social 
Research (IPSR) at Mahidol University, in Bangkok. In Senegal it is the Center for Advanced 
Management Studies (CESAG), which is located in Dakar. In Mexico the partner is the 
National Institute for Public Health (INSP) in Cuernavaca. 
A. Background 
In 1997, the United States Agency for International Development (USAID) funded the M/E 
project through a Cooperative Agreement with the Carolina Population Center in Chapel Hill, 
North Carolina. The project has two phases, running from 1997-2003 and 2003-2008. Its 
principal objective is to improve the monitoring and evaluation of PHN (population, health 
and nutrition) programs. Among its six components is one that attempts to strengthen the 
capacity of training institutions to implement M&E workshops. That, and the utilization of 
knowledge and skills gained in the workshops, is the subject of this evaluation. 
B. Scope of Work and Methodology 
USAID/Washington requested support from the Synergy Project to undertake an evaluation of 
what it called “one of the more innovative but difficult ‘Results’ in Phase II of the project:” 
strengthening the capacity of its regional training to conduct workshops on M&E. Two years 
into the project, USAID/W thought it timely to assess progress to date. USAID laid out two 
objectives for the evaluation: 
1. Assess progress in M/E project objective of increasing the organizational and technical 
capacity of its regional partners to undertake the Regional and General M&E short￾term training with gradually reduced technical, institutional and financial support from 
M/E and to replicate the M&E trainings; and 
2. Evaluate the results of this training as far as the utilization of new skills and knowledge 
of all participants…who completed structured M/E training.
The evaluation was limited to the workshops and participants carried out by the three training 
institutions between January 2004 and November 2005. The methodology included a review 
of relevant documents and reports, including M/E activity and trip reports, MEASURE 
approaches to M&E, and M/E approaches to capacity building. Site visits were made to M/E 
offices in North Carolina and to the three regional training institutes in Thailand, Senegal and 
Mexico. Two surveys were conducted. The first was a survey of “partner” assessments of 
their current capacity to: 1) conduct workshops on M&E; and 2) undertake each of the 16-25 
training modules that make up the course. The second survey was of the post-training 
experience of recent workshop participants, with an emphasis on their application of new 
material presented in the workshops. The samples were small because the target populations 
were small. The CESAG samples, for example, consisted of interviews with 5 “partner” 
representatives (1 institute director, 1 workshop coordinators and 3 workshop faculty), 26 
workshop participants, and 2 USAID stakeholders. A full discussion of the methodology, the 
samples and lessons learned can be found in Annex B. 
MEASURE Evaluation 
ME Final.doc 2 12/30/2005 
II. FINDINGS: PHASE II
A. Assessment and Selection of Partners 
The objective of MEASURE Evaluation is to strengthen the ability of individuals and 
organizations in developing countries to identify, collect, analyze and use data for improving 
population, health and nutrition (PHN) programs. Its focus is primarily on organizational and 
individual capacity and performance in monitoring and evaluation, with the overall goal of 
strengthening health system performance as a whole. MEASURE Evaluation has defined 
capacity as “the ability to carry out stated objectives.” Capacity development (or capacity 
building) is defined as “a process that improves the ability of a person, group, organization or 
system to meet objectives or to perform better.”3
F
During Phase I of the project, the emphasis was on building the capacity of individuals to 
conduct M&E. In phase II that changed to focus on building the capacity of institutions to 
train people in M&E. That change caused M/E to conduct an assessment of its current and 
potential training partners. The results of that assessment were to continue with the current 
partners in Thailand and Senegal, and to replace the partner at the University of Costa Rica 
with INSP in Mexico. See Annex C for a more detailed description of this selection process. 
All three organizations are well established and respected training institutions in their 
respective regions. CESAG is a private institution that concentrates on training in 
administration. IPSR is a training and research institution that is part of Mahidol University. 
INSP is a government training and research institution that includes a School of Public Health 
and which offers over 40 workshops each summer in various health areas. 
B. Vision, Objectives and Strategies 
CESAG’s vision is to continue to offer the basic workshop in M&E for PHN and to develop 
additional capacity to expand its M&E training and research activities in the region. IPSR and 
INSP have more ambitious visions. In addition to continuing to offer the M&E course, both 
would like to become the premier M&E training and research centers in their regions. 
M/E’s objectives are more modest, namely to build the capacity of each partner to be able to 
continue to offer the M&E workshop without further technical or financial assistance from 
MEASURE Evaluation after the current M/E project ends. 
The objectives for Phase II relevant to this evaluation are based on MEASURE Result 2:4
F
Result 2: Increased in-country individual and organizational technical capacity and 
resources for the identification of data needs and the collection, analysis and 
communication of appropriate information to meet those needs. 
Indicator 2.1: Number/percent of M/Eval-II training participants who are engaged in 
M&E activities after training 
Indicator 2.2: Increased organizational capacity to undertake activity as measured on 
 
3
 See Anne LaFond and Lisanne Brown, A Guide to Monitoring and Evaluation of Capacity-Building 
Interventions in the Health Sector in Developing Countries, MEASURE Evaluation Manual Series, No. 7, March 
2003, p. 7. 
4
 Draft Results Reporting Guidance, MEASURE Evaluation Phase II, Carolina Population Center, University of 
North Carolina at Chapel Hill, October 3, 2005. 
MEASURE Evaluation 
ME Final.doc 3 12/30/2005 
a continuum: 
This continuum has four stages: 
1) able to implement the course but with significant5
F technical assistance (TA); 
2) able to implement the course with limited6
F TA; 
3) able to implement the course with no TA; and 
4) able to help others to implement the course. 
Sub-Result 2a: Structured training courses and events provided and supported 
Indicator 2a.1: Number of M/Eval-II training events supported by M/Eval-II. 
Indicator 2a.2: Number of training participants who complete M/Eval-II training 
events. 
The objectives specific to this evaluation established between MEASURE Evaluation and the 
three partners are to cooperatively design, implement, monitor and evaluate a series of annual 
regional workshops. These workshops are intended to strengthen the monitoring and 
evaluation capacity of professionals working in population, health and nutrition programs in 
each region. These partners have committed themselves to ensuring the sustainability of the 
regional workshops by project’s end. 
Strategies include a package of interventions to build each institution’s capacity to design and 
deliver the M&E workshops. These include: capacity assessment, engaging leadership, 
training of key faculty, development/adaptation of curricula and training materials, expert 
course instructors, materials and logistics support, training of trainer workshops, TA and 
research opportunities for faculty, participant fellowships, administrative and financial 
planning and procedures, technical updates and M&E resource materials, and continuing work 
on sustainability – technical, logistical and financial. 
C. Agreement and Interventions 
M/E worked out individual Memoranda of Understanding (MOU) with each partner. These 
agreements specified what the partner would do and what M/E would do. The MOUs were 
tailored to each institution’s needs. However, there were standard provisions, as well. For 
example, the partners all agreed to take responsibility for administrative, logistical and 
financial management of the workshops. They all agreed to appoint a Workshop Coordinator, 
who would be the key contact with M/E on all administrative and technical matters in 
implementation. They also agreed to provide staff as faculty or co-facilitators. MEASURE 
Evaluation agreed to provide training for faculty, short-term visiting instructors to complement 
local instructors in course delivery, teaching materials, and information dissemination and 
publicity. The MOUs reflect true partnerships where both parties contribute to a mutual 
objective. 
MEASURE Evaluation has established a core set of interventions designed to build capacity 
with partner organizations. The standard interventions are listed in the first column. Examples 
of key activities are shown in the second column. For more detail on each partner’s 
 
5
 “Significant” assistance means that the partner would not be able to carry out the module without TA from M/E. 
6
 “Limited” assistance means that the partner could carry out the module, but it would be better to have M/E on 
hand in case questions or issues arise that the faculty may not be able to answer. 
MEASURE Evaluation 
ME Final.doc 4 12/30/2005 
interventions, please see the individual case studies. 
Table 2: Interventions to Build Capacity to Delivery M&E Workshops 
Intervention Activities Undertaken (General Examples) 
Assessment of M&E 
capacity and support to 
strengthen it 
• Capacity Self-Assessment tool developed 
• Capacity Assessment workshops 
• Capacity priorities set. 
Engaging leadership • Capacity Self-Assessment discussed with institute 
leaders. 
•
•
MOU updated 
Meetings with leadership on vision, roles, priorities 
Training of key faculty (in 
US or in-country) 
•
•
Institute faculty attend 2004 Gates/MEASURE 
workshop in Baltimore. 
Faculty observe workshops to learn presentation 
skills 
• See TOT workshops 
Technical update 
workshops on M&E 
• M/E workshop on Advanced Methods for 
Evaluation of Impact of Health and Social Programs 
Engaging key faculty in 
TA and evaluation research 
•
•
•
Institute sends visiting scholar to UNC for year 
Faculty participate in preparation of research 
proposals 
Institute develops program to involve faculty in 
M&E 
• M/E hires instructor as co-investigator of research 
project 
Training of trainer 
workshops 
•
•
M/E conducts TOT 2005 
Technical TOT planned for March 2006 
Fellowships for selected 
participants 
•
•
•
CESAG does not require fellowships 
IPSR awards 2 for 2004 course, five for 2005 course 
INSP awards four full scholarships, 2 partial; 2 
masters 
Development/adaptation of 
curricula and training 
materials 
•
•
Vision workshops (July 2004) includes plans to 
revise curricula and materials 
Curricula and materials revised in TOT (see above). 
Provision of instructors, 
materials and logistic 
support 
•
•
•
•
•
•
•
•
•
•
CESAG: In 2004 M/E taught eight sessions, co￾taught two. 
CESAG: In 2005 taught all but 5 of the modules 
INSP: In 2004 M/E taught 15, co-taught 5 
INSP: In 2005 M/E taught 15; INSP 11 
IPSR: In 2004 M/E taught 8; IPSR 5 
IPSR: In 2005 M/E taught 4 fundamentals, co￾taught 2 
M/E provides books and publications to partners 
M/E provides STATA software to INSP participants 
Partners handle most workshop management. 
M/E continues to support announcements and 
publicity. 
Administrative and 
financial planning and 
•
•
Applications received, many with own funding 
Training provided in USAID financial management 
MEASURE Evaluation 
ME Final.doc 5 12/30/2005 
procedures • M&E workshop integrated into INSP summer series 
• M/E assists in seeking funding from USAID, CDC, 
others 
Work on sustainability 
(technical, logistical, 
financial) 
• Break even estimates 
• Identification and training of core group of trainers 
• Faculty beginning to get involved in M&E projects 
• Listserv set up for participants 
• Participant data base developed 
D. Workshops 
All three of the partners have held three-week workshops in 2004 and 2005. All plan to hold 
another in 2006. The CESAG and IPSR workshops were the “standard” or “basic” 
introductory workshops. INSP’s workshops were different. They focused on evaluating the 
impact of PHN programs and emphasized statistical modeling techniques. 
Table 3: Workshops Conducted by Partner Institutes 
Institute Phase I 2004 2005 2006 
CESAG 6 August Jun-July Summer 
IPSR Nov-Dec Nov-Dec November
INSP NA July-Aug July-Aug Summer
The Standard Workshop. The main objective of the standard workshops is to improve the 
capacity of health professionals and decision makers at district and program levels in M&E. 
The principal output was to be an M&E plan, or a performance monitoring plan (PMP). To 
that end, the workshop covered the essential components of such a plan, called the 
“fundamentals.” These include developing frameworks, selecting indicators and identifying 
data systems. Participants were also expected to be exposed to state-of-the-art tools and 
methods used in M&E in the main programmatic areas, including family planning, maternal 
health, child health, nutrition and HIV/AIDS/STIs. Workshop activities included group 
projects whereby M&E plans were developed based on illustrative case studies, with 
participants being divided into groups depending on their topic of interest. During the last 
week of the workshop, the groups were to present their plans and received feedback from oth
participants and instructors. A field visit to health facilities provides participants the 
opportunity to practice M&E skills acquired in the workshop. Participant feedback was 
solicited at the end of the workshop. IPSR and CESAG also obtained daily assessments from
randomly-selected participants. All of the feedback obtained is summarized later in this 
report. 
The Impact Workshop. The primary objective of this workshop was to provide 
professionals responsible for evaluating PHN programs with appropriate concepts, methods 
and tools for examining the impact of their programs on health-related behavior and health 
outcomes. There was intensive training on statistical techniques for evaluating program 
impact. This hands-on training covered the following areas: fundamentals of M&E: overview
and uses of M&E results, conceptual frameworks, data and information systems, indicators, 
evaluation designs and building M&E plans; building empirical models from conceptual 
frameworks; evaluating programs using simple and multilevel regression analysis; problems 
usually encountered when evaluating programs and how to solve them. Sessions on measurin
socioeconomic status and designing facility surveys were also included. Sessions on design o
household surveys and problems usually encountered during their implementation were 
er 
g 
f 
included. Practical application of these skills included working in small groups to conduct 
MEASURE Evaluation 
ME Final.doc 6 12/30/2005 
H H H
H
analyses using real datasets and presenting the results. Exercises and practical applications 
used the STATA statistical software. While reviewing the statistical techniques, an important 
objective of the workshop was to develop the criteria for choosing the appropriate statistical 
tool to be able to properly evaluate the impact of programs given different scenarios of data 
availability and characteristics of the program being evaluated. Another emphasis was on the 
proper interpretation of the statistical results for assessing program impact. 
Workshop activities included group projects, which consisted of M&E plan development 
based on illustrative case studies. On the final day of the workshop, the groups presented their 
plans and received feedback from other participants and instructors. There were presentations 
on the following topics: the impact of the Oportunidades Program on the decision to use 
contraceptive methods in marginal urban areas of Mexico; the impact of the “Con todas las 
manos” program on hand washing by child caretakers during high-risk contamination 
outbreaks in Uchiza, Peru; the impact of the increase in health service supply on institutional 
deliveries in Peru; the impact of the Seguro Popular program on out-of-pocket expenditures in 
health in Aguascalientes-Mexico; and, the impact of the Seguro Popular program on total 
health expenditures of families affiliated to the Oportunidades program in Mexico. 
Agenda. The schedule, the sessions and the faculty of all three of the 2005 workshops are 
included in Annex A (see Table 23, Table 24, and Table 25). The 2004 workshops were 
essentially the same. 
Faculty. The faculty for these workshops came primarily from M/E and the training 
institutions. In all cases there were “guest instructors” who came from government 
institutions, such as the Ministries of Health, a few donors and technical experts from a few 
USAID cooperating agencies 
In Phase II, M/E was expected to assist their partners in taking on more responsibility for the 
workshops. However, M/E still played a key role in designing, developing and delivering the 
workshops. For example, in Senegal Abdou Billo Mounkaila helped to revise the curriculum 
of the course in July-August, 2004. He also participated as an instructor in the workshop. In 
2005 Anne LaFond and Mounkaila organized a stakeholder’s workshop on organizational 
capacity, which resulted in the identification of priorities for capacity building. They also 
participated in a Training of Trainers (TOT) workshop to strengthen the capacity of key 
trainers. In Thailand Bates Buckner developed the agenda and workshop content, in 
collaboration with Dr. Uraiwan. Dr. Buckner and Shelah Bloom provided instructional support 
at the workshop. Nina Frankel evaluated the draft PHN curriculum modules, which were field 
tested for the first time at this workshop. In Mexico, M/E also still plays a key role in 
designing, developing and delivering the workshops. The workshop is heavily dependent on 
Gustavo Angeles of M/E, in particular, especially in the design and delivery of the statistical 
modules. 
Table 4 lists the 2005 sessions and the faculty responsible for each session. In general, the 
partners are still relying on M/E staff to deliver workshop sessions. CESAG is the least reliant 
with only 5 of 18 sessions delivered by JSI-M/E. Actually, there are only two sessions for 
which CESAG needs help: Data Sources and Impact Evaluation. It can probably handle the 
other three fairly easily. 
IPSR is more dependent on M/E, especially for 9 of the 15 fundamentals sessions. These are 
the more technical areas where IPSR believes it lacks practical experience in applying the 
modules. INSP is the most dependent (20 of 33 sessions). Some of these are “team taught” by 
an M/E and INSP instructor. Others were dropped this year and may not be added next year. 
MEASURE Evaluation 
ME Final.doc 7 12/30/2005 
Also, some sessions could be turned over right away (Developing an M&E Plan, 
Communicating with Policy Makers, for example). Thus, there are nine impact sessions, in 
particular, where INSP will need a good deal of assistance – and where they will need to gain 
practical experience. Both IPSR and INSP believe that they can handle the theoretical parts of 
the sessions. Where they are least comfortable is in the application of theory to practice. 
Table 4: Training Modules and Assignments (M/E assignments shaded) 
Modules CESAG IPSR INSP
Fundamental Modules 
1. Overview/Theories /Importance of evaluation CESAG M/E MOH
2. Framework, Part I CESAG M/E INSP
3. Framework, Part II IPSR 
4. Developing an M&E Plan CESAG M/E M/E 
5. Indicators, Part I CESAG M/E INSP
6. Indicators, Part II IPSR 
7. Data Sources, Part I M/E-JSI M/E INSP
8. Data Sources, Part II M/E 
9. Experimental and quasi-experimental designs INSP 
10. Observational designs, reliability, validity M/E INSP
11. Calculating and interpreting coverage indicators M/E 
12. Impact evaluation M/E-JSI M/E 
13. Qualitative methods in M&E MOH IPSR INSP 
14. Sentinel & behavioral surveillance MOPH
15. Communicating to policy makers/Info for DM M/E-JSI IPSR M/E 
Program Modules 
1. M&E of Maternal Health programs MOPH 
2. M&E of Child Health programs MOH MOPH 
3. M&E of Family Planning programs UNFPA Pop Cncl 
4. M&E of Nutrition programs MOH MOPH 
5. M&E of STD/HIV/AIDS M/E-JSI-MOH FHI 
Impact Modules 
1. Framework for empirical models M/E 
2. Models for continuous events INSP
3. Models for binary events INSP
4. Models for multiple category events M/E 
5. Multilevel models M/E 
6. Socio-economic models M/E 
7. Problems in impact evaluation: endogeneity INSP-M/E 
8. Explicative endogenous models INSP-M/E 
9. Estimation of impact of focal programs M/E 
10. Models for panel data, double difference models M/E 
11. Models for cohort aggregation M/E 
12. Introduction to Propensity Score Matching M/E 
13. Use of data for program and policy changes M/E 
Sessions in 2004 that were dropped in 2005 
1. Multi-category models M/E 
2. Interaction models M/E 
3. Design of household surveys M/E 
MEASURE Evaluation 
ME Final.doc 8 12/30/2005 
4. Using evaluation results to improve programs M/E 
Practicum Modules 
1. Impact evaluation in Paraguay M/E 
2. Group Work on M&E Plans All IPSR-M/E INSP-M/E 
3. Field trip MOPH/Field work CESAG-MOH MOPH 
4. SPSS (optional)/Epi Info/STATA Consultant MOPH INSP-M/E 
Total Training Institute 5 6 12 
Total: M/E-JSI 5 10 20 
Total: Other (MOH, consultants, experts) 8 8 1 
Grand total 18 24 33 
Over the years, M/E has provided fewer and fewer trainers and resource personnel to CESAG 
as its capacity grew. Through a combination of CESAG faculty and experienced guest 
instructors they are moving steadily to what they call the “ownership” phase, where they will 
be able to carry out the workshops without any assistance from M/E. IPSR and INSP are 
attempting to implement the same strategy. 
E. Participants 
All three of the institutions are doing very well in attracting applicants. IPSR received over 70 
applications from all over the ANE region, 32 of which had secured funding. CESAG 
received 49 applications its first year, with almost 80 percent with their own funding. INSP 
received 44 applications, but only 7 with their own funding. These facts demonstrate three 
things: 1) there is a significant demand for M&E training, which exceeds the capacity of the 
training institutions; 2) the workshops could be filled with people who have their own funding; 
and 3) INSP is not able to attract many people who have their own funding. 
Some of the respondents interviewed were concerned that the workshops would not be able to 
attract enough applicants to survive financially. This does not seem to be the case. In addition 
to USAID Mission funding, applicants have received funding from a variety of donors, 
government agencies, NGOs and Cooperating Agencies. CESAG has done so well that it has 
not requested fellowships from M/E. Even INSP could survive if it limited applicants to 
Mexicans. 
The principal rationale for the fellowships is to make it possible for qualified applicants from 
poorer countries to attend the workshop. The need for fellowships is greatest in LAC where 
PHN funding has declined dramatically over the past decade. If the selection criterion were 
self-financing, none of the institutes would have any problem filling their workshop slots. But 
as a regional program, a concerted effort has been made to invite applicants from as many 
countries as need this help. The program has definitely been successful in that respect. The 
selection criteria are: 
ƒ Academic background 
ƒ Experience in health program M&E (with focus on the district level) 
ƒ Place of work 
ƒ Likelihood of using the M&E skills taught 
ƒ Recommendations 
ƒ Background in mathematics (INSP only) 
Not all of the applicants accepted were able to attend the workshops. Some dropped out due to 
lack of funding, changes in jobs and job descriptions, or visa and other administrative 
problems. 
MEASURE Evaluation 
ME Final.doc 9 12/30/2005 
In the five workshops undertaken in Phase II, the three training institutions have trained 132 
individuals from 38 countries in M&E. 7
F These participants have come from government 
agencies, donor organizations and NGOs. They have been involved in M&E at the national, 
provincial, district and community levels. 
Table 5: Workshop Participants by Country and Region 
ANE:IPSR LAC: INSP AFR: CESAG 
Country Number Country Number Country Number 
Bangladesh 4 Bolivia 1 Benin 5 
Cambodia 3 Brazil 1 Burkina Faso 7 
India 2 Dom. Rep. 2 Burundi 2 
Indonesia 1 Ecuador 2 Cote d’Ivoire 4 
Kenya 3 El Salvador 1 Gabon 1 
Kyrgystan 2 Honduras 3 Haiti 2 
Myanmar 2 Mexico 13 Madagascar 5 
Nepal 3 Nicaragua 2 Mali 5 
Pakistan 4 Paraguay 1 Mauritania 5 
Philippines 2 Peru 5 Moldavia 1 
Tajikistan 1 Niger 1 
USA 1 RCA 2 
 Rwanda 1 
 Senegal 10 
 Tchad 1 
 Togo 1 
Total=12 28 Total=10 31 Total=16 53 
F. Participant Evaluation 
The participants were asked to evaluate each session and a number of features of the workshop 
(food, hotel, etc.) at the end of the workshop. Some were asked to do daily assessments. The 
results are difficult to combine due to the use of different criteria and rating scales. Suffice it 
to say that the participants liked the workshops, were satisfied, learned a lot and found the 
material to be interesting, well presented and useful. 
1. Satisfaction 
CESAG. The majority of participants in both the 2004 and 2005 workshops were satisfied. 
Over 90 percent of the characteristics of the workshop were rated good or better (on a four￾point scale of excellent/very good/good/fair/poor). The top three features in 2005 were: course 
objectives met, course reading material and tools, and the workshop room. The lowest rating 
was for the time devoted to each session. In some cases it was too much, but in most it was 
not enough. 
IPSR. The 28 participants were generally satisfied with the workshop and found it both 
illuminating and useful. The respondents rated most of the organizational features at the high 
end. The highest rated features were overall satisfaction, computer facilities and other 
facilities. The lowest ratings were given to the quality of instruction and meeting course 
objectives. 
 
7
 At the time of its visit to Thailand, IPSR had not yet held its 2005 workshop. That was scheduled for November 
and 25 applicants had been accepted. 
MEASURE Evaluation 
ME Final.doc 10 12/30/2005 
INSP. The majority of participants in both the 2004 and 2005 were very satisfied with the 
workshops. Participants gave both workshops very high marks. All of the following were 
rated from 9.2-9.4 (on a 10-point scale): Perception of workshop in general; General level of 
satisfaction; Learning environment; Interpersonal relationships; Contents of workshop; and 
Quality of instruction. 
2. Content 
Course content was rated for each session. These ratings were even higher, although it is 
difficult to compare them because the rating scales and content were different. 
CESAG. Most assessments were of good or better in the 93-100 percent range in 2005 and 
2004. The table in the case study shows the two highest ratings were for Qualitative methods 
Drug logistic management and Introduction to M&E theories. The lowest rated modules were 
Impact evaluations, Quantitative methods and Strategic information for decision-making. 
IPSR. Most ratings were in the 82-93 percent range for content. The highest ratings for 
content were for Communicating with policy makers, M&E of STD/HIV/AIDS, and M&E of 
family planning. There were no low ratings. The lowest were at 75 percent: M&E for 
maternal health, M&E for Child Health, Qualitative methods, and Impact evaluation. 
INSP. In 2004 eleven sessions received 100 percent marks. In 2005 eight did. These were 
both mixtures of fundamental and impact modules. Nothing was rated below 60 percent in 
2005. Information systems and Data sources, Indicators for program evaluation, and 
Qualitative methods were the lowest rated modules. 
These ratings are for content and presentation of that content. This does not tell us much about 
utilization. What may be interesting here is the difference between immediate assessments of 
the training sessions and later assessments of what was actually utilized. In fact, we will see 
some shifts in rankings and ratings in the participant follow-up and Synergy surveys. For 
more details, see the Participant Evaluation sections of the case studies. 
In addition to these participant ratings obtained at the end of the workshops, in two cases M/E 
collected feedback cards daily. Most of the suggestions revolved around making the sessions 
more practical, reducing lecturing, increasing group work, providing more relevant examples 
and having more case studies. Another common suggestion was that the trainers should be 
more familiar with their presentations and should build in more examples from their own 
experience. This last suggestion will emerge again when we look at the partner ratings about 
their capacity to deliver a session. Practical experience is often cited as lacking in the 
feedback from IPSR and INSI workshops. 
III. ACHIEVEMENTS
Achievements are broken into two groups, corresponding with the objectives of the SOW for 
this assessment: institutional capacity building and participant use of new knowledge and 
skills. The assessment team combined survey results with secondary data and information 
gained from document reviews as well as interviews with partners and participants to rate the 
partners’ achievements to date. This rating also helps to identify areas where the training 
institutions do and do not need further technical assistance to move up the capacity continuum. 
A. Capacity Building: Moving up the Continuum 
As noted previously, two tools were used to assess capacity. The first is called an 
institutional capacity assessment tool. It assesses an institute’s capacity along 17 criteria, 
MEASURE Evaluation 
ME Final.doc 11 12/30/2005 
such as leadership, infrastructure and financial management (see Annex B.2 for an example of 
the instrument). The second is called a substantive capacity assessment tool. It assesses an 
institute’s ability to deliver each of the 16-25 training modules that make up the standard or 
impact workshop (see Annex B.3 for an example of the instrument used). 
Institutional capacity. The M/E Results Framework calls for an overall rating of institutional
capacity on a four-point scale (1=requires significant TA, 2=requires limited TA, 3=does not 
require TA, and 4=can provide TA to others). M/E is experimenting with a 10-point scale and 
the Team decided to use that instead, since that is how data were recorded in 2004 and 2005. 
Each of these tools organizes the criteria and modules into groups. For the institutional 
capacity assessment, these are Organizational Capacity (7 criteria), Technical Capacity in 
M&E (1), Technical Capacity in Training (6), and Organizational Motivation Capacity (3). 
The Team used assessments 
done by a key informant 
survey of representatives of 
IPSR and INSI in 2005,8
F
and a stakeholder 
assessment of CESAG that 
was carried out by M/E in 
Senegal in May, 2005. The 
results are shown in Table 
6. 
This table shows that INSP is well on its way to taking over the program. On a scale of 1-10, 
a score of 8 and above indicates that the institution has that capacity. INSP and IPSR both 
need to improve their technical capacity in M&E, but they are both doing very well on the 
other three components. CESAG has lower overall ratings but no specific weaknesses. The 
Team believes that the CESAG stakeholders rated their program too low and that CESAG is 
also on its way to gaining the institutional capacity needed to run the course independently. 
Substantive capacity. The team designed this tool and distributed it to the same key 
informants who received the institutional tool (see Annex B.4 for an example of the instrument 
used). The tool (which follows a four-point scale to conform with USAID’s continuum) calls 
for rating each module as 1) requires significant TA; 2) requires limited TA; 3) does not 
require TA; and 4) can provide TA to others. The training modules are in three groups: 
Fundamental Modules, Program or Impact Modules and Practicum Modules. The number of 
modules in each category varied among the workshops. 
A score of 3.0 indicates 
capacity to deliver the 
modules without 
external assistance. 
Table 7 shows that 
CESAG is the leader in 
this area, but it still 
needs to improve its 
capacity to deliver the 
 
Table 6: Overall Institutional Capacity Ratings 
CESAG IPSR INSP
Overall Institutional Capacity N=11 N=9 N=5 
Organizational Capacity 6.9 7.7 8.2 
Technical capacity in M&E 6.9 6.3 7.6 
Technical capacity in training 6.8 7.6 8.2 
Organizational Motivation 7.1 7.6 9.8 
Average 6.9 7.6 8.5
Table 7: Overall Substantive Capacity Ratings 
Overall Substantive Capacity CESAG IPSR INSP 
in M&E Training N=5 N=10 N=6 
Fundamentals: Core modules 2.5 2.2 3.1 
Program/impact modules 2.8 2.4 1.8 
Practicum modules 3.2 2.8 3.1 
Average 2.7 2.3 2.4 
8
 These data came from interviews with 5-9 individuals from each training institution (manager, coordinator and 
faculty) who are very familiar with the course. 
H
H
MEASURE Evaluation 
ME Final.doc 12 12/30/2005 
H
H
fundamental models, in particular. IPSR also needs to improve its capacity in this area and in 
the program modules. INSP, on the other hand, is strong in both the fundamental and 
practicum modules but weak in its capacity to deliver the impact modules. This was also 
reflected in Table 4 that describes responsibilities for workshop sessions. 
B. Institutional Capacity Building 
This section looks at all 17 of the capacity criteria, grouped into the four categories mentioned 
above. 
1. Organizational Capacity 
We begin with the seven criteria that make up Organizational Capacity, which are listed in 
order in Table 8. INSP is the 
highest with five categories 
in the 8-10 range. Only two 
criteria are rated low. These 
are human resources and 
financial management. As 
the case study shows, all 
three of the institutions are 
weakest in these two areas. 
These themes appear over 
and over again in the 
assessment. The institutions 
do not have enough qualified 
and experienced faculty. 
For INSP and IPSR the 
financial concern has more to do with the availability of fellowships (which IPSR and INSP 
are afraid will be decreased or eliminated) than their financial management. CESAG does not 
have that problem, but it does have a problem with paying its instructors adequately and on 
time. 
The strongest element for all three organizations is their expertise in hosting short-term 
workshops. They all rate highly in infrastructure. Inter-institutional linkages are also 
strong for all three, although IPSR needs to bolster its linkages with other national institutions, 
especially the Faculty of Public Health. Leadership has been strong and supportive, but both 
CESAG and IPSR are experiencing changes in leadership and there is some uncertainty as to 
where the new leaders will take the workshops. The Team believes that organizational 
processes and program management are high for all three organizations. There are some 
areas that need improvement; but overall, all three have a good deal of capacity in these areas. 
2. Technical Capacity in M&E 
There is only one criterion in the capacity assessment tool, but more detail is included in the 
substantive capacity section. Nonetheless, this institutional rating is important in that it 
reflects the need for continued assistance to sharpen the skills of faculty and local instructors 
in the newer areas that are 
introduced in the workshops, 
continued knowledge transfer 
of cutting-edge methods and 
tools, increased opportunity 
to gain more experience and 
Table 8: Organizational Capacity Ratings 
(lowest ratings shaded) 
CESAG IPSR INSP
Organizational Capacity N=11 N=9 N=5 
1. Strategic leadership 7.6 8.1 10.0 
2. Human resources 5.2 6.0 5.0 
3. Financial management 6.1 6.1 5.2 
4. Organizational processes 6.3 8.0 9.8 
5. Program management 6.7 7.9 9.2 
6. Infrastructure 8.1 9.2 9.8 
7. Inter-institutional linkages 8.3 8.6 8.6 
Average 6.9 7.7 8.2 
Table 9: Technical Capacity in M&E 
CESAG IPSR INSP
Capacity N=11 N=9 N=5 
 Technical capacity in M&E 6.4 6.3 7.6 
MEASURE Evaluation 
ME Final.doc 13 12/30/2005 
state-of-the-art knowledge, and new partnerships to expand professional and technical 
horizons. This is an area that the institutions see as the cornerstone of their assistance needs. 
It is rated relatively low here, but the substantive capacity section will show that needs are 
greatest in certain fundamental and impact modules, many of which are now given primarily 
by M/E staff. 
3. Training Capacity 
INSP is also leading in this area, especially in marketing, training content, and course logistics. 
IPSR is also doing well on most criteria, including logistics. CESAG is also doing well except 
for two areas. The two most critical areas that need work are Training planning and 
sustainability and 
Trainee follow-up, 
the latter being very 
low for all three 
institutions. 
Follow-up of 
trainees is an
acknowledged 
weakness of the 
project. M/E has a 
built-in follow-up 
strategy but 
response rates have 
been low. In our 
survey of participants, which included repeated email and telephone calls, the response rates 
were 29 percent (IPSR), 49 percent (CESAG), and 58 percent (INSP). 
Table 10: Training Capacity Ratings (low ratings shaded) 
CESAG IPSR INSP 
Technical capacity in training N=11 N=9 N=5 
1. Training content 8.8 7.9 9.4 
2. Training planning & sustainability 4.7 6.8 6.8 
3. Course logistics 7.9 9.2 9.4 
4. Marketing 7.7 8.1 10.0 
5. Course implementation 7.5 7.5 8.8 
6. Trainee follow-up 3.1 5.3 5.0 
Average 6.6 7.6 8.2 
4. Organizational Motivation 
Table 11: Organizational Motivation Capacity 
CESAG IPSR INSP 
Organizational Motivation N=11 N=9 N=5 
1. Mission 6.3 7.2 10.0 
2. Culture 6.8 7.9 9.8 
3. Incentives/Rewards 7.6 7.8 9.6 
Average 6.9 7.6 9.8 
INSP comes out on top 
again in this category, with 
an almost perfect score. 
IPSR will likely improve 
once the new Director 
returns and the organization 
clarifies its mission and 
vision for the future. 
CESAG’s scores are 
probably lower than they 
should be. At the moment CESAG is not sure where it wants to go with the M&E course. It is 
going through a transition phase with a new Workshop Coordinator, and it has had some 
problems paying guest trainers adequately and on time, which affects motivation. CESAG’s 
ratings on motivation should increase once the capacity assessment process is completed and 
these issues are addressed. 
C. Substantive Capacity Building 
This section deals with the substantive content of the workshop. Again, the basic question is: 
to what extent are the regional partners able to carry out the training modules without TA from 
M/E? The ratings are broken into three groups: fundamental (core) modules; health 
program/impact modules, and practicum modules. Please keep in mind that these modules 
were rated on the USAID four-point capacity scale, where three indicates the ability to carry 
out the module without technical assistance from M/E. There are quite a few individual 
MEASURE Evaluation 
ME Final.doc 14 12/30/2005 
ratings at level four, which indicate that some respondents believe that the institutions are able 
now to show other organizations how to deliver a number of these various modules. 
1. Capacity in Fundamental Modules 
Table 12 summarizes the fundamental modules taught in 2005 (2004 for IPSR). A blank cell 
means that module was not included in the curriculum. Many of these modules were unique to 
a particular workshop. For example, only IPSR introduced Sentinel and behavioral 
surveillance, which is usually associated with HIV/AIDS programs. INSP did not include 
Impact evaluation and Quantitative evaluation because those were the core of the Impact 
evaluation curriculum. The institutions usually spent the first week or week and ½ on 
fundamentals. 
H
Table 12: Capacity in Fundamental Modules (low ratings shaded) 
CESAG IPSR INSP 
Capacity in M&E Fundamentals N=5 N=10 N=6 
Qualitative methods in M&E 3.0 2.9 3.0 
Sentinel & behavioral surveillance 2.1 
Communicating to policy makers/info for DM 1.8 2.6 
Indicators 2.8 2.4 2.5 
Data sources 1.8 2.4 2.8 
Frameworks 3.0 1.9 3.8 
Developing an M&E Plan 3.2 1.9 2.0 
Calculating & interpreting coverage indicators 1.9 
Impact evaluation 2.0 1.8 
Quantitative evaluation 3.0 
Evaluating efficiency 2.2 
The importance of evaluation in the health sector 3.5 
Experimental & quasi-experimental designs 3.3 
Observational designs, reliability, validity 3.5 
Total number of modules taught 9 9 8 
Average 2.5 2.2 3.1 
As the table shows, the institutions already believe they have the capacity to carry out a 
number of these modules by themselves (those rated 3.0 and above). CESAG believes it can 
carry out four of the nine modules independently. IPSR does not think it is ready to carry out 
any of these modules by itself, although the Qualitative methods module is close. INSP is the 
most confident, believing that it can carry out five of its eight fundamental modules. 
At the opposite end, INSP does not believe that it needs significant TA in any of these 
modules (those rated 1.0-1.9). IPSR has four for which it will need significant help, and 
CESAG has two in that category. 
These ratings also reflect the capabilities of the individual instructors. In some cases a module 
is rated low because an M/E person or a guest instructor rather than a faculty member 
currently delivers it. The institution itself does not have the needed capacity to deliver it. In 
other cases a module is rated high because a local guest instructor (say from the MOH) 
delivers it and that person is expected to deliver that module every year. M/E staff currently 
delivers many of the fundamental modules in the IPSR workshop. IPSR will need to develop 
MEASURE Evaluation 
ME Final.doc 15 12/30/2005 
its own capacity to take over responsibility for these modules and/or find local experts who 
can be counted on to deliver them. 
2. Capacity in 
Modules for M&E 
in Health 
Programs 
These modules were 
delivered by CESAG
and IPSR. INSP did 
not include any healt
program modules, pe
se. The impact 
modules included 
examples and 
exercises that applie
a particular impact 
technique to a 
particular health 
program. 
IPSR reflects the set 
of health program 
h 
r 
d 
modules that are supposed to be included in the standard workshop. CESAG modified that to 
include health programs that were priorities in the region. It is clear that both institutions are 
more confident about their capacity in these modules than in the fundamentals, although the 
total scores are similar. There are no modules for which they would need significant TA, and 
CESAG believes that it can carry out four of the seven modules without any further TA. The 
Team believes that CESAG will have no trouble taking over the remaining modules. They 
have good relations with MOH experts who already deliver most of these health program 
modules. They just need to find more experts to deliver the two modules currently taught by 
M/E staff: STD/HIV/AIDS and Behavior change communication. IPSR also relies on guest 
instructors to carry out all of its health program modules. However, they may need assistance 
in standardizing the sessions. IPSR does not seem to have a standard curriculum for these 
modules. Rather, the presenters are given a list of learning objectives and a generic set of 
presentation suggestions. The presenters use their own materials and tailor them to fit the 
IPSR objectives. 
Table 13: Capacity in Health Program Modules (CESAG and 
IPSR) 
Capacity in M&E for Health CESAG IPSR INSP
Programs N=5 N=10 N=6 
M&E of STD/HIV/AIDS programs 3.2 2.6 
M&E of FP/RH programs 3.0 2.5 
M&E for maternal health programs 2.5 
M&E of child health programs 3.2 2.4 
M&E of nutrition programs 2.0 
M&E of vaccination programs (RED) 3.2 
M&E of malaria programs 2.4 
M&E for drug logistics management 2.2 
Behavior change communication 2.6 
Total number of modules taught 7 5 0 
Average 2.8 2.4 0 
MEASURE Evaluation 
ME Final.doc 16 12/30/2005 
2. Capacity in Modules for Impact Evaluation 
INSP is the only partner that includes these modules in its curriculum. This is the area where 
INSP needs the greatest amount of technical assistance. M/E staff taught eleven of the 13 
impact sessions in 2005, although co-teaching was used in three of the sessions. For M/E the 
entire burden was placed on one individual. These sessions are the core of the “Impact 
Evaluation” course and M/E and INSP will need to find ways to develop the capacity within 
INSP needed to deliver 
Table 14: INSP Capacity in Impact Modules them. 
(in rank order) 
Impact Modules (N=6) Avg.
Use of data for program and policy changes 2.5 
Framework for empirical models 2.3 
Models for continuous events 2.3 
Models for binary events 2.3 
Models for multiple category events 2.0 
Introduction to Propensity Score Matching 2.0 
Multilevel models 1.7 
Problems in impact evaluation: endogeneity 1.7 
Explicative endogenous models 1.7 
Models for panel data, double difference models 1.5 
Estimation of impact of focal programs 1.2 
Models for cohort aggregation 1.2 
Socio-economic models 1.0 
Total number of modules taught 13 
Average 1.8 
INSP staff was quick to 
point out that they have the 
capacity to address these 
impact models in the 
abstract. There are ample, 
highly qualified 
statisticians in INSP. What 
they lack is experience in 
applying these models to 
practical situations. That 
experience is being gained 
little by little as staff gets 
more field experience in 
such on-going evaluations 
as the Oportunidades and 
Seguro Popular programs. 
Some options that were 
discussed with INSP are: 
1) a special workshop for 
the INSP faculty 
responsible for these sessions; 2) team-teaching in 2006 with an expectation that full 
responsibility for these sessions will be transferred to INSP for the 2007 course; 3) 
concentrating on those modules that can be mastered quickly while folding the others into an 
“introductory” session; and 4) dropping those sessions that cannot be mastered over the next 
two years. 
3. Capacity in Practical Application Modules 
The practicum sessions seem to be the highlight of the workshop from both the partner and 
participant point of view. Only two cells are rated below 3.0. The training institutions are 
either delivering these 
sessions themselves or are 
ready to do so. CESAG does 
not seem to need any help in 
this area. IPSR has had some
problems with its field site 
visits (more lectures, 
language problems) and is 
trying another approach this 
year. The group work on 
M&E plans has also been 
revised to be introduced at 
Table 15: Capacity in M&E Practical Applications 
Capacity in M&E Practical 
Applications 
CESAG 
N=5 
IPSR 
N=10 
INSP
N=6 
Field/site visits/field work 3.4 3.1 
Group work on M&E plans 3.2 2.6 3.4 
SPSS/Epi Info/STATA 3.0 3.3 
Computer lab 3.3 
Impact evaluation of Paraguay 2.0 
Average 3.2 2.8 3.2 
MEASURE Evaluation 
ME Final.doc 17 12/30/2005 
the start of the workshop so that participants can better understand how the various 
components fit into that plan. INSP is very competent in this area. It should probably replace 
the Paraguay example with one of the Mexican examples (e.g., Oportunidades, which is 
already a special session). 
.D. Utilization of New Knowledge and Skills 
The next sections address the second objective of the SOW, participant use of new knowledge 
and skills gained at the workshop. This has been assessed in two ways. First, by examining 
the data from the follow-up surveys that M/E conducted 6 and 12 months after the workshops. 
Second, by conducting an email and phone survey of the 2004 and 2005 workshop 
participants. 
1. Follow up Survey Findings 
a. Most Useful Modules 
MEASURE Evaluation administered a follow-up questionnaire to the 2004 and 2005 M&E 
workshop participants in September 2005. Thirty responses were received from a total of 96 
participants (31 percent). The breakdown was CESAG (10/53), IPSR (11/28) and INSP 
(9/15). 
The questionnaire asked for the following information: 
• current employment and responsibilities in M&E 
• which topic areas from the workshop were most useful in their work, and 
• “success” stories on how the M&E workshop contributed to or helped their work or 
research. 
The table below lists the percentage of the respondents9
F who identified each module as “most 
useful.” Although the sample is very small, the results are consistent with participant feedback 
at the end of the workshops. Perhaps most important is the ranking of development of M&E 
plans as the most useful session by two of the cohorts and the relatively high ratings for other 
key components of planning (framework, indicators, data sources). 
Table 16: Modules Listed as Most Useful in M/E Follow-up Survey 
(highest ratings shaded) 
Modules CESAG IPSR INSP 
N=10/53 N=11/28 N=9/15 
Fundamentals: Core Modules 
Frameworks 33% 64% 33% 
Developing an M&E plan 67% 37% 78% 
Indicators 44% 73% 
Data sources/Information systems 11% 55% 22% 
Qualitative methods 22% 36% 
Information for decision-making 22% 
Impact evaluation 11% 55% 
Quantitative methods & data analysis 33% 36% 
Experimental & quasi-experimental designs 33% 
Evaluation of project efficiency 11% 
 
9
 One respondent identified all modules as useful and is not counted. 
MEASURE Evaluation 
ME Final.doc 18 12/30/2005 
Calculating and interpreting coverage 18% 
Sentinel and behavioral surveillance 
Program modules for PHN 
M&E for child health programs 11% 36% 
M&E for maternal health programs 36% 
M&E of vaccination programs 33% 
M&E of RH/FP programs 22% 45% 
M&E of nutrition programs 27% 
M&E of Malaria programs 22% 
M&E for drug logistics 0 
M&E of STI/AIDS & TB 22% 
Behavior change communication 33% 
Impact Modules 
Socio-economic models 
Models for panel data, double difference models 
Framework for empirical models 
Estimation of impact of focal programs 
Introduction to Propensity Score Matching 
Use of data for program and policy changes 
Multilevel models 11% 
Explicative endogenous models 
Models for cohort aggregation 
Models for multiple category events 
Models for continuous events 
Models for binary events 
Problems in impact evaluation: endogeneity 
Practicum Modules 
Group work on M&E plans 33% 
Field visit/Field work 9% 
EPI Info/STATA 11% 56% 
* Shaded cells are above 50 percent 
b. Most Applied Knowledge and Skills 
CESAG respondents tended to use the fundamental modules most, especially to develop and 
implement M&E plans, to develop indicators and to simplify reporting. New knowledge and 
skills acquired during the workshop enabled participants to10
F : 
1. Develop evaluation monitoring plans and tools. 
On my return I was given responsibility for developing a monitoring plan for the program 
with the government and UNICEF. The M&E course enabled me to develop and integrate 
a surveillance monitoring plan. This was introduced to the district chief medical officers to 
integrate monitoring activities at the district level. 
2. Develop and use indicators. 
The training enabled me to better develop indicators for my program. I used examples 
from the evaluation plan presented during the training. This has become an indispensable 
 
10 Translated from French 
MEASURE Evaluation 
ME Final.doc 19 12/30/2005 
tool for monitoring activity plans at the program and global levels of my organization. 
The Indicators session was important and necessary to my work. In the health system in 
Madagascar we revised and simplified all the reports submitted by the health centers to 
improve and increase regular reporting. 
3. Increase understanding and improve M&E practices. 
The three week training helped me understand M&E much better and to become much 
more effective in my work. 
My institution is a centralized structure in the MOH with responsibility for evaluation of 
hospitals in Mali. The M&E workshop contributed to strengthening our evaluation 
practices in general, especially of the hospitals. 
4. Train others in M&E knowledge and skills. 
On returning from the M&E workshop, I provided an M&E training workshop for 25 
people. 
IPSR respondents stated that the modules used most were developing an M&E plan and 
indicators. Those who mentioned development of evaluation plans said that not only were 
they using the principles and methods taught in the workshop in their own work, some were 
also assisting others in designing M&E plans. Of particular value, was the use of logframes in 
designing programs and M&E plans. The ability to define appropriate indicators and data 
sources for impact evaluation were also seen as significant benefits from the workshop and 
applied to their work. Overall, they applied what they learned to: 
1. Develop M&E plans 
For developing the M&E plan I followed many of the steps that I practiced during the training 
course. As a result it was not a difficult task for me. The M&E plan was widely accepted by 
USAID, and my leadership and guidance for developing such a comprehensive plan was 
highly appreciated by bosses. 
The workshop has helped contribute to my understanding of effective M&E and the 
importance of creating a solid M&E plan prior to the beginning of a new program or project. 
2. Develop Frameworks 
I have been able to use the methods on development of project logframes for development of a 
large project that my previous organization (that had supported my training) is now 
launching. It was widely appreciated within the organization and has been found very useful 
for getting a compact and comprehensive understanding of the project by all the team 
members managing the project. 
3. Develop indicators 
The whole range of discussion about indicators and the selection of different type of indicators 
for developing an M&E plan for a development project was very useful for me. 
“Do not measure if you do not need to use” learning has helped me to keep the number of 
appropriate indicators down, resulting in improved monitoring and management of data. 
4. Communicate with policy makers 
The learning I had in communicating results to policy makers provided me the knowledge on 
MEASURE Evaluation 
ME Final.doc 20 12/30/2005 
how to present results in most convincing way which policy makers will appreciate, even if it 
is a negative result. 
The Communicating to Policy Makers part of the workshop has been really useful in managing 
and advocating for new programs. 
As I have frequent and direct interaction with policymakers at federal level I feel more 
confident in communicating with policymakers 
INSP respondents tended to apply two modules most: the STATA software and Preparation of 
evaluation plans. None of the impact modules stood out. In general, the respondents seemed 
to be saying “I really learned a lot from the workshop, it was very helpful, and I would like to 
have the opportunity to use these skills in my job soon.” 
Qualitatively, some respondents reported that new knowledge and skills acquired during the 
workshop enabled them to11
F : 
1. Develop evaluation monitoring plans and tools. 
My participation in the workshop has made it possible for me to be more active in the 
development of the impact evaluation for this new project in Guatemala, including the 
conceptualization of the design. 
The training allowed me to prepare an M&E plan for fortifying food with micronutrients, 
which was approved by GAIN (Global Initiatives for Better Nutrition). 
2. Increase understanding and improve M&E practices. 
The Secretary of Health has incorporated an M&E plan for public health activities, which 
I am leading. 
Fortunately, program evaluation is receiving attention by decision-makers and they are 
requesting more of these types of studies. The workshop permitted me to open a channel in 
a new and relevant area in the institution where I work. 
3. Train others in M&E knowledge and skills. 
I made a presentation on program evaluation to the director and team of the Center of 
Equity and Gender and to the NGO where I work and they have requested printed 
materials as well.
2. Participant Survey Findings 
In November of 2005 the Evaluation Team conducted a participant email and phone survey to 
determine: 1) which of the modules imparted new knowledge and skills (K/S) to the 
participants; and 2) how often the newly acquired knowledge and skills were used in their 
work. The survey was structured so that responses that indicated that no new knowledge or 
skills were acquired were rated 0 and those modules that imparted new K/S were rated (1) 
when the new K/S was never used, (2) when it was seldom used, (3) when it was sometimes 
used, and (4) when it was often used. See Annex B.4 for a copy of the instrument. 
 
11 Translated from Spanish 
MEASURE Evaluation 
Responses were received from 52 of the 112 participants (46 percent) broken down as follows: 
CESAG (26/53), IPSR (8/28) and INSP (18/31). Follow-up telephone and face-to-face 
interviews were completed with 33 respondents (CESAG = 11, IPSR = 4, INSP = 18). Table 
17 summarizes the overall utilization results. It is somewhat surprising that INSP utilization 
was the lowest, not only overall but also in all categories. Also somewhat surprising is the 
finding that INSP rated the 
fundamental modules Table 17: Participant Utilization of Training Modules 
Modules CESAG IPSR INSP 
N=26 N=8 N=18 
Fundamental modules 2.8 2.5 2.4 
Program/impact modules 2.5 2.8 2.0 
Practicum modules 2.6 2.8 1.9 
Overall capacity 2.7 2.7 2.1 
higher than the impact and 
practicum modules. 
CESAG also rated 
fundamentals highest but 
IPSR rated them lowest. 
These differences are 
slight, however, given the 
small sample sizes. Perhaps more important is that the ratings are largely between two and 
three, indicating seldom to sometimes use. 
Fundamental Modules. Table 18 gives more detail on utilization of the fundamentals 
modules. Two things stand out. First is the number of sessions rated at 3 and above: 5 of 9 for 
CESAG, 2 of 10 for IPSR and 3 of 8 for INSP. This indicates that some modules are being 
use more often than would be expected given the overall ratings in the previous table. Second 
is the clustering around the development of M&E plans – the importance of M&E in health, 
frameworks, indicators and data sources, in particular. 
This is supported by the interviews, as well. Participants indicate that these 
fundamental/transversal modules are extremely important because they serve as the basis for 
working in M&E and provide the knowledge and skills that can be adapted to meet whatever 
needs a participant has. Some respondents thought that some of the fundamentals modules 
Table 18: Participant Utilization of Fundamental Modules 
(highest ratings shaded) 
Fundamentals: Core Modules CESAG IPSR INSP 
N=26 N=8 N=18 
The importance of M&E in health 3.1 3.1 
Frameworks 2.9 2.8 3.3 
Developing an M&E Plan 3.0 2.8 2.5 
Indicators 3.4 3.3 3.3 
Data sources 3.2 2.5 1.9 
Calculating & interpreting coverage indicators 2.6 
Impact evaluation 2.4 2.4 
Qualitative methods in M&E 2.5 2.5 1.0 
Sentinel & behavioral surveillance 1.8 
Communicating with policy makers 3.0 2.1 
Quantitative methods 3.1 
Evaluating efficiency 2.1 
Experimental & quasi-experimental designs 2.3 
Observational designs, reliability, validity 2.0 
Number of modules 9 10 8 
Subtotal 2.8 2.5 2.4 
ME Final.doc 21 12/30/2005 
H
H
MEASURE Evaluation 
ME Final.doc 22 12/30/2005 
were a very important part of the workshop. Developing an evaluation plan is something that 
almost everyone can use, no matter what his or her M&E role. Knowing what goes into an 
evaluation (a framework, indicators, data sources, and analytic methods) is useful in 
supervision and screening M&E proposals as well as in designing an evaluation. 
“From the ‘Fundamental concepts; frameworks’ module I learned about baselines and how to 
write conceptual frameworks in one page. I use the knowledge I learned in this module to 
develop proposals.” 
“This [Fundamental concepts; frameworks] was one of the most important modules for me. It 
was the clearest, most precise module for my present work. I learned about indicators and 
qualitative studies. I’m now helping design a new study and this gives me the ability to do it.” 
On the other hand, most participants said that they rarely or never use some of these M&E 
fundamentals because most of designs are not done daily or weekly, but once a year, if not 
less. Two participants summarized the sentiments of most participants: 
‘I learned a lot about fundamental concepts, but I only use them ‘at times’ or ‘rarely’ because 
it’s not every day that we are doing these types of studies [experimental or quasi-experimental 
designs]. 
“I am involved in the maintenance rather than an elaboration of an M&E plan.” 
Program modules. The survey results show that utilization of HPN program modules is 
slightly less for CESAG and slightly more for IPSR than was utilization of the fundamental 
modules. Interestingly enough, even though this is a PHN workshop, the STI/HIV/AIDS/TB 
module was one of the two 
program modules given by 
both workshops and was 
rated relatively highly. It 
was also one of the highest 
rated modules in terms of 
training capacity. What this 
may reflect is program 
priorities in the region. 
More people and more 
money are now directed 
toward HIV/AIDS. Thus, 
the need to learn how to 
monitor and evaluate 
HIV/AIDS programs is 
particularly pressing. 
Once again, job 
descriptions and opportunities affect utilization of program modules. Interviews suggest that 
their circumscribed portfolios. For other participants, the new K/S are used often because they 
fall within the scope of their duties whereas others may have learned new K/S but do not have 
the opportunity or need to use this new information. 
Table 19: Participant Utilization of CESAG and IPSR 
Program Modules (Highest ratings shaded) 
Program modules for PHN CESAG IPSR 
N=26 N=8 
M&E of maternal health programs 2.9 
M&E of child health programs 2.5 3.0 
M&E of FP/RH programs 2.8 
M&E of nutrition programs 2.9 
M&E of STD/HIV/AIDS 2.7 2.7 
M&E of vaccination programs (RED) 2.6 
M&E of malaria programs 2.4 
M&E for drug logistics management 2.3 
Behavior change communication 2.4 
Total number of modules 6 5 
Subtotal 2.5 2.8
several participants do not use all of the program level modules because some are outside of 
MEASURE Evaluation 
ME Final.doc 23 12/30/2005 
Impact modules averaged 2.0 (seldom used) for both 2004 and 2005 (see Table 20). Notice 
that there are no average scores of 3 in this component. That is, overall, participants were 
using these modules less than the fundamental and health program modules. 
Although the change Table 20: Participant Utilization of Impact Modules, 2005 
Impact Modules: N=18 INSP 
Socio-economic models 2.8 
Models for panel data, double difference models 2.8 
Framework for empirical models 2.5 
Estimation of impact of focal programs 2.2 
Introduction to Propensity Score Matching 2.0 
Use of data for program and policy changes 2.0 
Multilevel models 1.9 
Explicative endogenous models 1.9 
Models for cohort aggregation 1.8 
Models for multiple category events 1.6 
Models for continuous events 1.5 
Models for binary events 1.5 
Problems in impact evaluation: endogeneity 1.5 
Total number of modules 13 
Average 2.0 
was minimal, utilization 
rose in only five 
categories from 2004 to 
2005 and dropped in 
five others. 
Reasons for low 
utilization in all impact 
modules ranged 
between respondents. 
One participant stated, 
“I learned technical 
skills and tools that 
maybe I will use in the 
future, but the work [I 
am doing now] is not 
related to the technical, 
statistical side of 
evaluation.” 
Other participants 
commented that these are very specific modules and they don’t conduct these complex types 
of evaluations in their workplace very often, but would like to design and implement them, if 
given the opportunity. 
Practicum modules. The group work was the only common practicum module used by all 
three institutions. Most participants rated it relatively highly. Some participants have 
suggested that more time be allocated to group exercises because they consider them to be the 
most important aspect of the workshop. One of the examples given of utilization was the 
M&E plan for Togo. A multi-country group of participants developed a plan during the 
workshop and that is now being finalized in Togo for implementation. Participants within this 
group have maintained contact after the workshop, continue to provide input and will be 
invited to Togo for the official inauguration. Another respondent summarized the views of 
many participants: 
“The workshop helped me a lot. It gave us the ability to work with groups, and to see clearly 
what happens in the field.”
Epi Info and STATA was formal sessions in CESAG and INSP. SPSS was optional on the 
first Saturday. These modules are the least frequently used by participants. The most 
common explanation is that these programs are not used in their organization or using these 
programs is not within their sphere of work. However, there is evidence that this module is 
useful and used by some and that it can be transformational. One participant mentioned that 
he introduced Epi-Info after returning from the training. An INSP participant had a different 
comment: 
H
MEASURE Evaluation 
ME Final.doc 24 12/30/2005 
“I think most of us have a basic command of STATA but are still not at the point where we can 
interpret it without help. It would be nice to do more technical work to gain more actual 
experience. Examples, 
Table 21: Participant Utilization of Practicum Modules 
(Highest ratings shaded) 
Practicum modules CESAG IPSR INSP 
N=26 N=8 N=18 
Field/site visits 2.9 2.6 
Group work on M&E plans 2.7 3.0 3.0 
Computer labs 1.8 
SPSS/Epi Info/STATA 2.4 1.9 
Impact evaluation in Paraguay 1.3 
Total number of modules 3 2 4 
Subtotal 2.6 2.7 1.9
samples, and practice of 
skills are missing in the 
workshop. The opportunity 
to APPLY what we learned 
is missing, and yet is a very 
important component.” 
INSP did not have a 
fieldwork session. CESAG 
and IPSR participants 
reported that their field 
work exercises provided the 
most valuable and often 
used knowledge and skills This module enabled the participants to synthesize the classroom 
material and apply it to a real world situation. 
E. Facilitating Factors and Constraints 
A number of external factors affect the success of the workshop. The following is a summary 
of those that seem to be most important. 
1. Facilitating Factors 
Reputation and Experience: The reputation and experience of all three institutions are very 
important factors. They all have institutional experience in health and training. They are all 
regional institutions. They are all very experienced in designing and delivering short-term 
workshops. They have all attracted capable and qualified faculty, instructors and participants. 
Demand: The high demand for M&E training in each region gives these institutions a strong 
market position. They have all demonstrated that this demand is greater than their current 
capacity to meet it. 
Linkages with national government and regional and international agencies. All three 
institutions have strong linkages to their national governmental agencies as well as and other 
regional/international agencies. This increases their opportunities for M&E technical work 
and for expanding the M&E workshop. All have benefited positively from their affiliation 
with MEASURE Evaluation. 
Leadership: CESAG has lost its long-time coordinator and instructor for the M&E workshop 
to another opportunity. IPSR has a new Director who is out of the country on another 
assignment until March 2006. The transition to new leadership will present opportunities as 
well as challenges over the next 6-12 months. INSP has strong leadership that is very 
supportive of the M&E workshop. 
2. Constraints 
Limited Donor Funding for PHN: The shift of focus from PHN programs and services to 
HIV/AIDS priorities, and the subsequent loss of funding in PHN, affects the demand for PHN￾specific M&E training and thus the availability of resources to support training opportunities 
for those who may need it most. This is especially serious in the LAC region. However, this 
also opens a new opportunity for the institutes to offer HIV/AIDS workshops in M&E, which 
all three are doing. 
MEASURE Evaluation 
ME Final.doc 25 12/30/2005 
Experienced Personnel. All three organizations can call on competent faculty and guest 
trainers. However, there is a gap in practical experience in M&E. Filling that gap with local 
personnel is a challenge, especially for INSP and IPSR. 
Language and Culture. CESAG and INSP have a large advantage over IPSR with respect to 
a common language (French and Spanish, respectively) and cultures that have many 
similarities. IPSR has neither of these in the large ANE regional area. 
IV. CONCLUSIONS
A. The Partnerships 
1. Partnership. The partnerships between M/E and the three regional training 
institutions have been a productive and professional relationship. All parties would 
like it to continue. 
2. Vision. All three institutions and MEASURE share a common vision in terms of 
short-term capacity. All expect that the institutions will be able to continue to offer the 
M&E workshop in PHN before the M/E project comes to a close. Long-term, IPSR 
and INSP want to become the M&E leader in their regions, both in training and in 
evaluation studies. CESAG has not been considering such an expansion. 
3. Agreement. The formal agreements between the three partners and MEASURE have 
been focused on the dual objectives of capacity and utilization. All parties have done 
their best to carry out the activities agreed upon. 
4. Interventions. MEASURE has provided a great deal of technical assistance to help 
each institution to develop “sustainable capacity.” Among the most important and 
effective of these interventions have been: 1) TOTs; 2) TA in curriculum and module 
development; 3) promotion of the workshops; 4) the provision of experienced trainers 
for selected sessions; and 5) strengthening adult learner-centered approaches to training 
of faculty. 
5. Workshops. The partners and M/E have delivered five M&E workshops on PHN in 
2004 and 2005. A sixth was planned for the November 2005. A total of 112 
participants from 38 countries were trained in the first five workshops. Another 25 
were to be trained in the sixth workshop. All three institutions are planning to hold 
workshops in 2006. 
6. Curricula. The “standard” course is not yet standardized. The recent workshop in 
Ethiopia (August, 2005) is supposed to reflect the new standard course. It allocates 
most mornings to presentations and most afternoons to case studies, exercises and 
group work. Apparently, this curriculum will be used in the 2006 workshops in 
Thailand and Senegal. Whether this will be applied in Mexico is unclear. The demand 
for “impact” evaluation is strong in Mexico but that leaves the region without a 
standard course. It also raises a question about the overall M&E training strategy. 
7. Participants have been very positive in their assessments of the value of the 
workshops, their utility for them and their organizations, and the individual modules 
that make up the curriculum. All three training institutions and MEASURE have been 
responsive to participant feedback and have worked together to improve the content 
and delivery of each module. 
MEASURE Evaluation 
ME Final.doc 26 12/30/2005 
B. The Capacity Building Objective 
8. Capacity objective. All three institutions are on their way to achieving USAID’s 
capacity objective. All three could, if necessary, carry out the workshop now with no 
assistance from M/E. However, there are a number of capacity functions and training 
modules that they still needs help with. Over the next two years, with M/E assistance, 
all three should be able to address most, and probably all of those needs. 
9. Strengths and weaknesses. The data collected from documents, surveys and 
interviews in this assessment have identified both strengths and weaknesses in current 
capacity. The following table identifies areas of high (A), medium (B) and low (C) 
capacity. The Team believes that the institutions will not need further TA in the 57 
capacities listed in the A category (see Annex A, Table 26 and Table 27 for details). 
The organizations are already strong in these areas. Where each institution will need 
more TA to build capacity is in the 35 capacities listed in the B category (see Table 28) 
and especially in the 17 capacities in the C category (see Table 29). Although this is a 
large number of capacity elements and training modules, M/E and the three training 
institutions are already addressing many of them. The emphasis now should be on 
targeting those needs that are critical to independence. 
H H
H
H
Table 22: Summary of Institutional and Substantive Needs (priority needs shaded) 
 Institutional Substantive Total
Rating CESAG IPSR INSP CESAG IPSR INSP No. Percent
A. High capacity 11 8 12 13 5 8 57 52 
B. Medium capacity 5 6 4 6 4 10 35 32 
C. Low capacity 1 2 1 6 7 17 16 
Total elements/modules 17 16 17 19 15 25 109 100 
10. Moving to Level 4. Both IPSR and INSP want to become M&E centers of excellence 
in their regions. Both want to undertake evaluations as well as train people in M&E. 
Although this is commendable in the abstract, it will take considerable effort and 
resources to move to level 4. All or most of the modules will need to be raised to level 
4. IPSR and INSP will need to develop the in-house capacity to design, update and 
train trainers in a broader M&E curriculum. Right now this is done largely by M/E. In 
addition, they will need to develop the capacity to provide TA to other organizations in 
the region in planning, developing and delivering M&E courses (not just one course). 
They will also need to acquire the knowledge, skills and experience to develop and 
upgrade monitoring systems and to carry out evaluation studies in a variety of health 
programs and in a variety of countries in the region. 
C. The Utilization Objective 
11. The objective. The limited feedback from participants makes it difficult to generalize 
about utilization. Nevertheless, secondary data sources and interviews indicate that 
most, if not all, of the participants learned something new in the workshops that they 
have applied in their work. It appears that this objective has been achieved for many 
participants. 
12. Utilization. Almost all of the respondents continue to work in M&E, gained a deeper 
understanding of M&E, were energized by their learning and applied at least some of 
what they learned to improve M&E in their workplaces. The most important modules 
MEASURE Evaluation 
ME Final.doc 27 12/30/2005 
that they applied were those dealing with the fundamentals, in particular, the 
development and utilization of evaluation plans. Limitations on utilization seem to be 
job specific. For one reason or another some participants have not had the opportunity 
to apply what they learned, be it a concept or a statistical model. 
13. New knowledge. Some participants came to the workshop with prior knowledge and 
experience in M&E. For them, some of the sessions did not provide them with new 
concepts or techniques. What many learned, however, was how to apply these 
concepts and techniques to practical M&E problems. Many also learned from one 
another in the group work sessions. Some worked on problems in the workshops that 
they later applied back home. 
14. Follow-up is a significant gap in the program and one that is difficult to address. 
Without it, nobody knows whether these workshops are making a difference or not. 
Efforts to get feedback and to establish networks have not been very successful. It 
seems that the chances for developing a network of M&E participants depend on such 
factors as the number of participants from a given country and their proximity to one 
another. Local networks of participants living and working in the same country and 
area may be more likely to spring up than networks of individuals living and working 
in different areas and countries. At this point, neither M/E nor the institutions have 
developed an affordable and effective procedure for follow-up. 
D. The Intangible Benefits 
15. Synergy. The intangible outcomes of these workshops, while not measured directly, 
are apparent from interviews with participants and faculty alike. The “Hawthorne 
Effect” is clearly present. The mere fact of bringing like-minded people together for 
three weeks has had a significant effect on learning and motivation. Some lifelong 
friendships have resulted from these workshops. Some support systems have been 
developed. Some “radical” ideas have been introduced and internalized. Commitment 
to M&E has increased among participants and faculty alike. These “synergistic” 
outcomes should not be overlooked in any evaluation. 
E. The Methodology 
16. Ratings. The capacity assessment tools used in this evaluation lend themselves to 
periodic assessments of both functional and substantive capacity. Although admittedly 
limited in terms of the number of respondents polled for this evaluation, the 
instruments themselves appear to be extremely valuable, easy to administer and useful 
for both snapshots of current capacity (as in Senegal) and trends over time (as in 
Mexico). Regardless of the validity of these subjective assessments, they provide 
fodder for serious discussions as to where the institutions are right now and what will 
be needed to take them to the point where they can operate independently. 
V. RECOMMENDATIONS
A. Recommendations for MEASURE Evaluation 
1. Institutional Capacity Building should continue, but it should be more focused. M/E 
should continue with its capacity assessment in CESAG and its plans to conduct one in 
IPSR. It would be useful to do the same in INSP. These assessments should be 
broadened to include the training modules, as in this evaluation. The assessments will 
enable M/E and the institutions to focus on the greatest needs, develop appropriate 
strategies, and use the remaining time to implement appropriate interventions. M/E 
MEASURE Evaluation 
ME Final.doc 28 12/30/2005 
should then conduct focused TA and monitoring activities to ensure that crucial 
activities are undertaken in a timely way, and that they are producing the expected 
outcomes. 
2. Training capacity. M/E should continue to work with the institutions to help them 
develop the capacity needed to carry out the training modules now classified as B and 
C. This could require a large amount of TA. Thus, it would be prudent to develop 
some alternatives. Four that were raised are: 1) increase the amount of TA provided by 
M/E to build capacity; 2) introduce team teaching in 2006 with the understanding that 
the institutions will deliver these modules on their own in 2007; 3) focus on the 
modules for which capacity can realistically be developed in the next year or two and 
summarize the remaining modules in an overview session; and 4) drop those modules 
for which capacity cannot be developed within the next two years. 
3. Financial support is a critical issue for INSP, but it is also important for IPSR and 
CESAG, as well. M/E needs to help these institutions identify funding sources for 
regional fellowships and honoraria for instructors. 
4. Level 4. Although the idea to become a regional center of M&E expertise is up to the 
individual institutions, M/E could help each institution examine its options. M/E could 
provide technical assistance to conduct feasibility studies that would identify what it 
would take for each institution to become an M&E center of excellence. 
5. Follow up. In anticipation of this topic being at the center of attention in the capacity 
assessment process, perhaps M/E could expand its literature search. The report, 
“Measuring the Stars: A consultancy for M/E Evaluation, Spring 2005,” presents a 
brief literature review and addresses the opportunities for strengthening follow-up in 
M/E programs. Many of these recommendations would apply to the M&E workshop. 
Follow-on interaction should probably be built into the curriculum to ensure that 
meaningful contact can be maintained over a 1-2 year period. Other options worth 
exploring are: establishing alumni networks and newsletters; offer special rates for 
other trainings/workshops that could be paid from membership dues or subscription 
fees; set target dates for follow-up evaluations at 3, 6 and 12 months; identify 
“captains” during the workshop who will take responsibility for following up 10 
participants each. 
6. Curricula. There does not seem to be a clear link between the basic and impact 
courses. M/E should take a hard look at the overall strategy for the workshop. Should 
it be a one-shot exposure to M&E or should the basic course be the introduction to a 
continuing education series on M&E? Could other options be explored, such as 
distance learning on a range of evaluation topics and methods, such as rapid household 
surveys, cost-effectiveness analysis, formative evaluation, behavioral surveillance, 
randomized trials, evaluating communication programs, measurement, and so on? 
Such an approach might address the follow-up problem, as well, assuming that 
participants of the standard course would like to continue learning about M&E and 
would, therefore, have an incentive to stay in touch with their training institution. 
B. Recommendations for INSP 
7. Leadership and vision will be crucial over the next 6-12 months. Each institution will 
need to determine the scope and level of its standard M&E workshop as well as where 
the institution wants to go over the next decade. Should it strive to become an M&E 
regional leader, both in carrying out evaluations as well as training people in M&E? 
MEASURE Evaluation 
ME Final.doc 29 12/30/2005 
There is a market in each country for local M&E workshops for such programs as 
HIV/AIDS, tuberculosis, malaria and drug logistics. Should there be a graduated series 
of evaluation courses, as INSP is setting up? These, and other vision questions, will 
need to be addressed and examined. 
8. Curriculum. This evaluation did not look at the curriculum. Given the ancillary 
findings on utilization (and the large amount of TA that will be needed to develop each 
institution’s capacity to implement that curriculum) the institutions themselves should 
take a hard look at its curriculum. What are the M&E needs of the participants’ 
organizations? What knowledge and skills do the participants need in order to help 
their organizations reach their M&E objectives? Which modules are most useful (and 
used) to that end and what kind of continued interaction should be built into the 
curriculum to ensure continuity? 
9. Human resources are a major concern in all of the institutions. The vision for the 
future should guide the development of the human resources necessary to achieve that 
vision. How to give faculty experience in applying M&E is a critical question. 
Identifying and recruiting qualified local, guest instructors is another issue. 
10. Capacity development is not up to M/E alone. Each institution needs to complete a 
comprehensive capacity assessment (as in Senegal) and then use those results to set 
priorities for development of institutional and substantive capacity. 
11. Financial sustainability is a local issue, as well. All three institutions will need to 
determine how to pay for those elements of the workshops that M/E is currently 
supporting. This includes expert technical expertise, up-to-date materials, and 
advertising/promotion of the workshops, and fellowships. 
C. Recommendations for USAID 
12. USAID support of M/E and the three training institutions has been critical and useful. 
Of immediate concern to INSP and IPSR, in particular, is future support for 
fellowships. Although other donors support most participants, the fellowships have 
given the institutions a mechanism for supporting individuals from countries that do 
not have such support. 
13. Follow-on to M/E. Does USAID plan to continue to support this component of M/E 
throughout the life of the current project and beyond? Should M/E identify and 
support more training institutions and/or limit its assistance to X number of years so as 
to promote early graduation and the shifting of M/E resources to new sites? 
14. Curricula. As noted above, the current workshop may be better seen as an 
introduction to M&E that is followed by a series of graduated or specialized courses 
that could be provided in a variety of ways. Is USAID interested in exploring options 
like these? 
15. Centers of excellence. Two of the three institutions are interested in expanding their 
capacity to provide more M&E services in their regions. The third may be open to the 
idea. USAID needs to determine how far it wants to go in supporting regional capacity 
building in M&E. Is it enough to have an annual workshop or should the institutions 
do more? 
16. USAID priorities for M&E. It would help to know what USAID’s overall priorities 
MEASURE Evaluation 
ME Final.doc 30 12/30/2005 
are for M&E. Should there be some coordination between M/E and the “revitalization” 
effort? Are primary targets USAID staffs, host country staffs, academicians, 
cooperating agency staffs? 
MEASURE Evaluation 
ME Final.doc 31 12/30/2005 
ANNEXES
A. Tables 
Table 23: CESAG Agenda 
Table 24: IPSR Agenda 
Table 25: INSP Agenda 
Table 26: High Organizational Capacity Components, No TA Needed 
Table 27: High Substantive Capacity Components, No TA Needed 
Table 28: Technical Assistance Needs in Institutional Capacity Building 
Table 29: Technical Assistance Needs in Substantive Capacity Building 
Table 33: Criteria for selection of M/E local partner institutions. 
Table 34: Partner Institutions accepted/rejected for M/E II, by major reasons M/E gave for the 
selection/rejection 
H
H
H
H
H
H
H
H
H
MEASURE Evaluation 
ME Final.doc 32 12/30/2005 
Table 23: Agenda for the CESAG 2005 PHN Regional Workshop on Monitoring and Evaluation
Monday Tuesday Wednesday Thursday Friday Saturday 
8:30-10:30 8:30-10:30 8:30-10:30 8:30-10:30 8:30-10:30 8:30-10:30 
Welcome and Introductions 10:45-12:30 M&E Indicators M&E Info Sources Qualitative methods 10:45-12:30 
10:45-12:30 Intro to M&E Frameworks 10:45-12:30 10:45-12:30 10:45-12:30 Monitoring 
Intro to M&E Theory (Amani) M&E Indicators: Exercise Info Sources: Exercise Qualitative methods: Projects 
(Amani) (Amani) (Mounkaila) role play (Seck) Group work 
Week 
1 
(Seck/Maty)
14:00-15:30 14:00-15:30 14:00-15:30 14:00-15:30 14:00-15:30 Social 
Intro to M&E Planning 15:45-17:30 15:45-17:30 15:45-17:30 15:45-17:30 program
15:45-17:30 M&E Framework: M&E Indicators: M&E Info Sources: Monitoring projects: 
Introduction to group work Case study Group work case study Exercise, Group work 
(Amani) (Amani) (Amani) (Mounkaila/Maty) (Seck/Maty)
8:30-10:30 8:30-10:30 8:30-10:30 8:30-10:30 8:30-10:30 Free 
10:45-12:30 Impact evaluation Child health/IMCI Reproductive Health Logistics management 
Role of information & (Mounkaila) 10:45-12:30 10:45-12:30 10:45-12:30 
decision making 10:45-12:30 Child health/IMCI: Case Reproductive Health Logistics management: 
(Moreland/Gnassou) Quantitative methods study/group work Case study/group Case study/group work 
Week (Mane/Maty) (Seck/Maty) work (Ibnou/Aissatou)
2 (Lago)
14:00-15:30 14:00-15:30 14:00-15:30 14:00-15:30 14:00-15:30 
15:45-17:30 15:45-17:30 Child vaccination/RED Malaria HIV/AIDS/TB 
Role of information & Quantitative methods 15:45-17:30 15:45-17:30 15:45-17:30 Free 
decision making (Mane/Maty) Case study/group work TB: Case study/group Case study/group work 
(Moreland/Gnassou) 18:00-19:30: Epi Info (Seck/Maty) (Maunkaila/Maty) (Gnassou/Aissatoi) 
18:00-19:30: Epi Info 18:00-19:30: Epi Info/ 18:00-19:30: Epi Info/ 
(Mane/Thiam) STATA STATA (Mane/Thiam)
(Mane/Thiam)
8:30-10:30 8:30-10:30 8:30-10:30 8:30-10:30 8:30-10:30 
10:45-12:30Efficiency M&E plan development 10:45-12:30 10:45-12:30 10:45-12:30 
evaluation (Amani/Gnassou) Field work Analysis of field work Presentation of M&E 
(Amani/Mane) 10:45-12:30 (Amani/Seck/Mane/Maty) (Mane/Thiam) Plans (Amani) 
Week Field work preparation 
3 (Amani/Seck/Mane/Maty) 
14:00-15:30 14:00-15:30 14:00-15:30 14:00-15:30 14:00-15:30 
Behavior change 15:45-17:30 15:45-17:30 Analysis of field work Lessons Learned 
communication Field work preparation Field work 15:45-17:30 Course evaluation 
15:45-17:30 (Amani/Seck/Mane/Maty) (Amani/Seck/Mane/Maty) Finalization of field 15:45-17:30 
Group work 18:00-19:30: Data from 18:00-19:30: Data from work (Mane/Thiam) (Amani/Seck) 
(Gnassou/Kone) Internet (Faculty) Internet (Faculty) Closing ceremony 
MEASURE Evaluation 
ME Final.doc 33 12/30/2005 
Table 24: Agenda for the IPSR 2004 PHN Regional Workshop on Monitoring and Evaluation
MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY 
9:00-10:30 
Welcome and Introductions 
11:00-12:00 
1) Overview 
Bates Buckner
9:00-12:00 
Frameworks, Part II 
Uraiwan
9:00-12:00 
Indicators, Part I 
Bates Buckner 
9:00-12:00 
Data Sources, Part I 
Shelah Bloom 
9:00-12:00 
Calculating and 
interpreting coverage 
indicators 
Shelah Bloom 
9-12 
SPSS 
WEEK 
1 
1:30-3:30 
Frameworks, Part I 
Bates Buckner 
4:00-5:30 
Introduction to group work 
1:30-3:30 
Developing an M&E Plan 
Shelah Bloom 
4:00-5:30 
Group work on M&E Plans 
1:30-3:30 
Indicators, Part II 
Uraiwan 
4:00-5:30 
Group work on M&E Plans 
1:30-3:30 
Data Sources, Part II 
Bates Buckner 
4:00-5:30 
Group work on M&E 
Plans 
1:30 -4:30 
Group work on M&E 
Plans 
1-4 
SPSS 
9:00-12:00 
Impact Evaluation 
Bates Buckner 
9:00-11:30 
M&E of Maternal Health 
Programs 
(MOPH) 
Free 
WEEK 
2 
1:30-3:00 
Preparation for field visits 
3:30-5:00 
Group work on M&E Plans 
1:30-4:00 
M&E of Child health 
Programs (MOPH) 
4:30 – 5:30 
Group work on M&E Plans 
FIELD TRIP 
Day 1 
9.00-11.00 Ministry of 
Public Health (to hear about 
HMIS) 
Evening meeting to debrief 
on HMIS visit 
FIELD TRIP 
Day 2 
Evening meeting to 
debrief on site visits 
FIELD TRIP 
Day 3 
Lunch meeting to 
debrief on morning 
site visit 
No evening meeting 
Free 
9:00-12:00 
M&E of Family Planning 
Programs 
(Pop Council) 
9:00-12:00 
M&E of STD/HIV/AIDS 
Programs 
(FHI) 
9:00-12:00 
Qualitative Methods in 
M&E 
(IPSR) 
9:00-12:00 
Communicating to 
Policy Makers 
(IPSR) 
9:00-12:00 
Team presentations of 
M&E plans 
WEEK 
3 
1:30-4:00 
M&E of Nutrition Programs 
(Institute of Nutrition, 
Mahidol University) 
4:30 – 5:30 
Group work on M&E Plans 
1:30-4:30 
Group work on M&E Plans 
1:30 – 4:30 
Sentinel & Behavioral 
Surveillance 
(MOPH) 
4:30 – 5:30 
Group work on M&E Plans 
1:30-5:30 
Preparation for team 
presentations 
Closing 
MEASURE Evaluation 
ME Final.doc 34 12/30/2005 
Table 25: Agenda for the INSP 2005 PHN Regional Workshop on Impact Evaluation
Monday Tuesday Wednesday Thursday Friday Saturday 
9:00-9:45 
Inauguration 
9:45-11:00 
Importance of Evaluation 
(Sesma)
9:00-10:45 
Concepts, Frameworks 
11:00-12:45 
Framework examples 
(Urquieta)
9:00-10:45 
Indicators 
11:00-12:45 
Construction of indicators 
(Torres) 
9:00-10:45: Evaluation 
designs: true & quasi 
11:00-12:45 
Observation, reliability, 
validity, bias (Cruz) 
9:00-10:45 
Preparation of evaluation plans 
11:00-12:45 
(B. Plaza) 
9 :30-12 :00 
Group work 
Week 
1 
14:00-15:45 
Workshop themes & 
logistics (Urquieta, 
Reynales, T. Tellez) 
16:00-17:45 
Introduction to STATA 
(Reynales) 
14:00-15:45 
Framework: practicum 
(Urquieta) 
16:00-17:45: Final project: 
introduction (Urquieta, 
Reynales), Introduction to 
STATA (Reynales) 
14:00-15:45 
Info systems, data sources 
(Sosa) 
16:00-17:45 
Introduction to STATA 
(Reynales) 
14:00-15:45 
Qualitative methods 
(Allen) 
16:00-17:45 
Final project themes 
14:00-15:45 
Importance of data collection 
(Castro) 
16:00-17:45 
Opportunities Project Lessons 
Learned 
(B. Hernandez)) 
9:00-10:00: Review 
10:00-10:45 
11:00-12:45 
Framework for empirical 
models (Angeles) 
9:00-10:00: Review 
10:00-10:45 
Models for binary events 
11:00-12:45 (M. Tellez, 
La Madrid) 
9:00-10:00: Review 
10:00-10:45 
11:00-12:45 
Models for multiple 
categories (Angeles) 
9:00-10:00: Review 
10:00-10:45 
11:00-12:45 
Multilevel models 
(Angeles) 
9:00-10:00: Review 
10:00-10:45: Problems in impact 
evaluation: endogenicity 
11:00-12:45: Models with 
explicit endogenous variables 
(Urquieto, Angeles) 
Computer 
lab open 
Week 
2 14:00-15:45 
Models for continuous 
events (M. Tellez) 
16:00-17:45 
STATA exercises (M. 
Tellez) 
14:00-15:45 
STATA exercises 
(M. Tellez, La Madrid) 
16:00-17:45 
Group work: analysis plan 
13:00-14:45 
Presentation (Neufeld) 
15:00-17:00 
STATA exercises 
(Angeles, Urquieta, 
Reynales) 
14:00-15:45 
Measuring SES levels 
16:00-17:45 
Group work 
14:00-15:45 
STATA practicum 
Angeles, Urquieta) 
16:00-17:45: Impact evaluation 
in Paraguay (Angeles) 
9:00-10:00: Review 
10:00-10:45: Problems in 
impact evaluation: 
Endogenicity 
11:00-12:45: STATA 
practicum (Angeles, 
Urquieta) 
9:00-10:00: Review 
10:00-10:45 
11:00-12:45 
Panel data, double 
difference models 
(Angeles) 
9:00-10:00: Review 
10:00-10:45 
11:00-12:45 
Propensity Score 
Matching 
(Urquieta, Jimenez, 
Angeles) 
9:00-10:00: Review 
10:00-10:45 
11:00-12:45 
Use of data in 
programming and 
policy change 
(Morrison) 
9:00--10:45 
11:00-12:45 
Presentation of final projects 
Week 
3 
14:00-15:45: Impact of 
focal programs (Angeles) 
16:00-17:45 
Group work: final project 
14:00-15:45 
Models for cohort 
aggregation 
16:00-17:45 
Group work: final project 
14:00-15:45 
16:00-17:45 
Group work: final project 
14:00-15:45 
16:00-17:45 
Preparation of final 
presentation 
14:00-17:15 
Free, Evaluation, Closing 
 MEASURE Evaluation
ME Final.doc 35 12/30/2005 
Table 26: High Organizational Capacity Components, No TA Needed 
Organizational Capacity CESAG IPSR INSP 
Strategic leadership A A 
Organizational processes A A A 
Infrastructure A A A 
Human resources 
Program management A A A 
Financial management A 
Inter-institutional linkages A A 
Marketing A A A 
Course logistics A A A 
Training content A A 
Course implementation A A 
Mission A
Culture A A A 
Incentives/Rewards A A A 
Total number of modules 11 8 12 
Table 27: High Substantive Capacity Components, No TA Needed 
Substantive Training Capacity CESAG IPSR INSP
The importance of evaluation in health sector A 
Fundamental concepts; frameworks A A 
Developing an M&E plan A 
Indicators A
Quantitative methods and data analysis A 
Observational designs, reliability, validity A 
Experimental and quasi-experimental designs A 
Qualitative methods in M&E A A A 
Sentinel & behavioral surveillance A 
Communicating to policy makers A 
M&E of family planning programs A 
M&E of STD/HIV/AIDS programs A 
M&E for child health programs A 
M&E of vaccination programs (RED) A 
M&E of reproductive health programs A 
M&E of malaria programs A 
M&E for drug logistics management A 
Field work A 
Group work on M&E plans A 
STATA: introduction and exercises A 
EPI Info/STATA A A 
Computer laboratory: STATA A 
Total number of modules 13 5 8 
 MEASURE Evaluation
Table 28: Technical Assistance Needs in Institutional Capacity Building 
Organizational Capacity CESAG IPSR INSP 
Inter-institutional linkages B 
Strategic leadership B 
Training content B 
Training planning & sustainability B B B 
Course implementation B B 
Mission B B
Financial management B B 
Infrastructure 
Technical capacity in M&E B 
Marketing 
Trainee follow-up C C C 
Human resources B C B
Total TA needs 5B, 1C 6B, 2C 4B, 1C
Table 29: Technical Assistance Needs in Substantive Capacity Building 
Training Modules CESAG IPSR INSP 
Developing an M&E plan C B 
Frameworks C 
Indicators B B
Calculating & interpreting coverage indicators C 
Data sources B C B 
Impact evaluation B C 
Information for decision-making/Use of data B B 
Evaluation of project efficiency B 
Behavior change communication B 
M&E for maternal health programs B 
M&E of child health programs B 
M&E for STI/AIDS & TB B 
M&E of nutrition programs C 
Framework for empirical models B 
Models for continuous events B 
Models for binary events B 
Models for multiple category events B 
Introduction to Propensity Score Matching B 
Multilevel models C 
Problems in impact evaluation: endogeneity C 
Explicative endogenous models C 
Models for panel data, double difference models C 
Estimation of impact of focal programs C 
Models for cohort aggregation C 
Socio-economic models C 
Field/site visits: MOPH HMIS B 
Impact evaluation in Paraguay B 
Total TA needs 6 B 4B, 6C 10B, 7C 
ME Final.doc 36 12/30/2005 
 MEASURE Evaluation
ME Final.doc 37 12/30/2005 
B. Methodology Issues 
The evaluation was limited to the workshops and participants carried out by the three training 
institutions between January 2004 and November 2005. Thus, it did not include the IPSR workshop 
in November 2005 or the Ethiopia workshop in August 2005. If it had, the findings would probably 
have been even more favorable. That is because the IPSR workshop was improving. M/E had 
helped revise the curriculum, and training methods had been improved. The Ethiopia workshop 
was based on a new curriculum that is expected to be the standard for all basic workshops. 
1. Documents and Secondary Data 
M/E had known for quite awhile that this evaluation was scheduled. In preparation, M/E staff 
conducted a participant feedback survey, assembled data on the workshops and participants and 
alerted their partners about the evaluation. All of this was greatly appreciated by the Evaluation 
Team as it made our job much easier. M/E provided, for example, all of the participant evaluations, 
all of the reports on workshop schedules, all of the information on capacity-building interventions, 
and copies of all activity and trip reports. In addition to workshop-related material, M/E provided 
us with background documents on their approach to evaluation and to capacity building. 
2. Institutional Capacity Assessment 
Perhaps the most important M/E contribution was the data and instruments used in the CESAG 
stakeholder capacity assessment, which took place in May, 2005. We decided to use a short version 
of that instrument, which M/E had already prepared and tested. That instrument is shown in Table 
30. We decided to identify key informants at each institution and ask them to fill out the assessment 
form. We believed that this would be a simple way to get partner representatives (directors, 
coordinators and faculty) to do a rapid assessment of their institution’s capacity to carry out the 
workshops. We would then follow up with personal interviews to get clarifications and 
explanations for the ratings. We thought of this as a structured key informant survey. We learned 
early on that there were very few individuals at each institution who had a broad understanding of 
the workshops. So this approach would enable us to: 1) standardize our interview questions; and 2) 
quantify the responses. 
M/E had already administered the long version of the instrument as part of a broad capacity 
assessment exercise for CESAG. Eleven stakeholders filled out these instruments and then 
discussed the results. What was impressive was that the instruments actually identified the same 
strengths and weaknesses of CESAG that M/E had identified privately. We concluded, therefore, 
that it would be best to use the M/E instrument (short version) rather than to construct our own 
instrument. After all, the 17 elements listed had face validity, had been tested, and had identified 
key strengths (such as infrastructure) and weaknesses (such as trainee follow-up) that agreed with 
M/E views. We also believed that by using the same instrument, M/E and the institutions could 
repeat the assessment periodically and spot improvements – and problems that needed attention. 
M/E helped us identify the key informants at each site. We identified a few others when we visited 
the sites. We asked M/E to send a letter (email) to the prospective informants advising them as to 
what we were planning and that M/E supported it. After a week we emailed the instrument to each 
informant (in English or Spanish as appropriate – we had decided to use the stakeholder data for 
CESAG, which meant that we did not need to send this instrument out to CESAG informants). 
Responses were slow in coming, but we made up for that during the site visits. For example, if key 
H
 MEASURE Evaluation
ME Final.doc 38 12/30/2005 
informants had not sent in their forms, when we met with them we began by asking them to fill out 
the instrument first, then we followed up with questions. We learned that the informants could fill 
out the form in 10 minutes or less. We also learned that the responses tended to cluster, so that we 
did not need to ask about each item. We could ask our clarifying questions about each cluster. 
In general, the respondents were very cooperative and we had no problems getting them to elaborate 
on their responses. We learned that it was important to explain that we were not looking for 
problems but for TA that they might need to become self-sufficient. 
The survey results were tabulated in a simple Excel file that computed averages for each response 
category. Those averages were then transferred into the Word document for each case study. We 
learned to wait until all of the responses were in before making the tabulations. Otherwise, when an 
additional response came in we would have to update the formulas in the spreadsheets, which took 
time. 
In our view, this procedure worked very well. We were able to get a lot of information from a small 
number of key informants in a very short period of time. Instead of a notebook full of qualitative 
responses we had an organized format that quantified each capacity element and allowed for 
descriptive information to explain each rating. To us, this seems to be an approach that would allow 
USAID to collect qualitative information and quantify it. This could only happen if the CA had 
developed a quantified monitoring system, which M/E had. 
It is important to point out that we did not challenge the M/E capacity assessment instrument. M/E￾JSI staff, in particular, had invested a good deal of effort in developing the instrument. We asked if 
it had been validated. They said no, it had not. However, it was developed based on a variety of 
capacity building frameworks, and we decided to accept it as is. Therefore, we cannot state that the 
instrument has been validated, but we can state that it is comprehensive and just the type of 
instrument that USAID needs to carry out short-term evaluations. We would recommend that 
USAID (and its CAs) incorporate this type of monitoring instrument in all of its projects. 
3. Substantive Capacity Assessment. 
The substantive assessment followed the same lines as described above. M/E did not have an 
instrument for this type of assessment, so the Evaluation Team developed one that followed the 
M/E format. The instrument (see Table 31) lists all of the training modules and asks the key 
informant to rate each one on the four-point scale described in the text. The basic idea is to 
determine where the training institution stands with respect to its ability to deliver each training 
module. Obviously, the instrument has to be tailored to fit the curriculum of each workshop, which 
these did. 
The key informants were asked to fill out these forms, mail them back to us, and agree to a follow￾up interview. Again, we had the same problems getting responses, but were able to get the 
information we needed during the site visits. The data were processed the same way, on an Excel 
spreadsheet. 
4. Participant Utilization Assessment. 
Participants from the 2004 and 2005 workshops were identified from M/E records. They were sent 
an introductory letter followed by an email asking them to fill out the attached form (see Table 32). 
This form was similar to the substantive capacity form but the scale focused on utilization. 
Response rates were expected to be low, based on M/E experience with its Follow-up Survey. 
H
H
 MEASURE Evaluation
ME Final.doc 39 12/30/2005 
Therefore, we assembled a crew of English, Spanish and French-speaking interviewers to telephone 
each of the 112 participants to: 1) ask them to fill out the form and submit it (now, on-line or via 
email); and 2) set a time for a follow-up interview. 
This was easily the most time-consuming and expensive survey. Responses were received from 52 
participants (46 percent) and follow-up interviews were completed with 33 (29 percent). It was 
extremely difficult to find, much less speak with the participants. The reasons were legion: busy 
circuit lines, no answer, no one at the number dialed, no response when connected, person who 
answered the phone hung up, no forwarding phone number, not listed, and so forth. The response 
rate was lowest for the Thai workshop, largely due to language, distance and time zone problems. 
The interviewer noted that she made 50 calls for each response. When she made a connection there 
were other problems: one person was in a bar, another had a screaming baby in her arms, one was in 
a meeting, and another was driving. The interviewer was only able to interview four of the 28 
participants who attended the Thai workshop. CESAG yielded 11 (of 53) and INSP 18 (of 31). 
M/E had collected extensive contact information for each participant, but it seems that many 
changed telephone/cell phone numbers, moved, got new jobs, or for other reasons were not 
reachable. Once contacted, most people were quite willing to respond. The major problem was 
contacting them in the first place. 
Nevertheless, the information provided by those who were contacted was very helpful. Whether it 
is representative or not is unknown. 
5. Instrument scales 
The instruments used different scales. Some were 10 point and others were 4 point. Not all of them 
conformed to the USAID standard. It would be important in the future to standardize the scales so 
that they are uniform and can be compared. 
6. Lessons Learned 
Despite the limitations, the methodology applied seems to have yielded useful and accurate 
information. The quantification of qualitative responses from key informants and participants could 
be very helpful for future USAID evaluations, especially those that rely on unstructured interviews 
and observations. Uniform and standardized scales are needed for this type of evaluation, however. 
USAID should consider requiring its CAs to develop and utilize such scales. This evaluation has 
shown that such scales are relatively accurate, can identify strengths and weaknesses, and can 
facilitate follow-on explanatory questions. 
MEASURE Evaluation
ME Final.doc 40 12/30/2005 
Table 30: Institutional Capacity Assessment Tool for Training Institutions 
Please fill out this form by checking the appropriate box for each capacity element. The scale is from 1 (low) to 10 (high, as described below.) 
Scale: 1-3 = Strongly Disagree; 4-5 = Disagree; 6-7 = Agree; 8-10 = Strongly Agree; NA = Not applicable/do not know 
1 2 3 4 5 6 7 8 9 10 NA 
Organizational Capacity 
1. Strategic Leadership 
There is strong institutional commitment to the importance of training in M&E skills 
2. Human Resources 
The number of training staff is sufficient and skills are adequate (trainers have adequate 
knowledge/understanding of M&E principles, tool application and effective training techniques) 
3. Financial Management 
Budgetary resources are sufficient to support training interventions 
4. Organizational Processes 
Organization sets and maintains high standards of quality in all training activities 
5. Program Management 
There is strong organizational/management support for M&E and M&E training 
6. Infrastructure 
Organization has sufficient infrastructure, including dependable telephones, electricity, 
computers/internet access, appropriate software, etc. 
7. Inter-institutional Linkages 
Organization successfully networks/partners with universities/NGOs/community groups/media and 
other stakeholders, as appropriate 
8. Technical Capacity: M&E 
Staff members have sufficient knowledge of M&E activities and tools, experience applying this 
knowledge, and opportunities to continue to strengthen this knowledge/experience. 
Technical Capacity: Training 
9. Training Content 
Workshop agenda is appropriate for the audience and content is logically organized and technically 
accurate 
10. Training Planning and sustainability 
Systems for sustainability of training interventions in the medium and long-term have been developed 
11. Training Course Logistics 
Logistical arrangements for participants, trainers, venue, site visits, etc. meet high standards 
MEASURE Evaluation
ME Final.doc 41 12/30/2005 
Capacity Assessment Tool for Training Institutions (continued) 
Please fill out this form by checking the appropriate box for each capacity element. The scale is from 1 (low) to 10 (high, as described below.) 
Scale: 1-3 = Strongly Disagree; 4-5 = Disagree; 6-7 = Agree; 8-10 = Strongly Agree; NA = Not applicable/do not know 
1 2 3 4 5 6 7 8 9 10 NA 
12. Marketing 
Training responds to a real demand, is promoted effectively and attracts appropriate applicants 
13.Training Course Implementation 
Objectives and performance goals of training are met for each training intervention 
14. Trainee Follow-up 
Mechanisms are used to support participants after training intervention (TA, mentoring, materials, 
networks, websites, newsletters, trainee databases, etc.) 
Organizational Motivation 
15. Mission 
Organization has a clear mission statement that is known and agreed to by staff 
16. Culture 
Organization has a culture that is open to change and committed to the capacity building process 
17. Incentives/Rewards 
Participating in the training intervention is recognized by peers as a worthwhile and prestigious 
activity 
Please email a copy of the completed form to: Melanie Kindsfather at the Synergy Project (MKindsfather@s-3.com). During the follow-up 
interview, we will use your responses to: 
1. Verify the general strengths and weaknesses of IPSR capacity to carry out the M&E workshop; 
2. Identify the reasons for these strengths and weakness; and 
3. Explore what assistance IPSR might need to strengthen capacity. H
 MEASURE Evaluation
ME Final.doc 42 12/30/2005 
Table 31: Partner Assessment of Substantive Capacity 
We are interested in your assessment of (IPSR, CESAG, INSP) capacity to implement the M&E 
course for PHN. By capacity we mean the ability to implement the course independently, without 
technical or financial assistance from MEASURE/Evaluation. 
USAID uses a four-point continuum to assess this capacity: 
1 = able to implement the course but with significant 12
F technical assistance (TA); 
2 = able to implement the course with limited13
F TA; 
3 = able to implement the course with no TA; and 
4 = able to help others to implement the course. 
For each training module, please check the appropriate box. 
MEASURE/Evaluation PHN M&E Training 
modules 
Significant 
TA=1 
Limited 
TA = 2 
No TA 
=3 
Help 
others =4 
NA/ 
DK 
Fundamentals: Core Modules 
1. Role of strategic information in decision making 
2. Frameworks 
3. Indicators 
4. Information systems 
5. Evaluation designs 
6. Developing and implementing an M&E plan 
7. Calculating & interpreting coverage indicators 
8. Facilitating use of strategic information 
Program modules for PHN 
9. Child health 
10. Nutrition 
11. Maternal health 
12. Reproductive health/family planning 
13. Malaria and tuberculosis 
14. HIV/AIDS 
Other activities 
15. Field/site visits 
16. Group work 
17. Case studies/activities 
Please email a copy of the completed form to: Melanie Kindsfather at the Synergy Project 
(MKindsfather@s-3.com). During the follow-up interview, we will use your responses to: 
• Verify the general strengths and weaknesses of the workshop modules; 
• Identify the reasons for these strengths and weakness; and 
• Explore what assistance you might need to move the modules up the continuum. 
H
 
12 “Significant” assistance means that you would not be able to carry out the module without TA from M/Eval. 
13 “Limited” assistance means that you could carry out the module, but it would be better to have M/Eval on hand in 
case questions arise that the faculty may not be able to answer. 
 MEASURE Evaluation
ME Final.doc 43 12/30/2005 
Table 32: Participant Utilization of M&E Modules 
We are interested in learning: 1) what new knowledge and skills you learned at the M&E workshop 
that you attended in (date); and 2) what you have applied or used since you completed the M&E 
workshop. 
For each training module, first please enter Yes or No in the “New Information, Skills” box. For 
those that you answered “Yes,” please check the appropriate box to tell us how often you have used 
the new knowledge or skills. 
MEASURE/Evaluation PHN M&E Training 
modules 
New 
info, 
Used 
often=4 
Used 
some￾Seldom 
used=2 
Never 
Used=1 
skills=0 times=3 
Fundamentals: Core Modules 
1. Role of strategic information in decision making 
2. Frameworks 
3. Indicators 
4. Information systems 
5. Evaluation designs 
6. Developing and implementing an M&E plan 
7. Calculating & interpreting coverage indicators 
8. Facilitating use of strategic information 
Program modules for PHN 
9. Child health 
10. Nutrition 
11. Maternal health 
12. Reproductive health/family planning 
13. Malaria and tuberculosis 
14. HIV/AIDS 
Other activities 
15. Field/site visits 
16. Group work 
17. Case studies/activities 
Please email a copy of the completed form to: Melanie Kindsfather at the Synergy Project 
(MKindsfather@s-3.com). During the follow-up interview with you, we will use your responses to: 
4. Verify what new knowledge and skills you learned at the workshop; 
5. Learn more about what you have applied, in your work or elsewhere 
6. Learn more about what you have not applied and why; and 
7. Ask for your suggestions about increasing the utility of the workshop and its modules. 
H
 MEASURE Evaluation
ME Final.doc 44 12/30/2005 
C. Selection of Partner Institutions, by Charles Kenade 
1. Introduction 
M/E conducts its workshop in a USAID-supported country, in partnership with local training 
institutions that is selected based on the institution’s potential to promote and sustain PHN M&E 
training in a region. In Phase I, M/E partnered with IPSR in Thailand, Makerere University (MUK) 
in Uganda, and University of Costa Rica (UCR) to cover the Asia, Africa and the /Latin 
America/Caribbean regions respectively. 
Table 33: Criteria for selection of M/E local partner institutions. 
ƒ Institution has experience in M&E of PHN programs, activities and research 
ƒ Institution has experience in M&E training 
ƒ Institution has identified M&E as a key area for development and expansion (internal 
motivation) 
ƒ At least 4 faculty members with experience/interest in M&E are willing/able to serve as 
workshop presenters, and one of the above faculty members must be willing/able to 
serve as the key academic coordinator 
ƒ Institution has experience sponsoring international workshops 
ƒ Institution is located where there is high demand/requests for training in M&E 
ƒ Institution/location has adequate infrastructure (classrooms, computers, hotels, easy air 
accessibility, etc.) 
ƒ Ideally, institution has a relationship with a school of public health and has a mandate 
for training and service in public health. 
ƒ Institution is eager to participate and partner with us 
ƒ In Africa, institution is preferably located in East Africa 
At the end of Phase I, based on lessons learned, M/E assessed its existing and potential partners 
against a list of criteria shown in the following table. 
Based on this assessment M/E selected the following partners for Phase II: 
ƒ Institute for Population and Social Research (IPSR) of Mahidol University (IPSR), in 
Bangkok, Thailand; 
ƒ Instituto Nacional de Salud Pública (INSP), Cuernavaca, México (INSP). 
ƒ The Department of Community Health and the Demographic Training and Research 
Centre (DCH/DTRC), Addis Ababa University, Ethiopia - for Anglophone Africa. 
ƒ Centre Africain d'Etudes Superieures en Gestion (CESAG) in Dakar, Senegal (CESAG) 
for Francophone Africa. 
Table 34 presents the major considerations M/E ma H de in selection/non-selection of each partner. 
ƒ
MEASURE Evaluation
ME Final.doc 45 12/30/2005 
Table 34: Partner Institutions accepted/rejected for M/E II, by major reasons M/E gave for the selection/rejection 
Institutions selected 
for Phase II 
Reasons for selection 
IPSR, Thailand • Continuation of a previous close collaboration (mostly on demographic studies) since 1970s with UNC. 
• In 1970 IPSR seemed a good fit. It broadened the health scope, lately diversifying into other PHN areas including 
HIV/AIDS. 
• M/E is concerned that IPSR assigned the workshop responsibility to a faculty member who seems to have limited 
interest and experience, instead of assigning this responsibility to available faculty members that have more 
enthusiasm, experience and are better qualified to run this workshop. However, IPSR has a new director who is 
expected to change and address M/E’s concerns. The partnership’s future may depend on whether this 
expectation comes true. 
INSP, Mexico • INCS is a public health institution, which had identified M&E as a key area for development and expansion 
(internal motivation) because of the high demand for M&E training. 
• INSP leadership was interested in research as a mission. 
• INSP has a very high reputation in the region and, therefore, is well positioned to have an impact in region. 
• The Mexican Government could pay for Mexican participants – this guaranteed minimum demand necessary to 
sustain the workshop. 
CESAG, Senegal • Since 1998 M/E has collaborated with CESAG, primarily to teach the M&E course. 
• The M&E program fits well into the institution’s mandate to train middle level development program managers 
and technical personnel. 
• CESAG considered M&E a key area for development and expansion (internal motivation). 
• CESAG had ties with a regional development bank that has an interest in M&E. The bank encouraged CESAG to 
offer M&E courses. 
• CESAG had a faculty member that had interest and ability to coordinate this course. This faculty member acted 
as a champion and greatly promoted this workshop. 
Addis Ababa 
University, 
Ethiopia 
• The Center is interested in institutionalizing M&E into the School of Health. 
• The Center provided staff members to work with M/E to run the workshop. 
• The staff members showed great interest and commitment in the workshop. 
MEASURE Evaluation
ME Final.doc 46 12/30/2005 
Institutions not Reasons for not being selected 
selected for Phase II. 
Makarere 
University, Uganda 
• Poor performance on the capacity building continuum. MUK did not show progress towards the expected 
outcome – the ability to provide the workshop without M/E support. 
• Incidences of mismanagement during the 2003 workshop. 
• Weak collaboration with the Institute of Public Health. 
• Lack of interest and acceptance of responsibility by workshop faculty. 
• Note. Makerere has independently run two PHN M&E workshops. One UNFPA-funded workshop in 2003 and 
one international workshop, which attracted nine international participants, in 2005. 
University of Costa 
Rica, Costa Rica 
•
•
Better partner (INCS, Mexico) in the region. 
UCR was not as well known as INCS with respect to public health training in the region. 
• UCR did not (and did not expect to) establish a relationship with the school of public health. 
• The UCR workshop had difficulty attracting a critical number of sponsored participants necessary to break even. 
• The UCR two-year Masters program would enroll only two cohorts during the M/E II project lifetime, i.e., once 
every other year. 
• Note: Since end of the partnership, Costa Rica has independently conducted two PHN M&E workshops. 
University of Ghana, 
Ghana 
• The University did not follow up on the initial M/E contact. 
MEASURE Evaluation 
ME Final.doc 47 12/30/2005 
2. Why partnerships with Makerere and Costa Rica universities were terminated 
Makerere University 
M/E collaborated with the Department of Population Studies, Institute of Statistics, Makerere 
University, mainly through the head of the department. Throughout the partnership with Makerere, 
M/E conducted 4 workshops. 
Makerere was dropped because the department was not progressing towards conducting the 
workshop with minimal support from M/E. In particular, participation in the workshop was limited 
to almost only one member of the staff, the head of the department, and the Department had failed 
to secure functional collaboration with the Institute of Public health, as M/E had required. In 
addition, M/E noted serious mismanagement of the workshop conducted in 2003. 
Costa Rica 
The collaboration with Costa Rica University began with the training of five faculty members at the 
Summer Institute of 1998 in Chapel Hill. These faculty members returned to Costa Rica and in 
collaboration with M/E successfully conducted four M&E workshops. 
Although promising, the partnership with Costa Rica was discontinued in phase II because M/E did 
not have enough resources to work with two partners in Latin America, and had found a more 
strategic institution, INSP in Mexico. INSP had many advantages and was selected over Costa Rica. 
The former is better known and highly respected in the region and, therefore, has more potential to 
impact on the region. INSP has better sustainability prospects because it enjoys support and a ready 
market from the Mexico Ministry of Health. Moreover, INSP had internalized the idea of having 
the M&E workshop as a viable venture and, therefore, as an integral part of the institution’s strategy 
to become a premier public health institution in the region. 
Post partnership activities. 
Since the end of the partnerships, both Costa Rica and Makerere University have independently 
carried out the M&E workshop. Each has independently conducted two workshops. 
In 2003 Makerere University independently conducted a two weeks M&E workshop for 25 M&E 
and program managers working on UNFPA country program projects in Uganda. In 2005, Makerere 
conducted an international workshop for 20 participants, nine of them from neighboring countries in 
the region. 
MEASURE Evaluation 
ME Final.doc 48 12/30/2005 
D. Statement of Work 
STATEMENT OF WORK 
Evaluation of Regional Training Partners’ Capacity-Building in M&E 
Revised, November 2, 2005 
I. IDENTIFICATION OF THE TECHNICAL ASSISTANCE 
USAID/Washington requests support from The Synergy Project with evaluating one of the more 
innovative but difficult “Results” in Phase II of the project: strengthening the organizational and 
technical capacity of Regional Training Partners in the General Monitoring and Evaluation (M&E) 
Course under MEASURE/Evaluation (M/Eval) Phase II. 
While there are other Results in Measure Phase II, it is felt that this is one of the most complex 
changes to effect from Phase I to Phase II, and one that takes a lot of time, effective partnerships 
and sufficient resources to get off the ground. Thus, after two years into Phase II, it is timely to 
evaluate the movement in this new direction. 
The overall goal of the evaluation is to assess the capacity of M/Eval and its Regional Training 
Partners to deliver an effective, sustainable course with measurable results. This evaluation will 
extract lessons learned and recommendations for replication of program successes and 
modifications in areas of programmatic deficiency. 
II. GENERAL BACKGROUND
The M/Eval capacity-building strategy consists of three main components: (1) training courses and 
masters programs in M&E and Reproductive Health Information Systems; (2) institutional capacity 
building of training partners; and (3) training and capacity building in the context of country 
technical assistance activities. This evaluation will focus mostly on the first and second 
components, and specifically on the general three-week M&E course, as they encompass the bulk of 
the activities and participants in this new strategy. 
The prime cooperating agency, The Carolina Population Center of the University of North Carolina 
(UNC), has been implementing Phases I and II of M/Eval since 1999. Phase II began in 2003-4, 
and it is felt that considerable progress in capacity-building was made by Year 2 (2004-5) of Phase 
II of the project, but at a different rate in each institution. 
The project is currently working on short-term (three-week) Regional M&E Training Programs in 
Population Health and Nutrition (PHN) in four regions: 
• Centre Africain d’Etudes Superieures en Gestion (CESAG/Institut Superieur de Sante) of 
Senegal for Francophone Africa (AFR); 
• Addis Ababa University (AAU/Community Health Dept. and Demographic Training and 
Research Center) for Anglophone Africa; 
• The Institute for Population and Social Research (IPSR) at Mahidol University in Bangkok 
for Asia/Near East (ANE); and 
• The Instituto Nacional de Salud Publica, National Institute for Public Health (INSP) in 
Cuernavaca, Mexico for Latin America and the Caribbean (LAC). 
MEASURE Evaluation 
ME Final.doc 49 12/30/2005 
Under Phase I, M/Eval also worked with CESAG, as well as with the University of Costa Rica 
(Central American Population Center at the School of Statistics) and Makerere University 
(Population Program in the Institute of Statistics) in Uganda. The University of Costa Rica 
continues to offer similar courses on its own, while Makerere does not. 
Institutional capacity building of training partners and faculty has been carried out in Year Two in 
the two new sites, INSP and AAU, and on a continuous basis with CESAG, and to a lesser extent 
with Mahidol. These activities include: 
• technical update workshops at the Johns Hopkins University and the East-West Center 
(EWC) on M&E, 
• Training of Trainers workshops, 
• administrative and financial planning and procedures, 
• collaborative evaluation research proposal development, 
• collaborative evaluation assignments and referral for consulting 
• mentoring (on-site and outside) 
• website development, 
• capacity needs assessments and stakeholder meetings, 
• site visits to UNC and other centers of M&E training 
• curriculum development and case study preparation 
• master’s level internships 
• distance learning courses 
Since capacity-building of regional training partners is an innovative and important element of 
M/Eval. Phase II, USAID plans to undertake this evaluation after two years of implementation (of 
the five-year project). A new Results Reporting Guidance has been drafted by M/Eval., in 
accordance with the new PRH Strategic Framework concerning capacity building. This includes 
specific indicators of “increased in-country individual and organization technical capacity and 
resources for M&E …to meet data needs” (see Annex x) 
Visits will be planned to three of these regional training centers, one in each continent. USAID also 
wants to apply the new Guidelines on capacity building in M&E, as well as the new Population and 
Reproductive Health (PRH) strategic framework indicators for capacity building (Annex x). 
OBJECTIVES OF THE ASSIGNMENT (and illustrative questions per objective) 
The following questions are illustrative only. The Evaluation Team will prepare a final list of 
questions that will be addressed in the evaluation. 
1. Assess progress in M/Eval project’s objective of increasing the organizational and technical 
capacity of its regional partners to undertake the Regional and General M&E short-term training 
with gradually reduced technical, institutional and financial support from M/Eval (Phases I and 
II), and to replicate the M&E trainings independently. 
Q1. What changes has M/Eval made in supporting their regional training partners transition 
between Phase 1 and Phase 2, and within phase 2? What has been the response to these 
changes? 
MEASURE Evaluation 
ME Final.doc 50 12/30/2005 
Q. 2 How differently have the 3 evaluated regional partners moved along on the PRH-stated 
continuum from individual training to institutional capacity building in the first two years of 
Phase 2? Have different strategies been used for the different partners in this transition? 
Q.3 How much reduction, if any, has there been in technical, institutional and financial 
support per partner? What effects did these have? 
Q. 4 Have the financial resources to support the transition been adequate and appropriately 
spent? How supportive has GH been, as well as the Regional Bureaus and regional and 
country Missions? What kind of resistance is there to this type of technical and institutional 
assistance within USAID? 
Q. 5 How has the project’s process of assessment and selection of the partner M&E training 
institutions affected it’s outcomes? How were the two new partners selected (Mexico/INSP) 
and Ethiopia/AAU)? What regional and contextual factors influenced the selection process? 
Q. 6 How well is the formative and process monitoring and evaluation system working in 
support of the institutional capacity-building progress. Has the M/Eval. guide on the M&E 
of capacity-building been utilized? 
Q. 7 How was it decided that some Phase 1 partners should be graduated or dropped after 
Phase 1, and others continued? What happened to discontinue Uganda, and why was it’s 
initial replacement, Ghana, not selected. How was it decided that some partners (Costa Rica, 
Mexico, E-W Center) would focus on impact evaluation, others not? 
Q8. Is there a common or different understanding among the three partners, and between the 
donor and CA, and between the sub-partners within the CA, of what is meant by capacity￾building and its long term outcome? Is the same C-B objective expected for these three 
partners? 
Q. 9 Has there been any change in the type of technical assistance and direct M/Eval. 
teaching in the workshops between Phase 1 and 2, and during Phase 2? Has the strategic 
planning and training of trainers been adequate and appropriate? 
North/SouthTeam teaching? Have any of the older partners given technical assistance to the 
newer partners? 
Q. 10 How well and timely has M/Eval responded/adapted/applied the lessons learned and 
feedback from Phase 1 and the first two years of Phase 2? 
2. Evaluate the result of this training as far as the utilization of the new skills and knowledge of all 
participants (using convenience or representative sample) who completed structured M/Eval 
training, including mentoring activities. 
Q. 1 What have been the similarities and changes in the curriculum during phase 2? Why 
were the changes made, and by whom? What has been the balance between the substantive 
and the principles modules, and between the lectures and the case studies and laboratory 
work? How well done are the final group presentations? 
Q. 2 How do you assess the scientific, technical and didactic adequacy and appropriateness 
MEASURE Evaluation 
ME Final.doc 51 12/30/2005 
of the different training modules? 
Q.3 How did the training participants evaluate each workshop? What are the common 
issues, and what are course-specific issues? 
Q. 4 In addition to the three-week general M&E training, how else is M/Eval. contributing 
to increasing the individual capacity of the participants? 
Q. 5 How would you assess the investments made through Core-funded M/Eval 
participants, USAID Regional and Country Mission-supported participants, and other￾funded and self-funded participants? Is the proportion of Core-funded participants 
declining? Should some core funds continue to be used to sponsor participants? 
Q. 6 What follow-up has there been in country of the alumni in Phase 2? Has it increased 
over time? How do the alumni feel about it? 
Q. 7 Has there been much mentorship or on-the-job training by M/Eval on either the faculty 
or the alumni? How well has it been received by the mentorees? 
Q 8 How adequate is the candidate selection process? Assess the sub-regional balance 
among the countries of the participants (eg., in ANE, between the Arab and non-Arab 
countries; between large and small countries; between USAID and non-USAID countries)? 
How appropriate is the mix of career and professional backgrounds of the participants? 
METHODOLOGY 
Information should be collected through the following methods: 
a) Review of the RFA for the M/Eval project as well as its technical response; also the PRH 
strategic framework, and the project’s M&E plan for this Capacity-building Result, annual 
and quarterly work plans, annual reports and other relevant documents (identified by 
USAID, M/Eval and partners). 
b) Study Instruments (report annex): The Team Leader will coordinate the development of 
all study instruments, and take the lead in drafting flexible data collections guides for all key 
informants and target groups. The IDI and CSL M&E coordinator will draft the e-survey and 
phone interview guides for course alumni interviews, based on a review of 
MEASURE/Evaluation program documents (ie- interview guides for course participants, 
GH M/Eval Management, M/Eval Staff, and Regional Training Center Management). The 
case study guide will be developed by the TL and implemented by Synergy. These 
instruments will be pre-tested and adapted by the evaluation team. 
c) Data Tabulation and Analysis of Participant Evaluations and Alumni Interviews 
(report annex): The USAID/IDI and CSL M&E coordinator will compile the data from in￾course evaluations and alumni interviews, and analyze the data for inclusion in the 
evaluation report. 
d) Key informant interviews with the GH M/Eval Management team for Result 2-(Novak, 
Choi, Teller- see Part IX Relationships and Responsibilities), and with selected USAID/W 
Regional HPN staff. M/Eval. will suggest key institutional informants. 
e) Interviews with M/Eval staff with training and capacity-building responsibilities during 
Phases I and II (at UNC-Angeles, Bates, Bloom, Kulatilaki., Dunn, Frankel ; Macro-Eckert; 
MEASURE Evaluation 
ME Final.doc 52 12/30/2005 
Tulane-Gage, Escudero, Agha; and JSI-Abdou, LaFond, Mullin, Gnassou). Contact 
information will be provided by USAID. 
f) Field visits to 3 selected LAC, AFR and ANE regional training sites, Missions, host 
government and regional counterparts and other relevant stakeholders in these region (to be 
provided identified by USAID, Missions and M/Eval,.; While the M/Eval. Masters courses 
will not be evaluated directly, their existence will be taking into account, and links 
examined. Preliminary findings, conclusions and recommendations will be drafted as case 
studies.
g) DELIVERABLES 
1. Team Planning Meeting: The evaluation team will meet for two days in Washington, DC 
(and/or via teleconference, as necessary) to review the goals and objectives of the 
assignment and the roles and responsibilities of the team members. The USAID 
International Development Intern (IDI) will obtain and prepare a notebook of basic 
background documents for reading in preparation for the TPM, in collaboration with the 
PRH/Commodity Security & Logistics (CSL) evaluation coordinator. 
2. Detailed Work Plan: Based on discussions from the team planning meeting and in 
consultation with the USAID Project Advisors, the Team Leader will develop a detailed 
work plan for the assignment. M/Eval will assist in harmonizing the appropriate dates for the 
field visits, and by providing the contact names in each of the three institutions and 
respective countries. Study instruments and methods will be part of the details. 
3. Stand Alone Case Studies of the Regional Training Organizations The Synergy 
consultant team, under the direction of the Team Leader, will conduct site visits in Bangkok, 
Thailand; Dakar, Senegal; and Cuernavaca, Mexico; they are to compile their notes and 
observations and draft their case studies, allowing for revisits and follow-up phone calls. 
4. Draft Evaluation Report. The Synergy Team Leader will be responsible for writing the 
draft and final draft evaluation reports. The evaluation report will include a discussion of 
institutional capacity development in M&E, summaries of findings from the three case 
studies, relevant indicators and their required data for the evaluation, and all relevant 
annexes. The USAID/Washington team will give direction for its general content, length, 
and other USAID requirements. The Team Leader will submit via email a complete draft 
report concurrently to Charles Teller and The Synergy Project two weeks before the end of 
the assignment. 
5. USAID/Washington Debriefing. The Synergy consultant team will give a joint debriefing 
to a selected audience at USAID/Washington. The USAID/Washington team will organize 
the meeting. The Synergy Project will not provide any editing, formatting, or branding for 
the PowerPoint presentation for the USAID/Washington debriefing. 
6. MEASURE/Evaluation Debriefing. The Synergy consultant team will give a debriefing 
to a selected audience at MEASURE/Evaluation in Chapel Hill, NC. The M/Eval team will 
organize the meeting in conjunction with the Synergy Program and Project Managers. The 
Synergy Project will not provide any editing, formatting, or branding for the PowerPoint 
presentation for the M/Eval debriefing. 
7. Final Evaluation Report. The Synergy Team Leader will prepare the final draft evaluation 
report based on the feedback received from the debriefing. The report, which should not 
exceed 30 pages (excluding annexes), will be submitted to Charles Teller, the CTO John 
Novak, and The Synergy Project. The Synergy Project is responsible for editing, formatting, 
and printing 3 hard unbound copies of the final, written document, and producing 2 
electronic copies of the final report. The Synergy Project will edit, format, brand, and print 
MEASURE Evaluation 
ME Final.doc 53 12/30/2005 
the full 30-page report including annexes. The deadline for final report submission is 
December 30, 2005. Electronic versions shall be sent to Charles Teller and John Novak, or 
emailed to: (cteller@usaid.gov; jnovak@usaid.gov). 
REPORTING REQUIREMENTS (Products) 
• All products will be completed in Microsoft Word. 
• All deliverables will be submitted in hard copy and electronically to Charles Teller and John 
Novak. 
• The Synergy Project will edit, format, provide branding support, and print hard copies (3) of the 
final evaluation report. 
REFERENCE MATERIALS 
The USAID/Washington team will give guidance and provide Synergy with the reference materials 
(hard copy and/or electronic links) required or suggested for assignment preparation and the 
development of the e-survey instruments. M/Eval and PRH/PEC will be asked to provide access to 
the publications, workshop reports and relevant financial information. The following documents 
are suggested for the team’s review, and subsequently a short list will be required reading: 
• M/Eval RFA (Spring, 2003), and technical response to Result 2 questions. 
• The UNC Technical Application Phase II, Part A, Result 2: Capacity building, pp 9-14 (June 30, 
2003) 
• The UNC Final Proposal Revisions Technical Application (summer, 2003 
• Assessment of the Master’s level courses, UNC Graduate seminar, Spring, 2005 
• The M/Eval Results Reporting Guidance (e.g., PMP/M&E Plan), Aug.4, 2005 
• The PRH Strategic framework, September, 2004 
• Core and Regional Workplans (quarterly) 
• M/Eval Launch, Washington, DC, Feb. 2004 
• Minutes of technical meetings of the Working Group on Capacity-building, Washington, DC, 
April, 2005 
• M/Eval Annual Reports, July, 2004, and July 2005 
• PRH/PEC Results review, 2003, 2004 
• A guide to monitoring and evaluation of capacity building interventions in the health sector in 
developing countries, LaFond and Brown, UNC, 2003 
• Staff Handbook on capacity building in monitoring and evaluation, LaFond and Frankel, UNC, 
April, 2005 
• Trip reports to the regional training center general M&E workshops, 2003-2005 
• Activity Reports, including participant evaluations of each training course (2004-2005) 
• An Assessment of Monitoring and Evaluation of PRH Projects, Adamcheck and Reynolds, Sept. 
2004 
• List-serve of Alumni since Jan. 2004 of the General M&E Training Seminar 
• Other USAID/GH relevant documents (e.g., Health and Family Planning Indicators: Measuring 
Sustainability: Institutional Capacity (p. 15-24), AFR Bureau, Feb., 1999); ADS; CDIE/TIPS 
• Complete financial accounting for courses 
• Course curriculum (on CD for each partner by year) 
• Compilation of lessons learned 
• 6-month post-training evaluation with participants 
H H
MEASURE Evaluation 
ME Final.doc 54 12/30/2005 
• Results of capacity self-assessment and stakeholder workshop 
• Questionnaire sent to M&E staff at cooperating agencies 
TEAM COMPOSITION AND DESIRED QUALIFICATIONS 
General: Familiar with GH priorities, PRH results framework, USAID regional programming, 
excellent evaluation and writing skills, appropriate language skills, and broad experience living in 
developing countries and working with developing country training institutions. 
Specific: 
• Team leader: Consultant with successful experience as Team Leader and in evaluating capacity￾building in M&E in developing countries within the health sector 
• Capacity-building/Organizational Development expert: Consultant with extensive and 
successful experience in Franco-phone Africa, Asia/Near East and/or Latin America 
• USAID/GH/PEC Senior Technical Advisor, IDI point person and CSL M&E coordinator, with 
training in M&E and/or expertise in evaluation of capacity-building and language capacity in 
Spanish and/or French 
• Synergy program manager with experience in the coordination of logistics of a rigorous 
evaluation, including e-surveys and teleconferencing with third world alumni 
• One local consultant/interpreter for Senegal. 
IX. RELATIONSHIPS AND RESPONSIBILITIES 
USAID/Washington: USAID/Washington will provide overall direction, technical and in-country 
logistical guidance, and documents for review; approve the SOW and budget; and give final 
approval on deliverables. USAID/GH senior TA and IDI staff may also participate in one of the 
three site visits, respectively, and collaborate with the rest of the team in producing the assignment 
deliverables. MEASURE/Evaluation will contact the host partner institutions to inform them of 
possible dates for site visits; USAID/W will contact Missions to inform them of the assignment, 
dates, and needed support for identifying local consultants and interpreters, etc. USAID/W will 
provide general direction for developing the research design and the evaluation report, and the 
Team Leader is responsible for producing quality deliverables. Also, USAID/W will organize the 
Washington debriefing, and participate sparingly, as the need arises. 
The Synergy Project: The Synergy Project will provide the consultant team (team leader, CB/OD 
specialist, a local consultant/interpreter in Senegal) to USAID/Washington and logistical support as 
directed by USAID/Washington. A Senior Technical M&E Specialist will be available to provide 
guidance to the consultant team, upon the explicit request of the Team Leader, and provide 
technical review of draft report. A Program Manager will manage and support this assignment 
throughout the period of performance and will be the primary point of contact. A Program Assistant 
will provide logistical and additional administrative support to the consultants and other assignment 
activities as needed. Services from the Synergy Communications Unit will be provided for the 
preparation of the final report. The Synergy Project Communications Unit is responsible for editing, 
formatting, and printing copies of the final, written document, and producing electronic copies of 
the final report. The Synergy Project will edit, format, brand, and print the full 30-page report 
including annexes. Three unbound, hard copies of the final edited report and 2 CD ROM with PDF 
and Word files will be submitted to USAID/Washington by the Synergy Project. 
Synergy Consultant Team 
MEASURE Evaluation 
ME Final.doc 55 12/30/2005 
The Synergy lead consultant will serve as the Team Leader and principal report author. 
The Team Leader will be responsible for the overall organization of the work, developing the 
study instruments, conducting in-country briefings, and fostering cohesive and productive working 
relationships among team members. The Team Leader will consult with the client, 
USAID/Washington, as needed, throughout the assignment to ensure progress is sound and the 
scope of work is being followed. The Team Leader will facilitate the preparation of the draft 
Evaluation Report and other deliverables among the team members; assure that the draft and final 
products are prepared in accordance with the Scope of Work; and that the required revisions for the 
final deliverables are incorporated. The Team Leader will collaborate with the evaluation team to 
prepare the debriefing presentation and participate in the final briefing to USAID/Washington. The 
Team Leader will also manage local expenditures, including payment for local transport by car and 
local interpretation services as needed. 
The Synergy Senior M&E Specialist will collaborate with USAID, the Synergy Consultant Team 
and other members of the evaluation team as requested. He will guide the process of administering 
the course alumni interviews, and assist in the analysis of the data for inclusion in the evaluation 
report. He will participate, as requested, in reviewing the draft evaluation report and in the final 
debriefing. 
The Synergy OD/CB Expert will focus on assessing the organizational, behavioral and technical 
capacity of the regional training institutions, based on the assignment objectives. This consultant 
will collaborate with the team leader and other evaluation team members in preparing the 
deliverables outlined in this Scope of Work and participating in the debriefing. 
All Team Members will: 
• Participate in team planning and other meetings as directed by USAID/Washington 
• Participate in presentations and discussions with USAID officials and others as directed by 
USAID/Washington 
• Collaborate in the preparation of all deliverables 
• Engage in productive working relationships with team members 
• Maintain records and notes of all interviews and meetings 
USAID Position Title Telephone Number Email 
Charles Teller TA, Measure/Eval 202-712-5428 cteller@usaid.gov
IDI rotation: 202-712-5561 raugustin@usaid.gov Randolph evaluation point Augustin person 
John Novak CTO, M/Eval. 202-712-4814 jnovak@usaid.gov
Misun Choi HIDN, TA, M/Eval. 202-712-1722 mchoi@usaid.gov
Tanvi Pandit PRH/CSL M&E 202-712-4943 Tpandit@usaid.gov
Coordinator 
The Synergy 
Project 
H
H
H
H
H
MEASURE Evaluation 
ME Final.doc 56 12/30/2005 
Jack Reynolds Team Leader 808 395-8004 JackReynoldsHNL@msn.com
Laverne Webb CB/OD expert in PRH 410-745-3828 lwebb@encompassworld.com
Charles Katende Senior Technical 
M&E Specialist
202-842-2939 x139 ckatende@s-3.com
Melanie 
Kindfather
Program Managers 202-842-2939 x 169 mkindsfather@s-3.com
Virginia Lamb Program Assistant 202-842-2939 x129 vlamb@s-3.com
II. LOGISTICS 
The Synergy Project is responsible for: 
Arranging travel for consultants to and from the consultants’ home of record. 
Submitting finalized timeline for activity 
Providing office space to Synergy consultants, as needed. 
Providing copyediting, formatting, and USAID branding services for the final written reports 
resulting from the assignment. The Synergy Project will not provide editing or formatting for 
the PowerPoint presentations prepared for the USAID/Washington and/or 
MEASURE/Evaluation debriefings. 
Providing communications and other support associated with the assignment. 
USAID/W is responsible for: 
Arranging country clearances for consultants
Funding and sending 1-2 persons to each of the three countries
Organizing the USAID/W debriefing.
USAID/Missions are responsible for: 
Providing in-country logistical support, such as hotel reservations and local transportation, if 
needed 
M/Evaluation is responsible for: 
Assisting consultant team with scheduling in-country site visits and selected key informant 
interviews.
Providing background information, data on participant evaluations, etc.
Being available as respondents in key informant interviews.
XI. PERIOD OF PERFORMANCE 
The Synergy Project’s anticipated period of performance for this project will be from mid-October, 
2005 to mid- January, 2006. The assignment time line will be developed by USAID/W PRH and 
finalized by Synergy. For the fieldwork, two or three members of the evaluation team will visit each 
of the selected regional training sites. The Team Leader and Organizational Development 
MEASURE Evaluation 
ME Final.doc 57 12/30/2005 
Consultant will participate in all 3 site visits. The team members will spend approximately one 
week per country, including travel, a rest day, a country trip report, and holidays/delays (see below). 
Illustrative schedule of site visits by Synergy consultants and USAID (to be reviewed and 
modified by Team Leader and Synergy Program manager): 
Region Team Leader CB/OD expert CTO/TA IDI/CSL
Pre-field work (3 wks): 
USA- Chapel Hill, NC and 
Washington, DC area (Mid. 
Oct. to early Nov. 
X X X X 
AFR-Thailand (Oct. 31-
Nov. 1-4) 
X X 
ANE-Senegal (Nov. 14-18) X X X 
LAC-Mexico 
(Nov. 20-23) 
X 
The tentative timetable of events is as follows: 
Activity Who Days per 
Activity 
Dates 
Pre-field work activities (2 weeks) 
Send MEASURE Evaluation background materials 
nergy and/or USAID 
USAID/ 
Washington and 
MEASURE/Evaluation 
USAID & 
M/Eval. 
10/3-14 
 Document review Evaluation team 5 days 10/3-14 
Team Planning Meeting and development of work 
(Deliverables 1 & 2) 
Evaluation team 2 days 10-17/18 
Briefings and Guidance from PRH and the three 
onal Bureaus. 
Evaluation team 2-3 days 10/17-20 
Development of data collection, tabulation and 
sis instruments/field test and revisions (Deliverable 3) 
Team leader, OD expert 
and selected team 
members 
5 days 10/24-28 
Fieldwork (1 week)
US Interviews in Washington DC (UNC, MACRO, TL, OD, USAID and 
Synergy staff 
 10/21- 28
Data tabulation and analysis of participant 
ations and e-survey 
USAID/IDI and CSL 
M&E expert 
 Oct-Nov
Interviews with M&E Course Alumni 
-Data collection, tabulation, and analysis 
verable 4) 
USAID IDI, CSL M&E 
Expert and Synergy 
PM/PA 
 On-going
 
Travel to Bangkok, Thailand* TL and OD Expert 5-6 days 
each 
Thailand 
 -Country adaptation of final data collection 
uments/procedures 
 -Meets with USAID/regional staff 
 -Conducts program visits in collaboration with the 
TL and OD Expert Oct. 29-
Nov. 4 
MEASURE Evaluation 
ME Final.doc 58 12/30/2005 
ID/Washington team member
 -Draft country report 
Team Leader returns to home of record (Hawaii) for TL and OD Expert Nov. -5-
of interviews with the East-West Center Summer 11 
E Course directors; OD expert return to home of record 
yland) for more reading, time off (another short 
nment) 
Travel to Dakar, Senegal TL, CB/OD Expert, and 5-6 days Nov. 12-
Meets with USAID/HPN and Regional staff, etc. USAID/IDI each 18 
Draft country report 
OD Expert returns to home of record 
Travel to Mexico City/Cuernavaca site visit (etc.) TL, CB/OD Expert and 4 days each Nov. 19-
USAID/TA 23 
TL returns to homes of record for Thanksgiving TL and CB/OD expert Nov. 24-
day, then analysis and write-up Dec. 10 
Submit revised site visit trip reports to team leader Evaluation team Nov. 28-
members Dec. 2 
Post- Fieldwork 30+ days
Preparation of Draft Evaluation Report of site visits; TL (in conference calls Dec 5-9 
sis and write up of other data collection methods with team members) and 
ey, case study, phone interviews, etc. team 
(Deliverable 6) ; 
Travel to DC, and final drafts written, in TL and team Dec. 12-
hington, DC; e-mail survey and participants interviews 16 
Debriefing to USAID/Washington Team Leader and Dec 20 
(Deliverable 5) CB/OD Expert, USAID 
and Synergy staff 
Debriefing to MEASURE/Evaluation in Chapel Hill, Team Leader and Dec 21
Deliverable 6) CB/OD Expert and 
M/Eval staff 
Travel from Washington DC to home of record TL Dec 22 
Preparation of Final Draft Evaluation Report TL 
Submit final draft evaluation report based on TL Dec 30
ack from the debriefing to USAID and The Synergy 
ct
 
Synergy Communication Unit finalizes edits, Synergy Comm. Unit 
ats, and provides branding support on the final 
ment
Final evaluation report submitted to USAID Synergy Jan. ??
verable #8) 
 24 weeks
TOTAL (for the 2 
full-time 
consultants) 
* To be determined (also order of visits to countries illustrative only) 
 
XII. FUNDING 
Funding is provided mainly by PRH Core Support, with budget and final time line submitted by 
Synergy to USAID. Discussions will be held with OHA for additional support. 
MEASURE Evaluation 
12/30/2005 
E. Contacts 
USAID/Washington 
John Novak, CTO, MEASURE Evaluation 
Charles Teller, Ph.D., Technical Advisor, MEASURE Evaluation 
Subhi Mehdi, AFR/DP 
Mihira Karra, GH/PRH/RTU 
Sarah Harbison, GH/PRH/RTU 
USAID/Senegal 
Jennifer Adams, M.D., SO3 Team Leader, Chief, Health Bureau 
Brad Barker, SO3 Team member 
Ndiaye Sounka, SO3 M&E Specialist 
USAID/Regional Development Mission, Bangkok 
Patchara Rumakom, Ph.D., Project Specialist 
MEASURE Evaluation, University of North Carolina 
Gustavo Angeles, Ph.D., Deputy Director, MEASURE Evaluation/UNC 
Nina Frankel, M.A., Manager, Training and Capacity Building 
Hemali Kulatilaka, MSC, MPA 
Bates Buckner, PhD 
John Snow International 
Anne LaFond, M.Sc., Senior Technical Advisor 
Stephanie Mullen, Dr. P.H., Technical Advisor 
Maunkaila Abdou Billo, M.D., Ph.D., Senior Monitoring and Evaluation Officer 
GNASSOU Leontine, Resident Advisor M/E-JSI/Cote d’Ivoire 
The Futures Group 
Alan Johnston, M.A., Senior DDU Specialist, MEASURE Evaluation/The Policy Project 
Scott Moreland, Ph.D., Health Economist, MEASURE Evaluation/The Policy Project 
MACRO 
Erin Eckert, Ph.D., Principal Investigator, MEASURE Evaluation/ORC MACRO 
Institute for Population and Social Research (IPSR) 
Amara Soonthorndhada, Ph.D., Deputy Director 
Kritaya Archavanitkul, Ph.D., Deputy Director 
Sureporn Punpuing, Ph.D., Deputy Director 
Uraiwan Kanungsukkasem, Ph.D., PHN Workshop Coordinator 
Wassana Im-em, Ph.D., HIV/AIDS Workshop Coordinator 
African Center for Advanced Management Studies (CESAG) 
Mady Koanda, Director General 
Mo Mena, Director, Higher Institute of Health 
Koffi Amani, Workshop Coordinator 
National Institute of Public Health (INSP) 
Dr. Mauricio Hernandez Avila, Director General 
ME Final.doc 59 
MEASURE Evaluation 
ME Final.doc 60 12/30/2005 
Dr. Eduardo Cesar Lazcano Ponce, Executive Director of the Public Health Research Center 
Dra. Martha Maria Tellez-Rojo, Director, Quantitative Methods and Program Evaluation 
Mtro. Jose Urquieta, Workshop Coordinator 
Dr. Aurelio Cruz V., Assistant Director of Academic Services 
Dr. Bernardo Hernandez P., Director, Reproductive Health Department 
Lic. Tere Tellez, Executive Coordinator, Summer Session Program 
Mtra. Luz Myriam Reynales, Chief, Tobacco Research Department 
Cecilia Montes Jave, INSP Master’s student, MEASURE Evaluation scholarship recipient, 2005 
INSP workshop participant 
Ma. Del Carmen Arroyo, INSP Master’s student, MEASURE Evaluation scholarship recipient, 
2005 INSP workshop participant 
F. Documents Reviewed 
MEASURE Evaluation project, Exhibit A. Scope of Work, with IPSR. Undated, 3 pages. 
MEASURE Evaluation project, Workshop expenditures through August 2005, 1 page. 
MEASURE Evaluation project, “Expectations of partner institutions.” Undated, 1 page. 
MEASURE Evaluation project, “A Training and Capacity Building Partnership in ANE: 
MEASURE Evaluation and the Institute for Population and Social Research (IPSR), Mahidol 
University, Thailand.” Undated, 2 pages. 
MEASURE Evaluation project, “MEASURE Phase II Cooperative Agreement: RFA M/OP-03-812 
–A.3. Result 2, 30 June 2003, 8 pages. 
MEASURE Evaluation project, “MEASURE Phase II Cooperative Agreement: RFA M/OP-03-812 
– Response to Questions,” 13 August 2003. 4 pages. 
MEASURE Evaluation project, “Year 2 Annual Report Activity Reference Sheets,” (Year 2 
Workplan Scope of Work 07/01/2004-06/30/2005). Undated, 1 page. 
MEASURE Evaluation project, “MEASURE Evaluation Activity Sheets, ANE-1: Workshop on 
M&E of PHN Programs (Mahidol).” Undated, 1 page. 
MEASURE Evaluation project, “Year 3 Quarter 1 Activity Sheets-ANE-1: Workshop on M&E of 
PHN Programs (Mahidol).” Undated, 1 page. 
MEASURE Evaluation project, “MEASURE Evaluation Report of Results – October 2005.” 49 
pages. 
Handel, S., Kapaya, M., Tain, F., Villanueva, M., MEASURE’ing the Stars: A consultancy for 
MEASURE Evaluation, MHCH 315, UNC, Spring 2005. 49 pages. 
MEASURE Evaluation project, “MEASURE Evaluation Phase II: Results Reporting Guidance - 
Draft,” Carolina Population Center, UNC, 3 October 2005, 32 pages. 
MEASURE Evaluation project, “Monitoring and Evaluation of Population, Health and Nutrition 
Programs, Baltimore, Maryland, June 15-July 2, 2004: Workshop Activity Report.” 23 pages. 
MEASURE Evaluation 
ME Final.doc 61 12/30/2005 
MEASURE Evaluation project, “Activity Report: Monitoring & Evaluation of Population, Health 
and Nutrition Programs, Mahidol University, Bangkok, Thailand, November 15-December 3, 2004. 
24 pages. 
Angeles, G., MEASURE Evaluation project, “MEASURE Trip Report,” 18 October 2005. 2 pages. 
Buckner, B., Bloom, S., Frankel, N., “Trip Report,” Bangkok, Thailand, 5 January 2005, 2 pages. 
MEASURE Evaluation project, “Activity Report: Regional Workshop on Monitoring & Evaluation 
of Population, Health & Nutrition Programs in Anglophone Africa and Pre-Workshop Training of 
Trainers Session: Addis Ababa, Ethiopia, July 25-27, 2005 and August 1-19, 2005.” 35 pages. 
MEASURE Evaluation project, list of numbers of participants for INSP, Mahidol and CESAG, 
2004-2005. 1 page. 
MEASURE Evaluation project, “Appendix 3: Participant List” for IPSR 2004 workshop. Undated. 
MEASURE Evaluation project, “Proposed List of Participants, Regional Workshop on M&E of 
PHN Programs, Mahidol University, Thailand, November 24-December 2, 2005.” 1 page. 
MEASURE Evaluation project, “Announcement of Regional Workshop on: Monitoring and 
Evaluation of Population, Health and Nutrition Programs, November 15-December 3, 2004, 
Bangkok, Thailand.” 4 pages. 
CD-ROM of 2004 M&E workshop PowerPoint presentations. 
MEASURE Evaluation project, “Announcement of Regional Workshop on: Monitoring and 
Evaluation of Population, Health and Nutrition Programs, November 24-December 2, 2005, 
Bangkok, Thailand.” 5 pages. 
IPSR, Mahidol University, “Suggestions for Session Objectives,” provided to Philip Guest, 
instructor for M&E of Family Planning Programs, 2005. Undated, 1 page. 
MEASURE Evaluation project, “Follow-up Questionnaire.” Undated, 1 page. 
MEASURE Evaluation project, “Compiled Comments from Daily Feedback Cards,” 2004 
workshop. Undated, 5 pages. 
MEASURE Evaluation project, “Participants’ Current Employment, Success Stories, and Most 
Useful Topics: 2002 and 2004 Workshops on M&E of PHN Programs, Mahidol University, 
Thailand.” 6 October 2005, 9 pages. 
MEASURE Evaluation project, “MEASURE-Wide Training/Capacity Building Working Group,” 
April 21, 2005, 6 pages. 
MEASURE Evaluation project, “Outline of Proposed MEASURE/CESAG Capacity Building 
Activity (for 2004-2005).” Undated, 4 pages. 
MEASURE Evaluation project, “Institutional Capacity Building at IPSR, Thailand,” list of capacity 
development activities. Undated, 1 page. 
MEASURE Evaluation 
ME Final.doc 62 12/30/2005 
MEASURE Evaluation project, “Institutional Capacity Building at IPSR, Thailand,” - an overview 
of the program. Undated, 3 pages. 
MEASURE Evaluation project, “Framework for Diagnosing the Performance of an Organization,” 
from Enhancing Organizational Performance: A Toolbox for Self-Assessment. International 
Development Research Centre (IRRC). 1999. 
LaFond, A., Frankel, N., MEASURE Evaluation project, “Staff Handbook on Capacity Building in 
Monitoring and Evaluation.” April 2005, 58 pages. 
MEASURE Evaluation project, “Capacity Definitions.” Undated, 2 pages 
LaFond A., Brown, L., “A Guide to Monitoring and Evaluation of Capacity-Building Interventions 
in the Health Sector in Developing Countries.” MEASURE Evaluation Manual Series, No. 7. 
March 2003, 98 pages.