Research Technical Assistance Center
Midterm Performance Evaluation
(RTAC-MPE)
7200AA19F00008
EVALUATION REPORT
February 25, 2021
International Business & Technical Consultants, Inc. (IBTCI)
8618 Westwood Center Drive
Suite 400
Vienna, VA 22182
USA
This Evaluation Report was prepared for USAID’s Bureau for Development, Democracy,
and Innovation/Innovation, Technology, Research Hub/ Research Division (DDI/ITR/R)
by International Business & Technical Consultants, Inc. (IBTCI) under the EVAL ME IDIQ,
AID OAA I 15 00022. The views expressed in this Evaluation Report do not necessarily 
reflect the views of the United States Agency for International Developmentor the
United States Government.
RESEARCH TECHNICAL ASSISTANCE CENTER
MIDTERM PERFORMANCE EVALUATION
(RTAC-MPE)
7200AA19F00008
February 25, 2021 
International Business & Technical Consultants, Inc. (IBTCI) 
8618 Westwood Center Drive
Suite 400
Vienna, VA 22182
USA
This Evaluation Report was prepared for USAID’s Bureau for Development, Democracy, and
Innovation/Innovation, Technology, Research Hub/ Research Division (DDI/ITR/R) by
International Business & Technical Consultants, Inc. (IBTCI) under the EVAL-ME IDIQ, AID-OAA-I￾15-00022. Its authors are Christine Allison, Kathylynn Pierre Griff, and Elisabeth Nolan. The
views expressed in this report are those of the authors and do not necessarily reflect the views
of the United States Agency for International Development or the United States Government. 
RTAC Mid-term Performance Evaluation
February 25, 2021 i
ABSTRACT
This report summarizes the findings of the midterm performance evaluation of the Research and
Technical Assistance Center (RTAC) Contract (implemented by NORC at the University of Chicago and its
consortium of partners), focusing on the identification of successes and challenges in its implementation
to date and strategies for improving the program in the current contract period. NORC at the University
of Chicago has succeeded in building an institutionally and individually diverse network of academics
with interests and skills relevant to USAID programming, exceeding its life of project target by 100
percent. However, despite having over 900 members, the Network does not always contain the specific
academic or technical expertise that is requested for new USAID M/B/IO buy-ins, which then requires
RTAC to recruit outside of the network to fulfill newly identified requirements. The network has also not
met the expectations of many of its members to date. RTAC engages its consortium members and
research network members to offer research translation and communication products, convenings,
trainings, and additional research services. Amongst RTAC services and products offered, stakeholder
satisfaction was highest with research translation and communications products plans and lowest with
training activities. RTAC convenings generally had good outcomes, though utility of the information
shared was not rated as highly as hoped for in two of the three convenings held to date. Several of the
research buy-ins have been successful, but a few have been unsuccessful in meeting USAID’s needs.
RTAC Mid-term Performance Evaluation
February 25, 2021 ii
TABLE OF CONTENTS
Abstract i
Table of Contents ii
Acronyms iv
Executive Summary v
Findings vi
Conclusions vii
Recommendations ix
Introduction and Evaluation Purpose 1
Background 1
Evaluation Questions 3
Table 1. Evaluation Design Matrix 3
Evaluation Methodology 5
Table 2. Data Collection Method by Stakeholder Group 5
Sampling 6
Tool Validation 6
Data Analysis 6
Limitations 6
Findings: EVALUATION QUESTION 1 7
Findings: EVALUATION QUESTION 2 13
Task 4: Compose Publications and Products 14
Task 5: Convenings 15
Task 6: Trainings 16
Task 7: Additional Research and Technical Assistance 18
Findings: EVALUATION QUESTION 3 20
Key Enablers across RTAC Tasks 20
Key Enablers for Specific Tasks 21
Key Barriers across RTAC Tasks 24
Key Barriers for Specific Tasks 25
Findings: EVALUATION QUESTION 4 29
Task 4: Compose Publications and Products 29
Task 7: Additional Research and Technical Assistance 31
Task 1: Management 32
RTAC Mid-term Performance Evaluation
February 25, 2021 iii
Conclusions 32
Recommendations 35
Task 1 35
Task 2 35
Task 3 36
Task 4 36
Task 5 36
Task 6 37
Task 7 37
Annexes 40
Annex A: Statement of Work 
Annex B: Evaluation Methods
Annex C: Data Collection Instruments 
Annex D: List of Evaluation Participants
Annex E: COI Forms
Annex F: Evaluation Team Roles and Bios
Annex G: Statement of Difference 
 
RTAC Mid-term Performance Evaluation
February 25, 2021 iv
ACRONYMS
ASU Arizona State University 
CDCS Country Development Cooperation Strategy
CLA Collaborating, Learning and Adapting
COR Contracting Officer’s Representative 
CR Cost Reimbursement
EQ Evaluation Question 
FGD Focus Group Discussion
HESN Higher Education Solutions Network
IBTCI International Business & Technical Consultants,Inc.
IIE Institute for International Education
KII Key Informant Interview 
M/B/IOs Missions/Bureaus/Independent Offices
MEL Monitoring, Evaluation and Learning
MPE Midterm Performance Evaluation
MSI Minority-serving Institution
NORC NORC at the University of Chicago
PI Principal Investigator
PRB Population Reference Bureau
R2A Research to Action
RAN Resilient Africa Network at Makerere University
RFP Request for Proposal
RN Research Network
RTAC Research Technical Assistance Center 
ToC Theory of Change
USAID U.S. Agency for International Development 
RTAC Mid-term Performance Evaluation
February 25, 2021 v
EXECUTIVE SUMMARY
USAID’s Bureau for Development, Democracy, and Innovation/Innovation, Technology, Research Hub/
Research Division (DDI/ITR/R) selected International Business & Technical Consultants, Inc. (IBTCI) to
conduct a Midterm Performance Evaluation (MPE) of the Research and Technical Assistance Center
(RTAC) Contract to identify successes and challenges in its implementation to date and strategies for
improving the program in the current contract period. The RTAC project is managed by NORC at the
University of Chicago (NORC) and focuses on building an extensive network of research institutions and
individual researchers (based both in the U.S. and abroad) who can provide short-term technical
assistance to USAID Missions, Bureaus, and Operating Units through research, subject matter expertise,
and training. RTAC’s purpose is to support evidence-based programming at USAID to assist countries on
their journey to self-reliance through the provision of scientific- and research-based technical assistance.
Based on the original statement of work, RTAC was expected to have the capacity to produce evidence￾based policy briefs, technical reports, strategic research agenda setting, research activity consultations,
coordination of university-based researchers for expert panels, convenings, sector-focused trainings,
and other short-term technical assistance. The project was expected to establish and maintain the RTAC
research network (RN), including the RTAC RN HESN (Higher Education Solutions Network) website
(www.rtachesn.org); produce research publications, conduct research to action (R2A) products, and
develop communications products; support events, including development and delivery of trainings; and
manage additional research and technical assistance activities. 
As of November 2020, the RTAC RN comprised 912 academics and stretched over 74 countries. RTAC
operations experienced significant disruption from March 2020 through the end of the evaluation
period as a result of the coronavirus pandemic (COVID-19). All in-person activities were postponed, and
many research buy-ins had to revise their research designs to use remote data collection rather than in￾person data collection. The full impact of these limitations is not evident in this evaluation because only
those activities that were completed or nearing completion were included.
The purpose of the RTAC MPE was to inform mid-course program adaptations and to contribute to an
increased understanding of the research needs of USAID Missions, Bureaus, and Independent Offices
(M/B/IOs). The primary audiences for this evaluation are USAID RTAC program managers and NORC.
The MPE was intended to address four key evaluation questions with the intent of informing program
adaptation for the remaining years of the RTAC activity:
1. To what extent has the goal of building an institutionally and individually diverse partner
network been met? What have been the barriers and enablers to building a diverse partner
network?
2. To what extent does the training and technical assistance provided by the RTAC meet theneeds
and priorities of project stakeholders (such as M/B/IOs)? Which types of training and technical
assistance have been most successful in meeting the needs of project stakeholders (considering
timelines, relevance, and utility of product and services)? 
3. What were the barriers and enablers that made the difference between successful and
disappointing implementation and results of the RTAC assistance?
4. To what extent has the RTAC helped USAID M/B/IOs and affiliated counterpart NGOs and host
country governments make evidence-based decisions? 
The RTAC MPE team used an exploratory, sequential, mixed-methods design, which aligned with the
formative purpose of the evaluation. The data collection methods included key informant interviews 
RTAC Mid-term Performance Evaluation
February 25, 2021 vi
(KIIs) with different stakeholders, Focus Group Discussions with USAID technical and program staff and
with RTAC PIs, and a survey of RTAC Research Network Members. The MPE team collected data
between March and June 2020 remotely using telecommunication tools due to the COVID-19
restrictions. It was able to conduct 32 KIIs with USAID and NORC staff; 21 KII with RTAC research
principal investigators (PIs) and RTAC consortium members; and one KII with partner country
counterparts and beneficiaries. It also conducted three virtual FGDs with USAID staff and 10 FGDs with
RTAC PIs and RN members. The survey of RTAC RN members yielded 147 responses from 589 members
(25% response rate).
The strength of the MPE findings is somewhat constrained by the limited scope of the data collection
and the low survey response rates. Due to the unique nature of each activity conducted under RTAC, the
experiences of individual RN members are significantly different. Therefore, even the purposive
sampling for the qualitative data collection may not have presented all relevant views. It was also by
design not intended to provide statistically representative data that allows determining size of different
effects. The effectiveness of the designed qualitative data collection efforts was also lowered by the
inability to travel to the cooperating countries due to COVID-19.
The MPE team used collaborative learning approach for the evaluation, and the RTAC team (including
both USAID and NORC) has made changes to some of its planned activities and business practices based
on preliminary findings of the evaluations. This makes some findings presented in this report outdated.
However, we include them in the final evaluation report to provide the rationale for those adaptations
and suggest additional changes going forward.
Findings 
In the course of implementing this MPE, the team developed 51 findings, which are elaborated in the
body of this report. Overall, the team found the following:
NORC has succeeded in building an institutionally and individually diverse network of academics with
interests and skills relevant to USAID programming, exceeding all targets to date. Using word of mouth
and linking into existing networks as recruitment methods, RTAC was able to effectively overcome a few
barriers to building the network identified by the RN members.
Amongst RTAC services and products offered, stakeholder satisfaction was highest activities (as
reported by USAID personnel, PIs, and consortium members) with Task 4 and lowest with the Task 6
activities. Several of the research buy-ins have been successful, but a few had been unsuccessful in
meeting USAID’s needs at the time of data collection for the evaluation. Task 5 generally was perceived
as having good outcomes, but the utility of the information shared was not rated at the highest level for
two of the three events.
USAID and several RTAC RN researchers shared very different expectations about the final
deliverables and the extent of feedback under Task 7 research buy-ins. Evaluation team found that
even most of the Task 7 research buy-ins that were successful had issues with delays and quality control,
and few buy-ins had been unsuccessful in meeting USAID’s needs (at the time of data collection for the
evaluation). Most Task 7 research buy-ins are contracted as institutional subcontracts rather than
individual consultant agreements, which had implications for timing and may provide an explanation of
the patterns observed.
The top two enablers that made the difference in implementation and results of the RTAC assistance,
as reported by the key informants, were clarity and consistency in scopes of work provided by USAID
and PIs’ prior experience with USAID and/or with governmental agencies more broadly.Clear
RTAC Mid-term Performance Evaluation
February 25, 2021 vii
communication between technical clients and RTAC consortium members and researchers was another
key factor in successful activities. Among those research buy-ins under Task 7 that met or exceeded
expectations by both USAID and the researchers, clarity of purpose, prior experience with USAID or
government agencies on the part of the researcher, technical expertise well matched to the scope of
work, and effective communication between USAID and researcher at key points in the research process
were the key enabling factors.
Key enablers for convenings under Task 5 included effective use of feedback loops and learning
processes, effective registration processes, early identification of travel requirements, and use of soft￾copy materials. 
Most USAID technical clients very much appreciated that the RTAC buy-in process for additional
research and technical assistance under Task 7 was very simple. The RTAC team has undertaken a
range of strategies to reduce any lags in start time as a result of institutional contracting under Task 7.
Significant divergence in understanding of the scope of work and expectations for the trainings caused
a series of challenges in developing training modules under Task 6. Some consortium members felt
there was limited collaboration in the design of the convenings and noted the need for coordination of
logistics and content from the beginning.
RTAC’s inherent structure for research buy-ins inhibits the RTAC team’s ability to build client
relationships and trust over the course of activities. RN members ranked five key barriers that would
prevent them from bidding on research studies under Task 7: (1) short proposal time; (2) tight budget;
(3) inflexible timing of the study; (4) conflicting commitments; and (5) the use of firm fixed price
contracts. Consortium members and research partners reported inflexibility due to the use of firm fixed
price contracts, while USAID staff reported that Task 7 deliverables to date have not, on the whole, been
quick or agile in response to USAIDneeds.
RTAC Communications and R2A products are being used to make evidence-based decisions. M/B/IOs
reported that they are planning to use RTAC products for policy, strategy development, and project
design decisions. As most of the first round of research deliverables under Task 7 were just being
finalized at the time of data collection, the evaluation team is not able to fully address the extent to
which these products were used in decision-making. Some respondents found interim briefings to be
more useful than final deliverables because it enabled them to leverage the researcher’s expertise in a
targeted way.
Conclusions
Interpretation of findings led the evaluation team to the following conclusions:
NORC and its consortium members have exceeded all expectations in building a diverse network,
surpassing all goals set by USAID in the RFP and those set in the MEL plan for the project. As long as
researchers are affiliated with recognized academic institutions, very few barriers exist to joining the
network.
Some RTAC activities have excelled at meeting the needs and priorities of stakeholders while other
activities did not meet stakeholder’s needs. The most successful were activities with well-defined 
scopes that were well understood by all stakeholders, such as the R2A plans and communications
products; some research buy-ins; and convenings. Clear, consistent scopes of work, prior experience
with USAID, and clear communication with USAID at critical junctures were key enablers across tasks.
Conversely, lack of clarity around expected deliverables, turnover in USAID staff (and therefore changing
RTAC Mid-term Performance Evaluation
February 25, 2021 viii
expectations), and a lack of effective communication strategies were barriers to strong performance
across tasks. Several other enabling factors and barriers were also critical at a Task level. 
There is sufficient evidence that RTAC products are utilized by intended users. The R2A plans and
communications products are beginning to influence partners and host country governments.
Presentations, rather than reports or other documents, seem to have the greatest impact on those
audiences. Most of the research funded under RTAC, however, seems to be intended for USAID internal
use rather than to be shared with counterpart NGOs or host country governments.
Providing information in a timely manner is a key to the use of this information by intended US
audience, such as USAID M/B/IOs. M/B/IO respondents indicated that the presentations given by the
researchers at the conclusion of data collection were exceedingly helpful because they were able to
access the information they needed in a timely manner without waiting for a final document. It also
afforded them the opportunity to ask specific questions beyond what was included in the presentation.
Some also shared that although no programmatic changes had been made, the research findings
resulted in them asking different questions of their activities, which indicated an increased and deeper
understanding of the context and potential impacts. It is unclear if responses around the utility of
reports would have been similar if respondents had had enough time to apply the findings to their
portfolios.
The original project ToC may need revision, since there are some assumptions and expectations that
the project may not be able to meet. It was originally envisioned that the value of the RN would be in
aggregating and providing readily available access to global, diverse, and qualified academic researchers
with expertise in a wide variety of topics. However, despite the RTAC team’s success at assembling such
a network, recruitment of PIs has consistently needed to be done outside the network. The findings
raise questions of the actual value of the RN to its members and need for further expansion of the RN.
About twenty percent of RN survey respondents said that their expectations had not been met at all and
another 56.7% said that their expectations had been met only to a small extent. They also identified
conditions, most of which are present in each solicitation that would inhibit them from bidding on
studies commissioned by RTAC.
Greater focus on presentations and internal memos rather than reports for DEC publication might
provide as much value with less management and oversight required. The primary deliverables 
(reports) produced by RTAC-funded research to date have not been particularly quick or agile. This
happened due to some factors that are not taken into account by the current ToC,such as the learning 
curve for some researchers in writing for a policy-maker audience, occasional over-commitment by
researchers, and as well as the procedural delays (on both sides). 
The need for agility may also require shifting some activities from firm fixed price contracting to other
forms, such as cost plus fixed fee or time and materials if/when university conditions and policies
allow.
Research translation has been fairly effective thus far. The R2A and communications products have
been well received, as have the briefers and other distillations of RTAC-funded research reports.
The training on data use may not be a critical element of the ToC. During the implementation, the
focus of the RTAC trainings required under Tasks 6A and 6B was modified from use of data to USAID’s
research policy and training of USAID and partners has been delayed. Despite this shift in focus and the
RTAC Mid-term Performance Evaluation
February 25, 2021 ix
delay, some M/B/IOs are utilizing the findings of the initial phases of the activity in their planning and
programming without these trainings.
Recommendations
Drawing on the specific findings and conclusions discussed above, the evaluation team developed a
roster of recommendations for the RTAC team. Over the course of two virtual recommendations
workshops, several of these were refined jointly with USAID and the RTAC team, with a final list of 30
recommendations. As of the time of publication, several of these recommendations have also already
been implemented by USAID and the RTAC team. The eight Key Recommendations are in italics in the
body of the report.
Task Recommendations
Task 1 Key Recommendation 1 (USAID and NORC): Consider splitting the Research Director position into
two: one that oversees the administration of the activities and communication with partners and the 
other that provides expert methodological and feasibility review of scopes, guidance for partners
lacking USAID experience, and technical writing and editing, as needed. (Findings 12, 13, 14, 15, 28,
29)
Task 2 Key Recommendation 2 (USAID): USAID should conduct a cost-benefit analysis to determine whether the
development and use of the RN was the most cost-effective approach to filling its research needs.
(Finding 4)
Task 3 Key Recommendation 3 (USAID and NORC): Consider expanding resources for RN members on RTAC
HESN (Higher Education Solutions Network) site to include the following (Finding 3):
● Space to express interest in teaming on proposals
● Training resources (how to read/respond to RTAC RFPs, writing about research findings for
donor/policy audiences, etc.)
● Live webinars on various topics (multiple arcs – RTAC/USAID contracting, applied research
methodologies, USAID sector-focused, etc.)
● Space to share research
Task 4 Key Recommendation 4 (NORC): Add indicator(s) to MEL plan to capture influence of R2A plans and
communications products, aggregating metrics adopted within R2A plans. (Finding 51)
Task 5 Key Recommendation 5 (USAID): Consider relaxing the scope of work for Task 5 to include additional
virtual expert panels organized at low cost to engage RN members. (Findings 3, 4,8)
Task 6 Key Recommendation 6 (NORC): Follow the agreed-upon plan for finalizing training materials
currently in development. (Findings 11, 21)
Task 7 Key Recommendation 7 (USAID and NORC): Update the existing SOW template to ensure that it
contains clear guidance on identifying purpose, audience, deliverable format(s), and criteria for
deliverable inspection (including a flow-down provision for RTAC contractual clause E.3 on inspection
of deliverables). Provide links to sample deliverables for reference. (Findings 12, 13, 14, 16, 24, 26,
37)
Key Recommendation 8 (USAID and NORC): Include interim deliverables in scopes of work for
research buy-ins that will gather new data that allow NORC to monitor compliance with scope: data
collection instruments, data analysis plan, and data map; data collection debriefing; and draft report
outline. This will require an additional allocation of time in the timeline; however, it may reduce the
RTAC Mid-term Performance Evaluation
February 25, 2021 x
delays and extensions required to review and revise deliverables that do not meet USAID needs,
reducing total time overall. (Findings 18, 24, 26, 28, 37)
RTAC Mid-term Performance Evaluation
February 25, 2021 1
INTRODUCTION AND EVALUATION PURPOSE
USAID’s Innovation, Technology & Research Hub selected International Business & Technical
Consultants, Inc. (IBTCI) to conduct a mid-term performance evaluation (MPE) of the Research and
Technical Assistance Center (RTAC) Contract (contract no. 7200AA18C00057) to identify successes and
challenges in its implementation to date and strategies for improving the program in the current award
period. The RTAC project is managed by NORC at the University of Chicago (NORC) and focuses on
building an extensive network of university-based researchers (based both in the U.S. and abroad) who
can provide short-term technical assistance to USAID Missions and Operating Units through research,
subject matter expertise, and training. The purpose of the RTAC MPE was to inform mid-course
program adaptations and to contribute to an increased understanding of the research needs of USAID
Missions, Bureaus, and Independent Offices (M/B/IOs).
The primary audiences for this evaluation included USAID RTAC program managers and NORC. In line
with the purpose, the evaluation team used a collaborating, learning, and adapting (CLA) approach
within a mixed-methods evaluation framework to ensure that information was provided in a timely
manner to support project learning and adaptation. Throughout the course of the evaluation, the
evaluation team conducted regular learning sessions with USAID program managers and the NORC team
to review findings, reflect on adaptation approaches, and develop recommendations collaboratively. 
BACKGROUND
The purpose of RTAC is to support evidence-based programming on the journey to self-reliance through
the provision of scientific- and research-based technical assistance. RTAC leverages the technical and
research expertise of the higher education community, advertising technical assistance needs and
opportunities and working with USAID to select the academicians to complete the work. Based on the
original statement of work, RTAC was expected to have the capacity to produce evidence-based policy
briefs, technical reports, strategic research agenda setting and research activity consultations, and
coordination of university-based researchers for expert panels, convenings, sector-focused trainings,
and other short-term technical assistance. RTAC was awarded to NORC as a firm fixed price contract
(contract no. 7200AA18C00057, July 11, 2018 – July 10, 2020, and currently has the ability to exercise up
to three option years extending through July 10, 2023) for core activities and cost reimbursement or
incremental firm fixed price modifications for emergent tasks. RTAC was created to provide timely
research-based technical assistance to facilitate USAID’s capability to make quick, evidence-based
programming decisions by leveraging the technical expertise of the higher education community. Asof
RTAC Mid-term Performance Evaluation
February 25, 2021 2
November 2020, the RTAC Research Network (RN) comprised 912 academics1 and stretched over 74
countries.
2
The theory of change (ToC) underpinning the results for the RTAC is:
If RTAC provides global, diverse and qualified academic researchers with expertise in a wide variety of
topics who can be quick, agile, and responsive to USAID needs, and if research is translated for use and
dissemination and training is provided to USAID and partners to support research and use of data, then
USAID and development stakeholders will have an increased capability to make timely, sound,
evidence-based decisions that reflect a broad range of knowledge and perspectives. RTAC is designed
to offer learning opportunities in several ways, including producing applied research, creating
knowledge products, and conducting training and other convenings, all of which are aimed to inform
USAID programming decisions.
For management purposes, RTAC activities are categorized into seven discrete tasks:
1. Management of the RTAC mechanism;
2. Establishment and maintenance of the RTAC RN;
3. Development and maintenance of the RTAC RN website (www.rtachesn.org);
4. Production of research publications, research to action products, and communicationsproducts;
5. Support for convenings;
6. Development and delivery of trainings; and
7. Management of additional research and technical assistance activities. 
To execute tasks 1-6, NORC assembled a consortium of partners with diverse capacities, including:
Arizona State University (ASU), David Management Group, Duke University, Forum One, Institute of
International Education (IIE), Population Reference Bureau (PRB), Pulte Institute for Global Development
at the University of Notre Dame, and the Resilient Africa Network (RAN) at Makerere University. Each
additional research or technical assistance activity under Task 7 is contracted using a competitive
process to identify the best proposal, and any RN member can bid for work individually or jointly under
Task 7.
RTAC operations experienced significant disruption from March 2020 through the end of the evaluation
period as a result of the COVID-19 pandemic. All in-person activities were postponed, andmany
1Of these, approximately 38% are women researchers, 11% are researchers affiliated with U.S. minority-serving
institutions(MSI), and 50% are researchers affiliated with developing-country academic institutions (RTAC, 2020).
The figure cited represents the network as a single point in time, however, as it is continuously expanding. 2 Afghanistan, Argentina, Australia, Bangladesh, Belgium, Benin, Brazil, Burkina Faso, Cameroon, Canada, China, 
Colombia, Cote d’Ivoire, Denmark, Egypt, El Salvador, Eswatini, Ethiopia, France, Georgia, Germany, Ghana,
Guatemala, Haiti, Hungary, India, Indonesia, Iraq, Ireland, Israel, Italy, Jamaica, Japan, Jordan, Kazakhstan, Kenya,
Lebanon, Liberia, Malaysia, Mali, Mexico, Morocco, Mozambique, Nepal, Netherlands, Niger, Nigeria, Norway,
Pakistan, Peru, Philippines, Qatar, Romania, Rwanda, Saudi Arabia, Senegal, Serbia, Somalia, South Africa, Spain,
Sudan, Sweden, Tanzania, Thailand, Togo, Turkey, Uganda, United Arab Emirates, United Kingdom, United States
of America, Vietnam, Zambia, and Zimbabwe (RTAC, 2020).
RTAC Mid-term Performance Evaluation
February 25, 2021 3
research buy-ins had to revise their research designs to use remote data collection rather than in-person 
data collection. The full impact of these limitations is not evident in this evaluation because only those
activities that were completed or nearing completion were included.
EVALUATIONQUESTIONS
The RTAC MPE team used a mixed-method program evaluation design to answer four key evaluation
questions, as shown in the Table below. The following matrix provides an overview to the evaluation
design methodology developed for the study.
Table 1. Evaluation Design Matrix 
Evaluation Questions and Sub-questions Data Source Data Collection
Methods
Data Analysis
Method
1. To what extent has the goal of building an
institutionally and individually diverse partner
network been met? What have been the
barriers and enablers to building a diverse
partner network?
a. What results was USAID seeking in creating
the network? Are they being achieved? What
benefits or incentives does the RN offer to its
members?
b. What are the characteristics of individuals in
the network? What does their
“engagement” look like? What motivated
researchers to join the network? What do
they hope to gain? What would encourage
more researchers to join the RN?
c. Does the structure or management of the 
network help or hinder its effectiveness?
d. Was the RTAC theory of change (TOC) design
appropriate? Were key assumptions correct?
Should the TOC be revised?
Project documents
NORC staff 
USAID staff 
RTAC consortium
members
RTAC Research
Network members
Project document
review
Project data review
KIIs 
FGDs
Network member
survey
Content analysis of
qualitative data
Quantitative 
analysis of central
tendencies and
variance within
responses,
subgroup analysis
(sex, age group,
disability status,
length of time in
network, academic
discipline)
2. To what extent does the training and technical
assistance provided by the RTAC meet the
needs and priorities of project stakeholders
(such as M/B/IOs)? Which types of training and 
technical assistance have been most successful
in meeting the needs of project stakeholders
(considering timelines, relevance, and utility of
product and services)?
a. How is development evidence (i.e., research
results) being shared? Is RTAC providing
what the M/B/IOs are requesting?
b. Are efforts adequate to meet demand by
USAID? Has USAID been satisfied with
support from RTAC? Given the fixed price
Project documents
NORC staff 
USAID staff 
RTAC consortium
members
RTAC PIs
RTAC Research
Network members
Partner Country
Counterparts and
Beneficiaries
Project document
review
KIIs 
FGDs
Site visits
Content analysis of
qualitative data
RTAC Mid-term Performance Evaluation
February 25, 2021 4
modality, has the LOE allocated been
sufficient to meet USAID demand?
c. Which approaches were successful in
meeting stakeholder needs? Which
approaches were less successful? What other
approaches should be considered?
d. To what extent are research and
communications products being utilized?
How and by whom for what purpose? Is 
there a difference in extent of use comparing
buy-in projects with core-funded projects?
3. What were the barriers and enablers that made
the difference between successful and
disappointing implementation and results of
the RTAC assistance?
a. How are RTAC services and products
promoted throughout USAID (Washington 
and its missions)? Are efforts adequate? 
b. Are USAID staff pleased with RTAC products
and services? Are USAID technical and
program staff aware of RTAC’s RN and
services? How does satisfaction compare 
between USAID staff engaged through buy￾ins vs. USAID staff engaged through core
funded projects?
c. Are there other places or projects to obtain 
such services? In their view, does RTAC
compare favorably to other mechanisms
they might use to obtain research services?
d. What were the challenges, if any, that
hindered the achievement of desired results?
e. What were the enablers or facilitators, if any,
that accelerated the achievement of desired
results?
f. Are RTAC RN Members satisfied with the
research opportunities and other services
being requested through the network? How
are the relationships between RN members
engaged in specific RTAC activities and USAID
staff?
Project documents
NORC staff 
USAID staff 
RTAC consortium
members
RTAC PIs
RTAC Research
Network members
Partner Country
Counterparts and
Beneficiaries
Project document
and data review
Project data review
KIIs 
FGDs
Network member
survey
Content analysis of
qualitative data
Quantitative 
analysis of central
tendencies and
variance within
responses,
subgroup analysis
(sex, age group,
disability status,
length of time in
network, academic
discipline)
4. To what extent has the RTAC helped USAID
M/B/IOs and affiliated counterpart NGOs and
host country governments make evidence￾based decisions?
a. What are the key needs of USAID M/B/IOs
and affiliated counterpart NGOs and host￾country governments when it comes to
evidence-based decision-making? To what
USAID staff
RTAC PIs
Partner Country
Counterparts and
Beneficiaries
KIIs
FGDs
Site visits
Content analysis of
qualitative data
RTAC Mid-term Performance Evaluation
February 25, 2021 5
extent are RTAC products and services
meeting these needs?
b. Is evidence provided by RTAC being used by
USAID staff in policy, strategy development,
or project design decisions? How? Why or
why not? 
c. Is evidence provided by RTAC being used by
local and national decision-makers and
advocacy organizations in policy debate and
development, strategy development,
program design, or service delivery? How? 
Why or why not?
EVALUATION METHODOLOGY
The RTAC MPE team used an exploratory, sequential, mixed-methods design, which aligned with the
formative purpose of the evaluation and the CLA approach. The MPE team used eight data collection
methods to answer the evaluation questions and sub-questions listed above: (1) key informant
interviews (KIIs) with program managers (USAID and NORC personnel); (2) KIIs with USAID technical and
program staff; (3) KIIs with the principal investigators (PIs) of each RTAC research buy-in; (4) KIIs with
RTAC consortium members; (5) KIIs with partner country counterparts and beneficiaries; (6) focus group
discussions (FGDs) with USAID technical and program staff; (7) FGDs with RTAC Communications and
Research to Action (R2A) PIs and a sample of Research Network members; and (8) a survey of RTAC
Research Network members. Table 2 below shows the distribution of data collection methods for each
stakeholder group.
Table 2. Data Collection Method by Stakeholder Group
Stakeholder group KIIs Re-interviews FGDs Survey
NORC program managers 5 5 0 0
USAID staff (DC and field) 22 1 3 0
RTAC PIs and co-PIs (8,
representing 7 buy-ins)
9 0 2 0
RTAC Consortium Members 12 0 1 0
RTAC Research Network
Members 
0 0 7 1
Partner country counterparts
and beneficiaries
1 0 0 0
TOTALS 49 6 13 1
RTAC Mid-term Performance Evaluation
February 25, 2021 6
Data were collected between March and June 2020. The team collected all data remotely, as COVID-19
conditions prevented the team from traveling or conducting in-person meetings. As the evaluation used
a CLA approach, virtual learning sessions were conducted in March and May, and virtual
recommendations workshops were held in August and September. 
Sampling
As noted above, purposive sampling was used to conduct KIIs with 49 RTAC project stakeholders, as well
as with the USAID and RTAC PI and Consortium Member FGDs. The remaining FGDs comprised a
convenience sample of volunteers from the Research Network membership. Presumably those who
volunteered to participate are more willing or available to engage than their peers who did not
volunteer. A total of 90 participants participated in the KIIs (49) and FGDs (41) (see Annex D for a full list
of participants).
At the time of data collection, the RN network included approximately 600 members. RN members
were given an opportunity to opt out of the survey, and a few did. The remaining 589 received the
survey. A total of 147 RN members completed the survey, resulting in a response rate of 25%.
Tool Validation
The tool validation phase used a sample of 8 KIIs and 1 FGD for the qualitative data, as well as a sample
of 28 survey respondents. The interview protocols and the online survey were tested for
comprehension, relevancy to subject matter, and internal consistency (survey). Once the tools were
validated, the team proceeded with data collection as detailed in the Evaluation Design (see Annex B for
the full design).
Data Analysis
The MPE team used the re-emerging coding schemes from the KII and FGD notes as a guide to code each
of the KII and FGD transcriptions. After categorizing the emerging themes organized in a spreadsheet
through the identification of frequent emerging themes for each evaluation question and sub-question,
the team then thematically analyzed the coded segments to develop findings to conduct content
analysis of the transcripts (see Annex B). Concurrently, the raw data from the online survey were coded
and merged into a STATA file where descriptive statistics were run and frequency tables were produced
for each survey question. Further analyses were conducted using STATA to analyze central tendencies,
as well as differences among sub-groups such as sex, location of respondent (U.S.-based or USAID
partner country-based), and type of institution (MSI versus non-MSI). These data were then
triangulated to answer the four major evaluation questions and to find inconsistencies and
convergences in the data while documenting the RTAC program successes, challenges, design, and
implementation.
Limitations
The primary limitation of the study is the result of the design of the RTAC program model. Since the RN
members are recruited on a rolling basis, the experiences of each respondent differed based on time
involved in the RN, the services they were requested to provide (if selected at all), and their experience 
RTAC Mid-term Performance Evaluation
February 25, 2021 7
with an individual USAID Mission, Bureau, or Independent Office (M/B/IO). Similarly, USAID personnel
engaged with RTAC made differing requests and had varying needs and expectations. Therefore, the
experiences of any sample of respondents may not be representative of the whole. Additionally, as
much of the qualitative data collection (KIIs, FGDs) for this evaluation relied on purposive sampling, it
did not produce statistically representative samples. As a result, although these respondents
represented a variety of views on a topic, they may not have represented all relevant views. 
USAID determined that under Task 7, only buy-ins that were completed or had submitted initial drafts of
the research results would be included, as any buy-ins in earlier stages of the process would be unable
to provide information to answer the key evaluation questions. Due to the timing of the midterm
performance evaluation, only seven buy-ins were included in the evaluation. These seven represent the
Year 1 and some Year 2 activities, and an additional 10 buy-ins had been initiated by the time the
evaluation was completed. Therefore, the assessment of Task 7 activities includes fewer than half of the
activities. 
Another important limitation was the low response rate of 25% (147 respondents) for the research
network member survey, which was sent to 589 members. This was anticipated, as it was assumed that
the less engaged members may not participate in completing the online survey without an
incentive. Timing of data collection may have influenced whether or not RN members participated, but
the survey window was extended by two weeks, and multiple focus groups were scheduled to provide
flexibility for potential respondents.
The last foreseen limitation was the inability to travel to the cooperating countries due to COVID 19.
The evaluation design originally called for site visits to two Missions, which were changed to phone
interviews with identified stakeholders instead. Site visits to multiple geographical regions as previously
planned would have given the evaluation more depth and rigor and would have permitted the
evaluation team to survey a larger pool of participants within regions and possibly increased RN
member buy-in to complete the online survey. We wish to recognize, of course, that while COVID 19 had
a significant impact on our evaluation, it had a far more disruptive impact on the activities that the RTAC
implementers and beneficiaries have been planning to undertake.
A final limitation is that the data collection represents a specific point in time, and as this evaluation was
conducted using a CLA approach, the RTAC team (including both USAID and NORC) have made changes
to some of its planned activities and business practices based on the feedback provided to the
evaluation team, as well as a part of their own continuing CLA efforts and approaches. Therefore, some
of the findings are outdated, and some of the recommendations have already been implemented. This
final evaluation report, however, does provide the rationale for those adaptations and changes going
forward.
FINDINGS: EVALUATION QUESTION 1
As noted above, a fundamental task for NORC under the RTAC contract is the recruitment of an
institutionally and individually diverse network of academics whose interests and skills are relevant to
USAID’s strategies and programs. This goal is supported by Tasks 2 and 3 under the RTAC work plan,
establishment and maintenance of the RN and development, maintenance, and use of the RTAC website
(www.rtachesn.org), which was envisioned in the RFP as being the primary vehicle for communicating
RTAC Mid-term Performance Evaluation
February 25, 2021 8
with RN members. Therefore, the first evaluation question to be answered is: To what extent has the
goal of building an institutionally and individually diverse partner network been met? The most
significant findings are set off by italics. 
Finding 1: NORC has succeeded in building an institutionally and individually diverse network of
academics with interests and skills relevant to USAID programming.
In RTAC’s Monitoring, Evaluation, and Learning (MEL) Plan, the life of project goal for recruitment of RN
members was 454. With a network of more than 900 academics, NORC has succeeded greatly in
building the RN. Despite an unexpected delay for OMB approval of the membership application form
(which was beyond NORC’s control), NORC and its consortium members established the network and
launched the RTAC website.
The RFP set out performance targets of 15% of RN members being researchers affiliated with a U.S.
minority-serving institution (MSI), 20% being researchers from developing countries where USAID
works, and 35% overall being female researchers. In its initial recruitment of 254 members prior to the
award of the contract, NORC exceeded all those targets. As new members have joined, the balance of
those figures has shifted repeatedly, and MSI membership had dropped below the target, and as of
November 2020 was 11%, falling short of the target. MSI RN members do comprise 22% of US RN
members, however, and USAID is in the process of revising the target for this metric to be more
consistent with how MSI membership would customarily be counted (as a ratio of MSIs to overall U.S.
based higher education institutions). Female researchers comprise 38% of the RN membership,
remaining above the target. Membership among researchers in USAID partner countries expanded
rapidly, far exceeding the target at 50%.
One of the main purposes of having this large pool of research talent is to minimize the time required to
identify researchers to conduct rapid research on behalf of USAID. While the RTAC RN has exceeded its
membership demographic targets, almost every principal investigator (PI) for new research activities
under Task 7 had to be recruited outside the network. Some of the requests were highly specialized
from a technical perspective, but this fact highlights the challenge that NORC and its consortium face in
undertaking recruitment without advanced knowledge of the kinds of rapid research that may be
requested. NORC has succeeded in identifying experts and recruiting them into the RN to bid on those
research opportunities, but additional recruitment has implications for how rapidly opportunities can be
awarded. It also raises the question of the most effective use of resources from this point forward for
further expanding the network.
Finding 2: The most effective methods of recruitment were word of mouth and linking into existing
networks.
The outreach coordinators for the RN identified word of mouth and leveraging existing networks as the
most effective methods of expanding network membership. For U.S. academics in particular, it was
most effective to have a trusted source refer them to the network. The USAID brand was more relevant
for academics in USAID partner countries, but they also looked to referrals from trusted sources. Almost
all RN survey respondents, 94.4%, felt the process of joining the network was simple and timely. All RN
FGD participants thought the process was accessible to almost any university-based researcher who was
willing to complete the application. Survey respondents’ primary suggestions for improvingthe
RTAC Mid-term Performance Evaluation
February 25, 2021 9
application process were to clearly state the expected benefits to applicants and to relax the
requirement for institutional affiliation.
3
Finding 3: USAID’s and RN members’ expectations of the RN are mismatched.
In its design of the RTAC activity, USAID did not give sufficient attention to the circumstances under
which participation in a research network would be appealing to academic researchers, and perhaps
most critically, to researchers in U.S. universities. The key incentive structures within their own
institutions reward revenue generation, publications on topics within their personal research agendas in
relevant academic journals, effective instruction and advisement of students, and service to the
university. USAID’s intent to leverage their technical expertise did not account for those incentives
within the university structure. For example, a small, short-term study through an individual consulting
agreement represents additional work – nights and weekends – beyond an academic’s regular workload.
Unless it aligns very well with their personal research agenda, academics have little incentive to bid on
an opportunity, and even when it does align with their interests, the final report for USAID’s purposes
may not be their primary concern. Institutional subcontracts address some of these issues through
course buy-outs to allocate regular work time to a project, but few academics can or want to dedicate
their time completely to a short-term research project. Researchers also have an opportunity cost to
consider when they make choices about looking for sources of funding. The most attractive research
funding vehicles are predictable and consistent and allow the researcher to determine the scope of the
study. RTAC research opportunities do not match any of those criteria, and given the typically small
budgets, U.S. academics may be able to obtain more flexible research funds in similar amounts from
their own institutions. USAID partner country researchers often have heavier teaching loads than U.S.
academics, which may further limit their available time. Depending on the structure of their
employment agreements, however, they may have more flexible work arrangements than U.S.
academics. Either way, how RTAC opportunities relate to universities’ incentive structures is important
to consider.
NORC and its consortium members who are responsible for recruitment must make an appealing case
for researchers to join the RN given everyone’s limited time and attention. They have to provide a
compelling argument for why academics should join the RN. When the number and scope of research
opportunities do not match the expectations, researchers may be discouraged not only from engaging
with USAID through RTAC, but also through other mechanisms, making it harder for USAID to recruit
researchers for similar efforts in the future.
Another source of the researchers’ expectations was USAID’s utilization of the term “research network.”
A network implies connectivity between nodes, not just a one-way conduit of information (in this case,
research opportunities). Therefore, the use of the term network created expectation of connectivity
among members, including USAID technical staff that has not materialized as it was not a consideration
on USAID’s part in designing RTAC.
3 CDR program managers shared offline that relaxing the institutional affiliation requirement is not possible at this
time due to the fact that the RTAC program is funded primarily via congressionally directed funding for the Higher
Education Solutions Network.
RTAC Mid-term Performance Evaluation
February 25, 2021 10
In addition,
Although satisfaction among RN members was not a key concern for USAID in designing RTAC, it is
relevant for thinking about the design of similar activities in the future. Based on the survey results, to
which 147 RN members responded, satisfaction with the RN varied. Only about half of survey
respondents were satisfied or highly satisfied, as shown in Figure 1 below. Neutral responses may or
may not truly represent neutral feelings, assome respondents may have reported a neutral rating rather
than be seen to be critical of USAID; the evaluation team believes that to be true of at least a small
proportion, as 20.0% of survey respondents said that their expectations had not been met at all (more
than the highly dissatisfied and dissatisfied categories together). Another 56.7% said that their
expectations had been met only to a small extent, which largely aligns with the remaining neutral and
most of the satisfied responses. The responses in the FGDs were more positive than the survey
responses, but those groups were self-selected, and the responses may reflect a bias or performativity
(giving what they perceived to be socially desirable responses) among theparticipants.
FIGURE 1. SATISFACTION WITH RN AMONG RN MEMBERS
Although the survey results may overstate the lack of satisfaction among RN members, RTAC’s inability
to meet RN members’ expectations to date should be regarded with concern, as it potentially represents
reputational risk for USAID and NORC and its consortium members responsible for outreach and
recruitment. 
The RTAC contract requires the RTAC team to establish a process to incentivize commitment to the RTAC
RN (Subtask 2.A.4). However, at the time of data collection for the evaluation, the RTAC team had few 
tools at its disposal to generate any incentives. Other than bidding on the RFPs, there is no way to be
“active” in the RN.
From the member perspective, the RN offers little other than the announcement of RFPs, and those
have been relatively few (compared to the number of members) at about 6 per year and infrequent.
Among survey respondents, 73.7% agreed that the RTAC team effectively communicated new research 
RTAC Mid-term Performance Evaluation
February 25, 2021 11
opportunities to RN members, 20.4% somewhat agreed, and only 3.9% disagreed or somewhat
disagreed (2.0% did not know). (For more information about challenges to responding to those RFPs,
see discussion about Task 7 under Key Enablers for Specific Tasks.) Among those few members who
have successfully bid on rapid research activities, perspectives differ for US researchers and USAID
partner country researchers. Researchers from USAID partner countries greatly value the opportunity,
one calling it a “once in a career opportunity.” US researchers, on the other hand, noted that while the
RN might offer interesting opportunities, other, more predictable sources of funding that allow them
more flexibility to pursue their own research agendas were a higher priority for them.
Among researchers who had not successfully bid on a research opportunity, all were interested in
engaging with USAID and finding opportunities to conduct applied research that is relevant to USAID’s
portfolio. Some of the RN members were specifically interested in research opportunities that would
build their resumes, while others were interested in influencing USAID programming. The RN members’
perceptions of the potential reach of the research results also varied. For example, one respondent in
the Asia-based RN FGD indicated that he was specifically interested in influencing USAID programming,
which he believed would lead to influence on the government’s policies in his country. With a total of 6
to 10 opportunities per year, however, few of the RN members will ever have the opportunity to bid on
rapid research requests in their areas of expertise. This was a significant source of disappointment to
RN members participating in FGDs, both U.S.-based and USAID partner country-based.
Currently, the RN is of limited value to RN members, offering no sustained, predictable source of
research funds, no opportunities to engage directly with USAID, no networking opportunities, and no
other professional development opportunities. Given that the total number of research activities will
continue to be limited, RN members identified other ways in which they would be interested in
engaging. For example, RTAC could offer value to its RN members through networking within and across
disciplines and regions. RN members in USAID partner countries in particular were eager for the RTAC
website to serve as a platform to engage in interdisciplinary discussions and networking. Several also
were excited by the prospect of serving on expert panels (one of the selling points used for recruitment).
(For further discussion of expert panels, see Task 5: Convenings.) In one of the CLA discussions, NORC
indicated that the regulations governing the website prohibited the members’ information from being
published in a searchable format, however, other options that RN members suggested, such as
discussion groups, capacity or skill building seminars, or a space to advertise oneself as interested in
partnering on a specific solicitation, might be viable options. Any of these options would provide the
RTAC team with more tools to incentivize commitment and participation of a wide range of RN members
who are not successful in bidding on research opportunities.
Finding 4: It is not yet clear that the effort dedicated to building the RN itself has yielded better results
than what an approach exclusively using targeted recruitment would have yielded in the absence of a
network and for a lower cost.
Establishing a network of more than 900 researchers in 74 countries represents a significant investment
by USAID and its RTAC partners in terms of time, level of effort, and funding with approximately a
quarter of the budget initially allocated to this objective. The size of the RN is not aligned with the
number of opportunities that are released (6 to 10 per year), and the contract ceiling prevents the
number of opportunities from increasing significantly. Further, many of the PIs have had to be recruited
from outside the network, representing a significant additional level of effort on the part of the
Research Director (see Finding 36 for more information).
RTAC Mid-term Performance Evaluation
February 25, 2021 12
In exploring additional ways in which the RTAC RN could provide value to USAID, the evaluation team
found that awareness of the RTAC RN appears to be low outside of the Global Development Lab. In
FGDs with USAID staff in Bureaus and Independent Offices who frequently contract and oversee
research, only a few were aware of RTAC activities, and none had a clear idea of the actual services it
offered. While budgets in those Bureaus and Independent Offices are often obligated into their existing
award vehicles (which, in some sectors, include short-term research services often engaging academics
through other implementing partners), the technical experts do sometimes have the opportunity to
advise Missions on options for obtaining services, and it would be useful for them to have more
information on the RTAC vehicle, particularly if the contract ceiling was able to be raised to
accommodate more activities. Both Mission staff and Bureau and Independent Office staff noted that
RTAC potentially offers a unique value proposition to USAID Missions. While other contracting
mechanisms may offer research services, few, if any others, have the extensive reach into a range of
academics and institutions in USAID partner countries that RTAC offers.
Another key area of competitive advantage that RTAC does have over other vehicles that offer research
services is the research translation support, or how to communicate research results to general and
policy-maker audiences. While USAID respondents identified some other contracting vehicles that
support the development of research-based communications products (Translating Research in Action
(TRAction) and some Mission-based communication support platforms), for example, they identified
many more that supported short-term research (eight central mechanisms4 along with several Mission￾based MEL awards that include research components). Therefore, if the ceiling could absorb the
additional work, RTAC may be well positioned to supplement the services that Bureaus and Independent
Offices have within their own contracting vehicles in that area specifically. 
A corollary to the first evaluation question is: What have been the barriers and enablers to building a
diverse partner network?
Finding 5: RN members identified few barriers to building the network.
As noted above, the RN members felt that the application was accessible and relatively easy to complete
for any researcher affiliated with an academic institution. Two of the few barriers they identified were
the results of management decisions related to fidelity to scope of work and cost. For example, the
requirements for institutional affiliation and the use of English as the lingua franca within the network
do potentially exclude some researchers, but USAID intended RTAC to engage the academic community
and resources are not consistently available for translation. One other barrier identified related
specifically to MSI-based researchers, as many MSIs are mission-driven to focus on teaching rather than
research, and they may lack the internal support that enable research-focused universities to bid on
funded research opportunities. During a CLA session, the RTAC and evaluation teams discusseda
4 LAYER 2; START 2; PPL-LER MEL IDIQ; Global Health Center for Innovation and Impact (CII); MEASURE Evaluation
(Data for Impact); Science, Technology, Innovation, and Partnership (STIP) for Development; Data and Evidence for
Education Programs (DEEP); Education Performance Improvement, Communications, and Knowledge (EPIC);
Monitoring and Evaluation to Collaborate, Learn, Adapt and Knowledge Management activity (South Africa
regional); USAID Kenya and East Africa Evaluation, Assessment, and Analyses (EAA) (East Africa regional), and
Mission-based MEL awards that include research components. 
RTAC Mid-term Performance Evaluation
February 25, 2021 13
targeted outreach campaign focused on MSIs with a research focus, which NORC and its consortium
subsequently did, successfully increasing the percentage of RN members from MSIs.
The two remaining barriers identified are related: the limited number and disciplinary focus of
research solicitations that have come through the network and the lack of tangible benefit to the
university researchers who join beyond the opportunity to bid on those few solicitations. As word of
mouth and existing networks were most effective for recruitment, a perceived lack of benefits and
opportunities to bid on research solicitations may eventually hamper recruitment efforts, particularly 
among U.S. researchers. 
FINDINGS: EVALUATION QUESTION 2
The RTAC RN’s primary function is to assist NORC and its consortium in meeting USAID’s training and
technical assistance needs, which are organized into Tasks 4 through 7 as described above. The key
evaluation questions regarding these services, therefore, are: To what extent does the training and
technical assistance provided by the RTAC meet the needs and priorities of project stakeholders (such
as Missions, Bureaus, Independent Offices)? Which types of training and technical assistance have
been most successful in meeting the needs of project stakeholders?
Finding 6: Overall, RTAC’s performance has been mixed in meeting the needs and priorities of all
stakeholders across Tasks 4, 5, 6, and 7.
While a majority of stakeholders were satisfied (see Figure 2), it is of considerable concern that 24
percent of USAID staff and 30 percent of PIs reported being unsatisfied. Five of the USAID staff who
gave a neutral rating were ultimately happy with the final product, but they noted that the process to
get a satisfactory deliverable was onerous. It is also notable that the USAID staff who were dissatisfied
(5), neutral (7), and even one who was satisfied indicated in response to another question that they
would not recommend RTAC to a colleague, resulting in a net promoter score of 47% (n=21). In
addition, the PIs with the dissatisfied and neutral responses said they would not undertake another
RTAC activity without significant changes to how the project operated, and one would not undertake
another activity under any circumstances.
RTAC Mid-term Performance Evaluation
February 25, 2021 14
FIGURE 2. SATISFACTION WITH RTAC EXPERIENCE BY STAKEHOLDER GROUP
Task 4: Compose Publications and Products 
Finding 7: Amongst RTAC services and products offered, stakeholder satisfaction was highest with
Task 4 activities (as reported by USAID personnel, PIs, and consortium members). 
Under Task 4, RTAC collaborated with and supported other Global Development Lab research initiatives,
providing unique value by assisting them with research utilization strategies. Based on the information
collected in KIIs and FGDs, the deliverables that best met USAID and stakeholder needs and expectations
were the research translation communications products and research-to-action (R2A) plans. One
respondent said this component was “one of the strongest and most innovative components of the
project” and that there were opportunities in this area to design and test new approaches to improving
research use in data for decision making processes, not only with decision-makers in USAID partner
countries, but also within USAID itself. The R2A model allowed USAID to support researchers in
developing and implementing strategies to promote research dissemination and application to concrete
development challenges in their communities, which is aligned with USAID’s other efforts to promote
self-reliance in USAID partner countries. Task 4 effectively equipped research teams to promote their
research results and advocate for the utilization of evidence in decision-making.
One strength of RTAC’s technical assistance, according to consortium members, was the way that
throughout the design of communications products and R2A plans, RTAC (largely through the consortia
partner, PRB) played the role of facilitator, helping clients and other stakeholders to clarify their goals
for the use of research and evidence, to identify key audiences, and to determine what messages would
be most useful to accomplish their goals with each audience. These research translation activities were
not only reported by consortium members as a key stated goal of the research to action plans, but they
also played a major role in clarifying what might be the most useful scope for communications products
and other deliverables to achieve research utilization under Task 7. In this way, RTAC provided a key
“puzzle piece” that was missing between research and use for both USAID stakeholders and
international researchers. One consortium member stated, “USAID is making a really good investment
in helping researchers become more accountable and capable of making their research useful,” but
pointed out that this is only helpful when policymakers and institutions have enough capacity to create
an enabling environment open to improvements based on research. 
RTAC Mid-term Performance Evaluation
February 25, 2021 15
Other respondents suggested a variety of ways in which this Task could usefully be expanded. One
consortium member suggested that RTAC could be more involved with researcher recipients of
research-to-action technical assistance, offering support at more than one point. The current structure
of Task 4 activities supports a single instance of technical assistance to help with planning how to
disseminate research results to key audiences based on stakeholder needs, but a respondent noted that
coming in to offer strategic technical assistance at several points during and after research has been
completed could help researchers to develop capacity more rapidly, which is currently beyond the
technical assistance role in the RTAC scope. Similarly, focus group respondents who received R2A plans
or communications products provided several suggestions for how RTAC could help researchers with
other modes of dissemination. Some suggestions were for more intensive training of researchers on
how to best communicate with different audiences, for example, to translate research into fact sheets,
infographics and language that is more easily understood by laypeople, rather than providing one
instance of technical assistance. Another participant suggested that RTAC could help researchers
identify new audiences, such as the private sector or academic conferences. One focus group’s
participants also mentioned that examples of different types of products, such as fact sheets or policy
briefs, could help them to learn to reproduce similar documents presenting their research. 
Task 5: Convenings
Finding 8: The scope of work for convenings has been interpreted more narrowly by USAID than was
written in the RTAC RFP.
In the original RFP, Task 5 included the potential for convenings and expert panels every year, as
needed, around a series of topics to be determined. The current plan for this task, however, is to only
conduct convenings as part of a wider Global Development Lab effort to support innovation and
research in key priority areas with topics to be selected by Lab leadership. One of the main reasons
cited for this narrowed interpretation was that convenings are costly. NORC reported having proposed
additional convenings (in-person and virtual, which have a significantly reduced cost), but at this time,
USAID is not prepared to revise its interpretation of the scope. Therefore, there is no clear strategy on
how convenings could or should be used to support the broader goals of the project.
Finding 9: Convenings generally had good outcomes, though utility of the information shared was not
rated as highly as hoped for two of the three.
RTAC has supported three convenings thus far: Digital Tools and the Future of International
Development, Challenges and Best Practices for Research Translation in South East Asia, and Using
Research to Advance International Development: Women’s Empowerment as a Case Study. After each
of the events, NORC surveyed participants on their perceptions of the utility of the information
presented and held after action reviews to assess performance with USAID and consortium members.
All of the convenings were well received and appreciated by the participants, though the utility of what
they learned in the first two fell somewhat below the target RTAC set of 80% of convening participants
that say the event provided useful, actionable knowledge.
Finding 10: RTAC’s CLA process was very helpful in refining the approaches used in the second and
third convenings.
USAID stakeholders, NORC personnel, and consortium members identified a number of challenges with
the organization and planning processes in the first two events, particularly around roles and
RTAC Mid-term Performance Evaluation
February 25, 2021 16
responsibilities among partners for content and logistics and their coordination, but also in timing of
involvement of each partner.
RTAC has a strong CLA component to its activities, and effective use of this approach allowed the RTAC
team to modify certain elements of its planning and management of convening-related tasks for
improved results in the third convening. All stakeholders involved in more than one convening noted
improvements from the first to the third convening, particularly in the assignment of roles to partners,
the timing of partner involvement in the planning process, and the coordination of content and logistics.
Challenges remained in planning and executing the second and third convenings, and those issues were
mainly related to supporting international travel, logistics coordination, and communication for the
second convening.
Task 6: Trainings
Finding 11: Amongst RTAC services and products offered, stakeholder satisfaction was lowest with
Task 6 activities (as reported by USAID personnel, the RTAC team, and consortium members).
The training task was the area in which all midterm performance evaluation respondents expressed the
most frustration. The purpose of the trainings under Task 6 in the RFP was to provide the capacity for
training on demand in a breadth of sectors, starting with support for USAID personnel in the adoption of
effective strategies for using research in their work, including the incorporation of new digital tools.
Three initial subtasks were enumerated in the RFP under Task 6: a landscape review to assess training
needs, development of three training modules on use of research for global development, and the
development of a slide deck for use in webinars and in-person trainings on digital integration best
practices. One of the primary challenges for the activities under this task was that they were contracted
as firm fixed price activities, and in order to do that, all bidders had to make assumptions about
expected processes and outcomes, which is difficult to do when the task explicitly requires a needs
assessment for its design. One might argue that the needs assessment should have been included in the
firm fixed price activities, but that the scope for the training modules should have been issued as an
additional buy-in once parameters could be set in detail, if issued as firm fixed price, or to permit other
contract terms (cost plus fixed fee or time and materials). In addition, the technical staff who wrote the
RFP requirements initially for both sets of Task 6 activities were no longer the lead USAID points of
contacts for those tasks when the contract was awarded, so new personnel had to develop their own
understanding of what needs were and how they could be met under this task.
The landscape review on training needs for data and research use (Task 6a) was conducted at start-up,
as expected. USAID and the RTAC authors disagreed over the interpretation of some of the data,
however. The review was critiqued as presenting skewed data that supported the conclusions the RTAC
team expected rather than being a fresh, critical analysis of USAID’s needs. Since its publication, USAID
has re-analyzed the data and derived its own conclusions that differed from the conclusions in the
review. All publications produced by USAID contractors contain an explicit or tacit disclaimer that the
opinions contained therein reflect the authors’ views and not those of USAID. However, when the
review is intended to be the basis for designing a training, those differing interpretations can cause
challenges in the design process.
For the design of the training on research use itself (Tasks 6b), USAID technical activity managers
expected to be able to leverage the full membership of the network to develop the trainings (based on
RTAC Mid-term Performance Evaluation
February 25, 2021 17
specific language in the RFP), but as these were core activities with firm fixed price budgets, they had
been assigned in the proposal development process to consortium members for planning and budgeting
purposes. Changing that arrangement would have required modifications to multiple subcontracts by
the USAID Contract Officer, which is a burdensome request that is preferably avoided.
As noted above, a range of assumptions had to be used to plan for the development of training
materials based on the information in the RFP. USAID’s final decision on the content to be included in
the training on use of research for global development differed from what was expected by the RTAC
team. The number of modules requested was the same, and therefore USAID technical activity
managers expected the level of effort to be equivalent; however, some members of the RTAC team
perceived this as a shift in scope, largely because it required input from different subject matter experts
than they had expected and to whom they did not necessarily have ready access, i.e., USAID staff
themselves. Once the consortium members were able to be in collaboration with USAID as subject
matter experts, especially incorporating USAID internal processes, agendas, and examples from the
inside that contractors would not likely know, the team was able to better meet the needs of
stakeholders within USAID. During this process, USAID clients were able to think about their own
approaches to using research, which also helped the behavior change training goals to be more clearly
identified. 
The RTAC team noted that goals continued to be clarified, and USAID’s expectations shifted repeatedly
over time. USAID respondents indicated that the significant time lags from one round of reviews to the
next (the timing required by the RTAC team and its consortium partners to facilitate the execution of
other deliverables) meant that the training needed to be updated to reflect the most current situation
within the Agency. Stakeholders on both sides of the relationship wound up frustrated, with the RTAC
team feeling that the expectations kept changing and USAID feeling that the RTAC team was not
committed to producing top-quality training modules that met theirneeds.
To resolve the situation, direct engagement with the activity managers provided an enabling
environment to manage these changes and client expectations throughout the process. Being very clear
about agreements and putting things in writing really helped with this process. Ultimately, the RTAC
team delivered the “Using Research Evidence to Accelerate International Development Outcomes”
training. Its timeline was affected by multiple delays, and it required several iterations to produce a
final product that was acceptable to USAID, taking almost two years to complete. Ultimately, only a few
trainings were conducted. One notable comment from one of the RTAC team respondents was that
they thought there would be less substantial involvement from USAID on the design and developing
materials together every step of the way because RTAC is a contract and not a cooperative agreement.
This perception is a misreading of the extent of involvement in both, and that misunderstanding may
have caused some of the initial difficulty in responding effectively to the shift in approaches and the
subsequent changes USAID requested.
A second set of training materials has been developed under Task 6c, “Applying Digital Tools to Support
the Journey to Self-Reliance.” Turnover on the part of the technical activity managers at USAID limited
the amount of information the evaluation team was able to collect on this training, but others familiar
with the process noted that similar concerns and experiences arose around this activity.
The delays in getting to a final training product that USAID believes will meet its needs mean that
training delivery has not occurred as expected, which is a key element in increasing the capability of
RTAC Mid-term Performance Evaluation
February 25, 2021 18
USAID staff and partners to use research and data effectively – a primary focus of the RTAC theory of
change.
Task 7: Additional Research and Technical Assistance 
Finding 12: Several of the research buy-ins have been successful, but a few had been unsuccessful in
meeting USAID’s needs at the time of data collection for the evaluation.
One of the primary services RTAC was designed to provide was short-term, rapid research in support of
USAID programming. Each new research activity is contracted as an incremental modification to the
overall RTAC contract, and this allows NORC to add individual academics or institutional partners from
the RN specific to each task. The evaluation included seven of the early buy-ins in addition to another
research activity that was part of the original core activities.
5
Ultimately, most of the USAID technical clients were satisfied with the final products they received. For
more than half of the research buy-ins included in the evaluation, however, the USAID technical clients
felt that the first draft of the deliverable failed to address some of the key questions included in the
research scope. After the first instance, NORC instituted a technical review of all deliverables to ensure
they met the basic requirements outlined in the contract. This additional review was likely helpful in
some instances, but in at least two, the revisions NORC directed the researcher to make had to be
undone based on USAID comments. This demonstrates the ongoing learning needed among all
partners.
Two USAID technical clients were particularly dissatisfied with the final research product of one buy-in 
(as was another with an interim product at the time of data collection, and therefore no final
assessment of utility could be made). The technical clients who were not happy with one product felt
that RTAC did not have the right researchers with good local networks with people who could relate to
the local culture and gather information about the reality on the ground. The final research report
produced did not provide new information that was actionable for USAID programming.
Although the final products were generally satisfactory, several USAID clients found the process more
onerous than expected, and the process generally took longer than expected, reflecting multiple rounds
of revisions and, at times, delays on either the USAID and/or researcher side as other more time￾sensitive commitments took precedence. In some cases, the result of the delays was that the
deliverable could not be used as intended, for example, as an integrated part of the problem statement
analysis in an RFP, which had a firm deadline for release.
Finding 13: USAID and several RTAC RN researchers had very different expectations about the final
deliverable expected and modes and extent of feedback.
One key issue that emerged from discussions with USAID and RN researchers was the frequent
mismatch in their understanding about the expectations for deliverables. At the beginning of their
studies, researchers who had not worked with USAID in the past understood rapid research to be a less
formalized process with lower standards for the final product. They had not envisioned multiple rounds
5 As of the completion of the evaluation, 17 buy-ins had been contracted, and the findings should not be construed
to apply to the 10 additional buy-ins that were too early in the process to be examined in the evaluation. 
RTAC Mid-term Performance Evaluation
February 25, 2021 19
of revisions, nor did they expect the extent of editorial feedback they received from NORC and USAID.
Many of them also did not appreciate the key differences in writing for academic audiences and writing
for policy makers.
Those researchers who had worked with USAID in the past, or in some cases, with other governmental
agencies, had a better understanding of what was required of a “draft” report (that is, a final version
they believed to be ready to publish) and of the kinds of feedback they might receive. The writing was
also much more tailored to policy making audiences with concrete recommendations for programming
or policy adaptations (when requested/appropriate).
Some research deliverables also had multiple reviewers who provided contradictory comments. In
these cases, the researchers had to request that USAID streamline and coordinate comment through
NORC, which added to review time as well. 
Finding 14: Even most of the Task 7 buy-ins that were successful had issues with delays and quality
control.
One of the main reasons for the long review times required and heavy editing was consistent difficulty
with quality control. A common assumption among USAID stakeholders was that academic researchers
generally write well, but many of their articles and book chapters are rewritten and revised many times
before publication following a much longer timeline than afforded by RTAC’s rapid research activities. In
early products, communicating the key messages is often the main focus rather than producing polished
narratives. USAID technical clients noted that some of the initial deliverables had a number of basic
grammatical and spelling errors, and NORC instituted an additional copy-editing review for all
deliverables prior to submission as a result. Despite this addition, however, some deliverables 
continued to have basic quality issues as delays in receipt of deliverables from the researchers reduced
the time available for NORC technical and copyedit reviews. While the NORC team frequently worked
after hours and on weekends to complete reviews prior to submitting deliverables to USAID, the
reduction in time available for those reviews likely contributed to the continuing challenges with quality
control.
Finding 15: Most research buy-ins are contracted as institutional subcontracts rather than individual
consultant agreements, which has implications for timing. 
Several assumptions were made in the design of RTAC about how these services could be obtained. One
key assumption was that academics could be contracted as individual consultants, which would be a
very simple, streamlined contracting process. However, all but one of the research buy-ins included in
the evaluation necessitated institutional subcontracts with the researchers’ institutions. The purpose of
the institutional subcontracts is manifold: (1) it allows the researcher to do the USAID-funded research
as part of his or her normal workload rather than in addition to it (e.g., course buy-outs); (2) it allows the
researcher to engage graduate assistants, statistical support units, librarians, and others who provide
relevant services within the institution; and (3) it enables researchers to benefit from the incentive
systems within universities that reward revenue generation through research. While institutional
subcontracts are typically favorable for the researchers, they generally incur higher overhead rates and
require significant time for approvals on both the university’s end and on USAID’send.
RTAC Mid-term Performance Evaluation
February 25, 2021 20
FINDINGS: EVALUATION QUESTION 3
Having determined which forms of training and technical assistance was most successful in meeting
USAID needs, it is useful for current and future programming to explore why some elements were more
successful than others. Evaluation Question 3 therefore, focuses on: What were the barriers and
enablers that made the difference between successful and disappointing implementation and results
of RTAC assistance?
Some factors were relevant across tasks, while other enablers or barriers were specific to a particular
task. Each of these is discussed below.
Key Enablers across RTAC Tasks 
Finding 16: The primary enabler across tasks was clarity and consistency in scopes of work provided by
USAID.
Across tasks, one key enabler of success has been clarity and consistency in scopes and expectations of
deliverables. This finding is consistent with prior USAID experience in contracting research activities. 
USAID focus group participants from a variety of operating units who have overseen research activities
in the past highly recommended that scopes be thoroughly thought through to ensure that the product
is what USAID needs. Key lessons learned included ensuring that purpose and audience were clearly
articulated. Another was providing guidance as to the format for deliverables and the standards against
which the deliverable would be reviewed; being specific provides a quality checklist the researcher can
use. This also becomes a means of ensuring that any limitations on rounds of reviews by USAID are
meaningful and substantive (rather than copyediting and struggling with poorly formatted documents).
Audience designation is particularly critical, and it may be a difficult one for many researchers to grasp.
In many cases, in principle USAID technical managers could/would conduct the research themselves if
they had the bandwidth, but they do not. Therefore, the technical managers very much know what they
want out of the product. However, it may not be clear to the PI that the technical managers could
conduct the research themselves and/or what purposes they intend for the final product.
Finding 17: A second common factor across successful RTAC activities was PIs’ prior experience with
USAID and/or with governmental agencies more broadly. 
In general, PIs who had experience working with USAID in the past, or in a few cases with other
governmental agencies, had a clearer idea of the technical clients’ needs and types of deliverables
required. Although some of the concerns about editorial quality control arose with those deliverables,
the technical aspects of the deliverables were more aligned with USAID’s expectations upon initial
submission.
Finding 18: Clear communication between technical clients and RTAC consortium members and
researchers was another key factor in successful activities. 
Across tasks, clear and direct communication was critical. One consortium member reported that
having direct interaction with the activity manager at USAID improved outcomes and reduced scope
creep under Task 4. Another respondent reported that scopes of work, clarity of purpose, and
communication on deliverables were much improved when the subcontractor was in the lead in
communicating with USAID on Task 6. When NORC was translating USAID’s expectations to the
RTAC Mid-term Performance Evaluation
February 25, 2021 21
subcontractor, the vision for the scope varied between USAID and NORC, causing issues with providing
the correct services to the client. In the successful research buy-ins, it was also a key factor to have
clear communications at specific junctures: kick-off, completion of the study design, and completion of
data collection. Checking in at these three critical points allowed the researchers to validate their
approach with USAID and ensure that the design would capture the information needed and that the
final report would present it in a way that was useful to the intended audiences. Similarly, clear and
direct communication in the planning of the second and third convenings reduced time required for
certain tasks and facilitated rapid revisions to the agenda and invitee lists as needed.
Key Enablers for Specific Tasks 
Finding 19: Key enablers for the R2A and communications products under Task 4 included an effective
selection process, a co-creation approach to the product, and timely execution of the activities. 
The R2A plans and communications products are tailored to each research activity, but the processes 
are well-tested and deployed consistently. Three key factors emerged in discussions with Task 4
stakeholders as being critical to its success. The first of these was the selection process used to
determine what research studies would receive additional technical assistance through RTAC in the form
of R2A plans (at the mid-study point) or communications products (at the end of the study). PIs of
USAID-funded research studies under other award mechanisms (primarily PEER and HESN) are invited to
apply for additional technical assistance under RTAC. Self-selection is a first stage of filtering, and USAID
then conducts a technical review of the applications against criteria established with the RTAC team in
Year 1. Thus far, each of the studies selected has successfully utilized the additional technical assistance
offered, and the criteria established in Year 1 are serving RTAC and USAIDwell. 
A second key factor is the co-creation process utilized to develop the R2A plans and the communications
products. The PIs and other relevant experts are extensively involved in their development, and the
process utilized allows them to leverage their technical expertise and combine it with the RTAC team’s
advocacy and communications expertise. Two consortium respondents highlighted the engagement and
collaborative attitude both in USAID staff and local researchers as key contributors to the achievement
of desired results. The co-creation process grounds the R2A plan and communications products such
that they are tailored and relevant to the intended audiences identified. Stakeholders on all sides
appreciate the facilitation process, and for some PIs, the experience was revelatory, as they had not had
an opportunity to participate in a similar co-creation process in thepast.
Finally, the timing of the R2A plans and communications products has been kept to a tight schedule to
ensure that the timing coincided with the finalization of study results and, in the case of the R2A plans,
with windows of opportunity where other policy changes are already happening, allowing the PIs to
leverage and contribute to those changes. Keeping the Task 4 deliverables on schedule at times meant
that other deliverables that a consortium member had assigned to the same individual(s) were
sometimes delayed or put on hold, however.
Finding 20: Key enablers for convenings under Task 5 included effective use of feedback loops and
learning processes, effective registration processes, early identification of travel requirements, and
use of soft-copy materials. 
RTAC Mid-term Performance Evaluation
February 25, 2021 22
As noted above, USAID and the RTAC team effectively applied its CLA approach to its convenings and
applied what the team learned from one event to the next. In doing so, the team used its resources
more effectively in the later events. For example, learning from the first event, the team subsequently
managed registration for speakers, participants, subject matter experts through Google docs to
minimize any confusion and increased communication with and active participation from USAID. The
team used spreadsheets to allow convening organizations to approve and revise the invitee list and send
out reminders and confirmations, as well as to allow participants to update their RSVP status. The
management of the help desk was able to handle many questions that the participants had on
registration and track the feedback and follow-up on each participant’sinquiry.
Another key lesson from the second to the third convenings was the early identification of travel
requirements, particularly for those participants traveling to the U.S. or other countries on RTAC funds.
The timelines for obtaining visas are a critical concern for international travelers, and the RTAC team
documented this “lesson learned” in its after-action reviews to take forward into planning future events. 
Finally, the team eliminated paper convening materials. Using an electronic version of the convening
agenda, the organizers were able to send the agenda and biographies of the speakers/presenters in
advance. Participants were able to download the information on their smart devices, and in the end,
less paper was used. Another factor which two respondents thought was important was the use of an
event-specific design that incorporated the logos of various partners. The graphic design permitted
aspiring event participants to quickly recognize the partners affiliated with the training and encourage
them to participate.
Finding 21: Key enablers for trainings under Sub-task 6B included engagement by the COR in
mediating conflicting interpretations of scope and access to USAID staff as subject matter experts.
Although satisfaction was lowest with the trainings among Task 6A and 6B stakeholders, two key
enablers emerged as factors in overcoming the challenges identified. First, the COR was engaged in
mediating a satisfactory outcome for all parties. This can be a difficult role, but as of the time of data
collection, final agreements had been reached, and finalization of all remaining training materials was
proceeding.
Another key factor in moving toward a satisfactory deliverable was the recognition that under the final
scope for the trainings, USAID staff were the primary subject matter experts and had a significant role to
play in the instructional design of the training course. When USAID staff were able to work with the
RTAC team in that role, the decisions on learning objectives, content, format, and delivery were able to
be made more expeditiously. 
Finding 22: Most USAID technical clients very much appreciated that the RTAC buy-in process for
additional research and technical assistance under Task 7 was very simple. 
Most RTAC customers noted that the buy-in process itself was very easy once the scope was developed,
though some did note that the time frame for contracting was longer than they had hoped it would be.
This is consistent with the experience of other USAID staff who have overseen research activities. The
evaluation team conducted two focus groups with USAID staff who, although they had not used the
RTAC mechanism, had prior experience overseeing research services for their Bureaus or Independent
Offices. In addition to the information shared by stakeholders who had used RTAC, these staff offered a 
RTAC Mid-term Performance Evaluation
February 25, 2021 23
number of insights about what in their experience led to successful experiences with applied research to
support USAID programming.
The first factor they identified was mechanism selection. Short-term research is often conducted using
Office or Bureau mechanisms in which Washington-based Offices and Operating Units have already
obligated their resources (i.e., built into annual work planning for existing awards including research
activity funding lines rather than having funding set aside for that special purpose). Therefore, those
technical areas in which Operating Units do not have established research or broad program support
vehicles in place will have greater need for services like those RTAC offers than those that do. However,
USAID staff may be more open to using other mechanisms when they do not have a preferred method in
mind, when approaching a new strategy period for which they might need support, or when the work
requires an interdisciplinary approach. Missions may have more latitude with selection of a mechanism,
as they may allocate resources for activities that they anticipate obligating through a centralized
mechanism (and is therefore not built into the Mission’s activities’ funding lines) and resources for
research are often set aside as a country approaches a new CDCS period, which could be a critical
window in which RTAC could offer services. 
When unobligated short-term research funds become available, ease of contracting is a critical factor in
the selection of a mechanism, putting as little burden on the technical office as possible. The ease of
buying in reported by RTAC technical clients may give RTAC an edge over other mechanisms.
Finding 23: The RTAC team has undertaken a range of strategies to reduce any lags in start time as a
result of institutional contracting under Task 7.
Both USAID and NORC have taken most of the steps they can to reduce lag time between selection and
contract start. The COR works with the USAID Contracting Officer to pre-authorize spending for the
contract modification, and NORC subsequently pre-authorizes spending for the researcher’s institution.
NORC also has made it a practice to send all contract terms in advance for review; however, some
universities are more responsive than others. University contract officers may also identify new
concerns about contract terms that have to be addressed once they see the full scope of work to be
executed.
Finding 24: Among those research buy-ins under Task 7 that have met or exceeded expectations by
both USAID and the researchers, clarity of purpose, prior experience with USAID or government
agencies on the part of the researcher, technical expertise well matched to the scope of work, and
effective communication between USAID and researcher at key points in the research process were
the key enabling factors.
Clarity of purpose, prior experience, and effective communication were discussed above, but one
additional factor, technical expertise was also critical. Those researchers whose expertise was exactly
aligned with the scope of work had better experiences, and USAID staff were happiest with those
deliverables. Among U.S. researchers, there seemed to be a sense that studies that are relevant but not
a perfect match for their own academic interests were not terribly useful for them in a wider sense. As
one researcher whose expertise was not perfectly aligned with the scope of work indicated, “No
academic wants to do a lit review for the sake of doing a lit review…if it doesn’t build to something
useful for his research.” The non-U.S. researchers, however, seemed to be somewhat less concerned
about a funded study deviating slightly from their own bodies of work.
RTAC Mid-term Performance Evaluation
February 25, 2021 24
Finding 25: The RTAC team has adapted its staffing under Task 1 to accommodate a higher contracting
workload and broader administrative support.
NORC, with USAID’s approval, increased the time allocated for contracting support in response to the
need to issue institutional subcontracts rather than individual consulting agreements. At USAID’s
request NORC also expanded the staff support for the Research Director (both administrative and
editorial) to assist with tracking deliverables and editing deliverables prior to their submission to USAID.
Key Barriers across RTAC Tasks 
Finding 26: Lack of clarity around expected deliverables was a key barrier for several of the tasks.
Several PIs and consortium members highlighted the lack of clarity on the scope of work requested, or,
in some cases, perceived changes to scope or scope creep. Any requirements for deliverables
contracted under a firm fixed price arrangement should be specified in detail in the RFP and the contract
for the implementer to cost it appropriately. Negotiating issues arising from lack of clarity in scopes
represented a significant time burden for NORC staff and for USAID activity managers and the COR
and required CO review and approval when official scope modifications were requested.
Finding 27: Turnover in USAID technical activity manager role contributed to difficulties, particularly 
where scopes were not well defined.
In almost every case, across the board for Tasks 1-7, those USAID staff who were unsatisfied were not
the staff who were responsible for writing the original scope or terms of reference. While turnover is
built into USAID’s approach to doing business, this seems notable, and going forward, it might be useful
to have a protocol for working through those transitions to ensure as positive an experience as possible
for all parties. (However, turnover in the USAID technical activity manager role was not associated with
a reduction in satisfaction among NORC staff, consortium members, or PIs.)
Finding 28: A lack of effective communication at multiple levels was a barrier to smooth operations
and stakeholder satisfaction.
While all USAID technical clients felt that communication was strong at the beginning of their activities,
almost all noted a change. They felt that communication dropped off after activities began, and five
noted that communication about who was responsible for what at what stages was also lacking across
several tasks. Similarly, four consortium members identified weaknesses in internal project
communications that inhibited their understanding of how all the activities fit together. These
perceptions differed from NORC staff members’ perceptions. NORC staff described extensive efforts to
communicate clearly with USAID and subcontractors (which included both consortium members and RN
members); this indicates that some of those efforts were not effective and additional communication
strategies may be needed for certain audiences. 
Almost all USAID technical clients indicated that they would like to have direct contact with the
researchers or consortium members working on their activities. That was typically the case with
consortium members, but less so with the researchers. Some researchers were not interested in having
contact with what they perceived to be multiple layers of clients – NORC, USAID RTAC activity manager,
USAID technical client, and sometimes, an additional USAID technical expert. They foresaw, not entirely
incorrectly, that they were likely to receive multiple streams of input, some of which might be
RTAC Mid-term Performance Evaluation
February 25, 2021 25
contradictory. NORC also has a very legitimate concern about direct communications between
subcontractors and USAID that relate to scope and deliverables in which they might not be included, as
subcontractors are not in a position to accept any changes to scope or deliverables without NORC and
USAID COR or CO approvals.
Another concern raised about communication was the tone of e-mail communications some USAID staff
received from the RTAC leadership. They felt that the communications were directive, such as an
employer would use with an employee, not appropriate for a contractor to a client. These e-mails were
generally prompts for information needed or requests for feedback on deliverables. 
A reciprocal concern was identified in the tone of some of the USAID comments that were provided as
feedback on deliverables submitted by consortium members and researchers. In several cases, NORC
could not pass on the inline comments USAID M/B/IOs provided on deliverables without offending the
authors, possibly to the point of having them refusing to make the desiredchanges.
Finding 29: More than half of USAID technical clients and activity managers (52%) perceived RTAC to
lack a customer service orientation.
Several USAID respondents reported that they experienced a distinct lack of customer orientation in
their interactions with NORC. They felt that there was no real concern for managing the user
experience, particularly when there were performance issues. Some also alluded to a lack of initiative
on the part of NORC, particularly around requested/potential changes to scope. Four respondents
indicated that they would have been amenable to increasing the budget for the activity if they could
have made changes to their scopes, but that was never introduced as an option.
Finding 30: RTAC was not well known outside of the Global Development Lab, and only a few other
M/B/IOs have bought into the mechanism.
Although 50% (four) of the research buy-ins included in this evaluation were funded by units other than
the Global Development Lab, one final broad barrier to providing maximum value to USAID was that
RTAC and the services it offers were not well known throughout USAID, although there are obvious
drawbacks to the potential of creating demand that available resources could not satisfy. NORC’s
original outreach plan centered largely on information sharing through trainings that were planned for
M/B/IOs, however, the trainings did not proceed as originally anticipated. Therefore, the RTAC team
had little opportunity to market the mechanism. RTAC consortium members were eager to assist with
marketing RTAC services, but the USAID program managers noted that the ceiling on the contract would
limit the number of new buy-ins that could be supported. Further, the Global Development Lab needed
to ensure that there would be sufficient capacity for its own short-term research needs over the
remaining life of the mechanism and therefore did not indicate a need for aggressive marketing by the
NORC/RTAC team.
Key Barriers for Specific Tasks 
Finding 31: The planned level of effort for each R2A plan or communications product was not always
sufficient for optimal outcomes.
While one respondent who coached researchers through the development of an R2A plan thought the
time allocated was sufficient (after initial misgivings), others did not. One reported that having enough
RTAC Mid-term Performance Evaluation
February 25, 2021 26
time – a week versus a couple of days – improved the ability of the research-to-action process to meet
researcher needs of collaboratively learning the best way to craft messages for the specific audiences
being targeted.
Finding 32: Supporting international travel and visa acquisition in particular required more lead time
than the RTAC team had anticipated under Task 5.
In two of the convenings, some invitees were ultimately unable to attend as they had insufficient time to
obtain visas and organize their travel. The RTAC team learned from this experience, documented it in
their after-action reviews, and made plans to ensure that it would not be a problem in future
convenings.
Finding 33: Some consortium members felt there was limited collaboration in the design of the
convenings and noted the need for coordination of logistics and content from the beginning.
Consortium members noted that they would like to be included in the drafting of the scope of work to
limit the number of unforeseen challenges that may arise in the planning and delivery of convenings.
Another lesson learned from the convenings was the need to have the logistical and content planning
coordinated from the beginning. Some consortium members felt that in the first convening, the
individuals responsible for content had too many responsibilities to also support logistics and that
logistics specialists were engaged too late in the process. In the third convening, the logistical specialists
were engaged earlier to facilitate smoother planning processes aligning content and logistics.
Finding 34: Significant divergence in understanding of the scope of work and expectations for the
trainings caused a series of challenges in developing training modules under Task 6.
As part of the core tasks specified in the RFP, the response in the proposal was fairly short given the
page limitations, and NORC and consortium members had to make several assumptions about the
scope, content, and audience based on what information they had at the time. As noted under
Evaluation Question 2, the RTAC team and USAID disagreed on the interpretation of the data collected
for the landscape analysis. A further misunderstanding about the extent of direct USAID involvement
and direction on the part of one consortium member exacerbated the situation. To resolve the
situation, direct engagement with the activity managers provided an enabling environment to manage
these changes and client expectations throughout the process. Being very clear about agreements and
putting things in writing greatly aided with this process.
Finding 35: RN members who responded to the evaluation survey ranked five key barriers that would
prevent them from bidding on research studies under Task 7: (1) short proposal time; (2) tight budget;
(3) inflexible timing of the study; (4) conflicting commitments; and (5) the use of firmfixed price
contracts.
The most critical factor that RN members said would prevent them from bidding on a research
opportunity was a short proposal time. For simpler scopes, a short proposal time may be less of a
barrier, but researchers (and other USAID partners) may often incorrectly view a short proposal time as
an indicator that a particular individual or group is preferred and that the competition is for form only.
Unfortunately, many of RTAC’s solicitations have had a very short proposal time, and little if any
advanced notice during which interested researchers could start preparing a response. This factor alone
may be a significant contributor to the fact that NORC has often had to recruit PIs outside the RN.
RTAC Mid-term Performance Evaluation
February 25, 2021 27
The other four factors- tight budget, inflexible timing, conflicting commitments, and use of firm fixed
price contracts - all ranked very closely together. Tight budgets and firm fixed price contracts6 may be
most worrisome for researchers who have not worked with USAID in the past and who worry about the
level of risk they accept under such an arrangement. Inflexible timing and conflicting commitments 
reflect the fact that very talented researchers tend to be busy, and they are wary of making
commitments they cannot keep. For RTAC’s purposes, however, these pose a challenge as most of the
rapid research scopes of work have had tight budgets, limited flexibility in timing, and were issued as
firm fixed price contracts (although other types of contracting agreements are now also being used, and
USAID heavily encourages use of cost-reimbursement (CR) subcontracts under Task 7 when university
policies allow and the scope dictates CR as an appropriate approach).
It is also useful to note that only 46.7% of survey respondents agreed that solicitations provide clear
information about USAID’s information needs. Although another 36.8% somewhat agreed that the
solicitations provide clear information, given that short timelines for proposal preparation were
identified as a key barrier, it is unlikely that those who only somewhat understand the information
needs would prepare a proposal. A likely result is that fewer than 50% of RN members in relevant
disciplines would even consider submitting a bid on a given solicitation. For some solicitations this
might cause no difficulties, but for others, this could result in very few to zero applicants, which reflects
RTAC’s experience with some buy-ins. The major concern with this result is that it strains timelines that
were already short and requires additional effort on the part of the RTAC team to recruit prospective
candidates.
A similar issue exists for clarity of data rights. Only 50.0% agreed that information about data rights is
clearly delineated in RTAC solicitations. Given the short turn-around times for submitting responses to
solicitations, that represents another potential reduction in the pool of respondents. Although clarity
about data rights was not a high-ranking barrier for survey respondents, and despite the inclusion of
what seemed to USAID, to NORC, and to the evaluation team to be clear statements about data rights in
the most recent solicitations, another 27.6% somewhat agreed that information about data rights is
clearly delineated in RTAC solicitations, and the remainder disagreed. This lack of understanding may be
the result of inexperience with USAID and/or USG contracting and a reliance on a sponsored programs
office within the university to negotiate terms, but that might be a barrier for someone who would
otherwise be interested in applying for a research opportunity. The evaluation team was not able to
identify an optimal target for the percentage of respondents finding the solicitations and data rights
clauses to be clear, but RTAC should monitor response rates in the future and gather additional
information about critical barriers when solicitations receive fewer than three bids.
Finding 36: The need to recruit PIs for Task 7 research studies outside the network constitutes an
additional burden on the Research Director and may inhibit wide-spread recruitment.
All the PIs for the research buy-ins included in the evaluation were recruited outside the network. This
represents an investment of time on the part of the Research Director to identify the right types of
6 Although the RTAC contract permits cost reimbursement subcontracts, some universities’ own policies prohibit
accepting contracting on a cost reimbursement basis. Thus the source of the concern may be a university’s
contracting regulations rather than USAID’s contracting regulations.
RTAC Mid-term Performance Evaluation
February 25, 2021 28
candidates for each study. However, there is a disincentive to recruit more than two or three
candidates for each given the cost of recruitment time and opportunity cost for the potential applicants,
particularly those who do not have experience with competitive contractingprocesses.
Finding 37: Quality standards and intended audiences were not well understood by some buy-ins’ lead
researchers. 
Early on, NORC and some USAID technical managers assumed that all university-based researchers know
how to write effectively for policy audiences. As USAID heavily critiqued multiple early deliverables for
not clearly addressing the research questions for that audience, as well as significant editorial failings,
NORC added support to assist the Research Director in reviewing and, where necessary, editing
deliverables. Despite these investments, however, some deliverables from Year 2 buy-ins have had
similar issues. 
Finding 38: Task 7 deliverables to date have not, on the whole, been quick or agile in response to
USAID needs.
Almost all deliverables under Task 7 (at the time of data collection) were submitted after some delay.
Sometimes the ability for RTAC to be agile in responding to needs was at times hindered by the slowness
of USAID processes or clients giving conflicting feedback from multiple viewpoints, making it slower to
come to consensus on what should be the best way forward on revisions. One consortium respondent
reported that international researchers needed more support to develop a research report tailored to
USAID expectations, especially if they had not worked with USAID before, and that this hindered the
process of deliverable approval.
In addition, some USAID and in-country partner respondents noted that the final reports were not
necessarily the most useful format for getting the information they needed. Similar to the observations
of the R2A respondents of effective approaches, a few USAID stakeholders indicated that interactive
presentations where questions could be asked and answered were sometimes viewed as more effective
than even short, well-written summaries.
Finding 39: In several cases, USAID personnel felt that the researchers selected for Task 7 buy-ins were
overcommitted and had not allocated sufficient time for the activity.
Any shift in the timing of activities, including submission and review of deliverables, typically
exacerbated time and commitment issues for both USAID and researchers, as windows of availability
were missed and that lengthened some project timelines considerably. This may be an inherent risk of
engaging academics in short-term research but contributed to dissatisfaction on both sides. 
Finding 40: The RTAC team has almost no opportunity to gather information that would assist with
early planning in anticipation of customer needs.
RTAC has very little warning of when new tasks will arise and what the support needs might be. While
several consortium members and RN members noted that it would be very helpful if RTAC could make
information about potential activities available earlier, USAID and NORC program managers confirmed
that it is not possible at this time. This is partially because the LAB/CDR has no way to predict when new
M/B/IOs will reach out to CDR with interest in buying in, nor what the technical or geographic areas of
need they present may be. The inability to anticipate needs requires the RTAC team to function in a
reactive mode constantly, which is not optimal.
RTAC Mid-term Performance Evaluation
February 25, 2021 29
Finding 41: RTAC’s inherent structure for research buy-ins inhibits the RTAC team’s ability to build
client relationships and trust over the course of activities.
Given that most of the research buy-ins are of short duration, the relationship between USAID technical
clients and the RTAC team (NORC program managers and researchers) may be highly transactional. In
such an environment, it is difficult to build trust over the short-term, which can cause difficulties when
there has been a misunderstanding over scope or quality issues with a deliverable.
Finding 42: Consortium members and research partners may be inflexible due to the use of firm fixed
price contracts.
Consortium members and research partners may not be willing to accept adjustments to scopes, as is
typically USAID’s experience with its implementing partners, owing to the frequent use of firm fixed
price contracts. In general, USAID contractors make every attempt to keep the client happy, juggling
talent and other resources to do so. However, for short-term technical assistance, partners may have
limited options for substituting talent or adjusting research questions, for example. 
Finding 43: Several consortium members felt their organizations’ talents were underutilized in the
RTAC project.
Five consortium members indicated that their organizations brought great strength to the RTAC team,
but that given the scope of activities and the limited involvement they have had, their resources were
underutilized. One consortium respondent reported that this was partially the result of challenges in
the project team’s internal management practices. Specific examples included a lack of internal project
management systems, irregular or poorly structured team meetings,and unclear lines of accountability.
FINDINGS: EVALUATION QUESTION 4
As noted in the Introduction, ultimately, the aim of RTAC is to support USAID’s drive to increase the use
of evidence-based decision-making in its own programming and by USAID partner country counterparts
in their policies and programs. The final evaluation question, therefore, is: To what extent has the RTAC
helped USAID Missions, Bureaus, and Independent Offices and affiliated counterpart NGOs and host￾country governments make evidence-based decisions? 
Task 4: Compose Publications and Products 
Finding 44: RTAC Communications and R2A products are being used to make evidence-based
decisions.
RTAC has helped local governments to make evidence-based decisions through the communications
products and R2A plans, according to PI focus group respondents. One respondent reported that
disseminating the results of their research allowed provincial governments to look at incorporating
tenure rights into the land legal system using the researcher’s evidence to inform policy. This
respondent’s project was also invited to disseminate results in nearby countries to discuss whether it
could be implemented in other contexts. Another respondent mentioned that their research
dissemination allowed local agricultural ministry workers to distribute materials on techniques using
compost and biochar to extension agents, who then delivered training or materials to farmers in the
RTAC Mid-term Performance Evaluation
February 25, 2021 30
fields, improving crop yield according to the respondent. A few PIs indicated that their results and
research communication products were being used in discussions about local and national policy
changes but could not yet cite its use in decision-making, as they were in the early stages of those
processes.
RTAC helped with this process by working with researchers and trying to determine the ideal world
where their results are being used and to identify their goals, audiences, and key messages, which then
leads them to deliver the evidence to policy makers in a more usable format tailored to the big picture,
according to one consortium respondent.
Finding 45: The most common way of disseminating research results to stakeholders was through
presentations.
PIs involved with the communications products and R2A plans discussed how evidence was shared. For
example, in Indonesia, as part of the R2A plan, results were disseminated to various levels of
stakeholders (hospitals, Ministry of Health, District Health Office, and Provincial Health Office) through
presentations tailored to each stakeholder group focused on the key areas where maternal health care
can be improved using Kangaroo Mother Care based on each stakeholder’s sphere of influence. 
Finding 46: Research utilization under Task 4 has been hindered by the pace of policy or regulatory
change and lack of capacity within responsible agencies.
A key assumption of the R2A plans and communications products highlighted earlier is that potential
audiences for the information will be receptive to it and will act on it under the right circumstances.
Almost all R2A researcher respondents indicated, however, getting stakeholders to use research results
for policy making was a slow process, hindered by the pace at which government offices change policy
or regulations. In addition, staff turnover at potential key beneficiary agencies sometimes called for the
need to start the engagement process all over again. Lack of capacity in partner organizations may also
hinder the pace of change. For example, a provincial health office in Indonesia requested an additional
brief or manual to facilitate changing its official policy to incorporate findings from the USAID-sponsored
research into certified health care worker trainings but developing manuals for provincial health offices
was beyond the scope of the activity.
Finding 47: Academics do not often have the time readily available once research has been completed
to advocate for research utilization among decision makers given their other professional
responsibilities, which is a key assumption underlying the communications products and R2A plans.
While the PIs of the R2A and Communications activities believe strongly in the utilization of research
findings, they are academics with other responsibilities, and their positions do not generally give them
the time to campaign for long periods of time on one piece of research. At some point, the evidence has
to be taken up by other actors, and the decision-makers may not be ready for it at the point at which
the PIs must return to their other duties. Therefore, there may be a need to consider how to work with
other advocates or to otherwise continue the dissemination of information with a lower level of effort
required from the researcher. For example, two communication product recipient respondents cited
the need for strategies to disseminate results and lessons learned through mass media, where the
information could reach a much wider range of potential users.
RTAC Mid-term Performance Evaluation
February 25, 2021 31
Task 7: Additional Research and Technical Assistance 
Finding 48: As most of the first round of research deliverables under Task 7 were just being finalized
at the time of data collection, the evaluation team is not able to fully address the extent to which
these products were used in decision-making.
Given the timing of the evaluation’s data collection and the delays experienced in the finalization of
deliverables for several of the earliest research buy-ins, most evaluation respondents could not speak to
the utilization of the evidence generated by most RTAC research activities. Initial delays in deliverable
submission were aggravated by the curtailment of activities due to COVID-19, and all in-person 
information sharing activities were postponed prior to the data collection period. Those who could
speak to research utilization said the evidence generated from RTAC-led research has been used by
USAID to inform programming.
For example, USAID staff at one Mission indicated that they had begun to ask different questions about
their activity and participants as a result of the RTAC buy-in, and the research findings essentially revised
the theory of change underpinning one of its programs because the dynamics and interaction of
environment and intervention at the community level were playing out differently than anticipated.
One respondent indicated that future programs (and their MEL systems) may be structured differently
to use more effective approaches to achieve their objectives. 
One exception was the Mekong Political Economy Analysis, which was one of the earliest research
activities (technically conducted under Task 4, but more similar in format and structure to Task 7
activities than the other Task 4 activities). As one interviewee noted, “A bunch of Vietnamese media
picked up on it [the report], and there were a bunch of stories that were summarizing the report…and
even like an op ed in the People’s Dai ly about it.” However, as this activity was not aligned with the
Mission’s current portfolio, the evaluation team was unable to determine whether the media attention
resulted in (or, in fact may have resulted from) intentions by key stakeholders (USAID, governments,
NGOs, etc.) within Vietnam or other countries in the region to change policies orprograms.
Finding 49: M/B/IOs reported that they expected to use RTAC products for policy, strategy
development, and project design decisions.
When asked about the expected uses of RTAC products, most USAID respondents indicated that the
primary usage would be for internal advocacy supporting the goals of the relevant M/B/IO. Some USAID
respondents reported that they also intended to use the evidence provided in learning activities with
the implementers that support their activities; to influence policy; to launch requests for applications
(RFAs); and for drafting academic papers (with permission). Actual use was aligned with the
expectations of the respondents but did not fully align with the expected use cited in the original RFP,
which included dissemination to parties outside of USAID. While external dissemination of research
results might occur, it became clear that dissemination plans were typically only formulated after USAID
had the results of the research. Personnel from both the Mali and Kenya Missions indicated that they
planned to use final reports from their activities mainly for internal purposes. Outside of the Mission,
they thought they might share the final report with international non-governmental organizations and
some partners, but did not anticipate broad, public distribution.
RTAC Mid-term Performance Evaluation
February 25, 2021 32
Finding 50: Some respondents found interim briefings to be more useful than final deliverables
because it enabled them to leverage the researcher’s expertise in a targeted way.
Two end-user stakeholders noted that presentations at various stages of the research were very helpful
in allowing them to get to the core information they needed in a timely format. One went on to say that
although the research results presented had not shifted decision-making, it did influence her to begin
asking different questions about how programs could best serve participants. The shift she described
indicates that the research was effective – even if there is no change to programming, new areas of
inquiry will ultimately generate more evidence and inform program adaptations.
Task 1: Management 
Finding 51: As of the time of data collection, RTAC’s MEL Plan did not include any approaches for
measuring and collecting data on the utilization of research results. 
It is often the case that indicators and metrics drive activities. In this case, despite having an indicator of
research utilization, no concrete plan for collecting data on research utilization was included in the MEL
Plan originally, although the RTAC team had the development of a data collection strategy as an item in
its work plan for 2020. One challenge with the lag in adoption of an indicator, however, is that data on
utilization for the early R2A and communications products developed under Task 4 were not being
tracked systematically. Two consortium respondents reported that the role of RTAC was to deliver the
plan, but not to do follow up or monitoring, so the awareness of the R2A plan’s results was not
guaranteed. While consortium members, according to one respondent, would always let the R2A and
Communications PIs know they were very interested in hearing of how the plans’ implementation was
going, there were no clearly defined points at which research teams would report results to RTAC (as of
the data collection period for this project). Some of these concerns may be relieved with the addition of
subtask 4.C.2 - conduct an evaluation and develop a recommendations report of the process and
efficacy of the R2A planning process, which was added to the year 3 RTAC work plan.
Another challenge is that to date, most of the Task 7 research activities have not included a
dissemination and utilization plan. The USAID M/B/IOs and affiliated counterpart NGOs, and host￾country governments do not always track how they have used research results within programming or
decision-making.
A secondary but related issue is that DevResults, the platform used for data management, does not
allow RTAC to track Task 4 outcomes, as the funding for the original research (receiving TA under Task 4)
originated from other CDR programs (not RTAC). It may be possible to customize the database further
to accommodate a new indicator for outside funding sources to enable tracking, but if not, the system
itself inhibits good results tracking in this case.
CONCLUSIONS
This evaluation provided the opportunity for reflection on the successes achieved and the challenges
experienced under RTAC to date, not only by the evaluation team, but also by USAID and the wider
RTAC team as well. Based on the information shared by the evaluation participants, the evaluation team
has drawn a number of conclusions.
RTAC Mid-term Performance Evaluation
February 25, 2021 33
The summary response to Evaluation Question 1, to what extent has the goal of building an
institutionally and individually diverse partner network been met, is that NORC and its consortium
members have exceeded all expectations in building a diverse network, surpassing all goals set by USAID
in the RFP and those set in the MEL plan for the project. Very few barriers exist to those academics
affiliated with recognized institutions who would like to join.
The examination of evidence relation to Evaluation Question 2, to what extent does the training and
technical assistance provided by RTAC meet the needs and priorities of project stakeholders (such as
M/B/IOs), indicates that some RTAC activities have excelled at meeting the needs and priorities of
stakeholders while others have failed to meet their needs at least as of the time of data collection for
the evaluation. The types of training and technical assistance that have been most successful in
meeting the needs of project stakeholders are the R2A plans and communications products; research
buy-ins with straightforward, fairly narrow scopes of work; and convenings.
The key barriers and enablers that made the difference between successful and disappointing
implementation and results were many. Clear, consistent scopes of work, prior experience with USAID,
and clear communication with USAID at critical junctures were key enablers across tasks. Conversely, 
lack of clarity around expected deliverables, turnover in USAID staff (and therefore changing
expectations), and a lack of effective communication strategies were barriers across activities to strong
performance. Several other enabling factors were critical at a Task level, including the following:
● Selection process, co-creation process, and timely execution of the activities under Task4;
● Effective use of feedback loops and learning processes, effective information sharing processes,
and early identification of travel requirements under Task 5;
● Engagement of the COR in mediation and of USAID subject matter experts working on Task 6;
● Ease of the buy-in process, management approaches to reduce start up time, andtechnical
expertise well matched to scopes under Task 7; and
● Adaptations to staffing to increase capacity for contracting and administrative support in Task 1.
Similarly, a range of factors constituted barriers to successful performance at the Task level, including:
● Planned level of effort under Task 4;
● Lead time required to support international travel and limited collaboration in planning,
particularly for the early convenings under Task 5;
● Significant divergences in the understanding of the scope of work and expectationsfor trainings
under Task 6;
● Inhibitors to bidding on research opportunities (particularly short timelines and tight, inflexible
budgets), recruitment of PIs outside the RN, misunderstood quality standards, time delays,and
overcommitment of some researchers under Task 7; and
● RTAC’s design, which prevents the team from anticipating client needs, limits their ability to
build client relationships, limits flexibility as a result of firm fixed price contracting, and has
resulted in perceived underutilization of partner capacity within the wider management ofthe
RTAC project (Task 1).
Despite the challenges, the response to evaluation question 4, to what extent has the RTAC helped
USAID M/B/IOs and affiliated counterpart NGOs and host country governments make evidence-based
decisions, is that there is evidence that the R2A plans and communications products are beginning to
influence partners and host country governments. Presentations, rather than reports or other
RTAC Mid-term Performance Evaluation
February 25, 2021 34
documents, seem to have the greatest impact on those audiences. Most of the research funded under
RTAC seems to be intended for internal use. As all of the early activities had had delays, the evaluation
team was not able to elicit information about the utility of the reports, but some M/B/IO respondents
indicated that the presentations given by the researchers at the conclusion of data collection were
exceedingly helpful because they were able to access the information they needed in a timely manner.
Some also shared that although no programmatic changes had been made as a result, they were asking
different questions of their activities, which indicated a deeper understanding of the context and
potential impacts.
The evaluation team therefore concludes that RTAC has been quite successful in some of its activities.
However, in examining the ToC underpinning the project, it appears that several assumptions cannot be
fully validated at this point. As discussed earlier, the ToC states:
If RTAC provides global, diverse, and qualified academic researchers with expertise in a
wide variety of topics who can be quick, agile, and responsive to USAID needs, and if 
research is translated for use and dissemination and training is provided to USAID and
partners to support research and use of data, then USAID and development 
stakeholders will have an increased capability to make timely, sound, evidence-based
decisions that reflect a broad range of knowledge and perspectives. 
RTAC has provided global, diverse, and qualified academic researchers with expertise in a wide variety of
topics. However, it was originally envisioned that a key benefit of the RN would be in aggregating and
providing readily available access to that group of researchers. Despite the RTAC team’s success at
assembling such a network, recruitment of PIs has consistently had to be done outside the network, and
existing RN members themselves identified a range of conditions, most of which are present in each
solicitation, that would inhibit them from bidding on a study. That, therefore, highlights the question of
the realized value of the RN and any further expansion of the RN, particularly in light of the relatively
high degree of dissatisfaction and unmet expectations reported by RN members.
The next factor in the ToC is the provision of services that are quick, agile, and responsive to USAID
needs. Given the learning curve for some researchers in writing for a policy-maker audience, as well as
the delays (on both sides) and occasional overcommitment by researchers, the deliverables produced by
RTAC-funded research to date have not been particularly quick or agile. However, given that for many
scopes, USAID intends the information to be used internally, a greater focus on presentations and
internal memos rather than reports for DEC publication might provide as much value with less
management and oversight required. The need for agility may also require shifting some activities from
firm fixed price contracting to other forms, such as cost plus fixed fee or time and materials, if/as
university policies allow. 
Research translation has been fairly effective thus far. The R2A and communications products have
been well received, as have the briefers and other distillations of RTAC-funded research reports.
The next element of the TOC is training, specifically on the use of research and data. The focus of RTAC
training was modified from use of data to USAID’s research policy, and training of USAID and partners
has been delayed. Despite this shift in focus and the delay, some M/B/IOs are utilizing the findings of
the early research buy-ins in their planning and programming. The TOC includes “training is provided to
USAID and partners to support research and use of data” as one of its conditionalities, but it would
RTAC Mid-term Performance Evaluation
February 25, 2021 35
appear that training on data use may not be a critical element of the ToC, as M/B/IOs are using the
information and data provided in the absence of RTAC-provided training.
The only remaining element of the ToC to be addressed is RTAC’s agility and rapidity of response to
USAID needs. While activities under Tasks 4 and 5 were conducted in a timely manner, some activities
under Tasks 6 and 7 were less so. The gap to be addressed before the validity of the ToC can be tested,
therefore, is consistency in the agility and rapidity of response to USAID needs.
RECOMMENDATIONS
Drawing on the specific findings and conclusions discussed above, the evaluation team developed a
roster of recommendations for USAID and the RTAC team. Over the course of two recommendations
workshops, several of these were refined jointly with USAID and the RTAC team. As of the time of
publication, several of these recommendations had been implemented by USAID and the RTAC team.
Key Recommendations are marked in bold.
Task 1 
Recommendation 1.1 (NORC): Continue using the CLA approach to ensure continuous improvement, as
it represents a key strength of the management of RTAC. (Findings 10, 23, 25, 43)
Recommendation 1.2 (NORC): Ensure that RTAC staffing and subcontracting plans do not overburden
one person/partner. (Findings 11, 12, 13, 14, 15, 28, 29)
Key Recommendation 1.3 (USAID and NORC): Consider splitting the Research Director position into two:
one that oversees the administration of the activities and communication with partners and the other
that provides expert methodological and feasibility review of scopes, guidance for partners lacking
USAID experience, and technical writing and editing, as needed. (Findings 12, 13, 14, 15, 28, 29)
CLA Update: NORC and USAID have undertaken some general adaptations to increase communication,
consistency, and quality across RTAC activities. One notable adaptation is the delegation of some tasks
under Task 7 to NORC supporting team members to spread out the workload and alleviate pressure on
the RTAC Research Director. Another is the inclusion of a RTAC/CDR FO “special activities” fund that can
be leveraged if/when small adaptations are needed to budget/scopes mid-buy-in to increase flexibility. 
Task 2 
Recommendation 2.1 (NORC): As all targets have been met, cease active recruitment into the RTAC RN
unless additional activities will be planned to engage members. (Findings 1, 3, 4, 35, 36)
Recommendation 2.2 (NORC): If additional recruitment is needed to maintain balance on targets for
MSIs, develop a targeted outreach strategy to engage MSIs that have strong research records. (Findings
2, 5)K
Key Recommendation 2.3 (USAID): USAID should conduct a cost-benefit analysis to determine whether
the development and use of the RN was the most cost-effective approach to filling its research needs.
(Finding 4)
RTAC Mid-term Performance Evaluation
February 25, 2021 36
CLA Update: NORC and USAID had a CLA discussion in August 2020 and came to mutual agreement on a
path forward for de-emphasizing active recruitment under Task 2 and re-focusing on capacity building
activities and incentives for existing RN members.
Task 3 
Key Recommendation 3.1 (USAID and NORC): Consider expanding resources for RN members on the
RTAC HESN site to include the following (Finding 3):
● Space to express interest in teaming on proposals
● Training resources (how to read/respond to RTAC RFPs, writing about research findingsfor
donor/policy audiences, etc.)
● Live webinars on various topics (multiple arcs – RTAC/USAID contracting, applied research 
methodologies, USAID sector-focused, etc.)
● Space to share research 
CLA Update: As a result of discussions between NORC and USAID following an initial recommendations
workshop, examples of new activities planned for Year 3 include: featuring RN members on the RTAC
website, conducting internal webinars for RN members, and supporting selected RN members’
participation in professional conferences.
Task 4 
Recommendation 4.1 (NORC): Continue to use the selection process and R2A/product co-creation
processes that have been successful for participants to date. (Findings 7, 19)
Recommendation 4.2 (NORC): Ensure that future R2A plans include more concrete timelines and metrics
against which progress can be measured. (Finding 51)
Recommendation 4.3 (USAID): Explore the possibility of collaborating with Missions to provide support
for additional gaps in the uptake of research. (Finding 47)
Key Recommendation 4.4 (NORC): Add indicator(s) to MEL plan to capture influence of R2A plans and
communications products, aggregating metrics adopted within R2A plans. (Finding 51)
Recommendation 4.5 (USAID): Consider expanding the availability of research translation support
services to other USAID M/B/IOs if the ceiling can be raised to accommodate additional buy-ins.
(Finding 4)
CLA Update: NORC plans to continue to use its successful selection process for R2A and communications
products. NORC has adopted an approach to aggregate data for its indicator on utilization of research. 
In Year 3, RTAC will conduct an evaluation of the R2A plan activity to learn more about the 
implementation of the plans and how effective the activities have been in achieving the desired
outcomes. Per USAID’s request/revised and CO-approved scope of work for year 3, RTAC is also building
a research translation toolkit to provide tutorials for how to generate communications products,
stakeholder analysis, and other products, which addresses some of the wish list items identified byPIs.
Task 5 
Key Recommendation 5.1 (USAID): Consider relaxing the scope of work for Task 5 to include additional
virtual expert panels organized at low cost as a way to engage RN members. (Findings 3, 4, 8) This
RTAC Mid-term Performance Evaluation
February 25, 2021 37
would enable USAID to get greater value out of the RN on a variety of technical topics, and RN members
expressed great willingness to serve on expert panels and engage with USAID and their peers.
Recommendation 5.2 (NORC): Ensure that the planning process allows sufficient time for internal
approvals and coordination of logistics, particularly where visa support and other travel support will be
required. (Findings 9, 20, 32)
Recommendation 5.3 (NORC): Involve partners early in the planning process to ensure that logistics and
content are both planned out in parallel. (Findings 9, 20, 33)
Recommendation 5.4 (NORC): Ensure that outside experts are briefed on the composition and expertise
level of the audience for appropriate targeting of content. (Finding 9)
CLA Update: No further convenings are planned at this time. NORC plans to implement changes in its
process following the findings of its own after-action reports and this evaluation if there is an
opportunity to conduct future convenings.
Task 6 
Recommendation 6.1 (NORC): Follow the agreed upon plan for finalizing training materials currently in
development. (Findings 11, 21)
Recommendation 6.2 (USAID): Specify audience, learning objectives, expected behavior changes, and
expected training duration and iteration in initial scopes of work for firm fixed price activities. If those
elements must be informed by a needs analysis, use a different basis for contracting or provide a plug
figure to estimate level of effort. (Findings 11, 26, 34)
Recommendation 6.3 (NORC): Prior to training design, ensure that specifications for audience, learning
objectives, expected behavior changes, mode of delivery, and expected training duration and iterations
have been approved. (Findings 11, 26, 34)
Recommendation 6.4 (NORC): Ensure that appropriate content experts are engaged for the
development of the training module (i.e., sufficient familiarity with USAID operations, expertise in the
application of content in that environment – may require significant USAID input as content experts).
(Finding 21)
Recommendation 6.5 (NORC): Structure the development of training such that client approval is granted
in stages to reduce risk of dissatisfaction with the final product. (Findings 11, 21, 34)
CLA Update: NORC and USAID are following the agreed upon plan for finalizing training materials in
development.
Task 7 
Key Recommendation 7.1 (USAID and NORC): Update the existing SOW template to ensure that it
contains clear guidance on identifying purpose, audience, deliverable format(s), and criteria for
deliverable inspection (including a USAID E3/ED flow down provision for RTAC contractual clause E.3 on
inspection of deliverables). Provide links to sample deliverables for reference. (Findings 12, 13, 14, 16,
24, 26, 37)
RTAC Mid-term Performance Evaluation
February 25, 2021 38
Recommendation 7.2 (NORC): In the RTAC budget template, provide an instruction sheet that helps an
academic apply as an individual and one that helps a university apply as an institution. Include a
required budget breakdown for the anticipated pre-authorization amount so that few/no additional
budget revisions or recalculations are required for award. (Finding 15, 22, 23)
Recommendation 7.3 (NORC): Add a clearly labeled Assumptions section and instructions for that
section to the proposal template to ensure that interested researchers have clearly identified and
articulated the assumptions underlying the technical elements of their proposals(i.e., timing, access to
existing resources, support to be provided by NORC/USAID, etc.). (Findings 12, 13, 14, 16, 24, 26, 37)
Recommendation 7.4 (USAID and NORC): Continue to use pre-award authorizations in order to reduce
the downtime caused by contract negotiations. (Finding 23)
Recommendation 7.5 (NORC): Continue to request review of all terms by research partner CO prior to
sharing full subcontract. When general subcontract terms and conditions are shared with university
partners prior to final contract negotiations, identify the authorized signatory on their side and request
that review is completed by that person. (Finding 23)
Recommendation 7.6 (NORC): Provide a standard orientation on how USAID operates (relationships
between Missions, Regional Bureaus, and Functional and Central Bureaus), how buy-ins work (i.e., legal
duties and obligations of each party under this structure), and generally how USAID intends to use the
end product to help clarify appropriate format and writing structure. (Findings 24, 37, 38)
Recommendation 7.7 (USAID and NORC): Ensure that each scope of work and work plan for research
buy-ins that will gather new data includes opportunities for USAID, NORC, and the researcher to discuss,
clarify, and finalize/adjust deliverable parameters at key points (Findings 18, 24, 26, 28, 37):
● Kick-off
● Approval of data collection instruments
● Completion of data collection
Key Recommendation 7.8 (USAID and NORC): Include interim deliverables in scopes of work for
research buy-ins that will gather new data that allow NORC and USAID to monitor compliance with
scope (Findings 18, 24, 26, 28, 37):
● Data collection instruments, data analysis plan, and datamap
● Data collection debriefing
● Draft report outline
In CLA discussions, NORC and USAID identified barriers to this including tight timelines and limited
bandwidth by USAID technical experts for review. Those deliverables that were not immediately
acceptable on technical grounds, however, required a significant investment of USAID technical staff
time to review and provide guidance and many weeks for the researchers to revise, all of which caused
delays and extensions to the buy-in timelines. Planning for an additional two weeks in the timeline for
review of data collection instruments and the data analysis plan, as well as a draft outline of the report
would likely have ultimately reduced total time spent by both USAID staff and the researchers and
resulted in fewer delays from the planned timeline. In addition, for those buy-ins that conducted 
debriefings, the stakeholders (both USAID and PI) indicated that they were useful – for USAID to gain
earlier access to the information collected and to be able to probe on specific concerns and for the
RTAC Mid-term Performance Evaluation
February 25, 2021 39
researchers to gain a glimpse of the information that was most interesting or important to USAID.
Therefore, while the addition of interim deliverables may lengthen the planned timeline somewhat, it
should reduce the total time invested overall. 
Recommendation 7.9 (NORC): Consider engaging a writer/co-PI or other designated individual who can
provide quality control and bridge the gap between academic writing and writing for policy makers.
(Findings 17, 24, 38)
CLA Update: NORC and USAID are continuing to use the approaches that reduce subcontracting time. In
response to the discussions of initial findings, USAID and NORC worked together to undertake responses
to a number of these recommendations, including:
● Creation and utilization of a standard Buy-in Kick-Off MeetingAgenda.
● Development of a Buy-in Standard Operating Procedures (SOP) document for buy-in PIsand
activity managers.
● Development of a standard presentation to serve as an Introduction to USAID, CDR, and the
buy-in process that will be provided to new PIs on buy-ins movingforward.
● Use of post-buy-in assessments to gather additional feedback from buy-in partners in order to
support continuous improvement.
In addition, NORC has increased support for additional writing support and quality control.
RTAC Mid-term Performance Evaluation
February 25, 2021 40
ANNEXES
RTAC Mid-term Performance Evaluation
ANNEXES
 ANNEX A: RTAC-MPE Statement of Work 
1.2 The Assignment 
IBTCI proposes to carry out a 19-month mid-term performance evaluation of the RTAC program to identify successes
and challenges in the design and implementation of RTAC, and to better understand the degree to which RTAC is
meeting its objectives. Evaluation data will be used to inform mid-course program adaptations and contribute to
increased understanding of the research needs of USAID M/B/YOs. The evaluation will respond to four questions
provided by USAID:
1. To what extent has the goal of building an institutionally and individually diverse partner network been
met? What have been the barriers and enablers to building a diverse partnernetwork?
2. To what extent does the training and technical assistance provided by RTAC meet the needs and
priorities of project stakeholders? Which types of training and technical assistance have been most
successful in meeting the needs of project stakeholders? (Considering timeliness, relevance, and utility
of products and services).
3. What were the barriers and enablers that made the difference between successful and disappointing
implementation and results of RTAC assistance?
4. To what extent has RTAC helped USAID M/B/IOs and affiliated counterpart NGOs and host-country
governments make evidence-based decisions?
This mid-term performance evaluation is scheduled to take place between approximately September 2, 2019 and
April 30, 2021. While lengthy for a performance evaluation due to the fact that activities are implemented on a
rolling basis, this extended period of time will make it possible for the Evaluation Team (ET) to observe the planning
and development of new activities that take place during the life of the evaluation. This allows the ET to better
assess the effectiveness of the RTAC project model and network in responding to the needs of USAID, and assess
whether and how USAID is using the evidence provided to guide policy development and programming. Our review
will focus both on technical activities and organizational/administrative realities, to capture the ability of RTAC to
reach out to and actually make use of the skills of staff of minority-serving institutions and of higher education
institutions in USAID-assisted countries, and also to ascertain the extent to which RTAC’s products and services met
the requester’s needs and are being used by USAID to make evidence-based decisions.
In addition to data collection to be carried out in the U.S., IBTCI proposes making two six-day visits to two countries
where RTAC activities have been implemented, tentatively Honduras and Uganda, scheduled tentatively to occur
during the second quarter of CY 2020. During these trips data will be collected through KII interviews, a project
stakeholder survey, focus group discussions, interviews with USAID staff, affiliated counterpart NGOs, and host
country governments where relevant. Data collected will address all four evaluation questions.
During the third quarter of CY 2020, IBTCI plans to hold a two-day Recommendations Development workshop
expected to include 20 to 25 CDR and RTAC staff, representatives of the RTAC participating organizations, and CDR
and RTAC counterparts from other USAID offices with research units. Among the themes of this workshop will be an
examination of approaches to increase the participation of U.S. minority-serving institutions and of higher education
41
RTAC Mid-term Performance Evaluation
1
institutions in USAID-assisted countries and an exchange of information among colleagues that can make RTAC and
its partners more effective. 
1.3 Evaluation planning - initial desk review and consultations to develop evaluation design
The IBTCI ET will first develop a draft evaluation design and work plan and will conduct a desk review of relevant
RTAC background materials related to project design, implementation and reporting, including the RTAC RFP and
project agreement; work plans; bi-annual and other reports; the Monitoring, Evaluation and Learning (MEL) plan;
indicator monitoring plans and results; reports from RTAC partners and activities; RTC websites, social media and
marketing materials; gender analyses; RTAC-generated products and reports to date (including those from the
Artificial Intelligence and Public Health Activities); and any thematic reports from other sources, etc. This review will 
further include a review of the project materials (administrative and technical) for RTAC activities that have been
completed, that may be ongoing, and that may be in the works. In the course of this work, IBTCI’s team will review
relevant materials from the predecessor HESN and PEER projects and from the other CDR programs to best
understand USAID’s objectives and approaches in leveraging the higher education community to improve
development outcomes, and how RTAC fits into this process. Some other bureaus and offices also have research
units; we will review their materials and, with CDR concurrence, will conduct introductory conversations with their
staff to understand their approaches and see if there are lessons of relevance for RTAC. We should note that the
purpose of the document review is not to review the content of technical reports, etc., but rather reviewing the
processes of developing, maintaining, and enhancing institutional partnerships and making the partnerships more
effective partners with USAID.
1.4 Design revision and development of instruments 
After the desk review, the ET will prepare an evaluation design with a more detailed discussion of our anticipated
methodology included in the draft work plan presented in this proposal. This design will show an evaluation design
matrix linking the evaluation questions to data sources, methods and data analysis plans (see draft below in
Illustrative Evaluation Data Collection and Analysis Table). The team will prepare draft collection instruments
including surveys, questionnaires and Key Informant Interview (KII) and Focus Group Discussion (FGD) protocols. In
cooperation with USAID and NORC at the University of Chicago, IBTCI will draft a list of potential interviewees and
identification of sites to be visited, along with a proposed selection criteria and sampling plan. These lists will be
reviewed closer to the actual time of data collection to ensure that all relevant project beneficiaries, partners, host
government officials, researchers and USAID staff are included and that new respondents are added who may have
been identified in early stages of research. The evaluation design will include a sampling and data analysis plan and
a robust data quality assurance plan.
1.5 Proposed methodology (quantitative and qualitative)
The IBTCI ET will use a mixed-method design including quantitative and qualitative data collection techniques
(including surveys of universities participating in network, surveys of USAID staff, key informant interviews, focus
group discussions, and reviews of secondary data described in more detail below. Prior to developing all research
tools, the ET will carry out up to 10 preliminary in-person or phone interviews with key informants (i.e. university,
USAID Mission and Lab staff) to clearly identify priority issues and concerns regarding building and using the RTAC
partner network. These preliminary interviews will help ensure that the evaluation is focused on the issues of
highest relevance and importance to key RTAC stakeholders.
RTAC Mid-term Performance Evaluation
1
Data collection will focus on assessing the relevance and quality of RTAC assistance and products by identifying
which types of assistance were most successful in meeting stakeholder needs/priorities; identifying barriers and
enablers to successful or problematic implementation of assistance; examine what challenges exist and how they
are being addressed; identify tangible results of RTAC assistance; document actual application of RTAC services in
strategic choices and programming decisions; and examine the effectiveness of RTAC in increasing the number of
women researchers and minority-serving institutions actively engaged in the network.
Key methods include: 
• Survey of network participants: An electronic survey (via Survey Monkey or similar) will be sent to all RTAC
network participant organizations. Data collected will focus primarily on evaluation questions 1 and 3 to gather
input on satisfaction of participants with the network, levels of diversity and gender within the network,
partners’ experiences with using RTAC, and barriers or enablers to creating a diverse partner network, etc.
• Survey of RTAC researchers: An electronic survey will be sent to all RTAC researchers who have delivered
technical assistance, products or services. Data collected will assess to what extent providers felt that this
assistance met the needs and priorities of project stakeholders, identify what types of assistance were most
successful and identify successes of and challenges to providing RTAC assistance. The data will address all
evaluation questions. In addition to the survey, the ET will carry out selected interviews with RTAC researchers 
in person or by phone.
• Interviews of RTAC project staff with structured questionnaire: Data collected will identify barriers and enablers
to creating a diverse RTAC partner network and providing USAID with rapid, on-demand evidence, challenges in 
managing the network, meeting performance indicator targets and how they’re being addressed, and what
elements made a difference between the successful and disappointing implementation of RTAC assistance. The
data will address all evaluation questions and be carried out in person or byphone.
• Survey of USAID staff and other stakeholders: A short electronic survey (via Survey Monkey or similar) will be
sent to all USAID staff who have used RTAC services. Data collected will assess stakeholder level of satisfaction
with RTAC products, training and technical assistance delivered, vis-à-vis timeliness, relevance, utility, success,
and the extent to which RTAC assistance metstakeholder needs and priorities. Data will contribute primarily to
evaluation questions 2, 3 and 4. In addition to the survey, the ET will carry out selected, follow-up interviews
with USAID staff person or by phone.
• Key Informant Interviews with individuals with specialized knowledge about RTAC and the services it is providing: 
Data collected will focus on the technical and administrative aspects of implementing activities from the
standpoints of U.S. and international partners, how research topics are developed, feedback mechanisms, and
how such research is being used for better decision making at USAID. Data collected through KIIs will support all
research questions.
• FGDs with RTAC providers, clients/and other key stakeholders of RTAC assistance and services: Data collected
will assess the extent to which user priorities and needs were met with different RTAC products, identify
barriers and enablers to successful or problematic implementation, and pinpoint links between RTAC
services/products and related evidence-based strategic choices or programming decisions. Data will address all 
four evaluation questions.
• Review of project data: The ET will review RTAC partner network data; training and technical assistance
materials developed and used; Performance Indicator Data; any documentation of stakeholder requestsfor
assistance responses, stakeholder or user feedback; any assessment of RTAC services, products or training
provided; and other data deemed relevant. Data collected will address all evaluationquestions.
1.6 Data Analysis 
RTAC Mid-term Performance Evaluation
1
The RTAC data will be reviewed by the team after collection to ensure that it meets industry data quality standards
of validity, reliability, timeliness, precision and integrity, and to ensure the team members are in consensus about
key interpretations of the data collected. Quantitative data from RTAC partner network survey forms will be
checked for invalid responses, number of empty values and email bounce rates for on-line surveys and then entered
into a database for analysis according to response and other parameters related to the evaluation (e.g., gender
disaggregation, geographic location, ratings for timeliness, relevance, utility of RTAC products and services). In order
to analyze the non-numerical and unstructured RTAC qualitative data generated by interviews and focus group
discussions, the IBTCI evaluation team will use Atlas TI software which facilitates coding and analysis of qualitative
data, and allows charting, mapping and sorting and arranging information in order to identify relationships (e.g.
perception of satisfactory/disappointing RTAC implementation among various types of stakeholders) that support
data analysis. It also supports data collected from a wide variety of both qualitative and quantitative formats. The
IBTCI evaluation team will ensure triangulation of data by using multiple methods to collect data on the same topic
and will use multiple data sources to capture different dimensions of the same evaluation phenomenon.
Convergence of information from different sources and methods will support data validity. 
1.7 Sampling Strategy
A detailed sampling strategy will be developed in consultation with USAID and the RTAC implementer. The ET will
use a stratified sampling frame to ensure a sufficient sample based on gender or minority institution affiliation, and
purposive sampling for the two overseas sites with only one or two projects and a limited number of stakeholders.
Purposive sampling would be used to select focus group participants.
RTAC Mid-term Performance Evaluation
1
ANNEX B: EVALUATION METHODS
INTRODUCTION
International Business & Technical Consultants, Inc. (IBTCI) was selected by USAID to conduct a mid-term
performance evaluation (MPE) of the Research and Technical Assistance Center (RTAC) Contract to identify
successes and challenges in its implementation to date and strategies for improving the program in the current
award period. The RTAC project is managed by NORC at the University of Chicago (NORC) and focuses on building an
extensive network of research institutions and individual researchers (based both in the US and abroad) who can
provide short-term technical assistance to USAID Missions and Operating Units through research, subject matter
expertise, and training. The RTAC MPE started in September 2019 and will end in May 2021. The team is using a
collaborating, learning, and adapting (CLA) approach within a mixed-methods evaluation framework to ensure that
information is provided in a timely manner that supports project learning and adaptation.
PROGRAM SUMMARY
The RTAC was awarded to NORC as a firm fixed price contract (contract no. 7200AA18C00057, June 29, 2018 – June
30, 2020, with two option years extending through June 30, 2022) for core activities and cost reimbursement or
incremental firm fixed price for emergent tasks. RTAC was created to provide timely research-based technical
assistance to facilitate USAID’s capability to make quick, evidence-based programming decisions by leveraging the
technical expertise of the higher education community. As of January 2020, the RTAC Research Network (RN)
comprised 503 academics from 154 institutions7 and stretched over 38 countries.
8 Some of the anticipated activities
in the RTAC contract include, but are not limited to:
1. Creation of evidence-to-action policy briefs and technicalreports;
2. Strategic research agenda setting and research activity consultations;
3. Coordination of higher education institution (HEI)-based researchers for expert panels or convenings and sector￾focused trainings; and
4. Provision of additional technical assistance support.
EVALUATIONOBJECTIVES
The findings of the RTAC MPE will be used to inform mid-course program adaptations and contribute to the
increased understanding of the research needs of USAID Missions, Bureaus, and Independent Offices (M/B/IOs). The
7Of these, approximately 46% are women researchers, 16% are researchers affiliated with US minority-serving institutions
(MSI), and 26% are researchers affiliated with developing-country academic institutions (RTAC, 2019). The figure cited
represents the network as a single point in time, however, as it is continuously expanding.
8 Argentina, Canada, China, Colombia, Egypt, El Salvador, Ethiopia, Ghana, Honduras, Hungary, India, Indonesia, Iraq, Jordan, 
Kenya, Mali, Mexico, Morocco, Mozambique, Nepal, Nigeria, Norway, Pakistan, Peru, Philippines, Qatar, Romania, Saudi Arabia,
South Africa, Sweden, Tanzania, Turkey, Uganda, United Arab Emirates, United States of America, Yemen, and Zimbabwe (RTAC,
2019).
RTAC Mid-term Performance Evaluation
1
EQ 2a. To what extent does the training and technical assistance provided by the RTAC
meet the needs and priorities of project stakeholders (such as M/B/IOs, researchers,
Lab/CDR, local governments, etc.)?
RTAC MPE team will use a mixed-method program evaluation design to answer the following key evaluation
questions:
Table 1. Mid-term Performance Evaluation Question #1
Program
Task
Core
Evaluation Question # 1 
Sub-evaluation Questions and Specification Target Population
Core
Core
Core
Core
SEQ 1a. What results was USAID seeking in creating the network?
● Are they being achieved?
● What benefits or incentives does the RN offer toits
members?
SEQ 1b. What are the characteristics of researchers who are in the
network?
● What does their “engagement” look like? 
● What motivated researchers to join the network? Whatdo
they hope to gain?
● What would encourage more researchers to join the RN?
SEQ 1d. Was the RTAC theory of change (TOC) as described in theRFP
and its design appropriate?
● Were key assumptions correct? Should the TOC be revised? 
NORC, USAID staff
managing RTAC
Individual network
participants,
Institution partner
staff
NORC, USAID staff
managing RTAC,
USAID technicalstaff
Table 2. Mid-term Performance Evaluation Question #2
Program
Task
Core and
Buy-in 
Evaluation Question # 2 
SEQ 1c. How does the structure or management of the networkhelp NORC, Consortium
or hinder its effectiveness? members, USAID
staff managing RTAC,
USAID technicalstaff
EQ 1a: To what extent has the goal of building an institutionally and individually diverse
partner network been met?
EQ 1b: What have been the barriers and enablers to building a diverse partner network?
RTAC Mid-term Performance Evaluation
1
Sub-evaluation Questions and Specification Target Population
Buy-in 
Core and
Buy-in 
Core and
Buy-in 
Core and
Buy-in 
SEQ 2a. How is development evidence (i.e., research results) being
shared?
● Is RTAC providing what the M/B/IOs are requesting?
SEQ 2b. Are efforts adequate to meet demand by USAID?
● Has USAID been satisfied with support fromRTAC?
● Given the fixed price modality, has the LOE allocated been
sufficient to meet USAID demand?
SEQ 2c. Which approaches were successful in meeting stakeholder
needs?
● What approaches are lesssuccessful?
● What other approaches should be considered?
SEQ 2d. To what extent are research and communicationsproducts
being utilized? 
● How and by whom for what purpose?
● Is there a difference in extent of use comparing buy-in 
projects with core funded projects?
Comms/R2A leads,
Buy-in PIs, NORC,
USAID staff
managing RTAC,
USAID technical
staff
NORC, USAID staff
managing RTAC,
USAID technical
and program staff
Comms/R2A leads,
Buy-in PIs, NORC,
USAID staff
managing RTAC,
USAID technical
and program staff,
partner country
end-users
Comms/R2A leads,
Buy-in PIs, NORC,
USAID staff
managing RTAC,
USAID technical
staff, partner
country
counterparts
Table 3. Mid-term Performance Evaluation Question #3
Program
Task
Core
Evaluation Question # 3 
Sub-evaluation Questions and Specification Target Population
Core and
Buy-in 
SEQ 3a. How are RTAC services and products promotedthroughout
USAID (Washington and its missions)?
● Are efforts adequate?
NORC, USAID staff
managing RTAC,
USAID technical
and program staff
EQ 2b. Which types of technical assistance approaches (training, convening, research
projects, etc.) have been most successful in meeting the needs of project stakeholders
(considering timelines, relevance, and utility of product and services)?
EQ 3. What were the barriers and enablers that made the difference between successful and
disappointing implementation and results of the RTAC assistance?
RTAC Mid-term Performance Evaluation
1
SEQ 4a. What are the key needs of USAID M/B/IOs and affiliated
Core and
Buy-in 
SEQ 3b. Are USAID staff pleased with RTAC products and services?
● Are USAID technical and program staff aware of RTAC’s RNand
services? 
● How does satisfaction compare between USAID staff engaged
through buy-ins vs. USAID staff engaged through core funded
projects?
NORC, USAID staff
managing RTAC,
USAID technical
and program staff
Buy-in 
Core and
Buy-in 
Core and
Buy-in 
SEQ 3d. What were the challenges, if any, that hindered the
achievement of desired results? 
NORC, Consortium
members, USAID
staff managing
RTAC, USAID
technical staff
Core and
Buy-in 
SEQ 3f. Are RTAC RN Members satisfied with the research
opportunities and other services being requested through the
network?
● How are the relationships between RN members engaged in
specific RTAC activities and USAID staff?
Individual network
participants, buy-in 
PIs, Comms and
R2A leads,
Institution partner
staff, NORC, USAID
staff managing
RTAC, USAID
technical staff
Table 4. Mid-term Performance Evaluation Question #4
Program
Task
Core
Buy-in 
SEQ 3e. What were the enablers or facilitators, if any,that accelerated NORC, Consortium
the achievement of desired results? members, USAID
staff managing
RTAC, USAID
technical staff
SEQ 3c. Are there other places or projects to obtain such services? Individual network
participants,
● In their view, does RTAC compare favorably to other Institution partner mechanisms they might use to obtain research services? Why staff, NORC, USAID
or why not? staff managing
RTAC, RTAC
stakeholders
RTAC Mid-term Performance Evaluation
1
Evaluation
Question # 4 
Sub￾evaluation
Questions
and
Specification
Target
Population
c
o
u
n
t
e
r
p
a
r
t 
i
m
p
l
e
m
e
n
t
i
n
g 
EQ 4. To what extent has RTAC helped USAID M/B/IOs and affiliated counterpart
implementing partners and host country governments make evidence-based decisions?
RTAC Mid-term Performance Evaluation
1
partners
and host￾country
governm
ents
when it
comes to
evidence￾based
decision￾making?
USAID staff managing RTAC,
RTAC Mid-term Performance Evaluation
1
Buy-in SEQ 4b. Is evidence provided by RTAC being used by USAID staff in 
policy, strategy development, or project design decisions?
● How? Why or why not?
NORC, USAID staff
managing RTAC,
USAID technical
and program staff
Buy-in 
EVALUATION DESIGN MATRIX
To answer the evaluation questions listed above the following evaluation design matrix has been developed to
organize the method in which data is collected, and analyzed. RTAC MPE methodology will use a mixed-method
approach. The following matrix provides an overview to the evaluation design methodology developed for the
study.
Table 5. Evaluation Design Matrix
Evaluation questions and sub-questions Data Source Data Collection
Methods
Data Analysis
Method
1. To what extent has the goal of building an
institutionally and individually diverse partner
network been met? What have been the
barriers and enablers to building a diverse
partner network?
a. What results was USAID seeking in creating
the network? Are they being achieved? What
benefits or incentives does the RN offer to its
members?
b. What are the characteristics of individuals in
the network? What does their
“engagement” look like? What motivated
researchers to join the network? What do
they hope to gain? What would encourage
more researchers to join the RN?
Project documents
NORC staff 
USAID staff 
RTAC consortium
members
RTAC Research
Network members
Project document
review
Project data review
KIIs 
FGDs
Network member
survey
Content analysis of 
qualitative data
Quantitative 
analysis of central
tendencies and
variance within
responses,
subgroup analysis
(sex, age group,
disability status,
length of time in
network, academic
discipline)
c. Does the structure or management of the
network help or hinder its effectiveness?
● To what extent are RTAC products and servicesmeeting these USAID technical
needs? and program staff
SEQ 4c. Is evidence provided by RTAC being used by local and USAID technical
national decision-makers and advocacy organizations in policy debate staff, partner
and development, strategy development, program design, or service country end-users
delivery? 
● How? Why or why not?
RTAC Mid-term Performance Evaluation
1
d. Was the RTAC theory of change (TOC) design
appropriate? Were key assumptions correct?
Should the TOC be revised?
2. To what extent does the training and technical
assistance provided by the RTAC meet the
needs and priorities of project stakeholders
(such as M/B/IOs)? Which types of training and
technical assistance have been most successful
in meeting the needs of project stakeholders
(considering timelines, relevance, and utility of
product and services)?
Project documents
NORC staff 
USAID staff 
RTAC consortium
members
RTAC PIs
Project document
review
KIIs 
FGDs
Site visits
Content analysis of 
qualitative data
a. How is development evidence (i.e.,
research results) being shared? Is RTAC 
providing what the M/B/IOs are
requesting?
b. Are efforts adequate to meet demand by
USAID? Has USAID been satisfied with
support from RTAC? Given the fixed price
modality, has the LOE allocated been
sufficient to meet USAID demand?
RTAC Research
Network members
Partner Country
Counterparts and
Beneficiaries
c. Which approaches were successful in
meeting stakeholder needs? Which
approaches were less successful? What
other approaches should be considered?
d. To what extent are research and
communications products being utilized?
How and by whom for what purpose? Is
there a difference in extent of use
comparing buy-in projects with core
funded projects?
3. What were the barriers and enablers that made Project documents
NORC staff 
USAID staff 
RTAC consortium
members
RTAC PIs
RTAC Research
Network members
Partner Country
Counterparts and
Beneficiaries
Project document Content analysis of
the difference between successful and and data review qualitative data
disappointing implementation and results of
the RTAC assistance? Project data review
a. How are RTAC services and products
promoted throughout USAID (Washington 
and its missions)? Are efforts adequate? 
b. Are USAID staff pleased with RTAC products
and services? Are USAID technical and
KIIs 
FGDs
Network member
survey
Quantitative
analysis of central
tendencies and
variance within
responses,
subgroup analysis
program staff aware of RTAC’s RN and (sex, age group,
services? How does satisfaction compare disability status,
between USAID staff engaged through buy- length of time in
ins vs. USAID staff engaged through core network, academic
funded projects? discipline)
c. Are there other places or projects to obtain
such services? In their view, does RTAC
RTAC Mid-term Performance Evaluation
1
compare favorably to other mechanisms
they might use to obtain research services?
d. What were the challenges, if any, that
hindered the achievement of desired results?
e. What were the enablers or facilitators, if any,
that accelerated the achievement of desired
results?
f. Are RTAC RN Members satisfied with the
research opportunities and other services
being requested through the network? How
are the relationships between RN members
engaged in specific RTAC activities and USAID
staff? 
4. To what extent has the RTAC helped USAID
M/B/IOs and affiliated counterpart NGOs and
host country governments make evidence￾based decisions?
a. What are the key needs of USAID M/B/IOs
and affiliated counterpart NGOs and host￾country governments when it comes to
evidence-based decision-making? To what
extent are RTAC products and services
meeting these needs?
b. Is evidence provided by RTAC being used by
USAID staff in policy, strategy development,
or project design decisions? How? Why or
why not? 
c. Is evidence provided by RTAC being used by
local and national decision-makers and
advocacy organizations in policy debate and
development, strategy development,
program design, or service delivery? How?
Why or why not? 
USAID staff 
RTAC PIs
Partner Country
Counterparts and
Beneficiaries
KIIs 
FGDs
Site visits
Content analysis of 
qualitative data
EVALUATION METHODOLOGY
The RTAC MPE team will employ an exploratory, sequential, mixed-methods design, which aligns with the formative
purpose of the evaluation and the CLA approach selected to ensure that findings can be used for adaptive
management purposes. The choice of combining the use of qualitative and quantitative research enables more
flexibility in the investigation, because it allows the team to capture both breadth and depth of information to
address research questions that are complex in nature (Creswell, 2006), and provides the team with different types
of data through which results can be corroborated.
The MPE team will use seven data collection methods to answer the evaluation questions and sub-questions listed
above: (1) key informant interviews (KIIs) with program managers (USAID and NORC personnel); (2) KIIs with USAID
RTAC Mid-term Performance Evaluation
1
technical and program staff; (3) KIIs with RTAC consortium members; (4) KIIs with the principal investigators (PIs) of
each RTAC research buy-in; (5) focus group discussions (FGDs) with RTAC Communications and Research to Action
(R2A) PIs and a sample of Research Network members; (6) FGDs with USAID technical and program staff; (7) FGDs
with partner country counterparts and beneficiaries; and (8) a survey of RTAC Research Network members. Table 6
below shows the distribution of data collection methods for each stakeholder group. Each of these modes of data
collection and stakeholder groups from whom data will be collected is discussed below.
Table 6. Data Collection Method by Stakeholder Group
Stakeholder group KIIs
(expected)
Re-interviews
(expected)
FGDs
(expected)
Survey
NORC program managers 5 5 0 0
USAID staff (DC and field) 8 3 5 0
RTAC PIs 9 0 2 0
RTAC Consortium Members 12 0 0 0
RTAC Research Network
Members 
0 0 3 1
Partner country counterparts
and beneficiaries
6 0 2 0
TOTALS 40 8 12 1
Key Informant Interviews
Key Informant Interviews (KIIs) are in-depth, semi-structured and facilitated discussions conducted with individuals
or small groups of stakeholders (typically no more than 2 or 3) who have specialized and in-depth knowledge of the
program or a related issue. KIIs are semi-structured in that they utilize a discussion guide listing the questions to be
covered during the interview, while also following an open-ended format that: (1) gives interviewees the space to
respond at length in their own words; and (2) allows the interviewer to probe for information that will uncover new
insights into key aspects of stakeholder perceptions and to pursue other lines of inquiry as they emerge during the
interview. KIIs help answer the evaluation questions by providing information collected from a wide range of
people who have first-hand knowledge about the relevant topics. These key informants, with their specialized
knowledge and understanding, provide in-depth, first-hand insights on issues not easy to identify through other
data collection methods.
Draft KII protocols will be reviewed by USAID and NORC and adjusted as needed by the evaluation team after use
with 1-2 respondents prior to being finalized (after the first site visit). The MPE team will note any confusion the
respondents have with regard to wording and sequencing of questions, as well as any gaps the respondent
identifies, and make edits accordingly. Questions that are not easily understood by respondents or do not yield the
RTAC Mid-term Performance Evaluation
1
type of data sought will be removed from the final version of the protocol, and new questions may be added to
better respond to all evaluation questions. The re-interviews with NORC and USAID also will provide an opportunity
to probe for further information if the initial KII results in any information gaps.
All KIIs will be analyzed according to the themes of the evaluation, following a coding scheme in the data analysis
plan. If new themes emerge over the course of the coding process, they will be incorporated into the coding
scheme and used in the final analysis.
The team will utilize KII protocols with five audiences, as described below. 
NORC Program Managers
As the prime implementer of the RTAC activity, NORC’s personnel have extensive and specialized knowledge of its
operations. They are therefore critical to understanding the operations of the network, its successes and its
challenges. The team intends to interview up to five NORC personnel who will be purposively selected for their roles 
within the project, with up to three (3) of them likely to be re-interviewed toward the end of the study to capture
adaptations in the project over the life of the evaluation. The initial interviews will last approximately 60 minutes,
with the re-interviews lasting 30-45 minutes, and will be scheduled at the convenience of each respondent. The 
RTAC MPE team will contact these individuals directly with USAID’s approval once the data collection period begins.
The data collection instrument is provided in Annex 1: RTAC MIDTERM EVALUATION – NORC PROGRAM MANAGERS
KII PROTOCOL.
Prospective NORC Interviewees 
Sutherland Miller
Varuni Dayaratna 
Gabriela Alcaraz 
Alain Otarola 
Michelle Davis
USAID Program and Activity Managers and Leadership
The USAID personnel who have provided oversight and guidance to the RTAC activity and buy-ins have an important
perspective on how USAID uses (as well as could and should use) the RTAC mechanism, communications with
M/B/IOs on use of the mechanism, and the impacts to date of the work completed. The team intends to interview
up to eight USAID personnel who will be purposively selected for their roles overseeing and guiding the projects,
with up to three of them likely to be re-interviewed toward the end of the study to capture adaptations in the
project over the life of the evaluation. The initial interviews will last approximately 60 minutes, with the re￾interviews lasting 30-45 minutes. The RTAC MPE team will ask USAID to send an introductory note, and the team
will follow up with these individuals directly once the data collection period begins (or, if deemed more appropriate,
the activity manager for the evaluation may contact these individuals to schedule KIIs and FGDs). All KIIs will be
scheduled at the convenience of the respondent. The data collection instruments are provided in Annex 1:RTAC
RTAC Mid-term Performance Evaluation
1
MIDTERM EVALUATION – RTAC PROGRAM MANAGERS (USAID COORDINATORS, MANAGERS, and
ADMINISTRATORS) KII PROTOCOL and RTAC MIDTERM EVALUATION – RTAC Technical Clients - USAID KII PROTOCOL.
Prospective USAID Interviewees - Contract Administrators and Activity Managers
Tara Hill (RTAC COR)
Ashley Heiber (RTAC Lead Activity Manager)
Megan Miller (Activity Manager for Research to Action products, communication plans, Zimbabwe FFP
year 1, Digital Convening)
Prospective USAID Interviewees - Technical Clients 
LAB/CDD - Paul Nelson
LAB/CTP - Liz Chacko or Meron Paulos
GH/OHA - Hannah Marqusee
Kenya/East Africa - Nancy Peek
DCHA/FFP - Charles Davis or Daniel Houston 
RTAC Principal Investigators 
Nine research buy-ins have been contracted to date under the RTAC mechanism, and the principal investigators of
each of these buy-ins will have a unique perspective on the RTAC network and the use of research in achieving
development goals. The team intends to interview each of the buy-in PIs over the phone at a time convenient to
them. Each interview will last approximately 60 minutes. The MPE team will prepare initial communications for
each of the prospective respondents, and NORC staff will initiate contact with the buy-in PIs. The MPE team will
then follow up with the request and scheduling. The data collection instrument is provided in Annex 1: RTAC
MIDTERM EVALUATION – BUY-IN PRINCIPAL INVESTIGATORS KII PROTOCOL.
Prospective RTAC Research PI Interviewees 
Leon Perlman, Rebecca Rouse (Honduras)
Diana Ntamu (Uganda/Nigeria)
John Duffy (Fellowships)
RTAC Mid-term Performance Evaluation
1
Edmund Malesky, Erika Weinthal (research assistants Songkhun Nillasithanukroh, Ekta Patel)(Mekong
Political Economy Analysis)
Rahul Oka (Kenya)
Dominica Chingarande (Zimbabwe)
Moctar Tounkara (Mali)
Catherine Andrzejewski (SFI)
Gabriela Alcaraz (Increasing Research Impact) 
RTAC Consortium Members and Research Network Members
In addition to collecting data on the research-related activities of specific buy-ins, it will be useful to gather
perspectives from NORC’s consortium members and one highly active network member. The MPE team will work
with NORC program managers to purposively identify 11 consortium members personnel (tentatively identified
below) and one highly active network member (identified by NORC as having bid on almost all opportunities to
date), whose perspectives will help provide a range of opinions and experiences with the network. The interviews
will last approximately 45 minutes. The MPE team will prepare communication materials introducing the evaluation
and share that with NORC staff. The initial contact with the RTAC researchers will be made by NORC staff and the
MPE team will follow up with the request and scheduling. The data collection instrument is provided in Annex 1:
RTAC MIDTERM EVALUATION – RTAC CONSORTIUM MEMBER KII PROTOCOL.
RTAC Consortium Members Prospective Interviewees 
Arizona State University Ryan Johnson 
Institute of International Education David Simpson
Davis Management Group Sylvin Bynoe
Population Reference Bureau Carolyn Rodehauhau (Former employee; led
production of all communications products)
Tunisia R2A - Meredith Pierce (former
employee)
Indonesia R2A - Robin Johnson (former
employee/ independent consultant)
DRC R2A - Carolyn Rodehau and Marissa Yeakey
(former employee)
Philippines R2A - Reshma Naik
RTAC Mid-term Performance Evaluation
1
Forum One Mike Shoag
Duke University Erik Wibbels
Notre Dame Pulte Institute for Global Development Kevin Fink
Resilient Africa Network (RAN) at Makerere University Roy Mayega
USAID Partner Country Counterparts and Beneficiaries
Finally, the team will interview a limited number of counterparts and beneficiaries during each site visit to capture
the effectiveness and impact of applied research on progress towards development goals. The relevant principal
investigator and USAID Mission will be asked to recommend individuals to be interviewed, and they will also be
asked to share an introduction provided by the MPE team so the MPE team can follow up with the request and
scheduling. These interviews will last 30-45 minutes. The data collection instrument is provided in Annex 1: RTAC
MIDTERM EVALUATION – RTAC PARTNER COUNTRY COUNTERPARTS KII PROTOCOL.
Partner Country Counterparts and
Beneficiaries 
Site Visit #1 (Honduras)
Prospective Interviewees 
Site Visit #2 (Kenya)
Prospective Interviewees 
USAID Mission Staff 1-2 1-2 
Host country government partner(s) 0 2
(Interior Ministry, refugee
service providing agencies)
Primary local partner/end-user
institution staff*
3-5 
(financial institution
personnel)
1-2 
(UNHCR, WFP, etc.)
* Note: Final sampling for site visits will be decided in coordination with PIs for each of the activities.
Focus Group Discussions 
The FGDs are semi-structured and moderated small group discussions, typically with groups of 5 to 12 individuals
who have similar shared experiences and are not being asked to participate based on their technical knowledge or
expertise on a topic. FGDs are semi-structured in that they use a discussion guide but also involve in-depth probing
by the FGD facilitator, while also allowing the facilitator the flexibility to pursue other avenues of inquiry as they
emerge. FGDs help answer the evaluation questions by drawing on respondents’ attitudes, feelings, beliefs,
experiences and reactions, and in a manner that is harder to obtain through other data collection methods. The FGD
format also allows the evaluators to explore contextual factors that help explain data collected through other
approaches. In several cases, the potential respondents are scattered across long distances, and in those cases, the
team will use virtual focus group discussions (by phone, Skype, WhatsApp or another platform that is convenient for
all the participants). Although the interaction is somewhat more limited in a virtual format, the MPE team leader is
highly experienced at moderating virtual groups and fostering highly informative discussions using this modality. Key 
RTAC Mid-term Performance Evaluation
1
factors in successful virtual FGD facilitation include smaller numbers of participants than invited to in-person groups,
relative group homogeneity (for example, only those engaged in a specific activity, not any participant of any
activities), and effective tracking of each participant’s responses and the ability to link or relate experiences for
participants.
The FGD questions for USAID staff, partner country counterparts, and RN members will be tested during the pre￾sample trial period that will take place in Washington, DC and the Honduras site visit. After each FGD, the MPE team
will identify any poorly worded or vague questions and make edits accordingly. Questions that are not easily
understood by respondents or do not yield that type of data sought will be omitted from the final version of the
protocols, while new questions may be added to better cover all research inquiries.
All FGDs will be analyzed according to the themes of the evaluation, following a coding scheme in the data analysis
plan. If new themes emerge over the course of the coding process, they will be incorporated into the coding
scheme and used in the final analysis.
USAID Personnel
As USAID staff are a key audience for RTAC services, the team will conduct focus groups with a convenient sample of
relevant and available USAID personnel who have experience using the RTAC mechanism or who have considered
using the RTAC mechanism (but may have not yet decided or decided on another mechanism). The team will
conduct up to five (5) FGDs with USAID personnel. The initial pre-sample FGD will be draw from LAB/CDR personnel
(see table below). The other FGDs will be organized by sector/discipline: global health, agriculture and food security,
environment and global climate change, and democracy, human rights and governance. The team will invite staff to
participate who have significant responsibility for supporting program development (for both Washington-based
operating units and Missions). To ensure maximum participation, these groups will be limited to no more than six 
(6) people and one hour, given the time constraints most USAID personnel face. In the event that a number of
potential respondents work remotely, the team may offer a virtual FGD option to enable those individuals to
participate as well. The RTAC MPE team will contact these individuals directly with USAID’s approval once the data
collection period begins (or, if deemed more appropriate, the activity manager for the evaluation may contact these
individuals). The data collection instrument is provided in Annex 1: RTAC MIDTERM EVALUATION – RTAC USERS and
POTENTIAL USERS - USAID FGDPROTOCOL.
Prospective LAB/CDR FGD Participants
Michelle Jones, LAB/CDR/Fellowships 
Amanda Nataro, LAB/CDR/Fellowships 
Karen Fowle, Lab/CDR (Activity Manager Research Translation convening with UKRI, Activity
Manager Mekong PEA)
Lisa Patel, Lab/CDR (RTAC Task 6)
Holly Ferrette, Lab/CDR (RTAC Task 6)
Lily Sweikert, Lab/CDR (RTAC Task 6)
RTAC Mid-term Performance Evaluation
1
Cameron Bess (USAID/CDR/PEER)
Brent Wells (USAID/CDR/PEER)
RTAC PIs
To gain a better understanding of the efficacy of the communications and R2A approaches used under RTAC, the
team will conduct focus groups with the PIs of those activities (one for communications and one for R2A). Each
focus group will last approximately 60 minutes. The MPE team will prepare initial communications for each of the
prospective respondents, and NORC staff will initiate contact with the buy-in PIs. The MPE team will then follow up
with the request and scheduling. The data collection instrument is provided in Annex 1: RTAC MIDTERM
EVALUATION – RTAC PI FGD PROTOCOL.
Prospective RTAC Communications FGD Participants
Ghita Mezzour (Morocco)
Njukeng Nkengafac (Cameroon) 
Ikram Guizani (Tunisia)
Prospective RTAC R2A FGD Participants
Serges K. Vutegha (primary) and Edwin Price (DRC-HESN ConDev)
Flordeliza Bassiag (Philippines)
Hadi Pratomo and Asri Adisasmita (Indonesia) 
RTAC Research Network Members
The MPE team will invite a sample of RTAC Research Network members to participate in FGDs to capture a wider
range of experiences and further explore any issues that arise in the survey. The MPE team will work with NORC
program managers to invite network members to participate in three focus groups. One will include LMIC partner
institutions/members, one will comprise US-based minority serving institutions (MSI)/members based at MSIs, and
one will include representatives of the US-based non-MSI partner institutions/members based at non-MSI
institutions via virtual online meeting platforms, such as phone, Skype, go-to meeting, etc.). The FGDs will last
approximately 60 minutes. The evaluation team will assure that the discussions are dynamics through: (1) assuring
that everyone’s technical equipment is working (done through self-introductions); and (2) monitoring the online
chatroom/group roster list, recording the frequencies in which participants respond and encouraging them to
provide feedback through repeating probing questions. Initial contact with the RTAC researchers will be made by
NORC staff to facilitate an introduction, and the MPE team will follow up with the request and scheduling. The data
RTAC Mid-term Performance Evaluation
1
collection instrument is provided in Annex 1: RTAC MIDTERM EVALUATION – RTAC RESEARCH NETWORK MEMBERS
FGD PROTOCOL.
USAID Partner Country Counterparts and Beneficiaries
Finally, the team will conduct one FGD with partners and beneficiaries during each site visit to capture the impact of
applied research on progress towards development goals. The relevant principal investigator and USAID Mission
will be asked to identify possible participants, and they will also be asked to provide an introduction so the MPE
team can follow up with the request and scheduling. These FGDs will last approximately 60 minutes. The data
collection instrument is provided in Annex 1: RTAC MIDTERM EVALUATION – RTAC PARTNER COUNTRY
COUNTERPARTS FGD PROTOCOL.
Web-based Survey
The MPE team will design and administer a web-based survey to RTAC network members in two rounds using
Survey Monkey. As it is useful to gauge satisfaction with the network by a variety of subgroups (i.e., by sex, by
location, by area of specialization, etc.), a census of all network members is appropriate and will allow us to provide
granular information by the range of sub-groups included in the network. To increase response rate, the Evaluation
Team will assure:
1. The NORC Program Managers send an introductory letter to the network members (based on a sample
letter provide by the Evaluation Team);
2. The window for the web-based survey will be open up to 30 days to maximize participant response,and
that;
3. Bi-weekly reminders are sent to the network members to increase responserate.
The survey will require approximately 15-20 minutes to complete and will contain approximately 32 items (mostly
closed-ended, with a few open-ended to allow for inclusion of comments and recommendations). The first round
will be sent to the 254 original Research Network members who have been engaged with RTAC since its inception.
A second round will be sent to all newer network members some months later to allow them more time to become
engaged and share informed opinions about the network’s operations and effectiveness. 
The survey will be validated and tested for reliability using data collected from a sample of participants during pre￾testing. Nine (9) Research Network members will be randomly selected to test the survey. In the introductory letter
we will specify the first round of surveys will be used to test the survey. The participants will be provided a link,
and/or hard copy of the survey from which the Evaluation Team will follow up with participants for comments. The
validation process will be comprised of (whenever necessary) revising the sentence structure of the questionnaire to
assure that the participants understand the questions. Any misunderstood or confusing questions will be reworded.
The questionnaire will also be tested for reliability using Cronbach’s alpha to determine internal consistency, i.e.
whether or not the indicators reliably measure the expected outcomes. The Cronbach alpha coefficient will provide
evidence that the items on the instrument measure what they purport to measure. A Cronbach reliability coefficient
of .70 or higher is considered acceptable by researchers for most social science studies (Creswell, 2008). The data
collection instrument is provided in Annex 1: RTAC MIDTERM EVALUATION – RESEARCH NETWORK MEMBER
SURVEY.
EVALUATION PROCESS
RTAC Mid-term Performance Evaluation
1
In the initial, pre-testing phase of the evaluation, the team will use an exploratory sequential approach, which is a
process through which the team collects data while refining instruments through reflection with respondents and
use of other cognitive interviewing techniques. (See Diagram 1, pre-testing section.) The Evaluation Team will then
use the Concurrent Triangulation Strategy (CTS) to collect data during the formal data collecting phase of the
evaluation. In a CTS approach, both quantitative data and qualitative data are collected concurrently and then
compared to determine whether there is convergence, differences or some combination. This model is generally
used to offset the weakness inherent within one method with the strengths of the other (or conversely, the strength
of one adds to the strength of the other) (Creswell, 2009). Diagram 1 (below) provides a summary of the data
collection using the exploratory sequential and the CTS mixed method designs.
Diagram 1. Data Collection Sequence
SAMPLING PLAN
As noted above, purposive sampling will be used to conduct KIIs with up to 40 RTAC project stakeholders, as well as
purposive or convenience samples of FGD participants (approximately 72). Currently, NORC classified 12% of the
RTAC partner pool as being highly engaged. The Evaluation Team will assure that the highly engaged individuals and
institutions are represented when conducting KIIs, as they are very few in numbers (12%), and may be able to
provide more in-depth and informed experience about the RTAC project. The FGD sampling will include a broader
range that includes both less active and very active network members. As USAID has set quotas for women,
minority serving institutions, and foreign institutions, the MPE team will oversample these groups to ensure they
are adequately represented in the qualitative data collection.
RTAC Mid-term Performance Evaluation
1
A census of network members will be conducted for the survey. It is estimated that response rate will be between
50% and 60% based on the affiliation RTAC has established with its partners and stakeholders, and based on
previous studies that reported response rates in that range for online surveys administered when there is
organizational buy-in (Dillman et al., 2009). The team will conduct ongoing outreach to ensure a maximum
response rate, and if NORC has and can share basic demographics and other key variables of interest to the team,
the team may use an appropriate weighting or matching and bootstrapping approach to adjust the survey results to
ensure that they are representative. 
DATA ANALYSIS PLAN
The MPE team will develop a data analysis plan that will map each item on each data collection instrument to the
evaluation questions being answered. The plan will also include the coding scheme for the qualitative data and the
syntax to be used for the analysis of the survey data.
In general terms, the team will use the coding scheme to code each set of KII and FGD notes. The team will use
Dedoose or a similar coding software to code qualitative data. The team will then thematically analyze the coded
segments to develop findings. Concurrently, the survey data will be analyzed in STATA for central tendencies,
correlations, and sub-group differences. Key factors for sub-group analysis will include sex, location of respondent
(US-based or USAID partner country-based), and type of institution (MSI versus non-MSI).
The triangulation of the data will be used to find inconsistencies and convergences (Johnson and Christensen, 2004).
The pre-sample tool validation phase will be an initiation primarily seeking inconsistencies and some convergences.
Once the tools have been validated, the team will develop both quantitative and qualitative data sets collected
during the data collection phase to corroborate the results of the data in order to provide in-depth understanding of
the RTAC Project performance to date. Hence, the Evaluation Team will use particular data sets in order to answer
the four major research questions, and to document successes, challenges, design, and implementation.
ETHICS AND CONSENT
All of the potential respondents are highly educated adults affiliated with universities and research institutions. They
are fully capable of understanding the purpose of the research, their role in it, the parameters of data ownership
and use, and their right to refuse to participate. The team will use clearly drafted, broad consent forms for each of
the data collection instruments, and will allow any potential respondent who is not comfortable with the plans for
the study to opt out (before, during, or after data has beencollected). 
LIMITATIONS
The primary limitation of the study is based in the design of the RTAC program model. Since the program
participants are recruited on a rolling basis, the experiences of each participant will differ based on time involved in
the program, the services they were requested to provide and their experience with an individual USAID Mission,
Bureau, or Independent Office. Therefore, the experiences of any sample of respondents may not be representative
of the whole. Additionally, as much of the qualitative data collection (KIIs, FGDs) for this evaluation relies on
purposive sampling, it will not produce statistically representative samples. As a result, although these respondents
typically represent a variety of views on a topic, they may not represent all relevant views, and selection bias can
emerge. Timing of data collection may also influence whether or not network members and other potential
respondents participate. While the team will try to accommodate scheduling needs, focus groups and site visits may
not be able to be optimally scheduled for all respondents.
RTAC Mid-term Performance Evaluation
1
Another important limitation is the potential for non-response among less engaged members of the network. As
they may not be highly invested, it may be difficult to obtain their participation without incentives. As it is not
currently clear how more engaged and less engaged network members differ, if at all, it may not be possible to
eliminate non-response bias in the design.
The last foreseen limitation is the inclusion of only two site visits to cooperating countries. Currently, 80 (54%) of the
initial 146 participating institutions are in developing countries and are classified as Low to Medium Income
Countries (LMICs). Incorporation of another major region would have given the evaluation more depth and rigor,
and would have permitted the evaluation team to survey a larger pool of participants across regions and various
sites. Reaching out to more than two sites and for a longer duration than one week per country would have given a
wider sample from which to draw conclusions, and possibly increased participant buy-in to complete the online
survey.
DATAQUALITY ASSURANCE PLAN
The Evaluation Team will devise a check list for collecting data in the field to ensure that all questions are addressed
and that protocol forms are filled out properly and how to store data to guard confidentiality. While in the field, the
team will secure all notes and any recordings on password-protected devices. All KII and FGD notes and recordings
(where the respondent(s) consent) will be uploaded to and stored on IBTCI’s secure server. A plan will also be
devised on how to go back and obtain missing data from the KIIs and Focus Group Discussions, including follow up
phone calls and re-interviews for key respondents. For the online survey, auto-validation checks will be used and
program commands will be set in place in order to assure that participants answer all mandatory questions prior to
submitting the completed survey online.
DISSEMINATION PLAN
The MPE team will disseminate findings at key points during the data collection process and after analysis is
complete. In March 2020, the team will deliver the inception presentation, and LAB/CDR will determine the extent
of the audience for that presentation. In May 2020, the team is planning a CLA reflection meeting with CDR and
RTAC, along with its partners, which will include preliminary findings to that point for use in adaptive management
approaches. In addition, the team will conduct a half-day workshop in August 2020 to review the final results of the
data analysis and co-create recommendations to strengthen the RTAC program. Finally, the final report will be
submitted to the DEC and disseminated to evaluation respondents.
RTAC Mid-term Performance Evaluation
1
REFERENCES
Baruch, Y., & Holtom, B. C. (2008). Survey response rate levels and trends in organizational research.
SAGE Journals, 61(8), 1139-1160.
Creswell, J. (2009). Research Design Quantitative, Qualitative, and Mixed Methods Approaches 
(Ed. 3rd Edition). (V. Knight, S. Connelly, S. K. Quesenberry, & M. P. Scott, Eds.) Thousand Oaks, California,
United States: SAGE Publications, Inc.
Creswell, J.W. (2008). Educational research planning, conducting, and evaluating quantitative research
(3rd Ed). Upper Saddle River, NJ: Pearson Education, Inc.
Creswell, J. (2006). Mixed methods. Retrieved December 24, 2015, at SAGE:
http://www.sagepub.com/sites/default/files/upm-binaries/10981_Chapter_1.pdf
Dillman et al. (2009). Mail and Internet Surveys: The Tailored Design Method, Third edition. New York:
John Wiley and Sons. Retrieved from: http://joe.org/joe/2012december/tt7.php
Johnson, B. and Christensen, L. (2004). Educational research, quantitative, qualitative, and mixed
methods approaches (2nd Ed). Pearson Education Inc.: Boston, MA
Saldana, J. (2012). The coding manual for qualitative researchers. Thousand Oaks Publication: Sage
Publications.
RTAC. (2019). Research Technical Assistance Center. Retrieved from http://www.rtachesn.org
USAID.GOV. (2019, September 20). Research Technical Assistance Center (RTAC). Washington, D.C., United
States of America.
RTAC Mid-term Performance Evaluation
1
Data Collection Instruments 
Survey
RTAC MIDTERM EVALUATION – RESEARCH NETWORK MEMBERS SURVEY 
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Development Lab to
conduct a midterm evaluation of the Research Technical Assistance Center (RTAC) to assess the effectiveness of
program interventions to date and to strengthen future program interventions. As a partner and/or network
member of the project, your unique perspectives will assist us in ensuring RTAC effectively achieves its goals. The
evaluation will be used to make informed decisions about mid-course corrections and future programming efforts.
The survey contains 32 questions and should take no more than 10 to 15 minutes to complete.
Please note that your participation in this survey is voluntary, and you are free to end the survey at any time. By
clicking the “Consent and enter survey” button below, you are consenting to the following terms:
• Any response you provide may be reported in the final report as part of the aggregated quantitative
analysis or the de-identified qualitative analysis from open-ended responses.
• Responses may be reported by demographic category, discipline, orinstitutional characteristics. The
only identifying information used will be the demographic information provided in the final section
of the survey.
• De-identified data files will be submitted to USAID at the completion of the evaluation(without
names or any contact information).
• The data you provide may be reanalyzed at a later date for a follow-up study or other purpose as
approved by USAID.
If you have any questions about this survey or the RTAC evaluation more broadly, you can contact the IBTCI
evaluation team at [insert email address].
Please answer the questions to the best of your ability, and use the comment boxes to provide fuller answers and
more insight on your experiences with RTAC. Thank you in advance for your time and input!
CONSENT TO PARTICIPATE
By clicking the button to enter the survey below, you are giving your consent to participate in this evaluation. If you
do not wish to participate, please click the exit survey link below. 
Consent and Enter Survey ⃝ Refuse and Exit Survey ⃝
Thank you for agreeing to complete the survey! The estimated time to complete the survey is 10 to 15 minutes.
Please answer each question to the best of your ability. 
RTAC Mid-term Performance Evaluation
1
UNLESS OTHERWISE SPECIFIED, RESPONSE CHOICES: 1. DISAGREE 2. SOMEWHAT DISAGREE 3. SOMEWHAT AGREE 4.
AGREE 5. I DON’T KNOW 9. NOT APPLICABLE
QUESTIONS RESPONSES
Section 1. NETWORK BUILDING
1. How did you first become involved in the RTAC network? Please select
all that apply.
1. RECRUITED BY NORC 
2. RECRUITED BY ANOTHER RTAC PARTNER
(DUKE UNIVERSITY, MAKERERE UNIVERSITY,
NOTRE DAME PULTE INSTITUTE FOR GLOBAL
DEVELOPMENT, ETC.)
3. REFERRED BY A COLLEAGUE IN MY
INSTITUTION 
4. REFERRED BY A COLLEAGUE AT ANOTHER
INSTITUTION 
5. FOUND OPPORTUNITY THROUGH ONLINE
SEARCH 
6. OTHER: (WRITE IN)
2. Why did you join the RTAC network? What benefits, if any, did you
expect to receive? 
OPEN
3. Joining the RTAC network was a simple and timely process.
1. DISAGREE 2. SOMEWHAT DISAGREE 3.
SOMEWHAT AGREE 4. AGREE 5. I DON’T KNOW 9.
NOT APPLICABLE
4. What barriers or challenges do you see for individuals who want to join
the RTAC network?
OPEN
5. What changes, if any, would you propose to make recruitment more
effective and/or joining the network easier? 
OPEN
Section 2. LEVERAGING TECHNICAL EXPERTISE OF THE HIGHER
EDUCATION COMMUNITY
RTAC Mid-term Performance Evaluation
1
6. RTAC effectively communicates new research opportunities to its
members.
1. DISAGREE 2. SOMEWHAT DISAGREE 3.
SOMEWHAT AGREE 4. AGREE 5. I DON’T KNOW 9.
NOT APPLICABLE
7. The information and/or training needs are clearly identified by USAID in
the solicitations.
1. DISAGREE 2. SOMEWHAT DISAGREE 3.
SOMEWHAT AGREE 4. AGREE 5. I DON’T KNOW 9.
NOT APPLICABLE
8. Information about data rights is clearly delineated in the RTAC
solicitations. 
1. DISAGREE 2. SOMEWHAT DISAGREE 3.
SOMEWHAT AGREE 4. AGREE 5. I DON’T KNOW 9.
NOT APPLICABLE
9. Please rank the following factors in the order in which they would
discourage you from applying for a research project that is in your field for
which you have no conflict of interest:
TIGHT BUDGET
CONFLICTING COMMITMENTS 
TIGHT TIMELINES
POTENTIAL FOR UNEXPECTED DELAYS
POTENTIAL FOR LIMITATIONS ON DATA RIGHT
OTHER: WRITE IN (TO BE DEVELOPED FURTHER 
IN PILOT TESTING)
10. What change(s) would you make, if any, to improve the quality and
clarity of solicitations from RTAC?
OPEN
Section 3: EVALUATION AND RECOMMENDATIONS
11. How would you rate your overall satisfaction with the RTAC network on
a scale of 1 to 5, five being highly satisfied and 1 being highly dissatisfied?
1. Highly Dissatisfied 2. Dissatisfied
3. Neutral
4. Satisfied 5. Highly Satisfied
12. Have your expectations when joining the network been met?
1. NOT MET AT ALL
2. MET TO A SMALL EXTENT
3. MET TO A LARGEEXTENT
13. What recommendations would you make in addition to any changes
you have already described?
OPEN
14. Please add any additional comments regarding your experience with
RTAC.
OPEN
15. Are there other networks through which you have learned about
opportunities to conduct research for USAID?
YES/NO 
RTAC Mid-term Performance Evaluation
1
15a. If yes, please list the names of the network(s). OPEN
DEMOGRAPHICS
16. Academic department with which you are affiliated OPEN
17. Where is your (primary) institution located?
UNITED STATES
USAID-ASSISTED COUNTRY (SKIP NEXT)
OTHER COUNTRY (SKIP NEXT)
18. Is your institution designated a minority-serving institution, as defined
by the Higher Education Act? (a historically Black college or university, a
Hispanic-serving institution, a Tribal College or University, an Alaska Native￾serving institution or a Native Hawaiian-serving institution, a Predominantly
Black Institution, an Asian American and Native American Pacific Islander￾serving institution, or a Native American-serving nontribal institution)
YES/NO
19. Has your institution worked on a USAID award or sub-award (through
another partner) in the past?
YES/NO
20. Have you managed a USAID award or sub-award in the past? YES/NO
21. Sex FEMALE/MALE/OTHER
22. Age range
UNDER 25
25-29
30-34
35-39
40-44
45-49
50-54
55-59
60-64
OVER 65
23. Do you have difficulty with the following? (OPTIONAL) 1. No, no difficulty 2. Yes,
some difficulty 3. Yes, a lot of
RTAC Mid-term Performance Evaluation
1
o Seeing, even if wearing eyeglasses?
o Hearing, even if using a hearing aid?
o Walking or climbing steps?
o Remembering or concentrating
o Self-care (such as washing or dressing)?
o Communicating using your usual language (for example, 
understanding and being understood by others)?
difficulty 4. Cannot do it at all
5. I don’t know
RTAC Mid-term Performance Evaluation
1
Key Informant Interviews
RTAC MIDTERM EVALUATION – NORC PROGRAM MANAGERS KII PROTOCOL
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Development Lab to
conduct a midterm evaluation of the RTAC program to assess the effectiveness of program interventions to date and
to provide suggestions for strengthening future program interventions. As a core member of the program
management team, you have unique insights into network operations, management of activities, and progress
towards using research more effectively to achieve development goals. The evaluation will be used to make
informed decisions about mid-course corrections and future program efforts.
The information you share with us will be treated as confidential. The transcript and notes from this interview will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, RTAC program manager). You may choose not
to answer any question, and you may choose to end the interview at any time. Do you have any questions for me
about the evaluation, its purpose or use, or how we will handle the information you share with us today? [Answer
any questions that are asked.] Do you consent to participate in this evaluation? [If no, thank the respondent and
end the interview. If yes, continue with script.] 
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Please Complete Prior to Interview
First Name: Last Name:
Institution: Position:
Date of interview: Sex of Respondent (M/F/O):
Time Started: _ 
[Note to Interviewer: Some modules below may not be relevant to the interviewee’s role within the project. Prior
to the interview, or if not before then at the beginning of the interview, inquire about the role of the respondent
and determine which modules are most relevant to her/him.] 
Background questions
To start, I’d like to hear about your role with the Research Technical Assistance Center (RTAC) funded by USAID.
1. Please tell me about your role with RTAC.
2. Have you worked with USAID before? Please tell me a little about thoseexperiences.
RTAC Mid-term Performance Evaluation
1
I. RTAC RESEARCH NETWORK
For this section, we would like to focus on the RTAC research network membership.
1. What is the value proposition of the RTAC research network for USAID from yourperspective?
2. What have been the most successful approaches to expanding the network in your opinion? [Probe on
differences for individual and institutional members.] 
3. What have been the biggest hurdles in creating and expanding the network? [Probe on differences for
individual and institutional members.] 
4. How do members provide feedback on their experience with the network? What has been their most recent
feedback?
5. How would you describe the range of research opportunities promoted through the RTAC network? [Probe 
on whether or not the range meets expectations.] 
6. From your perspective, what are the benefits, if any, of belonging to the RTACnetwork?
7. From your perspective, what are the drawbacks, if any, of belonging to the RTACnetwork?
8. Are there any barriers to participation in the RTAC network that you could identify? 
9. Is the management of the RTAC network effective in your opinion? How could it be improved?
II. MEETING USAID NEEDS THROUGH RESEARCH AND TRAINING 
For the next section, we would like to focus on the extent to which RTAC has been to meet USAID’s identified needs,
starting with research. 
1. What are the USAID needs, in your opinion, has the RTACaddressed?
2. Has the stream of research requests been as expected? How has the stream of requests aligned with
planned levels of effort based on the information provided in the initial RFP?
3. To what extent have the requests for research you have received been well formulated and ready to
share with partners in the network? [Probe on the extent to which the research questions are solid and
well informed.] Has the dissemination of a template improved the quality of the research requests
received? 
4. To what extent have research projects been executed on their original schedules? If there have been
delays, at what stage or stages in the process have they occurred (i.e., scoping, contracting, design,
research approvals, data collection, analysis, or utilization)? If there were changes, were they initiated
by the researcher/institution or USAID?
5. What approaches does NORC take to address USAID's research and evidence needs?
6. Can you share any examples of how USAID (or other stakeholders) are using or plan to use RTAC
generated evidence? (Probe, have you been involved in research to action plans? To what extent has
that contributed to evidence use??? How are the R2A plans monitored and assessed, if atall?
7. Do you have any early sense of the ripple or spread effects of the research conducted – changes
resulting from its use? [Probe on concrete examples of use of evidence in decision-making and
programming.] 
8.
9. Would you please briefly tell us a little bit about the trainings that have been held for USAIDpersonnel?
10. In the evaluations you did of the training, how did participants rate the training? Has any follow up
been done to determine the extent to which participants have been able to apply it in their work? If so,
what were the results?
11. How are ongoing training needs identified, and what process is used to develop trainings or technical
assistance packages in response to those needs?
RTAC Mid-term Performance Evaluation
1
III. FINAL THOUGHTS
1. What have been the critical successes for RTAC thus far in youropinion?
2. What have been the critical challenges for RTAC thusfar?
3. Is there anything you would change in how RTAC supports USAID that you think would make RTAC more
effective in meeting those needs?
4. What have I not asked you about your RTAC experience that I should haveasked?
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Interviewer(s): 
Location:
Interviewer Comments:
RTAC Mid-term Performance Evaluation
1
RTAC MIDTERM EVALUATION – RTAC PROGRAM MANAGERS (USAID COORDINATORS, MANAGERS,
ADMINISTRATORS) KII PROTOCOL
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Development Lab to
conduct a midterm evaluation of the RTAC program to assess the effectiveness of program interventions to date and
strengthening future program interventions. As a core member of the program management team, you have
unique insights into RTAC’s operations, management, and results in support of using research more effectively to
achieve development goals. The evaluation will be used to make informed decisions about mid-course corrections
and future program efforts.
The information you share with us will be treated as confidential. The transcript and notes from this interview will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, RTAC program manager). You may choose not
to answer any question, and you may choose to end the interview at any time. Do you have any questions for me
about the evaluation, its purpose or use, or how we will handle the information you share with us today? [Answer
any questions that are asked.] Do you consent to participate in this evaluation? [If no, thank the respondent and
end the interview. If yes, continue with script.] 
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Please Complete Prior to Interview
First Name: Last Name:
Institution: Position:
Date of interview: Sex of Respondent (M/F/O):
Time Started: _ 
[Note to Interviewer: Some modules below may not be relevant to the interviewee’s role within the project. Prior
to the interview, or if not before then at the beginning of the interview, inquire about the role of the respondent
and determine which modules are most relevant to her/him.] 
Background questions
To start, I’d like to hear about your role with the Research Technical Assistance Center (RTAC) funded by USAID.
1. Please tell me about your role with RTAC.
2. What is your level of engagement within the RTACProgram?
RTAC Mid-term Performance Evaluation
1
I. RTAC RESEARCH NETWORK
For this first section, we would like to focus on the RTAC research network membership.
1. What is the value proposition of the RTAC network for USAID from your perspective? 
2. How would you describe the range of research opportunities promoted through the RTAC network? [Probe 
on whether or not the range meets expectations.] 
3.
4. From your perspective, what are the benefits, if any, of belonging to the RTACnetwork?
5. From your perspective, what are the drawbacks, if any, of belonging to the RTACnetwork?
6. Are there any barriers to participation in the RTAC network that you couldidentify?
7. Are there other mechanisms that you think provide the same or similar services? If so, what are they? If so,
do you know how they compare in terms of value for service received? 
8. Based on your experience thus far, is a mechanism like RTAC an effective way to meet information and/or
training needs (of your mission/OU/or office)? Are there other approaches you think would be equally or
more effective? How so?
9. Is the structure of the RTAC network effective in your opinion? How could it beimproved?
10. Is the management of the RTAC network effective in your opinion? How could it be improved?
11. How would you rate your overall satisfaction with the RTAC network on a scale of 1 to 5, five being highly
satisfied and 1 being highly dissatisfied? 
II. MEETING USAID NEEDS THROUGH RESEARCH AND TRAINING 
For the next section, we would like to focus on the extent to which RTAC has been to meet USAID’s identified needs,
starting with research. 
1. Has the stream of research requests been as expected? How has the stream of requests aligned with
planned levels of effort based on the information provided in the initial RFP?
2. How would you describe the support provided to clients, such as USAID buy-in entities, in the formulation of
scopes of work? [Probe on whether or not the template was used and/or co-creation of thescope.] 
3. To what extent have project activities been executed on their original schedules? If there have been delays,
at what stage or stages in the process have they occurred (i.e., scoping, contracting, design, research
approvals, data collection, analysis, or utilization)? If there were changes, were they initiated by USAID or
the researcher/institution? 
4. To what extent have the research results generated by RTAC addressed the needs USAID identified? (probe
other types of products such as trainings and convenings)
5. How have research findings and recommendations been utilized? Have the research to action plans been
effective? Have the R2A plans been co-created effectively? How so? How are the R2A plans being
monitored and assessed, if at all? 
6. Do you have any early sense of the ripple or spread effects of the research conducted – changes resulting
from its use? [Probe on concrete examples of use of evidence in decision-making and programming.] 
III. FINAL THOUGHTS
1. What have been the critical successes for RTAC thus far in youropinion?
2. What have been the critical challenges for RTAC thusfar?
3. Is there anything you would change in how RTAC supports USAID that you think would make RTAC more
effective in meeting those needs?
RTAC Mid-term Performance Evaluation
1
4. What have I not asked you about your RTAC experience that I should have asked? Whatelse should I know
about your experience with the program? How so?
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Interviewer(s): 
Location:
Interviewer Comments:
RTAC MIDTERM EVALUATION – RTAC Technical Clients - USAID KII PROTOCOL
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Development Lab to
conduct a midterm evaluation of the RTAC program to assess the effectiveness of program interventions to date and
strengthening future program interventions. As a core member of the program management team, you have
unique insights into RTAC’s operations, management, and results in support of using research more effectively to
achieve development goals. The evaluation will be used to make informed decisions about mid-course corrections
and future program efforts.
The information you share with us will be treated as confidential. The transcript and notes from this interview will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, RTAC technical client). You may choose not to
answer any question, and you may choose to end the interview at any time. Do you have any questions for me
about the evaluation, its purpose or use, or how we will handle the information you share with us today? [Answer
any questions that are asked.] Do you consent to participate in this evaluation? [If no, thank the respondent and
end the interview. If yes, continue with script.] 
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Please Complete Prior to Interview
First Name: Last Name:
Institution: Position:
RTAC Mid-term Performance Evaluation
1
Date of interview: Sex of Respondent (M/F/O):
Time Started: _ 
[Note to Interviewer: Some modules below may not be relevant to the interviewee’s role within the project. Prior
to the interview, or if not before then at the beginning of the interview, inquire about the role of the respondent
and determine which modules are most relevant to her/him.] 
Background questions
To start, I’d like to hear about your role with the Research Technical Assistance Center (RTAC) funded by USAID.
1. Please tell me about your role with RTAC.
2. What is your level of engagement within the RTAC program?
3. What services have you received to date through the RTAC? Could you tell me a little bit about your
experiences with these services?
I. RTAC RESEARCH NETWORK
For this first section, we would like to focus on the RTAC research network membership.
When we say RTAC network we are referring to the network of individual researchers that NORC maintains to
support USAID research requests.
4. From your perspective, what, if any, is the value of the RTAC network forUSAID?
5. Are there other mechanisms that you think provide the same or similar services? If so, what are they? Have
you drawn on them? If so, do you know how they compare in terms of value for service received? In what
circumstances would you choose them over RTAC, and when would you chooseRTAC?
6. Have you found the structure of the RTAC network effective? How so/why not? What are its strengths and
weaknesses? How could it be improved?
7. Is the management of the RTAC network effective in your opinion? How could it be improved?
8. How would you rate your overall satisfaction with the RTAC network on a scale of 1 to 5, five being highly
satisfied and 1 being highly dissatisfied? 
II. MEETING USAID NEEDS THROUGH RESEARCH AND TRAINING 
For the next section, we would like to focus on the extent to which RTAC has been able to meet USAID’s identified
needs, starting with research. 
9. How would you describe the support provided in the formulation of the scope(s) of work? [Probe on
whether or not the template was used and/or co-creation of the scope.] 
10. To what extent have RTAC tasks been executed on their original schedules? If there have been delays, at
what stage or stages in the process have they occurred (i.e., scoping, contracting, design, research
RTAC Mid-term Performance Evaluation
1
approvals, data collection, analysis, or utilization)? If there were changes, were they initiated by USAID or
the researcher/institution?
11. To what extent have the research findings generated by RTAC addressed the needs USAID hadidentified?
12. How have research findings and recommendations been utilized? (If they have not been utilized, probe on
whether they might be used in the future and, if so, how.) Do you have any early sense of the ripple or
spread effects of the research conducted – changes resulting from its use? [Probe on concrete examples of
use of evidence in decision-making and programming.] 
III. FINAL THOUGHTS
13. What have been the critical successes for RTAC thus far in youropinion?
14. What, from your perspective, have been RTAC’s critical challenges or failures thusfar?
15. In your opinion, are there any ways that RTAC could better support USAID, or be more effective in meeting 
USAID’s needs? 
16. What have I not asked you about your RTAC experience that I should have asked? What else should Iknow
about your experience with the program?
17. Based on your experience thus far, is a mechanism like RTAC an effective way to meet information and/or
training needs (of your mission/OU/or office)? Are there other approaches you think would be equally or
more effective? How so?
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Interviewer(s): 
Location:
Interviewer Comments:
RTAC Mid-term Performance Evaluation
1
RTAC MIDTERM EVALUATION – BUY-IN PRINCIPAL INVESTIGATORS KII PROTOCOL
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Development Lab to
conduct a midterm evaluation of the RTAC program to assess the effectiveness of program interventions to date and
strengthening future program interventions. As a member of the network, your observations will be very helpful in
these areas, and as a principal investigator of a study awarded under RTAC, you have a unique perspective to share
on the process of generating evidence and progress towards using research more effectively to achieve
development goals. The evaluation will be used to make informed decisions about mid-course corrections and
future program efforts.
The information you share with us will be treated as confidential. The transcript and notes from this interview will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, RTAC researcher). You may choose not to
answer any question, and you may choose to end the interview at any time. Do you have any questions for me
about the evaluation, its purpose or use, or how we will handle the information you share with us today? [Answer
any questions that are asked.] Do you consent to participate in this evaluation? [If no, thank the respondent and
end the interview. If yes, continue with script.] 
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Please Complete Prior to Interview
First Name: Last Name:
Institution: Position:
Date of interview: Sex of Respondent (M/F/O):
Title of Award:
Dates of Award:
Country of Residence: Country of Project:
Time Started: 
Background questions
To start, I’d like to hear about your role with the Research Technical Assistance Center (RTAC) funded by USAID.
1. Please tell me about your role with RTAC.
2. Have you worked with USAID before? Please tell me a little about thoseexperiences.
RTAC Mid-term Performance Evaluation
1
I. RTAC RESEARCH NETWORK
For this first section, we would like to focus on the RTAC network itself (as opposed to your project activities).
1. How did you first become involved in the RTAC network? How are you currentlyinvolved?
2. How would you describe the range of research opportunities promoted through the RTACnetwork?
3. , 2From your perspective, what are the benefits, if any, of belonging to the RTAC network?
4. From your perspective, what are the drawbacks, if any, of belonging to the RTAC network?
5. Are there any barriers to participation in the RTAC network that you couldidentify?
6. Is the structure of the RTAC network effective in your opinion? How could it beimproved?
7. Is the management of the RTAC network effective in your opinion? How could it be improved?
8. How would you rate your overall satisfaction with the RTAC network on a scale of 1 to 5, five being highly
satisfied and 1 being highly dissatisfied? 
II. MEETING USAID NEEDS THROUGH RESEARCH
For the next section, we would like to focus on your RTAC contract activities and its (expected) outcomes.
9. Would you please briefly share with us the focus of your research study and the identified information
and/or training needs that your work would help address?
10. To what extent were the information and/or training needs clearly identified by USAID in the solicitation to
which you responded? Was the initial timeline proposed reasonable?
11. Was there any evolution of those identified needs between the time of the solicitation and the time of
award [(where relevant) or the time that your research was completed]?
12. Have you been able to work with the original schedule proposed for your grant activities, or have there
been changes to the schedule? If there were changes, were they initiated on your end or USAID’s end?
13. Can you briefly tell us about your plan for disseminating the evidence generated by your research, if there is 
one? [(If so) To what extent were USAID and other partners involved in the development of the R2A plan?
How will you evaluate the impact of the R2A plan, if at all?] 
III. FINAL THOUGHTS
14. What have I not asked you about your RTAC experience that I should have asked? Is there anything you would
like to add?
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
RTAC Mid-term Performance Evaluation
1
Interviewer(s): 
Location:
Interviewer Comments:
RTAC Mid-term Performance Evaluation
1
RTAC MIDTERM EVALUATION – RTAC CONSORTIUM MEMBER KII PROTOCOL
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Development Lab to
conduct a midterm evaluation of the RTAC program to assess the effectiveness of program interventions to date and
to provide suggestions for strengthening future program interventions. As a core member of the program
management team, you have unique insights into network operations, management of activities, and progress
towards using research more effectively to achieve development goals. The evaluation will be used to make
informed decisions about mid-course corrections and future program efforts.
The information you share with us will be treated as confidential. The transcript and notes from this interview will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, RTAC program manager). You may choose not
to answer any question, and you may choose to end the interview at any time. Do you have any questions for me
about the evaluation, its purpose or use, or how we will handle the information you share with us today? [Answer
any questions that are asked.] Do you consent to participate in this evaluation? [If no, thank the respondent and
end the interview. If yes, continue with script.] 
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Please Complete Prior to Interview
First Name: Last Name:
Institution: Position:
Date of interview: Sex of Respondent (M/F/O):
Time Started: _ 
[Note to Interviewer: Some modules below may not be relevant to the interviewee’s role within the project. Prior
to the interview, or if not before then at the beginning of the interview, inquire about the role of the respondent
and determine which modules are most relevant to her/him.] 
Background questions
To start, I’d like to hear about your role with the Research Technical Assistance Center (RTAC) funded by USAID.
1. Please tell me about your role with RTAC. [Note to Interviewer: Focus on only those modules below that are
relevant to the individual’s or organization’s role.]
I. RTAC RESEARCH NETWORK
For this section, we would like to focus on the RTAC research network membership.
RTAC Mid-term Performance Evaluation
1
2. What is the value proposition of the RTAC research network for USAID from yourperspective?
3. What have been the most successful approaches to expanding the network in your opinion? [Probe on
differences for individual and institutional members.] 
4. What have been the biggest hurdles in creating and expanding the network? [Probe on differences for
individual and institutional members.] 
5. How do members provide feedback on their experience with the network? What has been their most recent
feedback?
6. From your perspective, what are the benefits, if any, of belonging to the RTACnetwork?
7. From your perspective, what are the drawbacks, if any, of belonging to the RTACnetwork?
8. Are there any barriers to participation in the RTAC network that you couldidentify?
9. Is the structure of the RTAC network effective in your opinion? How could it beimproved?
10. Is the management of the RTAC network effective in your opinion? How could it be improved?
II. MEETING USAID NEEDS THROUGH RESEARCH
For the next section, we would like to focus on the extent to which RTAC has been to meet USAID’s identified needs,
starting with research. 
1. What are the USAID needs, in your opinion, has the RTACaddressed?
2. What approaches does RTAC take to address USAID's research and evidenceneeds?
3. Please share a little with us about the development of research to action plans. How has the
development process worked?
4. To what extent has that contributed to evidence use by USAID (or otherstakeholders)?
5. Do you have any early sense of the ripple or spread effects of the research conducted – changes
resulting from its use? [Probe on concrete examples of use of evidence in decision-making and
programming.] 
6. How are the R2A plans monitored and assessed, if at all? 
III. RTAC WEBSITE
1. What have been the approaches does NORC take to address USAID's research and evidenceneeds? 
2. Can you share any examples of how USAID (or other stakeholders) are using or plan to use RTAC
generated evidence? (Probe, have you been involved in research to action plans? To what extent has
that contributed to evidence use??? How are the R2A plans monitored and assessed, if atall?
3. Do you have any early sense of the ripple or spread effects of the research conducted – changes
resulting from its use? [Probe on concrete examples of use of evidence in decision-making and
programming.] 
IV. RTAC TRAINING AND EVENTS
1. Would you please briefly tell us a little bit about the events and trainings that have been held for USAID
personnel?
2. In the evaluations you did of the training, how did participants rate the event/training? Has any follow up
been done to determine the extent to which participants have been able to apply it in their work? If so,
what were the results?
3. How are ongoing information and training needs identified, and what process is used to develop trainings or
technical assistance packages in response to those needs? 
V. FINAL THOUGHTS
RTAC Mid-term Performance Evaluation
1
1. What have been the critical successes for RTAC thus far in youropinion?
2. What have been the critical challenges for RTAC thusfar?
3. Is there anything you would change in how RTAC supports USAID that you think would make RTAC more
effective in meeting those needs?
4. What have I not asked you about your RTAC experience that I should haveasked?
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Interviewer(s): 
Location:
Interviewer Comments:
RTAC Mid-term Performance Evaluation
1
RTAC MIDTERM EVALUATION – RTAC PARTNER COUNTRY COUNTERPARTS KII PROTOCOL
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Lab to conduct a
midterm evaluation of the Research and Technical Assistance Center (RTAC) program, which undertook the [insert
project name] in this country. The goal of the evaluation is to assess the effectiveness of program interventions to
date and strengthening future program interventions. As someone familiar with the impacts of their work, your
observations will be very helpful in these areas. The evaluation will be used to make informed decisions about mid￾course corrections and future program efforts.
The information you share with us will be treated as confidential. The transcript and notes from this interview will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, cooperating country partner). You may choose
not to answer any question, and you may choose to end the interview at any time. Do you have any questions for
me about the evaluation, its purpose or use, or how we will handle the information you share with us today?
[Answer any questions that are asked.] Do you consent to participate in this evaluation? [If no, thank the
respondent and end the interview. If yes, continue with script.] 
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Please Complete Prior to Interview
First Name: Last Name:
Institution: Position:
Date of interview: Sex of Respondent (M/F/O):
Country:
Project:
Time Started: _ 
I. ACHIEVING DEVELOPMENT GOALS THROUGH RESEARCH
For this section, we would like to focus on the [insert project name] that was conducted under RTAC and its
(expected) outcomes.
1. We would like to start by understanding a little more about your role in [insert appropriate phrase:
regulation of financial markets in Honduras or meeting the needs of refugees in Kenya]. How do you (does
your institution) work in thatspace?
a. [Other key informants] Would you please tell us what you know, if anything, about the research 
conducted under this project?
RTAC Mid-term Performance Evaluation
1
2. How was the research conducted relevant to local or national development challenges?
3. Was the evidence generated by the research available in a timely manner?
4. How has information about the research findings been disseminated? [Probe on specific communications 
strategies and products for various audiences.] 
5. Have the methods of dissemination been effective? [Probe on why it was effective orineffective.] 
6. Can you share any examples of how the research findings have been used, or might be used in the future in
policy making or decision-making?
7. Do you think the [insert project name] has had an impact on the local community? If so, what has that
been?
II. FINAL THOUGHTS
8. What recommendations would you make, if any, to the funders of the [insert project name] about ways to
improve the effectiveness of similar efforts in the future?
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Interviewer(s): 
Location:
Interviewer Comments:
RTAC Mid-term Performance Evaluation
1
Focus Group Discussions 
RTAC MIDTERM EVALUATION – RTAC USERS and POTENTIAL USERS - USAID FGD
PROTOCOL
Hello, everyone, and thank you for agreeing to participate in today’s discussion group. My name is [state name],
and I will be leading today’s discussion. I am working on this project for IBTCI, an independent third-party
evaluation firm that has been contracted by the USAID Global Development Lab to conduct a midterm evaluation of
the RTAC program to assess the effectiveness of program interventions to date and strengthening future program
interventions. As an actual or potential user of the RTAC mechanism, you have unique insights into its strategic
value and utility for your office’s portfolio (and perhaps those of the other operating units you support). The
evaluation will be used to make informed decisions about mid-course corrections and future program efforts.
This focus group discussion will last approximately one hour. There are no right or wrong answers to the questions I
am going ask you. We want to capture the full range of opinions that you may have, so please feel free to disagree.
However, please speak one person at a time, because we are recording the focus group session. Recording the focus
group will allow me to concentrate on your responses and ensure that I do not miss anyone’s input during the
discussion. If at any time, anyone feels that they would prefer to speak with the recorder off, please let me know
and I will stop the recorder.
The information you share with us will be treated as confidential. The transcript and notes from this discussion will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, USAID staff). You may choose not to answer any
question.
Do you have any questions for me about the evaluation, its purpose or use, or how we will handle the information
you share with us today? [Answer any questions that are asked.] Do you consent to participate in this evaluation?
[If no, thank the respondent and ask him/her to excuse him/herself from the room. If yes, continue with script.] Is
everyone comfortable with me recording the discussion for my notes as I mentioned? [If yes, turn on the recorder
and continue. If no, do not record the FGD.] 
FGD Screener to be Completed for Each Respondent Prior to FGD
First Name: Last Name:
M/B/IO: Position:
Sex of Respondent (M/F/O):
Time Started: _ 
I. RTAC MECHANISM
RTAC Mid-term Performance Evaluation
1
We greatly appreciate your cooperation and time. To start us off, would you each please share your name, your
Operating Unit and/or Office, and how you first learned about the RTAC mechanism?
1. How do you usually find out about mechanisms when you have a need for a new service or product?
2. What is the value of the RTAC network for USAID from yourperspective?
3. How do you see research or academic expertise supporting the activities of your office or operating unit?
[Probe on how it has in the past and how it might, if differently, in the future.] 
4. As you considered using RTAC, were/are there other mechanisms that you also considered? If so, what
were/are they? What features or services did they offer that RTACA did not? Do you know how they
compare in terms of cost and value for service received? 
5. From what you know of how RTAC operates, is there anything you would change to make it more
responsive to USAID’s programmatic needs? [Probe on structure, management, range of partners involved,
partner characteristics, etc.] 
II. MEETING USAID NEEDS THROUGH RESEARCH AND TRAINING 
For those of you who have used the RTAC mechanism, we would like to ask a bit more about that experience.
1. How would you describe the initial support provided( for example, in thinking through whether RTAC was a
good fit, creating a scope for a research project or training, etc.? [Probe on whether or not the template was
used and/or co-creation of the scope.] What additionalsupport would have been useful to you?
2. To what extent have project activities been executed in a timely manner? [Probe on any delays at the
scoping, contracting, design, research approvals, data collection, analysis, or utilization stages and whether
those delays were on the side of USAID or researcher/institution.] 
3. To what extent has the evidence generated and/or technical assistance provided addressed theneeds
identified by USAID?
4. How has the evidence/findings been utilized? If the research is still in process, how do you envision that it
will be utilized? [Probe on any R2A plan, and whether or not USAID was engaged in its co-creation.] Have
you seen any changes as a result of the use of the evidence/findings? [Probe on concrete examples of use of
evidence in decision-making and programming.] 
III. FINAL THOUGHTS
1. Overall, how would you describe the performance of the RTAC mechanism based on yourexperience? 
2. What have I not asked you about your RTAC experience that I should haveasked?
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
RTAC Mid-term Performance Evaluation
1
Moderator(s): 
Location:
Moderator Comments:
RTAC Mid-term Performance Evaluation
1
RTAC MIDTERM EVALUATION – RTAC PI FGD PROTOCOL
Hello, everyone, and thank you for agreeing to participate in today’s discussion group. My name is [state name],
and I will be leading today’s discussion. I am working on this project for IBTCI, an independent third-party
evaluation firm that has been contracted by the USAID Global Development Lab to conduct a midterm evaluation of
the RTAC program to assess the effectiveness of program interventions to date and strengthening future program
interventions. As a member of the network, your observations will be very helpful in these areas. The evaluation
will be used to make informed decisions about mid-course corrections and future programefforts.
This focus group discussion will last approximately one hour. There are no right or wrong answers to the questions I
am going ask you. We want to capture the full range of opinions that you may have, so please feel free to disagree.
However, please speak one person at a time, because we are recording the focus group session. Recording the focus
group will allow me to concentrate on your responses and ensure that I do not miss anyone’s input during the
discussion. If at any time, anyone feels that they would prefer to speak with the recorder off, please let me know
and I will stop the recorder.
The information you share with us will be treated as confidential. The transcript and notes from this discussion will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, RTAC partner or RTAC network member). You
may choose not to answer any question. Do you have any questions for me about the evaluation, its purpose or use,
or how we will handle the information you share with us today? [Answer any questions that are asked.] Do you
consent to participate in this evaluation? [If no, thank the respondent and ask him/her to excuse him/herself from
the room. If yes, continue with script.] Is everyone comfortable with me recording the discussion for my notes as I
mentioned? [If yes, turn on the recorder and continue. If no, do not record the FGD.] 
FGD Screener to be Completed for Each Respondent Prior to FGD
First Name: Last Name:
Institution: Position:
Department or Discipline: Sex of Respondent (M/F/O):
Country of Residence: 
Time Started: _ 
We greatly appreciate your cooperation and time. To start us off, would you each please share your name, your
organization, and how you first learned about the RTAC research network?
I. ACHIEVING DEVELOPMENT GOALS THROUGH RESEARCH
1. Would you please briefly share with us the focus of your research study and the identified information
and/or training needs that your work would help address?
RTAC Mid-term Performance Evaluation
1
2. Was the research focused on a critical problem for this community or nation? For whom was this problem
urgent?
3. Can you briefly tell us about your approach to disseminating the results of your research? How was the
process of developing your [R2A plan or communications product(s)]?
4. [(For R2A) To what extent were USAID and other partners involved in the development of the R2A plan?
How will you evaluate the impact of the R2A plan, if at all?] 
5. [(For Communications) How was/were the communications product(s) developed? What, if any, audience
analysis was conducted and how were information formats and messages selected and tested? [Probe on
specific communications strategies and products for various audiences.] 
6. In your view, have the modes of communication/information dissemination been effective? [Probe on why it
was effective or ineffective.] 
7. In your opinion, would there have been more effective ways to disseminate information in the community
or at the national level? If so, what would they be?
8. Do you know if the research findings have been used to inform policy making or decision-making? Can you
share any examples? [Probe on use within their organizations and within the community at large.] 
9. Do you think the [insert project name] has had an impact on the local community? If so, what has that
been?
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Moderator(s): 
Location:
Moderator Comments:
RTAC Mid-term Performance Evaluation
1
RTAC MIDTERM EVALUATION – RTAC RESEARCH NETWORK MEMBERS FGD PROTOCOL
Hello, everyone, and thank you for agreeing to participate in today’s discussion group. My name is [state name],
and I will be leading today’s discussion. I am working on this project for IBTCI, an independent third-party
evaluation firm that has been contracted by the USAID Global Development Lab to conduct a midterm evaluation of
the RTAC program to assess the effectiveness of program interventions to date and strengthening future program
interventions. As a member of the network, your observations will be very helpful in these areas. The evaluation
will be used to make informed decisions about mid-course corrections and future programefforts.
This focus group discussion will last approximately one hour. There are no right or wrong answers to the questions I
am going ask you. We want to capture the full range of opinions that you may have, so please feel free to disagree.
However, please speak one person at a time, because we are recording the focus group session. Recording the focus
group will allow me to concentrate on your responses and ensure that I do not miss anyone’s input during the
discussion. If at any time, anyone feels that they would prefer to speak with the recorder off, please let me know
and I will stop the recorder.
The information you share with us will be treated as confidential. The transcript and notes from this discussion will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, RTAC partner or RTAC network member). You
may choose not to answer any question. Do you have any questions for me about the evaluation, its purpose or use,
or how we will handle the information you share with us today? [Answer any questions that are asked.] Do you
consent to participate in this evaluation? [If no, thank the respondent and ask him/her to excuse him/herself from
the room. If yes, continue with script.] Is everyone comfortable with me recording the discussion for my notes as I
mentioned? [If yes, turn on the recorder and continue. If no, do not record the FGD.] 
FGD Screener to be Completed for Each Respondent Prior to FGD
First Name: Last Name:
Institution: Position:
Department or Discipline: Sex of Respondent (M/F/O):
Country of Residence: 
Time Started: _ 
We greatly appreciate your cooperation and time. To start us off, would you each please share your name, your
organization, and how you first learned about the RTAC research network?
1. What were your expectations when you joined the RTAC research network? To what extent have those
expectations been met?
2. From your perspective, what are the benefits, if any, of belonging to the RTACnetwork?
3. From your perspective, what are the drawbacks, if any, of belonging to the RTACnetwork?
4. Are there any barriers to participation in the RTAC network that you could identify? 
RTAC Mid-term Performance Evaluation
1
5. What changes would you make to the RTAC network, if any? [Probe on structure, management, availability 
of research opportunities, availability of other consulting opportunities, etc.] Why and how would those
changes improve the outcomes?
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Moderator(s): 
Location:
Moderator Comments:
RTAC Mid-term Performance Evaluation
1
RTAC MIDTERM EVALUATION – RTAC PARTNER COUNTRY COUNTERPARTS FGD PROTOCOL
Hello, everyone, and thank you for agreeing to participate in today’s discussion group. My name is [state name],
and I will be leading today’s discussion. I am working on this project for IBTCI, an independent third-party
evaluation firm that has been contracted by the USAID Global Development Lab to conduct a midterm evaluation of
the RTAC program to assess the effectiveness of program interventions to date and strengthening future program
interventions. As a member of the network, your observations will be very helpful in these areas. The evaluation
will be used to make informed decisions about mid-course corrections and future programefforts.
This focus group discussion will last approximately one hour. There are no right or wrong answers to the questions I
am going ask you. We want to capture the full range of opinions that you may have, so please feel free to disagree.
However, please speak one person at a time, because we are recording the focus group session. Recording the focus
group will allow me to concentrate on your responses and ensure that I do not miss anyone’s input during the
discussion. If at any time, anyone feels that they would prefer to speak with the recorder off, please let me know
and I will stop the recorder.
The information you share with us will be treated as confidential. The transcript and notes from this discussion will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, cooperating country respondent). You may
choose not to answer any question. Do you have any questions for me about the evaluation, its purpose or use, or
how we will handle the information you share with us today? [Answer any questions that are asked.] Do you
consent to participate in this evaluation? [If no, thank the respondent and ask him/her to excuse him/herself from
the room. If yes, continue with script.] Is everyone comfortable with me recording the discussion for my notes as I
mentioned? [If yes, turn on the recorder and continue. If no, do not record the FGD.] 
FGD Screener to be Completed for Each Respondent Prior to FGD
First Name: Last Name:
Institution: Position:
Country: Sex of Respondent (M/F/O):
Project:
Time Started: _ 
I. ACHIEVING DEVELOPMENT GOALS THROUGH RESEARCH
Would you please start by telling us your name, the institution that you work for, and how you first learned about
the [insert project name]?
RTAC Mid-term Performance Evaluation
1
1. Based on what you know about the project, how was the research conducted relevant to challenges in the
local community or national development challenges? 
2. Was the research focused on a critical problem for this community or nation? For whom was this problem
urgent?
3. How was information about the research findings disseminated? [Probe on specific communications
strategies and products for various audiences.] 
4. Have the modes of dissemination been effective? [Probe on why it was effective orineffective.] 
5. In your opinion, would there have been more effective ways to disseminate information in the community
or at the national level? If so, what would they be?
6. Do you know if the research findings have been used to inform policy making or decision-making? Can you
share any examples? [Probe on use within their organizations and within the community at large.] 
7. Do you think the [insert project name] has had an impact on the local community? If so, what has that
been?
II. FINAL THOUGHTS
8. What recommendations would you make, if any, to the funders of the [insert project name] about ways
to improve the effectiveness of similar efforts in the future?
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Moderator(s): 
Location:
Moderator Comments:
RTAC Mid-term Performance Evaluation
1
Work plan 
RTAC Mid-term Evaluation Work Plan
Introduction
This work plan outlines the IBTCI team’s activities throughout the Research Technical Assistance Center Midterm
Performance Evaluation (RTAC MPE). The work is divided into five discrete phases: i) preparation and design; ii)
preliminary data collection and field testing; iii) data collection; iv) data analysis; and v) report writing. Primary
responsibility for all evaluation activities will lie with consultants, Christine Allison and Kathylynn Pierre Griff, with
support at key periods from IBTCI staff, Betsy Nolan, Ed Allan, and Kathryn Hornbeck. 
Over the life of the evaluation, the RTAC mid-term evaluation team will have monthly calls with USAID to review
progress, to identify any risks to adhering to the timeline, and to discuss lessons learned to date. In particular, the
team anticipates that these calls will represent key learning opportunities, particularly in January and May 2020,
which will complement the other learning activities built into the work plan below (including the inception
presentation in March 2020 and the recommendations development workshop in August 2020). Specific dates for
the monthly calls (set in general for the second Wednesday of the month at 2 pm) are included in the Timeline
below. 
Evaluation Team Roles and Responsibilities
The IBTCI team comprises five members, three of whom will be primarily responsible for the technical aspects of the
evaluation and two of whom will be responsible for oversight and the administrative aspects of the evaluation. Ed
Allan is the interim Project Director for IBTCI, and he will provide overall guidance on the project, as well as serving
as the corporate monitor for all activities. Christine Allison will serve as the Team Leader, and she will be
responsible for the day to day management of evaluation activities. She will work with other team members to
design the evaluation, to collect and analyze the data, to distill findings, and to lead the recommendations workshop
and final drafting of the report. Kathylynn Griff is the Evaluation Methodologist, and she will work hand-in-hand
with the team leader to design the evaluation and the instruments, to collect and analyze the data, to distill findings,
and to author the report. Betsy Nolan will also serve as an Evaluation Methodologist, and she will work with the
team to collect, manage, and analyze the data. Kathryn Hornbeck will provide assistance on contractual and
administrative tasks as needed.
Phase 1: Preparation and Design
The preparation and design work began at contract award (September 2019) and will be completed when USAID
approves the design and initial testing of the instruments (anticipated February 2020). In addition to introductory
meetings between the evaluation team and USAID and with NORC RTAC, the preparation and design phase includes 
two main activities: document review and evaluation design.
Document review (October 18, 2019 – November 15, 2019)
In order to fully understand the program and develop appropriate and well-targeted data collection instruments,
the team reviewed program documents, including the initial RFP and proposal, work plan, MEL plan, semiannual
reports, and year 1 deliverables. Drawing on the evaluation questions, proposed sub-questions (as approved by
USAID), and the documents reviewed, the team began designing the data collection instruments. It should be
noted, however, that the team will review additional documents as they become available over the course of the
evaluation. 
RTAC Mid-term Performance Evaluation
1
Evaluation design (November 18, 2019 – December 13, 2019)
The overarching evaluation design that IBTCI proposed and was accepted by USAID was a mixed- methods,
utilization-focused evaluation that can be used to inform program adaptation and learning. Per the proposal, the
team incorporated the following primary data collection approaches: (1) key informant interviews (KIIs) with
program managers (USAID and NORC personnel); (2) KIIs with USAID staff and key stakeholders; (3) KIIs with RTAC
researchers (n=7); (4) focus group discussions (FGDs) with RTAC providers; (5) FGDs with RTAC clients and
beneficiaries; (6) FGDs with host country counterparts; and (7) a self-administered survey of network participants
(n=503).9
To the extent possible, USAID and NORC will help the team identify and contact the universe of possible
respondents for each of the proposed data collection methods. This will follow the kick-off meetings, from which a
purposive sample of respondents will be drawn to provide the evaluation team with a broad range of experiences
on which to draw. The kick-off meetings and further consultation with USAID and RTAC guided the sampling
strategy for the KIIs and FGDs. The team expects to conduct a census of the RTAC network. For each respondent
group, a contact and communication protocol strategy will be developed and described in the final evaluation
design. The team developed draft data collection instruments for each target audience. The team also developed an
initial coding scheme for the qualitative data and is planning the syntax for the survey to ensure that all data
elements needed to answer the evaluation questions are being collected and structured appropriately. After
submission to USAID for review and receipt of comments and edits (expected mid-January 2020), the team will
finalize the evaluation design and data collection instruments, to be completed no later than February 7, 2020.
The team anticipates that the survey will be self-administered online in two rounds of data collection (an early one
for the original 254 network members and a later one for the 249 joining more recently)., We will attempt to
conduct a census of the network members, however, it is likely that the response rate will be significantly below 100
percent. In order to get the most informative responses, however, the survey will be field tested during the first site
visit (Honduras), anticipated in March 2020, and then resubmitted to USAID to review any changes to response
categories, instructions, or other survey elements. The approved, revised version will then be programmed into an
online survey platform and prepared for dissemination.
Concurrent with the submission of the final evaluation design and data collection instruments, the evaluation team
will submit necessary documentation and clearances (such as a country clearance request) for the first site visit to
be conducted in early March.
Phase 2: Preliminary Data Collection and Field Testing
Phase 2 will focus on initial data collection and any needed refinement of data collection instruments, as described
under each of the data collection methods below. Per the proposal, the team will conduct a total of 48 KIIs and 12
FGDs in addition to the survey10 in the field testing and data collection phases, as shown in the table below.
9 As of December 2019, the network has expanded to 503 participants. We anticipate that the original 254 will be surveyed
early in the data collection process, and the additional 249 added will be surveyed later in the data collection process to allow
them more time to engage with the network prior to asking for their input.
10 The team expects a 50% to 60% response rate for the survey of network members.
RTAC Mid-term Performance Evaluation
1
Data Collection
Method
Preliminary/Field 
Testing Phase
Data Collection Phase Total # 
Planned
Estimated #
of
Respondents
US Overseas/
Honduras
US Overseas/
Kenya 
KIIs 8 6 25 6 43 43
FGDs 1 1 9 1 12 72-96
Survey 1 251
In-briefing (February 2020)
Prior to the initiation of any data collection, the RTAC mid-term evaluation team will participate in an in-briefing 
with USAID. The team anticipates that the in-briefing will be conducted virtually in February 2020 after the final
evaluation design has been approved by USAID.
11
Sampling plan and outreach to field test respondents (February 2020)
After the in-briefing has been completed, the team will finalize the sampling plan and begin reaching out to the
initial sample of respondents for KIIs and FGDs with whom the initial interviews and field testing will be completed
(see below for a description of the target groups for field testing for each data collection method).
Initial KIIs (March 2-6, 2020 – to be adjusted around site visit)
The team will conduct an initial set of eight interviews to include the program and activity managers at USAID and
NORC, as well as at least one network participant and one USAID stakeholder (not a program manager). In addition
to collecting initial data, this first round of interviews will be used to fine tune all three interview protocols, allowing
the team to identify additional opportunities for probing or alternative ways of framing some questions that might
make them easier for respondents to answer and/or facilitate a higher participant response rate. To ensure that the
qualitative data coding is done rigorously and consistently, two of the initial KIIs will be coded by all team members
and then the results will be compared and worked through to ensure that all of the concepts included in the coding
structure are understood and applied consistently.
FGD field test (March 2-6, 2020 – to be adjusted around site visit)
Similar to the plan for KIIs, the team will conduct one FGD with Washington, DC-based USAID stakeholders to fine
tune the FGD protocol. The team will also use this opportunity to field test the survey to be used with USAID
11 The in-briefing should be scheduled as early as possible to allow the widest windows for Mission timing.
RTAC Mid-term Performance Evaluation
1
stakeholders and to make any edits that will improve the quality of the data. The FGD field test notes will also be
coded by all team members, similar to the KIIs, using the same process of comparing and working through the
results to ensure that all of the concepts included in the coding structure are understood and applied consistently,
as the team proceeds with data collection.
Site visit #1 Honduras (o/a February 24-March 13, 2020 – final dates to be determined with USAID and partners)12
The team will travel to Honduras for one week, and after an initial meeting with USAID, the team will conduct
approximately 6 KIIs and 1 FGD with key stakeholders. The team also intends to field test the survey of network
participants and the survey of RTAC researchers that will be administered online. At the completion of the visit, the
team will out brief with the Mission prior to departure.
Finalization of survey instruments (March 16-20, 2020)
At the completion of all of the initial data collection, the team will revise the three survey instruments based on the
results of the field tests. These will then be resubmitted to USAID for review and approval prior to being
programmed into an online platform. At the same time, the syntax drafted as part of the evaluation plan will be
updated to ensure that it is ready for use when data analysis begins.
Inception presentation (March 23-27, 2020)
At the completion of the design and field-testing stages, the team will develop the inception presentation based on
the results of the document review and the preliminary data collected through field testing and initial interviews.
This represents a key learning opportunity, and the team expects that the presentation will be structured as a
learning event that may involve USAID and the RTAC team.
Phase 3: Data Collection
The team will then move on to the remainder of the data collection tasks with the final instruments.
Site Visit #2 Kenya (April 20-24, 2020, May 4-8, or May 11-15 – final dates to be determined with USAID and
partners)13
The team will travel to Kenya in April or May, and after an initial meeting with USAID, the team will conduct
approximately 6 KIIs and 1 FGD with key stakeholders in Nairobi. At the completion of the visit, the team will
outbrief with the Mission prior to departure.
Domestic KIIs (April 2020)
12 If university personnel are to be included in in-country data collection, exams are in early March followed by graduation and a 
long Easter break. It may be difficult to get anyone from universities to participate in that time frame.
13 Limited date options in order to avoid MPE team external commitments, Easter celebrations, May Day, and the beginning and
end of Ramadan. All these options fall during second semester examination processing, however, so we may need to adjust if
respondents are not available then (exams start March 30 and run through mid-late April). Some universities have a summer
trimester and others do not, so availability may vary.
RTAC Mid-term Performance Evaluation
1
The team will conduct an additional 20 KIIs with US-based key informants identified in the sampling plan. The KIIs
will be written up and coded using the draft coding structure as they are completed, and any revisions/additions to
the coding structure will be shared with all team members to ensure consistency in its use.
Domestic FGDs (April 2020)
The team will also conduct 6 FGDs with USAID stakeholders and 3 FGDs with RTAC network members identified in
the sampling plan. The FGDs will be written up and coded using the draft coding structure as they are completed,
and any revisions/additions to the coding structure will be shared with all team members to ensure consistency in
its use.
Survey administration (April 15-May 15, 2020)
Once the final survey instruments have been approved by USAID, the survey will be programmed into an online
survey platform and disseminated by email to the potential respondent pool. We anticipate conducting a census of
all potential respondents, recognizing that self-selection bias will influence the final response rate for each
respondent group. Overall, we expect to have response rates of 50-60%, and the team will work with USAID and
RTAC to identify the most effective outreach strategies to encourage participation (to be detailed in the evaluation
design).
Re-interviews (June 2020-August 2020)
As this evaluation is adopting a CLA approach, the team anticipates that the RTAC program managers at USAID and
NORC may make adaptations to the program over the life of the evaluation. The team therefore plans to conduct
up to 2 FGDs and 8 KIIs with key stakeholders, including program managers, at the very end of the data collection
phase to capture any changes made to the RTAC program made in response to learning activities and after the initial
interviews conducted earlier in the evaluation. 
Phase 4: Data Analysis
During Phase 4, the team will conduct data analysis and develop preliminary findings, lessons learned, and
recommendations for USAID, NORC, and network members.
Data analysis and submission of data sets to DDL (June 2020)
Upon completion of the data collection, the team will analyze the cleaned survey data and triangulate the survey
data with the coded KII and FGD data. The team will then draft the preliminary findings of the evaluation. These
findings will be used as the basis for the Recommendations Development Workshop. The final, cleaned, de￾identified survey data will be submitted to DDL, to be embargoed until the final report is approved.
Recommendations Development Workshop planning (June-July 2020)
The team will work with LAB/CDR and RTAC to determine the best dates and location for the workshop, as well as
the list of people to be invited to participate. The team will develop an agenda and identify any AV support requests
it may have and will provide those to USAID and RTAC as early as possible. 
Recommendations Development Workshop (August 2020)
RTAC Mid-term Performance Evaluation
1
Working with the LAB/CDR and RTAC teams, the evaluation team will conduct a half-day workshop at a USAID or
RTAC venue in Washington, DC to review the results of the data analysis and develop recommendations to
strengthen the RTAC program. As no resources in the budget are allocated for direct workshop costs other than
labor, the IBTCI team will plan content and facilitate the workshop, but will not be responsible for securing the
venue or providing refreshments. The team anticipates that the results of the analysis will be presented, and the
group will identify priority areas for adapting the current program. Depending on the number of participants, the
group may work in plenary or divide into smaller groups to develop key recommendations for concrete, specific
program adaptations. The results of the workshop will be recorded, and the identified recommendations will be
incorporated into the final report.
Phase 5: Report Writing
During the final phase of the project, the team will draft the report and recommendations and respond to any
USAID comments in the final version.
Draft report (September 2020)
The team will develop a draft report that adheres to the outline included in the RFP/award to ensure that all key
evaluation questions have been clearly answered and that the evaluation process has been duly documented. The
team expects to submit the report to USAID in early October 2020 for initial review. 
USAID review and edits (October – December 2020)
The evaluation team expects that the LAB/CDR team will provide a first round of edits on the draft report, and that
the evaluation team will revise the report to address those comments (including the submission of a table that
tracks each comment and how the team addressed it). The team then anticipates submitting a second draft report
in November 2020 for broader stakeholder review and input, and will repeat the same process of presenting a
revisions and comment tracking table for the second round of review.
Final report (January 2021)
Having addressed all USAID comments, the team will submit the final report in January 2021.
Submission of final report to DEC (February 2021)
Having received final approval of the final report, the team will submit the approved public version of the final
report to the DEC.
Timeline
The table below shows the consolidated timeline for the RTAC mid-term evaluation. 
Task Dates (estimated
dates, may need to be
adjusted to
accommodate
schedules)
RTAC Mid-term Performance Evaluation
1
Phase 1 
Background reading (document summaries) Oct 18-Nov 15, 2019
Work plan submission November 8, 2019
Monthly call November 13, 2019
Kick-off meeting with RTAC November 18-22, 2019
Monthly call December 11, 2019
Draft Evaluation Design December 13, 2019
USAID Review of Evaluation Design December 16, 2019 – 
January 31, 2020
Monthly call – CLA discussion of lessons learned through document review January 8, 2020
Final Evaluation Design / Submission of travel request 1 February 7, 2020
Monthly call February 12, 2020
Phase 2 
In briefing February 21, 2020
Initial contacts to KII sample, FGD sample February 22, 2020
Initial KIIs March 2-6, 2020
Field test FGD protocol March 2-6, 2020
Site visit #1 Honduras TBD
Revise final survey instruments March 16, 2020
Monthly call (postponed one week due to field work) March 17, 2020
USAID review of final survey instruments March 17-31, 2020
RTAC Mid-term Performance Evaluation
1
Inception Presentation / Submission of travel request 2 March 23-27, 2020
Phase 3 
Domestic KIIs April 6-17, 2020
Domestic FGDs April 6-17, 2020
Monthly call April 8, 2020
Survey programming, testing, and launch April 13-15, 2020
Site visit #2 Kenya TBD
Monthly call – CLA discussion of qualitative data lessons learned May 13, 2020
Survey closes May 15, 2020
Clean survey data May 18-20, 2020
Re-interviews (domestic KIIs & FGDs) June 1-5, 2020
Phase 4 
Data Analysis June 8-19, 2020
Monthly call June 10, 2020
Submit survey data sets to DDL (to be embargoed until final report is
approved)
June 29, 2020
Recommendation Development Workshop planning June-July 2020
Monthly call July 8, 2020
Recommendation Development Workshop August 2020
Monthly call August 12, 2020
Phase 5 
RTAC Mid-term Performance Evaluation
1
Draft report September 2020
Monthly call September 9, 2020
USAID Review October-December 
2020
Monthly call October 14, 2020
Monthly call* (postponed 1 day due to Veteran’s Day holiday) November 12, 2020
Monthly call December 9, 2020
Final Report January 2021
Monthly call January 12, 2021
Submission of final report to DEC February 2021
RTAC Mid-term Performance Evaluation
1
ANNEX C: DATA COLLECTION INSTRUMENTS
1. RTAC MIDTERM EVALUATION – RTAC PROGRAM MANAGERS (NORC) 
SURVEY
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Development Lab to
conduct a midterm evaluation of the RTAC program to assess the effectiveness of program interventions to date and
strengthening future program interventions. As a core member of the program management team, you have
unique insights into network operations, management of awards, and progress towards using research more
effectively to achieve development goals. The evaluation will be used to make informed decisions about mid-course
corrections and future program efforts. The information you share with us will be treated asconfidential.
Please take a moment to respond to this satisfaction survey. The estimated time to complete the 38 question survey
is 10 minutes. Please answer each question to the best of your ability.
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Demographic Information
Highest Education Attained: Years working on the RTAC Project:
Institution: Position:
Date of interview: Sex of Respondent (M/F/O):
QUESTIONS RESPONSES
1. How did you first become involved in the RTAC
network? Please select all that apply.
2. I understand how the RTAC programfunctions
and the services it offers.
4. The techniques used to recruit RTAC members
is effective. 
5. There are many opportunities for research and
training in the RTAC Network.
6. The RTAC network expansion approach is
effective. 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
7. The RTAC website is easy to navigate.
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
RTAC Mid-term Performance Evaluation
1
QUESTIONS RESPONSES
8. The RTAC website information is consistently
updated.
9. I use the RTAC website as a platformto
disseminate RTAC information.
10. RTAC communicates mechanisms for new
service or products regularly. 
11. USAID contracting answers rapidly to my
and/or institution's service request. 
12. The RTAC adds value to the Global
Development Lab, and USAID Missions/ Bureaus
and IOs as a whole. 
13. The RTAC led activities address the needs
identified by USAID.
14. Overall, the RTAC Project model has been
implemented with ease. 
15. Timelines established to carry out research
are realistic and respected.
16. The research findings have been used to
change policy in the following ways (select all
that which apply):
17. Becoming a member of RTAC has encouraged
me and/or my institution to use other
mechanisms to support our professional and
strategic goals.
18. How would you rate your level of
engagement with the RTAC network?
19. The information and/or training needs are
clearly identified by USAID in the solicitations 
20. The focus of the RTAC research solicitations is
in line with the academic institutions’ needs.
21. Please select all the research and training
activities that you have been involved in since
becoming a member of the RTAC network.
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
RESPONSE SELCTION TO BE ATTAINED VIA KII AND FGD
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
1. Not engaged 2. Somewhat Engaged 3. Engaged
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
RESPONSE SELCTION TO BE ATTAINED VIA KII AND FGD
RTAC Mid-term Performance Evaluation
1
QUESTIONS RESPONSES
22. How would you rate the overall success of the
research and training activities lead by the RTAC?
23. How would you relate the level of pertinence
that your research and training activities bear on
your community / country?
24. Becoming a member of RTAC has encouraged
me and/or my institution to use other
mechanisms to support our professional and
strategic goals.
25. RTAC has supported my institution’s capacity
building needs.
1. NOT SUCCESSFUL 
2. SOMEWHAT SUCESSFUL 
3. SUCCESSFUL 
4. HIGHLY SUCCESSFUL 
1. NOT PERTINENT 2. SOMEWHAT PERTINENT
3. PERTINENT 4. VERY PERTINENT
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
26. The RTAC offers many benefits its members.
27. Becoming a member of RTAC has supported
my professional development needs.
28. The RTAC answers rapidly to my and/or
institution's service request. 
29. I (or my institution) is very involved in the
development of research to action (R2A)plans.
30. The activities supported through the R2A
plans include:
31. The research findings and impacts from the
R2A plans are easily disseminated. 
32. I have used the following method(s) to
disseminate research and training activities while 
a member at RTAC: (Please select all those that
apply)
33. RTAC communicates mechanisms for new
service or products regularly. 
34. The RTAC provides support for research
dissemination using many effective methods.
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
OPEN
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
OPEN AND TO BE ATTAINED THROUGH KIIS AND FGD
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
RTAC Mid-term Performance Evaluation
1
QUESTIONS RESPONSES
35. The research findings have been usedto
change policy. 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
1. HIGHLY DISSATISFIED
36. How would you rate your overall satisfaction
with the RTAC network on a scale of 1 to 5, five
being highly satisfied and 1 being highly
dissatisfied? 
37. Overall, how would you describe the
performance of the RTAC mechanism basedon
your experience?
38. Please add any additional comments
regarding your experience at RTAC.
2. NOT STATISFIED
3. SOMEWHAT SATISFIED
4. SATISFIED
5. HIGHLY SATISFIED
1. NOT EFFECTIVE
2. SOMEWHAT EFFECTIVE
3. EFFECTIVE
4. HIGHLY EFFECTIVE
OPEN
2. RTAC MIDTERM EVALUATION – RTAC PARTNERS AND NETWORK
MEMBERS SURVEY
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Development Lab to
conduct a midterm evaluation of the RTAC program to assess the effectiveness of program interventions to date and
strengthening future program interventions. As a partner and/or network member of the project, your experience
with the network is important for providing feedback for the RTAC program to effectively achieve development
goals. The evaluation will be used to make informed decisions about mid-course corrections and future program
efforts. The information you share with us will be treated as confidential.
Please take a moment to respond to this satisfaction survey. The estimated time to complete the 38 question survey
is 10 minutes. Please answer each question to the best of your ability.
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Demographic Information
Highest Education Attained: Years working on the RTAC Project:
Institution: Position:
RTAC Mid-term Performance Evaluation
1
Date of interview: Sex of Respondent (M/F/O):
RTAC Mid-term Performance Evaluation
1
QUESTIONS RESPONSES
1. How did you first become involved in the RTAC
network? Please select all that apply.
2. I understand how the RTAC programfunctions
and the services it offers.
4. The techniques used to recruit RTAC members
is effective. 
5. There are many opportunities for research and
training in the RTAC Network.
6. The RTAC network expansion approach is
effective. 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
7. The RTAC website is easy to navigate.
8. The RTAC website information is consistently
updated.
9. I use the RTAC website as a platformto
disseminate information.
10. RTAC communicates mechanisms for new
service or products regularly. 
11. The RTAC management team answers rapidly
to my and/or institution's service request. 
12. The RTAC adds value to my institution as a
whole. 
13. The RTAC led activities address the needs
identified by my affiliated institution.
14. Overall, the RTAC Project model has been
implemented with ease. 
15. Timelines established to carry out research
are realistic and respected.
16. The research findings have been used to
change policy in the following ways (select all
that which apply):
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
RESPONSE SELCTION TO BE ATTAINED VIA KII AND FGD
RTAC Mid-term Performance Evaluation
1
QUESTIONS RESPONSES
17. Becoming a member of RTAC has encouraged
me and/or my institution to use other
mechanisms to support our professional and
strategic goals.
18. How would you rate your level of
engagement with the RTAC network?
19. The information and/or training needs are
clearly identified by USAID in the solicitations 
20. The focus of the RTAC research solicitations is
in line with the academic institutions’ needs.
21. Please select all the research and training
activities that you have been involved in since
becoming a member of the RTAC network.
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. Not engaged 2. Somewhat Engaged 3. Engaged
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
RESPONSE SELCTION TO BE ATTAINED VIA KII AND FGD
22. How would you rate the overall success of the
research and training activities that you have
been involved in since becoming a member ofthe
RTAC?
23. How would you relate the level of pertinence
that your research and training activities bear on
your community / country?
24. Becoming a member of RTAC has encouraged
me and/or my institution to use other
mechanisms to support our professional and
strategic goals.
25. RTAC has supported my institution’s capacity
building needs.
1. NOT SUCCESSFUL 
2. SOMEWHAT SUCESSFUL 
3. SUCCESSFUL 
4. HIGHLY SUCCESSFUL 
1. NOT PERTINENT 2. SOMEWHAT PERTINENT
3. PERTINENT 4. VERY PERTINENT
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
development of research to action (R2A)plans.
26. The RTAC offers many benefits its members.
27. Becoming a member of RTAC has supported
my professional development needs.
28. The RTAC answers rapidly to my and/or
institution's service request. 
RTAC Mid-term Performance Evaluation
1
29. I (or my institution) is very involved in the
RTAC Mid-term Performance Evaluation
1
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT
AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T
KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT
AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T
KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT
AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T
KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT
AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T 
KNOW 
RTAC Mid-term Performance Evaluation
1
QUESTIONS RESPONSES
30. The activities supported through the R2A
plans include:
31. The research findings and impacts from the
R2A plans are easily disseminated. 
32. I have used the following method(s) to
disseminate research and training activities while 
a member at RTAC: (Please select all those that
apply)
33. RTAC communicates mechanisms for new
service or products regularly. 
34. The RTAC provides support for research
dissemination using many effective methods.
35. The research findings have been used to
change policy. 
OPEN
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
OPEN AND TO BE ATTAINED THROUGH KIIS AND FGD
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
1. HIGHLY DISSATISFIED
36. How would you rate your overall satisfaction
with the RTAC network on a scale of 1 to 5, five
being highly satisfied and 1 being highly
dissatisfied? 
37. Overall, how would you describe the
performance of the RTAC mechanism basedon
your experience?
38. Please add any additional comments
regarding your experience at RTAC.
2. NOT STATISFIED
3. SOMEWHAT SATISFIED
4. SATISFIED
5. HIGHLY SATISFIED
1. NOT EFFECTIVE
2. SOMEWHAT EFFECTIVE
3. EFFECTIVE
4. HIGHLY EFFECTIVE
OPEN
3. RTAC MIDTERM EVALUATION – RTAC COOPERATING COUNTRY PARTNERS
SURVEY
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Development Lab to
conduct a midterm evaluation of the RTAC program to assess the effectiveness of program interventions to date and
RTAC Mid-term Performance Evaluation
1
strengthening future program interventions. As a partner and/or network member of the project, your experience 
RTAC Mid-term Performance Evaluation
1
with the network is important for providing feedback for the RTAC program to effectively achieve development 
RTAC Mid-term Performance Evaluation
1
goals. The evaluation will be used to make informed decisions about mid-course corrections and future program
efforts. The information you share with us will be treated as confidential.
Please take a moment to respond to this satisfaction survey. The estimated time to complete the 38 question survey
is 10 minutes. Please answer each question to the best of your ability.
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Demographic Information
Highest Education Attained: Years working on the RTAC Project:
Institution: Position:
Date of interview: Sex of Respondent (M/F/O):
QUESTIONS RESPONSES
1. How did you first become involved in the RTAC
network? Please select all that apply.
2. I understand how the RTAC programfunctions
and the services it offers.
4. The techniques used to recruit RTAC members
is effective. 
5. There are many opportunities for research and
training in the RTAC Network.
6. The RTAC network expansion approach is
effective. 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
7. The RTAC website is easy to navigate.
8. The RTAC website information is consistently
updated.
9. I use the RTAC website as a platformto
disseminate information.
10. RTAC communicates mechanisms for new
service or products regularly. 
11. The RTAC management team answers rapidly
to my and/or institution's service request. 
RTAC Mid-term Performance Evaluation
1
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT
AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T
KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT
AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T 
KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT
AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T 
KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT
AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T 
KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT
AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T 
KNOW 
RTAC Mid-term Performance Evaluation
1
12. The RTAC adds value to my institution as a
whole. 
13. The RTAC led activities address the needs
identified by my affiliated institution.
14. Overall, the RTAC Project model has been
implemented with ease. 
15. Timelines established to carry out research
are realistic and respected.
16. The research findings have been used to
change policy in the following ways (select all
that which apply):
17. Becoming a member of RTAC has encouraged
me and/or my institution to use other
mechanisms to support our professional and
strategic goals.
18. How would you rate your level of
engagement with the RTAC network?
19. The information and/or training needs are
clearly identified by USAID in the solicitations 
20. The focus of the RTAC research solicitations is
in line with the academic institutions’ needs.
21. Please select all the research and training
activities that you have been involved in since
becoming a member of the RTAC network.
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
RESPONSE SELCTION TO BE ATTAINED VIA KII AND FGD
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. Not engaged 2. Somewhat Engaged 3. Engaged
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
RESPONSE SELCTION TO BE ATTAINED VIA KII AND FGD
22. How would you rate the overall success of the
research and training activities that you have
been involved in since becoming a member ofthe
RTAC?
23. How would you relate the level of pertinence
that your research and training activities bear on
your community / country?
24. Becoming a member of RTAC has encouraged
me and/or my institution to use other
mechanisms to support our professional and
strategic goals.
1. NOT SUCCESSFUL 
2. SOMEWHAT SUCESSFUL 
3. SUCCESSFUL 
4. HIGHLY SUCCESSFUL 
1. NOT PERTINENT 2. SOMEWHAT PERTINENT
3. PERTINENT 4. VERY PERTINENT
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
RTAC Mid-term Performance Evaluation
1
25. RTAC has supported my institution’s capacity
building needs.
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
26. The RTAC offers many benefits its members.
27. Becoming a member of RTAC has supported
my professional development needs.
28. The RTAC answers rapidly to my and/or
institution's service request. 
29. I (or my institution) is very involved in the
development of research to action (R2A)plans.
30. The activities supported through the R2A
plans include:
31. The research findings and impacts from the
R2A plans are easily disseminated. 
32. I have used the following method(s) to
disseminate research and training activities while 
a member at RTAC: (Please select all those that
apply)
33. RTAC communicates mechanisms for new
service or products regularly. 
34. The RTAC provides support for research
dissemination using many effective methods.
35. The research findings have been used to
change policy. 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
OPEN
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
OPEN AND TO BE ATTAINED THROUGH KIIS AND FGD
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW
1. STRONGLY DISAGREE 2. DISAGREE 3. SOMEWHAT AGREE
4. AGREE 5. HIGHLY AGREE 6. NON APPLICABLE 7. I DON'T KNOW 
1. HIGHLY DISSATISFIED
36. How would you rate your overall satisfaction
with the RTAC network on a scale of 1 to 5, five
being highly satisfied and 1 being highly
dissatisfied? 
37. Overall, how would you describe the
performance of the RTAC mechanism basedon
your experience?
RTAC Mid-term Performance Evaluation
1
2. NOT STATISFIED
3. SOMEWHAT SATISFIED
4. SATISFIED
5. HIGHLY SATISFIED
1. NOT EFFECTIVE
2. SOMEWHAT EFFECTIVE
3. EFFECTIVE
4. HIGHLY EFFECTIVE
RTAC Mid-term Performance Evaluation
1
38. Please add any additional comments
regarding your experience at RTAC.
OPEN
RTAC MIDTERM EVALUATION – RTAC PROGRAM MANAGERS (NORC)
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Development Lab to
conduct a midterm evaluation of the RTAC program to assess the effectiveness of program interventions to date and
strengthening future program interventions. As a core member of the program management team, you have
unique insights into network operations, management of awards, and progress towards using research more
effectively to achieve development goals. The evaluation will be used to make informed decisions about mid-course
corrections and future program efforts.
The information you share with us will be treated as confidential. The transcript and notes from this interview will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, RTAC program manager). You may choose not
to answer any question, and you may choose to end the interview at any time. Do you have any questions for me
about the evaluation, its purpose or use, or how we will handle the information you share with us today? [Answer
any questions that are asked.] Do you consent to participate in this evaluation? [If no, thank the respondent and
end the interview. If yes, continue with script.] 
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Please Complete Prior to Interview
First Name: Last Name:
Institution: Position:
Date of interview: Sex of Respondent (M/F/O):
RTAC Mid-term Performance Evaluation
1
Time Started: _ 
RTAC Mid-term Performance Evaluation
1
[Note to Interviewer: Some modules below may not be relevant to the interviewee’s role within the
project. Prior to the interview, or if not before then at the beginning of the interview, inquire about the
role of the respondent and determine which modules are most relevant to her/him.] 
RTAC Mid-term Performance Evaluation
1
I. RTAC NETWORK 
For this first section, we would like to focus on the RTAC network.
1. What is the value proposition of the RTAC network for USAID from your perspective?
2. How many institutions are currently in the network? How many individuals?
3. What have been the most successful approaches to expanding the network in your opinion? [Probe on
differences for individual and institutional members.]
4. What have been the biggest hurdles in creating and expanding the network?[Probe on differences for
individual and institutional members.]
5. In any network, some members are going to be more active than others. How do you monitor for activity or
engagement by network members? What characterizes a very activemember?
6. Thinking about a scale that goes from inactivity to low activity to regular activity to high activity, what
percentage of network members would you put in each category? [Prompt for each category as needed.]
7. How do members provide feedback on their experience with the network? What has been their most recent
feedback?
8. How would you describe the range of research opportunities promoted through the RTAC network? [Probe 
on whether or not the range meets expectations.]
9. What have been the most effective network activities in your opinion? [Probe on research, expert panels,
promotion of research findings, collaboration among researchers, etc.]
10. What network activities, if any, have not yielded the results expected thusfar?
11. From your perspective, what are the benefits, if any, of belonging to the RTACnetwork?
12. From your perspective, what are the drawbacks, if any, of belonging to the RTACnetwork?
13. Are there any barriers to participation in the RTAC network that you could identify?
14. Is the structure of the RTAC network effective in your opinion? How could it beimproved?
15. Is the management of the RTAC network effective in your opinion? How could it be improved?
II. MEETING USAID NEEDS THROUGH RESEARCH AND TRAINING
For the next section, we would like to focus on the extent to which RTAC has been to meet USAID’s identified needs,
starting with research. 
1. Has the stream of research requests been as expected? How has the stream of requests aligned with
planned levels of effort based on the information provided in the initial RFP?
2. To what extent have the requests for research you have received been well formulated and ready to share
with partners in the network? [Probe on the extent to which the research questions are solid and well
informed.] Has the dissemination of a template improved the quality of the research requestsreceived?
3. To what extent have research projects been executed on their original schedules? If there have been
delays, at what stage or stages in the process have they occurred (i.e., scoping, contracting, design, research
approvals, data collection, analysis, or utilization)? If there were changes, were they initiated by the
researcher/institution or USAID?
4. To what extent has the evidence generated addressed the needs identified byUSAID?
5. How has evidence been utilized? Have the research to action plans been effective? How have the R2A
plans been monitored and assessed, if at all?
6. Do you have any early sense of the ripple or spread effects of the research conducted – changes resulting
from its use? [Probe on concrete examples of use of evidence in decision-making and programming.]
7. Would you please briefly tell us a little bit about the trainings that have been held for USAIDpersonnel?
RTAC Mid-term Performance Evaluation
1
8. In the evaluations you did of the training, how did participants rate the training? Has any follow up been
done to determine the extent to which participants have been able to apply it in their work? If so, what
were the results?
9. Has the stream of training and technical assistance requests been as expected? How has the stream of
requests aligned with planned levels of effort based on the information provided in the initialRFP?
10. How are ongoing training needs identified, and what process is used to develop trainings or technical
assistance packages in response to those needs?
III. FINAL THOUGHTS 
1. What have been the critical successes for RTAC thus far in youropinion?
2. What have been the critical challenges for RTAC thusfar?
3. Is there anything you would change in how RTAC supports USAID that you think would make RTAC more
effective in meeting those needs?
4. What have I not asked you about your RTAC experience that I should have asked?
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Interviewer(s): 
Location:
Interviewer Comments:
RTAC MIDTERM EVALUATION – RTAC PROGRAM MANAGERS (USAID) 
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Development Lab to
conduct a midterm evaluation of the RTAC program to assess the effectiveness of program interventions to date and
strengthening future program interventions. As a core member of the program management team, you have
RTAC Mid-term Performance Evaluation
1
unique insights into RTAC’s operations, management, and results in support of using research more effectively to
achieve development goals. The evaluation will be used to make informed decisions about mid-course corrections
and future program efforts.
The information you share with us will be treated as confidential. The transcript and notes from this interview will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, RTAC program manager). You may choose not
to answer any question, and you may choose to end the interview at any time. Do you have any questions for me
about the evaluation, its purpose or use, or how we will handle the information you share with us today? [Answer
any questions that are asked.] Do you consent to participate in this evaluation? [If no, thank the respondent and
end the interview. If yes, continue with script.] 
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Please Complete Prior to Interview
First Name: Last Name:
Institution: Position:
Date of interview: Sex of Respondent (M/F/O):
Time Started: _ 
[Note to Interviewer: Some modules below may not be relevant to the interviewee’s role within the
project. Prior to the interview, or if not before then at the beginning of the interview, inquire about the
role of the respondent and determine which modules are most relevant to her/him.] 
RTAC Mid-term Performance Evaluation
1
I. RTAC NETWORK 
For this first section, we would like to focus on the RTAC network.
1. What is the value proposition of the RTAC network for USAID from your perspective? 
2. Are there other mechanisms that you think provide the same or similar services? If so, what are they? If so,
do you know how they compare in terms of value for service received? 
3. [Skip for buy-ins] How would you describe the range of research opportunities promoted through the RTAC
network? [Probe on whether or not the range meetsexpectations.] 
4. [Skip for buy-ins] What have been the most effective network activities in your opinion? [Probe on research,
expert panels, promotion of research findings, collaboration among researchers,etc.] 
5. [Skip for buy-ins] What network activities, if any, have not yielded the results expected thusfar?
6. [Skip for buy-ins] From your perspective, what are the benefits, if any, of belonging to the RTAC network?
7. [Skip for buy-ins] From your perspective, what are the drawbacks, if any, of belonging to the RTAC network?
8. [Skip for buy-ins] Are there any barriers to participation in the RTAC network that you couldidentify?
9. Is the structure of the RTAC network effective in your opinion? How could it beimproved?
10. Is the management of the RTAC network effective in your opinion? How could it be improved?
11. How would you rate your overall satisfaction with the RTAC network on a scale of 1 to 5, five being highly
satisfied and 1 being highly dissatisfied? 
II. MEETING USAID NEEDS THROUGH RESEARCH AND TRAINING
For the next section, we would like to focus on the extent to which RTAC has been to meet USAID’s identified needs,
starting with research. 
12. [Skip for buy-ins] Has the stream of research requests been as expected? How has the stream of requests
aligned with planned levels of effort based on the information provided in the initial RFP?
13. How would you describe the support provided in the formulation of research requests? [Probe on whether
or not the template was used and/or co-creation of thescope.]
14. To what extent have project activities been executed on their original schedules? If there have been delays,
at what stage or stages in the process have they occurred (i.e., scoping, contracting, design, research
approvals, data collection, analysis, or utilization)? If there were changes, were they initiated by USAID or
the researcher/institution?
15. To what extent has the evidence generated addressed the needs identified byUSAID?
16. How has evidence been utilized? Have the research to action plans been effective? Have the R2A plans
been co-created effectively with USAID? How have the R2A plans been monitored and assessed,if at all?
17. Do you have any early sense of the ripple or spread effects of the research conducted – changes resulting
from its use? [Probe on concrete examples of use of evidence in decision-making and programming.]
III. FINAL THOUGHTS 
18. What have been the critical successes for RTAC thus far in youropinion?
19. What have been the critical challenges for RTAC thusfar?
20. Is there anything you would change in how RTAC supports USAID that you think would make RTAC more
effective in meeting those needs?
21. What have I not asked you about your RTAC experience that I should have asked?
RTAC Mid-term Performance Evaluation
1
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Interviewer(s): 
Location:
Interviewer Comments:
RTAC Mid-term Performance Evaluation
1
RTAC MIDTERM EVALUATION – RTAC RESEARCHERS
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Development Lab to
conduct a midterm evaluation of the RTAC program to assess the effectiveness of program interventions to date and
strengthening future program interventions. As a member of the network, your observations will be very helpful in
these areas, and as a principal investigator of a study awarded under RTAC, you have a unique perspective to share
on the process of generating evidence and progress towards using research more effectively to achieve
development goals. The evaluation will be used to make informed decisions about mid-course corrections and
future program efforts.
The information you share with us will be treated as confidential. The transcript and notes from this interview will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, RTAC researcher). You may choose not to
answer any question, and you may choose to end the interview at any time. Do you have any questions for me
about the evaluation, its purpose or use, or how we will handle the information you share with us today? [Answer
any questions that are asked.] Do you consent to participate in this evaluation? [If no, thank the respondent and
end the interview. If yes, continue with script.] 
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Please Complete Prior to Interview
First Name: Last Name:
Institution: Position:
Date of interview: Sex of Respondent (M/F/O):
Title of Award:
Dates of Award:
Country of Residence: Country of Project:
Time Started: _ 
RTAC Mid-term Performance Evaluation
1
I. RTAC NETWORK 
For this first section, we would like to focus on the RTAC network itself (as opposed to your grant activities).
1. How did you first become involved in the RTAC network?
2. How would you describe the range of research opportunities promoted through the RTACnetwork?
3. In what network activities, if any, have you been engaged? [Probe on expert panels, promotion of their
research findings, collaboration with other researchers, etc.] 
4. How much time have you dedicated to RTAC network activities? Was that time well used, in your opinion?
5. From your perspective, what are the benefits, if any, of belonging to the RTACnetwork?
6. From your perspective, what are the drawbacks, if any, of belonging to the RTACnetwork?
7. Are there any barriers to participation in the RTAC network that you could identify? 
8. Is the structure of the RTAC network effective in your opinion? How could it beimproved?
9. Is the management of the RTAC network effective in your opinion? How could it be improved?
10. How would you rate your overall satisfaction with the RTAC network on a scale of 1 to 5, five being highly
satisfied and 1 being highly dissatisfied? 
11. How would you rate your level of engagement with the RTAC network: 1. Not engaged, 2. Somewhat
Engaged, 3. Highly Engaged.
II. MEETING USAID NEEDS THROUGH RESEARCH
For the next section, we would like to focus on your award under RTAC and its (expected) outcomes.
12. Would you please briefly share with us the focus of your research study and the identified information
and/or training needs that your work would help address?
13. To what extent were the information and/or training needs clearly identified by USAID in the solicitation to
which you responded?
14. Was there any evolution of those identified needs between the time of the solicitation and the time of
award [(where relevant) or the time that your research was completed]?
15. Have you been able to work with the original schedule proposed for your grant activities, or have there
been changes to the schedule? If there were changes, were they initiated on your end or USAID’s end?
16. Can you briefly tell us about your research to action plan for disseminating the evidence generated by your
research? To what extent were USAID and other partners involved in the development of the R2A plan?
How will you evaluate the impact of the R2A plan, if at all?
III. FINAL THOUGHTS 
17. What have I not asked you about your RTAC experience that I should have asked?
18. Based on your experience thus far, is a mechanism like RTAC an effective way for USAID to engage
partners in research to meet information and/or training needs? Are there other approaches you think 
would be equally or more effective?
If you think of anything that might further improve RTAC that we did not have achance 
to discuss, please call or email them to our attention. 
RTAC Mid-term Performance Evaluation
1
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Interviewer(s): 
Location:
Interviewer Comments:
RTAC Mid-term Performance Evaluation
1
RTAC MIDTERM EVALUATION – RTAC PARTNERS & NETWORK MEMBERS
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Development Lab to
conduct a midterm evaluation of the RTAC program to assess the effectiveness of program interventions to date and
strengthening future program interventions. As a member of the network, your observations will be very helpful in
these areas. The evaluation will be used to make informed decisions about mid-course corrections and future
program efforts.
The information you share with us will be treated as confidential. The transcript and notes from this interview will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, RTAC partner or RTAC network member). You
may choose not to answer any question, and you may choose to end the interview at any time. Do you have any
questions for me about the evaluation, its purpose or use, or how we will handle the information you share with us
today? [Answer any questions that are asked.] Do you consent to participate in this evaluation? [If no, thank the
respondent and end the interview. If yes, continue with script.] 
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Please Complete Prior to Interview
First Name: Last Name:
Institution: Position:
Date of interview: Sex of Respondent (M/F/O):
Country of Residence:
Time Started: _ 
RTAC Mid-term Performance Evaluation
1
I. RTAC NETWORK 
For this first section, we would like to focus on the RTAC network itself. 
1. How did you first become involved in the RTAC network?
2. How would you describe the range of research opportunities promoted through the RTACnetwork?
3. In what network activities, if any, have you been engaged? [Probe on expert panels, promotion of their
research findings, collaboration with other researchers, etc.] 
4. How much time have you dedicated to RTAC network activities? Was that time well used, in your opinion?
5. From your perspective, what are the benefits, if any, of belonging to the RTACnetwork?
6. From your perspective, what are the drawbacks, if any, of belonging to the RTACnetwork?
7. Are there any barriers to participation in the RTAC network that you could identify? 
8. Is the structure of the RTAC network effective in your opinion? How could it beimproved?
9. Is the management of the RTAC network effective in your opinion? How could it be improved?
10. How would you rate your overall satisfaction with the RTAC network on a scale of 1 to 5, five being highly
satisfied and 1 being highly dissatisfied? 
II. MEETING USAID NEEDS THROUGH RESEARCH
[Partners only] For the next section, we would like to focus on your award under RTAC and its (expected) outcomes.
11. Would you please briefly share with us what research and training activities you have been involvedin?
12. To what extent were the information and/or training needs clearly identified by USAID in the solicitation(s) 
to which you responded?
13. Was there any evolution of those identified needs between the time of the solicitation(s) and the time of
award(s) [(where relevant) or the time that your research was completed]?
14. Have you been able to work with the original schedule proposed for your activities, or have there been
changes to the schedule? If there were changes, were they initiated on your end or USAID’s end?
15. Can you briefly tell us about any research to action plan(s) for disseminating the evidence generated by
research? To what extent were USAID and other partners involved in the development of the R2A plan?
How will you evaluate the impact of the R2A plan, if at all?
III. FINAL THOUGHTS 
16. What have I not asked you about your RTAC experience that I should have asked?
17. Based on your experience thus far, is a mechanism like RTAC an effective way for USAID to engage
partners in research to meet information and/or training needs? Are there other approaches you think
would be equally or more effective?
If you think of anything that might further improve RTAC that we did not have achance
to discuss, please call or email them to our attention. 
THANK YOU FOR YOUR TIME AND COOPERATION!
RTAC Mid-term Performance Evaluation
1
Time Completed: 
Interviewer(s): 
Location:
Interviewer Comments:
RTAC Mid-term Performance Evaluation
1
RTAC MIDTERM EVALUATION – RTAC COOPERATING COUNTRY PARTNERS
IBTCI, an independent third-party evaluation firm, has been contracted by the USAID Global Lab to conduct a
midterm evaluation of the Research and Technical Assistance Center (RTAC) program, which undertook the [insert
project name] in this country. The goal of the evaluation is to assess the effectiveness of program interventions to
date and strengthening future program interventions. As someone familiar with the impacts of their work, your
observations will be very helpful in these areas. The evaluation will be used to make informed decisions about mid￾course corrections and future program efforts.
The information you share with us will be treated as confidential. The transcript and notes from this interview will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, cooperating country partner). You may choose
not to answer any question, and you may choose to end the interview at any time. Do you have any questions for
me about the evaluation, its purpose or use, or how we will handle the information you share with us today?
[Answer any questions that are asked.] Do you consent to participate in this evaluation? [If no, thank the
respondent and end the interview. If yes, continue with script.] 
We greatly appreciate your cooperation and time. Thank you for answering each question to the best of your
ability, and please don’t hesitate to ask for clarification if a question is unclear to you.
Please Complete Prior to Interview
First Name: Last Name:
Institution: Position:
Date of interview: Sex of Respondent (M/F/O):
Country:
Project:
Time Started: _ 
RTAC Mid-term Performance Evaluation
1
I. ACHIEVING DEVELOPMENT GOALS THROUGH RESEARCH
For this section, we would like to focus on the [insert project name] that was conducted under RTAC and its
(expected) outcomes.
1. We would like to start by understanding a little more about theproject.
a. [Research partner organizations only] Would you please briefly share with us what research and
training activities you have been involved in? [Probe on involvement in the research design,data
collection, data analysis, and dissemination.] 
b. [Other key informants] Would you please tell us what you know, if anything, aboutthe research
conducted under this project?
2. How was the research conducted relevant to challenges in the local community or national development
challenges? 
3. Was the evidence generated by the research available in a timely manner?
4. How has information about the research findings been disseminated? [Probe on specific communications
strategies and products for various audiences.] 
5. Have the modes of dissemination been effective? [Probe on why it was effective orineffective.] 
6. Can you share any examples of how the research findings have been used in policy making or decision￾making?
7. [Research partner organizations only] How are you assessing the impact of the research to action plan, if at
all? 
8. Do you think the [insert project name] has had an impact on the local community? If so, what has that
been?
II. FINAL THOUGHTS 
9. What recommendations would you make, if any, to the funders of the [insert project name] about ways
to improve the effectiveness of similar efforts in the future? 
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Interviewer(s): 
Location:
Interviewer Comments:
RTAC Mid-term Performance Evaluation
1
RTAC MIDTERM EVALUATION – RTAC USERS AND POTENTIAL USERS (USAID)
Hello, everyone, and thank you for agreeing to participate in today’s discussion group. My name is [state name],
and I will be leading today’s discussion. I am working on this project for IBTCI, an independent third-party
evaluation firm that has been contracted by the USAID Global Development Lab to conduct a midterm evaluation of
the RTAC program to assess the effectiveness of program interventions to date and strengthening future program
interventions. As an actual or potential user of the RTAC mechanism, you have unique insights into its strategic
value and utility for your office’s portfolio (and perhaps those of the other operating units you support). The
evaluation will be used to make informed decisions about mid-course corrections and future program efforts.
This focus group discussion will last approximately one hour. There are no right or wrong answers to the questions I
am going ask you. We want to capture the full range of opinions that you may have, so please feel free to disagree.
However, please speak one person at a time, because we are recording the focus group session. Recording the focus
group will allow me to concentrate on your responses and ensure that I do not miss anyone’s input during the
discussion. If at any time, anyone feels that they would prefer to speak with the recorder off, please let me know
and I will stop the recorder.
The information you share with us will be treated as confidential. The transcript and notes from this discussion will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, USAID staff). You may choose not to answer any
question. Do you have any questions for me about the evaluation, its purpose or use, or how we will handle the
information you share with us today? [Answer any questions that are asked.] Do you consent to participate in this
evaluation? [If no, thank the respondent and ask him/her to excuse him/herself from the room. If yes, continue with
script.] Is everyone comfortable with me recording the discussion for my notes as I mentioned? [If yes, turn on the
recorder and continue. If no, do not record the FGD.] 
FGD Screener to be Completed for Each Respondent Prior to FGD
First Name: Last Name:
M/B/IO: Position:
Sex of Respondent (M/F/O):
Time Started: _ 
I. RTAC MECHANISM
We greatly appreciate your cooperation and time. To start us off, would you each please share your name, your
office, and how you first learned about the RTAC mechanism?
1. How do you usually find out about mechanisms when you have a need for a new service or product?
RTAC Mid-term Performance Evaluation
1
2. What is the value of the RTAC network for USAID from yourperspective?
3. How do you see research or academic expertise supporting the activities of your office or operating unit?
[Probe on how it has in the past and how it might, if differently, in the future.] 
4. As you considered using RTAC, were/are there other mechanisms that you also considered? If so, what
were/are they? What features or services did they offer that RTC did not? Do you know how they compare
in terms of cost and value for service received? 
5. From what you know of how RTAC operates, is there anything you would change to make it more
responsive to USAID’s programmatic needs? [Probe on structure, management, range of partners involved,
partner characteristics, etc.] 
II. MEETING USAID NEEDS THROUGH RESEARCH AND TRAINING
For those of you who have used the RTAC mechanism, we would like to ask a bit more about that experience.
6. How would you describe the support provided in the formulation of service requests? [Probe on whether or
not the template was used and/or co-creation of the scope.] What additional support would have been
useful to you?
7. To what extent have project activities been executed in a timely manner? [Probe on any delays at the
scoping, contracting, design, research approvals, data collection, analysis, or utilization stages and whether
those delays were on the side of USAID or researcher/institution.]
8. To what extent has the evidence generated and/or technical assistance provided addressed theneeds
identified by USAID?
9. How has the evidence/findings been utilized? [Probe on any R2A plan, and whether or not USAID was
engaged in its co-creation.] Have you seen any changes as a result of the use of the evidence/findings?
[Probe on concrete examples of use of evidence in decision-making and programming.]
III. FINAL THOUGHTS 
10. Overall, how would you describe the performance of the RTAC mechanism based on yourexperience?
11. What have I not asked you about your RTAC experience that I should have asked?
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention. 
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Moderator(s): 
RTAC Mid-term Performance Evaluation
1
Location:
Moderator Comments:
RTAC Mid-term Performance Evaluation
1
RTAC MIDTERM EVALUATION – RTAC PARTNERS & NETWORK MEMBERS
Hello, everyone, and thank you for agreeing to participate in today’s discussion group. My name is [state name],
and I will be leading today’s discussion. I am working on this project for IBTCI, an independent third-party
evaluation firm that has been contracted by the USAID Global Development Lab to conduct a midterm evaluation of
the RTAC program to assess the effectiveness of program interventions to date and strengthening future program
interventions. As a member of the network, your observations will be very helpful in these areas. The evaluation
will be used to make informed decisions about mid-course corrections and future programefforts.
This focus group discussion will last approximately one hour. There are no right or wrong answers to the questions I
am going ask you. We want to capture the full range of opinions that you may have, so please feel free to disagree.
However, please speak one person at a time, because we are recording the focus group session. Recording the focus
group will allow me to concentrate on your responses and ensure that I do not miss anyone’s input during the
discussion. If at any time, anyone feels that they would prefer to speak with the recorder off, please let me know
and I will stop the recorder.
The information you share with us will be treated as confidential. The transcript and notes from this discussion will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, RTAC partner or RTAC network member). You
may choose not to answer any question. Do you have any questions for me about the evaluation, its purpose or use,
or how we will handle the information you share with us today? [Answer any questions that are asked.] Do you
consent to participate in this evaluation? [If no, thank the respondent and ask him/her to excuse him/herself from
the room. If yes, continue with script.] Is everyone comfortable with me recording the discussion for my notes as I
mentioned? [If yes, turn on the recorder and continue. If no, do not record the FGD.] 
FGD Screener to be Completed for Each Respondent Prior to FGD
First Name: Last Name:
Institution: Position:
Department or Discipline: Sex of Respondent (M/F/O):
Country of Residence:
Time Started: _ 
RTAC Mid-term Performance Evaluation
1
We greatly appreciate your cooperation and time. To start us off, would you each please share your name, your
organization, and how you first learned about the RTAC network?
1. What were your expectations when you joined the RTAC network? To what extent have those expectations
been met?
2. In what network activities, if any, have you been engaged? [Probe on research, expert panels, promotion of
their research findings, collaboration with other researchers, etc.]
3. How much time have you dedicated to RTAC network activities? Was that time well used, in your opinion?
4. From your perspective, what are the benefits, if any, of belonging to the RTACnetwork?
5. From your perspective, what are the drawbacks, if any, of belonging to the RTACnetwork?
6. Are there any barriers to participation in the RTAC network that you could identify?
7. What changes would you make to the RTAC network, if any? [Probe on structure, management, availability 
of research opportunities, availability of other consulting opportunities, etc.] Why and how would those
changes improve the outcomes?
If you think of anything that might further improve RTAC that we did not have a chance
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Moderator(s): 
Location:
Moderator Comments:
RTAC Mid-term Performance Evaluation
1
RTAC MIDTERM EVALUATION – RTAC COOPERATING COUNTRY RESPONDENTS
Hello, everyone, and thank you for agreeing to participate in today’s discussion group. My name is [state name],
and I will be leading today’s discussion. I am working on this project for IBTCI, an independent third-party
evaluation firm that has been contracted by the USAID Global Development Lab to conduct a midterm evaluation of
the RTAC program to assess the effectiveness of program interventions to date and strengthening future program
interventions. As a member of the network, your observations will be very helpful in these areas. The evaluation
will be used to make informed decisions about mid-course corrections and future programefforts.
This focus group discussion will last approximately one hour. There are no right or wrong answers to the questions I
am going ask you. We want to capture the full range of opinions that you may have, so please feel free to disagree.
However, please speak one person at a time, because we are recording the focus group session. Recording the focus
group will allow me to concentrate on your responses and ensure that I do not miss anyone’s input during the
discussion. If at any time, anyone feels that they would prefer to speak with the recorder off, please let me know
and I will stop the recorder.
The information you share with us will be treated as confidential. The transcript and notes from this discussion will
not be shared with any other organization. Reported information will be aggregated, and statements made will not
be attributed at an individual level, but at a role level (for example, cooperating country respondent). You may
choose not to answer any question. Do you have any questions for me about the evaluation, its purpose or use, or
how we will handle the information you share with us today? [Answer any questions that are asked.] Do you
consent to participate in this evaluation? [If no, thank the respondent and ask him/her to excuse him/herself from
the room. If yes, continue with script.] Is everyone comfortable with me recording the discussion for my notes as I
mentioned? [If yes, turn on the recorder and continue. If no, do not record the FGD.] 
FGD Screener to be Completed for Each Respondent Prior to FGD
First Name: Last Name:
Institution: Position:
Country: Sex of Respondent (M/F/O):
Project:
Time Started: _ 
RTAC Mid-term Performance Evaluation
1
I. ACHIEVING DEVELOPMENT GOALS THROUGH RESEARCH
Would you please start by telling us your name, the institution that you work for, and how you first learned about
the [insert project name].
1. Based on what you know about the project, how was the research conducted relevant to challenges in the
local community or national development challenges? 
2. Was the research focused on a critical problem for this community or nation? For whom was this problem
urgent?
3. How was information about the research findings disseminated? [Probe on specific communications
strategies and products for various audiences.] 
4. Have the modes of dissemination been effective? [Probe on why it was effective orineffective.] 
5. In your opinion, would there have been more effective ways to disseminate information in the community
or at the national level? If so, what would they be? 
6. Do you know if the research findings have been used to inform policy making or decision-making? Can you
share any examples? [Probe on use within their organizations and within the community at large.] 
7. Do you think the [insert project name] has had an impact on the local community? If so, what has that
been?
II. FINAL THOUGHTS 
8. What recommendations would you make, if any, to the funders of the [insert project name] about ways
to improve the effectiveness of similar efforts in the future?
If you think of anything that might further improve RTAC that we did not have a chance 
to discuss, please call or email them to our attention.
THANK YOU FOR YOUR TIME AND COOPERATION!
Time Completed: 
Moderator(s): 
Location:
Moderator Comments:
RTAC Mid-term Performance Evaluation
1
RTAC Mid-term Evaluation Work Plan
INTRODUCTION
This work plan outlines the IBTCI team’s activities throughout the Research Technical Assistance Center Midterm
Performance Evaluation (RTAC MPE). The work is divided into five discrete phases: i) preparation and design; ii)
preliminary data collection and field testing; iii) data collection; iv) data analysis; and v) report writing. Primary
responsibility for all evaluation activities will lie with consultants, Christine Allison and Kathylynn Pierre Griff, with
support at key periods from IBTCI staff, Betsy Nolan, Susan Morawetz, and Kathryn Hornbeck. 
Over the life of the evaluation, the RTAC mid-term evaluation team will have monthly calls with USAID to review
progress, to identify any risks to adhering to the timeline, and to discuss lessons learned to date. In particular, the
team anticipates that these calls will represent key learning opportunities, particularly in January and May 2020,
which will complement the other learning activities built into the work plan below (including the inception
presentation in March 2020 and the recommendations development workshop in August 2020). Specific dates for
the monthly calls (set in general for the second Wednesday of the month at 2 pm) are included in the Timeline
below. 
EVALUATION TEAM ROLES AND RESPONSIBILITIES
The IBTCI team comprises five members, three of whom will be primarily responsible for the technical aspects of the
evaluation and two of whom will be responsible for oversight and the administrative aspects of the evaluation. 
Susan Morawetz is the Project Director for IBTCI, and she will provide overall guidance on the project, as well as
serving as the corporate monitor for all activities. Christine Allison will serve as the Team Leader, and she will be
responsible for the day to day management of evaluation activities. She will work with other team members to
design the evaluation, to collect and analyze the data, to distill findings, and to lead the recommendations workshop
and final drafting of the report. Kathylynn Griff is the Evaluation Methodologist, and she will work hand-in-hand
with the team leader to design the evaluation and the instruments, to collect and analyze the data, to distill findings,
and to author the report. Betsy Nolan will also serve as an Evaluation Methodologist, and she will work with the
team to collect, manage, and analyze the data. Kathryn Hornbeck will provide assistance on contractual and
administrative tasks as needed.
PHASE 1: PREPARATION AND DESIGN
The preparation and design work began at contract award (September 2019) and will be completed when USAID
approves the design and initial testing of the instruments (anticipated February 2020). In addition to introductory
meetings between the evaluation team and USAID and with NORC RTAC, the preparation and design phase includes 
two main activities: document review and evaluation design.
Document review (October 18, 2019 – November 15, 2019) 
In order to fully understand the program and develop appropriate and well-targeted data collection instruments,
the team will review program documents, including the initial RFP and proposal, work plan, MEL plan, semiannual
reports, and year 1 deliverables. Drawing on the evaluation questions, proposed sub-questions (as approved by
USAID), and the documents reviewed, the team will begin designing the data collection instruments. It should be
noted, however, that the team will review additional documents as they become available over the course of the
evaluation. 
Evaluation design (November 18, 2019 – December 13, 2019) 
The overarching evaluation design that IBTCI proposed and was accepted by USAID was a mixed- methods,
utilization-focused evaluation that can be used to inform program adaptation and learning. Per the proposal, the
team will incorporate the following primary data collection approaches: (1) key informant interviews (KIIs) with 
RTAC Mid-term Performance Evaluation
1
program managers (USAID and NORC personnel); (2) KIIs with USAID staff and key stakeholders; (3) KIIs with RTAC
researchers (n=7); (4) focus group discussions (FGDs) with RTAC providers; (5) FGDs with RTAC clients and
beneficiaries; (6) FGDs with host country counterparts; (7) a self-administered survey of network participants
(n=254); and (8) a self-administered survey of USAID staff and key stakeholders (n=?).
To the extent possible, USAID and RTAC will help the team identify the universe of possible respondents for each of
the proposed data collection methods. This will follow the kick-off meetings, from which a purposive sample of
respondents will be drawn to provide the evaluation team with a broad range of experiences on which to draw. The
kick-off meetings and further consultation with USAID and RTAC will guide the sampling strategy for the KIIs and
FGDs. The team expects to conduct a census with each survey respondent group. For each target group, a contact
and communication protocol strategy will be developed and described in the evaluation design. The team will then
proceed to develop draft data collection instruments for each target audience. The team will also develop an initial
coding scheme for the qualitative data and plan the syntax for the surveys to ensure that all data elements needed
to answer the evaluation questions are being collected and structured appropriately. After submission to USAID for
review and receipt of comments and edits (expected mid-January 2020), the team will finalize the evaluation design
and data collection instruments, to be completed no later than February 7, 2020.
The team anticipates that all surveys will be self-administered online. In order to get the most informative
responses, however, the surveys will be field tested during the first site visit to Kenya, scheduled for March 2020,
and then resubmitted to USAID to review any changes to response categories, instructions, or other survey
elements. The approved, revised version will then be programmed into an online survey platform and prepared for
dissemination.
Concurrent with the submission of the final evaluation design and data collection instruments, the evaluation team
will also submit a country clearance request for the first site visit to be conducted in early March.
PHASE 2: PRELIMINARY DATA COLLECTION AND FIELD TESTING
Phase 2 will focus on initial data collection and any needed refinement of data collection instruments, as described
under each of the data collection methods below. Per the proposal, the team will conduct a total of 48 KIIs and 12
FGDs in addition to the surveys in the field testing and data collection phases, as shown in the table below. 
Data Collection
Method
Preliminary/Field
Testing Phase
Data Collection Phase TOTAL #
Planned
Estimated #
of
Respondents
US Overseas/
Kenya 
US Overseas/
Honduras
KIIs 8 6 22 6 42 42
FGDs 1 2 6 3 12 72-96
Surveys 3 TBD
In-briefing (February 21, 2020) 
RTAC Mid-term Performance Evaluation
1
Prior to the initiation of any data collection, the RTAC mid-term evaluation team will participate in an in-briefing 
with USAID. The team anticipates that the in-briefing will be conducted virtually toward the end of February 2020
after the final evaluation design has been approved by USAID.
Sampling plan and outreach to field test respondents (February 22, 2020) 
After the in-briefing has been completed, the team will finalize the sampling plan and begin reaching out to the
initial sample of respondents for KIIs and FGDs with whom the initial interviews and field testing will be completed
(see below for a description of the target groups for field testing for each data collection method).
Initial KIIs (March 2-6, 2020) 
The team will conduct an initial set of eight interviews to include the program managers at USAID and NORC, as well 
as at least one network participant and one USAID stakeholder (not a program manager). In addition to collecting
initial data, this first round of interviews will be used to fine tune all three interview protocols, allowing the team to
identify additional opportunities for probing or alternative ways of framing some questions that might make them
easier for respondents to answer and/or facilitate a higher participant response rate. To ensure that the qualitative
data coding is done rigorously and consistently, two of the initial KIIs will be coded by all team members and then
the results will be compared and worked through to ensure that all of the concepts included in the coding structure
are understood and applied consistently. 
FGD field test (March 2-6, 2020) 
Similar to the plan for KIIs, the team will conduct one FGD with Washington, DC-based USAID stakeholders to fine
tune the FGD protocol. The team will also use this opportunity to field test the survey to be used with USAID
stakeholders and to make any edits that will improve the quality of the data. The FGD field test notes will also be
coded by all team members, similar to the KIIs, using the same process of comparing and working through the
results to ensure that all of the concepts included in the coding structure are understood and applied consistently,
as the team proceeds with data collection.
Kenya site visit (o/a March 9-13, 2020 – final dates to be determined with USAID and
partners) 
The team will travel to Kenya in the first half of March, and after an initial meeting with USAID, the team will
conduct approximately 6 KIIs and 2 FGDs with key stakeholders in Nairobi. The team also intends to field test the
survey of network participants and the survey of RTAC researchers that will be administered online. At the
completion of the visit, the team will out brief with the Mission prior to departure.
Finalization of survey instruments (March 16, 2020) 
At the completion of all of the initial data collection, the team will revise the three survey instruments based on the
results of the field tests. These will then be resubmitted to USAID for review and approval prior to being
programmed into an online platform. At the same time, the syntax drafted as part of the evaluation plan will be
updated to ensure that it is ready for use when data analysis begins.
Inception presentation (March 23-27, 2020) 
At the completion of the design and field testing stages, the team will develop the inception presentation based on
the results of the document review and the preliminary data collected through field testing and initial interviews.
This represents a key learning opportunity, and the team expects that the presentation will be structured as a
learning event that may involve USAID and the RTAC team.
PHASE 3: DATA COLLECTION
The team will then move on to the remainder of the data collection tasks with the final instruments.
RTAC Mid-term Performance Evaluation
1
Domestic KIIs (April 2020) 
The team will conduct an additional 20 KIIs with US-based key informants identified in the sampling plan. The KIIs
will be written up and coded using the draft coding structure as they are completed, and any revisions/additions to
the coding structure will be shared with all team members to ensure consistency in its use.
Domestic FGDs (April 2020) 
The team will also conduct 4 FGDs with US-based stakeholders identified in the sampling plan. The FGDs will be
written up and coded using the draft coding structure as they are completed, and any revisions/additions to the
coding structure will be shared with all team members to ensure consistency in itsuse. 
Survey administration (April 15-May 15, 2020) 
Once the final survey instruments have been approved by USAID, the surveys will be programmed into an online
survey platform and disseminated by email to the potential respondent pool. We anticipate conducting a census of
all potential respondents, recognizing that self-selection bias will influence the final response rate for each
respondent group. Overall, we expect to have response rates of 5-15%, and the team will work with USAID and
RTAC to identify the most effective outreach strategies to encourage participation (to be detailed in the evaluation
design).
Honduras site visit (April 20-24, 2020) 
The team will travel to Honduras on April 20, and after an initial meeting with USAID, the team will conduct
approximately 6 KIIs and 3 FGDs with key stakeholders in Tegucigalpa. At the completion of the visit, the team will
outbrief with the Mission prior to departure.
Re-interviews (June 2020-August 2020) 
As this evaluation is adopting a CLA approach, the team anticipates that the RTAC program managers at USAID and
NORC may make adaptations to the program over the life of the evaluation. The team therefore plans to conduct
up to 2 FGDs and 8 KIIs with key stakeholders, including program managers, at the very end of the data collection
phase to capture any changes made to the RTAC program made in response to learning activities and after the initial
interviews conducted earlier in the evaluation. 
PHASE 4: DATA ANALYSIS
During Phase 4, the team will conduct data analysis and develop preliminary findings, lessons learned, and
recommendations for USAID, NORC, and network members.
Data analysis and submission of data sets to DDL (June 2020) 
Upon completion of the data collection, the team will analyze the cleaned survey data and triangulate the survey
data with the coded KII and FGD data. The team will then draft the preliminary findings of the evaluation. These
findings will be used as the basis for the Recommendations Development Workshop. The final, cleaned, de￾identified survey data will be submitted to DDL, to be embargoed until the final report is approved.
Recommendations Development Workshop planning (June-July 2020) 
The team will work with LAB/CDR and RTAC to determine the best dates and location for the workshop, as well as
the list of people to be invited to participate. The team will develop an agenda and identify any AV support requests
it may have and will provide those to USAID and RTAC as early as possible. 
Recommendations Development Workshop (August 2020) 
Working with the LAB/CDR and RTAC teams, the evaluation team will conduct a half-day workshop at a USAID or
RTAC venue in Washington, DC to review the results of the data analysis and develop recommendations to
strengthen the RTAC program. As no resources in the budget are allocated for direct workshop costs other than
RTAC Mid-term Performance Evaluation
1
labor, the IBTCI team will plan content and facilitate the workshop, but will not be responsible for securing the
venue or providing refreshments. The team anticipates that the results of the analysis will be presented, and the
group will identify priority areas for adapting the current program. Depending on the number of participants, the
group may work in plenary or divide into smaller groups to develop key recommendations for concrete, specific
program adaptations. The results of the workshop will be recorded, and the identified recommendations will be
incorporated into the final report.
PHASE 5: REPORT WRITING
During the final phase of the project, the team will draft the report and recommendations and respond to any
USAID comments in the final version.
Draft report (September 2020) 
The team will develop a draft report that adheres to the outline included in the RFP/award to ensure that all key
evaluation questions have been clearly answered and that the evaluation process has been duly documented. The
team expects to submit the report to USAID in early October 2020 for initial review. 
USAID review and edits (October – December 2020) 
The evaluation team expects that the LAB/CDR team will provide a first round of edits on the draft report, and that
the evaluation team will revise the report to address those comments (including the submission of a table that
tracks each comment and how the team addressed it). The team then anticipates submitting a second draft report
in November 2020 for broader stakeholder review and input, and will repeat the same process of presenting a
revisions and comment tracking table for the second round of review.
Final report (January 2021) 
Having addressed all USAID comments, the team will submit the final report in January 2021.
Submission of final report to DEC (February 2021) 
Having received final approval of the final report, the team will submit the approved public version of the final
report to the DEC.
TIMELINE
The table below shows the consolidated timeline for the RTAC mid-term evaluation. 
Task Dates
Phase 1
Background reading (document summaries) Oct 18-Nov 15, 2019
Work plan submission November 8, 2019
Monthly call November 13, 2019
Kick-off meeting with RTAC November 18-22, 2019
Monthly call December 11, 2019
Draft Evaluation Design December 13, 2019
USAID Review of Evaluation Design December 16, 2019 –
January 31, 2020
RTAC Mid-term Performance Evaluation
1
Monthly call – CLA discussion of lessons learned through document review January 8, 2020
Final Evaluation Design / Submission of travel request 1 February 7, 2020
Monthly call February 12, 2020
Phase 2
In briefing February 21, 2020
Initial contacts to KII sample, FGD sample February 22, 2020
Initial KIIs March 2-6, 2020
Field test FGD protocol March 2-6, 2020
Kenya field visit March 9-13, 2020
Revise final survey instruments March 16, 2020
Monthly call (postponed one week due to field work) March 17, 2020
USAID review of final survey instruments March 17-31, 2020
Inception Presentation / Submission of travel request 2 March 23-27, 2020
Phase 3
Domestic KIIs April 6-17, 2020
Domestic FGDs April 6-17, 2020
Monthly call April 8, 2020
Surveys programming, testing and launch April 13-15, 2020
Honduras field visit April 20-24, 2020
Monthly call – CLA discussion of qualitative data lessons learned May 13, 2020
Surveys close May 15, 2020
Clean survey data May 18-20, 2020
Re-interviews (domestic KIIs & FGDs) June 1-5, 2020
Phase 4
Data Analysis June 8-19, 2020
Monthly call June 10, 2020
Submit survey data sets to DDL (to be embargoed until final report is
approved)
June 29, 2020
Recommendation Development Workshop planning June-July 2020
Monthly call July 8, 2020
RTAC Mid-term Performance Evaluation
1
Recommendation Development Workshop August 2020
Monthly call August 12, 2020
Phase 5
Draft report September 2020
Monthly call September 9, 2020
USAID Review October-December 2020
Monthly call October 14, 2020
Monthly call* (postponed 1 day due to Veteran’s Day holiday) November 12, 2020
Monthly call December 9, 2020
Final Report January 2021
Monthly call January 12, 2021
Submission of final report to DEC February 2021
RTAC Mid-term Performance Evaluation
1
ANNEX D: LIST OF EVALUATION PARTICIPANTS
Organization Evaluation Participants
NORC Gabriela Alcaraz 
Michelle Davis
Varuni Dayaratna
Sutherland Miller
Alain Otarola
USAID Jessica Anderson
Cameron Bess
Chris Capacci-Carneal
Sara Carlson 
Charles Davis
Karen Duca
Luba Fajfer
Karen Fowle
Kyla Gregoire
Ashley Heiber
Tara Hill
Michelle Jones
Tyrell Kahan
Barbara Knox-Seith
Amadou Maiga
Hannah Marqusee
Megan Miller
John Mullenax
Amanda Nataro
Paul Nelson
Anita Oberai
Kristina Ortiz
Lisa Patel
Meron Paulos
RTAC Mid-term Performance Evaluation
1
Nancy Peek
Kyle Rearick
Rebecca Semmes
Lily Sweikert
Amadou Traore
Kelly Tucker
Elena Walls
Brent Wells
Shavonn Witten 
U.S. Department of State Ketura Brown
Arizona State University Ryan Johnson
David Schlinkert
Davis Management Group Sylin Bynoe
Beth Copes
Duke University Catherine Andrzejewski
Edmund Malesky
Erik Wibbels
Forum One Mike Shoag
Institute of International Education Lindsay Hillenberg
Pulte Institute for Global Development
University of Notre Dame
Kevin Fink
Population Reference Bureau Smita Gaith
Robin Johnson*
Reshma Naik
Carolyn Rodehau*
Marissa Yeakey*
Resilient Africa Network (RAN) at Makerere
University (Uganda)
Wilson Abigaba
Roy William Mayega
Christine Muhumuza
Dorothy Okello
Patrick Rubangakene
Anthony Ssebagereka
RTAC Mid-term Performance Evaluation
1
Julius Ssentongo
RTAC Mid-term Performance Evaluation
1
Nathan Tumuhamye
Columbia University Leon Perlman
Dairy Science Park Peshawar (Pakistan) M Subhan Qureshi
David Livingstone College of Education
(Zambia)
Amabile Manianga
Innovations for Poverty Action Rebecca Rouse
Institute of Agricultural Research for
Development (Cameroon) 
Njukeng Nkengafac 
Institute of Health Research, UHAS (Ghana) Desmond Klu
Institute of Social Work (Tanzania) Christina Fille
Instituto Nacional de Salud Pública (México) Caroline Irene Deschak
International University of Rabat (Morocco) Ghita Mezzour 
Lehigh University Whitney Szmodis
Makerere University (Uganda) Diana Ntamu
Andrew Tamale
Maryam Abacha American University (Niger) Abdulsalam Abdullahi
Research Center for Environmental Monitoring
and Modeling (CEMM), VNU University of
Science (Vietnam)
Ngo Tho Hung
Nguyen The Hung
Tanta University (Egypt) Yasmein Ali
Texas A&M University Edwin Price
Serges K. Vutegha
Uganda Management Institute Godfrey Bwanika
United StatesInternational University Susan Ndungu
Universidad Centroamericana José Simeón
Cañas (El Salvador)
Pauline Martin 
UniversitasIndonesia Asri Adisasmita
Hadi Pratomo 
University of Dar es Salaam (Tanzania) Marcelina K.Biro
RTAC Mid-term Performance Evaluation
1
University for Development Studies(Ghana) Gordana Kranjac-Berisavljevic
University of Alaska John Duffy 
University of Mississippi Jamiko Deleveaux
RTAC Mid-term Performance Evaluation
1
University of Notre Dame Rahul Oka
University of Sciences, Techniques, and
Technology of Bamako (Mali)
Moctar Tounkara
RTAC Mid-term Performance Evaluation
1
University of Sussex (UK but based in
Kathmandu, Nepal)
Sharada Pd Wasti
University of the West Indies, MONA (Jamaica) Sheena Francis
University of Wisconsin Gina Gallego
Zimbabwe Ezekiel Guti University Dominica Chingarande
* Denotes individual was no longer with the organization listed at the time of data collection
 
RTAC Mid-term Performance Evaluation
1
 ANNEX E: COI FORMS
Disclosure of Real or Potential Conflict of Interest for USAID Evaluations
Instructions: 
Evaluations of USAID projects will beundertaken sothattheyare notsubjectto theperception orreality of
biasedmeasurementorreporting due to conflictofinterest.1 Forexternalevaluations, allevaluation team
members will provide a signedstatementattesting to a lack of conflict of interestordescribing an existing
conflict of interest relative to the project being evaluated.*
Evaluators of USAID projects have a responsibility to maintain independence so that opinions, conclusions, judgments, and recommendations will be impartial and will be viewed as impartial by third parties. Evaluators and evaluation teammembers are to disclose allrelevant facts regarding real or potential conflicts of interest that could leadreasonable thirdpartieswithknowledge of the relevant facts and circumstances to conclude that the evaluator 
or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Operating Unit leadership, in close consultation with the Contracting Officer, will determine whether the real or potential conflict of interest is one that should disqualify an individual from the evaluation team or require recusal by that individual from evaluating certain aspects of the project(s).
In addition, if evaluation team members gain access to proprietary information of other companies in the process of conducting the evaluation, then they must agree with the other companies to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain fromusing the information for any purpose other than that for which it was furnished.
Real orpotential conflicts ofinterest may include, but arenot limitedto:
RTAC Mid-term Performance Evaluation
1
1. Immediate family or close family member who is an employee of theUSAID operating unit
managing the project(s)beingevaluated or theimplementingorganization(s)whoseproject(s)
arebeingevaluated.
2. Financial interest that is direct, or is significant/material though indirect, in the
implementing organization(s)whoseprojects arebeing evaluated orin theoutcome
of theevaluation.
3. Current or previous direct or significant/material though indirect experience with the
project(s) being evaluated, including involvementin the projectdesign orprevious iterations
of the project.
4. Current or previous work experience or seeking employment with the USAID operating unit
managing the evaluation or the implementing organization(s)whose project(s) are being
evaluated.
5. Current or previous work experience with an organization thatmay beseen as an industry 
competitor with the implementing organization(s) whose project(s) are being evaluated.
6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular
projects and organizations being evaluated that could bias theevaluation.
USAIDEvaluationPolicy (p. 8);USAIDContract InformationBulletin99-17; andFederalAcquisitionRegulations(FAR)Part
9.5,OrganizationalConflicts ofInterest, andSubpart 3.10,Contractor CodeofBusinessEthics andConduct.
USAIDEvaluationPolicy(p.
11) FAR 9.505-4(b)
RTAC Mid-term Performance Evaluation
1
Disclosure of Conflict of Interest for USAID Evaluation Team Members
Name Christine Allison
Title Team Leader
Organization IBTCI
Evaluation Position? Team Leader Team member
Evaluation Award Number (contract
or other instrument)
USAID Project(s) Evaluated
(Include project name(s),
implementer name(s) and award
number(s), if
applicable)
RTAC Midterm Evaluation
I have real or potential conflicts of
interest to disclose. 
Yes No
If yes answered above, I disclose
the following facts: 
Real or potential conflicts of interest mayinclude,
but are not limited to:
1. Close family member who is an employeeof
the USAID operating unit managing the
project(s) being evaluated or the
implementing organization(s) whose
project(s) are being evaluated.
2. Financial interest that is direct, or is
significant though indirect, in the
implementing organization(s) whose
projects are being evaluated or in the
outcome of theevaluation.
3. Current or previous direct or significant
though indirect experience with the
project(s) being evaluated, including
involvement in the project design orprevious
iterations of the project.
4. Current or previous work experience or
seeking employment with the USAID
operating unit managing the evaluationor
the implementing organization(s) whose
project(s) are being evaluated.
5. Current or previous work experience with
an organization that may be seen as an
industry competitor with the implementing
organization(s) whose project(s) are being
evaluated.
6. Preconceived ideas toward individuals, 
groups, organizations, or objectives of the
particular projects and organizations being
evaluatedthat 
could bias the evaluation.
I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this
disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies,
then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and
refrain from using the information for any purpose other than that for which it was furnished.
Signature
Date 17 July 2019
RTAC Mid-term Performance Evaluation
2
Disclosure of Real or Potential Conflict of Interest for USAID Evaluations
Instructions: 
Evaluations of USAID projects will beundertaken so thattheyarenotsubject to theperception orreality of
biasedmeasurement orreportingdue toconflict of interest.1 For externalevaluations, allevaluation team
members willprovide a signed statement attesting to a lack of conflict ofinterestordescribing an existing 
conflict of interest relative to the project being evaluated.*
Evaluators of USAID projects have a responsibility to maintain independence so that opinions, conclusions, judgments, and recommendations will be impartial and will be viewed as impartial by third parties. Evaluators and evaluation teammembers are to disclose allrelevantfactsregarding real or potential conflicts of interest that could leadreasonable third partieswithknowledge of the relevant facts and circumstancestoconclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Operating Unit leadership, in close consultation with the Contracting Officer, will determine whether the real or potential conflict of interest is one that should disqualify an individual from the evaluation team or require recusal by that individual from evaluating certain aspects of the project(s).
In addition, if evaluation team members gain access to proprietary information of other companies in the process of conducting the evaluation, then they must agree with the other companies to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished.
Real orpotential conflicts ofinterest may include, but arenot limitedto:
7. Immediate family or close family member who is an employee of theUSAID operating unit
managing the project(s)beingevaluated or theimplementingorganization(s)whoseproject(s)
arebeingevaluated.
8. Financial interest that is direct, or is significant/material though indirect, in the
implementing organization(s)whose projects arebeingevaluated orin the outcomeof
the evaluation.
9. Current or previous direct or significant/material though indirect experience with the project(s)
being evaluated, including involvement in the project design or previous iterations of the
project.
10. Current or previous work experience or seeking employment with the USAID operating unit
managing the evaluation or the implementing organization(s)whose project(s) are being
evaluated.
11. Current or previous work experience with an organization thatmay beseen as an industry
competitor with the implementing organization(s) whose project(s) are being evaluated.
12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular
projects and organizations being evaluated that could bias theevaluation.
RTAC Mid-term Performance Evaluation
3
USAIDEvaluationPolicy (p. 8);USAIDContract InformationBulletin99-17; andFederalAcquisitionRegulations(FAR)Part
9.5,OrganizationalConflicts ofInterest, andSubpart 3.10,Contractor CodeofBusinessEthics andConduct.
USAIDEvaluationPolicy(p.11)
FAR 9.505-4(b)
RTAC Mid-term Performance Evaluation
1
Disclosure of Conflict of Interest for USAID Evaluation Team Members
Name Elisabeth (Betsy) Nolan
Title Evaluation MethodsSpecialist
Organization IBTCI
Evaluation Position7 Team Leader X Team member
Evaluation Award Number (contract
or other instrument)
USAID Project(s) Evaluated (Include
project name(s), implementer
name(s) and award number(s), if
applicable)
RTAC
I have real or potential conflicts of
interest to disclose.
Yes XNo
Ifyes answered above,I disclose the
following facts:
fileaIorpotcntio/confIict>•ofintere•tmayinclude,
dut are not limited to:
1. Close familymember whoisan employee of
the USAID operating unit managing the
project(sJ being evaluated or the
implementing organization(s) whose
project(s) are being evaluated.
Finnnrinlintrrr.stthati.sdirect,or issignificant
though indirect, in the implementing
organization(s) whose projects are being
evaluatedor intheoutcomeoftheevaluation.
3. Currentor previausdirectorsigni/icont
though indirect experience with the
project(s} being evaluated, including
involvement in the project design or
previaus iterations of the project.
4. Current orpreviaus work experience or
seeking employment with the USAID
operating unit managingthe evaluationor
the implementing organization(sJ whose
project(s) are being evaluated.
5. Current or previous work experience with 
an organization t/ior may be seen as an
industry competitor with the
implementing organizotion(s) whose
project(s) are being evaluated.
6. Preconceivedideastowardindividuals, groups,
organizations, orobjectives of the particular
projectsand organizations beingevaluated that
could bias the evaluation.
I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this
disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies,
then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and
refrain from using the information for any purpose oper than t,hat.fpr w¿tich it was furnished.
Signature ” t/ , .
Date
RTAC Mid-term Performance Evaluation
2
Disclosure of Real or Potential Conflict of Interest for USAID Evaluations 
Instructions: 
Evaluations of USAID projects will be undertaken so that they are not subject to the perception or reality of
biased measurement or reporting due to conflict of interest.
1 For external evaluations, all evaluation team
members will provide a signed statement attesting to a lack of conflict of interest or describing an existing
conflict of interest relative to the project being evaluated.2
Evaluators of USAID projects have a responsibility to maintain independence so that opinions, conclusions, judgments, and recommendations will be impartial and will be viewed as impartial by third parties. Evaluators and evaluation team members are to disclose all relevant facts regarding real or potential conflicts of interest that could lead reasonable third parties with knowledge of the relevant facts and circumstances to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of
exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Operating Unit leadership, in close consultation with the Contracting Officer, will determine whether the real or potential conflict of interest is one that should disqualify an individual from the evaluation team or require recusal by that individual from evaluating certain aspects of the project(s).
In addition, if evaluation team members gain access to proprietary information of other companies in the process of conducting the evaluation, then they must agree with the other companies to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. 3
Real or potential conflicts of interest may include, but are not limited to:
1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 
2. Financial interest that is direct, or is significant/material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation.
3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 
5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 
6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. 
RTAC Mid-term Performance Evaluation
3
1USAID Evaluation Policy (p. 8); USAID Contract Information Bulletin 99-17; and Federal Acquisition Regulations (FAR) Part
9.5, Organizational Conflicts of Interest, and Subpart 3.10, Contractor Code of Business Ethics and Conduct.
2USAID Evaluation Policy (p. 11)
3 FAR 9.505-4(b)
RTAC Mid-term Performance Evaluation
1
Disclosure of Real or Potential Conflict of Interest for USAID Evaluations 
Instructions: 
Evaluations of USAID projects will be undertaken so that they are not subject to the perception or reality of
biased measurement or reporting due to conflict of interest.
1 For external evaluations, all evaluation team
members will provide a signed statement attesting to a lack of conflict of interest or describing an existing
conflict of interest relative to the project being evaluated.2
Evaluators of USAID projects have a responsibility to maintain independence so that opinions, conclusions, judgments, and recommendations will be impartial and will be viewed as impartial by third parties. Evaluators and evaluation team members are to disclose all relevant facts regarding real or potential conflicts of interest that could lead reasonable third parties with knowledge of the relevant facts and circumstances to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of
exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Operating Unit leadership, in close consultation with the Contracting Officer, will determine whether the real or potential conflict of interest is one that should disqualify an individual from the evaluation team or require recusal by that individual from evaluating certain aspects of the project(s).
In addition, if evaluation team members gain access to proprietary information of other companies in the process of conducting the evaluation, then they must agree with the other companies to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. 3
Real or potential conflicts of interest may include, but are not limited to:
1. Immediate family or close family member who is an employee of the USAID operating unit
managing the project(s) being evaluated or the implementing organization(s)whose project(s) are
being evaluated.
2. Financial interest that is direct, or is significant/material though indirect, in the
implementing organization(s) whose projects are being evaluated or in the outcome of
the evaluation.
3. Current or previous direct or significant/material though indirect experiencewith the project(s)
being evaluated, including involvement in the project design or previous iterations of the
project.
4. Current or previous workexperience or seeking employment with the USAIDoperating unit 
managing the evaluation or the implementing organization(s) whose project(s) are being
evaluated.
5. Current or previous work experience with an organization that may be seen as an industry 
competitor with the implementing organization(s) whose project(s) are being evaluated.
6. Preconceivedideas toward individuals, groups,organizations,or objectives of the particular
projects and organizations being evaluated that could bias theevaluation. 
RTAC Mid-term Performance Evaluation
2
1USAID Evaluation Policy (p. 8); USAID Contract Information Bulletin 99-17; and Federal Acquisition Regulations (FAR) Part
9.5, Organizational Conflicts of Interest, and Subpart 3.10, Contractor Code of Business Ethics and Conduct.
2USAID Evaluation Policy (p. 11)
3 FAR 9.505-4(b)
RTAC Mid-term Performance Evaluation
1
Disclosure of Conflict of Interest for USAID Evaluation Team Members
Name Kathylynn Pierre Griff
Title Evaluation Methods Specialist
Organization IBCTI
Evaluation Position? Team Leader X Team member
Evaluation Award Number (contract
or other instrument)
USAID RTAC MPE 19-189
USAID Project(s) Evaluated
(Include project name(s),
implementer name(s) and award
number(s), if
applicable)
RTAC Midterm Performance Evaluation
I have real or potential conflicts of
interest to disclose.
Yes X No
If yes answered above, I disclose
the following facts: 
Real or potential conflicts of interest may include,
but are not limited to:
1. Close family memberwho is an employee of
the USAID operating unit managing the
project(s) being evaluated or the
implementing organization(s) whose
project(s) are being evaluated.
2. Financial interest that is direct, or is
significant though indirect, in the
implementing organization(s) whose
projects are being evaluatedor in the
outcome ofthe evaluation.
3. Current or previous direct or significant
though indirect experience with the
project(s) being evaluated, including
involvementin the project design or previous
iterations of the project.
4. Current or previous work experience or
seeking employment with the USAID
operating unit managing the evaluationor
the implementing organization(s) whose
project(s) are being evaluated.
5. Current or previous work experience with 
an organization that may be seen as an
industry competitor with theimplementing
organization(s) whose project(s) are being
evaluated.
6. Preconceived ideas toward individuals,
groups, organizations, or objectives of the
particular projects andorganizations being
evaluated that 
could bias the evaluation.
I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this
disclosureform promptly ifrelevant circumstances change. If I gain accessto proprietary information of other companies,
then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and
refrain from using the information for any purpose other than that for which it wasfurnished.
Signature
Date October 22, 2020
RTAC Mid-term Performance Evaluation
2
Disclosure of Conflict of Interest for USAID Evaluation Team Members
Name Edward Jay Allan
Title Senior Technical Specialist
Organization IBCTI
Evaluation Position? Team Leader Team member x
Evaluation Award Number (contract
or other instrument) 
7200M19F00008
USAID Project(s) Evaluated (Include
project name(s), implementer
name(s) and award number(s), if 
applicable) 
Research Technical Assistance Center (RTAC)
NORC
I have real or potential conflicts of
interest to disclose.
Yes x No
If yes answeredabove, I disclose the
following facts:
Real or potential conflicts of interest may include,
but are not limited to:
1. Close family member who is an employee of the
USAID operating unit managing the project(s)
being evaluated or the implementing
organization(s) whose project(s) are being
evaluated.
2. Financial interestthat is direct, or issignificant
though indirect, in the implementing
organization(s) whose projects are being
evaluated or in the outcome of the evaluation.
3. Current or previous direct orsignificant though
indirect experience with the project(s) being
evaluated, including involvementin the project
design or previousiterations of the project.
4. Current or previous work experience orseeking
employment with the USAID operating unit
managing the evaluation or the implementing
organization(s) whose project(s) are being
evaluated.
5. Current or previous work experience with an
organization that may be seen as an industry
competitor with the implementing
organization(s) whose project(s) are being
evaluated.
6. Preconceived ideas toward individuals, groups,
organizations, or objectives of the particular
projects and organizations being evaluated that
could bias the evaluation.
I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this
disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies,
then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and
refrain from using the information for any purpose other than that for which it wasfurnished.
Signature
Date January 13, 2020
RTAC Mid-term Performance Evaluation
3
Disclosure of Real or Potential Conflict of Interest for USAID Evaluations 
Instructions: 
Evaluations of USAID projects will be undertaken so that they are not subject to the perception or reality of
biased measurement or reporting due to conflict of interest.
1 For external evaluations, all evaluation team
members will provide a signed statement attesting to a lack of conflict of interest or describing an existing
conflict of interest relative to the project being evaluated.2
Evaluators of USAID projects have a responsibility to maintain independence so that opinions, conclusions, judgments, and recommendations will be impartial and will be viewed as impartial by third parties. Evaluators and evaluation team members are to disclose all relevant facts regarding real or potential conflicts of interest that could lead reasonable third parties with knowledge of the relevant facts and circumstances to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of
exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Operating Unit leadership, in close consultation with the Contracting Officer, will determine whether the real or potential conflict of interest is one that should disqualify an individual from the evaluation team or require recusal by that individual from evaluating certain aspects of the project(s).
In addition, if evaluation team members gain access to proprietary information of other companies in the process of conducting the evaluation, then they must agree with the other companies to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. 3
Real or potential conflicts of interest may include, but are not limited to:
7. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 
8. Financial interest that is direct, or is significant/material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation.
9. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 
11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 
12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. 
RTAC Mid-term Performance Evaluation
4
1USAID Evaluation Policy (p. 8); USAID Contract Information Bulletin 99-17; and Federal Acquisition Regulations (FAR) Part
9.5, Organizational Conflicts of Interest, and Subpart 3.10, Contractor Code of Business Ethics and Conduct.
2USAID Evaluation Policy (p. 11)
3 FAR 9.505-4(b)
RTAC Mid-term Performance Evaluation
Disclosure of Conflict of Interest for USAID Evaluation Team Members
Name Yasmin Ibrahim
Title Project Associate
Organization IBCTI
Evaluation Position? Team Leader Team member x
Evaluation Award Number (contract
or other instrument)
7200M19F00008
USAID Project(s) Evaluated (Include
project name(s), implementer
name(s) and award number(s), if 
applicable) 
Research Technical Assistance Center (RTAC)
NORC
I have real or potential conflicts of
interest to disclose.
Yes x No
If yes answeredabove, I disclose the
following facts:
Real or potential conflicts of interest may include,
but are not limited to:
7. Close family member who is an employee of the
USAID operating unit managing the project(s)
being evaluated or the implementing
organization(s) whose project(s) are being
evaluated.
8. Financial interestthat is direct, or issignificant
though indirect, in the implementing
organization(s) whose projects are being
evaluated or in the outcome of the evaluation.
9. Current or previous direct orsignificant though
indirect experience with the project(s) being
evaluated, including involvementin the project
design or previousiterations of the project.
10. Current or previous work experience or
seeking employment with the USAIDoperating
unit managing the evaluation or the
implementing organization(s) whose project(s)
are being evaluated.
11. Current or previous work experience
with an organization that may be seen as an
industry competitor with the implementing
organization(s) whose project(s) are being
evaluated.
12. Preconceivedideastoward individuals,
groups, organizations, or objectives of the
particular projects and organizations being
evaluated that could bias the evaluation.
I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this
disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies,
then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and
refrain from using the information for any purpose other than that for which it wasfurnished.
Signature
Date October 9, 2020
RTAC Mid-term Performance Evaluation
February 25, 2021 1
ANNEX F: EVALUATION TEAM ROLES AND BIOS
• Ms. Christine Allison, Team Lead, has nearly 20 years of experience in research, monitoring,
and evaluation, primarily focused on USAID-funded education programs, including extensive
experience with higher education partnerships. For over five years, Ms. Allison served as
Project Director on a higher education leadership and management subcontract and co￾authored the Collaborative Assessment of 11 Higher Education Institutions in Indonesia. She
has also worked on the Higher Education Institute study with USAID’s Middle East and Asia
Bureaus where she oversaw fieldwork in four countries and developed a tool to guide strategic
outcomes for higher education partnership programming. Ms. Allison has also led and/or
participated in over 20 research studies or valuations in Europe, Asia, Africa, Central America,
and the Middle East. Allison will receive her PhD in Organizational Leadership and Policy
Development from the University of Minnesota in 2019. The Team Lead is responsible for the
day to day management of evaluation activities, and will work with other team members to
design the evaluation, to collect and analyze the data, to distill findings, and to lead the
recommendations workshop and final drafting of the report.
• Dr. Kathylynn Griff, Evaluation Methods Specialist, is an educator and evaluator with over 14
years of experience in community engagement, program evaluation, research methods and
applied statistics, much of it gained through her work in Haitian and US universities. Dr. Griff is
has been actively involved in the Haitian development sector for the past 21 years, providing
targeted capacity building services and technical support in the fields of education, research,
M&E, and data collection and analysis. She has worked with USAID in multiple assignments as
an evaluation and research expert. Dr. Griff received her doctorate in Education – 
Organizational Leadership from Nova Southeastern University in 2012. The Evaluation Methods
Specialist will work hand-in-hand with the team leader to design the evaluation and the
instruments, to collect and analyze the data, to distill findings, and to author the report.
• Ms. Elizabeth Nolan, Evaluation Methods Specialist, is an IBTCI employee with 11 years’
experience in research and evaluation, survey research, data management, statistical analysis,
and managing Excel-based data visualization and storage platforms. She is seasoned in
collecting and analyzing both quantitative and qualitative data, for which she is proficient in
using SPSS and Stata for quantitative analysis and Atlas.ti and Dedoose for qualitative data. Ms.
Nolan graduated with a Master of Arts in International Development Economics from American
University in 2016. The Evaluation Methods Specialist will work with the team to collect,
manage, and analyze the data.
• Mr. Oleksandr Rohozynsky, Project Director, is an IBTCI employee and is an economist with
over two decades of experience in monitoring, evaluation, and learning. He has worked with
several different clients including USAID, DOS, DOD, DOL, and WB. His technical expertise
includes development of project MELPs, development and implementation of project MEL
systems, including software solutions; development of methodology and implementation of
performance and impact evaluations; survey design and implementation; design and
implementation of quantitative and qualitative data collection, including fields work and
software solutions for remote data collection. Mr. Rohozynsky has subject expertisein several
sectors including macro and micro economics, economic development and forecasting; public
finance and budgets; HIV/AIDS and health systems development; ICT, tourism, and wine
sectors; training, education programs, and EGRA
• Ed Allan, Technical Specialist/Consultant, is a consultant at IBTCI with over 35 years
internationally-related experience, and over 20 years in development, implementation, and
evaluation of projects in education/human capacity development (emphasis on PEPFAROVC
RTAC Mid-term Performance Evaluation
February 25, 2021 2
and other especially vulnerable children), environmental awareness, and strengthening of civil
society funded by USAID, the World Bank, the State Dept., and other funders; experience
managing multi-sectoral evaluations. Regional focus on Middle East/North Africa and Sub￾Saharan Africa (particularly Horn of Africa), plus Asia experience. Ed Allan provides technical
expertise on key project activities and assists the project team in overall project monitoring.
• Yasmin Ibrahim, Project Associate, is an IBTCI employee with over two years of experience in
international development, serving as an intern at International Business and Technical
Consultants Inc.(IBTCI) and most currently as a Program Associate for the Global Health Practice.
Through her work at IBTCI, Yasmin has supported business development efforts for several
health-focused proposals, conducted literature reviews and research, and supported the global
health practice with routine project tasks. Before joining IBTCI, Yasmin served in an
administrative role for two years at a local university, where she provided assistance to students 
and faculty throughout the Arts and Sciences department. Ms. Ibrahim holds a Bachelor of Science
degree in Community Health with a concentration in Global Health from George Mason
University. As a Project Associate, Yasmin provides assistance on contractual and administrative
tasks as needed.
1
ANNEX G: Statement of Difference
Memo: Statement of Difference in Regard to the Research Technical 
Assistance Center Midterm Performance Evaluation
To: Tara Hill, Ashley Heiber, and Megan Miller, USAID
From: Sutherland Miller III, RTAC Project Director
CC: RTAC Staff
Date: April 15, 2021
Introduction
The Midterm Performance Evaluation (MTE) of the Research Technical Assistance Center (RTAC), led 
by NORC at the University of Chicago, was published on February 25, 2021. The evaluation report was 
prepared for the U.S. Agency for International Development (USAID) Bureau for Development, 
Democracy, and Innovation/Innovation, Technology, and Research Hub/Research Division (DDI/ITR/R) 
by International Business & Technical Consultants, Inc. (IBTCI). This memorandum highlights NORC’s 
priority observations in regard to errors of fact and differences in opinion regarding the findings, 
conclusions, and recommendations stated in the evaluation report. This statement of difference quotes 
from the evaluation report from the executive summary through the main body of the text and offers 
several corresponding clarifications.
In many instances, NORC agrees with the evaluation findings. Furthermore, we appreciate the 
acknowledgement that changes to RTAC’s operations and strategy have started to address many of the 
evaluation recommendations based on: a) open and routine communications and collaboration with 
USAID’s management team; b) RTAC’s internal collaborating, learning, and adapting (CLA) efforts with 
consortium members; and c) discussions with the IBTCI evaluation team. Indeed, RTAC has proposed 
that a CLA session occur with USAID project leadership to discuss the findings of the MTE so that 
further progress can be made. 
Responses to the Evaluation Findings and Interpretations
On the Research Network (RN)
Finding 1… “While the RTAC RN has exceeded its membership demographic targets, almost every 
principal investigator (PI) for new research activities under Task 7 had to be recruited outside the 
network. Some of the requests were highly specialized from a technical perspective, but this fact 
highlights the challenge that NORC and its consortium face in undertaking recruitment without 
advanced knowledge of the kinds of rapid research that may be requested.” (page 8)
Finding 4… “Further, many of the PIs have had to be recruited from outside the network, representing a 
significant additional level of effort on the part of the Research Director.” (page 11)
“RTAC has provided global, diverse, and qualified academic researchers with expertise in a wide variety 
of topics. However, it was originally envisioned that a key benefit of the RN would be in aggregating and 
providing readily available access to that group of researchers. Despite the RTAC team’s success at 
assembling such a network, recruitment of PIs has consistently had to be done outside the network, and 
existing RN members themselves identified a range of conditions, most of which are present in each 
2
solicitation, that would inhibit them from bidding on a study. That, therefore, highlights the question of 
the realized value of the RN...” (page 34)
These findings should be interpreted in light of the delays experienced in obtaining Office of 
Management and Budget (OMB) approval to launch the Research Network. Obtaining requisite 
USAID and OMB approvals for the registration of new RN members took over a year. During that 
period, which falls under the purview of this evaluation, RTAC was not able to accept any new 
members and, hence, was not able to tailor network membership to meet particular needs. 
Another factor to be considered is that RN membership is based on expertise matching USAID’s 
technical sectors (nine when RTAC started operations, eleven to date1). Areas of expertise with 
innovative methodologies (such as digital tools, machine learning, and artificial intelligence) 
requested in some activities are not naturally linked to any of these nine sectors. This fact helps 
explain the need to seek other type of highly specialized expertise outside the originally anticipated 
character of the Research Network. 
Furthermore, while some PIs were recruited outside the RN, this does mean that no other qualified 
RN members responded to a given solicitation. The pool of candidates applying for buy-in activities 
has been predominantly composed of RN members, but on occasion it has included researchers 
external to the network – generally due to specific specialization requirements and because USAID 
Missions, Bureaus, and Independent Offices (M/B/IOs) issuing the buy-ins have identified preferred 
researchers. RN members are responsive to buy-in opportunities, but technical USAID teams 
requesting the buy-in ultimately select the finalists.
Finding 3… “However, at the time of data collection for the evaluation, the RTAC team had few tools at 
its disposal to generate any incentives. Other than bidding on the RFPs, there is no way to be ‘active’ in 
the RN.” (page 10)
This statement is not fully accurate. In March 2020, NORC/Institute of International Education (IIE)
gathered inputs from 109 RN members about their expectations and hopes for the RN, which then 
informed subsequent RN incentive activities. Shortly thereafter, in April 2020, RTAC highlighted the 
work of some RN members on its website. We had similarly featured RN members on the website 
in 2019. In May 2020, RTAC carried out a webinar on proposal development and in June 2020, we 
conducted a RN member training activity. Hence, at the time of data collection for this evaluation, 
there had been a consultation, planning and a several activities that actively engaged the RN 
members in RTAC activities beyond responding to buy-ins.
Finding 4… “It is not yet clear that the effort dedicated to building the RN itself has yielded better 
results than what an approach exclusively using targeted recruitment would have yielded in the absence 
of a network and for a lower cost.” (page 11)
It is the size and skill diversity of the RN that enable RTAC to carry out external outreach quickly 
when needed. The RN is always the starting point for targeted recruitment and it is what enables 
RTAC to present a slate of candidates to USAID within one or two weeks of announcing a buy-in 
opportunity. Without the RN resources and contacts, which help mobilize interested and qualified 
researchers around a buy-in opportunity in a matter of a few days, RTAC would not be able to 
respond to opportunities around the diverse topics and countries of interest to USAID in such a 
short time window.
 1 https://www.usaid.gov/what-we-do
3
Finding 5… “For example, the requirements for institutional affiliation and the use of English as the 
lingua franca within the network do potentially exclude some researchers, but USAID intended RTAC 
to engage the academic community and resources are not consistently available for translation.” (page 
12).
At the onset of RTAC activities it was decided to use English as the lingua franca for RTAC activities 
and communications. Hence, translation to other languages would only be done as/when needed. 
This was a programmatic decision made at the start of the project, not a decision based on resource 
availability. 
On Task 4 activities
Finding 51… “As of the time of data collection, RTAC’s MEL Plan did not include any approaches for 
measuring and collecting data on the utilization of research results… despite having an indicator of 
research utilization, no concrete plan for collecting data on research utilization was included in the MEL 
Plan originally, although the RTAC team had the development of a data collection strategy as an item in 
its work plan for 2020. One challenge with the lag in adoption of an indicator, however, is that data on 
utilization for the early R2A and communications products developed under Task 4 were not being 
tracked systematically. Two consortium respondents reported that the role of RTAC was to deliver the 
plan, but not to do follow up or monitoring, so the awareness of the R2A plan’s results was not 
guaranteed. While consortium members, according to one respondent, would always let the R2A and 
Communications PIs know they were very interested in hearing of how the plans’ implementation was 
going, there were no clearly defined points at which research teams would report results to RTAC (as 
of the data collection period for this project). Some of these concerns may be relieved with the 
addition of subtask 4.C.2 – conduct an evaluation and develop a recommendations report of the process 
and efficacy of the R2A planning process, which was added to the year 3 RTAC work plan.” (page 32)
It should be noted that the RTAC MELP has been adjusted to measure impact-related results since 
the initial design of the project solicitation. Similarly, USAID has itself introduced some of its own 
monitoring and feedback mechanisms during Project Year 2 and 3. Specifically, RTAC is now 
tracking USAID standard indicator L3.S.2 – “Number of policy/programming recommendations 
adopted by the public sector, the private sector, USAID, or other development actors that are 
influenced by RTAC-funded research results or related scientific activities.” In the first half of Year 
3, RTAC pilot-tested and finalized the Policy/Program Change (PPC) Survey, a tool that we are using 
to track the number of policy/program changes that were influenced by RTAC-funded research and 
R2A plans. Because policy/program changes take time to influence, per protocol, we do not send 
out the PPC Survey until four months to one year after a buy-in is completed or an R2A plan has 
been finalized. Thus, RTAC has been able to go back and track data on all relevant buy-ins and most 
R2A plans since RTAC was established (the only R2A plans not receiving the PPC Survey are the 
first four, whose PPC data are being collected through subtask 4.C.2.).
It is also worth noting that USAID and RTAC jointly decided not to track policy/program change 
influenced by research communications products because it is too difficult to assess the influence of 
a specific communications product (e.g., a research brief or journal article) versus the dissemination 
effort (e.g., presentation, personal communication) used to get that product into the hands of a 
policymaker. Since RTAC’s Sope of Work (SoW) for research communication products does not 
extend to dissemination, we cannot be fully credited with or accountable for the utilization of those 
products. 
On trainings and convenings
4
“Significant divergence in understanding of the scope of work and expectations for the trainings caused a 
series of challenges in developing training modules under Task 6. Some consortium members felt there 
was limited collaboration in the design of the convenings and noted the need for coordination of 
logistics and content from the beginning.” (page v)
The text here seems to blend observations on the trainings (Task 6) with the project convenings 
(Task 5). In regard to the former, consortium members – Population Reference Bureau (PRB) and 
Arizona State University (ASU) – were principally involved in the design, logistics, and content 
development of the training courses in close collaboration with USAID and NORC. In regard to 
subtask 6.B, the development of the training design and content in Years 1 and 2 was guided by an 
intensive co-creation process spearheaded by several members of the relevant USAID team and the 
logistical burden (i.e. using a NORC conference room equipped with projectors) was nominal. 
Furthermore, for Task 6.B, multiple changes in USAID Activity Managers throughout the course of 
the activity led to changes in activity scope and expectations. This may have been viewed by new 
Activity Managers as a “significant divergence in understanding of the scope of work and 
expectations for the trainings,” but instead represented evolving and changing client expectations. 
As referenced, “The RTAC team noted that goals continued to be clarified, and USAID’s 
expectations shifted repeatedly over time. USAID respondents indicated that the significant time 
lags from one round of reviews to the next (the timing required by the RTAC team and its 
consortium partners to facilitate the execution of other deliverables) meant that the training needed 
to be updated to reflect the most current situation within the Agency.
 
In regard to subtask 6.C, RTAC partner ASU was responsible for content development, event 
logistics, and hosted the webinars, again with NORC support, on its own preferred online platform.
From the start of the activity, ASU worked closely with USAID Activity Managers to develop the 
curriculum and slide deck with speaker notes; there was close collaboration among ASU, USAID 
and NORC throughout the process. In regard to the online courses offered for subtask 6.B, which 
were planned in Project Year 2 to take place in Project Year 3, the design, logistics, and content 
were developed and managed by NORC itself in close collaboration with USAID, so no “consortium 
members” were involved.
“RTAC’s purpose is to support evidence-based programming at USAID to assist countries on their 
journey to self-reliance.” (page v)
The “journey to self-reliance” strategy was finalized after the RTAC project was designed. As such, 
project implementation was adjusted to fit in with emerging priorities under this strategy. At the 
time of writing, it is unclear to what extent the “journey to self-reliance” will remain a priority 
policy framework for the new Administration. 
Finding 9… “RTAC has supported three convenings thus far… After each of the events, NORC 
surveyed participants on their perceptions of the utility of the information presented and held after 
action reviews to assess performance with USAID and consortium members. All of the convenings were 
well received and appreciated by the participants, though the utility of what they learned in the first two 
fell somewhat below the target RTAC set of 80% of convening participants that say the event provided 
useful, actionable knowledge.” (page 15)
It is useful to note that each of the convenings were of a unique design and developed with different 
objectives in mind. In all cases, the design and planning of activities was done in close collaboration 
5
(co-creation) with USAID staff. Led by DevLab at Duke University, the expert panel on Digital 
Tools and the Future of International Development, which was programmed as part of the initial 
project Scope of Work, was centered on gathering a diverse cadre of experts to present various 
options for how digital tools could, broadly, advance development objectives and to highlight 
USAID’s thought leadership in the area. Indeed, seven USAID officials moderated panels or gave 
remarks at the event. The convening on Challenges and Best Practices for Research Translation in 
South East Asia, jointly sponsored by USAID and the United Kingdom Research and Innovation 
(UKRI) was tailored toward developing the awareness and appreciation for research translation and 
communication among sponsored researchers, practitioners, and government representatives from 
the Asian region and was one component within a larger buy-in activity. The Using Research to 
Advance International Development: Women’s Empowerment as a Case Study was focused on 
convening thought leaders and practitioners in the area to discuss approaches to advance women’s 
empowerment, specifically in four sectors, forge relationships for future collaboration, and enable 
USAID personnel to learn from presenters and participants. As such, drawing conclusions across all 
of these disparate events under the umbrella of “convenings” may be misleading, particularly because 
some of these gatherings were oriented to sharing thought leadership and state of the art 
knowledge, which is not always equivalent to actionable knowledge.
“To resolve the situation, direct engagement with the activity managers provided an enabling 
environment to manage these changes and client expectations throughout the process. Being very clear 
about agreements and putting things in writing really helped with this process. Ultimately, the RTAC 
team delivered the “Using Research Evidence to Accelerate International Development Outcomes” 
training. Its timeline was affected by multiple delays, and it required several iterations to produce a final
product that was acceptable to USAID, taking almost two years to complete. Ultimately, only a few 
trainings were conducted.” (page 17)
The two planned trainings led by RTAC personnel were completed in addition to an unanticipated
training course led by USAID staff and observed by RTAC staff. Ultimately, the timeline for the 
trainings were driven by USAID based on: a) the availability of trainees and core staff to take 
ownership of the sessions; b) evolving refinements requested by USAID; and c) changes in scope and 
focus based on changes in leadership among USAID Activity Managers. Based on the COVID 19 
pandemic, the contract was modified from a series of webinars to three on-line training courses 
based on the material developed for the in-person trainings. The first of these trainings, and an 
associated training-of-trainers course for USAID facilitators, was completed in December 2020.
“A second set of training materials has been developed under Task 6.C, ‘Applying Digital Tools to 
Support the Journey to Self-Reliance.’ Turnover on the part of the technical activity managers at USAID 
limited the amount of information the evaluation team was able to collect on this training, but others 
familiar with the process noted that similar concerns and experiences arose around this activity.” (page 
17)
It would be useful to better understand what “similar concerns and experiences arose around this 
activity” means. We note that at least two of the three lead USAID Activity Managers were 
involved throughout the design and delivery of the training. The USAID Activity Mangers, principally 
through regular planning calls and reviews of content and materials, worked closely with ASU faculty 
and consortium leadership in designing the training and developing the training materials. 
Significantly, the SoW was changed after the two in-person trainings were completed to shift from 
an internal training webinar for only USAID personnel to a publicly accessible event. This required 
significant adaptations to the training modules and materials based on the formalization of the 
emerging “Digital Strategy: 2020-2024,” which was under development while the training was being 
6
designed and implemented. Furthermore, a planned training session in Ghana was cancelled by 
USAID and shifted to being a U.S.-based training, prompting further adaptation in timeline, audience, 
budget, and logistics by ASU and NORC. 
Finding 32… “Supporting international travel and visa acquisition in particular required more lead time 
than the RTAC team had anticipated under Task 5… In two of the convenings, some invitees were 
ultimately unable to attend as they had insufficient time to obtain visas and organize their travel. The 
RTAC team learned from this experience, documented it in their after-action reviews, and made plans 
to ensure that it would not be a problem in future convenings.” (page 26)
Recommendation 5.2 (NORC)… “Ensure that the planning process allows sufficient time for internal 
approvals and coordination of logistics, particularly where visa support and other travel support will be 
required. (Findings 9, 20, 32)” (page 37)
The RTAC team is aware of the time needed to facilitate international travel, including visa 
requirements. Indeed, when hosting individuals in the United States the standard process of 
blocking hotel rooms requires an adequate planning horizon to facilitate domestic and international 
travel. Planning for international travel to a large conference in Bangkok did, however, pose 
challenges due to a confluence of factors beyond the control of either RTAC or USAID, including 
conflicting visa-related information provided by entities in different countries. In regard to the U.S. 
convenings problems related to finalizing international travel arrangements were primarily due to 
delays in USAID’s selection and approval of sponsored travelers. 
Finding 33… “Some consortium members felt there was limited collaboration in the design of the 
convenings and noted the need for coordination of logistics and content from the beginning…
Consortium members noted that they would like to be included in the drafting of the scope of work to 
limit the number of unforeseen challenges that may arise in the planning and delivery of convenings. 
Another lesson learned from the convenings was the need to have the logistical and content planning 
coordinated from the beginning. Some consortium members felt that in the first convening, the 
individuals responsible for content had too many responsibilities to also support logistics and that 
logistics specialists were engaged too late in the process. In the third convening, the logistical specialists 
were engaged earlier to facilitate smoother planning processes aligning content and logistics.” (page 26)
It is useful to note that for the first convening, the expert panel on Digital Tools and the Future of 
International Development, the SoW was written as part of the project solicitation. As such, 
neither the technical lead, DevLab at Duke University, nor NORC were afforded an opportunity to 
make changes to the SoW or budget. However, DevLab was fully involved in all content and 
program development decisions in close collaboration with USAID and NORC. The logistical 
subcontractor, Davis Management Group (DMG), was engaged approximately four months in 
advance of the event and NORC did not perceive any deficiencies with logistical planning or 
performance. 
The Using Research to Advance International Development: Women’s Empowerment as a Case 
Study was organized by NORC without the involvement of “consortium members” save for the 
logistical subcontractor DMG. DMG was engaged approximately four months in advance of the 
event and NORC did not perceive any deficiencies with logistical planning or performance. 
Recommendation 5.4 (NORC)… “Ensure that outside experts are briefed on the composition and 
expertise level of the audience for appropriate targeting of content.” (page 37)
7
As a matter of course, NORC shares a description of the intended audience and preliminary 
registration lists with convening presenters and event facilitators as part of the planning process.
Recommendation 6.3 (NORC)… “Prior to training design, ensure that specifications for audience, 
learning objectives, expected behavior changes, mode of delivery, and expected training duration and 
iterations have been approved. (Findings 11, 26, 34)” (page 37)
This must be at least a shared responsibility, as defining “learning objectives… and expected training 
duration and iterations,” for example, require clear articulation of preferences from USAID 
counterparts.
On buy-ins
“Two USAID technical clients were particularly dissatisfied with the final research product of one buy-in 
(as was another with an interim product at the time of data collection, and therefore no final assessment 
of utility could be made). The technical clients who were not happy with one product felt that RTAC 
did not have the right researchers with good local networks with people who could relate to the local 
culture and gather information about the reality on the ground.” (page 18)
It should be noted that while RTAC facilitates the solicitation of research teams, provides support 
for bidders in completing the required applications process, and offers a non-binding ranking of 
bidding teams, USAID is ultimately responsible for the selection of winning implementers of Task 7 
activities. As such, it is not possible for RTAC to “have the right researchers.” Despite the feelings 
of the technical clients, this observation may represent a reflection on the limitations of the RN to 
adequately yield researchers with the right set of skills to implement the wide range of buy-ins 
requested at specific points of time in specific countries, and/or the ability of USAID selection 
committees to choose the right expertise among bidders. It is also possible that the buy-in SOWs 
are not sufficiently detailed in listing the full set of qualifications USAID wishes to see in the ideal 
researcher or team.
Finding 17... “A second common factor across successful RTAC activities was PIs’ prior experience with 
USAID and/or with governmental agencies more broadly. In general, PIs who had experience working 
with USAID in the past, or in a few cases with other governmental agencies, had a clearer idea of the 
technical clients’ needs and types of deliverables required. Although some of the concerns about 
editorial quality control arose with those deliverables, the technical aspects of the deliverables were 
more aligned with USAID’s expectations upon initial submission.” (page 20)
We agree with this observation. However, given RTAC’s focus on engaging new and underutilized 
researchers from Low and Middle Income Countries (LMICs) and Minority-serving Institutions 
(MSIs), it may be helpful if USAID Activity Managers had different expectations and a higher 
tolerance when it comes to deliverable quality and fidelity to SoWs, particularly at the first draft 
stage and given the very tight timelines for buy-ins. The tight timelines leave little to no time for 
quality control and editorial oversight on the part of RTAC. We note that some of the Task 4 
products that USAID has been most pleased with have benefited from ample time for review and 
quality control for considerably simpler products, a luxury not afforded to Task 7 buy-in research 
products.
Finding 18… “Across tasks, clear and direct communication was critical. One consortium member 
reported that having direct interaction with the activity manager at USAID improved outcomes and 
reduced scope creep under Task 4.” (page 20)
8
While we agree with the benefits of direct communication, and have created spaces for this through 
kick-off meetings and regular checkpoints during implementation, unfettered direct interaction of
between Activity Managers and researchers under Task 7 have led to “scope creep” in some cases. 
Hence, there is a balance to be struck here between open dialogue and regular contact and 
adherence to a given SoW.
Finding 29… “Some also alluded to a lack of initiative on the part of NORC, particularly around 
requested/potential changes to scope. Four respondents indicated that they would have been amenable 
to increasing the budget for the activity if they could have made changes to their scopes, but that was 
never introduced as an option.” (page 25)
As a matter of course, USAID has been resistant to modify SoWs or budgets after the initial buy-in 
negotiation process has been concluded. However, USAID has showed great innovation and
flexibility in creating a DDI/ITR/R special projects fund which is often used to cover modest
additional costs incurred based on implementation of a particular SoW. The RTAC team tries to 
accommodate scope changes, whenever possible, within the allotted buy-in budget since adding 
funds, particularly through a centrally-managed vehicle like RTAC, is not a simple procedure 
administratively, and in addition to funding availability, there is also the question of how quickly the 
funds can be added. 
For these reasons, RTAC typically tries to manage within the existing budget as best it can. When 
we have had a scope change with a significant associated budget increment, the additional financing 
has often been covered by the special project fund, which does not require a contract modification. 
However, if the scope change is important to the client, then it would be helpful for them to signal 
that adding funds is a possibility, either initially or if the proposed action within the existing budget is 
not, in their view, sufficient.
Finding 30… “RTAC was not well known outside of the Global Development Lab, and only a few other 
M/B/IOs have bought into the mechanism… Although 50% (four) of the research buy-ins included in this 
evaluation were funded by units other than the Global Development Lab, one final broad barrier to 
providing maximum value to USAID was that RTAC and the services it offers were not well known 
throughout USAID, although there are obvious drawbacks to the potential of creating demand that 
available resources could not satisfy.” (page 25) 
To date, RTAC has completed or is presently implementing 19 buy-ins from various M/B/IOs 
throughout USAID; of these 15 have been funded by units outside DDI/ITR/R. Given the lag 
between the time the evaluation was conducted and the release of the associated report, the 
numbers in the report do not accurately reflect the trajectory of RTAC through February 2021 
(report date).
Finding 35… “Although the RTAC contract permits cost reimbursement subcontracts, some 
universities’ own policies prohibit accepting contracting on a cost reimbursement basis. Thus the 
source of the concern may be a university’s contracting regulations rather than USAID’s contracting 
regulations.” (footnote 6, page 27)
This statement is not accurate in the framework of Task 7 activities. With few exceptions, all buy￾ins have been framed under a fixed price contract modality. Exceptions that have allowed budgetary 
components to be managed under the cost reimbursement modality are linked to activities with 
significant travel, event organization costs, or scenarios where the pandemic may make in-person 
9
gatherings impossible and a virtual option much be created and re-budgeted. In these cases, these 
costs have been budgeted separately from the fixed price research component.
Finding 36… “The need to recruit PIs for Task 7 research studies outside the network constitutes an 
additional burden on the Research Director and may inhibit wide-spread recruitment... All the PIs for 
the research buy-ins included in the evaluation were recruited outside the network. This represents an 
investment of time on the part of the Research Director to identify the right types of candidates for 
each study. However, there is a disincentive to recruit more than two or three candidates for each 
given the cost of recruitment time and opportunity cost for the potential applicants, particularly those 
who do not have experience with competitive contracting processes.” (page 27)
This information is not accurate. All buy-in activities implemented by the time of data collection 
went through a competitive process where RN members and/or RTAC consortium partners 
submitted proposals for USAID’s consideration. In some cases, where the SoW emphasized a 
particular methodology, tool, or highly specialized field of expertise not naturally aligned with 
USAID’s technical sectors, or a country not covered by the RN members, outreach was also done 
to non-RN members with such expertise (thematic or geographic). In such cases, those recruited 
for such an activity and then encouraged to join the RN. Outreach is supported by RN members, 
RTAC partners, NORC, and in a few instances by USAID itself. The short turnaround time for 
submitting proposals (one or two weeks, depending on the activity) dictates the extent of external 
outreach, as these individuals or institutions have a shorter proposal submission timeline because
they typically learn about the opportunity a few days later, compared to the RN members.
Furthermore, USAID makes the ultimate selection of the implementing team. In some cases, the 
selected researcher/teams were researchers those identified by RTAC through outreach external to 
the RN. However, the pool of candidates submitted to the USAID team also includes RN members.
Finding 37… “Quality standards and intended audiences were not well understood by some buy-ins’ 
lead researchers… Early on, NORC and some USAID technical managers assumed that all university￾based researchers know how to write effectively for policy audiences. As USAID heavily critiqued 
multiple early deliverables for not clearly addressing the research questions for that audience, as well as 
significant editorial failings, NORC added support to assist the Research Director in reviewing and, 
where necessary, editing deliverables. Despite these investments, however, some deliverables from 
Year 2 buy-ins have had similar issues.” (page 28).
This finding can be enhanced with more precision about the characteristics of the deliverables. For 
the most part, buy-ins are research activities and the SoWs require a report as main deliverable. 
Academic researchers tasked to carry out a research activity typically have the experience to 
produce a research report. However, differing expectations about how a report should be drafted 
for research and policy purposes is indeed a major challenge experienced by buy-in teams, 
particularly those without previous experience working with USAID. Also, each USAID team has
different expectations for the “report” and these are usually discovered during implementation of 
the activity after the draft report is prepared (as Finding 49 explains).
Over time, NORC and USAID Activity Managers have been quicker to request clarifications from
the USAID technical teams about the expected characteristics of the deliverables and their intended 
audience and use. In more recent buy-ins, additional communications-oriented deliverables have 
been incorporated in the SoWs, such as briefs, presentations, data visualizations, and infographics. 
These additional deliverables, intended to meet the needs of USAID technical teams, have generated 
additional challenges for the academic researchers who are not experienced in drafting these types
10
of products. As needed, NORC has stepped in and supported the research teams in producing
these additional communication products.
Finding 43… “Several consortium members felt their organizations’ talents were underutilized in the 
RTAC project… Five consortium members indicated that their organizations brought great strength to 
the RTAC team, but that given the scope of activities and the limited involvement they have had, their 
resources were underutilized. One consortium respondent reported that this was partially the result of 
challenges in the project team’s internal management practices. Specific examples included a lack of 
internal project management systems, irregular or poorly structured team meetings, and unclear lines of 
accountability.” (page 29)
Given that all consortium partners have specific and detailed subcontracts that have been discussed 
and negotiated during the annual work planning process, this conclusion is surprising. All university￾based consortium members are able to bid on any and all Task 7 buy-ins. Some consortium 
partners, such as PRB, Forum One, and IIE, with significant responsibilities on the project have 
regular meetings with RTAC management to discuss scopes of work and activity progress. Other 
consortium members – ASU, Duke University, Notre Dame and ResilientAfrica Network (RAN) –
have been assigned more discrete tasks on the projects, including leading buy-ins. As such, they 
meet with RTAC leadership and USAID Activity Managers on a regular basis around those specific 
activities while they last. 
The perceived underutilization of talents is not a result of poor management practices, given that 
RTAC’s internal management systems do not correlate with the generation of demand for services 
by USAID. In the Year 3 workplan, RTAC leadership included several innovative ideas submitted by 
consortium members; however, USAID chose not to fund them. Further, after Year 1, USAID has 
not requested convenings/panels that could draw on the skills and expertise of consortium 
members. As a result, the engagement of consortium members like Duke, ASU, Notre Dame, and 
RAN in RTAC is limited to buy-ins (when they bid and are selected) and, more recently, leading 
Webinars for Research Network members. By contrast, PRB, Forum One, and IIE, have a more 
integrated and ongoing role on the project.
Key Recommendation 7.8… “Include interim deliverables in scopes of work for research buy-ins that 
will gather new data that allow NORC and USAID to monitor compliance with scope (Findings 18, 24, 
26, 28, 37)”. (page 38)
Buy-in activities do include meetings with USAID staff at kick-off and certain points in time to 
provide updates (“check-in calls”). Hence, there are opportunities to hear about progress, 
challenges, and initial insights early in the process.
More deliverables are not an optimal solution given tight timelines and small fixed price budgets as 
they add time, LOE, and costs. Based on the experience so far, we have shared with USAID the 
disadvantages of having researchers allocate their limited time to the simultaneous drafting of 
multiple deliverables as this process affects the quality of all deliverables. Hence, a more fitting 
alternative is to change the nature of certain deliverables and touch points to accommodate 
information exchange and learning, without adding burden on the researchers that could deviate 
them from their priority task (i.e. generating the knowledge needed to answer the research 
questions).
Multiple touch points with USAID teams along the activity are certainly useful and can help 
accommodate these exchanges, but it is important to keep in mind that this has sometimes resulted 
11
in scope creep, unexpected technical adjustments, and delayed timelines due to the difficulty in 
scheduling the meetings. Hence, these meeting points should be defined such that take into 
consideration the activity timeline, budget, and the phases in the research where exchanges are 
useful.