April 2014
This publication was produced at the request of the United States Agency for International Development. It was 
prepared independently by Robert Potter. 
Performance Evaluation of the Program for Biosafety 
Systems
Cover photo: Confined field trial of genetically engineered rice in the Philippines. Photo credit: 
International Rice Research Institute
PERFORMANCE 
EVALUATION OF THE 
PROGRAM FOR 
BIOSAFETY SYSTEMS:
SUPPORTING PARTNER COUNTRIES BY FACILITATING USE OF 
BIOTECHNOLOGY FOR THE EVENTUAL DEPLOYMENT OF 
BENEFICIAL AGRICULTURAL PRODUCTS
April 22, 2014 
Grant No: AID-EEM-A-00-03-00001; Evaluation Mechanism Number: AID-BFS-P-13-00001
DISCLAIMER
The author’s views expressed in this publication do not necessarily reflect the views of the United States
Agency for International Development or the United States Government.
CONTENTS
Acronyms ...................................................................................................................................................... 1
Executive Summary....................................................................................................................................... 2
Project Background ....................................................................................................................................... 6
Evaluation methods & limitations................................................................................................................. 10
Evaluation Design.................................................................................................................................... 10
Data Collection and Analysis Methods ......................................................................................................... 10
Methodological Limitations ...................................................................................................................... 12
Findings, Conclusions & Recommendations ............................................................................................... 14
Findings................................................................................................................................................... 14
Country Case Studies.............................................................................................................................. 22
Conclusions and Recommendations ............................................................................................................ 25
Concluding Remarks ............................................................................................................................... 27
Annexes
Annex I: Evaluation Statement of Work................................................................................................... 32
Annex II: Evaluation Methods and Limitations......................................................................................... 45
Annex III: Data Collection Instruments .........................................................................................................48
Annex IV: Sources of Information............................................................................................................ 52
Annex V: Disclosure of any Conflicts of Interest...................................................................................... 54
Annex VI: Statement of Differences ........................................................................................................ 55
ACRONYMS
AATF African Agricultural Technology Foundation
ABNE African Biosafety Network of Expertise
ABSPII Agricultural Biotechnology Support Project II
BCGEP Biosafety Commission for Genetically Engineered Products (Indonesia)
COMESA Community for East and Southern Africa
ECOWAS Economic Community for West and Central Africa
GM Genetically Modified
ICGEB International Centre for Genetic Engineering and Biotechnology
IFPRI International Food Policy Research Institute
ISAAA International Service for the Acquisition of Agribiotech Applications
ISBGMO International Society for Biosafety of Genetically Modified Organisms
M&E Monitoring and Evaluation
NBRC National Biosafety Regulation Committee (Malawi)
NGO Non-Governmental Organization
OECD Organization for Economic Cooperation and Development
OFAB Open Forum for Agricultural Biotechnology
PBS Program for Biosafety Systems
SIAB Society for Indonesian Agricultural Biotechnology
SOP Standard Operating Procedure
UBBC Uganda Biotechnology and Biosafety Consortium
USAID/BFS U.S. Agency for International Development/Bureau for Food Safety
1 
EXECUTIVE SUMMARY
EVALUATION PURPOSE AND EVALUATION QUESTIONS
The purpose of this external evaluation of the Program for Biosafety (PBS) is to assess the performance
of the preceding 5-year period, and to identify ways in which the program can be improved during the
next 5-year period. The scope of this review isthe entire range of PBS activities, including policy
research and all in-country programs, but excluding programs that just began in FY2013. The primary
audience for this evaluation is USAID/BFS and IFPRI staff. The evaluation will identify successes and
lessons learned over the past 5 years of program implementation, and help guide the design of the new
IFPRI award to continue PBS activities.
Questions were posed on five topics by USAID to guide the evaluation:
1. PBS Country Programs Staff and Support from International Technical Consultants
Are the roles and responsibilities of the in-country country lead appropriate compared with
international short-term consultants? Should the mix be adjusted to support program expansion and
sustainability of provider support in the next PBS award?
2. Policy Research
Has the policy research carried out by PBS been relevant to the technical support and capacity
development activities being conducted by PBS? Provide specific examples of where PBS policy research
has made a direct and measurable contribution to the achievement of PBS country-level objectives. Are
the research products developed by the PBS team of high quality and contribute to a literature base that
is being utilized by partners and stakeholders?
3. Project Management
How effectively have the PBS Director and Africa and Asia regional program leads supported learning
and ensured quality across countries? How could these roles be strengthened in the next PBS award?
How useful is the current PBS monitoring and evaluation system for program management, results
reporting and learning? Given that many outcomes are outside the manageable interest of the program,
are impacts being captured appropriately across all aspects of the program? How might the M&E system
be strengthened in the next PBS award?
4. Financial Management
Given the goals of the projects as outlined in project documentation, how appropriately have BFS
central resources been allocated amongst countries, partners and activities, to supplement and
complement Mission funding and to support global technical leadership?
5. Communications, Outreach and Partnerships
Has PBS effectively responded with appropriate and timely communications products and outreach
directed to the appropriate audiences as technical and political issues arise? What could be improved? Is
PBS effectively collaborating and working with other biosafety and biotechnology programs in the
regions where these projects are active?
2 
PROJECT BACKGROUND
USAID has been funding efforts in biotechnology and the regulation of agricultural biotechnology for
nearly two decades across several partners and host countries in Africa and Asia. The Program for
Biosafety Systems (PBS) is the largest of USAID’s investments that support the development of science￾based regulatory systems that govern the application of these new agricultural technologies. PBS has
been supported for 10 years by both USAID Washington and USAID field missions through a contract
with the International Food Policy Research Institute (IFPRI).
PBS is active in seven countries in Africa and three in Asia, although this number has changed over time
and remains flexible based on funding and the situation in countries in these regions. PBS supports
partner countries by facilitating use of biotechnology for the eventual deployment of beneficial
agricultural products. The indirect outcome is sustainably enhanced productivity and climate resilience in
agricultural production systems to reduce poverty, enhance nutritional status and improve quality of life
for resource-poor farmers and consumers.
The USAID Bureau for Food Security (BFS) supports the PBS core budget. This budget includes
management staff, technical consultants, global policy research, participation in international meetings
and in-country program and staff costs for some PBS countries and regions. USAID missions support 
PBS to strengthen and deepen programmatic activities in their respective countries. PBS currently
receives mission funding from Ghana, Indonesia, Malawi, Nigeria, Tanzania and Uganda.
EVALUATION DESIGN, METHODS AND LIMITATIONS
Data were collected using two methods: document review and semi-structured interviews with key
informants. Field visits to Indonesia, Uganda and Malawi allowed for interviews with key informants at a
country level as the program is based around country activities. This was supported by interviews with
regional and global collaborators, as well as interviews with senior PBS staff and long-term consultants.
The evaluation team conducted a preliminary desk review of relevant project documents and key sector
background material in order to gain a thorough understanding of the specific activities over the 5 year
period of the project (2008-2013), together with the country context in which each program is
operating. Key informants were identified by the contractor in-country staff in addition to others
identified during the data collection process. Based on a review of project documents the evaluation
team identified several stakeholder groups that are the focus of program activities. These stakeholder
groups comprise (i) USAID and other US Government agencies, (ii) PBS Implementing staff in both the
headquarters and the countries visited, (iii) Government regulatory agencies as well as research
institutes and universities acting as agents of regulators, and (iv) Partners/collaborators (including
technology development groups and the private sector). For each such stakeholder group, specific
questions were designed that focus on their interest in the PBS project. The same base sets of
questions were asked of all interviewees.
In the initial planning of the review and for the first of the three field visits, a specialist in Monitoring and
Evaluation was accompanied by a specialist in biotechnology regulatory affairs as the review team. Due
to the inability of the monitoring and evaluation specialist to continue with the review, the second and
third field visits and the overall drafting of the review were completed by the regulatory affairs specialist,
using the template and methods developed by the monitoring and evaluation specialist as much as
3 
possible.
It should be stressed that this review is based on the responses to questions posed to a large number of 
different stakeholders, many of whom may not be completely aware of the full range of activities 
performed by PBS. Statements in this document include common responses, even if these are not 
completely factual, as they represent the understanding and perception of certain stakeholders.
FINDINGS AND CONCLUSIONS
1. PBS Country Programs Staff and Support from International Technical Consultants
The role of the country coordinator was reported as being the key to the success as this person is the
main interface with the various stakeholders in each country. The technical consultants were highly
praised for their technical ability, but there was a common request for more diversity and more use of
experts from within the region. To date, the technical focus of PBS is well served by the country staff
and experts, however the demands are now moving from primarily technical support to advocacy and
there are different skills required. The value of having external experts from a wider source than just
the United States is tempered by the need to have well-qualified experts and to have them involved in
PBS activities over a long enough period to be familiar with and accepted by local stakeholders.
Development of local expertise, through training and joint research studies, can provide a resource for
the region as well just the country in which they are working.
2. Policy Research
The PBS Policy team, working within and through IFPRI, has played an important role in producing a 
body of work recognized globally as being critical to the understanding of biosafety policy and regulation.
However, there seems to be little recognition of this by country stakeholders and there appears to be
some conflict between IFPRI demands for publications in peer-reviewed journals and products which can
be used to help countries develop practical regulatory systems. This is recognized by PBS management
and attempts are being made to use the research more widely and also to conduct more country-level
studies. The challenge for PBS is to work within IFPRI to allow for recognition of more practical studies
for implementation as a valid output. With regard to recognition of the role of PBS research by local
collaborators, it is less important that these studies are identified as IFPRI publications than that PBS is
seen as a source for documents relevant to the local policy and planning needs. The PBS summaries and
short briefs are where there needs to be “brand recognition” as this is where most of the government
regulators and policy makers will be getting their information.
3. Project Management
Annual work plans and semi-annual reports appear deficient in many respects for proper management,
results reporting and learning. The main problems include:
• Exclusive focus on inputs (and outputs), rather than on outcomes and results.
• Milestones should be reviewed and refined using USAID standard definitions and differentiated
from inputs.
• There is a need to present realistic time frames for achievement of critical milestones, outputs,
and outcomes, so that program progress can be measured and monitored for decision-making.
• Project level intermediate outcomes and results need to be identified. USAID Feed the Future
standard indicators do not capture outcomes and results in sufficient detail to know whether
the inputs are achieving the desired impacts.
• Previous project indicators for capacity building within government agencies should be reviewed
4 
for potential utility. The time for an application to progress through each step of the regulatory
process can be used as a measurement of the effectiveness of the regulatory system, as can the
responsiveness of the various committees to applicants.
4. Financial Management
PBS has been successful to date in juggling the demands on the core budget to provide services and
support in a number of countries which have USAID mission support as well as operating in some key
countries which don’t have such support. There was no suggestion that lack of funds was a key factor in
those cases where political changes have become a road-block to further progress.
A challenge for PBS lies in how much they can expand into new countries, given the demands on the
core budget, recognizing that for each country funding is not only required for country level staff, but
also to support the country activities with external experts and central PBS activities. There is the
potential to reduce effectiveness over the whole program by stretching core staff and external
consultants too far.
5. Communications, Outreach and Partnerships
PBS has entered into partnerships with a number of collaborators to increase the reach of PBS outreach
activities, in some cases forming formal or semi-formal consortia. A number of issues concerning
targeting, the role of communications and outreach and the professionalization of advocacy and PR need
to be addressed. Answers to the following will be helpful in defining and implementing a strategy:
• For whom do the in-country consortia speak? The tighter the connection to PBS, the more the
group becomes a proxy for the US government, which could backfire if PBS loses the trust of
local stakeholders.
• Is there a long-range plan for engagement and institutional support? A clear plan is needed to
determine the relationship in both short and long term financial terms and otherwise.
• What is the capacity of the consortia to do public relations and policy work?
• What is the appropriate target group(s) for the consortia to focus on, what clear and
measureable results are expected from their outreach, and how will they be measured?
• Is policy outreach an appropriate direct PBS role, using respected policy experts both
international and national?
• Are professional communications services needed to complement and support PBS local staff
and volunteers?
• What results are desired, and what indicators will measure the impact of different inputs on
different stakeholders?
5 
PROJECTBACKGROUND
USAID has been funding efforts in biotechnology and the regulation of agricultural biotechnology for
nearly two decades across several partners and host countries in Africa and Asia. Amongst these
programs are those that support Research and Development (R&D) and deployment of new agricultural
technologies using biotechnology and a separate set of programs that support the development of
reasonable, science-based regulatory systems that govern the application of these new agricultural
technologies. The Program for Biosafety Systems (PBS) is the largest of USAID’s investments in this
latter group. PBS has been supported for 10 years by both USAID Washington and USAID field missions
through a contract with the International Food Policy Research Institute (IFPRI).
The role of biosafety systems and facilitative policies to insure that Agency-funded research yields
practical benefits cannot be underestimated since it is widely recognized that biosafety considerations
and concerns, especially in developing countries, can be a primary limiting factor in biotechnology
adoption.
PBS is currently active in six primary countries in Africa and three in Asia, although this number has
changed over time and remains flexible based on funding and the situation in countries in these regions.
PBS supports partner countries by facilitating use of biotechnology for the eventual deployment of
beneficial agriculture products, such as new genetically-modified (GM) varieties of crop plants that are
resistant to a variety of biotic and abiotic stresses (e.g. insect resistant cowpeas and drought tolerant
maize) or GM varieties with enhanced nutrient profiles (e.g. Vitamin A enriched cassava). The indirect
outcome is sustainably enhanced productivity and climate resilience in agriculture production systems to
reduce poverty, enhance nutritional status and improve quality of life for resource-poor farmers and
consumers.
The objective of PBS’s support for developing functional biosafety systems is to ensure that countries
have and evidenced-based legal framework and the laws in place which are consistent with global best 
practices employed by other countries who have opted to adopt the use of biotech crops as part of 
their agriculture system.. PBS support and guidance is intended to provide sustained, credible, and
impartial biosafety legal and policy advice and to enhance capacity of decision-makers. Coupled with
this, targeted, practical/technical support is supplied to regulators so that a system is in place to evaluate 
products for eventual distribution to farmers.. In parallel with the increased number of GM products 
approaching release in developing countries, ongoing public debate and legal challenges are more
frequently emerging as obstacles to progress. PBS has strategic, multi- stakeholder outreach activities,
coupled and integrated to the work it does on the regulatory front, to confront the challenges of 
misinformation and to to avoid backpedaling to more precautionary policies.
The final outcome of the layered support and integrated approach provided by PBS is to establish a 
science-based regulatory system and local capacity to insure that products are evaluated from a point 
of evidence. Developing a country’s capacity would impact and further its national goal of increased
food security, poverty reduction and also better climate resilience.
The PBS program works with local agricultural organizations and stakeholders to build functional
science-based regulatory pathways for the adoption of new biotechnology products. Its diverse team of
6 
scientific, legal, commercial and communications experts builds biosafety capacity through an integrated
program of policy analysis, development and implementation for practical, achievable results. Activities
are designed and implemented through a country-led approach with local PBS country leads. Further,
the ability to draw on an in-house independent policy research team at IFPRI is expected to bring added
value and academic credibility for informed decision-making. IFPRI is a public international organization
and member of the Consultative Group on International Agricultural Research.
USAID Bureau for Food Security (BFS) supports the PBS core budget. This budget includes management
staff, technical consultants, global policy research, participation in international meetings and in-country
costs for some PBS countries (Malawi, the Philippines, Vietnam) and regional activities within the
Common Market for Eastern and Southern Africa (COMESA) and the Economic Community for West
and Central Africa (ECOWAS). USAID missions support PBS to strengthen and deepen programmatic
activities in their respective countries. PBS currently receives mission funding from Ghana, Indonesia,
Nigeria, Tanzania and Uganda.
PBS’s approach to biosafety capacity development is unique among a variety of biosafety service
providers (most of which are funded by other donors) in several aspects, as it emphasizes:
• A comprehensive approach to biosafety framework development, including the development of
workable implementing regulations, detailed operating procedures, and guiding biosafety dossier
development and reviews, among others;
• A sustained and consistent on-the-ground presence in the form of local country coordinators
who facilitate discussions of needs and concerns between local decision-makers and the
project’s technical and managerial experts;
• Development of biosafety frameworks by involving international and local legal and scientific
expertise in support of effective functioning biosafety decision-making;
• Supportive outreach and communications strategies for national competent authorities and the
broader stakeholder community that is a pre-emptive, guiding and responsive to complement
the technical work that is undertaken by the project;
• Capacity development, including hands-on training, around a specific product, policy/legal
instrument, or GM application in the pipeline in partner countries;
• De novo, targeted research to inform the decision-making process and effect policy
development, adoption and implementation.
7 
EVALUATION PURPOSE &
EVALUATIONQUESTIONS
EVALUATION PURPOSE
The purpose of this external evaluation of the Program for Biosafety Systems (PBS) is to assess the
performance of the preceding 5-year period (2008-2013), and to identify ways in which the program
can be improved during the planned next 5-year period. The scope of this review will be the entire
range of PBS activities, including policy research and all in-country programs, but excluding programs
that just began in FY2013 (Ghana, Tanzania and ECOWAS).
The primary audience for this evaluation is USAID/BFS and IFPRI staff. The evaluation will identify
successes and lessons learned over the past 5 years of program implementation, and help guide the
design of the new IFPRI award to continue PBS activities.
EVALUATION QUESTIONS
1. PBS Country Programs Staff and Support from International Technical 
Consultants Are the roles and responsibilities of the in-country country lead appropriate compared
with international short-term consultants? Should the mix be adjusted to support program expansion 
and sustainability of provider support in the next PBS award?
Sub-questions to consider: Is the skill set of the in-country country leads appropriate to achieve PBS
objectives? In what areas of their essential job functions have country leads been effective and where
have they struggled? Do stakeholders feel country leads have enough/the right technical expertise, as
compared to skills required to initiate and keep activities moving forward, and to communicate,
advocate and build consensus among national stakeholders? Has the support provided by international
consultants been effective in achieving the objectives of their STTA and overall PBS country 
objectives?
2. Policy Research
Has the policy research carried out by PBS been relevant to the technical support and capacity
development activities being conducted by PBS? Provide specific examples of where PBS policy
research has made a direct and measurable contribution to the achievement of PBS country-level
objectives.
Are the research products developed by the PBS team of high quality and contribute to a literature
base that is being utilized by partners and stakeholders?
8 
3. Project Management
How effectively have the PBS Director and Africa and Asia regional program leads supported learning
and ensured quality across countries? How could these roles be strengthened in the next PBS award?
How useful is the current PBS Monitoring and Evaluation (M&E) system for program management, results
reporting and learning? Given that many outcomes are outside the manageable interested of the
program, are impacts being captured appropriately across all aspects of the program? How might the
M&E system be strengthened in the next PBS award?
4. Financial Management
Given the goals of the projects as outlined in project documentation, how appropriately have BFS
central resources been allocated amongst countries, partners and activities, to supplement and
complement Mission funding and to support global technical leadership?
5. Communications, Outreach and Partnerships
Has PBS effectively responded with appropriate and timely communications products and outreach
directed to the appropriate audiences as technical and political issues arise? What could be improved?
Is PBS effectively collaborating and working with other biosafety and biotechnology programs in the
regions where these projects are active (including AATF, ABNE, ICGEB and OFAB communications
programs)
9 
EVALUATION METHODS &
LIMITATIONS
EVALUATION DESIGN
The scope of this review is the entire range of PBS activities over the past five year period including
research and in-country programs but excluding those activities that just started in 2013. As the time
and evaluation budget did not allow for visits to all participating countries or of a review of every
research document produced, a purposive sampling method was used.
Field visits to Indonesia, Uganda and Malawi allowed for interviews with key informants. The three
countries have different characteristics such as length and depth of assistance, and stage of preparedness
and institutional capacity in biosafety regulation. Furthermore, in each country there are specific cultural
and political factors which influence outcomes. Three different cases allow for an assessment of the
impact of outside factors and variables on outcomes and results and enable a comparison of how the
program is being implemented in each country and the effect this has on meeting objectives. Each
country will be summarized as a separate case study, tracking the processes and mechanisms that led to
specific results to understand causality, and from which relevant lessons can be generalized and applied
to the design and management of the project in other countries as recommendations. In addition, global
policy research inputs have been considered and compared with outputs in policy and regulatory
changes with the countries to determine the extent of causality and usefulness.
In the initial planning of the review and for the first of the three field visits, a specialist in Monitoring and
Evaluation was accompanied by a specialist in biotechnology regulatory affairs as the review team. Due
to the inability of the monitoring and evaluation specialist to continue with the review, the second and
third field visits and the overall drafting of the review were completed by the regulatory affairs
specialist, using the template and methods developed by the monitoring and evaluation specialist as
much as possible.
DATA COLLECTION AND ANALYSIS METHODS
DATA COLLECTION
The project’s design is based on assisting country-specific institutions to develop successful and fully
functioning regulatory frameworks and methods for assessing biosafety of products, and implementing
the associated directives with appropriate measures, laws and regulations. As this is a process
evaluation, it focuses mainly on the effectiveness of the processes and the inputs in producing project
success, rather than on the specific outputs or results. Therefore, a large part of the data collection has
involved gathering information to assess the extent to which the different activities of the program and
its management have contributed to achieving the general program objectives and specific country
objectives. To a great extent the answers to the evaluation questions involve subjective opinions,
10
however, by employing the techniques of cross- verification and triangulation between respondents and
with existing secondary data, objective findings are derived. Data was collected for this evaluation
using two methods: document review and key informant semi-structured interviews.
DOCUMENT REVIEW AND SECONDARY SOURCES
The evaluation team conducted a preliminary desk review of relevant project documents and key sector
background material in order to gain a thorough understanding of the specific activities over the 5 year
period of the project, together with the country context in which each program is operating.
Document review is an important source of data on project activities that can be used to triangulate
interview data, derive qualitative information and continually inform the data collection process.
KEY INFORMANT INTERVIEWS
Key informants - identified by the contractor in-country staff in addition to others identified during the
data collection process – were interviewed either face-to-face or by telephone conference. Based on a
review of project documents the evaluation team identified several stakeholder groups that are the 
focus of program activities. These stakeholder groups comprise (i) USAID and other US Government
agencies. donors, (ii) PBS Implementing staff in both the headquarters and the countries visited, (iii)
Government regulatory agencies as well as research institutes and universities acting as agents of
regulators, and (iv) Partners/collaborators, including technology development groups such as
USAID’s Agricultural Biotechnology Support Project (ABSPII), the African Agricultural Technology
Foundation (AATF) and regulatory groups such as the International Centre for Genetic
Engineering and Biotechnology (ICGEB), and the private sector. Attempts were made to interview
representatives of all stakeholder groups in each country visited, including representatives of those who
have received training through PBS. Initially, two additional stakeholder groups were included – policy
makers and media, however discussions during field work with USAID and the implementers were used
to better define the target groups and individuals to interview and there were not enough 
individuals in these two groups to maintain them as separate groups. There is considerable cross￾over in the roles of certain stakeholders and parliamentarians, government officials and members of
media organizations were included in either the collaborators or regulators group based on initial
discussions with the interviewee. The full list of key informants is included in Annex IV, with affiliations
and the stakeholder group in which they were included.
For each such stakeholder group, specific questions were designed that focus on their interest in the
PBS project. The same base sets of questions were asked in all three sample countries visited - see
Annex III for interview instruments. Questions were asked in the same order where possible. In 
Indonesia, key informant interviews were conducted with both evaluation experts present, however
this was not possible in Uganda and Malawi.
Data collection was aided by the use of an Excel workbook to organize and store data culled from
interviews. At the end of the interview day, interview notes were synthesized and entered into the 
data collection spreadsheet. A separate sheet in the excel workbook was maintained for each of the
stakeholder groups identified above. Each of these sheets contained all potential interview questions to
be asked to that stakeholder group. Each interview respondent was listed in one sheet of the excel
workbook to allow recording information from each protocol question from each interviewee.
Confidentiality of respondents was maintained by anonymizing the data (i.e. removing names, titles,
and other information that specifically links a participant with a specific response).
11
This method of organizing and assembling interview information allowed for the identification of
patterns in the responses and common themes that are supported by evidence in the data. To ensure 
that the data are internally valid and reliable, interview data was triangulated between respondents and
between sources such as work plans, performance reporting, research reports, financial reports, or
other secondary data.
Data collected and transcribed into the excel workbook was systematically analyzed to answer the five
evaluation questions. This content analysis process involved review of the data in the excel workbook
for emergent themes. Attempts were made to quantify the prevalence of certain themes in the data and
indicate the frequency that represents the prevalence of any given theme for one or more stakeholder
groups or respondents. The identified and analyzed themes allow for the formulation of findings for 
each of the evaluation questions.
The evaluation report will be structured on the basis of these questions – i.e., the discussion of each
evaluation question, by itself, forms the basis of a complete section, based on analysis of the themes and
response across the range of stakeholders.
METHODOLOGICAL LIMITATIONS
PARTICIPANT SELECTION AND BIAS
Key informant interviewees were selected non-randomly based on availability and engagement with
program activities and as such, there were no comparison groups. A non-random selection is standard
for qualitative “purposive” evaluations. As there is a small universe of stakeholders in each country, it
was not be possible to randomly select interviewees. Attempts were made to interview representatives
of every organization that was a recipient or object of the PBS project in the three sample 
countries, as well as with regional organizations whose own interests and work coincide with the
donors. However as most interviewed beneficiaries are direct participants in the project and were
recommended by either USAID or the contractor, it can be expected that they will say positive things 
about the project.
Where possible, to minimize bias informants were sought who represent “public stakeholders” with
different ‘causal distances’ from the activities to qualitatively triangulate/check information from
beneficiary organizations and who cannot be perceived as “friendly” or biased stakeholders but who
have sufficient knowledge to comment.
RECALL BIAS
To minimize recall bias, the interview questions were designed to draw upon specific knowledge or
viewpoints that the organization or the individual acquired from direct association with the project. As
the project is currently actively being implemented recall issues should be minimal.
INTERVIEWEE RESPONSE BIAS
All participants come with biases. An attempt was made to organize a sample of organizations with
different perspectives and playing different roles to provide different viewpoints. The degree to which
there is agreement/disagreement in viewpoints is indicated and assertions are triangulated with other
data sources and with other stakeholder groups.
12
Undoubtedly a larger problem, difficult to assess, is that of deliberately inaccurate responses because
the respondent wants to keep a good relationship with the implementer and donor. This shortcoming 
will be mitigated through triangulation of what was said by different organizations, comparison with
known facts, and the interviewer’s perceptions of points of view.
ABILITY TO GENERALIZE FROM THE DATA
The method of organizing the data by question and respondent through an excel workbook allowing for
coding for recurrent themes, opinion agreements and disagreements allow the identification of areas
where the data clearly justifies generalizations. Where possible the number of respondents is quantified
although the quantification is subject to interpretation since often the respondents’ answers to “open￾ended questions” are imprecise and opinion-related, and it is difficult to get explicit answers or
examples. Hence the development of recurring themes and concurrence across different types of
respondents provides the best source of data. USAID Evaluation Policy is followed, placing emphasis on
anchoring conclusions in ‘findings of fact,’ and not in individual interviewee or evaluator opinions or
second-hand reports. Clear, reproducible evidence for findings and conclusions, supported by specific
examples and quotes, have been used to tie findings to conclusions, and from there to
recommendations.
13
FINDINGS, CONCLUSIONS &
RECOMMENDATIONS
As described in the methodology, the findings are arranged in the order of the questions which form the
basis of the evaluation. Separately, a summary of the situation, progress and impacts in each of the three
countries visited are included as a set of cases, which are then used together with the findings as the 
basis for conclusions.
FINDINGS
1. PBS Country Programs Staff and Support from International Technical Consultants
• Are the roles and responsibilities of the in-country country lead appropriate compared with international
short-term consultants? Should the mix be adjusted to support program expansion and sustainability of
provider support in the next PBS award?
Interview responses from the implementers are in agreement that the role of the country coordinator is
key to the operations of PBS, with the country staff supplying both the direct implementation of PBS
programs and providing critical insight into the political situation in their respective countries. USAID
and other US government stakeholders support the role that the country staff play, with comments
noting the connections that country staff have with policy makers and other stakeholders who have
influence.
There is also no disagreement about the role played by external consultants, who are seen as providing
valuable expertise and a focal point for activities. At the same time, there is a broad feeling that the
range of consultants could be increased, with specific note that consultants from within their region
would be appreciated by stakeholders. High marks were consistently given for the level of expertise of
the outside consultants, but the only ones named were those with long term staff affiliation with PBS and
not short term experts. Criticism was voiced by several interviewees of the ability of consultants to
present information and examples of how developing countries achieve goals, with more emphasis
desired on non-US models and examples. Other individuals commented that the external consultants
were all from the US and a broader source would be appreciated.
• Sub-questions to consider: Is the skill set of the in-country country leads appropriate to achieve PBS
objectives? In what areas of their essential job functions have country leads been effective and where
have they struggled? Do stakeholders feel country leads have enough/the right technical expertise, as
compared to skills required to initiate and keep activities moving forward, and to communicate, advocate
and build consensus among national stakeholders? Has the support provided by international consultants
been effective in achieving the objectives of their STTA and overall PBS country objectives?
Implementers all agree that a certain level of technical expertise is required (a science or agriculture
background), but that biosafety expertise per se is not a pre-requisite given that all country staff receive
training from PBS and experts can be called on when needed. However, it appears that to the majority
of country-level collaborator and regulatory contacts, the level of technical expertise exhibited by the
14
country coordinators is adequate or above what is required – perhaps reflecting the lower level of
experience in these interviewees. Some less favorable comments were noted in other areas often
relating to specific failings with certain Ministries in certain countries i). One theme noted across the 
three countries visited was a lack of the requisite skills for outreach and communications. This
changing focus of the type of role played by the country level staff was also mentioned by 
implementers and a stronger focus on this aspect – either through staff training or even bringing in
new supporting staff - would be an option to consider. There was almost unanimous support for the 
expertise and work-rate of the coordinators which could be a result of selection bias, but does seem
to reflect a genuine appreciation of their activities and achievements.
Consultant support was highly rated, with the previously noted caveat of some interviewees that
consultants were US-centric. Where specific outcomes were accomplished through PBS activities
(approval of field trial applications, legal reviews, development/adoption of guidelines and SOPs etc.) the
input of external consultants was acknowledged by the stakeholders involved as being a key factor.
2. Policy Research
• Has the policy research carried out by PBS been relevant to the technical support and capacity
development activities being conducted by PBS? Provide specific examples of where PBS policy research
has made a direct and measurable contribution to the achievement of PBS country-level objectives.
There is a general lack of awareness of PBS policy research among stakeholders, unless they are already
familiar with and working in the area of the research. There is more familiarity with IFPRI as a research
organization, which suggests that some of this is a confusion of the role of PBS within IFPRI with few
people appreciating that related research published by IFPRI is a PBS activity. However, a bigger issue
seems to be the level at which policy research is published/presented makes it less relevant to the actual
stakeholders involved in the country level activities. A few specific examples of PBS policy research 
were cited, most notably the socio-economic study and the environmental risk analysis done for
eggplant. On the other hand, when asked how policy research has proven useful to their organization
and to meeting PBS objectives, respondents referred to exposure to the US and Brazilian regulatory
system and OECD guidelines (which may not have come from research but from workshops), in addition
to socio-economic studies.
The contribution of the policy research is hard to measure directly, given that its impact occurs in
combination with other factors, however it was reported by PBS implementers that research on labeling
thresholds had had a noted impact in Vietnam. In addition, the work on socio-economic benefits was
recognized by a number of stakeholders as being useful in getting policy makers to understand the role
that biotechnology can play in increased agricultural productivity. In Indonesia, the development of a
socio-economic assessment as part of the national regulatory system was reported by three interviewees
as being helped by research from PBS detailing how this was done elsewhere.
In Indonesia, the most frequently noted contribution to the technical committee’s work were the
workshop which led to establishing the socio-economic team and the workshops on technical
assessment from which committees drafted their regulations (which presumably included the research
that was used as background information). For members of the biosafety commission a better
understanding of the issues surrounding biosafety regulation influenced them in understanding and
acceptance of findings of the technical committees. Probing for how workshops led to actions, the
15
technical committees acknowledged that they were able to prepare their own guidelines based on the
workshop inputs, but there was little differentiation from policy studies. Respondents noted that the
examples from other countries cited in workshops are useful, but it is was not stated whether anyone
read the publications.
In Uganda there is awareness and use of the research by some of the collaborators. However some
respondents understood that the research was focused on trade issues and was therefore not relevant
at the present time to Ugandan researchers who are involved in early-stage development of modified
crops. In Malawi, the interest was in socio-economic benefits, but there was a call for specific data on
benefits within Malawi directly. There were few examples of direct impact of the research studies in
these countries, but it was also stated that they expected these to be more relevant as modified
products moved further along the development pathway.
A common theme in responses was that the research was not directly relevant to the respondents own
country and that there was an (unmet) demand for research done within the country, especially on
potential benefits at the country level for specific crops. PBS is addressing this to some extent, with
studies under way in Indonesia on policy and legal issues, but this issue of showing relevance of policy
research at the country level seems to be an important aspect.
• Are the research products developed by the PBS team of high quality and contribute to a literature base
that is being utilized by partners and stakeholders?
Where stakeholders were aware of the research being done by IFPRI, there was almost unanimous
agreement that papers were of high quality. As noted above, there is little awareness that the research
is part of the PBS program and the regional collaborators in particular look to IFPRI research as being
the major source of academic information. More common than awareness of the research publications,
however, was exposure to the research through shorter policy briefs and training workshops where
participants were presented with data from the research rather than papers themselves. These types of
exposure were regarded as useful and shorter documents – such as the policy briefs – were used by
collaborators and government regulators for their own use.
One issue raised by implementers was the conflict between meeting IPFRI requirements for high-level
publications in peer-reviewed journals and the needs of PBS collaborators and government regulators for
simpler policy advice and also more locally-relevant studies. Although the Director noted that PBS is
used as a positive example within IFPRI with regard to “policy impact”, there is still an issue with
addressing country-level demands for research which may not be compatible with IFPRI publication
standards.
3. Project Management
• How effectively have the PBS Director and Africa and Asia regional program leads supported learning
and ensured quality across countries? How could these roles be strengthened in the next PBS award?
Interviews with the Director and regional coordinators detailed the management structure being
followed in PBS – with a combined role in developing the workplans, but more reliance on the regional
coordinators for day to day implementation. Quality control for individual activities is provided by the
external experts, once the central management team (Director and regional coordinators) have
approved these activities in the workplan. As the external experts operate across the program and are
16
responsible for delivery of the individual activities, this should ensure that standards are maintained
across the countries. The role of the country coordinators in providing the links to stakeholders and
determining the political situation within each country was noted as a positive aspect by the
implementers. Attempts are being made to develop more independence in the country programs and
more direct connections between them, with direct communication between the country coordinators
in West Africa being encouraged by the Director. However, a combination of factors such as the level
of experience of the in-country teams and budgetary constraints reducing the scope for independent
actions, mean that most budgetary decisions affecting programs are taken at a higher level. Although this
does support the quality control aspects, this will make it difficult to expand the program into more
countries without an expansion of the core staff.
Very few respondents were able to assess the effectiveness of the PBS Director and the Regional
Program Leads as most reported little interaction with this level of PBS management. Those that did
report interaction gave high marks to the Regional Program Leads and the communications advisor,
noting that they are easy to communicate with and respond quickly to emails. Positive feedback cited
recent communications and outreach oversight and quality control, including assistance on developing
focused stakeholder targeting, and supervision of newly developed outreach products and methods of
delivery. There was more familiarity with the senior PBS staff in their role as technical experts, which
was addressed in relation to question 1 above.
• How useful is the current PBS monitoring and evaluation system for program management, results
reporting and learning? Given that many outcomes are outside the manageable interested of the
program, are impacts being captured appropriately across all aspects of the program? How might the
M&E system be strengthened in the next PBS award?
The USAID Feed The Future criteria provide a very basic set of M&E requirements that are not
considered by the implementing staff to be useful in assessing the impact of the PBS Program. The
reports made available to the evaluation team include the numbers of people attending training events,
(this is higher than the indicator, which call for the number of people trained due to a small number of
people who attend more than one training event, but this is estimated at around 5% by the program
staff). However, this number does not provide any measure of impact and there needs to be some way
to capture the progress countries have made in developing a cadre of scientists who have had
experience in the regulatory arena. In both Indonesia and Uganda, the various technical committees
are now operating to review and make decisions on applications submitted to them. Ugandan scientists
have been used as resources for training workshops both in Uganda and in other countries in the region. 
Even in Malawi, it was noted that although the National Biosafety Regulation Committee has only
reviewed two applications for confined field trials, there was already a noted increase in confidence
among the members about their role. This capacity building impact does not appear to be being
captured in the progress reports submitted to USAID and it is difficult to provide quantifiable measures.
Measurement of the operational effectiveness of the government committees (time between receipt of
applications to approval of field trials, for example) would be possible, but will require the monitoring of
agencies not directly under the authority of PBS. Reference was made to the previous monitoring
framework which PBS operated under as having broader indicators (number and types of government
organizations strengthened) and there would be value in re-assessing whether the previous indicators
were more relevant.
17
Project monitoring currently rests on conference calls between different levels of PBS staff at set
periodic intervals, and country lead reports on activities and semi-annual country reporting. Reporting
to USAID Washington is made under two indicators: 4.5.2(7): Number of individuals who have received
USG supported short-term agricultural sector productivity or food security training; and 4.5.1(24): Numbers of
Policies/Regulations/Administrative Procedures in each of the following stages of development as a result of USG
assistance in each case: (Stage 1/2/3/4/5) in addition to a narrative which is more qualitative and focusses
more on activities than outcomes. The USAID mission in Uganda have an on-line reporting system for
monitoring, but other USAID missions rely on reports from Washington and presence at PBS events to
provide oversight. In Malawi, the US Embassy relies specifically on such presence and prefers not to
receive detailed reports. Although this is adequate in the sense that the directly responsible project
officers are satisfied, there is little scope in this system to quantify impacts across the project and to use
the impacts as an illustration of the value of the project to other potential investors.
4. Financial Management
• Given the goals of the projects as outlined in project documentation, how appropriately have BFS
central resources been allocated amongst countries, partners and activities, to supplement and
complement Mission funding and to support global technical leadership?
Reviewing the financial documents supplied by PBS showed a complex funding situation requiring quite
some effort to manage successfully. Interviews with the Director and regional coordinators confirmed
the complicated nature of the financial situation which arises from a number of different USAID donors
(Missions) being combined into a single line grant to IFPRI, but these funds often requiring managing and
reporting separately. As mission funds are rarely more than single year grants, the challenge has been
to maintain a constant presence in countries where continuity is considered to be important. Balancing
core funds to be able to support all of the country programs, regardless of whether mission funds are
available, is a key role of the Director.
The countries where PBS supports activities from core funds are included for a number of reasons,
although it was explained that there are no fixed criteria. Continuity of presence is a key factor, and this
is supported by the fact that a number of stakeholders credited the impact of PBS to this continuity. In
fact, three regional collaborators commented that it is the long term presence of PBS and the contacts
that PBS had built up in certain countries that are among the most useful aspects that PBS brought to
the collaboration. Other factors considered in funding decisions are the political climate in a country,
for example in Malawi, PBS considered that the situation was positive towards biotechnology and that
the demand for external regulatory help would have support from within the country.
Some issues with availability of funds has caused problems in recent years. Delays in confirming funding
from USAID have meant delays in disbursements from IFPRI to country operations as IFPRI
management are cautious in committing funds to PBS operations until they receive the confirmation
from USAID. In addition, retaining a pipeline of funds in the core to cover potential delays in availability
of new funding results in an apparent under-spending by the project and risks reduction in funding from
USAID. Better financial management by USAID and IFPRI would improve the running of PBS by 
preventing uncertainty over the availability of funds.
18
5. Communications, Outreach and Partnerships
• Has PBS effectively responded with appropriate and timely communications products and outreach
directed to the appropriate audiences as technical and political issues arise? What could be improved?
The PBS program originally concentrated communications activities on high level information sharing
and technical training of a targeted group of decision-makers most related to implementation of 
regulatory policy, namely the various biosafety commissions and the technical committees who
support them and the policy-makers and influencers who set the agenda for biotechnology. Outreach
to technical and academic collaborators appears to have been accomplished through small scale training
activities. In recognition of the fact that communications and outreach is important to the policy 
development process, and to insure that a range of stakeholders understand the policies and policy 
decisions, the program has expanded its communications work to include an exapanded group of 
stakeholders who figure prominently in the both policy creation and implementation. The approach is 
more strategic-- to better inform the policy process, and to anticipate issues which may impact policy 
development and implementation. he program emphasis has, as a result, evolved to focus on building
partnerships with a wider group of collaborators and providing information to sensitize stakeholder
groups who can influence policy-makers to make biosafety a policy priority and insure that evidence￾based biosafety policy is enacted and implemented. This represents a much broader outreach and
communications program, which has been recognized by the hiring of a communications specialist as a 
long-term consultant, who assists in guiding this work, in addition to a number of other roles. 
With the small resources that PBS has in each country it is not possible to operate alone - even if this
were considered desirable - and so in each country PBS has either developed or associated itself with a
broad consortium of government agencies, academic and government scientists and industry
organizations to perform the broad scale sensitization that is required. The role of PBS in all of these 
efforts or organizations is to provide a conduit for science-based, accurate information of biosafety. In 
Indonesia, communications and messaging is currently being done by the Society for Indonesian 
Agricultural Biotechnology (SIAB). The group was established with some guidance and ongoing 
technical support from PBS. In Uganda, the Uganda Biotechnology and Biosafety Consortium (UBBC)
and the Open Forum for Agricultural Biotechnology (OFAB) (a consortium established with AATF and
which operates in a number of countries in Africa) are the main vehicles for this and PBS guidance was 
useful in helping these groups organize and develop, especially with respect to building their capacity to 
respond to regulatory issues.. In Malawi, PBS works with a number of local organizations including
working with an association of agricultural journalists to insure that they receive evidence-based 
information about regulatory issues and practices. This certainly has an effect in multiplying the reach
that PBS can have, with a far greater number of people able to be contacted than could be achieved with
PBS funding alone. At the same time, however, there is a potential that the message is harder to control
and the goals and targets of the partners may not be in alignment with PBS goals. In many cases, the 
partners are as interested in information and education about biotechnology itself and the message
about the biosafety aspects of the regulation of biotechnology products may be diluted or missed
altogether.
Furthermore, there is no consensus among the various respondents on who would be prime target
groups for PBS as the various collaborators all have slightly different goals. While all respondents
seemed aware of outreach and communication as a tactic, their targets of choice for PBS varied
depending on where they themselves stand. Government regulators express interest in increasing focus
on messaging on policy makers in government, believing that knowledge of biosafety is still restricted to
19
a very limited policy group. The non-governmental groups want to work with a wide variety of target
groups including university and secondary school students, farmer groups that may benefit as users of
agricultural biotechnology products, and also elected and appointed officials - diverse groups and
widespread across each country. The private sector often have their own programs to target farm
groups and farmers and would prefer that PBS should take advantage of its close ties with key
government officials, drawing on the stature that experts bring to bear on policy makers, rather than
with farmers whom industry can reach and with whom PBS has no connections. Hence, the issue of
what is the appropriate audience or set of target groups for the future is not resolved and currently
efforts appear to be going in many directions.
Another issue is the sustainability of these organizations should the program cease to operate in these
countries. Transitioning to expanded and diverse funding sources and development of a mature management 
structure is a goal, but progress in this respect is dependent on the development of local skill sets for 
fund raising and management. These are in process. This will result in less guidance provided by PBS and 
therefore less quality control with respect to the regulatory messages and information, especially as 
other donors may have different goals.
It is not known what, if any, of the past and current efforts have been effective because there are no
criteria to apply or methods for assessing impact (other than the “body count” of participants at
outreach activities). Several respondents mentioned that the communications training has helped them
to interact in public with journalists and to communicate with lay public, but at the same time, several
others stated that PBS needs to bring in competent local PR experts to guide the process. Impact has
been assessed through results in other areas, such as passing bills and/or regulations and approval of field
trials, although the role of broad outreach in these outcomes is hard to estimate. There have been
examples of publicly stated changes of opinion of influential individuals – such as the public support of
one religious group in Indonesia and the change of position (from negative to positive) of the major
consumer activist in Malawi, but it was noted that much effort is needed just to maintain current
positions and prevent “backsliding” among prominent stakeholders and this may not be captured in
assessment of impact. A more explicit monitoring of public awareness and opinion, through direct
surveys is an option, however these are expensive and require repeating to determine the impact of PBS
activities. Collecting and analyzing media reports would address the impact that the program is having on
the press and could be used as a proxy, especially if there is also monitoring of feedback on websites etc.
Internally, PBS has a relatively large number of front-line staff (at least one in each country) and works
with as number of different organizations both in individual countries as well as regionally and – in some
cases – globally. Ensuring a coherent message across these is a major effort and responding to issues as
they arise requires a number of capabilities. For the most part, response is through the expert
consultants who rely on the rapid communications that they are able to achieve using electronic media.
It is an aim stated by a majority of the PBS implementers (country staff and experts) that PBS should try
to get ahead of the issues rather than reacting to issues as they arise.
PBS has not tried to garner a high profile for itself (a position noted by several US government
interviewees) and works more with and through partners with regard to communications and outreach.
Sets of frequently asked biosafety questions have been prepared by some partnering organizations,
sometimes using PBS documents where applicable. To the extent that a common set of answers can be 
useful to show a united front, this is a valid aim and could be expanded more explicitly.
20
• Is PBS effectively collaborating and working with other biosafety and biotechnology programs in the
regions where these projects are active (including AATF, ABNE, ICGEB and OFAB communications
programs).
Direct collaboration with regional organizations is an expanding theme, but some non-formal
collaboration has been going on for a longer period and has been the subject of a more recent, formalized 
effort since 2011 which marked the start of an annual biosafety service providers meeting) and more 
recently evidenced by closer joint programs in countries where numerous potential collaborators exist 
(such as the joint workshop between PBS and ICGEB which took place in March 2014). Close links with
ISAAA have been established over a relatively long time and links with AATF at the country level
through OFAB have also been a feature. The noted issues with such collaboration is the somewhat
different goals and mandates of these organizations, with some being explicit promoters and/or
developers of agricultural biotechnology products (AATF and ISAAA) and ICGEB targeted towards
training more than policy.
Interviewees from all collaborating organizations reported benefit from their collaboration with PBS,
often with respect to visibility within a country and access to the relevant stakeholders. The fact that
PBS has been on the ground in many of their target countries for a number of years is seen as a major
positive factor by the collaborators.
21
COUNTRY CASE STUDIES
Although the main themes noted in the interviews have been collected and reported across all of the
countries in the sections above, the actual situation in each of the three countries visited was quite
different. The individual situation reports below illustrate the broad range - in terms of the status of
biosafety regulations - in the countries where PBS operates. These snapshots are included to provide
some country specific analysis.
Case 1- Indonesia
Snapshot of Situation
• In Indonesia, the USAID Mission has provided significant funding for the past 3 years in tandem
with a re-launch of the program which had fallen dormant due to negative publicity over BT￾cotton and low perceived interest on the part of the government to proceed.
• In 2010 the Government of Indonesia appointed the Biosafety Commission for Genetically
Engineered Products (BCGEP) based on a Presidential Decree (Government regulation 3910)
whose role and authority is to advise government on importation and marketing of GM
commodities including locally developed GM products. The thrust of PBS program activities has
been to support the knowledge of biosafety and to advise in preparation of regulations for
reviews and approvals by three technical committees (food, feed and environmental safety) that
advise the Commission. Technical workshops have provided support to develop such regulations
for technical committee review and advice to the Commission.
• There are strong undercurrents of negative feeling about international biotech industries -
mainly US companies - as well as worries about food safety on the part of a small part of the
population. This is manifested in negative publicity by NGOs. The Ministry of Environment has
conducted its own outreach campaign on the guidelines for environmental risk assessment
independent of PBS and they worry about the diverse issues surrounding GM crop cultivation.
• In mid-2013 the BCGEP mandate expired and the Ministry of Environment, the responsible
agency, has not appointed (or re-appointed) a Commission that meets with the agreement of
President’s office. In addition, biotechnology and biosafety is a contentious issue, especially as it 
relates to the role of multi-national companies and especially sensitive in an election year. 
Thirteen food products have been approved for import using newly developed regulations, while
regulations for feed regulations are awaiting ratification by the new Commission when it is
appointed. Since the BCGEP has no formal authority at this time, no further formal approvals
may be made by the Commissioners until the impasse over re-appointing the Commission is
resolved, although technical committees continue to function. In the case of the one product 
developed, a drought-tolerant sugar cane, a dossier has been prepared, but the product cannot
be approved in the absence of a functional Commission. In the meantime PBS is looking to
develop an outreach strategy working with a locally based coalition to share information about 
the regulatory process for the GM sugarcane and other GM products.
• USAID Indonesia has preferred to keep a low profile on biotechnology because of past and
potential future bad publicity. The new USAID Country Strategy will eliminate agriculture as an
22
intervention area and mission support for PBS in the future is in question as the program objectives 
may not be consistent with the Mission strategy.
Analysis
• The signal for programming and for USAID interest was the renewed interest in biotech
through the Presidential Decree establishing the BCGEP. However, in retrospect, the
foundation for the Commission was not sufficiently strong to keep it free of political
manipulation and electoral pressures. This demonstrates the importance of a clear and firm
institutional basis for regulation of biosafety so that progress is not dependent on waxing or
waning political will or driven by outside factors. In Indonesia, the stakeholder interest in
biotech does not appear to be strong enough to maintain the momentum in the face of negative
pressure.
• PBS must determine whether it can, or should, attempt to develop more stakeholder support for 
biosafety and continue to support the government of Indonesia in the next project period and
without, potentially, the active collaboration of a Mission program after the next year. The 
program success to date has been in the technical arena dependent on outside experts. As key 
products near commercial release, one might expect the need for more effort in outreach to 
materialize, while needing to maintain a continued level of technical support. The support of the 
country mission would be an important asset for policy outreach, not only from a funding 
perspective but also in enabling access to decision-makers. Without mission support, biosafety 
policy outreach efforts might be more difficult.
Case 2 – Uganda
Snapshot of Situation
• In Uganda the USAID Mission has provided significant support since 2004 through an Associate
Award which was closed in 2013 and a new Associate Award which will run until 2017. During
this period, the local development of expertise in biotechnology research has progressed and
field trials of a number of GM crops have been carried out. PBS was instrumental in developing
the capacity of Ugandan regulatory authorities to review applications, approve and manage these
field trials.
• The system for confined field trials is functional and a number of locally developed and external
GM events are currently being developed towards commercial deployment. PBS played a large
role in developing SOPs for confined field trials and training staff to manage and inspect the
trials. PBS has contributed to the development of a significant Ugandan capacity in
biotechnology and biosafety, such that Ugandan scientists are now involved as resource for
training in the region as well as in Uganda.
• A bill has been submitted to Parliament to cover the commercial approval of GM crops. This Bill
was introduced by the government, but is currently awaiting a second reading and proponents 
of the bill are concerned that in the current climate it will not pass and are anticipating that a 
second reading will be delayed.. Although it is a government-sponsored Bill, outside interests
and coalitions are providing most of the technical and outreach backstopping related to passage 
of the Bill. With an election approaching, the Bill may be contentious; as a result, momentum and 
support for passing the Bill has stalled.
• PBS has been focused on insuring that those in Parliament have access to technically sound 
information to support an informed decision making process with respect to passage of this Bill. 
23
Visits to the confined field trials and international study tours to visit commercial plantings of
GM crops have been effective, via a “seeing is believing” approach, to expose key policymakers 
to regulatory progess in both Uganda and other countries. This is regarded as successful, but only
limited numbers of people can be exposed.
• Stakeholders remain confident that the Bill will be passed, but there is currently no timetable.
Attention appears to have been so focused on getting the Bill passed that there is a worry
among some stakeholders that little consideration has been given to what is needed to 
implement the Bill. There needs to be planning now for the next stage of support to get the 
Bill fully implemented and operational.
Analysis
• PBS has been successful in gaining support of individual MPS though targeted activities, but this
support does not seem sufficient to ensure passage of the Bill. Although this is a Ministry of -
Planning sponsored Bill, support for the bill among some government Ministries is not consistent 
within and between ministries. 
• PBS should support those working on the Bill should by providing analytical direction regarding the 
various stakeholders who can impact the Bill to insure that they have accurate information about the 
purpose of the Bill, its contents, and the implications for agriculture productivity and safe deployment 
of new GM varieties in Uganda. 
• Consideration is needed on what PBS role will be once the Bill is passed.
Case 3 – Malawi
Snapshot of Situation
• In Malawi, PBS has no current support from the USAID Mission and is operating on core funding, 
but is in receipt of funds from the US Embassy in Lilongwe to do outreach related to the 
regulatory work This situation is likely to change in the near future as AATF are planning to
apply for field trials of an insect-resistant cowpea event and the USAID Mission is willing to
support this under the Feed the Future program.
• PBS was instrumental in developing implementing regulations under the original Biosafety Act
which was passed in 2002, but had not been implemented. Technical and financial support was
provided by PBS to put in place Biosafety Regulations and guidelines under which field trails can
take place.
• Confined field trials with insect resistant cotton are under way in Malawi. This is the second
season and funding for the trials (in three locations) is being provided by Monsanto as the 
technology developer with Bunda College. PBS has provided support in the form of 
infrastructure and training of regulatory staff, inspectors and the National Biosafety Committee. 
Progress on capacity development of the National Biosafety Regulation Committee (NBRC) has
been made – with the approval of the second round of cotton trials much more straight forward
than the first round of trials and the Biosafety Registrar speaking of increased confidence within
the NBRC as to their role.
• The confined field trials have provided an opportunity to sensitize a wide range of stakeholders
in addition to those who were given the opportunity of international visits. Lack of interest by
the Ministry of Agriculture has meant that the trials are being sponsored by Bunda College of
Agriculture; support from the Department of Agricultural Research (under the Ministry of
Agriculture) is lukewarm although some research sites have been used for trial work. In contrast,
24
there is enthusiastic support from the Ministry of Environment and Climate Change, which
houses the Biosafety Registrar.
Analysis
• The “seeing is believing” visits to the confined field trials have increased interest in the
technology and the challenge is to maintain this momentum. At present, there is only one
potential new product on the horizon and insufficient local resources and projects for public 
sector GM research
• PBS needs to consolidate the gains (a regulatory system that has approved two rounds of 
confined field trials) by encouraging other developers to work in Malawi and thus increase the
experience of the local regulatory system.
CONCLUSIONS AND RECOMMENDATIONS
1. PBS Country Programs Staff and Support from International Technical Consultants
To date, the technical focus of PBS is well served by the country staff and experts, however the
demands are now moving from primarily technical support to advocacy and there are different skills
required. The value of having external experts from a wider source than just the United States is
tempered by the need to have well-qualified experts and to have them involved in PBS activities over a
long enough period to be familiar with and accepted by local stakeholders. Development of local
expertise, through training and joint research studies, can provide a resource for the region as well just
the country in which they are working.
2. Policy Research
The PBS Policy team, working within and through IFPRI, has played an important role in producing a body
of work recognized globally as being critical to the understanding of biosafety policy and regulation.
However, there seems to be little recognition of this within the country stakeholders and there appears
to be some conflict between IFPRI demands for publications in peer-reviewed journals and products
which can be used to help countries develop practical regulatory systems. This is recognized by PBS
management and attempts are being made to use the research more widely and also to conduct more
country-level studies. The challenge for PBS is to work within IFPRI to allow for recognition of more
practical studies for implementation as a valid output. With regard to recognition of the role of PBS
research by local collaborators, it is less important that these studies are identified as IFPRI publications
than that PBS is seen as a source for documents relevant to the local policy and planning needs. The PBS
summaries and short briefs are where there needs to be “brand recognition” as this is where most of 
the government regulators and policy makers will be getting their information.
3. Project Management
Annual work plans and semi -annual reports appear deficient in many respects for proper management,
results reporting and learning. The main problems include:
• Exclusive focus on inputs (and outputs), rather than on outcomes and results. The work plans
do not indicate what outcomes (except USAID standard measures which are of little value) are
25
expected from a given input (activity) and provide no adequate measure of whether the activity
resulted in expected and desired outcomes (which are often not defined). Hence, the planning
and reporting do not give a clear picture of what objectives and outcomes were expected or
achieved in the given period.
• There is considerable confusion over what a milestone is, what is an input and what is an output.
These should all be reviewed and refined using USAID standard definitions with critical
milestones clearly defined, and differentiated from inputs.
• There is a need to present realistic time frames for achievement of critical milestones, outputs,
and outcomes, so that program progress can be measured and monitored for decision-making,
both from the standpoint of financial cost benefit analysis and allocation of funds against
expected progress, but especially for Missions that need clarity on what they have paid for and
what is being achieved from year to year leading to desired outcomes.
• Project level intermediate outcomes and long term outcomes and results need to be identified,
with clearly defined indicators for measurement (based on USAID indicator development
methodology) so that achievements can be measured at the project level for better reporting,
and learning from experience. USAID Feed the Future standard indicators do not capture
outcomes and results in sufficient detail to know whether the inputs are achieving the desired
impacts; as for one of the two indicators used, the number of people trained, the raw number of
attendees at a seminar or a lecture is meaningless for project management and planning purposes 
(these are outputs) because it does not reveal what the attendees did with the 
information. A similar problem is found with use of indicators such as number of regulations
produced or reviewed (which are inputs and outputs) that tell nothing about results. Carefully
refined indicators and means of measurement are needed to capture project impacts and goal
achievement.
• Previous project indicators for capacity building within government agencies should be reviewed
for potential application. The time for an application to progress through each step of the
regulatory process can be used as a measurement of the effectiveness of the regulatory system,
as can the responsiveness of the various committees to applicants.
4. Financial Management
PBS has been successful to date in juggling the demands on the core budget to provide services and
support in a number of countries which have USAID mission support as well as operating in some key
countries which don’t have such support. Based on interviews and assessment of country situations,
progress has been made in all countries in which PBS operates and there was no suggestion that lack of
funds was a key factor in those cases where political changes have become a road-block to further
progress. It is not possible to speculate on whether PBS would have been more successful by focusing
only on countries where there were mission funds as there are examples of success both with and
without mission funds. However, the interest of the country mission is valuable and in the case of
Indonesia and Uganda has provided more than simply funding such that - for the most part PBS – should
concentrate on those countries where there is USAID country mission support.
One challenge for PBS lies in how much they can expand into new countries, given the demands on the
core budget, recognizing that for each country funding is not only required for country level staff, but
also to support the country activities with external experts and central PBS activities. There are many
26
demands on the core budget and hard decisions need to be taken about when to stop funding activities
in a country or even to turn down mission funds to begin working in a new country if this would stretch
core funds too thinly. There does not appear to be any formal benchmarking of different countries with
respect to the return on investment of core funds and this would be very difficult, given the very
different situations (and funding levels) in the countries where PBS operates. However, some
understanding of what is needed from the core budget to support a useful country program - over and
above the costs for the country team itself is – is required to determine how many countries PBS can
productively support from the existing core budget. There is the potential to reduce effectiveness over
the whole program by stretching core staff and external consultants too far.
5. Communications, Outreach and Partnerships
A number of issues concerning targeting, the role of communications and outreach and the
professionalization of advocacy and PR are pending. Answers to the following will be helpful in defining
and implementing a strategy:
• For whom do the in-country consortia (SIAB, UBBC, OFAB etc.) speak? Is it the mouthpiece of
PBS? The tighter the connection, the more the group becomes a proxy for the US government
and PBS, which could backfire if PBS loses the trust of local stakeholders.
• Is there a long-range plan for engagement and institutional support? PBS is providing partial 
support to a few of these consortia in addition to funding travel costs of speakers and venues.
USAID has a long history of supporting and financing NGOs that eventually fail when financial
support is withdrawn. A clear plan is needed to determine the relationship in both short and
long term financial terms and otherwise. 
• What is the capacity of the consortia to do public relations and policy work? Neither the 
members nor PBS country staff are advocacy or PR professionals nor are they receiving
professional advice at present.
• What is the appropriate target group(s) for the consortia to focus on, what clear and
measureable results are expected from their outreach, and how will they be measured? To date
PBS considers “results” the number of persons attending a lecture. It is not clear what result or
outcome has been defined as the goal for speaking to high school students, or a farmers groups,
for example.
• Who should focus on other target groups, and is PBS involvement needed or desirable? Is policy
advocacy an appropriate direct PBS role, using respected policy experts both international 
and national?
• Are professional communications services needed to complement and support PBS local staff
and volunteers?
• What results are desired, and what indicators will measure the impact of different inputs on
different stakeholders?
CONCLUDING REMARKS
The country visits have confirmed this reviewer’s previous experiences of biosafety regulation in other
developing countries in that there is often a lack of real commitment by governments to developing a
regulatory system for products of biotechnology. The fact that there is no government budget for the
various committees, in certain cases, and little government support for a bill originating with the 
27
ministry of Agriculture in Uganda, indicates that biosafety regulations are still not seen as critical to the 
functioning of governments. In this atmosphere, international commitments to biosafety regulation
(chiefly the Cartagena Protocol) become something to address at the minimum level required and only
where external funds can be obtained. The fact that it is largely overseas agencies which are involved in
both developing biotechnology products and supporting the implementation of a regulatory system
compounds the erroneous perception that these things are being pushed onto a country, rather than
addressing an internal need or being a positive factor in the country’s development.
Within this difficult political environment, PBS has been an effective force in increasing the technical
capacity of countries to handle the regulatory aspects of agricultural biotechnology up to the stage of
confined field trials. In all three of the countries visited, PBS can take considerable credit for the fact
that there is a system for the application, review, approval and monitoring of confined field trials of
transgenic plants, together with some form of regulations and standard operating procedures to ensure
that these are carried out with minimal risk to the environment and human or animal health. There is
even noted progress in addressing the technical aspects of risk assessments for approval of foods and
feeds and environmental release in both Indonesia and Uganda. However, taking this to the next step
– getting products approved for commercial cultivation – has not been achieved and remains elusive in
many countries across the globe. This failure is not a criticism of PBS as there are still relatively few
countries which have approved transgenic plants for cultivation (and even some of these cannot be
described as having a functional regulatory system as there is just a single modified crop being grown
with approval being given as a one-off dispensation). It is simply a recognition that more than technical
expertise and support are needed to develop and implement a functional regulatory system to the
stage of approving GM products for commercial cultivation. In order to achieve such commercial level
approval, there needs to be a broader political commitment, supporting the passage of enabling law(s)
and/or the establishment of a permanent government system for implementing cultivation approvals. It
is the opinion of this Reviewer that promoting this broader commitment will require a different 
strategy to that taken by PBS in the past.
In achieving the success that it has in the technical areas, PBS has focused quite narrowly on the relatively
few people who need to be involved in the process of implementing confined field trials and addressing
the technical aspects of risk assessment. This has been successful because the governments in the
selected countries were supportive of the technology – as long as there was relatively little political cost.
While the funding was from external sources and there was little negative public reaction, the activities
in implementing a confined field trial system had little downside to government officials or politicians.
However, as the products moved further towards commercial approval, decisions on legal frameworks
(potentially requiring laws to be passed) and funding for regulatory activities now have the potential to be 
controversial. As the situation evolves, PBS has become more focused on broadening its outreach to 
other stakeholders that may impact the regulatory system. This has required an augmentation of 
communication skills, expertise and resources in the program.program and has resulted in a more 
integrated approach such atht technical capacity building and communications capacity building are now 
developed in tandem. PBS has a challenge to effect this change in their focus – while at the same time
retaining the credibility and trust which has been built up during the technical training and development.
As the situation is changing, the future role of PBS could be considered through how to strike a balance
between two extreme options, detailed below. It is not suggested that PBS chooses one of these two
extreme positions, and in reality, it is expected that some course between these two will be chosen, but
in choosing the particular course, the potential for conflict between these goals needs to be recognized.
28
Option 1. PBS remains an agency for technical support, narrowly focused on getting a relatively small
number of people trained in the review and monitoring of confined field trials and risk assessment of
applications for commercial release, staying away from the political advocacy areas needed to get such
cultivation applications approved. PBS is certainly good at this – and has a track record in a number of
countries. This would retain the reputation of PBS as an honest broker and might even have benefit for
USAID as it will be seen not to be an arm of big US companies. However, this will miss the major goal
of USAID – giving farmers in these countries more options to improve their productivity and
profitability and thus not be compatible with the Feed the Future mandate.
Option 2. PBS converts to an advocacy organization, utilizing its linkages and contacts to push for an
enabling environment through the adoption of (possibly) new laws, regulations and the setting up of
regulatory agencies. This will require quite a large alteration in the make-up of PBS staff and also – quite 
likely – a different kind of budgetary system, given the potential need for increased funds to be spent on
public relations and – possibly – covering the funding of positions within local government agencies to
facilitate the process. This is something which will more directly advance the Feed the Future goals, but
is quite different to the current PBS structure and may be difficult to operate within the current system
of an award to IFPRI.
One of the problems for PBS is that it is operating in countries which are at different stages along the
pathway towards a functional regulatory system. Countries such as Malawi are still at the stage of
needing technical support to handle applications and manage confined field trials, while Indonesia has a set
of technical committees which are now actively involved in handling approvals for food use and
environmental release. Even as each country moves towards the commercial end-point and advocacy
becomes a bigger part of the PBS role, there will still be a need for technical support to regulators and –
possibly – developers and so PBS will need to retain the technical expertise and, to a large extent, an un￾aligned non-partisan stance.
The need to be essentially neutral and outside the government means that PBS will never have full
control over the outcomes in relation to getting government procedures in place and products 
approved. Thus there will always be a need to have other measures of impact than a simple counting of
bills and regulations put in place by governments. In both planning and monitoring, the inability of PBS
to control many of the factors which are directly related to its success needs to be understood and
milestones and their attendant criteria developed to incorporate this.
29
ANNEXES
30
ANNEX I: EVALUATION STATEMENT OF WORK
Aug 1, 2013
STATEMENT OF WORK
Performance Evaluation of the Program for Biosafety Systems
Complete Title: Performance Evaluation of the Program for Biosafety Systems 
Begin Work Date: 08/10/13
End Work Date: 10/30/13
Total Days: 64
1) BACKGROUND
A. Problem Addressed by PBS
USAID has been funding efforts in biotechnology and the regulation of agricultural
biotechnology for nearly two decades across several partners and host countries in Africa 
and Asia. Amongst these programs are those that support R&D and deployment of new 
agricultural technologies using biotechnology and a separate set of programs that support 
the development of reasonable, science based regulatory systems that govern the 
application of these new agricultural technologies. PBS is the largest of USAID’s 
investments in this latter group. PBS has been supported for 10 years by both USAID
Washington and USAID field missions.
The role of biosafety systems and facilitative policies to insure that Agency-funded 
research yields practical benefits cannot be underestimated since it is widely recognized
that biosafety considerations and concerns, especially in developing countries, can be a 
primary limiting factor in biotechnology adoption.
B. Target Areas and Groups
PBS is currently active in the following countries:
Africa
• Kenya
• Uganda
• Nigeria
• Malawi
• Mozambique
• Tanzania (new in 2013)
• Ghana (new in 2013)
Asia
• Indonesia
• The Philippines
31
• Vietnam
Regional Organizations
• COMESA
• ECOWAS (new in FY13)
C. Identifying Information
• Activity Title: Program for Biosafety Systems (PBS)
• Grant No: AID-EEM-A-00-03-00001
• Implementing Partner: International Food Policy Research Institute (IFPRI)
• Award Dates and Amounts:
Leader Award: 05/14/2003 to 09/30/2013 $25,762,721 
Leader Award Funding from:
USAID/BFS 05/14/2003 to 09/30/2013 ($23,451,154)
USAID/Ghana: 10/01/2012 to 09/30/2013 ($250,000)
USAID/Tanzania: 10/12/2012 to 09/30/2013 ($200,000)
USAID/Indonesia: 10/01/2010 to 09/30/2013 ($931,567)
USAID/Malawi: 10/01/2007 to 09/30/2013 ($300,000)
USAID/Nigeria: 03/22/2007 to 09/30/2013 ($550,000)
USAID/Mozambique: 10/01/2006 to 09/30/2013 ($80,000)
Associate Awards:
USAID/Kenya: 10/01/2010 to 09/30/2013 $600,000
USAID/Kenya 03/12/2007 to 03/11/2010 $479,000
USAID/Uganda: 08/03/2004 to 01/31/2013 $2,419,284
USAID/Ghana: 03/01/2005 to 04/30/2008 $750,000
D. PBS Theory of Change and Intended Results
PBS supports partner countries by facilitating use of biotechnology for the eventual deployment 
of beneficial agriculture products, such as new genetically-modified (GM) varieties of crop
plants that are resistant to a variety of biotic and abiotic stresses (e.g. insect resistant cowpeas 
and drought tolerant maize) or GM varieties with enhanced nutrient profiles (e.g. Vitamin A 
enriched cassava). The indirect outcome is sustainably enhanced productivity and climate 
resilience in agriculture production systems to reduce poverty, enhance nutritional status and 
improve the quality of life for resource poor farmers and consumers.
The theory of change in PBS’s support for developing functional biosafety systems is to ensure 
that countries have the legal framework and the laws which facilitate development and adoption 
of biotech crops. PBS initial support and guidance is required for sustained, credible and, 
impartial biosafety legal and policy advice and capacity building of decision-makers. Coupled 
with this, targeted, practical/technical support will be essential to actualize the benefits of these 
products for farmers and consumers alike. In parallel with the increased number of GM
products approaching release in developing countries, ongoing public debate and legal
32
challenges are more frequently emerging as obstacles to progress. PBS has strategic, multi￾stakeholder outreach activities, coupled and integrated to the work it does on the regulatory 
front, to confront these challenges “head on” and to avoid backpedaling to more precautionary 
policies.
The final outcome of the layered support and integrated approach provided by PBS is to
enhance adoption of biotech crops, coupled to an increased understanding of the underlying 
science by decision makers in order to withstand negative influences of anti- GM groups which
might affect eventual product adoption and limit impacts. Developing a country’s capacity
would impact and further its national goal of increased food security, poverty reduction and also
better climate resilience. Figure 1 presents PBS’s Impact Pathway, which describes the Inputs 
and Outputs for which PBS is responsible, and the Outcomes and Impacts that should result in
part due to PBS successfully implementing planned inputs and producing planned outputs. 
33
Input Output Outcome Impact
National policy and regulatory framework
development
a. Legal training and small
drafting team retreats to
lawyers and regulators
b. Support to legal and regulatory
actors to review and approve
commercial product
applications
c. Educate regulators about the
biosafety regulatory system
and facilitate the establishment
of policies and guidelines that
allow for simultaneous use of
overlapping data and analysis
to satisfy other legal obligations
d. Train regulators on how to
write better legal language into
the laws and defend these
regulatory decisions against
potential challenges
e. Socio economic analysis of
major crops or traits in the
regulatory pipeline
Strategic outreach and communications
a. NetMapping to orient, update
and refresh the country
strategies
b. dynamic use of the PBS
website as a capacity building,
information sharing and
regulatory services tool
c. systemize communications
training for regulators
d. establish a more formal
networkmechanism whereby
1. Regulatory
framework developed
2. Biosafety law passed
3. First products submitted
for CFT
4. Competent biosafety
officials in place;
5. Draft law developed;
6. Regional harmonization
framework developed
7. Analysis of biosafety
decision-making options
for CPB
8. Research used to inform
biosafety frameworks
9. Communication
materials developed and
disseminated
10. Advocacy of
biotechnology as a
science for enhancing
support within the
country
11. Implementation of
enabling regulations
(e.g., circulars, guidance
documents)
Functional
biosafety systems
established in
partner countries
Commercial
release of GM
crops
Coalition built at
various levels
within the
agriculture value
chain of people
knowledgeable
about the benefits
of biotechnology
Capacity within
country to counter
pressures from
anti-GM lobbies
Better
understanding of
the science of
biotechnology for
enhanced food
security and
poverty reduction
Sustainably
enhanced
productivity and
climate resilience
Enhanced food
security in
partner
countries and
the region
Reduction of
agriculture loss
from climate
change effects
Poverty
reduction in
adopting
countries due to
increased yields
and 
better trade
Expand
producer choice
Inspire
consumer
confidence
Facilitate trade
Promote
agricultural
R&D
34
developing country regulators
can share information, best
practices, lessons learned and
issues across countries
Regulatory and technical support
a. support in the development and
implementation of enabling
regulations
b. Assist regulatory agencies in
developing policies surrounding
food and feed safety
c. Assist regulatory agencies in
defining and implementing insect
resistancemanagement practices
d. Assist regulatory agencies and
technology providers
e. Foster partnerships between in￾country R&D institutions and
regulatory agencies
Supportive decision-making research
a. Inform decision-makers on the
socio-economic consequences of
various policy options
b. Assess existing constraints and
policy obstacles that impact the
development and implementation
of functional biosafety systems
c. Analyze current and upcoming
issues at a global scale and
determine the impacts of sub￾regional, regional, and global
regulatory instruments and trade
considerations on national
regulatory systems
d. Analyze the impacts of GM
regulatory policy on product access
from a gender sensitive 
perspective
12. Capacity developed in
food-feed safety
assessments
13. Reconciliation of
biosafety laws and
procedures with other
laws and across
ministries
14. More positive political
climate and enabling
policy system across
ministries
15. GMplants/products
tested and evaluated in
compliance with national
guidelines
16. System for commercial
release incl. IRM
guidelines
in agriculture
production
systems
developed
Figure 1. Impact pathways of the Program for Biosafety Systems
E. Approach and Implementation
The PBS program works with local agricultural organizations and stakeholders to build 
functional science-based regulatory pathways for the adoption of new biotechnology 
products. Its diverse team of scientific, legal, commercial and communications experts 
builds biosafety capacity through an integrated program of policy analysis, development 
35
and implementation for practical, achievable results. Activities are designed and
implemented through a country-led approach with local PBS country leads. Further, the 
ability to draw on an in house independent policy research team at IFPRI brings added
value and academic credibility for informed decision-making. IFPRI is a public
international organization and member of the Consultative Group on International
Agricultural Research.
BFS supports the PBS core budget. This budget includes management staff, technical
consultants, global policy research, participation in international meetings (i.e. Cartegena
Protocol, ISBGMO) and in-country costs for some PBS countries (Malawi, the Philippines, 
Vietnam, COMESA and ECOWAS). USAID missions support PBS to strengthen and 
deepen programmatic activities in their respective countries. PBS currently receives
additional mission funding from Ghana, Indonesia, Kenya, Nigeria, Tanzania and Uganda.
PBS’s approach to biosafety capacity development is unique among a variety of biosafety
service providers (most of who are funded by other donors) in several aspects, as it
emphasizes:
• A comprehensive approach to biosafety framework development, 
including the development of workable implementing regulations, 
detailed operating procedures, and guiding biosafety dossier 
development and reviews, among others;
• A sustained and consistent on-the-ground presence in the form of local
country coordinators who facilitate discussions of needs and concerns 
between local decision-makers and the project’s technical and 
managerial expertise;
• Development of biosafety frameworks by involving international 
and local legal and scientific expertise in support of effective 
functioning biosafety decision-making;
• Supportive outreach and communications strategies for national 
competent authorities and the broader stakeholder community that is 
a pre-emptive, guiding and responsive to complement the technical
work that is undertaken by the project;
• Capacity development, including hands-on training, around a specific
product, policy/legal instrument, or GM application in the pipeline in 
partner countries;
• De novo, targeted research to inform the decision-making process
and effect policy development, adoption and implementation.
F. Existing Documents
PBS will provide the evaluation team with the following documents at the beginning of the 
assignment:
• Original Grant
• Performance Management or Monitoring and Evaluation Plan
36
• Annual workplans and reports
• Publications and Major communications pieces produced by PBS
• Any additional technical reports
• Yearly Summary Financial Reports from period of FY2008-FY2013
2) TASKS
A) Evaluation Purpose
The purpose of this external evaluation of PBS is to assess the performance of the 
preceding 5-year period, and to identify ways in which the program can be improved 
during the planned next 5-year period. The scope of this review will be the entire range of 
PBS activities, including policy research and all in-country programs, but excluding 
programs that just began in FY2013 (Ghana, Tanzania and ECOWAS).
B) Audience and Intended uses
The primary audience for this evaluation is USAID/BFS and IFPRI staff. The evaluation 
will identify successes and lessons learned over the past 5 years of program
implementation, and help guide the design of the new IFPRI award to continue PBS 
activities.
C) Evaluation Questions
1) PBS Country Programs Staff and Support from International Technical
Consultants
(a) Are the roles and responsibilities of the in-country country lead appropriate 
compared with international short-term consultants? Should the mix be 
adjusted to support program expansion and sustainability of provider support 
in the next PBS award?
Sub-questions to consider: Is the skill set of the in-country country leads 
appropriate to achieve PBS objectives? In what areas of their essential job
functions have country leads been effective and where have they struggled? 
Do stakeholders feel country leads have enough/the right technical expertise,
as compared to skills required to initiate and keep activities moving forward,
and to communicate, advocate and build consensus among national 
stakeholders? Has the support provided by international consultants been
effective in achieving the objectives of their STTA and overall PBS country 
objectives?
2) Policy Research
(a) Has the policy research carried out by PBS been relevant to the technical
support and capacity development activities being conducted by PBS?
Provide specific examples of where PBS policy research has made a direct
and measurable contribution to the achievement of PBS country-level
objectives.
37
(b) Are the research products developed by the PBS team of high quality 
and contribute to a literature base that is being utilized by partners and 
stakeholders?
3) Project Management
(a) How effectively have the PBS Director and Africa and Asia regional
program leads supported learning and ensured quality across countries? 
How could these roles be strengthened in the next PBS award?
(b) How useful is the current PBS monitoring and evaluation system for 
program management, results reporting and learning? Given that many
outcomes are outside the manageable interested of the program, are impacts 
being captured appropriately across all aspects of the program? How might 
the M&E system be strengthened in the next PBS award?
4) Financial Management
(a) Given the goals of the projects as outlined in project documentation, how 
appropriately have BFS central resources been allocated amongst countries, 
partners and activities, to supplement and complement Mission funding and 
to support global technical leadership?
5) Communications, Outreach and Partnerships
(a) Has PBS effectively responded with appropriate and timely 
communications products and outreach directed to the appropriate 
audiences as technical and political issues arise? What could be improved?
(b) Is PBS effectively collaborating and working with other biosafety and
biotechnology programs in the regions where these projects are active
(including AATF, ABNE, ICGEB and OFAB communications 
programs)
D) Evaluation Design
This is a performance evaluation of PBS that focuses on PBS’s implementation processes
and efficiency. There are no comparison groups or randomized assignment involved, nor 
is there an expectation that the evaluation team will be able to attribute outcomes or 
impacts to PBS activities. USAID anticipates the team will use a combination of three
evaluation designs:
1. Snapshot: Looks at a group receiving an intervention at one point in time during the 
intervention. Most of the performance evaluation questions will be answered with
this design (Questions #1,3-5)
2. Before and After: Aggregate measure of outcomes for a group before and after the 
intervention. This design will be useful to document policy and regulatory changes 
as a result of PBS policy research (Q#2)
3. Case Study: Used to gain in-depth understanding of a process event or situation and 
explain why results occurred – from the perspective of a case, a cultural context, the 
38
lived experiences of recipients. The evaluation team may consider using this design
to document particularly useful and informative successes or lessons learned.
E) Data Collection and Analysis Methods
The evaluation questions are descriptive in nature and the evaluation team will rely
largely on qualitative methods to collect the data required to answer them. Likely
methods include individual and group structured interviews with, among others, USAID
BFS and Mission staff, PBS team members, host country government officials,
technology researchers and developers, and civil society participants in advocacy and
communication activities; desk reviews of project and other documentation to understand 
the biotechnology and biosafety policy climate and changes in policy and regulatory 
decisions made over the previous 5 years in PBS countries; and possibly network
analysis to understand how information is communicated and whether the right people 
are being involved to reach the desired project audience.
The evaluation team will be required to submit an evaluation plan that describes in detail
the data collection methods, including triangulation, that will be used for each 
evaluation question, and how unbiased information will be assured (see evaluation plan
template in Annex 1.) The evaluation plan should also detail the analytical methods to
be used for each data collection method. The plan should specifically address how 
qualitative interview information will be analyzed, including transcription, coding and
interpretation/theory development, and what types of validity checks will be used.
The evaluation should be based entirely on the data collected (interviews or 
documents) for the evaluation. No personal opinions should be included and no 
inferences made. All conclusions in the evaluation report should be based on objective 
data; biased or subjective statements and opinions should not be included in the report.
All statements/conclusions on performance/progress should be linked to project
objectives (and objective measures), such as those outlined in a project document or 
work plan, and measured against objective standards for performance. Performance
should be measured first and foremost by the goals and objectives laid out in project
planning, and statements about performance provided by respondents can be included as 
additional supporting information.
The evaluation team should develop written instruments for structured interviews and
systematic document review in advance of field work, and share them with USAID and
PBS for review and comment prior to initiating fieldwork. All instruments must be 
included in an annex in a final report.
F) Methodological Strengths and Limitations
Qualitative methods can be relatively flexible and adjusted as the evaluation team gains
understanding of the PBS context and approach. The methods can provide very 
39
accurate information if appropriate qualitative validity checks were employed, and will 
be useful for understanding how PBS interventions and project management operates 
and what fosters and hinders their success. Limitations of qualitative methods in
general include difficulty in replicating (NB: USAID does not intend to replicate this 
evaluation); and in generalizing beyond the specific activity being evaluated (NB: 
USAID intends to use the findings of this evaluation in the design of the next round of 
PBS, not more generally).
Quality depends heavily on skills of the individual evaluation team members and the use 
of non-biased methods. Team member qualifications include experience with evaluation
of these types of programs using these types of methods.
G) Reporting Guidelines
The evaluation team must ensure that the final report meets USAID’s Criteria to Ensure 
the Quality of the Evaluation Report (see Attachment 2). The team should also reference
PPL’s How To Note on preparing evaluation reports 
http://www.usaid.gov/sites/default/files/documents/1870/How-to-Note_Preparing-Evaluation￾Reports.pdf, and, to the extent feasible, submit the report using the USAID Evaluation 
template http://usaidlearninglab.org/library/evaluation-report-template.
All findings and statements of conclusions must be based on data, not opinion. 
Paragraphs or sections should be written primarily to provide a summarized finding, 
followed by an explanation of the supporting evidence that substantiates the finding.
There should be no personally identifiable information in the report. Do not include 
names, including in the list of interviewees in the annex. Do include titles or positions of 
interviewees.
H) Logistics
Three one-week field visits are anticipated, to Indonesia, Uganda and Kenya. Both team
members will participate in all three visits. The review team will consult with PBS staff
and USAID for logistic arrangements and contacts in host countries. PBS will be
responsible for assisting the team to organize the field visits, including scheduling 
individual interviews, arranging for any necessary in-country lodging and travel, and 
making work space available to the team, if available. The costs of all travel and in￾country activities will be covered by the USDA/FAS contract as will the per diems and
reimbursements for time spent.
40
Scheduling
The following schedule is suggested with a begin date in mid-late September 2013.
Tasks Wk 1 Wk 2 Wk 3 Wk 4 Wk 5 Wk 6 Wk 7 Wk 8 Wk 9 Wk 10
Organizational
meetings
Work and
Evaluation plans
Document review
Field
visits/consultations
Write and present
draft report
Write final report
Present final report
41
[43]
3) DELIVERABLES
Deliverable Due
Initial Meeting with USAID and PBS Within two days of
start date
Evaluation Team Meeting (internal) Within two days of
start date
Work Plan Within one week of
start date
Evaluation Plan Within one week of
start date
Draft Report Within three weeks of
completion of field
visits
Presentation on Draft Report (conference call or 
in person)
Within three weeks of
completion of field
visits
Written comments from USAID and PBS Within two weeks of
receiving draft 
report
Final Revised Report Within two weeks of
receiving 
comments
4) QUALIFICATIONS
The review team will be composed of 2 senior consultants, an evaluation specialist and a 
biosafety expert in the technical areas listed below.
The Evaluation Specialist will be the Team Leader. His/her responsibilities will be to
coordinate all activities and ensure all aspects of the review are covered by the team. The
Team Leader/Evaluation Specialist is responsible for developing the evaluation plan, and
ensuring the technical team member uses appropriate data collection and analytical
techniques as defined in the evaluation plan. The Team Leader/Evaluation Specialist will
ensure that the report is completed according to the plan above.
Final decisions on which team member is responsible for which evaluation question will
be determined by the team based on composition and expertise. However, illustratively,
the team Leader/Evaluation Specialist could be primarily responsible for evaluation
questions 3 and 4, while the technical team member focus on questions 2 and 5. The
team would jointly be responsible for question 1.
Education and Experience: All members of the team will hold advanced degrees in
relevant fields. The team will have combined work experience in the following areas:
[44]
research management and/or administration, regulatory compliance, product stewardship, 
human and institutional capacity building – particularly in biosafety/regulation/law, and
international development program evaluation of similar types of projects.
Knowledge and Ability Requirements: Team members will possess expertise in: (1)
biosafety regulation; (2) agriculture biotechnology and its applications; (3) international
agriculture development issues; (4) research methodology and implementation; (5)
human and institutional capacity building; and (6) performance evaluation design, and
data collection and analysis methods.
5) PERIOD OF PERFORMANCE
Activities under this Statement of Work will be implemented in two (2) months or until 
11/30/2013.
[45]
ANNEX II: EVALUATION METHODS AND LIMITATIONS
DATA COLLECTION
The project’s design is based on assisting country-specific institutions to develop successful and fully
functioning regulatory frameworks and methods for assessing biosafety of products, and implementing
the associated directives with appropriate measures, laws and regulations. As this is a process
evaluation, it focuses mainly on the effectiveness of the processes and the inputs in producing project
success, rather than on the specific outputs or results. Therefore, a large part of the data collection has
involved gathering information to assess the extent to which the different activities of the program and
its management have contributed to achieving the general program objectives and specific country
objectives. To a great extent the answers to the evaluation questions involve subjective opinions,
however employing the techniques of cross- verification and triangulation between respondents and with 
existing secondary data objective findings are derived. Data was collected for this evaluation using two
methods: document review and key informant semi-structured interviews .
DOCUMENT REVIEW AND SECONDARY SOURCES
The evaluation team conducted a preliminary desk review of relevant project documents and key sector
background material in order to gain a thorough understanding of the specific activities over the 5 year
period of the project, together with the country context in which each program is operating.
Document review is an important source of data on project activities that can be used to triangulate
interview data, derive qualitative information and continually inform the data collection process.
KEY INFORMANT INTERVIEWS
Key informants - identified by the contractor in-country staff in addition to others identified during the
data collection process – we interviewed either face-to-face or by telephone conference. Based on a
review of project documents the evaluation team identified several stakeholder groups that are the
focus of program activities. These stakeholder groups comprise (i) USAID and other US Govt. donors,
(ii) PBS Implementing staff in both the headquarters and the countries visited, (iii) Government
regulatory agencies as well as research institutes and universities acting as agents of regulators, and
(iv)Partners/collaborators (including technology development groups such as ABSPII, AATF and
regulatory groups such as ICGEB, and the private sector). Attempts were made to interview
representatives of all stakeholder groups in each country visited, including representatives of those who
have received training through PBS. Discussions during field work with USAID and the implementers
were used to better define the target groups and individuals to interview.
For each such stakeholder group, specific questions were designed that focus on their interest in the
PBS project. The same base sets of questions were asked in all three sample countries visited - see
Annex III for interview instruments. Questions were asked in the same order where possible. In
Indonesia, key informant interviews were conducted with both evaluation experts present in order to
assure two independent sources, however this was not possible in Uganda and Malawi.
Data collection was aided by the use of an Excel workbook to organize and store data culled from
interviews. At the end of the interview day, interview notes were synthesized and entered into the data
collection spreadsheet. A separate sheet in the excel workbook was maintained for each of the
stakeholder groups identified above. Each of these sheets contained all potential interview questions to
be asked to that stakeholder group. Each interview respondent was listed in one sheet of the excel
workbook to allow recording information from each protocol question from each interviewee.
Confidentiality of respondents was maintained by anonymizing the data (i.e. removing names, titles, and
other information that specifically links a participant with a specific response).
[46]
This method of organizing and assembling interview information allowed for the identification of
patterns in the responses and common themes that are supported by evidence in the data. To ensure
that the data are internally valid and reliable, interview data was triangulated between respondents and
between sources such as work plans, performance reporting, research reports, financial reports, or
other secondary data.
Data collected and transcribed into the excel workbook was systematically analyzed to answer the five
evaluation questions. This content analysis process involved review of the data in the excel workbook
for emergent themes. Attempts were made to quantify the prevalence of certain themes in the data and
indicate the frequency that represent the prevalence of any given theme for one or more stakeholder
groups or respondents. The identified and analyzed themes allowed the formulation of findings for each
of the evaluation questions.
The evaluation report is structured on the basis of these questions – i.e., the discussion of each
evaluation question, by itself, forms the basis of a complete section, based on analysis of the themes and
response across the range of stakeholders.
PARTICIPANT SELECTION AND BIAS
Key informant interviewees were selected non-randomly based on availability and engagement with
program activities and as such, there were no comparison groups. A non-random selection is standard
for qualitative “purposive” evaluations. As there is a small universe of stakeholders in each country, it
was not be possible to randomly select interviewees. Attempts were made to interview representatives
of every organization that was a recipient or object of the PBS project in the three sample 
countries, as well as with regional organizations whose own interests and work coincide with the donors. 
However as most interviewed beneficiaries are direct participants in the project and were recommended
by either USAID or the contractor, it can be expected that they will say positive things about 
the project.
Where possible, to minimize bias informants were sought who represent “public stakeholders” with
different ‘causal distances’ from the activities to qualitatively triangulate/check information from
beneficiary organizations and who cannot be perceived as “friendly” or biased stakeholders but who
have sufficient knowledge to comment.
RECALL BIAS
To minimize recall bias, the interview questions were designed to draw upon specific knowledge or
viewpoints that the organization or the individual acquired from direct association with the project. As
the project is currently actively being implemented recall issues should be minimal.
INTERVIEWEE RESPONSE BIAS
All participants come with biases. An attempt was made to organize a sample of organizations with
different perspectives and playing different roles to provide different viewpoints. The degree to which
there is agreement/disagreement in viewpoints is indicated and assertions are triangulated with other
data sources and with other stakeholder groups.
Undoubtedly a larger problem, difficult to assess, is that of deliberately inaccurate responses because the
respondent wants to keep a good relationship with the implementer and donor. This shortcoming will
be mitigated through triangulation of what was said by different organizations, comparison with known
facts, and the interviewer’s perceptions of points of view.
ABILITY TO GENERALIZE FROM THE DATA
[47]
The method of organizing the data by question and respondent through an excel workbook allowing for
coding for recurrent themes, opinion agreements and disagreements allow the identification of areas
where the data clearly justifies generalizations. Where possible the number of respondents is quantified
although the quantification is subject to interpretation since often the respondents’ answers to “open￾ended questions” are imprecise and opinion-related, and it is difficult to get explicit answers or examples. 
Hence the development of recurring themes and concurrence across different types of respondents
provides the best source of data. USAID Evaluation Policy is followed, placing emphasis on anchoring 
conclusions in ‘findings of fact,’ and not in individual interviewee opinions or second-hand reports. 
Clear, reproducible evidence for findings and conclusions, supported by specific examples and quotes, 
have been used to tie findings to conclusions, and from there to recommendations.
[48]
ANNEX III: DATA COLLECTION INSTRUMENTS
USAID • Do you suggest policy research topics to PBS staff?
• Do you receive requests for publications from any of the stakeholder groups you work with
(regulatory agencies, policy makers or collaborators)?
• Are you aware of communications products or messaging from PBS? Have you used PBS
communications products in your own contact with media or in explaining the PBS program or in
other USAID programs?
• Are there specific communications products which you have either developed jointly with PBS or
which PBS have developed which you use? In either case, how have they been used and what
has been their impact on targeted groups ?
• In your opinion (on a scale of 1-5) how effective has PBS been in targeting influential groups to
change attitudes about GM? Are there groups they have not targeted or methods unused?
• How timely has PBS been in providing messaging when there is a political issue?
• Describe the monitoring information you get from PBS and please rate how satisfactory it is to
your needs for monitoring and decision making about the program, its funding, and its coherence
within the USAID country portfolio.
• What is the basis (criteria) for deciding whether to fund PBS out of Mission funds? What
information do you rely on from PBS?
• Would USAID want more information on outcomes and impacts (in addition to outputs)?
• How effective is the country staff at performing their duties? Do they have the appropriate skills
to deal with policy makers and regulatory agencies?
• How effectively is the communication between the Mission and PBS country staff and experts?
Implementers • Can you describe what systems you have in place for communications, consultation and
information flows between field and home staff and experts? What are the strengths and
weaknesses of the current system? Are there gaps and how do you address them?
• Who ensures quality control of inputs and products within and across different countries?
• How do you see the role of the director and two regional team leaders compared to Technical
Consultants for quality control and programming?
• How do you define the role of the in-country leads? How do you define their role compared to
that of technical Consultants and/or to country advisors (where present)?
• To what extent is technical background necessary or important for country leads? What other
qualities and experience have you found important?
• Please rank on a scale of 1-5 the capacity of the in country leads to fulfill the following functions:
advocacy, outreach, logistics, management, technical direction. What makes you rank them so
high/low?
• Is the mix of staff and consultants adequate for needs? Are there gaps that need to be filled?
• What is the Country Team? What is its role?
• Do you have country-specific indicators? Are the indicators the same throughout the projects?
Are there indicators for regional programs and if not, how do you measure outcomes?
• In your reporting, how do you differentiate between milestones and outputs? Where (and how)
do you track outcomes and results and relate this to programming decisions?
• What are the definitions for your indicators and how do you assess meeting definitions?
[49]
• For what purposes do you use the information in your monitoring plan (budget decision,
programming, reporting to USG)?
• Do you benchmark in any way amongst the countries where you actively have projects, for
example annual central grant amount, inputs provided such as consultant days, outputs
achieved, targets from your indicators?
• Who decides topics for policy research? Where do ideas come from and what is the input of
USAID Missions, collaborators including international or national groups, country leads,
Consultants, policy makers and regulatory agencies? What are the criteria for selecting topics?
• How (and who) determines the likely audience for policy research and how to target this
audience? Who is tasked with the promotion and dissemination of policy research products and
according to what plan?
• Please rank the general usefulness of policy research in promoting the project objectives on a 
scale of 1-10. If applicable, what makes you rank them so high/low?
• What do you see as the most effective dissemination methods for different types of research, with
most payback?
• Is there research to determine the effectiveness and influence of different policy research and on
what target groups?
• Can you cite examples where the research has clearly contributed to achieving a program goal or
objective or outcome? What did it achieve and what factors made it useful or influential? Can you
cite examples of where research was not relevant?
• How (and for what) do you use the research for training and capacity development activities?
Where has it been of direct application?
• What criteria do you use in allocating central program resources? Is this a function of need,
based on amount of mission support, or lack thereof or other criteria?
• What constraints are placed on funds from different sources (USAID missions etc.)?
• If a mission stops its funding, what are the financial consequences? Do you shift more core funds
into the country to make up the deficit ? Reduce activities?
• Are there instance you can cite where the funding was not sufficient to meet the objectives the
PBS program was committed to achieving? Has funding level been an issue?
• What kind of collaboration with regional governmental bodies on harmonization has proved most
successful? Had the most impact? What has proved less-useful? In what way has association
with these bodies improved or added to the impact of your program success? What work with
other, global organizations, has proved successful and had an impact?
• Do you collaborate with other groups on communications and outreach activities and has this
been useful and in what way?
• What has been the impact of the communications activities in the past? Do you do any research
to determine what the impact has been on the groups you target?
• What gaps have been identified in outreach and how do you plan to improve communications to
reach relevant target groups in each country?
• Is there a demand for communications and outreach materials from local sources? Which
sources have requested this and how have you responded?
• Do you have a communications plan for specific occurrences – such as a set of prepared
responses etc.?
• Are there any anti-GM groups active in your country and what has been their effect on public
opinion? How has PBS addressed their views?
• To ask of country leads: what do you find most challenging in your assignment? How do you
define your role and responsibility compared to that of the experts? Is the support of the
international consultants adequate to your needs? Is more in country assistance needed? Is the
[50]
support of central staff adequate to your needs? Does your scope of work reflect the real job
requirements?
• To ask of consultants (consulted): Is the support of the country staff adequate to your needs? If
not what is needed? Do you get sufficient time in-country, or to do the necessary quality control or
communications and technical support?
Regulatory agencies • What is the nature of your interaction with the country program staff? With the central PBS staff?
• On a scale of 1-10 can you assess the adequacy of the country program staff experience in
biosafety issues? If applicable can you explain the reason for your high/low score?
• On a scale of 1-10 asses how well program staff manages the activities in which you collaborate
with them? If applicable, can you explain the reason for your high/low score?
• Are sufficient staff consultant or expert time and input available to meet needs that country leads
cannot fill? What are significant contributions of staff consultants?
• Please rate the usefulness of the expert technical assistance on a scale of 1-10. Please rate the
professional level of the assistance on a scale of 1-10.
• What technical assistance or training has proven most useful to your organization and had the
most impact? What in your view was not useful? Why?
• Have expert technical assistance inputs been sufficient in (terms of time and presence) to meet
your needs? Are there unmet needs?
• What policy research are you aware of from the PBS? How do you know about research that is
done through the PBS program - through publications, directly through the assistance or through
other means?
• What has been your input into determining policy research topics or products?
• Do you see policy research outputs in publications?
• How do you rate the quality of the research on a scale of 1-10?
• To what extent has the policy research related to or added to the technical support or training that
was given. (rarely, mostly, always)
• Can you cite examples where the policy research has clearly contributed to achieving one of your
organizations’ biotechnology or regulatory objectives or activities? How have you used research,
what did it achieve and what factors made it be influential or useful?
• Do you work with PBS on communications activities?
• Are there specific communications products which you have either developed jointly with PBS or
which PBS have developed which you use? In either case, how have they been used and what
has been their impact on targeted groups?
• Has awareness of biosafety regulation changed over the period of PBS? In your view has the
training and outreach through PBS had an effect and if so on what groups?
• Do you think the outreach and communications messages reached the right target groups? Are
there any agencies or groups which you think should be included in outreach and
communications which have not been included? Are there gaps?
• Are there any anti-GM groups active in your country and what has been their effect on public
opinion? How has PBS addressed their views and the issues they raise?
CollaboratingOrganizations • Do you interact directly with the country program staff and/or the central PBS staff?
[51]
• If so, do you find the country program staff to be adequately trained/experienced in biosafety
issues to be of assistance to you?
• How effective is the country program staff at managing the activities in which you collaborate with
them?
• Please evaluate the quality of PBS experts with whom you have collaborated (scale 1-10) If
applicable can you explain the high/low score?
• What has been your input into deciding policy research topics that PBS has funded?
• How do you rate the quality of the research on a scale of 1-10?
• In your view, has the PBS policy research contributed to achieving a program goal or objective or
outcome of your organization? What in particular? What did it achieve and what factors made it
influential?
• Have you been in receipt of funds from PBS for joint activities? If so for what?
• Are there constraints on what the funds can be used for?
• Do you have specific reporting rules for jointly-funded projects?
• What collaboration with PBS has been most useful to advancing joint goals?
• In what way has association with PBS improved or added to the impact of your program success?
What is the added value of association?
• Do you work with PBS on communications activities? Do you have your own target groups?
• Are there specific communications products which you have either developed jointly with PBS or
which PBS have developed which you use?
• Do you operate joint outreach activities, or do you have input into PBS outreach activities? (Does
PBS involve other agencies when conducting communications activities either at the central
management level , at the level of USAID or within countries where the project is active?)
• Are there organizations which you think should be included in outreach and communications
which have not been included?
• Can you evaluate on a scale of 1-10 the effectiveness of their targeting messaging and
communications? If applicable can you explain the high/low score? Are they reaching key groups
(differentiate by country )? What evidence do you use to evaluate?
• Do you have a communications plan for specific occurrences – such as a set of prepared
responses etc.?
[52]
ANNEX IV: SOURCES OF INFORMATION
NAME AFFILIATION COUNTRY GROUP
John McMurdy/
Saharah Moon Chapotin
USAID, Washington DC USA USAID
Thom Wright/
Ali Absi
USDA-FAS, Jakarta Indonesia USAID
Brian Dusza/
Donal Tambunan
USAID Indonesia Mission Indonesia USAID
Simon Byabagambi USAID – Uganda Mission Uganda USAID
Cybill Siegler USAID – Malawi Mission Malawi USAID
Heather Dresser/
Piston Msika
US Embassy, Lilongwe Malawi USAID
Mark Petry USDA-FAS, Washington DC USA USAID
Sidi Asmono PBS Country Coordinator Indonesia Implementer
Jeff Stein PBS Regional Coordinator USA Implementer
Donna Ramaeker Zahn PBS External Consultant USA Implementer
Herbert Oloka/
Theresa Sengooba
PBS Country Team Uganda Implementer
Boniface Mkoko PBS Country Coordinator Malawi Implementer
Jose Falck-Zapeda PBS Policy Team Leader USA Implementer
Greg Jaffe PBS External Consultant USA Implementer
John Komen PBS Regional Coordinator Netherlands Implementer
Judy Chambers PBS Director USA Implementer
Prof. Bambang Sugiharto University of Jember Indonesia Collaborator
Prof. Sugiono
Mulyoprawiro
Crop Life Indonesia Indonesia Collaborator
Herry Krisanto Monsanto Indonesia Collaborator
Dr Widodo Hadisaputro Society for Indonesian Agricultural
Biotechnology (SIAB)
Indonesia Collaborator
Prof. Mohammed Herman ABSPII Country Coordinator Indonesia Collaborator
Hon. Beatrice Anywar Member of Parliament, Uganda Uganda Collaborator
Imelda Kashaija National Research Organization Uganda Collaborator
Andrew Kiggundu National Agricultural Research
Laboratory, Kawanada
Uganda Collaborator
Peter Wamboga SCIFODE Uganda Collaborator
Dr Peter Ndemere Uganda National Council for Science and
Technology (UNCST)
Uganda Collaborator
Dr Tushemereirwe National Agricultural Research
Laboratory, Kawanada
Uganda Collaborator
Tilahun Zeweldu ABSPII Country Coordinator Uganda Collaborator
Erostus Nsubuga Uganda Biotechnology and Biosafety
Consortium
Uganda Collaborator
Dr David Mutekanga Uganda national Academy of Sciences Uganda Collaborator
Prof. Rubaihayo Makerere University Uganda Collaborator
Dennis Kibirige/
Susan Nakabuye
Ministry of Justice and Constitutional
Affairs
Uganda Collaborator
Paul Nampala Regional Universities Forum for
Acapacity Building in Agriculture
Uganda Collaborator
[53]
NAME AFFILIATION COUNTRY GROUP
(RUFORUM)
Prof. Kwapata Bunda College of Agriculture Malawi Collaborator
Anthony Muyepa-Phiri/
Pongolani Msakambewa/
National Commission for Science and
Technology
Malawi Collaborator
Dt Wisdom Changadeya Biotechnology-Ecology Research and
Outreach Consortium (BioEROC)
Malawi Collaborator
Felix Jumbe Private sector (seed production) Malawi Collaborator
Duncan Warren Famers Union of Malawi, Cotton
Development Trust
Malawi Collaborator
Alick Maulawo Monsanto-Malawi Malawi Collaborator
Wendy Craig ICGEB Italy Collaborator
Craig Rickerd CropLife International USA Collaborator
Adelaida Harries BIGMAP Argentina Collaborator
Nancy Muchiri AATF Kenya Collaborator
Dr Agus Pakpahan Biosafety Commission for Genetically
Engineered Products (BCGEP)
Indonesia Regulator
Yusra Egayanti/
Siti Maemunati/
Saffian Dewi/
Desy Fasta Waty
National Agency for Food and Drugs
Control (NAPDCP
Indonesia Regulator
Dr Antung Deddy Ministry of Environment Indonesia Regulator
Dr Karden Mulya Indonesian Center for Agricultural
Biotechnology and Genetic Resources
(ICABIOGRAD)
Indonesia Regulator
Prof Bahagiawati Amir
Husin
Indonesian Center for Agricultural
Biotechnology and Genetic Resources
(ICABIOGRAD)
Indonesia Regulator
Dr Satya Nugroho Indonesian Institute of Sciences (LIPI Indonesia Regulator
Dr Erna Maria Lokollo Indonesian Center for Agriculture
Socio-Economics and Policy Studies
Indonesia Regulator
Dr Friday Agaba National Drug Authority Uganda Regulator
Dr Isa Katwesigye Ministry of Water and Environment Uganda Regulator
Prof. Opuda-Asibo National Council for Higher Education Uganda Regulator
Dr Yona Baguma National Crops Resources Research
Institute, Namulonge
Uganda Regulator
Caroline Thaka Environmental Affairs Department Malawi Regulator
Dr Yanira Ntupanyama Ministry of Environment and Climate
Change Management
Malawi Regulator
Dr Mackson Banda Ministry of Agriculture Malawi Regulator
[54]
ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST
Name Robert Potter
Title Dr
Organization Robert Potter Consulting
Evaluation Position? X Team Leader Team member
Evaluation Award Number
(contract or other instrument)
USAID Project(s) Evaluated
(Include project name(s),
implementer name(s) and award
number(s), if applicable)
I have real or potential conflicts
of interest to disclose.
Yes X No
If yes answered above, I disclose
the following facts:
Real or potential conflicts of interest may
include, but are not limited to:
1. Close family member who is an employee
of the USAID operating unit managing the
project(s) being evaluated or the
implementing organization(s) whose
project(s) are being evaluated.
2. Financial interest that is direct, or is
significant though indirect, in the
implementing organization(s) whose
projects are being evaluated or in the
outcome of the evaluation.
3. Current or previous direct or significant
though indirect experience with the
project(s) being evaluated, including
involvement in the project design or
previous iterations of the project.
4. Current or previous work experience or
seeking employment with the USAID
operating unit managing the evaluation or
the implementing organization(s) whose
project(s) are being evaluated.
5. Current or previous work experience with
an organization that may be seen as an
industry competitor with the implementing
organization(s) whose project(s) are being
evaluated.
6. Preconceived ideas toward individuals,
groups, organizations, or objectives of the
particular projects and organizations being
evaluated that could bias the evaluation.
I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update
this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other
companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains
proprietary and refrain from using the information for any purpose other than that for which it was furnished.
Signature
Date 22 April, 2014
[55]
ANNEX VI: STATEMENT OF DIFFERENCES
(1) Background
PBS is pleased to offer its comments on the findings of the External Performance Evaluation of the 
Phase II Program for Biosafety Systems (Grant No. AID-EEM-A-00-03-00001 for the period 2008-
20013). In general, we were very appreciative of the reviewer’s comments and regard this review 
as a positive assessment of the program, its works and its staff. We believe the review identifies a 
number of issues and recommendations that will assist our program in its next phase of funding 
(2013-2019). However, in the interest of accuracy, there are a few points which require additional 
comment and clarification. A summary of our general concerns is shown below. Individual and 
specific comments are presented, in addition, referencing the topic or issue in question by page and 
paragraph.
(2) General Comments Related to Conclusions and Recommendations
Structure of the review: We believe that the review could have benefited from a larger team, with 
varying expertise, to better inform the project. While it is imperative that at least one member of 
the review team have a technical understanding of the subject matter (as was the case here), a 
diverse team comprised of those with understanding of policy and/or legal challenges in a 
development context and as well as those with specialized skills set in impact analysis, specifically for 
policy interventions, would have been additionally helpful to the project.
Program Staff and Technical Consultants: The review accurately identifies the need to balance use of a 
strong technical team to maintain consistency and quality of results across the program and to 
insure technical accuracy vs. the need to build capacity among country level personnel to assume 
this role in the future. Unfortunately, for most of the countries in which we work, the capacity of 
local experts is still evolving, and building the capacity of such experts is a significant part of the 
program’s long term objectives. To a certain extent, PBS has used the Philippines study tours to 
address this need and one member of our team was formally a regulator in Argentina and has been 
a valuable resource to this end. However, it is quite possible and reasonable to cast a wider 
geographical net for advisory expertise (i.e. Brazil, Australia) and the program has begun to identify 
and engage experts beyond those that reside in the United States. Nevertheless, PBS appreciates 
the concern about the reliance on the use of US-based consultants. Capacity building to further 
develop local resources will continue, and the use of other global and locally placed consultants will 
be pursued to augment the existing core of primarily US-based consultants. 
Policy Research Issues: Significant strides have been made in recent years to conduct policy research 
that is more relevant to the decision-making challenges faced at the country level. This has been an 
evolution in the program, and the need to do so is certainly recognized by senior management and 
the PBS policy team. From a branding perspective, PBS is less concerned about whether the 
research is seen as PBS research or IFPRI research: our greater concern is to insure the research is 
used and impactful on the setting evidence-based regulatory policy. 
Plans are underway to develop the appropriate format and locally relevant materials to insure that 
research findings are accessible to key decision makers. A number of country level policy papers 
[56]
have been developed or are underway to specifically address the national decision making 
challenges, such as socio-economic impact studies on Bt cotton (Uganda, Kenya and Malawi) or 
labeling threshold impacts (Vietnam). We recognize that the format for the presentation of this 
information is often in academic journals or somewhat technical policy briefs. An identified goal of 
the program going forward is to insure that information is appropriate packaged for audiences 
which may have an authoritative role in policy formation but may lack detailed technical knowledge. 
Project Management: Many of the points raised with respect to project management, reporting and 
measuring of impacts are very useful suggestions which we will incorporate into future reporting 
structures and documents. PBS intends to acquire expertise to assist our Monitoring and Impact 
work to insure that we develop more appropriate metrics to measure project success. We agree 
that fine tuning is needed in work plans and progress reports that differentiate between inputs and 
outcomes and that realistic milestones should be developed to track our work progress, manage 
expectations and more effectively guide resources and expenditures. In general, impact 
measurements and reporting will benefit from the hiring of an M&E specialist. There is a need to 
reconcile overall reporting requirements to Mission needs for monitoring and evaluation and we are 
hopeful an M&E specialist can also assist in that area. 
Financial Management: Delays in our annual obligation from USAID place an extraordinary 
administrative and financial burden on the project. To date, the project has managed to cope by 
using reserve funds to support operations, although at a reduced level of effort during the 
intervening time until new program funds arrive. However, this has the undesirable effect of 
needing to preserve pipeline funds in order to protect the program against these delays. The effect, 
when funds finally arrive, is pressure to accomplish a year’s work plan of activities in a shortened 
time frame (6-9 months). The pressure on consultants and collaborators is also increased due to 
funding delays as contracts cannot be executed due to IFPRI policy until the obligation is in place. 
On a related note, retaining focus only on those countries where mission support is provided is also 
note tenable, as mission interests can change (e.g Indonesia). Instead, we believe the project should 
continue to focus on those countries where there is ample interest and need for the project’s 
support. Benchmarking of country progress, while an enviable goal, is also not entirely realistic due 
to the fact that countries are starting from a different place on the regulatory progress continuum; 
this is not always reflective of the product environment or political will. All of these factors are
taken into consideration, in addition to the possibility of mission support, when considering PBS 
engagement in a given country. 
Communications, Outreach and Partnerships: Consortiums speak for the range of interests and 
stakeholders which are members of the consortium. The role of PBS is to provide technical, 
evidenced-based formation to allow the development and implementation of science-based 
regulatory systems. 
(3) Specific Point by Point Comments Related to Specific Review Topics 
and Issues Raised
[57]
Evaluation Methods & Limitations
Page 15, Key Informant Interviews
Incomplete Review Team, para 2: Given that the reviewers were not standardized across all three 
countries, caution should be exercised in any attempt to compare and evaluate across the three 
considered – Indonesia, Uganda and Malawi.
Findings, Conclusions & Recommendations
1. PBS Country Programs Staff and Support from International Technical 
Consultants
Page 18
Use of US-based models, para 2: The use of examples in PBS training and capacity building work has 
been somewhat dictated by the fact that regulatory case studies and examples for most of the 
commercially released GM varieties come from the US (a majority) or other developed counties 
(i.e. Canada, Australia). As more developing countries, in addition to the Philippines, South Africa, 
Brazil and India, pursue commercial adoption of GM crops (such as Bangladesh recently did), the 
availability of non-US/developed country examples will increase and will be incorporated into PBS’s 
work. 
Expertise of PBS Country Coordinators, para 3: It is not necessarily a fair assumption to state that 
PBS coordinators are perceived to have a high level of competency due to the relative low level of 
competency of their government counterparts. PBS coordinators are chosen for their scientific 
competency, to start, and then are given extensive and constant training and, by virtue of working 
with the senior technical staff, have consistent and repetitive exposure to best regulatory principles 
and practices.
2. Policy Research 
Page 19
IFPRI vs. PBS Acknowledgement for Policy Research, para 1: This is essentially an issue about 
branding and not necessarily germane to the quality and the impact of the work. To some extent, 
the fact that the research findings are considered to be those of IFPRI actually raises the visibility 
and credibility of the work, as IFPRI is well known among most of the government policymakers. 
There is a need to better “package” the research in a user-friendly format, and to insure a more 
targeted outreach plan to those who can most benefit from having the information generated from 
the research conducted by the PBS policy team. This is a major focus for the next phase of PBS 
funding. 
Page 20 
Relevance and Balance of Policy Research, para 3-5: Considerable progress has also been made in 
Phase II over Phase I funding to insure that the research actually conducted is more client driven, 
and focused on addressing key decision making issues related to national biosafety discussions. To 
some extent, this is also a funding related issue. The PBS policy team is only partially supported by 
PBS funding. The team has other research objectives which they pursue with other funds. Current 
[58]
PBS funding levels in the leader award do not permit the “luxury” of a full time policy research team 
which can respond to all of the needs presented by the operational side of the program.
3. Project Management
Page 21 
Familiarity with Senior Management, para 2: Less familiarity with the PBS Director and regional 
directors is a function of project design. The in-country coordinators are expected to be the first 
line and most consistent, face of interaction with the government. This should be interpreted as a
neutral comment.
M and E Assessments, para 3: The Feed the Future criteria for M&E were not in place at the time 
this award was developed and initially funded. PBS has been responding to impact measurements 
based on guidance we have received from USAID, especially for capacity building, which is captured 
via proscribed, quantitative reporting guidance. The project agrees that the current methodologies 
and metrics for evaluating PBS impact are insufficient to fully account for project performance. 
With the addition of staff expertise in M&E for the new award period, we expect to revisit M & E 
reporting along lines which are more consistent with both quantitative and qualitative program-level 
impacts. Any additional guidance to assist in this from USAID, in terms of specific parameters that 
should be considered, would be welcome by the project management team.
4. Financial Management
Page 22, 
Mission Funding and Obligation Delays, para 2&3: The delays in PBS’s yearly obligation from USAID 
(which were extensive in FY 2013) cannot be underestimated in terms of the negative impacts it can 
potentially have on program success. This is compounded by the fact that CGIAR centers no longer 
have “core “ funds to cover temporary project shortfalls and therefore lack flexibility to address the 
problem. This pattern of delay with respect to the yearly obligation has necessitated that the 
project operate with an intention of retaining some reserve funding so that project operations can 
continue, if even at a reduced level of effort, until the USAID obligation is received. This however, 
results in criticism from USAID with respect to the project pipeline. The program is caught in the 
middle and ability to complete a 12 month work plan in a compressed 6-9 month effective time 
frame puts unnecessary stress on program staff, contractors and in-country collaborators. In 
addition, unless a separate contracting mechanism is employed, the mission “buy-in” mechanism of 
funding creates added stress as missions typically base commitments on a year by year time frame. 
PBS remains hopeful that these trends will not continue for the remainder of the Phase III award.
5. Communications, Outreach and Partnerships
Page 23
Shift in Communications and Outreach, para 1: Added emphasis on this function stems from a 
heightened awareness of the importance that good communications and outreach functions have in 
policy enactment and implementation. The project consultant was hired for a number of reasons, 
not just this one. The consultant’s role, as a member of the senior management team, is also to 
assist on internal communications across a complex and growing program, to establish standards 
[59]
and quality control in our training programs and communications materials, and to also build 
capacity of regulators to enunciate their policies and policy decisions with clarity and confidence. 
The role of this position extends beyond outreach. 
Coalitions, para 2: The role and interaction of PBS with these various coalitions needs some 
clarification. These groups are, for the most part, self-assembled and self-identified and have a 
variety of functions and activities which go beyond the PBS interests in regulatory policy. Other 
aspects of biotechnology are also a target of coalition work but PBS stays focused on the regulatory 
issues. Some guidance and administrative support has been initially provided by PBS, as well as 
program support specific to the activities which are directly in line with the PBS mission. We 
remain engaged with these groups to support the work that they do in bringing a technical and 
evidenced-based voice to biosafety policy discussions. 
Page 24
Public Relations vs. Evidenced-based Communications and Outreach, para 3: Public relations, and 
therefore hiring of a PR (local or otherwise) expert, is not really the role of PBS, as our efforts in 
this area are relegated to providing sound technical information about the regulatory process and 
safety issues, and we believe the channels to do so are fairly obvious and exist (with media being 
only one such channel). Monitoring of sentiment, particularly of those who are in a position to 
influence the regulatory process, is achievable. While this may have been expressed by some who 
were interviewed, this is not our role and is indicative of the nuanced differences between working 
on the establishment of sound regulatory policy vs. advocating for the technology, per se.
PBS Low Profile, para 5: The fact that PBS has maintained a low profile is not an issue from our 
perspective. We are not seeking accolades for the work that we do. We are instead focused on 
providing credible information, advisory services, research and capacity building in support of 
effective national regulatory systems. PBS believes our credibility is enhanced by our low key 
approach. 
Country Case Studies
(1) INDONESIA
Election Cycles and political appointments: The situation with respect to delays in appointments as 
a result of the election cycle is somewhat overstated. These appointments are often political 
appointments, as is the case in many developed countries, including the US. It is not unreasonable 
to expect that a Presidential election may have impacts on this type of appointment. It does not 
mean that the regulatory system is necessarily flawed or weak. The key thing to consider is to 
assess whether or not the regulatory functions proceed, along expected lines commensurate with 
policy, once such appointments are made. Also, it is not possible at this point to judge the impact of 
stakeholder interest in the face of negative pressure. To date, except for an understandable delay in 
the appointments made due to an electoral process, the system has been functioning, in spite of 
negative pressure. 
Indonesia Mission Support: The issue of mission support for PBS in Indonesia is an area of critical 
concern, but not necessarily for the reasons cited in the review. There is strong Government of 
Indonesia interest, expressly stated in many policy briefs and pronouncements, to use this 
[60]
technology to address issues related to food security and agriculture productivity likely to be 
impacted by climate change. With this as a clear policy pronouncement, it is imperative that the 
country builds a regulatory system that is based on sound science, global best practices and includes 
a staff that is skilled and confident in its regulatory authority. PBS has made great strides in assisting 
the government in this regard in a fairly short period of time (3 years). However, significant 
capacity building needs remain, especially as the first generation of public and private sector 
products approach general, commercial release. Without continued mission support, additional 
pressure will be placed on the PBS core funds to maintain a robust level of activity in Indonesia, a 
large, strategic country with many food security demands. This will generate the need for stock 
taking across the entire program, with the result that an Indonesia program may be funded at a 
lower level, resulting in less progress, or another PBS program may need to be eliminated, with 
local consequences for that country. 
(2) UGANDA
PBS has continued to provide technical support while progress on the Bill continues. We have been 
focused on guidelines for commercial release, and updating the training of the National Biosafety 
Committee.
PBS has given consideration to our role and plans upon passage of the Bill. We expect to provide 
significant technical expertise in training and capacity building for the new regulatory authority and 
to finalize various guidelines and requirements needed to implement the Bill, including support to 
various Ministries and Agencies who will have a defined regulatory role.
(3) MALAWI
PBS has been effective in encouraging others to engage in Malawi. Other regulatory service 
providers are offering assistance (the African Biosafety Network of Expertise) and the Bt cowpea 
project will begin work in Malawi in late 2014 via USAID Malawi mission support. The historically 
slow developments of the regulatory system and delays in the first field trial have been factors in the 
lack of more robust mission involvement. It is likely that in the face of continued regulatory 
progress, other donor-funded or commercial efforts in biotechnology will develop in Malawi.