End of Project Evaluation Final Report Enhancing Participatory Governance and Accountability (EPGA) Prepared by: 2 Table of Contents LIST OF TABLES 4 LIST OF FIGURES 6 ACRONYMS 7 KEY TERMS 7 1. EXECUTIVE SUMMARY 8 2. INTRODUCTION AND BACKGROUND 11 2.1 EPGA PROJECT OUTCOME 1 12 2.2 EPGA PROJECT OUTCOME 2 13 2.3 EPGA PROJECT OUTCOME 3 14 2.4 IMPLEMENTING PARTNERS 15 3. EVALUATION PURPOSE AND OBJECTIVES 16 4. EVALUATION METHODOLOGY 18 4.1 STUDY DESIGN 18 4.1.1 STUDY DESIGN SUMMARY 18 4.1.2 COMMENTS ON REVISIONS TO STUDY DESIGN 18 4.1.3 DESK REVIEW 18 4.1.4 QUANTITATIVE SURVEY 20 4.1.5 QUALITATIVE DATA COLLECTION 20 4.2 SAMPLING PROCEDURES 22 4.2.1 QUANTITATIVE SURVEY 22 4.2.2 FOCUS GROUP DISCUSSIONS 22 4.2.3 KEY INFORMANT INTERVIEWS WITH LOCAL GOVERNMENT 23 4.2.4 NATIONAL LEVEL AND PROJECT IMPLEMENTATION KEY INFORMANT INTERVIEWS 23 4.3 DATA COLLECTION PROCEDURES AND DATA ANALYSIS 24 4.3.1 QUANTITATIVE SURVEY 24 4.3.2 QUALITATIVE DATA COLLECTION 24 4.4 ETHICAL CONSIDERATIONS 24 4.5 LIMITATIONS 25 5. ANALYSIS OF FINDINGS 26 5.1 RELEVANCE 26 5.1.1 END VALUES FOR PROJECT IMPACT AND OUTCOME INDICATORS 26 5.1.2 APPROPRIATENESS AND USEFULNESS OF PROJECT INDICATORS 41 5.1.3 APPROPRIATENESS OF PROJECT DESIGN 41 5.2 EFFICIENCY 51 5.2.1 ACHIEVEMENT OF PLANNED OBJECTIVES AND CONSTRAINTS 51 5.2.2 ADEQUACY OF PLANNED RESOURCES 53 3 5.2.3 DELIVERY OF PROJECT FUNDS AND ACTIVITIES 54 5.3 EFFECTIVENESS 55 5.3.1 EFFECTIVENESS OF PROJECT MANAGEMENT ARRANGEMENTS 55 5.3.2 STAKEHOLDER INVOLVEMENT AND PROJECT OWNERSHIP 56 5.4 IMPACT 57 5.4.1 INTENDED AND UNINTENDED IMPACT 57 5.4.2 USE OF TECHNOLOGY 60 5.5 SUSTAINABILITY 62 5.4.1 SUSTAINABILITY OF PROJECT INTERVENTIONS 62 5.4.2 INSTITUTIONAL SUSTAINABILITY 63 6. CONCLUSIONS AND LESSONS LEARNED 64 7. RECOMMENDATIONS 65 7.1 PROGRAMMATIC RECOMMENDATIONS 65 7.2 OPERATIONAL RECOMMENDATIONS 67 8. ANNEXES 69 ANNEX 1: EVALUATION TOR 69 ANNEX 2: DATA COLLECTION TOOLS 76 ANNEX 2.1: BENEFICIARY SURVEY 76 ANNEX 2.2: WOMEN AND YOUTH FOCUS GROUP DISCUSSION/INTERVIEW GUIDE 78 ANNEX 2.3: COMMUNITY COMMITTEE MEMBER DISCUSSION/INTERVIEW GUIDE 81 ANNEX 2.4: CSO MEMBER DISCUSSION/INTERVIEW GUIDE 83 ANNEX 2.5: IMPLEMENTING PARTNER INTERVIEW GUIDE 85 ANNEX 3: LIST OF INTERVIEWS 87 ANNEX 4: SURVEY PARTICIPANT CHARACTERISTICS 88 ANNEX 4.1: GENDER AND AGE DISTRIBUTION 88 ANNEX 4.2: HOUSEHOLD SIZE DISTRIBUTION 88 ANNEX 4.3: ENGAGEMENT IN AGRICULTURAL SECTOR 88 ANNEX 4.4: ENGAGEMENT IN LEADERSHIP 89 ANNEX 4.5: SURVEY PARTICIPANT BY SURVEY TYPE 89 ANNEX 5: SUPPLEMENTAL QUANTITATIVE SURVEY DATA TABLES 90 ANNEX 6: PROJECT RESULTS FRAMEWORK 93 4 List of Tables TABLE 1: SUMMARY OF EPGA PROJECT IMPACT AND OUTCOME INDICATORS.....................................................9 TABLE 2: GEOGRAPHIC COVERAGE OF THE EPGA PROJECT .................................................................................12 TABLE 3: EVALUATION CRITERIA ..........................................................................................................................16 TABLE 4: EVALUATION QUESTIONS .....................................................................................................................16 TABLE 5: LIST OF KEY PROGRAM DOCUMENTS REVIEWED..................................................................................18 TABLE 6: LIST OF KEY EXTERNAL DOCUMENTS REVIEWED...................................................................................19 TABLE 7: QUALITATIVE DATA SOURCES................................................................................................................21 TABLE 8: QUANTITATIVE SURVEY SAMPLING.......................................................................................................22 TABLE 9: FGD/KII SAMPLING ................................................................................................................................23 TABLE 10: LOCAL GOVERNMENT KIIS ...................................................................................................................23 TABLE 11: PERCENT OF RESPONDENTS AWARE OF AGRICULTURAL POLICIES IN THEIR DISTRICT.......................26 TABLE 12: MAIN AGRICULTURAL ACTIVITY BY DISTRICT AS PER NISR AGRICULTURAL SURVEY (2017) AND PERCENT OF RESPONDENTS AWARE OF AGRICULTURAL POLICIES IN THEIR DISTRICT, BY DISTRICT.........27 TABLE 13: INDICATOR 2.A PERCENTAGE OF TARGET BENEFICIARIES WHO ARE SATISFIED WITH THEIR ROLE IN THE IMPLEMENTATION OF AGRICULTURAL RELATED POLICIES .................................................................27 TABLE 14: REPORTED MECHANISMS FOR PARTICIPATION IN IMPLEMENTATION OF AGRICULTURAL POLICIES.28 TABLE 15: INDICATOR 2.B PERCENTAGE OF TARGET BENEFICIARIES WHO ARE SATISFIED WITH THEIR ROLE IN MONITORING OF IMPLEMENTATION OF AGRICULTURAL RELATED POLICIES ............................................29 TABLE 16: REPORTED MECHANISMS FOR PARTICIPATION IN MONITORING THE IMPLEMENTATION OF AGRICULTURAL POLICIES ............................................................................................................................31 TABLE 17: INDICATOR 2.C PERCENTAGE OF TARGET BENEFICIARIES WHO ARE SATISFIED WITH THEIR ROLE IN MONITORING OF BUDGETS ON AGRICULTURAL RELATED POLICIES ..........................................................32 TABLE 18: REPORTED MECHANISMS FOR PARTICIPATION IN MONITORING THE BUDGETS OF AGRICULTURAL RELATED POLICIES.......................................................................................................................................33 TABLE 19: INDICATOR 1-1.A PERCENTAGE OF TARGET CITIZENS WHO ARE CONFIDENT OF THEIR CONTRIBUTION IN THE DEVELOPMENT OF AGRICULTURAL POLICY ..........................................................35 TABLE 20: INDICATOR 1-1.B PERCENTAGE OF TARGET CITIZENS WHO ARE CONFIDENT OF THEIR ROLE IN BUDGET MONITORING................................................................................................................................36 TABLE 21: INDICATOR 3-1.A PERCENTAGE OF TARGET CITIZENS WHO CONSIDER THAT LOCAL AUTHORITIES RESPOND TO THEIR EXPRESSED PRIORITIES WITH REGARD TO SUSTAINABLE AGRICULTURE ...................37 TABLE 22: PERCENT OF PARTICIPANTS REPORTING PRIORITIES RESPONDED TO IN THE PAST 12 MONTHS.......38 TABLE 23: INDICATOR 3-1.B PERCENTAGE OF CSOS WHO CONSIDER THAT LOCAL AUTHORITIES RESPOND TO THEIR EXPRESSED PRIORITIES WITH REGARD TO SUSTAINABLE AGRICULTURE.........................................39 TABLE 24: SUMMARY OF CSO RESPONSES TO INDICATOR 3-1.B .........................................................................39 TABLE 25: AGRICULTURE POLICY ISSUES RESPONDED TO BY LOCAL AUTHORITIES IN THE PREVIOUS 12 MONTHS ....................................................................................................................................................................40 TABLE 26: IMPACT INDICATOR 1-NUMBER OF PUBLIC POLICIES INTRODUCED, ADOPTED, REPEALED, CHANGED OR IMPLEMENTED CONSISTENT WITH CITIZEN INPUT ...............................................................................40 TABLE 27: OUTCOME INDICATOR 1.2 FREQUENCY OF INTERACTIONS BETWEEN CITIZENS AND LOCAL AUTHORITIES ON AGRICULTURAL POLICIES AND BUDGET .........................................................................40 TABLE 28: OUTCOME INDICATOR 1.2 NUMBER OF ISSUES RELATED TO AGRICULTURAL POLICY RAISED BY CITIZENS AND ADDRESSED BY PUBLIC AUTHORITIES IN THE PAST 12 MONTHS ........................................40 TABLE 29: OUTCOME INDICATOR 2.1 NUMBER OF DECISIONS TAKEN BY TARGET DISTRICT AND SECTOR LEADERS WITH CSO’S INPUTS IN THE PAST 12 MONTHS............................................................................40 TABLE 30: OUTCOME INDICATOR 2.2 NUMBER OF DECISIONS MADE AT THE NATIONAL LEVEL WITH CSO’S INPUTS IN THE PAST 12 MONTHS ...............................................................................................................40 TABLE 31: PERCENTAGE PARTICIPATION BY FEMALES AND MALES, YOUTHS AND ADULTS IN THE EPGA PROJECT BY DISTRICT*...............................................................................................................................................42 TABLE 32: PERCENTAGE PARTICIPATION BY FEMALES AND MALES, YOUTHS AND ADULTS IN COMMUNITY COMMITTEES BY DISTRICT..........................................................................................................................43 TABLE 33: PERCENT OF SURVEY PARTICIPANTS REPORTING SEEING OR READING POSTERS ..............................44 TABLE 34: PERCENT OF SURVEY PARTICIPANTS REPORTING SEEING OR READING POSTERS ..............................46 TABLE 35: POLICY ISSUES RAISED DURING NATIONAL-LEVEL DIALOGUE MEETINGS...........................................49 TABLE 36: ACHIEVEMENT AGAINST PLANNED ACTIVITIES FOR PROJECT OUTCOME 1........................................51 TABLE 37: ACHIEVEMENT AGAINST PLANNED ACTIVITIES FOR PROJECT OUTCOME 2........................................52 5 TABLE 38: ACHIEVEMENT AGAINST PLANNED ACTIVITIES FOR PROJECT OUTCOME 3........................................53 6 List of Figures FIGURE 1: TRÓCAIRE’S GLOBAL THEORY OF CHANGE ..........................................................................................11 FIGURE 2: OUTCOME 1, OUTPUTS AND ACTIVITIES AS PER THE EPGA PROJECT PROPOSAL ...............................13 FIGURE 3: OUTCOME 2, OUTPUTS AND ACTIVITIES AS PER THE EPGA PROJECT PROPOSAL ...............................14 FIGURE 4: OUTCOME 3, OUTPUTS AND ACTIVITIES AS PER THE EPGA PROJECT PROPOSAL ...............................15 FIGURE 5: IMPACT AND OUTCOME LEVEL PROJECT INDICATORS MEASURED ....................................................20 FIGURE 6: PERCENT OF RESPONDENTS AWARE OF AGRICULTURAL POLICIES IN THEIR DISTRICT BY AGE AND GENDER.......................................................................................................................................................26 FIGURE 7: PERCENT OF RESPONDENTS AWARE OF AGRICULTURAL POLICIES IN THEIR DISTRICT, BY DISTRICT..27 FIGURE 8: INDICATOR 2.A PERCENTAGE OF TARGET BENEFICIARIES WHO ARE SATISFIED WITH THEIR ROLE IN THE IMPLEMENTATION OF AGRICULTURAL RELATED POLICIES BY AGE AND GENDER ..............................28 FIGURE 9: INDICATOR 2.A PERCENTAGE OF TARGET BENEFICIARIES WHO ARE SATISFIED WITH THEIR ROLE IN THE IMPLEMENTATION OF AGRICULTURAL RELATED POLICIES BY DISTRICT .............................................28 FIGURE 10: REPORTED MECHANISMS FOR PARTICIPATION IN IMPLEMENTATION OF AGRICULTURAL POLICIES BY AGE AND GENDER..................................................................................................................................29 FIGURE 11: REPORTED MECHANISMS FOR PARTICIPATION IN IMPLEMENTATION OF AGRICULTURAL POLICIES BY DISTRICT.................................................................................................................................................29 FIGURE 12: INDICATOR 2.B PERCENTAGE OF TARGET BENEFICIARIES WHO ARE SATISFIED WITH THEIR ROLE IN MONITORING OF IMPLEMENTATION OF AGRICULTURAL RELATED POLICIES BY AGE AND GENDER.........30 FIGURE 13: INDICATOR 2.B PERCENTAGE OF TARGET BENEFICIARIES WHO ARE SATISFIED WITH THEIR ROLE IN MONITORING OF IMPLEMENTATION OF AGRICULTURAL RELATED POLICIES BY DISTRICT........................30 FIGURE 14: REPORTED MECHANISMS FOR PARTICIPATION IN MONITORING THE IMPLEMENTATION OF AGRICULTURAL RELATED POLICIES .............................................................................................................31 FIGURE 15: REPORTED MECHANISMS FOR PARTICIPATION IN MONITORING THE IMPLEMENTATION OF AGRICULTURAL RELATED POLICIES BY DISTRICT.........................................................................................31 FIGURE 16: INDICATOR 2.C PERCENTAGE OF TARGET BENEFICIARIES WHO ARE SATISFIED WITH THEIR ROLE IN MONITORING OF BUDGETS ON AGRICULTURAL RELATED POLICIES BY AGE AND GENDER.......................32 FIGURE 17: INDICATOR 2.C PERCENTAGE OF TARGET BENEFICIARIES WHO ARE SATISFIED WITH THEIR ROLE IN MONITORING OF BUDGETS ON AGRICULTURAL RELATED POLICIES BY DISTRICT......................................33 FIGURE 18: REPORTED MECHANISMS FOR PARTICIPATION IN MONITORING THE BUDGETS OF AGRICULTURAL RELATED POLICIES BY AGE AND GENDER....................................................................................................34 FIGURE 19: REPORTED MECHANISMS FOR PARTICIPATION IN MONITORING THE BUDGETS OF AGRICULTURAL RELATED POLICIES BY DISTRICT...................................................................................................................34 FIGURE 20: INDICATOR 1-1.A PERCENTAGE OF TARGET CITIZENS WHO ARE CONFIDENT OF THEIR CONTRIBUTION IN THE DEVELOPMENT OF AGRICULTURAL POLICY BY AGE AND GENDER .......................35 FIGURE 21: INDICATOR 1-1.A PERCENTAGE OF TARGET CITIZENS WHO ARE CONFIDENT OF THEIR CONTRIBUTION IN THE DEVELOPMENT OF AGRICULTURAL POLICY BY DISTRICT ......................................35 FIGURE 22: INDICATOR 1-1.B PERCENTAGE OF TARGET CITIZENS WHO ARE CONFIDENT OF THEIR ROLE IN BUDGET MONITORING BY AGE AND GENDER ............................................................................................36 FIGURE 23: : INDICATOR 1-1.B PERCENTAGE OF TARGET CITIZENS WHO ARE CONFIDENT OF THEIR ROLE IN BUDGET MONITORING BY DISTRICT ...........................................................................................................36 FIGURE 24: INDICATOR 3-1.A PERCENTAGE OF TARGET CITIZENS WHO CONSIDER THAT LOCAL AUTHORITIES RESPOND TO THEIR EXPRESSED PRIORITIES WITH REGARD TO SUSTAINABLE AGRICULTURE BY AGE AND GENDER.......................................................................................................................................................37 FIGURE 25: INDICATOR 3-1.A PERCENTAGE OF TARGET CITIZENS WHO CONSIDER THAT LOCAL AUTHORITIES RESPOND TO THEIR EXPRESSED PRIORITIES WITH REGARD TO SUSTAINABLE AGRICULTURE BY DISTRICT ....................................................................................................................................................................37 FIGURE 26: PERCENT OF PARTICIPANTS REPORTING PRIORITIES RESPONDED TO IN THE PAST 12 MONTHS BY TOPIC BY AGE AND GENDER .......................................................................................................................38 FIGURE 27: PERCENT OF PARTICIPANTS REPORTING PRIORITIES RESPONDED TO IN THE PAST 12 MONTHS BY TOPIC BY DISTRICT ......................................................................................................................................38 FIGURE 28: PERCENT OF PARTICIPANTS REPORTING PRIORITIES RESPONDED TO IN THE PAST 12 MONTHS ....39 FIGURE 29: PERCENT OF SURVEY PARTICIPANTS REPORTING SEEING OR READING POSTERS ............................45 FIGURE 30: PERCENT OF SURVEY PARTICIPANTS REPORTING LISTENING TO EPGA RADIO PROGRAMS BY AGE AND GENDER...............................................................................................................................................46 7 FIGURE 31: PERCENT OF SURVEY PARTICIPANTS REPORTING LISTENING TO EPGA RADIO PROGRAMS BY DISTRICT......................................................................................................................................................46 Acronyms CEJP Episcopal Commission for Justice and Peace CSO Civil Society Organization DAC Development Assistance Committee EPE End of Project Evaluation EPGA Enhancing Participatory Governance and Accountability FGD Focus Group Discussion GDPR General Data Protection Regulations JADF Joint Action Development Forum KII Key Informant Interview MDC Mobile Data Collection MEL Monitoring, Evaluation and Learning MINAGRI Ministry of Agriculture and Animal Resources OECD Organization for Economic Cooperation and Development PIRS Performance Indicator Reference Sheet PSTA 4 Strategic Plan for Agriculture Transformation 2018-24 RAB Rwanda Agricultural Board RDO Rwanda Development Organization SEDO Social Economic and Development Officer SMART Specific, Measurable, Achievable, Relevant and Time-bound ToC Theory of Change ToT Training of Trainers UNICOOPAGI The Union des Cooperatives Agricoles Intégrées VAC Village Agricultural Committee Key Terms Girinka One cow per family program, with the aim of reducing childhood malnutrition and increasing household income of poor farmers. Imihigo Government performance contracts used for periodic appraisals. Smart Nkunganire A supply chain management system to digitalize the end to end value chain of the Agro-Input Subsidy program. Survey CTO A mobile data collection platform. Twigire Muhinzi An agricultural extension model that is a combination of Farmer Field Schools and the Farmer Promoter approach. Umuganda A monthly community workday for all Rwandan citizens that takes place on the last Saturday of each month and can be a platform for addressing community issues. 8 1. Executive Summary The Enhancing Participatory Governance and Accountability in Rwanda project is a three year project (April 2017 to April 2020) implemented by Trócaire and funded by USAID with the specific objective of promoting the effective participation of citizens, especially women and youth, and CSOs in the development, implementation and monitoring of agricultural policy and budget in five districts in order to ensure that they respond to the priorities of vulnerable people and contribute to sustainable and inclusive development in Rwanda. In accordance with Trócaire’s global Theory of Change (ToC), the project aimed to empowered citizens, particularly women and youths, to be able to engage in agricultural policy and budget monitoring through capacity development, develop local village level structures to support community-based advocacy, strengthen the capacity of civil society to support citizen engagement and sensitize local government in participatory governance. The project was implemented by Trócaire, in partnership with the Rwanda Development Organization (RDO), the Episcopal Justice and Peace Commission (CEJP) and the Union des Cooperatives Agricoles Intégrées (UNICOOPAGI), implemented the EPGA project in 63 villages, across 14 sectors in five districts of Rwanda. The districts covered by the project are Gakenke and Rulindo Districts in Northern Provence and Nyanza, Nyamagabe and Nyaruguru in Southern Province. The purpose of the final evaluation was to measure progress in achieving the project objectives since the start of the EPGA project and provide an opportunity to highlight key successes, lessons learned and recommendations. In addition, the evaluation also assessed the project status against the following OCED/DAC criteria: 1) Relevance of the project design, 2) Efficiency of the project delivery, 3) Effectiveness of management structures, 4) Impact and 5) Sustainability. The final evaluation employed a mixed-methods approach, with a focus on generating comprehensive quantitative and qualitative information. A quantitative survey of 362 project beneficiaries was undertaken using tablet-based data collection. Qualitative data collection included Focus Group Discussions and Key Informant Interviews with project beneficiaries including Women, Youth, Community Committee members, members of civil society organizations, local government, project implementing partners and Trócaire staff. Due to the COVID-19 pandemic and subsequent government measures to limit the spread, all field work ceased after one week of data collection, with the remaining data collection taking place by phone. At the baseline, those surveyed reported that limited knowledge in the area of modern farming techniques and limited access to agricultural information prevented their participation in the implementation and monitoring of agricultural policies.1 However, findings from the final evaluation highlight that the project contributed to a visible increase in engagement of citizens and CSOs in advocacy on agricultural policy implementation as a result of improved knowledge of agricultural policies and rights to participation. End values for impact and outcome level indicators demonstrate an increase in citizen satisfaction and confidence with the role that they play in the implementation of and the monitoring of implementation and budgets for agricultural policies. Of those surveyed, 95% were satisfied with the roles they played in both the implementation of and monitoring of agricultural policies as compared to 93% and 77%, respectively, at baseline and 91% were confident in their contribution toward the development of agricultural policies as compared to 81% at baseline. Both satisfaction and confidence were highest for male participants, with both male youth and adults reporting 97% satisfaction and 95% confidence with the role that they play whereas female adults reported 96% satisfaction and 90% confidence and female youth reported 91% satisfaction and 88% confidence in the role that they played in implementation of policies. Of those surveyed, 64% reported attending meetings where policies were discussed and 61% reported attending meetings where they were able to comment on policies. Only 3% of surveyed participants reported that there were no opportunities for participation. While no male youth reported a lack of opportunities for participation, 5% of female youth reported that there were no opportunities for their participation, reflecting societal norms surrounding the role of women in rural communities. 1 Trócaire (2017) Baseline survey for “Enhancing participatory governance and accountability” project 9 While citizens reported that budgets were too complex for them to understand, they still reported high satisfaction with their role in monitoring of budgets with 83% of those surveyed reporting satisfaction, an increase from 5% at baseline, and 75% reporting being confident in their role in monitoring budgets, but from 29% at baseline. Male youth reported the greatest satisfaction and confidence at 91% and 86% respectively, while female youth reported the lowest at 83% and 62%, with female youth noting that they still lacked sufficient understanding and confidence to contribute, which is also reflected in societal gender roles where women are not encouraged to participate. With increased engagement with local leaders through community, sector and district level meetings to resolve issues related to implementation of agricultural policies as well as the two dialogue meetings that were able to address policy implementation issues at a national level, both CSOs and citizens reported that they felt that local authorities responded to their priorities, with 88% of citizens reporting that the local authorities responded to their priorities, as compared to 57% at baseline and CSOs reporting that local authorities respond to their priorities 88% of the time as compared to 48% at baseline. Table 1: Summary of EPGA Project Impact and Outcome Indicators Impact and Outcome Indicators Data Source Adult (>30 Yrs) Youth (18-30 Yrs) Total Female Male Female Male Indicator 2.A Percentage of target beneficiaries who are satisfied with their role in the implementation of agricultural related policies Target 55% 50% 77% 70% Baseline 59% 66% 94% 100% 93% Midterm 81% 86% 48% 58% n/a Final 96% 97% 91% 97% 95% Indicator 2.B Percentage of target beneficiaries who are satisfied with their role in monitoring of implementation of agricultural related policies Target 65% 50% 94% 94% Baseline 51% 37% 92% 93% 77% Midterm 83% 84% 66% 88% n/a Final 94% 97% 95% 95% 95% Indicator 2.C Percentage of target beneficiaries who are satisfied with their role in monitoring of budgets on agricultural related policies Target 23% 20% 23% 25% Baseline 5% 5% 5% 8% 5% Midterm 34% 38% 36% 33% n/a Final 81% 87% 83% 91% 83% Indicator 1-1.A Percentage of target citizens who are confident of their contribution in the development of agricultural policy Target 62% 62% 86% 93% Baseline 58% 58% 84% 92% 81% Midterm 66% 68% 73% 69% n/a Final 90% 95% 88% 95% 91% Indicator 1-1.B Percentage of target citizens who are confident of their role in budget monitoring Target 25% 20% 60% 65% Baseline 18% 13% 47% 58% 29% Midterm 53% 50% 50% 54% n/a Final 77% 76% 62% 86% 75% Indicator 3-1.A Percentage of target citizens who consider that local authorities respond to their expressed priorities with regard to sustainable agriculture Target 60% 55% 65% 80% Baseline 58% 48% 54% 77% 57% Midterm 72% 77% 76% 81% n/a Final 92% 87% 79% 86% 88% CSOs Indicator 3-1.B Percentage of CSOs who consider that local authorities respond to their expressed priorities with regard to sustainable agriculture Target 52% Baseline 48% Midterm n/a Final 88% 10 The project’s success lies in educating citizens on their rights to participate in the agricultural policy dialogue, developing a forum for their advocacy on a local level through the establishment of Community Committees, developing a receptive local leadership through awareness of participatory governance and creating opportunities for advocacy to take place at district and national levels. In addition, the use of mobile data collection to capture advocacy needs and outcomes ensured that the project could more swiftly identify common issues and concerns that were not able to be resolved at a local level and bring them to a national audience. As a result, policies, such as timely access to improved seeds and fertilizers were addressed at the national level through National Dialogues, the impact of which can be seen at the level of the farmers who are now reporting greater yields and greater confidence in their role in advocating for policy implementation. By increasing engagement, and awareness of rights there is space now available for ensuring policies are more responsive to needs. The report also presents a number of recommendations based on the findings including: • The need for further training and capacity building opportunities for citizens and CSOs throughout the life of the project, with extra time allocated to understanding budgets; • Increased integration of program components that meet the specific needs and priorities of youth and women, taking into account and addressing societal norms that may limit participation; • The recognition of the contribution of the Community Committees by government as a needed forum for sharing of needs and priorities surrounding the development and implementation of agricultural policies; • The need to investigate means for the sustainable continuation of mobile data collection, including increasing the capacity of implementing partners and CSOs to continue using mobile platforms for developing evidenced-based policy recommendations beyond the life of the project; and • The recognition that rural citizens want opportunities to participate in policy discussions and the government should be encouraged to shift the focus of policy making from a top-down approach to a more collaborative, participatory approach that engages with CSOs and rural citizens in the identification and formulation of agricultural policies. 11 2. Introduction and Background Rwanda’s economy is strongly dependent upon agriculture as it is a significant driver of economic growth in the country. However, Rwanda is one of the most densely populated countries in the world, with a population density of 498.72 people per square kilometer. Agriculture is dominated by small-scale subsistence farms relying on traditional practices that leave them vulnerable to climate change and other shocks. Currently, 96% of rural households are directly reliant upon agriculture as their main or only source of income, cultivating an average of 0.6 hectares3. With an annual growth rate of 2.6%4, there will be increasing pressure for sustainable land use. In order to develop a resilient agricultural sector, farmers, particularly small-scale farmers, need access to technology and practices to sustain agricultural growth. Women and youth are the most vulnerable populations, as 92% of women in rural areas engage in the agricultural sector, as compared to 77% of men, and working age youth make up 77% of the rural population.5 Women are also most vulnerable to climate change and external shocks as they often have no other source of income. While some mechanisms have been put in place through recent government decentralization processes to ensure women and youth engage in decision making, they often lack the skills and an enabling environment to do so. The Enhancing Participatory Governance and Accountability in Rwanda is a three year project (April 2017 to April 2020) co-funded by USAID and Trócaire with the specific objective of promoting the effective participation of citizens, especially women and youth, and CSOs in the development, implementation and monitoring of agricultural policy and budget in five districts in order to ensure that they respond to the priorities of vulnerable people and contribute to sustainable and inclusive development in Rwanda. In accordance with Trócaire’s global Theory of Change (ToC) which proposes that change happens at and across four key levels: individual, community, civil society, and institutional levels, the project aimed to strengthen the existing governance structures in five districts, targeting 3,675 citizens by providing them with training on their roles and responsibilities in terms of policy and budget monitoring and advocacy, primarily related to agriculture. In addition, the project planned to increase the capacity of 50 CSOs to analyze policies and budgets, with a focus on agricultural programs to ensure that they take into account the needs of vulnerable populations, particularly women and youth. Finally, the project planned to improve the capacity of local authorities in the five target districts to better engage with citizens and civil society so that they can better respond to their priorities during planning and budgeting activities. The project results framework can be found in Annex 6. Trócaire, in partnership with the Rwanda Development Organization (RDO), the Episcopal Justice and Peace Commission (CEJP) and the Union des Cooperatives Agricoles Intégrées (UNICOOPAGI), implemented the EPGA project in 63 villages, across 14 sectors in five districts of Rwanda. The districts covered by the project were Gakenke and Rulindo Districts in Northern Provence and Nyanza, Nyamagabe and Nyaruguru in Southern Province. 2 World Bank https://databank.worldbank.org/ 3 MINAGRI (June 2017) National Agriculture Policy 4 World Bank https://databank.worldbank.org/ 5 MINAGRI (June 2017) National Agriculture Policy Figure 1: Trócaire’s Global Theory of Change 12 Table 2: Geographic Coverage of the EPGA Project Province District Sector Cells Villages Northern Province Gakenke Mataba 3 3 Minazi 4 4 Janja 4 4 Rulindo Base 3 3 Cyungo 3 3 Rukozo 4 4 Southern Province Nyanza Nyagisozi 5 5 Rwabicuma 6 6 Nyamagabe Gatare 5 5 Kitabi 5 5 Uwinkingi 6 6 Nyaruguru Mata 5 5 Kivu 5 5 Ruramba 5 5 Total: 5 Districts 14 Sectors 63 Cells 63 Villages 2.1 EPGA Project Outcome 1 The first expected outcome of the EPGA project was for empowered citizens, particularly women and youths, to be able to engage in agricultural policy and budget monitoring in 5 districts and hold public authorities accountable. Trócaire proposed to achieve this outcome by building awareness and capacity of citizens around their roles and rights and by forming Community Committees6 at the village level responsible for advocacy around sustainable agricultural policy and budgeting. This outcome is in line with Trócaire’s first two pillars of their Global ToC supporting empowered individuals and resilient and sustainable communities. Individual Level Change: In the first year of the project, 3,595 local citizens and 830 village and cell leaders were trained on their roles and responsibilities for agricultural policy and budget monitoring. This was accompanied by advocacy messages including posters and booklets detailing citizens’ rights and responsibilities, radio broadcasts and translation of the most relevant policy documents to make them accessible at the community level. Citizens were also invited to participate in exchange meetings with local leaders on agricultural planning. A documentary film on the role of citizen participation was made in Year 3 of the project. Community Level Change: In the first year of the project, 63 Community Committees were formed, one within each target village. Five members were selected from the local village for each Community Committee and were intended to serve for three years in their position. Community Committee members were trained on their role and responsibilities and were supported to conduct feedback meetings with the citizens in their communities. Each Community Committee was supplied with a smartphone with SurveyCTO software installed. Monitoring data collected by the Committee was entered on the smartphone and is shared with Trócaire and implementing partners in real time. 6 The terms Village Agricultural Committee (VAC) and Community Committee were used interchangeably in the program documents and in discussions. However, the VAC is a government institution made up of local leaders, whereas the Community Committee is made up of farmers and is the group formed by the EPGA project. 13 Figure 2: Outcome 1, Outputs and Activities as per the EPGA Project Proposal Outcome 1: Empowered citizens, particularly women and youths, who are able to engage in agricultural policy and budget monitoring in 5 districts and hold public authorities accountable Output 1.1: Increased knowledge of 3675 citizens and 819 (900) local authorities on their role and responsibilities in terms of agriculture policy and budget monitoring related to agriculture in terms of policy and budget monitoring related to agriculture Activity 1.1.1: One training of 3675 citizens especially women and youth and 819 village and cell leaders, on their roles and responsibilities in terms of policy and budget monitoring and Advocacy regards to agriculture issues. Activity 1.1.2: 12 feedback meetings conducted by community committee members Activity 1.1.3: Organize six exchange meetings between citizens and local leaders around citizen participation in planning especially in agriculture planning Output 1.2: Increased awareness of 3675 citizens’ roles and responsibilities through production and dissemination of information and education materials Activity 1.2.1: Translate and design user friendly 5000 copies of summarized versions of agricultural policies over 3 years Activity 1.2.2: Produce and distribute 14,000 posters on citizens’ rights and responsibilities, 5000 copies of booklets on citizen’s role and responsibilities in terms of policy and budget monitoring regards to agriculture Activity 1.2.3: Produce and disseminate 36 radio broadcasts and 1 documentary film on citizen participation in policy monitoring especially in the domain of agriculture and budget monitoring Output 1.3: 63 Community committees made up of five people are established and operational Activity 1.3.1: Establish 63 community committees at village level for policy and budget monitoring and advocacy with regard to sustainable agriculture Activity 1.3.2: Organize one training for community committees on policy, budget monitoring and advocacy Output 1.4: Improved mechanism of exchange of information between community committees, partner program staff and Trócaire Activity 1.4.1: 72 telephones with adequate software for collecting and sharing information are purchased and distributed to each of 63 community committees, and 9 staff on agricultural policy and budget monitoring Activity 1.4.2: One training community committees, and program staff on telephone software operations 2.2 EPGA Project Outcome 2 The second project outcome is increased capacity of CSOs to engage in policy analysis, budget monitoring and advocacy and ability to use a collective approach to influence decisions related to sustainable agriculture. This outcome is in line with Trócaire’s third pillar of their Global ToC for a strong and representative civil society. Under the project, CSOs were trained on policy budget, monitoring and advocacy and were supplied with training materials and a strategy guide in order to better support their stakeholders. 14 Figure 3: Outcome 2, Outputs and Activities as per the EPGA Project Proposal Outcome 2: Increased capacity of civil society to engage in policy analysis, budget monitoring and advocacy and ability to use collective approach to influence decisions related to sustainable agriculture Output 2.1: Training manual developed for CSOs covering policy and budget monitoring and advocacy; Strategy guide in policy and budget monitoring and advocacy and recommendation papers about priority needs with regards to agriculture policies and programs are produced Activity 2.1.1: Develop a training manual on policy and budget monitoring and advocacy Activity 2.1.2: Develop a strategy guide in policy and budget monitoring and advocacy and guidelines for agriculture Output 2.2: 50 Civil Society Organizations in five Districts have sufficient capacity for policy analysis, budget monitoring and advocacy and are aware of participative approach for decision influencing Activity 2.2.1: One, five-day training of the trainers targeting the three implementing partners and Trócaire staff on policy and budget monitoring and advocacy Activity 2.2.2: One training of civil society organizations based in the districts on policy and budget monitoring and advocacy with focus on agricultural policy and gender analysis and integration Activity 2.2.3: Conduct three policy dialogue meetings and advocacy activities at district level between communities, civil society and public authorities Activity 2.2.4: Conduct two policy dialogue meetings and advocacy at national level with stakeholders including civil society and national government officials Activity 2.2.5: Facilitate six exchanges of experiences and learning among stakeholders intervening in agriculture and policy monitoring through JADF and other networks in country and in neighboring countries Activity 2.2.6: Document and disseminate lessons learnt and best practices in local governance, policy and budget monitoring Activity 2.2.7: Conduct one research on the involvement of citizens particularly women in local governance, policy and budget monitoring in agriculture area Output 2.3: Increased engagement on participatory governance and agricultural policy Activity 2.3.1: Organize 2 radio broadcasts on agricultural policies by using most popular radio stations 2.3 EPGA Project Outcome 3 The final expected outcome is increased knowledge of local authorities on participatory governance and are committed to address citizen’s concerns corresponds with Trócaire’s fourth pillar of just and accountable institutions. Under this outcome, Trócaire developed a training module and 844 local government actors at the sector and district level participated in a two-day a training on participatory governance in order to better support citizen and CSO engagement in sustainable agriculture. Accountability meetings were also organized at the district level to follow￾up on the expressed needs of citizens. 15 Figure 4: Outcome 3, Outputs and Activities as per the EPGA Project Proposal Outcome 3: Increased knowledge of local authorities on participatory governance and are committed to address citizens' concerns Output 3.1: Increased knowledge of local authorities on participatory governance and are committed to address citizens’ concerns Activity 3.1.1: One training of local authorities and community leaders in participatory planning and budgeting Activity 3.1.2: Develop 100 participatory governances modules targeted at local authority officials Activity 3.1.3: Organize six accountability meetings with local authorities at district level to follow up the implementation of recommendations submitted by CSOs concerning the expressed needs of citizens 2.4 Implementing Partners The EPGA project was implemented in collaboration with three implementing partners: RDO, CEJP and UNICOOPAGI. In addition to implementation, partners also contributed to project monitoring as members of the Project Steering Committee. RDO: The Rwanda Development Organization is a national non-governmental organization founded in 1995 to support Rwandan initiatives in social and economic development. The mission of RDO is to empower communities through human and institutional capacity development, food security and nutrition and social promotion and protection for vulnerable groups. RDO supported the EPGA project through training of CSOs on issue-based advocacy and organizing policy dialogues at the national level. CEJP: The Episcopal Commission for Justice and Peace is a commission of the Catholic Church of Rwanda. CEJP was the lead EPGA project partner in Nyanza, Gakenke and Rulindo Districts working in the areas of governance. CEJP specifically works to ensure the accountability of local leaders and public institutions towards citizens, particularly women and youth, through transparent governance, accountability of public authorities and inclusiveness of public policy and budgeting processes. UNICOOPAGI: The Union des Cooperatives Agricoles Intégrées is a union of agricultural cooperatives headquartered in Nyamagabe District in Southern Province. UNICOOPAGI was formed in 1991 to foster community development by empowering farmers. As an implementing partner, UNICOOPAGI supported mobilization and capacity building in Nyamagabe and Nyaruguru Districts. 16 3. Evaluation Purpose and Objectives The purpose of the final evaluation is to measure progress in achieving project objectives since the start of the EPGA project and provide an opportunity to highlight key successes, lessons learned and recommendations. In addition, the evaluation also assessed the project status against the following OECD/DAC criteria: Table 3: Evaluation Criteria Relevance of the project design Establishing the extent to which the project meets predefined indicators and their validity and relevance to partners, and key stakeholders. Efficiency of the project delivery Establishing how available inputs have been utilized with the highest possible effect, the quality of program management and the factors affecting efficiency. Effectiveness of the project delivery (Management) Establishing the degree to which the project has been able to achieve the objectives as outlined in the project document and the factors affecting effectiveness in delivery. Impact of the project Examining what impact the project is having on both direct and indirect beneficiaries. Sustainability of the project Measuring whether the benefits of the project are likely to continue after the project and the factors likely to affect the project’s sustainability. The evaluation was designed in order to address the following evaluation questions as pre-defined by Trócaire. Table 4: Evaluation Questions # Evaluation Criteria Evaluation Questions 1 Relevance of the project design 1.1 What are the actual project end values for the impact and outcome indicators as set out in the project results framework? 1.2 How appropriate and useful were the indicators described in the project results framework in assessing the project’s progress? Were the targeted indicator values realistic and could they be easily tracked? 1.3 What are the recommendations and rationale for changes in program design? 2 Efficiency of the project delivery 2.1 Has the project been able to achieve its planned objectives? What were the main constraints to achieving the planned objectives? 2.2 Were the planned resources adequate? What steps were taken to ensure that the project participants got the most out of the resources that were deployed? 2.3 Have project funds and activities been delivered in a timely manner? 17 # Evaluation Criteria Evaluation Questions 3 Effectiveness of the project delivery (specifically management arrangements) 3.1 Did project management facilitate good results and efficient delivery? 3.2 Is there a clear understanding of roles and responsibilities by all parties involved? 3.3 Did Trócaire provide a conducive environment for partners to effectively implement the project? Did project implementing partners receive appropriate technical and administrative support from Trócaire and USAID? 3.4 How effectively did the project management monitor project performance and results? 3.5 How have stakeholders been involved in project implementation? How effective has the project been in establishing community and national level ownership? 4 Impact of the project 4.1 What have been the intended and unintended impact in the target population that can be directly or indirectly attributed to the project? 4.2 How did use of technology enable the project to achieve intended results? What has been the intended and unintended use of technology by the target population? 4.3 Do rural people and farmers have a greater voice in agricultural policy decisions? How is that visible? How has government policy or strategy changed at the micro, meso or macro level for project participants? 5 Sustainability of the project 5.1 What are the project interventions that are likely to remain in the community after the project lifetime? What measures have been taken to ensure this? 18 4. Evaluation Methodology 4.1 Study Design 4.1.1 Study Design Summary The final evaluation employed a mix-methods approach, with a focus on generating comprehensive quantitative and qualitative information. The methodology included: Desk Review • Program documents including annual reports, studies and Lessons Learnt, government reports and relevant research and reports Quantitative • Survey of program participants Qualitative • Focus group discussions (FGDs) with female participants, youth participants, VAC members and CSOs. • Phone-based interviews with female participants, youth participants, VAC members and CSOs. • Key informant interviews (KIIs) with district, sector and cell officials • Semi-structured interviews with project staff, implementing partner staff and government partners 4.1.2 Comments on Revisions to Study Design Due to the COVID-19 pandemic and policies put in place in the country of Rwanda to limit the spread of the virus announced mid-way through field based data collection, the study design was modified to ensure the continuation of data collection while remaining in compliance with government policies and protecting both Trócaire beneficiaries and the data collection team. Revisions to the design are noted where applicable and included the use of telephone-based surveys and interviews to replace in-person data collection. 4.1.3 Desk Review A literature review of existing program documents and relevant external documents was undertaken in order to inform the design and the analysis of findings. In addition, a review of program implementation reports, monitoring data and studies were used to respond to the key evaluation questions and validate the quantitative and qualitative data collected as a part of this evaluation. The documents reviewed included: • Key Program Documents: Project proposal, baseline report, midterm evaluation, research reports, case studies, lessons learnt documents and donor reports, specifically: Table 5: List of Key Program Documents Reviewed RDO (2020) Enhancing Participatory Governance and Accountability Project: Lessons Learnt and Best Practice Trócaire (2016) EPGA Technical Proposal Trócaire (2016) Rwanda Country Strategic Plan 2016 – 2020 Trócaire (2017) Baseline survey for “Enhancing participatory governance and accountability” project Trócaire (2017) EPGA Year 1 Annual Report to USAID Trócaire (2019) EPGA Year 3 Annual Report to USAID 19 Trócaire (2019) Key Recommendations from the National Policy Dialogue on Agriculture Conducted on December 6th Trócaire (2019) Midterm Evaluation Report 2019 Trócaire (2020) Case study: Assessing the impacts of the use of SurveyCTO and the response of the Rwandese government Trócaire (2020) The Involvement of citizens, particularly women, in local agricultural governance, policy and budget monitoring in Rwanda: A study from Nyamagabe, Nyaruguru, Nyanza, Gakenke and Rulindo Districts Trócaire (December 2019) Agricultural issues paper to inform national policy dialogue; Trócaire (November 2017) The Role of Citizens and Local Entities in Policy and Budget Monitoring and Advocacy Related to Agriculture, English Version Trócaire (2018) EPGA Year 2 Annual Report to USAID • Government and Policy Documents: relevant research, national policy documents, government statistics, and agricultural policy documents, specifically: Table 6: List of Key External Documents Reviewed ActionAid Rwanda (October 2015) Annual Analysis of Rwanda's Agricultural Budget Expenditure 2015-16 Franzel, S., Kinyua, H., Rucibigango, M., Davis, K., and Makh, S. (2019) Youth in Extension and Advisory Services: Rwanda. Developing Local Extension Capacity Project Gender Monitoring Office (March 2019) The State of Gender Equality in Rwanda from Transition to Transformation International Alert (2018) Towards Sustainable Agriculture: An analysis of farmers' participation in agricultural programmes in Rwanda MINAGRI (June 2017) National Agricultural Policy MINAGRI (June 2018) Strategic Plan for Agriculture Transformation 20018-24 MINECOFIN (2012) Vision 2020 MINECOFIN (2017) 7 Years Government Programme: National Strategy for Transformation (NST1) 2017-2024 Ministry of Gender and Family Promotion (July 2010) National Gender Policy Never Again Rwanda (2018) Local Government Imihigo Process: Understanding the Factors Contributing to Low Citizen Participation NISR (2017) Agricultural Household Survey ODI (2006) How Civil Society Can Be More Effective Republic of Rwanda (2003) Rwandan Constitution Republic of Rwanda (May 2019) Beijing +25 Rwanda Country Report RGB (2018) Rwanda Governance Scorecard, 5th Edition Rwanda Governance Board (June 2018) Rwanda Governance Review: Good Governance and Decentralization in Rwanda The New Times (14 February 2017) Govt seeks more youth engagement in agriculture UNESCO Rwanda Country Statistics (http://uis.unesco.org/en/country/rw) USAID, FHI360 and ICNL (2018) The 2018 CSO Sustainability Index for Rwanda World Bank Databank (https://databank.worldbank.org/) 20 4.1.4 Quantitative Survey The quantitative survey was designed to gather the data necessary to measure impact and outcome level indicators as per the EPGA Results Framework and be comparable with data collected during the baseline and midterm evaluations. Figure 5: Impact and Outcome Level Project Indicators Measured Impact Indicator 2: Percentage of target beneficiaries who are satisfied with their role in the implementation and monitoring of agriculture policy and related budget. Based on the challenge of measuring this indicator, the EPGA project refined the indicator into three separate measures at the baseline: 1. Percentage of target beneficiaries who are satisfied with their role in the implementation of agricultural related policies. 2. Percentage of target beneficiaries who are satisfied with their role in monitoring of implementation of agricultural related policies. 3. Percentage of target beneficiaries who are satisfied with their role in monitoring of budgets on agricultural related policies. Outcome Indicator 1.1-1: Percentage of target citizens who are confident of their contribution in the development of agricultural policy and budget monitoring. Based on the challenge of measuring this indicator, the EPGA project refined the indicator into two separate measures: 1. Percentage of target citizens who are confident of their contribution in the development of agricultural policy. 2. Percentage of target citizens who are confident of their role in budget monitoring. Outcome Indicator 3.1-1: Percentage of citizens, especially women and youth, and members of civil society who consider that local authorities respond to their expressed priorities with regard to sustainable agriculture. The EPGA project refined the indicator into two separate measures: 1. Percentage of target citizens who consider that local authorities respond to their expressed priorities with regard to sustainable agriculture. 2. Percentage of CSOs who consider that local authorities respond to their expressed priorities with regard to sustainable agriculture. Where feasible, quantitative survey questions measuring the aforementioned indicators were identical or similar to those asked during midterm evaluation to ensure comparability of findings. The quantitative beneficiary survey can be found in Annex 2.1. 4.1.5 Qualitative Data Collection The qualitative component of the evaluation consisted of FGDs and telephone-based interviews with program beneficiaries and Key Informant Interviews (KIIs) with Trócaire staff, implementing partners and local government partners. Qualitative data was collected to address the following evaluation questions: 21 Table 7: Qualitative Data Sources Evaluation Questions FGD/ Interviews Interviews Women Youth VAC7 CSO Local Gov. Imp. Partners Trócaire 1. Relevance 1.1 What are the actual project end values for the impact and outcome indicators as set out in the project results framework? Measured via quantitative survey 1.2 How appropriate and useful were the indicators described in the project results framework in assessing the project’s progress? Were the targeted indicator values realistic and could they be easily tracked? 1.3 What are the recommendations and rationale for changes in program design? 2. Efficiency 2.1 Has the project been able to achieve its planned objectives? What were the main constraints to achieving the planned objectives? 2.2 Were the planned resources adequate? What steps were taken to ensure that the project participants got the most out of the resources that were deployed? 2.3 Have project funds and activities been delivered in a timely manner? 3. Effectiveness 3.1 Did project management facilitate good results and efficient delivery? 3.2 Is there a clear understanding of roles and responsibilities by all parties involved? 3.3 Did Trócaire provide a conducive environment for partners to effectively implement the project? Did project implementing partners receive appropriate technical and administrative support from Trócaire and USAID? 3.4 How effectively did the project management monitor project performance and results? 3.5 How have stakeholders been involved in project implementation? How effective has the project been in establishing community and national level ownership? 4. Impact 4.1 What have been the intended and unintended impact in the target population that can be directly or indirectly attributed to the project? 4.2 How did use of technology enable the project to achieve intended results? What has been the intended and unintended use of technology by the target population? 4.3 Do rural people and farmers have a greater voice in agricultural policy decisions? How is that visible? How has government policy or strategy changed at the micro, meso or macro level for project participants? 5. Sustainability 5.1 What are the project interventions that are likely to remain in the community after the project lifetime? What measures have been taken to ensure this? 7 “VAC” refers to Community Committees established as a component of the EPGA project 22 4.2 Sampling Procedures 4.2.1 Quantitative Survey The survey reached 362 respondents from the 3,675 primary beneficiaries, for a 95% confidence interval and 5% margin of error. The sample size calculation used was: 𝑛 = 𝑁𝑥 ((𝑁 − 1)𝑒2 + x) 𝑤ℎ𝑒𝑟𝑒 𝑥 = 𝑍𝑎2(𝑝(100 − 𝑝)) where n is the sample size, N is the total population (3,675), e is the margin of error (5%), Za is the factor that corresponds with the desired significance level, in this case a 95% significance level has a Za of 1.96, and p is the level of prevalence, assumed to be 50%. Participants were selected using stratified sampling, first identifying two sectors per district at random for inclusion in the survey framework, then using a random number to select participants from the list of beneficiaries per sector proportional to the total number of beneficiaries per sector. Lists of beneficiaries were provided for each district by Trócaire implementing partners. As a result of changes in the sampling design from in-person surveys to telephone-based surveys halfway through the data collection and the inability of the survey team to obtain working telephone numbers for some beneficiaries, the proportion of participants responding to the survey in Rulindo District was less than for other districts. Table 8: Quantitative Survey Sampling District Sector Total Number Beneficiaries Number of Beneficiaries Planned In-Person Surveys Conducted Phone-Based Surveys Conducted Nyaruguru Mata 221 35 36 Ruramba 357 42 42 Nyamagabe Kitabi 355 42 43 Uwinkingi 424 50 50 Nyanza Rwabicuma 446 53 57 Nyagisozi 367 44 41 Gakenke Mataba 221 26 26 Minazi 295 35 32 Rulindo Base 247 29 19 Cyungo 241 29 16 Total 3,247 383* 228 134 Total Surveys: 362 *348 participants were needed to obtain a 95% confidence interval with a 5% margin of error in line with Trócaire’s evaluation guidance. A 10% oversampling was planned, and ultimately 383 participants were selected to ensure sufficient level of response. 4.2.2 Focus Group Discussions Focus group discussions were conducted at the sector level in those sectors selected to participate in the quantitative survey. Participants were randomly selected from the pool of survey participants. FGDs were conducted with four project target groups: women, youth, Community Committee or “VAC” members and CSO members. Given time constraints and the project’s emphasis on women and youth, male beneficiary FGDs were not conducted. FGDs were conducted with 6-8 participants from each category in mixed gender groups, aside from the “women” FGDs which consisted of female participants only. With the change in sampling design, FGDs were revised to KIIs with participants from the same four project target groups. In each remaining district, five participants per target group were randomly selected from the list of beneficiaries for participation in the KII. FGD question guides were minimally revised for the interview format to ensure comparability of findings between FGDs and KIIs. See Annexes 2.2-2.4. 23 Table 9: FGD/KII Sampling District Sector Women Youth VAC8 CSO FGD Phone KII FGD Phone KII FGD Phone KII FGD Phone KII Nyaruguru Mata 1 1 Ruramba 1 1 Nyamagabe Kitabi 1 1 1 Uwinkingi 1 Nyanza Rwabicuma 1 1 Nyagisozi 3 5 Gakenke Mataba 3 2 3 Minazi 1 1 2 2 Rulindo Base 3 Cyungo 1 Total: 2 7 3 3 2 14 3 2 4.2.3 Key Informant Interviews with Local Government Local government partners were invited to participate in KIIs at the sector and district levels. Local government partners includes the Sector Economic Development Officer (SEDO), the Sector Agronomist and the District Agronomist. With changes to the sampling design, in-person interviews were revised to take place by telephone mid-way through data collection. Table 10: Local Government KIIs District Sector SEDO Sector Agronomist District Agronomist In person Phone In Person Phone In Person Phone Nyaruguru Mata 1 n/a Ruramba 1 Nyamagabe Kitabi 1 1 Uwinkingi 1 Nyanza Rwabicuma 1 1 1 Nyagisozi Gakenke Mataba 1 1 1 Minazi Rulindo Base 1 Cyungo 1 Total: 3 2 3 2 2 1 4.2.4 National Level and Project Implementation Key Informant Interviews Trócaire identified national level actors to participate in KIIs to contribute to the final evaluation. Unfortunately, due to the COVID-19 pandemic and ensuing government policy of restricted movement in Rwanda, it was not possible to conduct national level interviews with the Ministry of Agriculture and Animal Resources (MINAGRI) and Rwanda Agricultural Board (RAB) or with identified INGOs and USAID. A combination of telephone-based and in-person interviews were conducted with implementing partners: RDO, CEJP and UNICOOPAGI. 8 “VAC” refers to Community Committees established as a component of the EPGA project 24 4.3 Data Collection Procedures and Data Analysis 4.3.1 Quantitative Survey Quantitative Survey questions were developed in English and translated into Kinyarwanda. The survey was then digitized using KoBo Toolbox and loaded onto tablets for use by the data collection team. The data collection team participated in a training on the tools and Trócaire’s informed consent procedures followed by testing of the survey tool. Data was uploaded and exported to Excel for quality assurance purposes, daily analysis and troubleshooting. All data analysis was conducted in Excel. The Survey Form can be found in Annex 2.1. 4.3.2 Qualitative Data Collection Guides for FGDs and KII with local government were developed in English and translated into Kinyarwanda. The data collection team was trained on the FGD and KII forms and participated in a testing of the FGD and KII tools. FGDs were conducted with a maximum of eight and minimum of six participants. FGDs and KIIs were conducted in Kinyarwanda and documented by note-taking. Responses to each question were transcribed into English and compiled into individual Excel spreadsheets for each respondent category for analysis. Responses were then reviewed and coded for major themes and outcomes. Guides for FGDs and KIIs can be found in Annex 2.2-2.4. As previously noted, FGDs were revised to take place as KIIs in five of the ten districts mid-way through data collection. The interviews and remaining KIIs with local government were conducted by telephone in Kinyarwanda using a revised version of the original FGD and KII guides. Revisions are noted in the FGD forms. The data collection team documented and entered responses into the existing Excel spreadsheets. Interviews with implementing partners took place in-person and via telephone interviews in either English or Kinyarwanda following an interview guide (see Annex 2.5). Notes were taken in English and responses compiled into an Excel spreadsheet for analysis. 4.4 Ethical Considerations Data collection complied with Trócaire’s policy for informed consent in accordance with international good practice regarding research ethics and protocols including General Data Protection Regulations (GDPR). Enumerators were trained in obtaining informed consent and all researchers were required to participate in and pass a research ethics short course to conduct field research on behalf of Three Stones. This research ethics course includes informed consent and child protection considerations. All protocols and policies regarding participant selection and data collection were agreed upon with Trócaire Rwanda and were observed in the training and throughout the duration of the fieldwork. In order to ensure responsible data collection specific consideration was given to: • Administrative confidentiality of those participating in research; • Physical safeguards for those conducting research; • Data protection and secure maintenance procedures for personal information; and • Age- and ability-appropriate assent processes based on reasonable assumptions about comprehension of individuals involved in the research, including in the development of data collection tools. The evaluation team aimed to adequately account for how age, gender and disability present dynamic factors of vulnerability amongst the project beneficiaries. Data collection tools were developed to be gender-sensitive and inclusive and data collection was carried out using a multidisciplinary team composed of female and male consultants and local enumerators, including those familiar with working with varying disabilities. Inclusivity is of primary concern so both gender￾sensitive and accessible language and methodologies were considered in the data collection instruments. Data is disaggregated by age and gender to allow for deeper analysis and understanding of these dynamic factors. 25 4.5 Limitations Due to the COVID-19 pandemic, the evaluation methodology was revised mid-way through data collection to incorporate phone-based surveys and interviews. All attempts were made to ensure comparable data collection between both survey delivery mechanisms. In order to ensure a sufficient number of respondents, additional beneficiaries were randomly selected for inclusion in the survey in the event of non-response. An analysis of responses shows that while gender distribution by age was similar for both in-person and phone-based surveys, reported survey respondent household size was smaller with 44% of phone-based interview respondents reporting household sizes of 4 or less, while 36% of in-person survey respondents reporting household sizes of 4 or less. According to the beneficiary files, the majority of beneficiaries had a listed phone number. However, actual phone use was dependent upon the ability to maintain a phone, specifically the availability of electricity to charge a phone. Therefore, the use of phones may have created a bias towards beneficiaries with sufficient income to maintain a phone. Please see Annex 4.5 for additional information on beneficiary characteristics between in-person and phone-based survey participants. At the national level, MINAGRI, RAB and INGO participants were unavailable to participate in interviews due to their response to the COVID-19 pandemic. The lack of participation on the part of national level stakeholders may result in a loss of nuanced or meaningful recommendations for Trócaire. However, the evaluation did include a review of key government policies and research conducted by both government and relevant INGOs and incorporated findings where relevant. 26 5. Analysis of Findings This section presents an analysis of the findings from both the quantitative survey and qualitative data collection as they respond to the evaluation questions as defined by Trócaire in the Evaluation TOR (Annex 1). 5.1 Relevance 5.1.1 End Values for Project Impact and Outcome Indicators9 5.1.1.1 Impact Indicator 2: Percentage of target beneficiaries who are satisfied with their role in the implementation and monitoring of agriculture policy and related budget. Prior to asking beneficiaries about their involvement in the implementation and monitoring of agricultural policies and budgets, the evaluation sought to establish the respondents’ awareness of agricultural policies. Nearly all beneficiaries surveyed (99%) were able to provide at least one example of an agricultural policy being implemented in their district with 96% of those surveyed able to provide two examples. Table 11: Percent of Respondents Aware of Agricultural Policies in their District Adult (>30 Yrs.) Youth (18-30 Yrs.) Total Total Total Female Male Female Male Female Male Adult Youth Fertilizer Policy 92% 89% 93% 95% 92% 90% 91% 94% 92% Seed Policy 92% 91% 91% 86% 91% 90% 91% 90% 91% Land Use Policy 78% 78% 71% 77% 77% 78% 78% 73% 77% Irrigation Policy 19% 14% 14% 18% 17% 15% 17% 15% 17% Art. Insemination Pol. 14% 16% 10% 18% 13% 17% 15% 13% 14% Girinka Policy 14% 14% 14% 0% 14% 11% 14% 10% 13% Figure 6: Percent of Respondents Aware of Agricultural Policies in their District by Age and Gender Source: EPGA Final Evaluation Survey Results March 2020 When analyzing survey data by district, there were notable differences in policies mentioned. Respondents in Nyanza district were more likely to mention the artificial insemination policy (32%) as compared to the other project districts, contrasting their reliance on livestock as their main agricultural activity (see Table 9 below). In addition, the percent of participants mentioning the irrigation policy was greater in Northern Province project districts, Gakenke (33%) and Rulindo (31%), as compared to the districts in Southern Province (average of 11%). 9 Tables of survey respondent characteristics may be found in Annex 4 and additional data tables from the beneficiary survey may be found in Annex 5 14% 14% 19% 78% 92% 92% 14% 16% 14% 78% 91% 89% 14% 10% 14% 71% 91% 93% 18% 18% 77% 86% 95% 0% 20% 40% 60% 80% 100% Girinka Policy Artificial Insemination Policy Irrigation Policy Land Use Policy Seed Policy Fertilizer Policy Adult Female Adult Male Youth Female Youth Male 27 Table 12: Main Agricultural Activity by District as per NISR Agricultural Survey (2017) and Percent of Respondents Aware of Agricultural Policies in their District, by District Main Agricultural Activity10 Fertilizer Policy Seed Policy Land Use Policy Irrigation Policy Artificial Insem. Policy Girinka Policy Mention at least 1 policy Crops Only Livestock Only Crops & Livestock Gakenke 8% 0 92% 93% 97% 90% 33% 9% 14% 100% Nyamagabe 12% .2% 88% 94% 96% 68% 8% 2% 13% 99% Nyanza 25% 2% 73% 92% 98% 85% 14% 32% 12% 99% Nyaruguru 10% 0% 90% 87% 73% 67% 12% 10% 15% 100% Rulindo 22% 2% 76% 94% 91% 83% 31% 14% 9% 100% Total 92% 91% 77% 17% 14% 13% 99% Figure 7: Percent of Respondents Aware of Agricultural Policies in their District, by District Source: EPGA Final Evaluation Survey Results March 2020 5.1.1.1.1 Participation in Implementation of Agricultural Related Policies There was a demonstrated increase between the Midterm and Final Evaluation in the percentage of target beneficiaries who are satisfied with their role in the implementation of agricultural related policies. Overall, 95% of beneficiaries surveyed (95% female and 97% male) said that they are satisfied with their role in the implementation of agricultural related policies. The change is especially dramatic for both male and female youth with an increase in satisfaction from 48% to 91% for female youth and an increase from 58% to 97% for male youth between the Midterm and Final Evaluations. Overall 93% of youth beneficiaries surveyed were satisfied with their roles. As noted in the baseline, a lack of understanding of agricultural policies and improved agricultural limited participation in engagement with the implementation of policies,11 however FGDs and KIIs with youth conducted during the final evaluation found that youth reported increased confidence in their understanding of policies and improved agricultural practices. Table 13: Indicator 2.A Percentage of target beneficiaries who are satisfied with their role in the implementation of agricultural related policies Adult (>30 Yrs) Youth (18-30 Yrs) Total Total Total Female Male Female Male Female Male Adult Youth Target 55% 50% 77% 70% Baseline 59% 66% 94% 100% 91% 97% n/a 96% 93% Midterm 81% 86% 48% 58% n/a n/a n/a n/a n/a Final 96% 97% 91% 97% 95% 97% 96% 93% 95% 10 NISR (2017) Agricultural Household Survey 11 Trócaire (2017) Baseline survey for “Enhancing participatory governance and accountability” project 10% 2% 32% 9% 14% 87% 94% 92% 93% 94% 15% 13% 12% 14% 12% 8% 9% 14% 33% 31% 67% 68% 85% 90% 83% 73% 96% 98% 97% 91% 0% 20% 40% 60% 80% 100% Nyaruguru Nyamagabe Nyanza Gakenke Rulindo Artificial Insemination Policy Fertilizer Policy Girinka Policy Irrigation Policy Land Use Policy Seed Policy 28 Figure 8: Indicator 2.A Percentage of target beneficiaries who are satisfied with their role in the implementation of agricultural related policies by Age and Gender Sources: EPGA Final Evaluation Survey Results March 2020, EPGA Midterm Evaluation Survey Results March 2019, EPGA Baseline Survey Results October 2017 Figure 9: Indicator 2.A Percentage of target beneficiaries who are satisfied with their role in the implementation of agricultural related policies by District Source: EPGA Final Evaluation Survey Results March 2020 Reported mechanisms for participation included attending meetings where policies were discussed (64% of beneficiaries surveyed), participating in meetings where able to comment on agricultural policies (61% of beneficiaries surveyed) and participating in trainings to support implementation of policies (48% of beneficiaries surveyed). Only 3% of surveyed beneficiaries (4% for females) reported that there were no opportunities for participation. Table 14: Reported Mechanisms for Participation in Implementation of Agricultural Policies Adult (>30 Yrs) Youth (18-30 Yrs) Total Total Total Female Male Female Male Female Male Adult Youth Attended mtg. where policies discussed 66% 60% 62% 68% 65% 62% 64% 64% 64% Attended mtg. where able to comment 59% 71% 53% 55% 57% 68% 50% 40% 61% Trained to support implementation 52% 47% 34% 55% 48% 49% 63% 54% 48% No opportunities for participation 3% 2% 5% 0% 4% 2% 3% 4% 3% 59% 66% 94% 100% 81% 86% 48% 58% 96% 97% 91% 97% 0% 20% 40% 60% 80% 100% Female Male Female Male Adult Youth Baseline Midterm Final 12% 25% 18% 9% 14% 72% 52% 55% 44% 66% 16% 24% 27% 47% 20% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Gakenke Nyamagabe Nyanza Nyaruguru Rulindo Very satisfied Satisfied Not satisfied 29 Figure 10: Reported Mechanisms for Participation in Implementation of Agricultural Policies by Age and Gender Source: EPGA Final Evaluation Survey Results March 2020 Figure 11: Reported Mechanisms for Participation in Implementation of Agricultural Policies by District Source: EPGA Final Evaluation Survey Results March 2020 5.1.1.1.2 Participation in Monitoring of Implementation of Agricultural Related Policies There was an increase in the percentage of beneficiaries reporting satisfaction with their role in monitoring the implementation of agricultural related policies, with 95% of all beneficiaries surveyed reporting that they are satisfied compared to 77% at baseline. Reported participation in the monitoring of the implementation of agricultural policies was also high, with 97% of beneficiaries surveyed (96% female and 98% male) reporting that they participated in at least one mechanism for monitoring the implementation of agricultural policies. Across districts, Nyanza reported the highest levels of satisfaction at 97% while Nyaruguru reported the highest levels of being ‘very satisfied’ at 51%. Table 15: Indicator 2.B Percentage of target beneficiaries who are satisfied with their role in monitoring of implementation of agricultural related policies Adult (>30 Yrs) Youth (18-30 Yrs) Total Total Total Female Male Female Male Female Male Adult Youth Target 65% 50% 94% 94% Baseline 51% 37% 92% 93% 78% 74% n/a 91% 77% Midterm 83% 84% 66% 88% n/a n/a n/a n/a n/a Final 94% 97% 95% 95% 94% 97% 95% 95% 95% 66% 59% 52% 3% 60% 71% 47% 2% 62% 53% 34% 5% 68% 55% 55% 0% 10% 20% 30% 40% 50% 60% 70% 80% Attended Meetings Where Policies are Discussed Attended Meetings Where Able to Comment on Agricultural Priorities Trained to Support Implementation of Policies No Opportunities for Participation Adult Female Adult Male Youth Female Youth Male 47% 74% 43% 68% 59% 47% 3% 73% 54% 48% 3% 60% 59% 56% 4% 63% 66% 40% 6% 0% 10% 20% 30% 40% 50% 60% 70% 80% Attended mtg. where policies discussed Attended mtg. where able to comment Trained to support implementation No opportunities for participation Gakenke Nyamagabe Nyanza Nyaruguru Rulindo 30 Figure 12: Indicator 2.B Percentage of target beneficiaries who are satisfied with their role in monitoring of implementation of agricultural related policies by Age and Gender Sources: EPGA Final Evaluation Survey Results March 2020, EPGA Midterm Evaluation Survey Results March 2019, EPGA Baseline Survey Results October 2017 Figure 13: Indicator 2.B Percentage of target beneficiaries who are satisfied with their role in monitoring of implementation of agricultural related policies by District Source: EPGA Final Evaluation Survey Results March 2020 When asked how participants participate in the monitoring of the implementation of agricultural policies, the majority of respondents, 84% (85% female and 82% male) participated in community forums (including Umuganda, parent’s meetings and evening sessions). Male beneficiaries surveyed were more likely to report participating in accountability meetings (53%) and performance contract implementation (21% as compared to female beneficiaries (46% and 13% respectively). Adult male beneficiaries were also more likely than youth male beneficiaries to participate in performance contract implementation (23% for adult and 14% for youth males). At the district level, participants in Nyanza were more likely (90%) to participate in community forums compared to those in other districts, but less likely to participate in accountability meetings (38%). Rulindo participants reported greater participation in accountability meetings (60%) as compared to other districts. 51% 37% 92% 93% 83% 84% 66% 94% 88% 97% 95% 95% 0% 20% 40% 60% 80% 100% Female Male Female Male Adult Youth Baseline Midterm Final 5% 6% 3% 5% 6% 71% 66% 68% 44% 69% 24% 28% 29% 51% 26% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Gakenke Nyamagabe Nyanza Nyaruguru Rulindo Very satisfied Satisfied Not satisfied 31 Table 16: Reported Mechanisms for Participation in Monitoring the Implementation of Agricultural Policies Adult (>30 Yrs) Youth (18-30 Yrs) Total Total Total Female Male Female Male Female Male Adult Youth Participated in community forums 84% 81% 86% 86% 85% 82% 83% 86% 84% Participated in accountability mtgs. 49% 54% 38% 50% 46% 53% 50% 41% 48% Collected information by visiting activities 22% 22% 21% 23% 21% 22% 22% 21% 22% Part. in performance contracts 12% 23% 16% 14% 13% 21% 16% 15% 15% Completed monitoring report 14% 13% 9% 18% 13% 14% 13% 11% 13% Completed a scorecard 10% 9% 9% 23% 10% 14% 10% 13% 10% Did not participate 4% 2% 3% 0% 4% 2% 3% 3% 3% Figure 14: Reported Mechanisms for Participation in Monitoring the Implementation of Agricultural Related Policies Source: EPGA Final Evaluation Survey Results March 2020 Figure 15: Reported Mechanisms for Participation in Monitoring the Implementation of Agricultural Related Policies by District Source: EPGA Final Evaluation Survey Results March 2020 84% 49% 22% 12% 14% 10% 4% 81% 54% 22% 23% 13% 9% 2% 86% 38% 21% 16% 9% 9% 3% 86% 50% 23% 14% 18% 23% 0% 20% 40% 60% 80% 100% Participated in Community Forums Participated in Accountability Meetings Collected Information by Visiting Activities Participated in Performance Contract Implementation Completed a Monitoring Report Completed a Scorecard Did Not Participate Adult Female Adult Male Youth Female Youth Male 83% 48% 22% 9% 14% 10% 83% 45% 16% 11% 4% 4% 4% 90% 44% 22% 19% 13% 10% 3% 77% 50% 32% 21% 17% 12% 4% 86% 66% 9% 17% 26% 23% 3% 0% 20% 40% 60% 80% 100% Participated in community forums Participated in accountability mtgs. Collected information by visiting activities Part. in performance contracts Completed monitoring report Completed a scorecard Did not participate Gakenke Nyamagabe Nyanza Nyaruguru Rulindo 32 5.1.1.1.3 Participation in Monitoring of Budgets on Agricultural Related Policies There was a significant difference between the reported satisfaction of beneficiaries at the Baseline, Midterm and Final Evaluation. Overall, 83% of beneficiaries surveyed were satisfied with their role in monitoring of budgets on agricultural related policies compared to 5% at baseline. This is figure is also significantly higher than the country statistics of 47.7% of citizens who are satisfied with their participation in elaboration of budgets and plans12. The majority of beneficiaries surveyed, 70%, participated at the Community Committee level, with equal participation between males (70%), females (70%) and youth (68%). While 30% of participants reported participation at the Cell level, the majority were adult male beneficiaries (45%), while female and youth reported less participation (30% and 33% respectively). FGD and interviews with women and youth indicate that there is some confusion between a personal agricultural budget for purchasing of seeds and other inputs and the budget for implementation of agricultural policies. However, others noted that having an understanding of the agricultural budget encouraged them to develop their own personal agricultural budgets. Findings from the KIIs and FGDs with youth do not provide a clear reason for the increase in their satisfaction, as some youth commented that they still do not have the skills or confidence to participate in budgets however others commented that authorities were taking their needs into account during meetings where budgets were discussed. At the district level, Nyaruguru reported the highest levels of satisfaction with their role in monitoring budgets, with overall satisfaction of 91% with 47% reporting to be ‘very satisfied’. Nyamagabe had the lowest reported satisfaction at 75%, followed by Nyanza at 82%. Table 17: Indicator 2.C Percentage of target beneficiaries who are satisfied with their role in monitoring of budgets on agricultural related policies Adult (>30 Yrs) Youth (18-30 Yrs) Total Total Total Female Male Female Male Female Male Adult Youth Target 23% 20% 23% 25% Baseline 5% 5% 5% 8% 5% 5% n/a 6% 5% Midterm 34% 38% 36% 33% n/a n/a n/a n/a n/a Final 81% 87% 83% 91% 81% 88% 83% 85% 83% Figure 16: Indicator 2.C Percentage of target beneficiaries who are satisfied with their role in monitoring of budgets on agricultural related policies by Age and Gender Sources: EPGA Final Evaluation Survey Results March 2020, EPGA Midterm Evaluation Survey Results March 2019, EPGA Baseline Survey Results October 2017 12 RGB (2018) Rwanda Governance Scorecard, 5th Edition 5% 5% 5% 8% 34% 38% 36% 33% 81% 87% 83% 91% 0% 20% 40% 60% 80% 100% Female Male Female Male Adult Youth Baseline Midterm Final 33 Figure 17: Indicator 2.C Percentage of target beneficiaries who are satisfied with their role in monitoring of budgets on agricultural related policies by District Source: EPGA Final Evaluation Survey Results March 2020 Of the beneficiaries surveyed, 23% reported no opportunity for participation, with female youth reporting the least opportunities for participation at 28%, which is reflected in the lower levels of satisfaction for this group. Of those reporting no opportunity for participation, 39 beneficiaries surveyed (or 11% of all respondents) reported being satisfied with their level of participation. It is not clear if this is due to a lack of understanding surrounding the question or if there are other factors contributing to their satisfaction. Interviews and FGDs with beneficiaries indicate that beneficiaries report increased satisfaction from the knowledge that they have the right to participate, even if they have yet to develop the skills to meaningfully participate. At the district level, Gakenke and Rulindo survey respondents were more likely to report participating at the Community Committee (Gakenke 83%, Rulindo 80%) and Cell levels (Gakenke 45%, Rulindo 37%) as compared to the other three districts. Gakenke (7%) and Rulindo (9%) were also less likely to report no opportunities for participation, as compared to Nyamagabe (31%), Nyanza (24%) and Nyaruguru (29%). Table 18: Reported Mechanisms for Participation in Monitoring the Budgets of Agricultural Related Policies Adult (>30 Yrs) Youth (18-30 Yrs) Total Total Total Female Male Female Male Female Male Adult Youth Participated at Community Committee level 70% 70% 69% 64% 70% 69% 70% 68% 70% Participated at Cell level 30% 27% 28% 45% 30% 30% 29% 33% 30% Participated at Sector level 13% 13% 12% 23% 13% 15% 13% 15% 14% Participated at District level 1% 3% 2% 5% 1% 3% 2% 3% 2% No opportunities for participation 22% 23% 28% 18% 23% 22% 22% 25% 23% 12% 25% 18% 9% 14% 72% 52% 55% 44% 66% 16% 24% 27% 47% 20% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Gakenke Nyamagabe Nyanza Nyaruguru Rulindo Very satisfied Satisfied Not satisfied 34 Figure 18: Reported Mechanisms for Participation in Monitoring the Budgets of Agricultural Related Policies by Age and Gender Source: EPGA Final Evaluation Survey Results March 2020 Figure 19: Reported Mechanisms for Participation in Monitoring the Budgets of Agricultural Related Policies by District Source: EPGA Final Evaluation Survey Results March 2020 5.1.1.2 Outcome Indicator 1-1: Percentage of target citizens who are confident of their contribution in the development of agricultural policy and budget monitoring. There was a marked increase in the confidence of citizens in their contribution to the development of agricultural policy and budget monitoring between the midterm and final evaluations, with 91% of surveyed beneficiaries (90% female and 95% male) reporting the they were confident in their contribution to the development of agricultural policy and 75% of surveyed beneficiaries (73% female and 78% male) reporting confidence in their role in budget monitoring compared to baseline figures of 81% and 29% respectively. Data collection for the midterm evaluation took place in March of 2019, one year prior to the data collection for the final evaluation. During this time, it is feasible that policy changes surrounding timely access to seeds and inputs resulting from the first national agricultural policy dialogue conducted in March 2018, became apparent. At the district level, 100% of surveyed participants in Gakenke and 94% in Rulindo reported that they were confident in their contribution in the development of agricultural policy, with no reports of inability to participate. Nyamagabe reported highest rates of not having an opportunity to participate at 11% and had the lowest rates of confidence at 82% compared to the other districts survey. 70% 30% 13% 1% 22% 70% 27% 13% 3% 23% 69% 28% 12% 2% 28% 64% 45% 23% 5% 18% 0% 10% 20% 30% 40% 50% 60% 70% 80% Participated at the Community Committee Level Participated at the Cell Level Participated at the Sector Level Participated at the District Level No Opportunities for Participation Adult Female Adult Male Youth Female Youth Male 83% 65% 69% 62% 80% 45% 22% 28% 28% 37% 14% 7% 13% 3% 13% 3% 14% 1% 14% 31% 24% 29% 9% 0% 20% 40% 60% 80% 100% Gakenke Nyamagabe Nyanza Nyaruguru Rulindo Participated at Community Committee Participated at Cell level Participated at Sector level Participated at District level No opportunities for participation 35 Table 19: Indicator 1-1.A Percentage of target citizens who are confident of their contribution in the development of agricultural policy Adult (>30 Yrs) Youth (18-30 Yrs) Total Total Total Female Male Female Male Female Male Adult Youth Target 62% 62% 86% 93% Baseline 58% 58% 84% 92% 80% 83% n/a 86% 81% Midterm 66% 68% 73% 69% n/a n/a n/a n/a n/a Final 90% 95% 88% 95% 90% 95% 92% 90% 91% Figure 20: Indicator 1-1.A Percentage of target citizens who are confident of their contribution in the development of agricultural policy by Age and Gender Sources: EPGA Final Evaluation Survey Results March 2020, EPGA Midterm Evaluation Survey Results March 2019, EPGA Baseline Survey Results October 2017 Figure 21: Indicator 1-1.A Percentage of target citizens who are confident of their contribution in the development of agricultural policy by District Source: EPGA Final Evaluation Survey Results March 2020 Female youth, and youth in general, were less likely to report confidence in their role in budget monitoring, with 62% of female youth reporting that they were confident, compared to 73% for all women. While 77% of adults reported being confident in their contribution, only 69% of youth felt confident. At the district level, Nyamagabe (66%, Nyanza (73%) and Nyaruguru (74%) reported less confidence and were more likely to report that they did not participate in monitoring of budgets. Gakenke and Rulindo reported higher rates of confidence with 84% of survey respondents in Gakenke reporting that they were confident in their role and 89% in Rulindo. 58% 58% 84% 92% 66% 68% 73% 69% 90% 95% 88% 95% 0% 20% 40% 60% 80% 100% Female Male Female Male Adult Youth Baseline Midterm Final 3% 5% 6% 100% 86% 92% 90% 94% 11% 3% 4% 6% 0% 20% 40% 60% 80% 100% Gakenke Nyamagabe Nyanza Nyaruguru Rulindo Not confident in contribution Confident in contribution Did not participate 36 Table 20: Indicator 1-1.B Percentage of target citizens who are confident of their role in budget monitoring Adult (>30 Yrs) Youth (18-30 Yrs) Total Total Total Female Male Female Male Female Male Adult Youth Target 25% 20% 60% 65% Baseline 18% 13% 47% 58% 28% 30% n/a 51% 29% Midterm 53% 50% 50% 54% n/a n/a n/a n/a n/a Final 77% 76% 62% 86% 73% 78% 77% 69% 75% Figure 22: Indicator 1-1.B Percentage of target citizens who are confident of their role in budget monitoring by Age and Gender Sources: EPGA Final Evaluation Survey Results March 2020, EPGA Midterm Evaluation Survey Results March 2019, EPGA Baseline Survey Results October 2017 Figure 23: : Indicator 1-1.B Percentage of target citizens who are confident of their role in budget monitoring by District Source: EPGA Final Evaluation Survey Results March 2020 5.1.1.3 Outcome Indicator 3-1: Percentage of citizens, especially women and youth, and members of civil society who consider that local authorities respond to their expressed priorities with regard to sustainable agriculture. 5.1.1.3.1 Percent of Target Citizens Who Consider that Local Authorities Respond to their Priorities Of those beneficiaries surveyed, 88% (89% of females and 87% of males) reported that local authorities respond to their expressed priorities, as compared to 57% at baseline, and 84% of the beneficiaries surveyed (85% female and 82% male) were able to give an example of how local authorities have responded to an expressed agricultural need in the previous 12 months. Surveyed 18% 13% 47% 53% 58% 50% 50% 54% 77% 76% 62% 86% 0% 20% 40% 60% 80% 100% Female Male Female Male Adult Youth Baseline Midterm Final 16% 10% 4% 13% 3% 84% 66% 73% 74% 89% 25% 22% 13% 9% 0% 20% 40% 60% 80% 100% Gakenke Nyamagabe Nyanza Nyaruguru Rulindo Not confident in contribution Confident in contribution Did not participate 37 participants in Nyanza (93%) and Nyaruguru (97%) were more likely to report that local authorities responded to their needs as compared to the other three districts. Table 21: Indicator 3-1.A Percentage of target citizens who consider that local authorities respond to their expressed priorities with regard to sustainable agriculture Adult (>30 Yrs) Youth (18-30 Yrs) Total Total Total Female Male Female Male Female Male Adult Youth Target 60% 55% 65% 80% Baseline 58% 48% 54% 77% 57% 56% n/a 63% 57% Midterm 72% 77% 76% 81% n/a n/a n/a n/a n/a Final 92% 87% 79% 86% 89% 87% 90% 81% 88% Figure 24: Indicator 3-1.A Percentage of target citizens who consider that local authorities respond to their expressed priorities with regard to sustainable agriculture by Age and Gender Sources: EPGA Final Evaluation Survey Results March 2020, EPGA Midterm Evaluation Survey Results March 2019, EPGA Baseline Survey Results October 2017 Figure 25: Indicator 3-1.A Percentage of target citizens who consider that local authorities respond to their expressed priorities with regard to sustainable agriculture by District Source: EPGA Final Evaluation Survey Results March 2020 Of those beneficiaries reporting that their priorities were addressed, the majority cited timely access to higher quality seeds (27%) and timely access to fertilizers or lime for soil health (31%). Other priorities responded to by local authorities included access to pesticides (14%) and the sharing of information or provision of trainings on topics such as kitchen gardens and horticulture (10%). Participants in Rulindo were more likely to mention fertilizers (43%) compared to the other districts, while all districts reported access to seeds at approximately the same rate. 58% 48% 54% 72% 77% 76% 77% 81% 92% 87% 79% 86% 0% 20% 40% 60% 80% 100% Female Male Female Male Adult Youth Baseline Midterm Final 10% 3% 5% 6% 9% 13% 2% 3% 14% 81% 84% 93% 97% 80% 0% 20% 40% 60% 80% 100% Gakenke Nyamagabe Nyanza Nyaruguru Rulindo I do not have any priorities/there are no platforms to share No Yes 38 Table 22: Percent of Participants Reporting Priorities Responded to In the Past 12 Months Adult (>30 Yrs) Youth (18-30 Yrs) Total Total Total Female Male Female Male Female Male Adult Youth Fertilizer/ soil health 33% 32% 24% 36% 31% 31% 33% 28% 31% Seeds 28% 24% 31% 27% 29% 24% 27% 30% 27% Pesticides 13% 12% 17% 18% 14% 13% 13% 18% 14% Info sharing/ trainings 12% 13% 5% 5% 10% 11% 12% 5% 10% Markets 3% 4% 3% 5% 3% 4% 3% 4% 3% Water/ irrigation 3% 4% 0% 0% 2% 3% 3% 0% 2% Animal health/ art. insemination 2% 2% 0% 5% 2% 3% 2% 1% 2% Able to provide an example 88% 81% 78% 86% 85% 82% 85% 80% 84% Figure 26: Percent of Participants Reporting Priorities Responded to In the Past 12 Months by Topic by Age and Gender Source: EPGA Final Evaluation Survey Results March 2020 Figure 27: Percent of Participants Reporting Priorities Responded to In the Past 12 Months by Topic by District Source: EPGA Final Evaluation Survey Results March 202 33% 28% 13% 12% 3% 3% 2% 32% 24% 12% 13% 4% 4% 2% 24% 24% 17% 5% 3% 36% 36% 18% 5% 5% 5% 0% 5% 10% 15% 20% 25% 30% 35% 40% Fertilizers/ Soil Health Seeds Pesticides Information Sharing and Trainings Markets Water/ Irrigation Animal Health/ Insemination Adult Female Adult Male Youth Female Youth Male 29% 31% 35% 31% 29% 19% 28% 35% 16% 43% 17% 17% 14% 9% 6% 19% 11% 5% 10% 6% 9% 3% 3% 1% 3% 7% 1% 3% 1% 2% 5% 0% 10% 20% 30% 40% 50% Nyaruguru Nyamagabe Nyanza Gakenke Rulindo Fertilizers/Soil Health Seeds Pesticides Information Sharing and Trainings Markets Water/Irrigation Animal Health/Insemination 39 Figure 28: Percent of Participants Reporting Priorities Responded to In the Past 12 Months Source: EPGA Final Evaluation Survey Results March 2020 5.1.1.3.2 Percent of CSOs Who Consider that Local Authorities Respond to their Priorities There has been a significant increase in reported response by local authorities to the priorities of CSOs since the start of the EPGA project. At baseline, 10 CSOs surveyed reported that local authorities responded at a rate of 48% while the 10 CSOs surveyed for the final evaluation reported that local authorities respond at a rate of 88%, an increase of 40%. CSOs in Gakenke and Nyaruguru generally reported high levels of response, while those in Nyanza and Nyamagabe reported response rates of 70-80%. CSOs reported sharing their priories with local government in cell, sector and district meetings, including JADF meetings. One CSO reported preparing policy papers. Table 23: Indicator 3-1.B Percentage of CSOs who consider that local authorities respond to their expressed priorities with regard to sustainable agriculture CSOs Target 52% Baseline 48%* Midterm n/a Final 88%* *Both baseline and end-line values measure the percent to which CSOs feel that local authorities respond to their expressed priorities, not the percent of CSOs who consider local authorities respond to their expressed priorities. Table 24: Summary of CSO responses to Indicator 3-1.B District Consider local authorities respond Percent to which local authorities respond CSO-Nyaruguru Yes 100% CSO-Nyaruguru Yes 90% CSO-Nyaruguru Yes 90% CSO-Nyamagabe Yes 70% CSO-Nyanza Yes 80% CSO-Nyanza Yes 70% CSO-Gakenke Yes 98% CSO-Gakenke Yes 70% CSO-Gakenke Yes 87% CSO-Gakenke Yes 95% CSO-Gakenke/Rulindo Yes 99% Total 100% 89% 81% 76% 90% 92% 77% 19% 24% 10% 8% 23% 0% 20% 40% 60% 80% 100% Gakenke Nyamagabe Nyanza Nyaruguru Rulindo Provided an example Unable to provide an example 40 Table 25: Agriculture Policy Issues Responded to by Local Authorities in the Previous 12 Months District Agricultural Policies Responded To By Local Authorities Nyaruguru Terracing and agro-forestry for soil erosion Nkunganire program Animal husbandry Nyamagabe Access to fertilizers Access to markets Nyanza Increased purchase price of milk Reduced taxes on swamplands Gakenke Fertilizer pricing and access to fertilizers at agro-dealers Seed pricing and delays in access Agroforestry seedlings Gakenke/Rulindo Training on long-term storage of crops to avoid spoilage Small-scale irrigation-provision of water pipes 5.1.1.4 Results for Other Impact and Outcome Indicators The following indicators were not measured as a component of the final evaluation as they were measured and reported on during project implementation but are included here for reference. Table 26: Impact Indicator 1-Number of public policies introduced, adopted, repealed, changed or implemented consistent with citizen input Planned Actual Year 1 0 0 Year 2 4 2 Year 3 4 4 Year 4 4 2 (as of Q1) Total 12 8 Table 27: Outcome Indicator 1.2 Frequency of interactions between citizens and local authorities on agricultural policies and budget Planned Actual Year 1 0 0 Year 2 6 10 Year 3 4 26 Year 4 37 27 (as of Q1) Total 47 63 Table 28: Outcome Indicator 1.2 Number of issues related to agricultural policy raised by citizens and addressed by public authorities in the past 12 months Planned Actual Year 1 0 0 Year 2 4 7 Year 3 4 4 Year 4 4 3 (as of Q1) Total 12 14 Table 29: Outcome Indicator 2.1 Number of decisions taken by target district and sector leaders with CSO’s inputs in the past 12 months Planned Actual Year 1 0 0 Year 2 18 7 Year 3 6 8 Year 4 3 1 (as of Q1) Total 27 16 Table 30: Outcome Indicator 2.2 Number of decisions made at the national level with CSO’s inputs in the past 12 months Planned Actual Year 1 0 0 Year 2 10 0 Year 3 2 2 Year 4 1 1 Total 13 3 5.1.2 Appropriateness and Usefulness of Project Indicators As written and defined, the indicators were generally appropriate and useful for measuring project achievement against results framework. In addition, all impact and outcome indicators were appropriately defined in the Performance Indicator Reference Sheet (PIRS), disaggregated by gender and age groupings as appropriate and generally met the criteria as per the SMART (specific, measurable, achievable, relevant and time-bound) framework for indicators. However, the three indicators measured through quantitative surveys (Impact Indicator 2, Outcome Indicator 1-1 and Outcome Indicator 3-1) required further disaggregation into what ultimately constituted two to three separate indicators. In addition, Outcome Indicator 3-1B as defined in the PIRS and as defined during the baseline data collection and subsequent evaluation exercises are inconsistent. As written in the project PIRS, the indicator intends to measure the percentage of CSOs who consider that local authorities respond to their expressed priorities. However, the measurement at baseline is the percentage to which CSOs feel that local authorities respond to their expressed priorities. In addition, measuring this indicator requires a separate quantitative survey tool in order to capture CSO responses rather than using responses during FGDs to accurately measure achievement against this indicator as attempted during the midterm evaluation. While project staff noted the usefulness of and satisfaction with the project indicators as written, a review of project annual reports and annual MEL (Monitoring, Evaluation and Learning) plans submitted to USAID highlight significant changes to the organization and wording of outcome and output indicators without providing rationale or justification for changes. While the changes in the indicators did not impact performance, the shifting placement, removal and addition of outcome and output indicators makes it difficult to consolidate and report on project progress and achievement across the life of the project. Project learning: The project included multiple opportunities for project learning in addition to regular monitoring and reporting on indicators and evaluation exercises. Specific activities imbedded within the project design that supported project learning included the development of a documentary film on citizen participation in policy monitoring, exchange experiences and learning through JADF, the development of a lessons learned and best practices report, a research piece on the involvement of women. In addition, quarterly stakeholder meetings provided opportunities for implementing partners to share findings and best practices. 5.1.3 Appropriateness of Project Design The EPGA project was designed in alignment with Trócaire’s Global ToC, in which empowered women and men, sustainable and resilient communities, a strong and representative civil society and just and accountable local government are necessary for sustainable outcomes. The project is also in alignment with the Rwanda Strategic Plan for Agricultural Transformation (PSTA 4) Priority Area 4: “Enabling Environment and Responsive Institutions” which highlights the needs for strengthening forums for farmers participation in joint planning from the local to national levels through farmer organizations and cooperatives and using IT tools for data collection to support planning and implementation.13 Interviews with Trócaire and implementing partner staff highlighted the appropriateness of the project components in achieving the overall project objective. In addition, the placement of activities within the project life cycle was well thought-out during the project design process. The following analysis of the appropriateness of the project design will look at beneficiary selection and individual project components by Outcome. 5.1.3.1 Beneficiary selection Women and youth are the most vulnerable populations, as 92% of women in rural areas engage in the agricultural sector, as compared to 77% of men, and working age youth make up 77% of the 13 MINAGRI (June 2018) Strategic Plan for Agriculture Transformation 20018-24 42 rural population.14 Recognizing the unique vulnerabilities of women and youth, the project targeted these two groups in the project design (60% women and 20% youth). Project beneficiary lists provided by the implementing partners illustrate that the majority citizens reached (70%) were female (11% female youth and 59% female adults). The project also reached its intended target of 20% youth participation. However, with working age youth comprising 77% of the rural work force with 54% of young men working in agriculture and 79% of young women working in agriculture primarily on family farms15, there in a need to place greater emphasis on youth participation in advocacy projects. Youth in particular face significant challenges in agriculture as they lack access to land, capital and knowledge and often lack an interest in agriculture, with female youth facing even greater challenges due to unequal power relations. With an aging agricultural workforce, with an average age of 55 years, MINAGRI recognizes a need to engage more youth in agriculture.16 Table 31: Percentage participation by Females and Males, Youths and Adults in the EPGA Project by District* Adult Youth Total Female Total Male (>30 Yrs) (18-30 Yrs) Female Male Female Male Gakenke 55.88% 24.09% 13.35% 6.68% 69.23% 30.77% Nyamagabe 60.52% 19.53% 10.62% 9.33% 71.14% 28.86% Nyanza 57.53% 21.77% 9.89% 10.81% 67.43% 32.57% Nyaruguru 59.93% 19.98% 10.50% 9.59% 70.43% 29.57% Rulindo 59.31% 23.56% 12.19% 4.94% 71.50% 28.50% Total 58.81% 21.51% 11.22% 8.46% 70.03% 29.97% 80.32% 19.68% Source: Project Beneficiary Files Shard by Implementing Partners *Note: These figures may not be accurate due to incomplete and inaccurate data entry In addition to target beneficiaries, the project created Community Committees at the village level. These committees are made up of local rural farmers elected by their community. The project intended for Community Committee membership to include 60% women and 20% men, with 40% youth (or 2 adult women, 1 adult man, 1 youth female and 1 youth male) as members. As the table below illustrates, females comprised the majority of Community Committee members (project average of 56%, or 2.8 females per group) and youth comprised 24% of the members. While Gakenke, Nyamagabe and Nyanza were able to achieve close to 3 females per group, both Nyaruguru and Rulindo averaged 2.3 and 2.5 females respectively, with Nyaruguru reporting no female youth participation. As noted in the Lessons Learnt document, the participants who “self￾select” into these processes are more likely to be privileged and more engaged in agricultural issues as compared to other community members. They also noted that while more than half of the participants were female, the majority of committees were chaired by wealthier, more educated male farmers.17 14 MINAGRI (June 2017) National Agriculture Policy; 15 Franzel, S., Kinyua, H., Rucibigango, M., Davis, K., and Makh, S. (2019). Youth in Extension and Advisory Services: Rwanda. Developing Local Extension Capacity Project. 16 The New Times (14 February 2017) Govt seeks more youth engagement in agriculture 17 RDO (2020) Enhancing Participatory Governance and Accountability Project: Lessons Learnt and Best Practice 43 Table 32: Percentage participation by Females and Males, Youths and Adults in Community Committees by District Adult Youth Total Total (>30 Yrs) (18-30 Yrs) Female Male Female Male Female Male Gakenke 46.51% 23.26% 16.28% 13.95% 62.79% 37.21% Nyamagabe 40.91% 34.09% 18.18% 6.82% 59.09% 40.91% Nyanza 42.11% 28.95% 15.79% 13.16% 57.89% 42.11% Nyaruguru 46.34% 48.78% 0.00% 4.88% 46.34% 53.66% Rulindo 29.17% 37.50% 20.83% 12.50% 50.00% 50.00% Total 42.11% 34.21% 13.68% 10.00% 55.79% 44.21% 76.32% 23.68% Source: Project Beneficiary Files Shard by Implementing Partners 5.1.3.2 Outcome 1: Empowered citizens, particularly women and youth, who are able to engage in agricultural policy and budget monitoring in 5 districts and hold public authorities accountable Activities under Outcome 1 focused on empowering citizens to engage in agricultural policy by supporting both the individual level through trainings and provision of educational materials and at the community level through the development of structures to support collective engagement with local government. The importance of citizen participation in governance is documented in the Constitution of Rwanda, Article 48: “All Rwandans have the duty to participate in the development of the country”. In addition, since 2000, the Government of Rwanda began the process of decentralization with the intent of bringing about a citizen centered governance, with citizen participation in the identification of priorities, implementation of development projects and hold local authorities accountable.18 However, while there is intent for participation, actual avenues for participation are limited to implementation of policies, rather than meaningful contribution to the discourse on policies or the monitoring of their implementation. These are further hindered by lack of understanding of rights to participation and a general sense that leaders make the decisions, not citizens.19 Below is a summary of the specific project components. Training of citizens on their roles and responsibilities: In the first year of the project, 3,675 citizens and 830 local leaders were trained at the village level in their roles and responsibilities for agricultural policy monitoring of implementation and budgets. These trainings were carried out by UNICOOPAGI and CEJP. When asked, those who recalled the training, found the content appropriate however, as the majority of women and youth had no previous experiences with budgets, some noted that the training was insufficient for them to meaningfully participate in the monitoring of budgets. This finding contradicts survey findings reporting high levels of satisfaction and confidence in participation in monitoring agricultural policy budgets. However, while beneficiaries reported not understanding budgets, findings from FGDs and interviews highlighted the satisfaction gained from the knowledge that they have a right to participate in budget monitoring. Many of the respondents in the Youth and Women FGDs and interviews were unable to distinguish trainings provided under the EPGA project with other agricultural trainings provided by the implementing partners on specific agricultural topics. 18 Rwanda Governance Board (June 2018) Rwanda Governance Review: Good Governance and Decentralization in Rwanda 19 Never Again Rwanda (2018) Local Government Imihigo Process: Understanding the Factors Contributing to Low Citizen Participation 44 A training manual was developed after the trainings had already been completed, based upon the training materials.20 The manual was initially intended for distribution to community leaders, but additional copies were printed and distributed more widely. Rather than a guide on how to train citizens in advocacy, the document is actually a reference guide to the role of citizens in policy implementation and monitoring of implementation and budgets and advocacy. The training component consists of five lessons, each in a small text box at the start of each chapter, with little guidance offered as to how to facilitate each session. The manual also lacks any mention of the role of women and youth play in implementation, monitoring and advocacy for agricultural policies. Establishment and Training of Community Committees: Community Committees were established in 63 villages during Year 1 of the project and 315 members were trained during the Year 2 (October-December 2017). Community Committees are made up of five community members and elected by village members. Community Committees have been an integral component of the project as they have served as an intermediary between farmers and local government, advocating on behalf of farmers and sharing responses from local government. Community Committees have also become a source of information on agricultural policies in their communities and supported initiatives including development of kitchen gardens and supporting farmers to register for the Smart Nkunganire program for agricultural subsidies. In addition, through the use of Mobile Data Collection via smartphones installed with the SurveyCTO software, Community Committee members were able to report the advocacy needs of farmers in their community to EPGA project staff. Feedback and Exchange Meetings: Feedback meetings took place at least once per quarter as an opportunity for Community Committees to share information with the community. These meetings often took place within existing meeting spaces, including Umuganda and parent’s nights, and were reported to occur in greater frequency than the proposed once per quarter, indicating that communities found these meetings useful. Exchange meetings were an opportunity for Community Committee members to bring issues raised by community members to the local leaders in a formal capacity and for local leaders to respond to issues. Educational Materials on Rights and Responsibilities: Posters and booklets developed by the project on topics related to roles and rights of citizens in agricultural policy implementation and budgeting were well received by the target audience as evidenced by the need to increase production from the 5,000 booklets distributed in December 2017. In April 2018, an additional 20,000 booklets were printed for a total of 25,000 booklets. Booklets were distributed to citizens, local authorities including sector and district agronomists, village leaders and CSOs in the project districts. In addition, JADF (Joint Action Development Forum) members also received copies. Posters were also well received by beneficiaries with 85% of those surveyed reporting having seen at least one poster on the role of citizen in monitoring budgets. Youth were more likely to have seen the posters as compared with adults, with 91% of youth surveyed reporting having seen or read the posters compared with 83% of adults. FGDs and interviews with women and youth also highlight that the majority saw and read the posters as they were posted in prominent areas. However, men were more likely than women to report having actually read the posters, with 73% of men reporting reading compared to 53% of women. These findings are consistent with Rwanda literacy rates as 78% of men are literate whereas 69% of women are literate.21 Posters used cartoon depictions and few words, making the information accessible for a low literacy population. Findings from the FGDs and interviews showed that those who reported reading the posters found the information very useful and that posters contributed to a better understanding of their rights. Table 33: Percent of Survey Participants Reporting Seeing or Reading Posters 20 Trócaire (November 2017) The Role of Citizens and Local Entities in Policy and Budget Monitoring and Advocacy Related to Agriculture, English Version 21 UNESCO Rwanda Country Statistics http://uis.unesco.org/en/country/rw 45 Adult (>30 Yrs) Youth (18-30 Yrs) Total Total Total Female Male Female Male Female Male Adult Youth Have seen Poster 26% 16% 45% 14% 31% 16% 23% 36% 26% Have Read Poster 55% 71% 45% 82% 53% 73% 60% 55% 59% Never Saw/Read 19% 13% 10% 5% 17% 11% 17% 9% 15% Figure 29: Percent of Survey Participants Reporting Seeing or Reading Posters Source: EPGA Final Evaluation Survey Results March 2020 Policy Translations: Recognizing that agricultural policies put forth by the national government are inaccessible to the majority of local leaders and communities who are tasked with implementing the policies, the project planned to translate three policies deemed to be most useful for local leaders into an accessible format, including less technical jargon and in Kinyarwanda. However, delays in the formal approval process meant that only one policy, the Seed Policy, has been distributed to local leaders by the end of Year 3 with the Small-Scale Irrigation Policy and MINAGRI Citizen’s Charter to be approved and distributed by project completion. In addition, policies for translation were ultimately determined by MINAGRI, rather than the beneficiaries and local government. While the intention of the policy translations is to support citizen understand the most relevant agricultural policies, this activity would be better placed under Outcome 3, as citizens are unlikely to use the policies themselves due to low literacy rates. Rather, these policies can be used by local government institutions to support their engagement with their constituency. Distribution of the translated Seed Policy and the MINAGRI Citizen’s Charter to districts took place in December 2019 and January 2020 respectively and distribution to beneficiaries had not yet been completed at the time of the evaluation. Radio Programs: During Years 2 and 3 of the project, a total of 43, 30-minute radio programs on “Your Role as a Farmer” were broadcast on Radio Huguka. Radio Huguka was selected due to the coverage, broadcasting from Muhanga District in Southern Province. The radio reports a following of 4 million listeners, more that 70% of whom are farmers. However, survey and interview findings indicate that radio programs were not listened to regularly or at all, with 42% of respondents reporting having never listened to the program and 43% reporting having listened to the program on occasion. Of those surveyed, 14% reported regularly listening to the program and 3% reporting having called in or sent a message to the program. Rulindo district reported the least participation with only 49% of those surveyed reporting every having listened to the program while Nyaruguru reported the highest participation with 67% reporting listening to the program. 26% 16% 45% 14% 55% 71% 45% 82% 19% 13% 10% 5% 0% 20% 40% 60% 80% 100% Female Male Female Male Adult Youth Have seen Poster Have Read Poster Never saw poster 46 Table 34: Percent of Survey Participants Reporting Seeing or Reading Posters Adult (>30 Yrs) Youth (18-30 Yrs) Total Total Total Female Male Female Male Female Male Adult Youth Listen regularly 14% 16% 9% 41% 13% 21% 15% 18% 15% Listen sometimes 40% 49% 45% 32% 41% 46% 41% 41% 43% Never listened 46% 34% 47% 27% 46% 33% 42% 41% 42% Figure 30: Percent of Survey Participants Reporting Listening to EPGA Radio Programs by Age and Gender Source: EPGA Final Evaluation Survey Results March 2020 Figure 31: Percent of Survey Participants Reporting Listening to EPGA Radio Programs by District Source: EPGA Final Evaluation Survey Results March 2020 Male survey participants were more likely to report listening with 67% reporting having ever listened to a program, while only 54% of women reported ever having listened. While adults and youth had similar rates of reported listening, with 42% of adults and 41% of youth reporting having ever listened to a broadcast, there is a significant different between male and female participation. Survey findings showed that 73% of male youth and 66% of male adults reported having ever listened, while 53% of female youth and 54% of female adults reported having ever listened. The majority of women and youth participating in FDGs and interviews reported that they only listen to Radio Rwanda. Others reported not having a radio or not having time to listen to radio programs as barriers to participation. 14% 16% 9% 41% 40% 49% 45% 32% 46% 34% 47% 27% 0% 20% 40% 60% 80% 100% Female Male Female Male Adult Youth Listen regularly Listened sometimes or 1-2 times Never listened 36% 45% 45% 33% 51% 42% 47% 46% 39% 40% 46% 43% 16% 8% 13% 22% 13% 2% 1% 3% 5% 3% 3% 0% 20% 40% 60% 80% 100% Gakenke Nyamagabe Nyanza Nyaruguru Rulindo All Districts never listened sometimes or 1-2 times listen regularly called in or sent message 47 5.1.3.3 Outcome 2: Increased capacity of civil society to engage in policy analysis, budget monitoring and advocacy and ability to use collective approach to influence decisions related to sustainable agriculture Activities under Outcome 2 focused on building the capacity of civil society to engage in agricultural advocacy through trainings and provision of educational materials and facilitating opportunities for CSOs to use a collective approach to influence decisions. In order for CSOs to effectively engage in policy development or implementation and hold duty bearers to account, CSOs must have the capacity to do so. These capacities include understanding of policies, institutions and actors; ability to develop a strategy for policy engagement and use evidence to ensure recommendations are credible; capacity for communicating appropriately, either in the form of policy briefs or in presentations; and the ability to organize into networks for collective advocacy. 22 While CSOs may have the technical capacity in their given field, they often lack the ability to use evidence to prepare communication pieces or the capacity to effectively monitor budgets. As noted in the Lessons Learnt report commissioned by RDO23, CSOs noted that they lacked basic office facilities and resources and many reported working on a voluntary basis. In addition, while networks for joint advocacy exist at the national level, there is limited coordination between CSOs at the district level. Finally, in order for effective CSO engagement, local CSOs must have external opportunities for engagement at the local or national levels. However, as noted in the 2018 CSO Sustainability Index for Rwanda, CSO engagement with high-level decision makers is generally deemed low and their contribution is primarily limited to attending meetings. Often CSOs are not informed of policy issues in a timely manner and their comments come after meetings have taken place and policies are already adopted. Overall, CSOs are seen as reactive rather than proactive on policy issues.24 In addition, existing formal mechanisms for district based CSO advocacy at the local level is primarily limited to their participation in JADF meetings and currently there are no opportunities for their engagement at the national level. Below is a summary of the specific project components and how they contribute to the achievement of Outcome 2: Civil Society Capacity Building: The project activities related to capacity building included: • Conducting a Training of Trainers (ToT) for Trócaire staff and implementing partners; • Conducting trainings for members of civil society on policy and budget monitoring and advocacy with a focus on agricultural policy and gender analysis and integration; • Developing and distributing a training manual for CSOs on ‘the role of citizens and local entities in policy and budget monitoring and advocacy related to agriculture’ intended for use with citizen with limited skills or knowledge of public policy engagement and advocacy to understand how they can participate in the development, implementation and monitoring of public policies and programs in the agricultural sector; and • Developing and distributing a ‘strategy guide on policy and budget monitoring and advocacy in agriculture’ for CSOs with the intention of supporting CSOs to harmonize methodology, tools and activities related to agricultural policy and budget monitoring and advocacy. The intention behind the design of the capacity building component of Outcome 2 was appropriate, as it provided relevant skills to CSOs to participate in advocacy around agricultural policy as well as support CSOs to engage with each other. However, given the existing capacity of CSOs, one training was insufficient to achieve the desired outcome. In addition, in the absence of both formal and informal mechanisms for CSOs to engage with each other at the district level, the extent to which CSOs can harmonize methodology, tools and activities is limited, thus reducing the potential for joint advocacy. Specific findings from the evaluation include: 22ODI (2006) How Civil Society Can Be More Effective 23 RDO (2020) Enhancing Participatory Governance and Accountability Project: Lessons Learnt and Best Practice 24 USAID, FHI360 and ICNL (2018) The 2018 CSO Sustainability Index for Rwanda 48 Training and Training Manuals: During FGDs and interviews CSOs appreciated the training manual design, particularly the use of pictures for low literacy populations. “All of us read it because even illiterate people understand everything in the manual. The manual is still used because everything we teach our people it is from the book. I normally use it when showing new methods or how things are done because this book shows the difference of before practicing agriculture policy and after putting it in action.” –CSO member, Nyanza District However, at least one CSO noted that the training on budgets was insufficient for them to meaningfully participate in budget monitoring while another CSO used the training to support their internal budgeting: “We are thankful because of the trainings we got about budget monitoring, even though we haven't understood it well, at least we can now say that we have an idea about it.”—CSO member, Nyanza District “I can say that we got some trainings about policy and budget monitoring and advocacy in agriculture which was useful for our cooperative because for now we gain skills on how we can do budget monitoring which helps us to know if we get a profit or loss.” –CSO Member, Gakenke District Strategy Guide: According to RDO’s Lessons Learnt report, CSOs reported not having the strategy guide, a finding which was supported during FGDs and interviews with CSOs as a part of the final evaluation, with at least one CSO reporting not having received the strategy guide and other CSOs confusing the strategy guide with the training manual. Forums for advocacy: In addition to training CSOs, the project set up various forums for CSO engagement in advocacy on agricultural policies as identified by citizens. These forums included: • District level policy dialogue meetings; • National level policy dialogue meetings; and • Exchange meetings between agricultural stakeholders through JADF. In order for effective CSO engagement, there needs to be opportunities for their involvement at the district and national levels. The forums developed as a component of the EPGA project provided an opportunity for CSOs to advocate on behalf of their constituency on agricultural priorities and resulted in concrete actions to be taken at district and national levels to address the identified needs. During FGDs and interviews with CSOs, some commented on improved responses by local government to citizen’s expressed needs. In addition, survey findings highlight that CSOs feel that the majority of their concerns (88%), are responded by authorities. “We advocated for our citizens about Nkunganire25 and, at the following meeting, the mayor came. He was asked directly by the citizens and he promised to look into it. After a short period that was solved”—CSO member, Gakenke District The national policy dialogue meetings in particular played a significant role in the success of the EPGA project as the forum provided an opportunity for visible changes in policy delivery at the local level. However, the Lessons Learnt26 report noted that no local CSOs attended the national dialogues. Specific policy issues raised at the national policy dialogue meetings included: 25 This refers to The Smart Nkunganire System (SNS) which is a supply chain management system built by BK TecHouse Ltd in collaboration with Rwanda Agriculture and Animal Resources Development Board (RAB) to digitalize the end to end value chain of the Agro-Input Subsidy program. 26RDO (2020) Enhancing Participatory Governance and Accountability Project: Lessons Learnt and Best Practice 49 Table 35: Policy Issues Raised During National-Level Dialogue Meetings27 Policy/Issue Status Response National Policy Dialogue 1 Decentralize the process of seed and fertilizer distribution as reflected in MINAGRI guidelines Addressed MINAGRI issued guidelines decentralizing inputs distribution Facilitate farmers in soil testing Addressed RAB launched a mobile soil testing truck and motorcycles through the Caravan Project in April 2019, however there is still need to analyze fertilizer types for distribution. Production of local hybrid seeds better adapted to local soils Addressed RAB and Western SEED Company have initiated production of local Hybrid seeds, RHM (Rwanda Hybrid Maize) has been produced in Rwanda and farmers are planting it in season 2019/20 A. Increase access of mixed and high nutrient fertilizers to farmers Addressed RAB is in process of putting up the facility to produce blended fertilizers National Policy Dialogue 2 Gaps in Twigire Muhinzi extension Model Pending RAB is in the process of increasing the number of farmer promoters so that each village has one and will continue to build their capacity. RAB will assess Farmer Field School Facilitators to see which cells have/do not have them, trainings will match the priority crops grown in each area. Low productivity in acidic soils due to lack of lime Addressed The government subsidized lime at 50%. The districts should help all farmers who need lime to access it at subsidized price. Lack of agriculture loans for small holder farmers Pending Determined a comprehensive analysis should be carried out to understand why Financial institutions including Imirenge SACCOs do not have agricultural financing packages. Abrupt increase of maize seed prices Addressed Government will maintain subsidies for hybrid maize seeds developed in Rwanda and continue reducing subsidies for imported maize seeds. Unavailability of Irish Potato seeds in the districts of Nyamagabe, Nyaruguru and Rulindo Pending This issue has not yet been addressed Insufficient agro-dealers Addressed MINAGRI and RAB in collaboration with the Districts committed to increase the number of agro dealers so that each sector has 2-3 agro-dealers 5.1.3.4 Outcome 3: Increased knowledge of local authorities on participatory governance and are committed to address citizens' concerns Outcome 3 activities were designed to raise awareness of local government institutions about participatory governance to create institutions that are responsive to the needs of citizens and include citizen and CSOs in agricultural policy decision making, planning and implementation. Despite efforts and commitment on behalf of the government to develop a citizen centered governance, there are still existing barriers to citizen participation, including the costs, both in 27 Trócaire (December 2019) Agricultural issues paper to inform national policy dialogue; Trócaire (2019) Key Recommendations from the National Policy Dialogue on Agriculture Conducted on December 6th, 2019; feedback from implementing partners. 50 terms of time and logistics, and the general feeling that citizens and civil society, particularly those in rural areas, do not have the necessary level of competence to participate in decision making processes.28 Within this context, the EPGA project developed a training and accompanying materials to support local government in understanding citizen’s rights to participation in the agricultural sector as well as organized accountability meetings. These specific components are further described below: Training and distribution of training materials: Trócaire developed a training manual on “Participatory Governance for the Development of the Agricultural Sector” in both English and Kinyarwanda. This was followed by a two-day training for leaders at the sector level and district levels in order to improve knowledge and capacity of local authorities on participatory governance and engender a commitment to respond to the priorities expressed by citizens and civil society. Participants at the sector level included village committees, coordinators of the National Women’s Council at the village level and cell levels, coordinators of the National Youth Council at the village and cell levels, the cell executive secretary and SEDOs. At the district level, participants included the sector executive secretaries, agronomists, coordinators of the National Women's Council at sector and district level, coordinators of the National Youth Council at the sector and district level, staff of the Directorate of Agriculture, staff of the Directorate of Good Governance, the Organizations of Civil Society, cooperatives and churches involved in agriculture as well as JADF permanent secretaries and JADF committee chairpersons. Accountability meetings: Accountability meetings were an opportunity for local leaders to exchange information on their experiences and challenges in responding to issues raised by citizens and civil society. Common unmet needs were identified for inclusion in the national policy dialogue meetings. Interviews with local government, including SEDOs and sector and district level agronomists, indicate that the activities under Output 3 was appropriately designed and implemented. “Before the project, it was not easy for us to visit the community to find out the challenges they face because we often had a lot of work and it was difficult for us to have time to visit the community, but when the project started it helped us reach out to the community. The project organized different meetings where we have participated in discussions on how we can solve community issues and even encourage them to participate in agricultural policy”—Agronomist, Gakenke District 28 Never Again Rwanda (2018) Local Government Imihigo Process: Understanding the Factors Contributing to Low Citizen Participation 51 5.2 Efficiency 5.2.1 Achievement of Planned Objectives and Constraints The EPGA project was successful in achieving the project objectives as set form in the Project Results Framework with few reported delays. Specific challenges as noted in the Annual Reports to USAID are highlighted below. Note that figures are inclusive of up to the first quarter of Year 4. 5.2.1.1 Outcome 1: Empowered citizens, particularly women and youth, who are able to engage in agricultural policy and budget monitoring in 5 districts and hold public authorities accountable Table 36: Achievement Against Planned Activities for Project Outcome 1 Out￾puts Activities Results Achieved Planned Actual 1.1 Activity 1.1.1: One training of 3675 citizens especially women and youth and 819 village and cell leaders, on their roles and responsibilities in terms of policy and budget monitoring and Advocacy regards to agriculture issues. • 3,675 citizens • 819 village and cell leaders • 3,595 citizens • 830 village and cell leaders Activity 1.1.2: 12 feedback meetings conducted by community committee members 12 meetings per year Data between Yr2 and Yr3 not comparable Missing Yr. 4 data Activity 1.1.3: Organize six exchange meetings between citizens and local leaders around citizen participation in planning especially in agriculture planning 6 exchange meetings 1.2 Activity 1.2.1: Translate and design user friendly 5000 copies of summarized versions of agricultural policies over 3 years 5,000 copies of 3 agricultural policies • 1,667 copies of the Seed Policy • 1,667copies of the MINAGRI charter • copies of the Small-Scale Irrigation Policy to be printed in Year 4, Q2 Activity 1.2.2: Produce and distribute 14,000 posters on citizens’ rights and responsibilities, 5000 copies of booklets on citizen’s role and responsibilities in terms of policy and budget monitoring regards to agriculture • 14,000 posters • 5,000 booklets • 4,667 posters in Year 2 • 4,678 posters in Year 3 • 4,668 posters in Year 4 • 25,000 booklets in Year 2 Activity 1.2.3: Produce and disseminate 36 radio broadcasts and 1 documentary film on citizen participation in policy monitoring especially in the domain of agriculture and budget monitoring • 36 radio shows • 1 documentary film Radio shows broadcasted • 19 in Year 2 • 12 in Year 3 • 3 in Year 4 1 documentary film completed 1.3 Activity 1.3.1: Establish 63 community committees at village level for policy and budget monitoring and advocacy with regard to sustainable agriculture 63 community committees established 63 community committees established Activity 1.3.2: Organize one training for community committees on policy, budget monitoring and advocacy One training per sector for 315 • 315 members trained • Refresher training for 32 community committee leaders 1.4 Activity 1.4.1: 72 telephones with adequate software for collecting and sharing information are purchased and distributed to each of 63 community committees, and 9 staff on agricultural policy and budget monitoring 72 telephones purchased and distributed • 63 phones distributed to Community Committees • 9 phones distributed to project staff • 10 additional phones purchased to replace broken phones 52 Out￾puts Activities Results Achieved Planned Actual Activity 1.4.2: One training for community committees, and program staff on telephone software operations 72 people trained • 63 Community Committee members • 6 implementing partner staff • 2 Trócaire staff • 63 Community Committee leaders participated in re￾fresher trainings 5.1.1.2 Outcome 2: Increased capacity of civil society to engage in policy analysis, budget monitoring and advocacy and ability to use collective approach to influence decisions related to sustainable agriculture Table 37: Achievement Against Planned Activities for Project Outcome 2 Outputs Activities Results Achieved Planned Actual 2.1 Activity 2.1.1: Develop a training manual on policy and budget monitoring and advocacy Develop 1 training manual Developed in Year 2 Activity 2.1.2: Develop a strategy guide in policy and budget monitoring and advocacy and guidelines for agriculture Develop 1 strategy guide Developed in Year 2, 400 copies distributed 2.2 Activity 2.2.1: One, five-day training of the trainers targeting the three implementing partners and Trócaire staff on policy and budget monitoring and advocacy Conduct 1, 5- day training 16 participants in the ToT conducted in Year 2 Activity 2.2.2: One training of civil society organizations based in the districts on policy and budget monitoring and advocacy with focus on agricultural policy and gender analysis and integration Conduct 1 training of CSOs 346 CSO members trained in Year 2 Activity 2.2.3: Conduct three policy dialogue meetings and advocacy activities at district level between communities, civil society and public authorities. Conduct 3 policy dialogue meetings 3 policy dialogue meetings conducted: • 389 participants in 5 dialogue meetings in Year 2 • 160 participants in 2 dialogue meetings in Year 3 • 358 participants in Year 4 Activity 2.2.4: Conduct two policy dialogue meetings and advocacy at national level with stakeholders including civil society and national government officials. Conduct 2 policy dialogue meetings 2 conducted, 1 in Year 3 and 1 in Year 4 Activity 2.2.5: Facilitate six exchanges of experiences and learning among stakeholders intervening in agriculture and policy monitoring through JADF and other networks in country and in neighboring countries. 6 exchange experiences • 5 exchange meetings in Year 2 • 11 exchange meetings in Year 3 • 2 exchange meetings in Year 4 Activity 2.2.6: Document and disseminate lessons learnt and best practices in local governance, policy and budget monitoring. 1 document Completed in Year 4 Activity 2.2.7: Conduct one research on the involvement of citizens particularly women in local governance, policy and budget monitoring in agriculture area 1 document Completed in Year 3 53 Outputs Activities Results Achieved Planned Actual 2.3 Activity 2.3.1: Organize 2 radio broadcasts on agricultural policies by using most popular radio stations 2 broadcasts planned • 1 conducted in February 2018 • 1 conducted in August 2018 • 1 conducted in October 2019 in preparation for the national policy dialogue meeting 5.1.1.3 Outcome 3: Increased knowledge of local authorities on participatory governance and are committed to address citizens' concerns Table 38: Achievement Against Planned Activities for Project Outcome 3 Outputs Activities Results Achieved Planned Actual 3.1 Activity 3.1.1: One training of local authorities and community leaders in participatory planning and budgeting One training • One training conducted with 844 participants (local authorities and community leaders) in Year 2 • Refresher conducted for 410 participants in Year 4 Activity 3.1.2: Develop 100 participatory governance modules targeted at local authority officials 100 governance modules developed • 300 copies of the training module printed in Year 2 • 200 copies of participatory governance module in Year 4 Activity 3.1.3: Organize six accountability meetings with local authorities at district level to follow up the implementation of recommendations submitted by CSOs concerning the expressed needs of citizens 6 accountability meetings conducted • 5 accountability meetings (1 per district) conducted in Year 2 • 5 accountability meetings in Year 3 • 2 accountability meetings in Year 4 5.1.1.4 Constraints to Achieving Project Objectives While the majority of the project outputs were accomplished with minimal changes to the proposed delivery or workplan, the following outputs experienced delays: Translation of Agricultural Policies: The project planned to translate three policies deemed to be most useful for local leaders into an accessible format, including less technical jargon and printed in Kinyarwanda. RDO, through consultation with local leaders and CSOs, identified policies during the first year of project implementation which included the Irrigation Policy, Land Use and Consolidation Policy and the Seed and Fertilizer Distribution Policy. In Year 2, the Irrigation Policy, Seed Policy and Fertilizer Policies were translated and submitted to MINAGRI for approval. However, due to the adoption the PSTA 4 a number of agricultural policies needed to be reviewed for alignment with the new agricultural sector strategic plan. Therefore, only the translation of the Seed Policy was approved and MINAGRI proposed translation of the Livestock Policy and the MINAGRI Citizen’s Charter as replacements. The project finally settled on the translation of the MINAGRI Citizen’s Charter and the Small-Scale Irrigation Policy which is still under review and awaiting approval. 5.2.2 Adequacy of Planned Resources As the EPGA project was an advocacy project, the project resources primarily consisted of trainings, meetings, and the development of information pieces including training manuals, policy documents, information booklets and posters. In the case of the information booklets, it was noted 54 that the original quantity printed was insufficient to meet the demand and a surplus in the budget was able to cover additional printing of the materials. 5.2.3 Delivery of Project Funds and Activities While asked about timeliness of project activities, beneficiaries all noted that they were well informed of the timetable for activities in advance and that when changes occurred, they were informed in advance. In addition, they appreciated that project staff respected their time and attended meetings and trainings on time. Project staff also noted that they were able to implement activities as planned, except on occasion when meetings were delayed due to last minute commitments by government officials. When planned resources were deem inadequate, surplus funding was used to address these needs, such as in the case of booklets on rights and responsibilities of citizens. The project initially printed 5,000, but increased printing to include an additional 20,000 copies. Trócaire and implementing partners noted that funds were, with few exceptions, distributed on time to meet the project needs. The distribution of funds to project partners improved with the adoption of mobile payments as it made reconciliation of funds dispersed to beneficiaries easier to document. 55 5.3 Effectiveness 5.3.1 Effectiveness of Project Management Arrangements The EPGA project was designed in coordination and collaboration with the implementing partners, with each organization agreeing to their roles and responsibilities at the project design phase. As a result, partners reported being aware of their roles and responsibilities for project implementation. Steering committee meetings were conducted on a quarterly basis, during which time roles and responsibilities were further clarified. These roles and responsibilities included opportunities for implementing partners to take on management of specific project components. Project implementation was monitored through quarterly steering committee meetings. Steering committee meetings were noted to be democratic in that each partner took turns hosting and chairing the meeting. One implementing partner appreciated that this approach ensured that Trócaire management could be questioned and be held accountable. In addition, implementing partners submitted monthly financial reports and quarterly narrative and financial reports coupled with quarterly visits by the program accountant to review the financial reports with clear recommendations on improvements to internal controls made and eligibility of costs. Trócaire also developed a Monitoring and Evaluation Plan in August 2018 that specified internal roles for monitoring and reporting on project implementation. Technical components of the project were addressed during quarterly technical meetings, field visits and ad hoc meetings as required. Trócaire and implementing partners also noted that the project made regular use of email, phone calls and WhatsApp to share project information. In addition, Trócaire organized a study visit to Malawi for program staff and implementing partners during the third year of the project, in March 2019, with the objective of sharing experiences, particularly on CSO involvement in participatory governance, creating alliances to optimize advocacy efforts and methods to engage communities, particularly women and youth, in policy and budget implementation and monitoring. Capacity building is a central element of Trócaire’s approach to working with implementing partners. Trócaire, through their Partner Capacity Assessment and Support framework, provided on-going organizational capacity building support to each partner based on assessment findings. In addition, Trócaire offered a range of specific trainings for implementing partners throughout the duration of the project, including: Organizational Capacity: • USAID Regulations • Rwandan Tax Law • Disaster Risk Reduction • Training on Sage Accounting Software • Supply Chain and Logistics • Website Design and Maintenance • Risk Management and Updating/Development of Procurement Policies Project Management Capacity: • Results Based Management • Integrated Water and Resource Management and Climate Change Adaptation • Writing Proposals, Reports and Success Stories • Advocacy Using Trócaire’s Citizen Advocacy and Monitoring Framework • 10 Principals of Social Norms Change Findings from interviews with implementing partners indicate varying levels of administrative and technical support received by Trócaire project management. One partner reported a very conducive working relationship with Trócaire program management and noted that Trócaire was always 56 available to meet. They specifically highlighted technical support provided by Trócaire in the preparation of policy dialogues as they did not have previous experience in organizing high-level national meetings. Trócaire and the implementing partner formed a joint team to organize the meetings and support engagement of development partners. While they noted receiving appropriate technical support, project funds and materials on time, some reported less satisfaction with the support received by Trócaire project management. They specifically highlighted a lack of coordination between implementing partners in the development of project action plans and timelines to ensure uniform implementation of activities across the project districts. However, Trócaire addressed this by improving coordination through quarterly review meetings and field visits with implementing partners during the second half of the project. In addition, some implementing partner staff reported a lack of Trócaire involvement in activities, including monitoring project implementation at the level of beneficiaries. As Trócaire is not an implementing agency, this finding may reflect a lack of understanding of Trócaire’s role on behalf of implementing partner staff. Field visit reports demonstrate Trócaire staff regularly visited implementing partners in the field and provided support to address challenges surrounding program implementation as well as organizational capacity as identified through partner capacity assessments. 5.3.2 Stakeholder Involvement and Project Ownership While there was no formal mechanisms to engage beneficiaries in the design of the project, the implementing partners had significant experience working with rural communities in agriculture and were aware of needs, particularly in the area of participation. The design of the project was in collaboration with the implementing partners to utilize their knowledge of the proposed beneficiaries. Once the project was approved, the project engaged with stakeholders to explain the project and verify project components. Stakeholders reported that the EPGA project staff were very responsive to their needs. During interviews, SEDOs, sector agronomists and district agronomists all stated that the EPGA project sought their input into the design of the project, from the selection of the beneficiaries to project components. All CSO members interviewed also noted that they were consulted in the implementation of the project and that their experiences and knowledge of the beneficiaries was respected by the project. FGDs and interviews with Women and Youth beneficiaries indicate that they felt that they were able to contribute to the EPGA project. However, findings also suggest that they attribute the positive changes in agricultural support received to the EPGA project rather than attributing it to their own advocacy at the level of the Community Committees. There is also some indication of community ownership over the project as evidenced by project beneficiaries developing their own initiatives including forming savings groups as a component of the Community Committees and planting of kitchen gardens. One field officer noted that through their participation in the EPGA project, they began to identify solutions to their own problems. 57 5.4 Impact 5.4.1 Intended and Unintended Impact Increased Awareness of Rights and Roles: Beneficiaries reported that trainings on roles and responsibilities coupled with community meetings and posters effectively increased their understanding of and participation in the implementation and monitoring of agriculture policies and budgets. “I participated in the 2017 training on our roles and responsibilities for agricultural and budget monitoring, where we learned that the budget was found in the government gazette. I learned to follow up and we now have a facilitator in the sector level who follows it and shares information during sector meetings about what the budget is going to do for us.”–Female Beneficiary, Gakenke District With an awareness of their rights, the majority of respondents also reported a significant increase in the confidence of rural people and farmers to share their priorities with local government officials and hold them accountable. “Now if a citizen has ordered seeds or fertilizers and it is too late to come, he himself asks and finds out where it is and knows why he didn’t get it on time.”–SEDO, Nyaruguru District “We didn’t know before that we had to be involved in budget preparation and our role in controlling its use, we thought it was just for leaders. Knowing this helped me to have the courage to comment and share my ideas at village meetings. We dare to talk about what’s going wrong”—Female Beneficiary, Nyanza District “There was a leaflet that had someone asking an agronomist when they may get fertilizers. This gave me courage to ask the agronomist why my passion fruit was not growing well, and the agronomist helped me to get pesticides. I used to be afraid of asking but now I know that is what the leaders are there for.” –Male Youth Beneficiary, Nyanza District Increased Voice in Agricultural Policy Decisions: Evaluation findings suggest that rural people do have a greater voice in agricultural policy decisions, particularly around implementation of existing policies, as a result of the project interventions. As reported in section 5.1, 81% of beneficiaries surveyed reported that they feel that local authorities respond to their expressed priorities with 84% being able to list at least one example of how authorities have responded to their priorities within the preceding 12 months. Prior to the start of the project, only 57% of those surveyed reported that local authorities responded to their expressed priorities.29 Interviews and FGDs with beneficiaries and implementing partners also highlight that there is greater collaboration and sharing of issues between individual farmers and local leaders through the Community Committees or “VACs.”30 The evaluation also found that information sharing is bi-directional with farmer’s priorities and needs being shared with local leaders and the response by local leaders shared with farmers. “Before this project came about, we felt that the farmer was worthless . . . Farmers are now valued and at all village meetings, they do not end the meetings without mentioning agricultural policies.”—Female Beneficiary, Gakenke District 29 Trócaire (2017) Baseline survey for “Enhancing participatory governance and accountability” project 30 Due to the interchangeable use of VAC and Community Committee, it is impossible to determine if beneficiaries are referring to the government sponsored VAC or the project sponsored Communities Committees. 58 “We attend village agriculture committee meetings, and that’s where the agricultural budget is being prepared and when we give ideas based on our needs. They will prepare this budget based on our ideas. This showed us that our participation is valued which further increases our participation.”—Male Youth Beneficiary, Nyanza District “Sometimes we attend meetings and you find that a citizen is asking about budget monitoring or why the fertilizers quantity has been reduced. This shows that they have started to engage in budget policy.”—CSO member, Gakenke District Policy Decisions: Rural farmers in the project catchment area directly impacted changes in national government level policy implementation as a result of the project. This was achieved through the national policy papers and the two national agricultural policy dialogue meetings organized by RDO. Policy papers used the data collected by Community Committees through the SurveyCTO and verified through FGDs with beneficiaries and interviews with local leaders to identify agricultural policy issues that could not be addressed by local government institutions. Policy decisions mentioned during the final evaluation that were based on direct feedback from beneficiaries included the following: • Timely access to improved seeds and inputs (2018 national dialogue) through decentralization of the responsibility from RAB to the Districts. • Lime subsidies (2019 national dialogue) and mobile soil testing units (2018 national dialogue) to ensure soil fertility. • Increased availability of agro-dealers in rural areas in order to increase access to agricultural inputs (2019 national dialogue). Agricultural Production: Some beneficiaries noted that their increased engagement in advocacy for agricultural policies has had a direct impact on their agricultural production. This is attributed to timely access to improved seeds and other inputs, such as fertilizers and pesticides as well as the confidence to approach sector and district level agronomists for support when needed. “Before we lacked seeds and fertilizers and, at that time, we thought that no one else would listen to us and we kept quiet. After the training, we realized that we have a say. Now, we have village agricultural committees and agronomists. They use villages meetings where everyone has the right to speak to get our feedback and address the issues we have. We now get seeds and fertilizers on time, which increases productivity. Where we used to harvest 10kgs of corn, we now harvest 1 ton. –Female Beneficiary, Nyanza District My attitude has changed a lot because we are starting to implement agricultural policies that we see as different from the way we used to cultivate. We have become more knowledgeable through trainings, village meetings with the agronomist and visits from agricultural extension agents. All of these encourage us to implement agricultural policies and anyone who has a problem would be given a chance to ask what he or she doesn’t understand. Also, when you put things in practice and you see that the more productivity, you keep doing it and it even helps you become even more knowledgeable. –Female Beneficiary, Gakenke District Youth: Findings from both the survey and qualitative data collection show that youth feel that they can meaningfully participate in agricultural policy decision making. Findings from the survey, highlight an increase confidence in their participation in their role in the development of agricultural policy (88% female and 95% male) and their role in monitoring budgets (62% female and 86% male). While the final evaluation findings did not highlight any unintended impact on youth participants, findings from the Lessons Learnt report show that some youth farmers reported finding ways to make money through agriculture and that youth are more interested in income￾based agricultural work as compared to subsistence after attending trainings that focused 59 particularly on the increasing demands for mechanization, a connection to the outside world, and evidence that agriculture can be a profitable. The findings also noted that citizens and village leaders noted the positive work and involvement of the youth, their lack of fear and admirable self confidence in speaking with local officials and in holding duty bearers to account.31 “While we used to feel that these are government programs, so we treated them as if they didn’t concern us, but today we know it is our program that is being done for us.” –Male Youth, Nyaruguru “Before the EPGA project started, it was difficult to comment on something during the meetings because we were afraid of our leaders. But the project showed us that they really need our feedback. This built our confidence and gave us courage to ask for what we want.” –Female Youth, Nyanza Sustainable Community Committees: Community Committees are an accessible entry point for community members to collectively advocate for their agricultural needs. While it was unintended by the program, many committees, encouraged by project field officers, set up savings groups to support the purchase of agricultural inputs in order to ensure their sustainability as a community structure. In addition, Community Committee members also became a resource for their local communities, adopting and providing information on improved agricultural methods. Civil Society Capacity: Civil society organizations reported an increased understanding of the need for citizens to engage in agricultural policy and budgets. Some CSOs also reported that the EPGA project has supported their own work with community members by increasing their awareness of agricultural policies and the use of improved agricultural methods. In addition, the project contributed to evidence based advocacy with the development of issues papers leading up to the national policy dialogue meetings as well as research conducted on women in agriculture. “We work as partners, and some beneficiaries in this project are also our beneficiaries. So, you find that it has simplified our work, if my duty is to work on women’s empowerment and she has already been trained in budget advocacy or participation, you will find that me telling her about savings will not be difficult”— CSO Member, Gakenke District As a Civil Society organization and implementing partner, RDO reported a significant change in their capacity to advocate at the national level through development of policy papers and organizing policy dialogue meetings. Responsive Local Government: As a result of the EPGA project, local government institutions reported that they now recognize the importance of citizens advocating for their needs and now rely on farmers and communities to do so in order to improve their own work. “In terms of agricultural policy and agricultural budgets before the project we used methods called ‘Top downing planning’ where we used to say, ‘the district has prepared this, now the community is supposed to follow this’ and then you find that the community doesn't want to implement it and activities are unused. Now we see that the district should consider first and ask the community what they feel is most important and we do it through budgeting consultation where we go to listen to the public’s opinion before preparing the budget. We ask the community: ‘what do you need most in next year’s budget in maize farming’ and they say, ‘We need a dryer.’ When I prepare the agricultural budget, I consider their opinion. The project helped farmers to feel they needed to raise their voices to meet their needs but also helped us to change our mindset and make us feel that we have to work with the community.” —Agronomist, Nyanza District 31RDO (2020) Enhancing Participatory Governance and Accountability Project: Lessons Learnt and Best Practice 60 “Now that people can advocate for themselves, it has become very easy for us because sometimes you can find they are problems or issues which public employees can't just address to the authorities due to fear of losing our job. But now that people are aware that they have to speak for themselves they rise their problems during meetings. We thank this project a lot because it became as a channel between local leaders and people. Before it was very difficult for people to come to our offices due to fearing us, feeling that we are superior to them. But, because of this project, we now work hand in hand.”—Sector Agronomist, Nyaruguru District 5.4.2 Use of Technology The use of technology in the EPGA project included payment of incentives via mobile money platforms and the use of mobile data collection (MDC). E-payments: Mobile money platforms were used for the transfer of incentives to beneficiaries. These platforms are already well developed and in wide-spread use throughout Rwanda. One implementing partner noted that the move to mobile money to transfer incentives to beneficiaries improved their financial reporting as account managers could easily track all payments made. Prior to implementing this system, the implementing partner noted difficulties in obtaining approval of financial reports, delaying the subsequent release of funds from Trócaire. Mobile Data Collection: MDC was rolled out in August 2018, through distribution of smartphones installed with SurveyCTO software for recording and sharing of priorities by Community Committees with implementing partners and Trócaire. Smartphones with the installed software were distributed to 63 leaders of the Community Committees with the expectation that they would complete monthly reports highlighting issues and priorities around sustainable agriculture. Prior to the distribution of the smartphones, Community Committees completed monthly handwritten reports for submission to the project’s implementing partners. Community Committee members and implementing partners surveyed reported that this required significant time and resources in both the preparation of the report and transportation of the report to the implementing partner. With the introduction of the MDC platform, Community Committee members and implementing partners received a one-day training on the use of the SurveyCTO software. The SurveyCTO form captured information on a monthly basis based upon priorities and challenges identified during monthly community meetings at the village level. Beneficiaries reported ease of data entry as much of the data was pre-entered for selection from drop-down lists. Information collected on a monthly basis included location information on who submitted the data, date of submission, key highlights from the month and responses to the following questions: Did you participate in any advocacy issues during the reporting period? List of advocacy issues To which authority were advocacy issues raised? What was the outcome of advocacy? Number of meetings organized during the reporting period related to agriculture Who organized meetings on agriculture during the reporting period? Agricultural issues raised during the reporting period-selected from a list of options Data was analyzed monthly by Trócaire for both accuracy of data entry and to highlight advocacy needs and shared with implementing partners. Implementing partners reported that a one-day training was insufficient and that they lacked capacity to analyze the data themselves. Community Committees and implementing partners surveyed all reported satisfaction with the use of MDC for submission of monthly reports. As previously noted, Community Committees and implementing partners both highlighted ease of use and submission of monthly reports and implementing partners and Trócaire highlighted the real-time access to community level agricultural advocacy issues. Challenges raised by surveyed beneficiaries included reports that the 61 training was not sufficient for them to fully understand how to use the software or troubleshoot problems encountered. Implementing partners also noted that they needed to provide in-person technical support to Community Committee members, including re-installation of software. Community Committee members also reported challenges associated with poor network and unavailability of electricity to charge the smartphone. 5.4.2.1 How did the use of technology enable the project to achieve intended results? Analysis of data collected from Community Committees was used by implementing partners and Trócaire to determine the advocacy issues raised at the community level that could not be addressed by local government institutions. Key issues identified were then compiled and addressed in policy papers and in the two national dialogue meetings organized by RDO and Trócaire. The ability to report in real-time and the observable outcomes from the national dialogue meetings supported the projects intended objective of promoting effective participation of citizens in the development, implementation and monitoring of agricultural policy to ensure that the priorities of vulnerable people are responded to. During interviews, some Community Committee members noted that when they submitted paper-based reports, they were not sure who would read them, but with MDC they knew that their priorities were being seen by more than just the project field officer. 5.4.2.2 What have been the intended and unintended use of technology by the target population? Findings from FGDs and interviews with Community Committee members, implementing partners and Trócaire staff show that the smartphones installed with the SurveyCTO software were primarily used as intended to share information collected through monthly community meetings with project staff. However, the smartphones were also used to create a WhatsApp group for Community Committee members to share information between sectors and districts. This unintended use was mentioned in two interviews in different districts. 62 5.5 Sustainability 5.4.1 Sustainability of Project Interventions Citizen Engagement: One of the successes of the project is the increase in citizen engagement and advocacy in agricultural policy. With the knowledge of their rights and responsibilities, rural people will continue to hold local authorities accountable and engage with local agronomists for support. However, as previously noted, the majority of engagement takes place at the Community Committee therefore, continued engagement at levels seen during the life of the project may be dependent upon the continuation of the Community Committees. In addition, beneficiaries primarily advocated for better implementation of existing policies, including timely distribution of inputs, rather than changing existing policies to better meet the needs of rural farmers. Community Committees: Community committee members and project staff are confident that the Community Committees will continue to function as a village level organization, even in the absence of the project as they have cemented a role for themselves in supporting engagement of rural farmers in agricultural policy implementation. In addition, while meetings may not take place monthly as during the project implementation, project staff noted that the savings groups organized by Community Committee will continue, which will continue to bring the members together on a regular basis. “Our role didn't change because we are still doing our responsibilities. People in our community ask us why is EPGA not giving us more trainings yet. But the project gave us enough trainings which we shall continue to practice even if it comes to an end.” –Female Community Committee Member, Nyanza District “I think the lasting effects of the EPGA project will be the community engagement and savings. The fact that the community works together and saves and sees that is what benefits them the most and will keep doing it. Now that the public knows that while saving it will never run out of seeds during planting, this gives hope that the community will continue to do so even when the project is over.”—Sector Agronomist, Nyanza District Use of MDC: During the last quarter of the project, Trócaire changed the MDC software to CommCare, with whom Trócaire has a global contract to ensure that data collection will continue beyond project completion and it is expected that Community Committees will remain with the smart phones installed with the CommCare software. In addition, Community Committee members were tasked with developing plans for how they continue to provide for phone credit, which included the development of savings groups. While the Committee members reported that the project has not communicated with them post-project expectations for continued reporting, the majority of those interviewed responded that they will continue to collect data and submit reports to the implementing partners after the end of the project. However, there are notable constraints: • Use and Analysis of Data: It is not clear to what extent this information will be used given that the implementing partners report lacking the capacity to analyze the data. While some suggested that sector and district government also receive the reports, they will also lack the capacity to analyze and use the data collected. As implementing partners will remain working in the project districts, they can presumably continue to use the data however, it will unlikely be compiled and shared at the national level as during the life of the project. • Troubleshooting: As noted by project staff, the use of software requires regular updates and troubleshooting, which was managed by implementing partner staff during the life of the project. • Use of Phones. In addition, there is no plan for phone replacement as smartphones if lost or no longer working as smartphones have a limited lifespan. Local and National-level Engagement: Building the capacity of local leaders to support citizens in identifying agricultural priorities and implementation has successfully increased engagement between citizens and their local leaders. As noted, citizens have more confidence to engage local 63 leaders and local leaders understand the benefit of this engagement. However, as there is high turnover in government leadership at all levels, maintaining government institutions that support citizen engagement, will require continued training and advocacy. Another successful component of the project was the formation of policy papers and engagement of national-level leaders to address agricultural policy issues raised at the community level. With the training of local sector and district leaders through the project coupled with an increasingly engaged rural constituents, local government will likely continue to engage with and respond to the needs of local farmers to their ability. However, in the absence of an external actor analyzing the unmet needs, summarizing into policy documents and organizing meetings for advocating for policy changes at the national level, the extent to which unmet needs at the local level will reach national-level decision-makers is less likely. 5.4.2 Institutional Sustainability Trócaire’s partnership approach highlights capacity building of implementing partners as demonstrated through provision of regular training opportunities as well as the Partner Capacity Assessment and Support Framework whereby organizations were assessed and provided with support tailored to their individual needs in both project management and organizational management. Findings from key informant interviews with implementing partners highlight the institutional support provided by Trócaire during the life of the project and given the financial resources, it is feasible to expect partners to be able to continue to engage with communities on agricultural policy and advocacy. Implementing partners have already indicated that they will continue to follow-up on specific policies relevant to their communities. In addition, RDO, through support from Trócaire and various consultants, reported having the capacity to continue developing policy paper and engaging with national and international level stakeholders to carry out agricultural dialogues in the future. 64 6. Conclusions and Lessons Learned Findings from the final evaluation highlight that the project contributed to a visible increase in engagement of citizens in advocacy on agricultural policy, principally focused on the implementation of agricultural policies rather than advocating for the development of new policies. End values for impact and outcome level indicators illustrate an increase in citizen satisfaction and confidence with the role that they play in the implementation of and the monitoring of implementation and budgets for agricultural policies. While actual monitoring of agricultural budgets is beyond the capacity of the majority of participants, they now understand that they have a right to participate in the monitoring of how the budgets are spent and are requesting for further capacity building in order to actively engage. The project’s success lies in educating citizens on their rights to participate in the agricultural policy dialogue, developing a forum for their advocacy on a local level through the establishment of Community Committees, developing a receptive local leadership through awareness of participatory governance and creating opportunities for advocacy to take place at district and national levels. In addition, the use of MDC to capture advocacy needs and outcomes ensured that the project could identify needs that were not able to be resolved at a local level and bring them to a national audience. As a result, problems related to implementation of policies, such as timely access to improved seeds and fertilizers, were addressed at the national level, the impact of which can be seen at the level of the farmers who are now reporting greater yields. Lessons learned: Specific lessons learned are noted below and are intended to complement the lessons learned highlighted in the Enhancing Participatory Governance and Accountability Project: Lessons Learnt and Best Practice report commissioned by RDO in April 2020. • Outcomes from advocacy activities, particularly at the national level, take time to be realized. Projects need to ensure adequate time commitment for beneficiaries to see the results of their advocacy work and even when an advocacy project is targeted to a few sectors, the impact is far reaching as citizens located outside of the project catchment area will also benefit from improved policy implementation. • When working with low-literate, rural populations, low literacy materials are effective for both sharing information directly with citizens and providing those who work with rural citizens, including local government and CSOs, effective tools for sharing information. • It is important to define the extent to which a project expects participants, both citizens and CSOs, to engage in monitoring of budgets. In a context where many people lack basic literacy skills, it is not feasible to expect that they would have the numeracy or financial literacy skills to participate in budget monitoring. In addition, many local leaders themselves lack the capacity to meaningfully participate in budget monitoring32 or access to the budgets. • Understanding the rights of citizen participation in governance is necessary, but in order for successful engagement with citizens to take place, local government actors need to realize the benefits of participatory governance and the added value of citizen contributions to agricultural policy decision making. • Farmers want to be seen as full participants in decision-making processes, not just as beneficiaries of government programs and there is a need for the focus of policy making to shift from a top down policy development approach to a participatory approach that engages and enables farmers to make and shape the policies that affect them.33 • Mobile Data Collection is an effective and efficient mechanism for sharing local needs and priorities with project staff. However, with the adoption of any new technology, it takes time to ensure that community members understand how to manage the technology and for those responsible for collecting the data to develop the skills to be able to analyze and effectively use the data. 32 Never Again Rwanda (2019) Local Government Imihigo Process: Understanding the factors contributing to low citizen participation 33 RDO (2020) Enhancing Participatory Governance and Accountability Project: Lessons Learnt and Best Practice 65 7. Recommendations The following programmatic recommendations are based on analysis of the findings from survey respondents, FGDs and interviews with beneficiaries and key informants. The operational recommendations are based on an analysis of program documents as well as key informant interviews. 7.1 Programmatic Recommendations General Recommendations • The use of the radio to share messages should be analyzed to ensure that it is the appropriate medium to reach the target population, particularly women. At the project design, beneficiaries should be consulted to determine media types (radio, social media, print), sources (Radio Rwanda, etc.) and timing (time of day, day of the week) used to access educational information in order to design interventions that reach the target population. After programs or broadcasts air, small-sample surveys should be conducted to determine if multi-media components are reaching the target population and if the messaging was understood or if adjustments to approach need to be made. • With low literacy rates amongst the target population (27% of the adult population in Rwanda is illiterate, 31% of women and 22% of men)34 low literacy materials developed by the EPGA project were the most appreciated by citizens, CSOs and local governments. Given the delays encountered with translating official agricultural policies, consider developing low-literacy policy briefs for distribution and display. This would also expand the number of policies that the project can address. • As participants noted that the timing of trainings was not sufficient given the material to be covered and that many of the topics were too novel or complex to be understood within the context of a single training event, trainings should be organized into modules and be carried out through the life of the project, occurring in coordination with events where learning can be demonstrated. Modules should build upon each other, first introducing the participatory governance as a right and benefits of citizen participation in agricultural policy. This would be followed by modules covering the different levels of participation, from policy implementation, monitoring and budgets. The content should be flexible to address specific policies of interest to the participants and facilitators should be knowledgeable in the subject and sufficiently flexible to be able to address participant interests and needs. • As low-literacy and low numeracy rates make understanding budgets difficult, more time should be devoted to this topic. In addition, beneficiaries need opportunities to see and comment on budgets. A key area to focus on is to further develop CSO capacity in budgets in order to ensure CSOs are able to skillfully translate key budget components into easy to digest information to support citizen participation while also involving citizens in their own budget analysis processes. Policy Environment • In order to shift rural citizen participation from passive recipients of policies into fully engaged participants in decision-making processes, the government should be encouraged to shift the focus of policy making from the development of policy products via a top down approach, to a more collaborative, participatory approach that engages with CSOs and those citizens most affected in the identification and formulation of agricultural related policies.35 Youth Engagement • As agreed upon during the National Policy Dialogue meeting in December 201936, the role of youth in development projects should be reflected in the activities, not just in number or percent of participants. Youth have different agricultural priorities and needs, specifically around agriculture for income generation, including mechanization and access to markets, and projects 34 UNESCO Rwanda Country Statistics http://uis.unesco.org/en/country/rw 35 RDO (April 2020) Lessons Learned and Best Practices Report 36 Trócaire (2019) Key Recommendations from the National Policy Dialogue on Agriculture Conducted on December 6th 66 activities should be designed to specifically address these unique needs at the design phase including incorporating information on existing relevant policies in trainings and project media, as well as supporting the implementation of the PSTA 4 by engaging local government institutions to develop linkages between youth and private sector opportunities including internships and trainings.37 • Youth have higher rates of literacy (84% for male youth and 89% for female youth)38 and more readily adopt new technology, therefore, communication mediums to reach youth may be different as compared to adults. At project design, youth beneficiaries should be consulted to determine types of medium best for transmitting information and confirm access to the appropriate technology. Engagement of Women • As highlighted in the Lessons Learnt report commissioned by RDO39, women’s participation in projects is often limited by societal constraints, including deep-rooted socio-cultural norms and practices that put women and girls in a much lower position relative to men and boys. Women are more likely to engage in unpaid care work, spending an average of 25.3 hours per week on unpaid care as compared to 13.5 hours for men. In addition, 84.7% of women in Rwanda depend on firewood as their mains source of cooking energy, for which women and children are primarily responsible for procuring.40 Many women are operating on a time deficit, leaving little time for engagement in external activities such as advocacy. Therefore, to increase engagement of women in advocacy projects, there first must be increase in self-esteem and resilience as well as awareness of equitable distribution of workloads through trainings for both men and women and social mobilization. Policy and advocacy training and educational materials should emphasize the role of women in agriculture and the significance of their experience in formulating advocacy messages. • Women are less likely to have access to resources, such as radios, and have lower literacy (69% for women compared to 78% for men)41 and ICT literacy rates (7% for women as compared to 11% for men of the same age)42 as compared to men and have less time available to engage with educational resources. Therefore, projects should include an analysis on how to best to reach women with project messaging recognizing that media or ICT may not be the most effective, rather inter-personal interactions through project identified agents of change, such a Community Committee members, may be required. Community Committees • The formation of Community Committees has created a local space for rural farmers, particularly women, to share their priorities for agricultural policy and has been key to the overall success of the project. While the project has put mechanisms in place to sustain these groups beyond the project end date by challenging Committees to discuss and develop plans, it is unclear the extent to which or in what capacity these groups will continue to function beyond the life of the project without continued support and feedback provided through the project. In order to maintain these groups as a key source of information on community level needs and priorities as well as improve evidence-based decision making by CSOs, the project should consider formally linking Community Committees to other local CSOs. In addition, the project should advocate for formal recognition of Community Committees and/or integration into the Village Agricultural Committee within the Twigire Muhinzi extension model. CSO Capacity and Engagement • CSO capacity building is necessary for the sustainability of project interventions, as CSOs have a wider geographic coverage beyond the village level and are able to identify issues that go beyond 37 MINAGRI (June 2018) Strategic Plan for Agriculture Transformation 20018-24 38 UNESCO Rwanda Country Statistics http://uis.unesco.org/en/country/rw 39RDO (April 2020) Lessons Learned and Best Practices Report 40 Republic of Rwanda (May 2019) Beijing +25 Rwanda Country Report 41 UNESCO Rwanda Country Statistics http://uis.unesco.org/en/country/rw 42 Republic of Rwanda (May 2019) Beijing +25 Rwanda Country Report 67 local cell or sector. CSOs also have greater access to higher level government institutions and forums, such as JADF, for their advocacy. However, in order for CSOs to be able to fulfill this role, they require further capacity development. This includes additional support in understanding relevant agricultural policies, including budgets, developing evidenced based advocacy messages and have the opportunity to engage with other CSOs for joint advocacy. CSOs should also be supported to initiate forums for advocacy and information exchange between citizens and government at district level to ensure that opportunities for advocacy remain beyond the life of the project. • A plan should be put in place to monitor the policy issues raised during the national policy dialogue that have not yet been addressed or that have only been partially addressed. The plan should include who will be responsible for monitoring and follow-up and to whom information should be communicated. As implementing partners will continue to engage with the beneficiaries in the project coverage area upon project completion, their participation in monitoring implementation should be encouraged. Local Government Engagement • Understanding of participatory governance was a key component of the success of the project. However, with turnover in local government, support and training for local government engagement with citizens will need to be on-going for sustained citizen participation in agricultural policy decision making. In addition, as there is a financial and logistical cost associated with bringing people together in formal exchange meetings, the project should advocate for time to be formally designated within existing meeting structures for government officials to engage with citizens on agricultural issues. Mobile Data Collection • Implementing partners noted that the training in the use of the smartphones and MDC software was insufficient to build their capacity to conduct data analysis on their own. Supporting implementing partners in data analysis should be integrated into monitoring visits to build capacity of partners to continue using and analyzing community-based data. In order to minimize loss of capacity due to staff turnover, multiple staff members should be trained and supported throughout the project. In addition, to foster collective advocacy amongst CSOs as well as develop the skills for effective use of evidence when developing advocacy messages, MDC findings should be shared broadly amongst CSOs and CSOs should be provided technical skills in simple data analysis. • Develop a handover plan for the use of MDC once the project ends to share with implementing partners, Community Committee members and other stakeholders. The handover plan should include the frequency of reporting, who will be responsible for receiving and analyzing the data and a plan for how the data will be used and disseminated. 7.2 Operational Recommendations • At project design, define indicators and how they will be measured to ensure feasibility and reliability of reporting against the proposed indicators. Consider rapid surveys and other forms of engagement with proposed project beneficiaries in order to ensure a community engagement at the very beginning and to inform decisions surrounding the best manner in reaching youth and women especially. • Use consistent terminology in communication pieces. The project used the term Village Agricultural Committee (VAC), a local official structure of the Twigire Muhinzi extension model, interchangeably with the Community Committee developed as a component of the project. • Develop a centralized database to track beneficiaries, record services provided and monitor the policy priorities and needs raised by community members, the level (cell, sector, district or national) where advocacy took place and the outcome of the advocacy. As noted in the Lessons Learnt document, farmers had influence at many levels of government, with each serving a different purpose and requiring a different way for farmers to have influence. The policy database would assist the project staff in tracking advocacy activities as well as analyze the types of 68 decisions made at each government level to support citizens and CSOs better plan for future advocacy activities. • As both SurveyCTO and CommCare are paid services, future projects should look to use free software for MDC such as KoBO Toolbox43. Use of free software will enable implementing partners to continue collecting data beyond the life of the project, without continued Trócaire support, as well as develop skills in low-cost MDC. • At project close, ensure that all relevant documents are uploaded to USAID Development Experience Clearing House 43 https://www.kobotoolbox.org/ KoBoToolbox is a suite of tools for field data collection for use in challenging environments. The software is free and open source. Most users are people working in humanitarian crises, as well as aid professionals and researchers working in developing countries. Kobo developers are based around the world monitor the functionality of the software and are available for trouble shooting 69 8. Annexes Annex 1: Evaluation TOR 70 71 72 73 74 75 76 Annex 2: Data Collection Tools Annex 2.1: Beneficiary Survey Section 1: Enumerator Identification 1.1 Form/Participant ID (note: this is assigned by the evaluation team) 1.2 Name of Enumerator 1.3 Date 1.4 District Gakenke Rulindo Nyanza Nyamagabe Nyarguru 1.5 Sector Janja Base Nyagisozi Gatare Kivu Mataba Cyungo Rwabicuma Kitabi Mata Mainazi Rukozo Uwinkingi Ruramba Section 2: Respondent Socio-Economic Profile 2.1 Category of respondent: Individual farmer Trader Processor Agro-dealer Other (specify) 2.2 Age 18-25 years 26-30 years 36-40 years 41-45 years 46-50 years Above 50 years 2.3 Gender Female Male 2.4 Disability None Physical Visual Hearing Other (specify) 2.5 Marital status Single Married Divorced Separated Widowed 2.6 Household size (if 1, respondent lives alone) 1 to 2 3 to 4 5 to 6 7 or more 2.7 Other income generating activities None Employed (specify) Self-employed (specify) Retired Other (specify) 2.8 Leadership positions held (currently or in the past) None Government Church Women Youth Cooperative Other (specify) Section 3: EPGA Project Awareness 3.1 Yes No 3.2 No Yes, I have seen them, but I have not read them Yes, I have read them 3.3 No Yes, I have seen them, but I have not read them Yes, I have read them 3.4 No Once or twice I listen sometimes I listen sometimes and I have called in or sent a WhatsApp message I listen regularly I listen regularly and I have called in or sent a WhatsApp message Are you aware of the EPGA (Enhancing Participatory Governance and Accountability) project? Have you seen posters in your community on “Itegurwa ry’ingengo y’imari y’Akarere, uburenganzira bwawe, uruhare rwawe mu iterambere ry’igihugu” (district budget planning, your right and civic responsibility in National development)? Have you ever seen or read posters in your community on “Ingengo y’imari yagenewe ubuhinzi izatuma dukomeza kwihaza mu biribwa” (does the current budget allocation for agriculture give confidence for food security)? Have you ever heard Uruhare Rwawe Muhinzi (“Your Role as a Farmer”) radio broadcasts on 105.9 FM or 107.5 FM? 77 Section 4: Beneficiary capacity 4.1 None Seed Policy Land use Policy Irrigation Policy Fertilizer Policy Girinka Policy Artificial insemination Policy Other (specify) Don't know 4.2 No Yes, I participated in discussions with the Village Agricultural Committee Other (specify) Yes, I participated in discussions at Cell level meetings Yes, I participated in discussions at Sector level meetings Yes, I participated in discussions at District level meetings Yes, I provided comments to my agricultural extension agents 4.3 I did not participate I am confident in my contribution I am not confident in my contribution I am very confident in my contribution 4.4 None Attended meetings where policies were discussed Other (specify) Attended meetings where I was able to comment on agricultural policies I have been trained to support implementation of policies 4.5 Not satisfied Satisfied Very satisfied 4.6 No Yes, I have collected information by visiting activities Other (specify) Yes, I have completed a monitoring report Yes, I have completed a scorecard Yes, I have participated in accountability meetings Yes, I have participated in community forums (like Umuganda, parents meetings, evening sessions) Yes, I have participated in performance contract implementation 4.7 Not satisfied Satisfied Very satisfied 4.8 No Yes-with the Village Agricultural Committee Other (specify) Yes-at the Cell level Yes-at the Sector level Yes-at the District level 4.9 I did not participate I am confident in my contribution I am not confident in my contribution I am very confident in my contribution 4.10 Not satisfied Satisfied Very satisfied 4.11 I do not have any priorities With the Village Agricultural Committee Other (specify) There are no platforms to share During local (cell, sector or district) meetings During visits by agricultural extension agents 4.12 Yes No I do not have any priorities/there are no platforms to share 4.13 Are you involved in monitoring how the budget allocated to the agricultural sector is being used? (tick all that apply) What agricultural policies are being implemented in your community? (Note for enumerator: Do not read the list, ask participant to list policies that they are aware of and tick off those mentioned) Did you participate in the development of or revision of any agricultural policies? (tick all that apply) How confident are you that your participation contributed to the development of or revision of agricultural policies? (NOTE: this is worded differently than at the midterm) What role(s) do you play in implementing these agricultural policies? (tick all that apply) What is your level of satisfaction with the role you play in implementing agricultural policies? Have you been involved in monitoring of the implementation of agricultural policies? (tick all that apply) What is your level of satisfaction with the role you play in monitoring the implementation of agricultural policies? How confident are you that your participation contributed to the monitoring of how the budget allocated to the agricultural sector is being used? (NOTE: this is worded differently than at the midterm) What is your level of satisfaction with the role you play in monitoring how the budget allocated to the agricultural sector is being used? (NOTE: this is worded differently than at the midterm) How do you express (share) your priorities for sustainable agricultural practices? Do you feel that local authorities respond to your expressed (shared) priorities? If respondent answered "Yes" in question 4.12 : Can you provide any examples of priorities that local authorities have responded to in the past 12 months? 78 Annex 2.2: Women and Youth Focus Group Discussion/Interview Guide With the change from in-person FGD Discussion to phone-based interviews, the guide was revised. Revision are noted in the document. Section 1: Enumerator Identification FGD ID # Name of Enumerator Date Start Time: End Time: District Gakenke Rulindo Nyanza Nyamagabe Nyaruguru Type of FGD Women Youth Number of Participants Female: Male: Section 2: Focus Group Discussion Guide # FGD Question Comment 2.1 Did you participate in any trainings or meetings organized by CEJP (Northern Province)/ UNICOOPAGI (Southern Province) as a part of the EPGA project? If so, how relevant were these activities in meeting your agricultural needs? (probes: Did you participate in the 2017 training on your roles and responsibilities for agricultural policies and budget monitoring? Were the activities sufficient or appropriate?) 2.2 Was the timing of the support/activities of the EPGA project appropriate? Removed for Interviews (probes: Did you receive advance notice of activities? Did activities take place as planned or communicated to you? Did activities take place at an appropriate time to encourage participation? Do you have any recommendations how to improve the timing of activities?) 2.3 Have you seen the posters/leaflets on “District budget planning, your right and civic responsibility” and “Does the current budget allocation for agriculture give confidence for food security” in your community? How did these change your understanding of your role and rights, if at all? (probes: Were these posted in visible locations? Was the information appropriate for your needs? Did you make any changes based upon reading these posters? Can you give examples of changes made?) 2.4 Have you heard any radio broadcasts on “Your role as a farmer” 105.9 or 107.5FM? How useful were these radio programs, if at all? (probes: Was the timing of the programs appropriate? Was the information appropriate for your needs? Did you make any changes based upon the broadcasts? Can you give examples of changes made? Did you call in or message during the broadcasts? Why do you think people did/did not listen to these broadcasts? What additional information should be covered by these radio programs?) 79 Section 2: Focus Group Discussion Guide # FGD Question Comment 2.5 How did the EPGA project (Trócaire, RDO, CEJP and/or UNCOOPAGI) seek your input in the design, implementation or monitoring of the project, if at all? Do you feel that you were able to contribute to the EPGA project in a meaningful way? (probes: how did you provide feedback to the project implementors￾Trócaire, RDO, CEJP and/or UNCOOPAGI-if at all? Did the project seek your input? If you had questions or concerns about the project, who could you ask? Do you feel that project staff were interested in your concerns? How can communication be improved in the future?) 2.6 How has your awareness of agricultural policies changed since the start of the EPGA project in 2017? (probes: What systems (such as local government meetings, Village Agricultural Committee meetings, visits from agricultural extension agents, etc.) are in place that encourage active participation in monitoring implementation? What discourages participation? ? Do feel that leaders and policy makers actively seek your input and support for monitoring policy implementation?) 2.7 How have you participated in the development or implementation of agricultural policies in your community? How has this changed since the start of the EPGA project? (probes: What systems (such as local government meetings, Village Agricultural Committee meetings, visits from agricultural extension agents, etc.) are in place that encourage participation? What discourages participation? Do feel that leaders and policy makers actively seek your input? ) 2.8 How have you participated in the monitoring of implementation of agricultural policies? How has this changed since the start of the EPGA project? Removed for Interview (probes: What systems (such as local government meetings, Village Agricultural Committee meetings, visits from agricultural extension agents, etc.) are in place that encourage active participation in monitoring implementation? What discourages participation? ? Do feel that leaders and policy makers actively seek your input and support for monitoring policy implementation?) 2.9 Do you feel that your participation in the development and/or implementation of agricultural policies is meaningful? What examples have you seen that your participation is valued, if at all? (probes: Have you seen any changes in policy implementation after giving feedback? To whom do you give feedback or share priorities? What systems exist that support your participation (ex: local government meetings, Village Agricultural Committee meetings, visits from agricultural extension agents, etc.)? 2.10 What opportunities exist for your participation in the development and spending of the agricultural budget, if any? How has this changed since the start of the EPGA project? 80 Section 2: Focus Group Discussion Guide # FGD Question Comment (probes: How are you involved in monitoring the allocation of the agricultural budget, if at all? Where do discussions of the agricultural budget take place? What discourages your participation? Do you feel that you have the skills and/or enough information to meaningfully contribute to discussions of budget allocation and monitoring of budget spending? Why/why not?) 2.11 Do you feel that your participation in the development and/or monitoring of the agricultural budget is meaningful? What examples have you seen that your participation is valued, if at all? (probes: Have you seen any changes in budget allocations after giving feedback? To whom do you give feedback or share concerns about the budget? What systems exist that support your participation (ex: local government meetings, Village Agricultural Committee meetings, visits from agricultural extension agents, etc.)? How can this be further improved?) 2.12 What are your agricultural priorities at this time? With whom will you engage to ensure that your priorities are taken into account by local and national leaders, if at all? What additional support, if any, do you need in order to do this? 81 Annex 2.3: Community Committee Member Discussion/Interview Guide With the change from in-person FGD Discussion to phone-based interviews, the guide was revised. Revision are noted in the document. Section 1: Enumerator Identification FGD ID # Name of Enumerator Date Start Time: End Time: District Gakenke Rulindo Nyanza Nyamagabe Nyaruguru Type of FGD VAC Number of Participants Female: Male: Section 2: Focus Group Discussion Guide # FGD Question Comments 2.1 What is the role of the Village Agricultural Committee in your community? Question removed for Interviews (probes: Was a VAC needed in your community? Why? What training did you receive? Was this sufficient? What additional support do you need? 2.2 Is your community aware of the role of the Village Agricultural Committee? How does the VAC engage with the community? (probes: Do you think that the community sees value in having a VAC? How does the community engage with the VAC—is this during formal meetings, informal discussions?) 2.3 What opportunities exist for the VAC to meaningfully participate in policy development, implementation and budget discussions? (probes: Is the VAC invited to sector, district or national level meeting? Can you give any examples of how your participation has changed policy, implementation of policy or budgets? What encourages meaningful participation? What discourages meaningful participation?) 2.4 How do you see the role of the VAC changing with the end of the project, if at all? What, if anything, has been done to sustain the role of the VAC in your community? (probes: Is there a mechanism to sustain the VAC? Do you think your access to meetings and leaders will change once the project ends? What has been done to ensure that the VAC will still be operational at the end of the project? Is this adequate?) 2.5 How did the EPGA project (Trócaire, RDO, CEJP and/or UNCOOPAGI) seek your input in the design, implementation or monitoring of the project, if at all? Do you feel that you were able to contribute to the EPGA project in a meaningful way? 82 (probes: how did you provide feedback to the project implementors-Trócaire, RDO, CEJP and/or UNCOOPAGI-if at all? Did the project seek your input? If you had questions or concerns about the project, who could you ask? Do you feel that project staff were interested in your concerns? How can communication be improved in the future?) 2.6 Were the support/activities and materials provided by the project (Trócaire, RDO, CEJP and/or UNCOOPAGI) sufficient and appropriate? (probes: What other project supported activities did you participate in? What types of support did you receive? How relevant was the support? Did you receive any materials from the project? If so, what types of materials? Did these materials meet your needs? If not, what would have been more useful?) 2.7 Was the timing of the support/activities of the EPGA project appropriate ? (probes: Did you receive advance notice of activities? Did activities take place as planned or communicated to you? Did activities take place at an appropriate time to encourage participation? Do you have any recommendations how to improve the timing of activities?) 2.8 How has the Survey CTO (smartphone) changed how you shared information with the EPGA project? (probes: How was information shared before? Who is in-charge of the smartphone? What types of information is shared? Who is this information shared with? How is this information used? How else has the phone been used (both for agricultural reporting and other uses? What have been the benefits of using the Survey CTO? Have there been any challenges with using Survey CTO?) 2.9 How will the Survey CTO/smartphone be used now that the EPGA project is ending? (probes: Who will be responsible for the phone? With whom will information be shared using the phone? Has this been discussed with you by CEJP or UNICOOPAGI?) 2.10 What are your communities’ agricultural priorities at this time? With whom will you engage to ensure that your priorities are taken into account by local and national leaders, if at all? What additional support, if any, do you need in order to do this? 83 Annex 2.4: CSO Member Discussion/Interview Guide With the change from in-person FGD Discussion to phone-based interviews, the guide was revised. Revision are noted in the document. Section 1: Enumerator Identification FGD ID # Name of Enumerator Date Start Time: End Time: District Gakenke Rulindo Nyanza Nyamagabe Nyaruguru List of CSOs Present: CSO Name # Female # Male Section 2: Focus Group Discussion Guide # FGD Question Comments 2.1 What support did you received from the EPGA project? 2.2 How useful was the training manual developed by the EPGA project entitled: “The role of citizens and local entities in policy and budget monitoring and advocacy related to agriculture”? (probes: How have you used this training manual? Is this manual still being used? Was the content sufficient to support trainings of citizens to engage in agricultural policy and budget monitoring? What changes have you seen in citizen capacity after conducting trainings using this manual? Was anything missing from the manual? If you have not used the manual, what prevented you from using it? 2.3 How useful was the Strategy Guide on Policy and Budget Monitoring and Advocacy in Agriculture for your organization? (probes: Did this guide help your CSO harmonize your activities related to agriculture policy and budget monitoring and advocacy? Did you read or use the guide? If not, what prevented you from using the guide?) 2.4 How did the EPGA project (Trócaire, RDO, CEJP and/or UNCOOPAGI) seek your input in the design, implementation or monitoring of the project, if at all? Do you feel that you were able to contribute to the EPGA project in a meaningful way? (probes: how did you provide feedback to the project implementors-Trócaire, RDO, CEJP and/or UNCOOPAGI-if at all? Did the project seek your input? If you had questions or concerns about the project, who could you ask? Do you 84 feel that project staff were interested in your concerns? How can communication be improved in the future?) 2.5 Were the support/activities and materials provided by the project (Trócaire, RDO, CEJP and/or UNCOOPAGI) sufficient and appropriate? Removed for (probes: What other project supported activities did you participate in? interviews What types of support did you receive? How relevant was the support? Did you receive any materials from the project? If so, what types of materials? Did these materials meet your needs? If not, what would have been more useful?) 2.6 Was the timing of the support/activities of the EPGA project appropriate? (probes: Did you receive advance notice of activities? Did activities take place as planned or communicated to you? Did activities take place at an appropriate time to encourage participation? Do you have any recommendations how to improve the timing of activities?) 2.7 What opportunities exist for CSOs to meaningfully participate in policy development, implementation and budget discussions? (probes: Are CSOs invited to sector, district or national level meetings? What encourages meaningful participation? What discourages meaningful participation? 2.8 Do you feel that your expressed priorities are responded to by local authorities? (Note: each CSO member should provide their individual response) (probes: Do you have any examples of which priorities have been responded to in the past 12 months? Were you satisfied with the response? Why, why not? What more needs to be done to ensure local authorities respond to CSO and local citizen priorities in the agricultural sector?) 2.9 How has support from the EPGA project changed your CSO’s role in supporting local citizens and advocacy for engagement in agricultural policy and budget development, implementation, monitoring? Which types of support from the EPGA project has been most useful? Which support has been least useful? Removed for interviews (probes: How useful were the trainings? Policy dialogue meetings? Exchange visits? Do people have a greater voice now than they did before the project started?) 2.10 What are your organizations’ agricultural priorities at this time? With whom will you engage to ensure that your priorities are taken into account by local and national leaders, if at all? What additional support, if any, do you need in order to do this? Removed for interviews 85 Annex 2.5: Implementing Partner Interview Guide Section 1: Interview Information Name of Implementing Partner Name of Interviewee Title of Interviewee Date of Interview Section 2: Project Design and Implementation Questions # Interview Question Comments 2.1 What was your organizations role in implementing the EPGA project? 2.2 Can you describe the impact of the project on the target communities (both intended and unintended)? 2.3 Do you think that rural people and farmers have a greater voice in agricultural policy decisions as a result of the project? How is that visible? How has government policy or strategy changed at the micro, meso or macro level for project participants? 2.4 Do you think the EPGA program design was appropriate? What changes, if any, would you recommend in order to increase citizen participation in agricultural policy and budget development, implementation and monitoring? 2.5 Were the planned resources adequate? What steps were taken to ensure that the project participants got the most out of the resources that were deployed? 2.6 Do you think project activities were delivered in a timely manner? Why/why not? What constraints existed that may have prevented timely delivery of project activities? 2.7 How effective was the use of technology in this project? Do you believe that this use of technology helped achieve the projects intended results? 2.8 Was the use of Survey CTO appropriate? How was the data collected used? Was the information shared with the participants? Was there any unintended use of the phones? 2.9 What are the project interventions that are likely to remain in the community after the project lifetime? What measures have been taken to ensure this? 2.10 How did Trócaire seek your input in the design, implementation or monitoring of the project, if at all? Do you feel that you were able to contribute to the EPGA project in a meaningful way? 2.11 How successful was the project in generating community and national ownership over the project? What could be done to improve community and national ownership? 2.12 What was your organizations role in monitoring the project implementation? 2.13 How appropriate and useful were the indicators described in the project results framework in assessing the project’s progress? Were the targeted indicator values realistic and could they be easily tracked? 86 Section 3: Project Management and Relations with Trócaire Questions # Interview Question Comments 3.1 Did Trócaire provide a conducive environment for partners to effectively implement the project? Please describe. 3.2 Did your organization have a clear understanding of your role in implementing the project? Please explain. 3.3 Do you feel that your organization received appropriate administrative support (including financial) from Trócaire? What additional support did you need? 3.4 Were project funds released in a timely way in order to implement activities appropriately? What challenges existed? 3.5 Do you feel that your organization received appropriate technical support from Trócaire? What additional technical support did you need? 3.6 How did Trócaire monitor project performance and results? Was this effective? 3.7 What recommendations do you have for improving coordination and management of the project by Trócaire? 3.8 Is there anything else that you would like to share about the EPGA project? 87 Annex 3: List of Interviews Trócaire Dony Mazingaizo Country Director 30 March 2020 Modeste Sibomana Program Manager 27 March 2020 Fred Ntakirutinka Project Coordinator Implementing Partners Eugene Rwibasira, John Mugabo RDO Executive Secretary, Project Officer 17 March 2020 Father Niragire Valens, Ruzindana Joseph CEJP General Secretary, Project Officer 13 March 2020 Eugene Murwanashyaka CEJP Field Officer-Gakenke 26 March 2020 Habyarimana Mathaias CEJP Field Officer-Nyanza 27 March 2020 Leo Bienvenue UNICOOPAGI Project Coordinator 26 March 2020 Emmanuel Hategekimana UNICOOPAGI Field Officer￾Nyamagabe/Nyaruguru 25 March 2020 Other Joseph Rurangwa USAID AOR 3 April 2020 Jean Claude Ngendandumwe CCOAIB Executive Secretary 2 April 2020 88 Annex 4: Survey Participant Characteristics Annex 4.1: Gender and Age Distribution Table 1: Gender distribution by age category and district District Adult (>30 Yrs) Youth (18-30 Yrs) Total Female Male Female Male n % n % n % n % Nyaruguru 43 55.1% 22 28.2% 10 12.8% 3 3.8% 78 Nyamagabe 46 49.5% 26 28.0% 17 18.3% 4 4.3% 93 Nyanza 49 50.0% 25 25.5% 14 14.3% 10 10.2% 98 Gakenke 26 44.8% 16 27.6% 12 20.7% 4 6.9% 58 Rulindo 25 71.4% 4 11.4% 5 14.3% 1 2.9% 35 Total 189 52.2% 93 25.7% 58 16% 22 6.1% 362 Table 2: Gender distribution by age category at baseline, midterm, and final evaluation Adult (>30 Yrs) Youth (18-30 Yrs) Female Male Female Male Baseline 60% 20% 21% Midterm 48% 32% 12% 8% Final 52.2% 25.7% 16% 6.1% Annex 4.2: Household Size Distribution Table 3: Household size distribution by district District 1-2 3-4 5-6 7 + Total n % n % n % n % Nyaruguru 3 3.8% 19 24.4% 28 35.9% 28 35.9% 78 Nyamagabe 6 6.5% 26 27.96% 34 36.6% 27 29.0% 93 Nyanza 8 8.2% 27 27.6% 43 43.9% 20 20.4% 98 Gakenke 9 15.5% 21 36.2% 19 32.8% 9 15.5% 58 Rulindo 2 5.7% 17 48.6% 11 31.4% 5 14.3% 35 Total 28 7.7% 110 30.4% 135 37.3% 89 24.6% 362 Annex 4.3: Engagement in Agricultural Sector Table 5: Engagement in Agricultural Sector by District District Individual Farmer Processor Trader Agro-dealer Other Nyaruguru 95.1% 0% 2.4% 1.2% 1.2% Nyamagabe 94.8% 1.0% 0% 0% 4.2% Nyanza 98.0% 0% 1.0% 0% 1.0% Gakenke 96.6% 0% 0% 0% 3.4% Rulindo 97.2% 0% 0% 0% 2.8% Total 96.2% 0% 0.8% 0% 2.4% 89 Table 6: Engagement in Agricultural Sector distribution at baseline, midterm and final evaluation Individual Farmer Processor Trader Agro-dealer Other Baseline n/a n/a n/a n/a n/a Midterm 97% 1% 0% 0% 1% Final 96.2% 0% 0.8% 0% 2.4% Annex 4.4: Engagement in Leadership Table 7: Engagement in Leadership Positions (past or present) District Church Cooperative Government Women Club Youth Club Other None F% M% F% M% F% M% F% M% F% M% F% M% F% M% Nyaruguru 15% 12% 15% 8% 43% 60% 21% 0% 2% 4% 2% 0% 34% 28% Nyamagabe 5% 7% 21% 13% 25% 53% 16% 0% 0% 0% 0% 3% 49% 30% Nyanza 13% 11% 11% 9% 27% 40% 22% 0% 5% 6% 0% 0% 43% 46% Gakenke 11% 25% 8% 0% 29% 35% 5% 0% 3% 0% 0% 0% 61% 40% Rulindo 0% 0% 7% 0% 33% 60% 7% 0% 0% 0% 0% 0% 57% 40% Total 9% 12% 13% 8% 31% 48% 16% 0% 2% 3% 0.4% 0.9% 47% 37% Annex 4.5: Survey Participant by Survey Type Due to the COVID-19 pandemic, the data collection methodology changed mid-way through data collection from in-person interviews to phone-based interviews. All attempts were made to ensure comparable data collection between both survey delivery mechanisms. As noted below, gender distribution by age was similar for both survey types. However, survey respondent household size was smaller with 44% of phone-based interview respondents reported household sizes of 4 or less, while 36% of in-person survey respondents reported household sizes of 4 or less. Table 8: Gender Distribution by Age and Survey Type District Adult (>30 Yrs) Youth (18-30 Yrs) Total Female Male Female Male n % n % n % n % In-Person 118 52% 61 27% 35 15% 14 6% 228 Phone 71 53% 32 24% 23 17% 8 6% 134 Table 9: Household Size by Survey Type District 1-2 3-4 5-6 7 + Total n % n % n % n % In-Person 15 7% 65 29% 87 38% 61 27% 228 Phone 13 10% 45 34% 48 36% 28 21% 134 90 Annex 5: Supplemental Quantitative Survey Data Tables Table 1: Indicator 2.A Percentage of target beneficiaries who are satisfied with their role in the implementation of agricultural related policies by District District Not satisfied Satisfied Very satisfied Gakenke 5% 76% 19% Nyamagabe 5% 60% 34% Nyanza 3% 70% 27% Nyaruguru 5% 44% 51% Rulindo 3% 74% 23% Total 4% 63% 32% Table 2: Reported Mechanisms for Participation in Implementation of Agricultural Policies by District District Attended meetings where policies discussed Attended meetings where able to comment Trained to support implementation No opportunities for participation Gakenke 47% 74% 43% 0% Nyamagabe 68% 59% 47% 3% Nyanza 73% 54% 48% 3% Nyaruguru 60% 59% 56% 4% Rulindo 63% 66% 40% 6% Total 64% 61% 48% 3% Table 3: Indicator 2.B Percentage of target beneficiaries who are satisfied with their role in monitoring of implementation of agricultural related policies by District District Not satisfied Satisfied Very satisfied Gakenke 5% 71% 24% Nyamagabe 6% 66% 28% Nyanza 3% 68% 29% Nyaruguru 5% 44% 51% Rulindo 6% 69% 26% Total 5% 63% 32% Table 4: Reported Mechanisms for Participation in Monitoring the Implementation of Agricultural Related Policies District Participated in community forums Participated in accountability mtgs. Collected information by visiting activities Part. in performance contracts Completed monitoring report Completed a scorecard Did not participate Gakenke 83% 48% 22% 9% 14% 10% 0% Nyamagabe 83% 45% 16% 11% 4% 4% 4% Nyanza 90% 44% 22% 19% 13% 10% 3% Nyaruguru 77% 50% 32% 21% 17% 12% 4% Rulindo 86% 66% 9% 17% 26% 23% 3% Total 84% 48% 22% 15% 13% 10% 3% 91 Table 5: Indicator 2.C Percentage of target beneficiaries who are satisfied with their role in monitoring of budgets on agricultural related policies by District District Not satisfied Satisfied Very satisfied Gakenke 12% 72% 16% Nyamagabe 25% 52% 24% Nyanza 18% 55% 27% Nyaruguru 9% 44% 47% Rulindo 14% 66% 20% Total 17% 56% 28% Table 6: Reported Mechanisms for Participation in Monitoring the Budgets of Agricultural Related Policies by District District Participated at Community Committee Participated at Cell level Participated at Sector level Participated at District level No opportunities for participation Gakenke 83% 45% 14% 0% 7% Nyamagabe 65% 22% 13% 3% 31% Nyanza 69% 28% 13% 3% 24% Nyaruguru 62% 28% 14% 1% 29% Rulindo 80% 37% 14% 0% 9% Total 70% 30% 14% 2% 23% Table 7: Indicator 1-1.A Percentage of target citizens who are confident of their contribution in the development of agricultural policy by District District Not confident in contribution Confident in contribution Did not participate Gakenke 0% 100% 0% Nyamagabe 3% 86% 11% Nyanza 5% 92% 3% Nyaruguru 6% 90% 4% Rulindo 0% 94% 6% Total 4% 91% 5% Table 8: Indicator 1-1.B Percentage of target citizens who are confident of their role in budget monitoring by District District Not confident in contribution Confident in contribution Did not participate Gakenke 16% 84% 0% Nyamagabe 10% 66% 25% Nyanza 4% 73% 22% Nyaruguru 13% 74% 13% Rulindo 3% 89% 9% Total 9% 75% 16% 92 Table 9: Indicator 3-1.A Percentage of target citizens who consider that local authorities respond to their expressed priorities with regard to sustainable agriculture by District District I do not have any priorities/there are no platforms to share No Yes Gakenke 10% 9% 81% Nyamagabe 3% 13% 84% Nyanza 5% 2% 93% Nyaruguru 0% 3% 97% Rulindo 6% 14% 80% Total 4% 7% 88% Table 10: Percent of Participants Reporting Priorities Responded to In the Past 12 Months by Topic by District Priorities Nyaruguru Nyamagabe Nyanza Gakenke Rulindo Total Fertilizers/ Soil Health 29% 31% 35% 31% 29% 31% Seeds 19% 28% 35% 16% 43% 27% Pesticides 17% 17% 14% 9% 6% 14% Information Sharing and Trainings 19% 11% 5% 10% 6% 10% Markets 9% 0% 3% 3% 0% 3% Water/Irrigation 1% 0% 3% 7% 3% 2% Animal Health/ Insemination 1% 1% 2% 5% 0% 2% Total 29% 31% 35% 31% 29% 31% 93 Annex 6: Project Results Framework Goal Objective Strategic Objective Outcome 1 Empowered citizens, particularly women and youths, who are able to engage in agricultural policy and budget monitoring in 5 districts and hold public authorities accountable Outcome 2 Increased capacity of civil society to engage in policy analysis, budget monitoring and advocacy and ability to use collective approach to influence decisions related to sustainable agriculture Outcome 3 Increased knowledge of local authorities on participatory governance and are committed to address citizens' concerns Output 1.1 Increased knowledge of 3675 citizens and 819 (900) local authorities on their role and responsibilities in terms of agriculture policy and budget monitoring related to agriculture. Output 2.1 Training manual developed for CSOs covering policy and budget monitoring and advocacy; Strategy guide in policy and budget monitoring and advocacy and recommendation papers about priority needs with regards to agriculture policies and programs are produced Output 3.1 Increased knowledge of local authorities on participatory governance and are committed to address citizens’ concerns Output 1.2 Increased awareness of 3675 citizens’ roles and responsibilities through production and dissemination of information and education materials Output 2.2 50 Civil Society Organizations in five Districts have sufficient capacity for policy analysis, budget monitoring and advocacy and are aware of participative approach for decision influencing Output 1.3 63 Community committees made up of five people are established and operational Output 2.3 Increased engagement on participatory governance and agricultural policy Output 1.4 Improved mechanism of exchange of information between community committees, partner programme staff and Trócaire Contribute to transparent governance, accountability of public authorities and inclusiveness of public policy and budgeting processes Contribute to the strengthening of civil society organizations (CSOs) for transparent governance, public leaders' accountability and inclusiveness in public policy development Promote the effective participation of citizens, especially women and youth, and CSOs in the development, implementation and monitoring of agricultural policy and budget in five districts in order to ensure that they respond to the priorities of vulnerable people and contribute to a sustainable and inclusive development in Rwanda 94