MID-TERM PERFORMANCE EVALUATION REPORT USAID/INDIA MILLENNIUM ALLIANCE PROGRAM May 2019 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Rodd Myers (Team Leader) and Leah Ghoston of Social Impact and Lina Sonne and Rajarshi Singh of Athena Infonomics for Social Impact, Inc. ii | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV MID-TERM PERFORMANCE EVALUATION REPORT USAID/INDIA MILLENNIUM ALLIANCE PROGRAM Submitted: May 31, 2019 USAID/India Contract No. AID-486-1-14-00001; Task Order No. AID-486-I-14-00001/72038619F00001 Cover Photo Credit: USAID/India DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV TABLE OF CONTENTS Abstract ............................................................................................................................................................................... vi Executive Summary ......................................................................................................................................................... vii Introduction ........................................................................................................................................................................ 1 Background Context......................................................................................................................................................... 3 Evaluation Design and Methodology ............................................................................................................................. 6 Responses to Evaluation Questions.............................................................................................................................. 9 EQ1: Innovation .................................................................................................................................................. 9 EQ2: Leveraging ................................................................................................................................................15 EQ3: Development Outcomes......................................................................................................................22 EQ4: Sustainability ............................................................................................................................................31 EQ5: Management ............................................................................................................................................36 EQ6: Gender......................................................................................................................................................48 EQ7: Knowledge Management and Communications..............................................................................51 EQ8: Program Design......................................................................................................................................55 Recommendations ...........................................................................................................................................................58 Annex 1: Statement of Work .......................................................................................................................................66 Annex 2: Data Collection Instruments and Consent..............................................................................................78 Annex 3: Return on Investment Method...................................................................................................................91 Annex 4: Data Analysis Variables................................................................................................................................. 95 Annex 5: Schedule of Work..........................................................................................................................................98 Annex 6: List of Key Informants and Awardees Interviewed ...............................................................................99 Annex 7: List of Documents Reviewed................................................................................................................... 102 Annex 8: Medium and Low Priority Recommendations..................................................................................... 104 Annex 9: Innovation Analysis Considerations....................................................................................................... 110 Annex 10: Evaluation Team Members and Responsibilities................................................................................ 116 Annex 11: List of FICCI Capacity Building Events................................................................................................ 121 Annex 12: Signed Disclosures of Conflict of Interest.......................................................................................... 123 iv | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV FIGURES, TABLES, AND BOXES Figure 1: Definitions of Innovation by MA Partners................................................................................................10 Figure 2: Numbers and Share of Awards by Stage of Innovation- Rounds 1 to 5 ...........................................12 Figure 3: Proportion of Total Funding Received by MA as of December 31, 2018 by Source (USD Million)................................................................................................................................................................................16 Figure 4: Details of MA Operational Expenses (Indirect Costs) Excluding Salaries (USD 2012-2018)......20 Figure 5: Number of Awards by Stage (1-3) and Round for Rounds 1-5...........................................................24 Figure 6: Value of Awards by Innovation Stage (1 to 3) and Rounds 1-5 (Million INR)................................. 24 Figure 7: Number of Awards Issued by Duration in Rounds 1-4 ........................................................................26 Figure 8: Number of MA Awards Granted by State in Round 1 to 5................................................................. 27 Figure 9: MA Disbursements by Round and Sector (Million INR).......................................................................40 Figure 10: MA Disbursements by Round and Sector (Number of Awardees).................................................40 Figure 11: Average Award Size by Round and Sector (Million INR)...................................................................41 Figure 12: Data Availability from Awardees Round 1-4 by Sector......................................................................45 Table 1: Summary of Evaluation Questions................................................................................................................. 1 Table 2: Summary of Interviews by Location and Sector......................................................................................... 7 Table 3: Types of Innovation Among MA Awardees..............................................................................................11 Table 4: Non-MA Financing Secured by Awardees Round 1 to 5 (USD)..........................................................17 Table 5: Illustrative Examples of Return on Investment.........................................................................................28 Table 6: Stages of Development and Award Caps ..................................................................................................41 Table 7: High-Priority Recommendations..................................................................................................................59 Box 1: WASH Success Story ........................................................................................................................................13 Box 2: Health Success Story .........................................................................................................................................18 Box 3: Agriculture Success Story................................................................................................................................. 19 Box 4: Clean Energy Success Story .............................................................................................................................23 Box 5: Education Success Story ...................................................................................................................................34 Box 6: Gender Success Story .......................................................................................................................................48 v | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ACRONYMS BoP Bottom of the Pyramid CDCS Country Development Cooperation Strategy CSR Corporate Social Responsibility DfID Department for International Development (UK) EQ Evaluation Question FICCI Federation of Indian Chambers of Commerce & Industry GoI Government of India ICCo Innovative Change Collaborative IIT Indian Institutes of Technology KII Key Informant Interview MA Millennium Alliance MoU Memorandum of Understanding NGO Non-governmental Organization RoI Return on Investment SAC Strategic Advisory Committee SI Social Impact, Inc. TTC Thinkthrough Consulting USAID United States Agency for International Development USD U.S. Dollars WASH Water, Sanitation, and Hygiene WISH Wadhwani Initiative for Sustainable Healthcare vi | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ABSTRACT In 2012, the United States Agency for International Development (USAID) Mission in India launched the Millennium Alliance Program to leverage Indian creativity, expertise, and resources to identify innovations, test promising solutions, and scale effective interventions and products. USAID/India contracted Social Impact, Inc. to conduct a performance evaluation to examine project effectiveness, identify the factors that promoted social innovation, and document best practices and lessons learned to inform the future of USAID’s involvement in the Millennium Alliance Program. USAID provided evaluation questions related to innovation, leveraging, development outcomes, sustainability, management, gender, knowledge management and communications and the design of the Program. The Evaluation Team used mixed methods including document review, secondary data analysis, semi-structured interviews and group interviews to identify findings that address the evaluation questions. This report outlines key findings that the Program has been effective at identifying and financing inclusive innovation in India that address the development needs of the bottom of the pyramid, however, there are several design and implementation challenges that impede efficiency (staffing limitations, monitoring and evaluation system weaknesses), effectiveness (development outcome, innovation, sustainability, and leveraging limitations), and equity (regional and gender). The report makes 35 recommendations (17 high priority) for Millennium Alliance partners to address in the short, medium and long-term. Overall, the evaluation concludes that there are many areas for improvement of Millennium Alliance, but that continued investment into the project, with FICCI as the managing partner, would likely be beneficial for developing the inclusive innovation space in India. vii | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV EXECUTIVE SUMMARY In 2012, the United States Agency for International Development (USAID) Mission in India launched the Millennium Alliance (MA) Program, its flagship innovation platform that leverages Indian creativity, expertise, and resources to identify game changing innovations, rigorously test promising solutions, and scale effective interventions and products. Since launching with three founding partners, this program has grown from direct financial contributions of $7.7 million to $19.9 million and to nine financial and in-kind contributing partners between 2012 and 2018. USAID/India contracted Social Impact, Inc. to conduct a performance evaluation to examine project effectiveness, identify the factors that promoted social innovation, and document best practices and lessons learned to inform the future of USAID’s involvement in the MA platform. The evaluation questions (EQs) examine innovation, leveraging, development outcomes, sustainability, management, gender, knowledge management and communications, and program design. The Evaluation Team disaggregated awardees by funding rounds and stage and randomly selected awardees from this list. The final sample included interviews with 20 stakeholders and 28 awardees, comprising 30% of all awardees across each sector in Rounds 1 to 4 with two additional awardees from Round 5. There were 14 awardees at the proof-of-concept stage, nine awardees at the scale-up within India stage, and five awardees at expanding globally. The sample also included awardees identified by Federation of Indian Chambers of Commerce & Industry (FICCI) as “failed” and denoted which awardees were women-owned businesses. The Evaluation Team spent six weeks in the field conducting semi-structured interviews with Millennium Alliance awardees and key stakeholders. MA has been important in raising the profile of inclusive innovation in India and crucial in funding early stage impact organizations and ventures. There is no one clear definition of what innovation means for MA, which makes assessing the extent to which MA captures innovation, or produces innovation impact, challenging. MA has supported a wide range of organizations under the remit of innovation. However, the extent to which they are innovative is subjective, so in some cases it is not clear how they differ from traditional development projects. Overall, awardees have been able to use awards for what they intended to, including test and pilot new solutions, start reaching markets or expand to new markets domestically or internationally. MA has successfully leveraged a pool of funding from the Government of India and the United Kingdom’s Department for International Development (DfID) in addition to USAID. Private sector funding contribution directly to MA is very limited; however, awardees have leveraged grant, debt and equity financing to finance their innovations. Furthermore, involvement of private sector players, especially impact investors and venture capitalists as proposed at the onset of MA has not been realized. MA and its awardees have experienced limited gains from FICCI’s vast member network. MA networking and partnership events have a finite scope and do not cater to targeted matchmaking required for awardee￾level leveraging. Gains in branding, prestige, credibility and market contacts have been leveraged by awardees but not tracked systematically. MA funding has enabled innovators to make incremental development impacts by improving access to health, water, sanitation, and hygiene (WASH), clean energy, agriculture and educational products and resources not previously guaranteed for many and has the potential to further improve the quality of life for targeted “bottom of the pyramid” populations. Development outcomes are affected by the amount of viii | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV their MA award and the time given to implement. While MA has awardees across India, most remain clustered around the main cities and the South and West of India together with the National Capital Region, and it has been challenging to support those who can design appropriate innovations and solutions for their regions in remote corners of India. Return on Investment calculations are challenging due to inconsistent data availability and reliability. In terms of sustainability, past awardees were still operational and continued to pursue the growth and development of the innovations that benefited from MA. However, granting to innovators in different stages of development, at different scales, in different sectors, and operating on both for profit and not￾for-profit models challenges , the nature of support that MA can provide to support the sustainability of awardees MA award amounts do not allow entrepreneurs to reach the full breadth and depth of impact needed for an innovation to be successful over a longer term. Overall, there is a lack of systematic follow￾up from MA on awardees after the completion of their award. An enquiry into the management of the MA yielded mixed results. Overall, there was a high regard to the effort that the small MA team put into ensuring MA runs smoothly. However, MA is seen broadly as a grant-making platform, with little support for awardees aside from financing. The application and award approval process is generally considered robust, but there are some issues that advantage some types of applicants over others. MA has consistently issued awards within its priority sectors, many of which overlap with USAID development outcomes. Several awardees reported receiving awards significantly smaller than their proposed amount, which can be partially attributed to either misalignments of self￾assessment or MA-assessment of stage of innovation. The terms and conditions under which awardees can apply capacity building funds remains unclear for many awardees and feedback mechanisms for the types of capacity building requested by awardees have not resulted in systematic changes to how capacity building activities are implemented. Monitoring and evaluation has been identified as a struggle for MA by its management and awardees. Progress towards improving monitoring and evaluation remains slow, and in many cases problematic, despite an increased focus over the last three years to establish regular reporting and common indicators. The Theory of Change remains elusive and by subcontracting monitoring and evaluation, FICCI has lost some opportunities for building this skill set internally. Overall, the monitoring and evaluation framework has not been documented well with the effect of unreliable data to the extent that impacts of the MA as a whole are difficult to assess. There is no evidence that MA has a gender inclusive focus with respect to the selection process and criteria, or the composition of the jury. There has not been a concerted effort to create a more gender balanced portfolio or to bring on board more women owners and managers. MA does not engage with awardees on gender inclusion in any meaningful way. Overall, the current design of MA does not have gender at its core, hence, gender equality is not systematically tracked or reported on. MA sponsored activities and events have been insufficient to promote consistent, ongoing knowledge sharing and communication among awardees and with key stakeholders both in and outside of MA. Awardees do not currently have a platform to systematically share lessons learned with other awardees about technical, management and other business processes. This lack of peer learning and connection in turn affects early-stage entrepreneur’s abilities to scale-up, within or outside of their state or even outside of India. There is also a lack of a regulated institutional knowledge base within MA to capture, respond to, and share feedback and lessons learned from awardees and also from its own learnings which prevents the wider ecosystem from generating valuable information about how best to support early stage ix | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV innovators through their entrepreneurial life cycle. While successes are easily lauded and promoted, failure is recognized as a reality by FICCI, but not systematically addressed. There is a missed opportunity for deeper engagement with the limited awardees in areas outside of heavily populated capital areas. The design of MA has been largely reactive and, while there is general support for FICCI as the lead implementer, the design is not without critiques. There are a number of systematic gaps in terms of linking FICCI members with awardees, but the MA partners, including USAID have played an important role in MA implementation, including overall guidance and the provision of technical expertise. Failure of awardees is difficult to measure in MA because there are very few awards that have been ceased due to performance issues. The report concludes with 35 recommendations, 17 of which are assigned high priority by the evaluation team. Recommendations relate to service provision, management and governance, focus of the MA, and monitoring and evaluation. Overall, the evaluation concludes that there are many areas for improvement of Millennium Alliance, but that continued investment into the project, with FICCI as the managing partner, would likely be beneficial for developing the inclusive innovation space in India. 1 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV INTRODUCTION In 2012, the United States Agency for International Development (USAID) Mission in India launched the Millennium Alliance (MA) Program, its flagship innovation platform that leverages Indian creativity, expertise, and resources to identify game changing innovations, rigorously test promising solutions, and scale effective interventions and products. Since launching with three founding partners, this program has grown from direct financial contributions of $7.7 million to $19.9 million and nine partners making direct financial or in-kind contributions between 2012 and 2018: Federation of Indian Chambers of Commerce & Industry (FICCI), USAID, Government of India (GoI), Innovative Change Collaborative (ICCo), ICICI Foundation, Department for International Development (UK) (DfID), Wadhwani Initiative for Sustainable Healthcare (WISH), World Bank and Facebook.1 USAID/India contracted Social Impact, Inc. (SI) to conduct a performance evaluation to examine project effectiveness, identify the factors that promoted social innovation, and document best practices and lessons learned to inform the future of USAID’s involvement in the MA platform. The evaluation questions (EQs) examine the major areas in Table 1, below (a full list of the EQs and sub-questions is included in the evaluation Statement of Work in Annex 1): Table 1: Summary of Evaluation Questions EQ 1 INNOVATION: To what extent has the MA program succeeded in identifying and selecting innovative solutions for implementation that will support USAID’s technical sectors and corresponding activities? EQ 2 LEVERAGING: To what extent has the MA project been able to leverage funding and non-financial value services from various sources like private sector, government, and other bilateral and multilateral donor agencies for additional development outcomes? EQ 3 DEVELOPMENT OUTCOMES: To what extent have the innovative solutions made an incremental contribution to USAID’s development results? EQ 4 SUSTAINABILITY: To what extent have the innovative solutions become sustainable? EQ 5 MANAGEMENT: To what extent has the MA platform been effective in terms of management of awards and capacity building of the awardees in technical, managerial, business administration, and sustainability issues? EQ 6 GENDER: To what extent did the MA platform consider gender issues in the awards in each sector? EQ 7 KNOWLEDGE MANAGEMENT AND COMMUNICATION: To what extent has the MA project succeeded in serving as a knowledge platform for the innovations supported and established with the wider Indian social innovation and development ecosystem? EQ 8 USAID/INDIA’S DESIGN OF THE MA PLATFORM: Is a Chamber of Commerce an appropriate mechanism for what USAID is trying to achieve via the MA platform? 1 Facebook and World Bank are non-financial partners 2 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV SI identified an experienced four-person Evaluation Team including a Team Leader, two Evaluation Specialists, and a Sr. Public-Private Partnership/Innovation Specialist supported by a Technical Advisory Panel of part-time sector experts and a Financial Analyst. SI also partnered with Athena Infonomics, a firm that brings expertise in public-private partnerships in India, as well as the five technical areas of the MA platform. The Evaluation Team used a mixed-methods approach that combines both qualitative and quantitative data collection gathered through document and secondary data reviews and key informant interviews. Additionally, the team incorporated a sector-specific analysis created by technical advisory panel members, conducted a participatory innovation workshop to gauge strengths, weaknesses and opportunities in the innovation ecosystem (see Annex 2), and utilized the Financial Analyst to perform a return on investment (RoI) analysis. 3 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV BACKGROUND CONTEXT USAID defines innovation as “novel business or organizational models, operational or production processes, or products or services that lead to substantial improvements (not incremental “next steps”) in addressing development challenges…[it] may incorporate science and technology but is often broader to include new processes or business models.”2 In 2010, the “Decade of Innovation” was declared by the GoI, which set up the National Innovation Council along with State and Sectoral Innovation Councils to discuss, analyze and help implement strategies for inclusive innovation in India, and to prepare a Roadmap for Innovation 2010-2020. USAID/India has established itself as an in-country branch of the USAID Development Lab, set up to produce “innovations by sourcing, testing, and scaling proven solutions.”3 In 2012, USAID/India launched its flagship innovation initiative, the MA Program, to put to work Indian creativity, expertise, and resources to find and support creative solutions to some of India’s and the world’s most significant development issues. In collaboration with the GoI’s Technology Development Board and FICCI, USAID has developed a public-private partnership platform to provide seed money, human resources, and technical assistance to Indian organizations and individuals with innovative ideas, technologies, and business models in the inclusive innovation space. USAID/India’s Country Development Cooperation Strategy (CDCS) envisions a combination of innovative ideas, capital investments and strengthened, sustained local partnerships as a means of achieving development results. INCLUSIVE INNOVATION IN INDIA While this report uses the term “inclusive innovation,” there are many related and overlapping definitions that emphasize innovation with a focus on development outcomes and low-income consumers. These include social innovation, market-based solutions to poverty reduction, social entrepreneurship, impact ventures, products and services for the bottom of the pyramid (BoP).4 Inclusive innovation in India operates within the context of social change, and broadly encapsulates the idea of creating access to products and services for the underserved, or livelihood opportunities as producers, primarily from the low-income segment at the bottom of the pyramid. Innovation is taken to be something new in the local context, rather than “new to the world.” This understanding of inclusive innovation goes beyond technologies to the need to create processes, create new user habits, and deal with complex last mile (delivery or distributor) challenges. Dynamic inclusive innovation comes about through a mixture of invention, innovation, and entrepreneurship. Organizations must be entrepreneurial to facilitate access to and for new users and markets, while also building new types of business models that generate sustainable solutions for low-income populations. Inclusive innovation has a long history in India developed from grassroots innovation and indigenous knowledge movements and emerges from a range of sources and stakeholders. These range from inventors’ tinkering, to social entrepreneurs (often urban and well educated) and non-governmental 2 https://web.archive.org/web/20190402055036/https://www.usaid.gov/fallsemester/usaid-101/innovation-development 3 Request for Task Order Proposal, Millennium Alliance Performance Evaluation 4 BoP is a proximate term often used in India to describe the population living under the poverty line. Globally, it is sometimes used for people living under $2.50 per day (https://web.archive.org/web/20190319010803/http://lexicon.ft.com/Term?term=bottom-of-the-pyramid-(bop) 4 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV organizations (NGOs) experimenting with new ways of working with communities, to corporates and startups that focus on low-cost products. At the same time, India’s strong domestic social innovation foundation operates alongside a mix of foreign organizations operating in the country. The ecosystem supporting and enabling inclusive innovation in India has evolved rapidly in the past decade, from a handful of incubators and impact investment funds, to a large number of entities offering specialized financial and non-financial support to organizations. These entities may focus their support on specific sectors, issue areas, organizational types, or venture growth stages. MA, therefore, operates within an ecosystem that is continuously changing. At the same time, the GoI has redirected its development policy towards supporting inclusive innovation and entrepreneurship. The GoI has sharply reduced the inflow of foreign aid into the country, and overall, foreign donors and multilateral agencies have changed their strategies from funding traditional aid programs to supporting inclusive innovation and market-based solutions to poverty. It is in this context that MA serves as a pioneer program in which a partnership between foreign donors and the GoI seeks to spur development through inclusive innovation. The inclusive innovation ecosystem will be discussed further in the Responses to Evaluation Questions and in Annex 9. MILLENNIUM ALLIANCE PROGRAM OVERVIEW MA is comprised of three components that enable FICCI, the Executive Secretariat of the platform, to enable MA to form partnerships and programs that identify, support, and scale innovations to achieve concrete development outcomes for the BoP population in India and globally. These three components are: (1) Seeding and scaling innovations to achieve concrete development outcomes; (2) Platform and partnership development; and (3) Knowledge management and communications. MA provides multi-tier funding via calls for proposals referred to as “Rounds.” By the conclusion of evaluation fieldwork, MA had awarded Rounds 1-5, and the application window had closed for Round 6. MA funds innovations across different stages of development: ● Stage 1: Proof of concept ● Stage 2: Demonstrated success at pilot level or small-scale and scaling up ● Stage 3: Credible and rigorous evidence of development impacts at significant scale in India expanding globally The size and duration of the awards vary, and are discussed in greater depth in response to the Management evaluation question. In the past five years, MA supported 115 innovators and 124 innovations at all stages of innovation across five development sectors: health; education; agriculture; clean energy; water, sanitation, hygiene (WASH); and “other.” Since its inception, the MA platform has leveraged additional resources from DfID, ICCo, World Bank, and Facebook. DfID participates in the transfer of resources to new countries, and World Bank provides intermittent technical assistance. 5 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV REPORT OBJECTIVES This evaluation assesses the level of effectiveness of the MA platform to support, nurture, and learn from innovations in education, clean energy, WASH, health, and agriculture to positively affect the lives of India’s BoP populations. The evaluation seeks to inform USAID/India, FICCI, the Government of India and its counterparts invested in MA of project effectiveness, identify the factors that facilitated or hindered change, and document feedback from both awardees and key stakeholders to provide recommendations to inform the future of USAID’s involvement. This evaluation also considers how key lessons learned can be applied to future innovation grantmaking mechanisms in USAID/India’s key priority sectors, as well as opportunities for capacity building of existing awardees and the potential for innovations in India and beyond to be scaled-up to address development challenges. The evaluation report is intended for the use of USAID and its Indian implementing partner, FICCI, for the purpose of informing any follow-up implementation plans, and other Missions that may wish to replicate the model. 6 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV EVALUATION DESIGN AND METHODOLOGY The Evaluation Team used a mix of methods to understand and assess MA, including a document review, secondary data review, and key informant interviews (KIIs). Beginning in January 2019, the Evaluation Team used secondary data (listed in Annex 7) from MA reports and literature pertaining to innovation in India to familiarize itself with the context. From this review, the Evaluation Team developed data collection instruments (see Annex 2), which were designed to address the Statement of Work. Prior to field work, the Evaluation Team held an in-briefing with USAID on Feb 4, 2019 to review and prioritize evaluation questions, determine key geographic locations for data collection, and review data collection methods and sampling strategy to make necessary adjustments with USAID input. Following the debrief, the Evaluation Team piloted all final data collection tools with one stakeholder and one awardee in Delhi. The Evaluation Team conducted field work from February 7 to March 14, 2019. Annex 5 details the schedule of work. Semi-structured interviews were administered with awardees and key informants, which took about 90 minutes to complete. The Evaluation Team held a workshop on March 1, 2019 with key informants to help to contextualize MA within the greater inclusive innovation ecosystem. Documents studied in the desk review also served as important sources of quantitative data to assess costs and determine the outcomes of the various awardees. A further sector-specific study was conducted by the Evaluation Team’s Technical Advisory Panel to identify how innovation is contextualized within the key development sectors reflected in MA. Additionally, following field data collection, the Evaluation Team conducted an innovation workshop with several Delhi-based stakeholders to collect feedback on the past, present, and future of engagement with the innovation ecosystem. Information captured from both the Technical Panel and the workshop will inform the context of the Evaluation Team’s recommendations in this report. SAMPLING The Evaluation Team, in consultation with USAID, developed a sampling frame from which respondents from all sectors, key regions of India, MA call rounds, and stages of business development were interviewed. Social Impact purposively selected key informants from USAID, FICCI, and MA partner organizations to ensure that KIIs were conducted with individuals who have the deepest knowledge of project activities. The Evaluation Team selected key informants from a listing of MA stakeholders provided by USAID and FICCI, who were met both in person and over the phone. MA supported 88 innovations during its first four rounds of awards, which spanned the key sectors of education, clean energy, health, WASH, agriculture and “other.” Social Impact performed a multi-stage sample selection, first identifying awardee locations, which informed travel sites for data collection. Interviews were conducted in person in Delhi, Bangalore, Pune and Mumbai, where MA awardee headquarters appeared to be clustered, as well as in Kolkata and Bhubaneswar. The Evaluation Team conducted remote interviews with awardees located and/or operating in the North Eastern states of Assam, Sikkim and Bihar, as well as with awardees located outside of India in Nepal, Bangladesh, and Afghanistan. The Evaluation Team identified which sectors were represented in the overall MA awardee portfolio at all locations, and within each sector, disaggregated awards by rounds and stage.5 Awardees were selected randomly from this list; the final sample included 28 total awardees, or 32% of all awardees across each sector in Rounds 1 to 4, with two additional awardees from Round 5. There were 14 awardees at Stage 1, nine awardees at Stage 2, and five awardees at Stage 3. The Evaluation Team’s sample also 5 In the context of this report, “stage” refers to the level of maturity of the innovation. Stage 1 is proof of concept; Stage 2 is domestic scale-up and Stage 3 is International Scale-up. 7 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV included awardees identified by FICCI as “failed” and denoted which awardees were women-owned businesses. While the initial sample included up to nine awardees denoted as women-owned, it was determined that the list of awardees to sample did not accurately depict women-owned businesses; applying more accurate criteria after data collection revealed that of those sampled, one awardee was indeed women-owned (see Limitations). In total, 48 interviews were conducted with 28 awardees (23 male and 7 female)6 and 20 interviews with key stakeholders (10 male and 10 female) (see Annex 6 for the full list of awardee and key informant consultations made). Table 2, below shows the summary of interviews by location of interview and sector of activity. Table 2: Summary of Interviews by Location and Sector LOCATION HEALTH AGRICULTURE WASH CLEAN ENERGY EDUCATION INTERVIEWS WITH STAKEHOLDERS Delhi 3 2 2 1 16 Mumbai 1 2 1 1 1 Bangalore 5 1 1 1 Kolkata 1 2 Bhubaneswar 1 Remote 2* 3* 1* TOTAL 8 9 5 3 3 20 * Remote interviews were conducted with awardees based in the Indian states of Assam, Telangana and Bihar and in the countries of Afghanistan, Nepal and Bangladesh. DATA ANALYSIS The Evaluation Team conducted a systematic analysis of qualitative and quantitative data. During fieldwork, the Evaluation Team took key informant interview notes in real-time, cleaning and synthesizing electronic notes and summaries after site visits, and then used content and comparative analyses to identify patterns and elucidate themes and contextual factors from qualitative data. The Evaluation Team used these themes, along with the evaluation questions, to generate a coding structure to systematically code all qualitative notes using qualitative analysis software Dedoose,7 organizing thematic findings by evaluation question and sub-question. All qualitative data was disaggregated by stakeholder type and sex, awardee sector, round, stage, and location to capture any divergences between male and female respondents and geographic perspectives. The Evaluation Team analyzed programmatic quantitative data from MA’s monitoring and reporting database using Microsoft Excel and Google Sheets to produce summary statistics and comparative tabulations, as well as financial data to generate RoI, a detailed methodology for which is shown in Annex 3. LIMITATIONS ● Because the Evaluation Team used both purposeful and random sampling for respondents, and has a relatively small sample of clean energy, education and agriculture awardees, the overall qualitative sample cannot be considered statistically representative of all awardees. The Evaluation 6 The total calculated number of male and female awardee respondents exceed the total number of interviews due to two interviews with more than one respondent. 7 http://dedoose.com 8 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Team mitigated this limitation by analyzing data received from FICCI and USAID from across all awardees, not only those in the sample for quantitative analysis. ● The Evaluation Team sampled a list of 23 women-owned businesses that was provided by FICCI; however, using the Evaluation Team’s definition of women-owned, it was determined that sampled awardees did not fit these criteria (discussed more in the response to evaluation question 6). This means that any disaggregation by status of women-owned is limited to secondary data in this evaluation. ● The methods only included successful awardees, some of whom were rejected in previous or subsequent rounds; the evaluation did not interview applicants who did not receive an award. The perspectives of applicants who were not granted awards could have added another perspective to the analysis, which was still somewhat captured by successful applicants who had been rejected in prior or subsequent rounds. ● There was a higher proportion of MA awardees at the Stage 1 piloting stage who did not yet market their product, which prohibited the Evaluation Team from conducting group interviews with beneficiaries as planned (discussed more in the Development Outcomes evaluation question). Additionally, where awardees did have beneficiaries, the Evaluation Team was unable to travel to these locations in the time allotted, as they were outside of the city or state in the sample area. This would have provided more direct insight on utility of innovations. ● The Evaluation Team also experienced some limitations in terms of the availability of or quality of data, which limited the depth of analysis that could be completed with confidence. These data were mostly related to development outcomes, failed projects, and finances, and the implications of this limitation are listed throughout this report, with the overall effect of limiting the extent to which, or the way in which, the Evaluation Team could directly address the evaluation questions. For example, due to unavailable or unreliable impact data, the Evaluation Team could not adequately address questions around whether some types of innovation were more impactful than others. ● Due to the composition and expertise of the Evaluation Team, the two pairs of evaluators split into teams of two women and two men. This structure could have influenced interview responses if respondents were uncomfortable speaking with evaluators of the opposite sex. 9 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV RESPONSES TO EVALUATION QUESTIONS In this section, the core findings, conclusions, and recommendations are presented for each evaluation question in the order presented in the Statement of Work (see Annex 1). Each evaluation question has a number of sub-questions (see Annex 1) which are not reproduced here to make this report more accessible to the reader. EQ1: INNOVATION EQ1 To what extent has the MA program succeeded in identifying and selecting innovative solutions for implementation that will support USAID’s technical sectors and corresponding activities? FINDINGS There was broad consensus among respondents that MA has been important in raising the profile of inclusive innovation in India. This was especially true in its early years of operations, when there was next to no alternative early-stage grant funding and no major platforms that promoted inclusive innovation in the country (other than the Honeybee Network for grassroots innovation). Several respondents attested to the multi-stakeholder platform calling attention to inclusive innovation as an alternative means to alleviate poverty and spur development. One stakeholder working across conventional start-ups and inclusive innovation noted that, “MA was influential at the beginning - it was a very good idea and had the potential to bring need-based innovation out of the woodwork to leverage the domestic market. That was its real power potential.” The MA platform continues to be a unique and crucial source of early-stage grant finance, as noted by nearly all awardees interviewed. One key informant suggested, “The role that MA plays of money with no equity is amazing. It is an important and great ecosystem need that must continue.” An awardee stated that “there [are] very few programs aimed at providing grants to entrepreneurs at the early stage - it is like getting money for experimenting.” While MA internally has classified awardees into three stages, the first two, and many at the third stage still fall under early stage entrepreneurs and early stage finance in the wider ecosystem (e.g. they are pre impact venture capital). Despite the recognized importance of MA in raising the profile of inclusive innovation, and filling a large gap in early stage impact finance by offering grants, more than half of the stakeholder and partner respondents indicated that MA has become relatively less visible and less prominent in the ecosystem today. According to one key informant, “People are not aware of MA. Today MA happens in isolation.” According to respondents outside MA, it has not been active in recent years at the major inclusive innovation and social enterprise convenings, which are major focal points for the ecosystem and ecosystem strengthening activities. Additionally, according to several respondents, the support ecosystem has evolved rapidly, and is now increasingly crowded. In 2013, when MA started, there were few early stage alternatives for funding other than incubators such as UnLtd India and Villgro (primarily incubation with small grant) and impact investors including Acumen and Aavishkaar (who then invested at a relatively nascent stage), as well as one primary yearly conference for networking. Today, however, there are several domestic and international impact funders, a large number of incubators and accelerator programs, and multiple networks of ecosystem stakeholders (e.g. India Impact Investment Council), as well as multiple conferences. A key informant explained, “In the early days there weren’t too many ecosystem players looking at early stage innovators, so MA played a significant role in that. In the last few years there have 10 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV been other players.” Furthermore, while the MA has remained the only large-scale grant-funding program of its kind and size, the ecosystem has increasingly specialized along sectors and stages of venture development. Today a range of financial instruments and non-financial support are available, as more and more Indian and international support organizations have emerged. For more detail on the innovation landscape in India, see Annex 9. It is difficult to pinpoint what MA means by innovation. There is no singular, clear definition of innovation among the partners, as different partners use different definitions. Furthermore, FICCI’s definition is in stark contrast to the level of innovation of many of the awardees in its portfolio. Figure 1, below highlights some of the differences among definitions of MA partners. Figure 1: Definitions of Innovation by MA Partners Partners and awardees offer a range of understandings of innovation, but broadly focus on accessibility to new products and services for low-income populations. The emphasis is on innovation affecting social change, as noted by a key informant: “We are looking at innovation in the context of social change- we define it differently in India. In India, the innovation is about access- how to do you get life changing solutions to those who can’t normally access it? Trying to get a solution working for the people who aren’t normally the beneficiaries.” Key elements of innovation highlighted by awardees and partners include: ● Technology driven, generated through research and development, and solution-oriented; ● Innovations that fill gaps and address the needs of beneficiaries: how appropriately services and products are delivered to the hard to reach markets; ● Reverse engineering: taking something existing and changing the color or shape. Otherwise, reducing its cost; 11 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ● “Lego block” innovation, where different aspects are pulled together, or a modular approach where certain aspects are pulled from the existing solution to make it more accessible and user friendly; ● Merging the business side with the product/engineering side: how to take it to society and get commercial value is a major part of the innovation. The selection process includes innovation among multiple criteria but does not focus on innovation above other criteria. For example, at the first applicant evaluation step, there is a yes / no response required to “Is the innovation substantially incremental or novel?” In the second applicant evaluation step, innovation counts for 22% of the overall score. In its evaluation guidelines, MA defines innovation in the following manner in its guide for evaluators: “MA takes a broad approach to the idea of innovation. Apart from considering new technological advances, we also look at process related innovations that improve upon existing approaches to significantly enhance the quality of a particular product or service.” There are no further instructions on how to evaluate innovation at any of the selection stages. To the public, applicants are only told that MA takes a “broad approach to defining innovation.”8 There are different facets of innovation. Awardees address innovation in different ways. Table 3 below summarizes the types of innovation in MA and provides some examples of each. Table 3: Types of Innovation Among MA Awardees TYPE OF INNOVATION EXAMPLES IN MA Developing, testing or rolling out a never￾before-seen technology (or new to India) ● Heart disease detection technology ● Grain storage ● Biodegradable feminine hygiene pad Successful model or product in a different location ● Micro grids ● Solar powered water filters and pumps ● Pregnancy health mobile technology Adapting existing technologies to suit local needs ● Water collection and distribution New business models ● Finding ways to remain sustainable while working with low-income producers and consumer ● New networks or creating new last mile distribution and collection models for water storage Although the Evaluation Team was unable to decipher whether some types of innovation had more impact than others due to the lack of availability of reliable data, the issue of Development Outcomes is analyzed later in this report. This analysis suggests that Stage 1 awards are developing proof-of-concept and are therefore less likely to have development impact than Stage 2 and 3 awards. MA supports different stages of enterprise or solutions development that are related to making an innovation successful - from prototyping, testing and piloting to rolling out and replicating in India and expanding internationally. Figure 2 shows that from Rounds 1 to 5, 44% of awards were issued for Stage 8 https://web.archive.org/web/20190402072406/http://www.millenniumalliance.in/faqs.aspx 12 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 1 innovations, 37% for Stage 2, and 19% for Stage 3. This is different from, for example, an innovation challenge grant that would focus on a particular invention-to-market stage in the innovation cycle. Figure 2: Numbers and Share of Awards by Stage of Innovation- Rounds 1 to 5 The MA platform has enabled awardees to scale up in India and abroad. Awardees in more advanced stages of innovation (Stage 2 and 3) applied for MA funds with a proven model and with the intent to scale up - either to additional locations or populations in India, or to global markets. Several awardees scaled-up within India successfully, whether expanding into neighboring communities or neighboring states. MA’s direct support primarily entailed funding awards. However, some awardees reported receiving help with linkages as well as access to mentors. Awardees further reported that informally, MA and the USAID connection helped improve the brand value, which meant new potential collaborators took them more seriously. However, there are certain considerations for working in resource-poor environments of the North East, and awardees cited logistics, human resources, language challenges, lack of exposure and limited networks that affect operations and constrained innovation. As one agriculture awardee noted, “Finding the diamond-in-the-rough entrepreneurs who are out in rural areas who can really scale is hard.” Awardee expansion into Uganda, Rwanda, Kenya and Ethiopia, in particular, has been notable and encouraging according to both awardees and stakeholders. Apart from funding, the connections that the MA team has helped to broker through FICCI partnerships in Africa such as the India Rwanda Innovation Growth Program9 have been very helpful in opening doors and entering new markets, for example. Likewise, FICCI and partners, including DfID, have helped establish connections with new partners in African countries. One partner responded, “It is great to see low-cost innovation in India that has a potential to go outside of India.” Awardees interviewed also scaled-up to Nepal, Bangladesh, and Afghanistan. 9 http://www.irigp.org/ 13 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV However, there are challenges with global expansion that come with a need for networks, partnerships and cultural competence. One awardee stated, “For a non-Indian country, we have to wait for someone to approach us from that country first so we know the demand is there...can we solve a problem there? Is it needed there? Then we know what our added value is.” Another noted that for awardees looking to scale globally, there are “unique challenges” that come with understanding the political and regulatory environment which necessitate strong partnership with other countries’ governments and embassies. Respondents see these kinds of partnerships as critical for global scale-up; several MA awardees used their funds for outreach activities, scoping, and partnership development. While most of the Stage 2 and 3 respondents confirmed that the MA funding allowed them to scale up, others commented that MA's tendency to partially fund proposals stymied innovation. The Evaluation Team found several examples of projects that were only partially funded (i.e. the awarded amount was less than the amount requested), resulting in the elimination or severe truncation of the most innovative part of programming, which limited the awardees ability to scale up entirely as planned or to make significant improvements to product development. As one awardee remarked, “essentially what made the proposal innovative had to be left out.” A full chart of the grant asks and awards is discussed in more detail under the Management evaluation question. It is unclear what constitutes “failed” awardees on the MA platform. FICCI was unable to provide a consistent definition of what “failure” was in the context and life cycle of the award timeline. The criteria provided to the Evaluation Team singled out awardees who faced challenges leveraging funds, acquiring certification for development or commercialization of their product or producing an affordable product. However, the Evaluation Team determined that “failure” is subjective under MA. While five awardees were identified as failures by FICCI, there were several additional awardees in the sample that faced similar challenges but were not branded as failures. Furthermore, it was unclear to the Evaluation Team after speaking with awardees that those classified as failures by FICCI were in fact failures, namely because several did have successful product development. For further discussion on failure, see the responses to Knowledge Management and Communications and Management evaluation questions. After identifying issues with contaminated drinking water and water borne disease, Round 2 WASH awardee Swayamsiddha Mahila Utkarsha Foundation focused on water purification at the local community level in Maharastra. Their approach, “Jaldoot,” used reverse osmosis or ultrafiltration technology to remove harmful chemicals and heavy metals from local water sources. They realized that remote communities were not receiving equitable access and suffered from adverse health effects at often higher rates. With MA funding, Swayamsiddha adapted the technology to the local context, and scaled up the successful Jaldoot model based on the demand from other villages. It takes water from the village source to a distribution center run by local women, purifying it with their established stationary water filtration system and utilizing their vehicle mounted technology to take the clean water to remote communities. While the grant ended in 2016, staff noted, “MA funding allowed us to upscale 30-40%;” they started with two vehicles and now have 20 and have scaled up to additional schools and villages in the most remote areas, covering almost 9000 students and over 1200 households. Box 1: WASH Success Story 14 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV CONCLUSIONS At the ecosystem level, MA has been important in raising the profile of inclusive innovation in India and crucial in funding early stage impact organizations and ventures. However, as the innovation ecosystem has been maturing, the centrality of the MA platform has been reduced. The Evaluation Team did not focus specifically on USAID connections, especially considering the limited sample interviewed. However, there are several overlaps between awardees of the MA platform and other USAID sponsored programs and initiatives enabling innovation. There is no single, clear definition of what innovation means for MA among partners, as each partner uses different language. The varied portfolio of different levels of innovation, including initiatives where notions of innovation are elusive, is likely an outcome of this lack of coherent understanding of what constitutes innovation and what the best strategy for seeking out, selecting, and supporting such awardees should be. Likewise, based on the available information on the portfolio, including existing monitoring and evaluation, it is not possible to provide a definitive response on what kinds of awardee innovations have had the greatest impact. MA has supported a wide range of organizations under the remit of innovation. However, the extent to which they are innovative is subjective, so in some cases it is not clear how they differ from traditional development projects. The criteria and process used to select organizations do not focus primarily on innovation, unlike innovation challenge grants and competitions. The result is a more varied portfolio, which includes awardees that resemble conventional development programs more than innovative new approaches. There are no indicators for innovation, nor is ‘innovativeness’ tracked, which, in turn, makes it impossible for the Evaluation Team to conclude the extent to which the awardee solutions are innovative. The Evaluation Team was provided with no information on failed awardees other than a set of names of awardees that FICCI considered failures. However, on closer inspection, it was not clear that any of these were indeed failures, nor why. Overall, awardees have been able to use awards for what they intended, including to test and pilot new solutions, start reaching markets, or expand to new markets domestically or internationally. In that sense, they have been able to move along the enterprise lifecycle. However, there has been a lack of focus on graduating from one stage of the program to a next stage, as discussed further in the Capacity Building under Management section of this report. 15 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV EQ2: LEVERAGING EQ2 To what extent has the MA project been able to leverage funding and non-financial value services from various sources like private sector, government, and other bilateral and multilateral donor agencies for additional development outcomes? FINDINGS In this section, the Evaluation Team addresses leveraging at several levels and types: first, MA level funding, examining the sources of funds it has received since its inception and how it is tracked; and second, awardee-level leveraging, both financial and non-financial from various other sources, including the extent to which FICCI has been able to leverage its network for awardee benefit. MA LEVEL LEVERAGE OF FUNDING USD 19.91 million has been added to MA’s collective fund. Six funders have contributed to this fund since MA’s inception. FICCI has therefore used the USAID funds to leverage in excess of the 1:1 matching rate to which it committed in its Agreement with USAID. The largest three contributors are USAID, DfID and GoI, making up 92% of total MA funding as shown in Figure 3, below.10 Other partners, such as ICCO, WISH and ICICI Foundation, together contributed 8% to MA. MA applies a “pooled fund” principle (also referred to as the “kitty”) in which funders agree to contribute to MA. FICCI is not required to report back granular financial utilization details to USAID/India, nor does MA track which funder’s funds are used on which awards. Funders, including USAID, have placed some minimal stipulations on MA on how funds can be spent. USAID funds, for example, go to specific administrative expenses and funding awards in USAID’s five core sectors. DfID specifically supports innovations that are taken internationally, and more recently, awards focusing on persons with disabilities. At the onset of MA, proponents announced that a variety of partners, including social impact funds, venture capitalists, and corporate foundations, would bring in private sector investments to grow the funding pool, bringing the total amount leveraged to $50 million.11 MA has not reached this leveraging target yet through direct arrangements with financial supporters. The Evaluation Team was unable to identify a formal strategy outlining how MA would achieve this target. Further, there was no time-bound target for this leveraging target, and therefore it is not clear whether FICCI has failed to reach that target, or if it is merely a work in progress. However, according to reports by awardees to FICCI, the total enterprise level funding raised by the awardees stands at $49.018 million from Rounds 1 to 5 (see awardee- level financial leveraging below). Given the issues related to reporting (see Management section), this figure is likely to be higher albeit unconfirmed. In terms of private-sector investing, five of the eight private sector stakeholders interviewed claimed that they were interested in participating further in MA, and that they would be willing to engage with the MA system to screen candidates for equity investment, but that MA has distanced itself from them to the point that three investor network representatives no longer felt connected to the platform or knew what support MA would want them to provide. 10 GoI has committed another USD 714,000 in March 2019. 11 https://economictimes.indiatimes.com/news/economy/finance/usaid-ficci-millennium-alliance-launched-to-fund￾innovators/articleshow/14804825.cms?from=mdr 16 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Figure 3: Proportion of Total Funding Received by MA as of December 31, 2018 by Source (USD Million) Source: FICCI Records MA has tapped into FICCI’s large private sector network to a limited extent. FICCI has a network of 250,000 companies12, but only has anecdotal examples of robust member engagement in MA. Marico Foundation, with close ties to FICCI (the founder is the ex-president of FICCI), is also a new MA partner. FICCI is also under discussion with CITI Bank, another of its members, to become an MA partner. Facebook provided advertising support, but not direct financing. CISCO, Abbot, and Marico Foundation (whose founder is an ex-president of FICCI, according to FICCI staff) has also funded awards in MA. The involvement of the GoI has also proven useful to attract private sector participation in MA. According to an officer from a partner organization, “The government is involved, and they have a good platform to attract investors,” but have failed to do so. The importance of leveraging through networking and partnership development was further substantiated by a representative of an inclusive innovation acceleration firm, who noted, “Networks are super important - you can leverage the network to get what you want.” The lost opportunity of leveraging the FICCI network was seconded by a key informant and MA partner representative, who reported they had “only seen one letter from FICCI asking their membership to engage.” Multiple stakeholders from the inclusive innovation space noted that MA engaged them to leverage resources, pipelines of successful awardees, networking and investment opportunities. However, these did not progress beyond initial conversations and information sharing (one respondent suggested that it became a lot of work to figure out how to work together and the leaders of both institutions are busy so the conversation simply fell by the wayside). There has been no systematic action to build a network of a wider set of partners One of these stakeholders felt that FICCI did not believe that “they had a role to play in keeping everyone together, not just involved but synthesizing across many people and utilizing and 12 https://web.archive.org/save/http://ficci.in/about-us.asp 17 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV leveraging strengths.” Another key informant noted that FICCI is reputed for its ability to organize events, but that it needs to be a “master orchestrator” to maximize resources from its networks. AWARDEE-LEVEL FINANCIAL AND NON-FINANCIAL LEVERAGE Some awardees have successfully raised additional funds after or during their participation in MA. Some awardees participated in a Development Impact Bond after their MA experience, receiving funding from UNICEF to expand their operations across other states respectively. Although FICCI estimates based on reports from awardees suggest that $49 million has been secured by awardees from a mixture of grant, equity and debt sources throughout the lifecycle of the program/. Attributing awardee￾level financial leverage to MA is a challenge because the evaluation captured the perspectives of MA partners and awardees, while funders’ voices were beyond the scope of this evaluation, so it is unclear whether or not additional funding was because of MA funding. Table 4: Non-MA Financing Secured by Awardees Round 1 to 5 (USD) CATEGORIES ROUND 1 ROUND 2 ROUND 3 ROUND 4 ROUND 5 TOTAL GRANTS (GOVT) 787,519 62,463 2,122,895 306,555 0 3,279,432 GRANTS (PRIVATE) 2,236,318 2,603,018 1,828,623 770,453 723,163 8,161,575 DEBT 0 0 8,701,940 0 146,972 8,848,912 EQUITY 3,858,025 14,939,638 9,829,218 838 100,000 28,727,719 TOTAL 6,881,862 17,605,119 22,482,676 1,077,846 970,135 49,017,638 Source: FICCI Records as reported from awardees Between Rounds 1 and 5, a total of 124 projects were funded by MA. Of these only 46 projects (37 percent) reported securing external funding from various sources. MA expected that awardees would have the ability to raise funds from alternate sources, using MA funds as catalytic funding. Close to 63% of the projects did not report any external support, the reasons for which were varied, but according to several respondents, often came back to the ‘savvy’ of the awardee to secure additional funds. Awardees were divided about the effectiveness and value of MA’s partnership and network development. Awardees who had successfully utilized the grant and were able to make full use of the FICCI and MA network were highly appreciative of FICCI’s ability to make business grade contacts. These awardees had successfully used the platform to make connections to Corporate Social Responsibility (CSR) donors, make possible successful implementation of Stage I innovation and subsequently connect to funders, receive grant support from private sector financers like Ananya Finance, and make inroads into the India-Rwanda partnership development. An awardee who obtained multiple funds from MA scaled-up to increase field level impact. They described MA as “one of the best investments in India.” 18 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV An awardee commented that MA improves awardee visibility to other funders, media coverage, and exposure and brand validation in the market. They said, “Getting MA recognition is what put us into a global platform and gave our name a brand… they gave us the confidence that we can do anything.” This sentiment was corroborated by other awardees who had successfully leveraged MA for their national operations. One of them noted, “Nobody listens to you at first. MA helped us so that people began to listen to us.” Another awardee expressed appreciation of the MA marketing platform and leverage, suggesting it is a definitive “stamp” of the social enterprise’s proof of concept. Other awardees benefited from awardee meet￾ups, exhibitions, roadshows, and investor￾awardee connections, facilitated by MA. FICCI’s network at the Innovation Business Acceleration Center (Kolkata) and XLR8AP (Vishakapatnam) connected awardees with market opportunities and further support. Seven of 41 founding businesses in the Innovation Business Acceleration Center received MA awards and were receiving in-depth mentoring and coaching to advance their businesses. Another respondent noted that MA creates a bridge to GoI, which has in some cases been able to facilitate linkages between MA awardees and other companies and/or investors. MA has helped awardees create Memoranda of Understanding (MoUs) and partnerships with State Governments or Government of India. An awardee who successfully developed such a partnership noted that, “FICCI is great at linking us with partners - we have been linked monthly with conference exhibitions, government partnerships.” Outside of India, inductions and introductions to the India-Rwanda, India-Ethiopia and other government partnerships helped awardees not only showcase their solutions internationally, but also enter new geographies that brought them increased operations and learnings of scaling and growth. An awardee who had expanded to Africa and Bangladesh, piggy-backing on FICCI’s networks said, “Initially we never had any idea how to scale up. In the last few years - we have developed a partnership model to use in each country. Before going to Bangladesh, we didn’t have this model.” Round 2 Health awardees Pratimesh / MicroX Labs aimed to develop a prototype for a low-cost, complete blood count test at the point of care which would identify diseases in the blood such as dengue and could also assist with testing for anemia. After being connected to a potential manufacturer, who was later found to have lacked FDA clearance to develop the innovation, FICCI deemed Pratimesh a “failure” due to its inability to acquire the medical certificate needed for product development. Despite this setback within their grant timeline, the award permitted them to begin to develop the prototype, and Pratimesh leveraged its connections within MA to find additional funding from departments within the Ministry of Science and Technology and DBS Bank to continue product development after MA. In 2017, the India Institute of Science (IISc) provided incubation and legal assistance to enable Pratimesh / MicroX Labs to file a patent with the successful prototype. They also received additional financial support from the ICICI Foundation and India US Science Technology Forum. Achieving financial independence through leveraging their first grant with MA has paid off; presently the company is valued at USD 10 million with a buyout offer and has four patents and one trademark. Box 2: Health Success Story 19 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Despite the positive evidence shared above, some awardees did not gain as much as they had expected from MA, stating that they were unable to leverage MA into better connections for more support. One Mumbai-based awardee noted that FICCI made connections with investors, but that the awardee was not able to parlay that into any kind of business relationship. A Bengaluru-based awardee commented that in FICCI events, “The entrepreneur doesn’t get to meet customers, only gets to meet other awardees. They would tell us that we would have government figures, investors, private sector there, but they were always badly attended.” Another Delhi-area awardee observed that despite having done good work on the ground, they had been unsuccessful in showcasing their work. Other Delhi-based awardees called for more pro-active facilitation by FICCI among awardees, investors, and collaborators. Despite the positive examples of how awardees leveraged opportunities provided by MA, there is a lack of continuous, systematic and well curated leveraging in MA. A social impact investor summarized this succinctly: “The easiest thing is to give grants and say you are part of MA. MA means you become a part of platform where you actively get ecosystem stakeholders and awardees to meet each other. If they stay true to that platform that can really help. I would assume MA is something where you bring in entrepreneurs with potential to become game changers and unicorns.” While MA awardees reported securing $49 million externally, an investor critiqued MA leveraging by suggesting that they fail to obtain “super-leveraging” by connecting awardees with investors in the form of debt and equity, and therefore lose out on both additional capital and the vested interests of active investors. Aside from these regional chapter initiatives mentioned above, FICCI’s offices across India, which primarily act as liaison centers for their headquarters, have helped in dissemination of information, mobilization for roadshows and key MA events, and implementation support but their participation has been limited. MA draws from several budget lines to support leveraging for awardees. From 2012 to 2018, MA has spent USD 544,733 on operational costs (excluding salaries). These expenditures are shown in Figure 4, below. Secretariat expenses (logistical support for meetings, workshops, events such as capacity building programs, award ceremonies, and meetings for due diligence, evaluations, pitch sessions and Apex Round 2 Agriculture awardee Naireeta Services aimed to address small holder farmers’ pervasive problem of unproductive land and food insecurity due to drought and water shortages. Naireeta’s innovation, “Bhungroo,” is a proven water management system to store excess rainfall from monsoons to be used by farmers facing water scarcity during the rest of the year. Starting in Gujarat and Ghana, Naireeta applied for MA funding to successfully scale Bhungroo to Bangladesh. Naireeta has developed numerous partnerships through engaging with MA, including eight partners and the government in Bangladesh; staff stated that “getting MA recognition is what put us into a global platform and gave our name a brand.” As a model for stakeholder engagement, Naireeta developed a partnership model to use in each country for non-profits, entrepreneurs, government and donors and now operates in 16 countries. To date, they have received 11 more inquiries from African nations about collaborations, as well as initiating new partnerships in Myanmar, Laos and Vietnam. According to staff, “Money comes, and money goes but connections are important.” Box 3: Agriculture Success Story 20 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Committee meetings) comprise the largest share, followed by program outreach (local state offices, industry chambers and stakeholders from the local entrepreneurial ecosystem), program management (day-to-day operations, office expenses, communication, IT), business development (networking events, seminars, B2B portals), and others. Figure 4: Details of MA Operational Expenses (Indirect Costs) Excluding Salaries (USD 2012-2018) Source: FICCI Records CONCLUSIONS MA has successfully gathered a large pool of funding from two primary funders in addition to USAID, namely, GoI and DfID. The funds are pooled and used to fund the awardees. Although each funding partner has separate ledgers, the appropriation of funds is through the pooled fund, the only exception being programmatic foci such as DFID’s support for State 3 enterprises, which meant awardees had a more direct relationship with DFID although the funds were still disbursed from the common pool. Although the platform has successfully leveraged USAID’s funding to bring and cement the position of these two partners, direct private sector funding contribution to MA is very limited. Furthermore, involvement of private sector players, especially impact investors and venture capitalists as proposed at the onset of the project, has not been fully realized. This has led to a situation where a majority of the direct MA funding has been through grants and awards, and very little through equity-based investments, which would have introduced more ownership from investors, although there is evidence of external funding from a variety of sources obtained by about a third of awardees but outside the MA funding scheme. FICCI’s primary strength is networking and consensus building through its reach to 250,000 companies and public institutes. MA and its awardees have had limited gains from FICCI’s vast member network. Even where FICCI members have come forward to fund awards, there is little evidence of systematic support in terms of expertise-sharing from its members, and support from FICCI members in 21 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV general is nascent. Partnerships have developed in silos, mostly through ad-hoc introductions by FICCI and proactive networking by business savvy awardees. A systematic channel to develop awardee to private sector partnerships through FICCI’s network is missing. Despite the expenditure on networking events, the direct outcome of these events has created limited impact and mixed reviews of awardees. Awardees and stakeholders suggest a more targeted partnership development exercise. MA networking and partnership events have finite scope and do not cater to targeted matchmaking required for awardee-level leveraging, with few exceptions. Attendance and number of events are key metrics tracked for partnership events, although there are no means of tracking the quality of “connections” made during the events due to the absence of a systematic events tracking mechanism. State-level FICCI-run platform in West Bengal and Telangana are providing support to some MA awardees through targeted market-linked growth and development support; however, the other three state chapters of FICCI are far less active. At the moment, most regional offices are primarily involved in outreach programs for MA, but not support for awardees. Gains in branding, prestige, credibility and market contacts have been noted and leveraged by awardees, but not tracked systematically. MA awards help awardees gain visibility and credibility (including opportunities for national and international profiling e.g. television appearances or on other platforms like Tie Silicon). Awardees have utilized this branding to generate additional funding from other sources. While the platform has showcased some awardees, there is no systematic plan to do the same for all awardees. Much of the marketing and branding is left up to the awardees themselves without substantial push from FICCI. 22 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV EQ3: DEVELOPMENT OUTCOMES EQ3 To what extent have the innovative solutions made an incremental contribution to USAID’s development results? FINDINGS One of the two sub-goals in USAID/India’s CDCS states the intent for “Indian innovations (to) accelerate development outcomes in India and globally.” Under this goal, the objective of impacting people’s lives at the BoP is envisioned through the identification and scaling of effective health solutions, agricultural innovations, low carbon models, and early grade reading initiatives. Innovations funded under MA have been shown to produce a variety of positive development outcomes in line with USAID/India’s development objectives. Of the respondents in the sample: Agriculture awardees have noted significant gains in livelihood improvements for women and smallholder farmers through developing efficiencies for storing and marketing products, improving produce quality and farmer skill development, creating employment, and utilizing previously overlooked natural resources and land. Health and WASH awardees reported the ability to roll out novel research and tools to improve health outcomes at both the community and health facility level, improve wastewater management and water purification and for those in the pilot stage, the potential to impact diagnostics and improve emergency medical response in rural areas. Education awardees shared that they designed language-appropriate educational material for hard-to￾reach schools, improved literacy, emphasized in-depth teacher capacity development and encouraged a culture of reading for pleasure for primary school students. All of the education awardee respondents described scaling up their initiatives to different states. Clean energy awardees noted successes in applying international standards and best practices to improve rural electrification through affordable solar grids and providing clean cookstoves to improve respiratory health and reduce dependence on fossil fuels. Gender and development appeared to be a focus for a few awardees whose key beneficiary groups were women, especially those in rural and resource-poor environments. These awardees also credited their innovations with creating jobs and opportunities for women. One health awardee operated a manufacturing unit for women’s sanitary napkins staffed completely by women, an undertaking described by the women working in the facility as a catalyst for behavior change around hygiene practices and menstrual taboo in Muslim majority communities. Workers describe using their salary to buy insurance, educate their children, and for savings and investments, without having to rely on their husbands. A WASH awardee started a water purification on wheels initiative that employs all women to check and operate water tanks, transport and deliver affordable clean water to hard-to-reach areas and conduct training on water purification. This awardee also completed a baseline and endline survey to assess improvements in health outcomes for women and their families resulting from access to clean water, including reductions in illnesses and time needed to walk to access water. Lastly, an agriculture awardee who employs women to engage in solar drying of produce and marketing noted that women are “breadwinners” who are 23 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV “committed to improving the lives of their families,” and noted that they have “seen women change from being in the house to being very well recognized,” citing an increase in independence for women farmers. Social development benefits were also noted. Two WASH awardees witnessed a “paradigm shift” not easily quantified, where social benefits emerged from the ways low- and middle-income wage earners work together and treat each other. For example, respondents reported instances of waste pickers allowed into a family compound, not stigmatized, and invited for tea as equal citizens. While several awardees were able to point to concrete or emerging improvements for populations at the BoP, most of the early-stage entrepreneurs have not yet reached this phase. Awardees received funds based on the stage of their innovation, and because the MA award portfolio has a higher representation of Stage 1 awardees who have not yet been able to roll out a product, it is premature to attempt to measure development outcomes for those awardees as a direct result of MA. Figure 5 below shows the number of awards by stage in each Round. Until Round 4, MA issued fewer awards at higher stages, but in Round 5, it issued more Stage 3 awards than Stage 2. Overall, 43.5% of all awards have been Stage 1, 37.1% Stage 2, and 19.4% Stage 3. Figure 6 also shows that the value of awards has become higher for Stage 3 awards over time (with the exception of Round 4, in which there were no Stage 3 awards). Notably, the proportion of Stage 2 awards, which are more likely to be on a path to develop and market their product, has decreased after Round 3. Firewood and biomass are used for daily cooking in many parts of India, with negative implications for both personal health and the environment. Round 3 clean energy awardee Gram Oorja sought to give customers more options and initiated a biogas cooking grid to reduce their reliance on firewood for fuel. Starting with a central biogas plant where cow dung is collected daily, the biogas is transferred using a pipeline to households who pay tariffs for the biogas based on their income and levels of use. The use of biogas has shown to reduce smoke inhalation for women and children, time spent collecting firewood and deforestation, and provides organic fertilizer from the byproducts of the dung; it is also beneficial for those in remote or tribal areas who face difficulty refilling petroleum gas cylinders. With a hope that biogas could be 75% of their business going forward, Gram Oorja is currently identifying larger villages for implementation and researching cattle-dense populations that can provide a sustainable source of dung to keep costs low. As staff noted, “Sustainability and quality over quantity is key to us.” They have begun developing new partnerships to scale within India, and connections with universities have helped them to identify interests in the African market. Box 4: Clean Energy Success Story 24 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Figure 5: Number of Awards by Stage (1-3) and Round for Rounds 1-5 Source: FICCI records Figure 6: Value of Awards by Innovation Stage (1 to 3) and Rounds 1-5 (Million INR) Source: FICCI records To effectively reach broader development outcomes, awardees and stakeholders understand that early￾stage innovators will need additional support, namely, more time to fully develop their product, build institutional capacity, identify additional financial resources, build market linkages, and then reach beneficiaries. 25 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV The amount of award funding directly impacted awardees’ abilities to carry out their work, and many awardees received less than the amount requested in their application. Half of the awardees interviewed (13) who did not receive the full amount for which they applied stated that they needed to reduce the intended scope of their work and revise their workplan. This included reducing or limiting the site locations of work and reach of the innovation, the activities planned under their project (often the more technical or risky innovative activities), and the breadth of targeted beneficiaries. An agriculture awardee stated that “in order to do what we could with what they gave us we looked at the total number of farmers and we had to change the number of those we could meet.” Others felt that their innovations were truncated, preventing them from developing strategies necessary to ensure sustainability or scale-up; a clean energy awardee stated that as a result of less funding than requested, “we cut down our scale and marketing plan, we curtailed our staff involvement…we wanted to expand outside but instead we just had to focus on Delhi.” An agriculture awardee remarked that “there was a large portion of the project that was cut because of the budget cuts. The proposal included consumer awareness raising of organic products and a new brand specifically for organics. That was an important part of sustainability but now it is not happening at all.” The award funding constraints also often included not only reduced funding but, in many cases, also reduced time given to achieve concrete development outcomes, affecting awardees in all sectors. One education awardee stated, “We didn’t get the whole amount we asked for…we proposed for two educational blocks but got approved for half. We also proposed for three years and got 1.5 so the library program was cut short.” Ten of the awardees interviewed who did not receive the full amount requested needed to utilize internal resources, such as existing equity or CSR1314 funds, while others sought alternate, often multiple sources of funding (e.g. grants) to make up the difference and still try to reach the intended beneficiaries or sites originally proposed. MA states that key reasons for not funding a project in its entirety include instances where the amount requested is not justified against the workplan shared, or in instances where MA believes the awardee has the ability to raise funds from alternate sources, using their MA funds instead as catalytic funding. These internal justifications are not shared with awardees, as of the respondents who received less than their proposed budget, only DFID-funded grants were informed of the reason why, and many awardees did face challenges with identifying additional funds. For those who were successful in securing funding from additional sources, they were unable to assign attribution at the outcome level to MA specifically. In the findings for Leveraging, the Evaluation Team discusses in more detail the additional funds raised to support entrepreneurs under MA and the implications of these benefits throughout the innovation lifecycle. This issue is discussed more in the Management section of this report. 13 Since 2013, India has made it mandatory for companies over a certain size and revenue, to spend two percent of its earnings on corporate social responsibility (CSR), under Section 135 of the Companies Act, 2013. The funds can be spent on a wide range of areas under the umbrella of development and poverty alleviation, as stipulated by the Act. Additionally, there are stipulations on the entities that receive CSR, including non-profit status and a minimum of three years of operations. This means that most MA platform awardees, who are for-profit and often younger than three years, are not qualified for CSR funds. Furthermore, the only incubator programs that can receive CSR funding are Government-approved Technology Business Incubators. Other incubators and impact investors are not allowed to take CSR funding that counts towards the two percent rule. This has kept the impact of CSR funding to a minimum within inclusive innovation and impact investment ecosystem. 14 See also https://ssir.org/articles/entry/new_frontiers_in_indian_corporate_social_responsibility# & https://www.villgro.org/csr-in-social-enterprise-incubators-the-what-why-how/ 26 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV DEVELOPMENT CHALLENGES MA awards range from one year to five years. As one awardee stated, health solutions often require “a lot of incubation…but in the time between incubation and approval there is a long gap,” making the overall timeline to develop and scale health innovations longer compared with other sectors. As Figure 7 below, shows, MA has tended toward shorter duration of awards over time. Rounds 1 and 2 featured four and five-year awards, but subsequent rounds did not. In Round 3 and 4, the majority of awards were for one and two years. Figure 7: Number of Awards Issued by Duration in Rounds 1-4 Source: FICCI Records (available data only) The specific timelines associated with the academic year and agricultural cycle also affect awardees’ ability to make development impact within the limitations of their award. One education awardee stated that “[the receipt of award] was delayed for a long time and this is difficult for education because our timeline is with the academic year...the process for MA was in the middle of the academic year, so we couldn’t start anything in the field until school was out.” Similarly, several agricultural awardees noted that the time needed to realize results is crop-specific and compromised by the realities of weather and the patience necessary when planting produce; agriculture is usually on a one-year cycle, making it difficult to see progress or success quickly. As one awardee stated, this makes it “time consuming to scale up an agriculture project because you need to see results for people to get on board...people want to see change- when we talk to farmers in Kenya, they don’t have confidence until a year has passed and they can see the results from their neighbors.” Several early-stage health awardees noted specific challenges in navigating the complex regulatory environment and standards for compliance with medical technology, along with lengthy approvals, personnel and space needed to conduct clinical trials and train health workers. These processes dominated the often-limited implementation timelines before even rolling out a product for market use. 27 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV For others who have successfully developed a product for use at the community level, outreach and development progress could be hindered by language barriers, the need to respect cultural boundaries and general relationship building with community members, with a WASH awardee remarking that “community awareness and response can be slow so the sensitization needs to be very repetitive and can take time,” and one agriculture awardee confirming that relationships take “nearly two years” to develop. As Figure 5 shows, over time MA has provided awards in shorter duration, and this can constrain an awardees’ ability to get their innovation operationalized and see measurable effects within that time, making results-based expectations more difficult. Innovations funded under the MA platform target the populations and environments that are hard to reach or resource poor. Awardees reported a focus on improving the lives of those in rural areas, for communities facing shortages in water and natural resources, those experiencing infrastructure shortfalls, and those affected by inconsistent or unavailable health services. While the operating sites for awardees in Rounds 1-5 tended to focus on implementation in resource poor environments, awardee headquarters were substantially clustered in and around large cities. While one MA stakeholder noted that awardee headquarters location was not a concern and that “even if it is an innovation in Bangalore, quite often it is implemented and is helping populations outside of the capital city,” there is a noticeable lack of representation of awardees outside of major capital areas (Figure 8). While there are 19 states where entrepreneurs have received the 124 awards in Rounds 1 to 5, over half of the awards come from three large states - Delhi (North), Maharastra (West) and Karnataka (South). The East and North East are underrepresented in both applications and awards. For those limited awardees who are based in the North East or implementing projects there, their operating environment can hinder development progress, namely due to environmental, political and logistical challenges. One awardee operating in a North Eastern state remarked that because of these challenges, entrepreneurs must prepare for a more difficult terrain to navigate and the potential for setbacks to implementation, noting that “you have to go with the mindset of helping farmers, and not focused on RoI or transaction costs,” otherwise it will be frustrating to attempt to measure development. The North East awardees interviewed also felt that the level of support received from the MA platform to adjust to hurdles around achieving development outcomes was limited. The support provided to awardees in the North East is further explored under Knowledge Management and Communications. Source: FICCI Records. Note: there are no states with 9-14 awards. Figure 8: Number of MA Awards Granted by State in Round 1 to 5 28 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV The Evaluation Team assessed the extent to which Return on Investment could be analyzed for each project, with feasibility based on available data ranging from not possible to unreliable. This assessment is limited by: (a) the unavailability of baseline data; (b) the lack of appropriate indicators to assess return on investment; (c) attribution slippage since for most awardees, funding comes from multiple sources and there are several variables that contribute to returns; (d) limited scope within the evaluation SoW to undertake a full RoI assessment and address attribution through in-depth qualitative analysis; and, (e) the challenge to engage awardees to discuss sensitive and often proprietary financial data. Since MA comprises awardees working on both for-profit and not-for-profit models, RoI may not be a logical assessment for all awardees. Further, not all respondents were convinced that RoI is appropriate in the MA context. As one awardee respondent mentioned, “They want us to have co-investment - because it makes them look better and it grows the entire award.” Another, quoted earlier in this section, suggested that development outcomes and RoI are not always applicable, especially in remote areas where implementation costs are high and development results are slow. Nevertheless, the Evaluation Team developed a method that could indicate RoI at the project and sector levels, with the caveats that there could be significant returns that are not captured by this model due to lack of data and should therefore be considered indicative at best. The model and limitations are listed in Annex 3. Table 5 shows some illustrative RoI calculations based on individual awards. Due to the extent of assumptions that would be necessary by costing each of the returns, the Evaluation Team has followed Social Return on Investment best practices and recorded raw data. As the data show, even in these illustrative examples, there is high variability among some of the variables. Three of the four samples are for awards of 5 to 6 million rupees (1), yet the reach per million rupees (4) ranges from 32.5 to 827 people. Other variables, like direct job multiplier (6), show less variation. As a percent of the award, award value (1) compared to additional funds raised (2) also remain quite low, but constant at about 5 to 6% of the award value. A common indicator across all sectors in terms of development outcomes is likely not useful, which makes comparison among sectors difficult. Table 5: Illustrative Examples of Return on Investment NO. ROI INDICATOR AGVENTURES CORP. (AGRICULTURE) SIKSHASANDHAN (EDUCATION) WATERLIFE INDIA (WATSAN) SATTVA MEDTECH (HEALTH) 1 Direct portfolio outlay (INR mn) 5.7 5 20.9 5.8 2 Indirect portfolio outlay (prorata) (INR mn) 0.34 0.3 1.25 0.35 3 Total MA portfolio outlay (INR mn) 6.14 5.3 22.15 6.15 4 Reach (persons impacted/ INR mn)15 827 667 226 32.5 15 No. beneficiaries does not capture depth of involvement (e.g. Sattva provides fetal monitoring systems which are very high value but low scale). 29 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 5 Leverage multiplier (outside program)16 123% NA NA 536% 6 Direct jobs multiplier (per INR mn) 1.3 1.7 2 1.3 7 Total jobs multiplier (per INR mn) 15.1 1.7 0.3 NA 8 Sectoral Outcomes17 8.1 Increase in income of farm households 10% - - - 8.2 Increase in test scores of students - 45% - - 8.3 Increase in access to clean water - - 20% - 8.4 No. deaths averted - - - NA NA = Not Available CONCLUSIONS MA funding has enabled innovators to make incremental development impacts by developing new or improved access to health, WASH, clean energy, agriculture and educational products and resources not previously guaranteed for many and has the potential to further improve the quality of life for targeted BoP populations. However, very few awardees are at a phase in which they are or can be achieving real impact; timelines may be unrealistic to achieve substantial, if any, development outcomes, especially for those in agriculture and health sectors. The overall development impact depends on factors often outside the control of awardees, and, to a large extent, is affected by the amount of their MA award and the time given to implement. This stunting of project work caused by unexpected, insufficient funding has reduced the ability of the MA platform to ensure consistent, concrete development outcomes through the awardee innovation lifecycle. While each of the awardees in the key sectors of health, WASH, education, clean energy and agriculture sampled shared successes, there are significant sector-specific considerations regarding technical or community support needed, policy, regulations and logistics that affect implementation and may not be wholly addressed by MA. While MA has created a platform to seed innovations, it has a limited geographic reach to be able to broadly serve remote corners of India that house entrepreneurs to support those who can design appropriate innovations and solutions for their region. The under-representation of awardees outside of a select few larger cities, while not necessarily intentional, is a missed opportunity for MA to tap into underserved populations of entrepreneurs that may need exposure and resources. 16 Many awardees have not provided the details of externally raised funds (debt, grant, equity). Otherwise, leverage remains the most comparable quantitative RoI metric 17 Sectoral outcomes are sometimes not comparable even within the sector (e.g. health). Additionally, the timing of these outcomes is not provided, making attribution to MA challenging. 30 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV RoI data are challenging to obtain due to inconsistent availability. Extracting generalizations from the varied results presented would be premature; however, it is clear that the variability of data among awardees, sectors, and stages of innovation makes comparison unreliable. 31 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV EQ4: SUSTAINABILITY EQ4 To what extent have the innovative solutions become sustainable? FINDINGS CRITERIA AND MEASURING SUSTAINABILITY USAID/India addresses sustainability in the CDCS related to institutional capacity, Indian ownership, linkages with government strategies, and self-sustaining innovation platforms engaging a range of public and private stakeholders. MA factors sustainability into its assessments of applications for awards,18 and weights it 20% in the second step of evaluation; however, the Evaluation Team could find no documented shared understanding of sustainability in MA documentation. One document describing application scoring criteria identifies financial sustainability (ensuring continued financial support to grow an innovation), environmental sustainability (reducing the innovation’s risk to the environment), and social sustainability (making strong development impact) for consideration, but the meaning of each is left unclear and not mentioned again in any other assessment documents outside of the broad term of sustainability. The guidelines provided by MA assess the likelihood the business model will sustain the project beyond the MA grant period and its scalability beyond MA. Applicants are charged with demonstrating this to be the likely case.19 While the jury views and scores all awardees with these criteria in mind, there are no specific indicators related to assessing and scoring sustainability that would merit an objective review; for example, there is a stated directive that financial sustainability should indicate a “viable plan post the MA grant which is not grant based,”20 but no other specified criteria to guide the scoring around the subjective measurements of what is “viable.” Also, there is no differentiation of how to measure sustainability based on the stage and sector of the inclusive innovation, where awardees have noted that innovations at different stages have very different parameters and metrics when it comes to determining their own sustainability. There is a notable lack of focus on the non-financial sustainability criteria highlighted in other USAID funded innovation platforms, such as consumer demand, cost and accessibility, private sector engagement, and ongoing political support.21 The singular definition of sustainability was highlighted by a MA partner, who observed that “sustainability is a challenge because there is no single strategy in development.” 18 https://web.archive.org/web/20190403025206/http://millenniumalliance.in/images/Solicitation_Document_MA%20R5.pdf 19 Pitch guideline document, Discussion on evaluation sheet for MA full application, Due Diligence Report for MARV00112 (example), and Evaluation Criteria for Stage 1 MA (Round 4) 20 MA Application Cycle http://www.millenniumalliance.in/application_cycle.aspx 21 Sourced from application materials from the USAID U.S. Global Development Lab’s Flagship innovation initiatives of Grand Challenges, Development Innovation Ventures and Global Innovation Fund https://www.usaid.gov/GlobalDevLab/about/innovation 32 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV FINANCING As noted in the above section on Development Outcomes, 54% (67 of the 124 awards) of awards received less than 50% of the award they requested, which had implications not only for implementation, but also affected sustainability in some cases. MA did not provide any structured mechanism to support innovation sustainability when reducing awards. With the exception of DfID-funded Stage 3 projects, awardees granted less than proposed budgets were asked by MA to submit a revised budget with no specific support to do so. All respondents interviewed who experienced these cuts claimed they received no response from FICCI on revised budgets. This issue is addressed in more detail in Knowledge Management and Communication. MA expects that awardees “use the MA funds catalytically by raising funds from other sources”22 and, as discussed in the Leveraging section, some awardees have successfully done so. While this may be the case, it opens further discussion among both awardees and stakeholders about the pros and cons of becoming grant dependent if the funds catalyzed are from other grant funds. There was evidence of a tendency for numerous awardees to “hop from grant to grant” without becoming market ready. As one stakeholder noted, “[entrepreneurs] get into a grants cycle and it feeds them, but they never get down to selling the product.” Innovations of some not-for-profit organizations (such as those in education) have no sustainable commercial elements, as beneficiaries are not expected to pay for the product or service; thus, grant-based implementation is core to their operations. On the other hand, grant dependency by private sector organizations to push their innovation is not sustainable, as it diminishes organizations’ efforts to tailor their innovation for the market. Some awardees pool finances from multiple sources, and MA funding is a small portion of their total available funding. As a result, these organizations are not completely dependent on the MA award, and some even use the MA award as seed capital, leveraging the visibility of the platform to reach and market to broader audiences with their product and linking their commercial sustainability to new donors. Others use the MA fund to take risks that they might not be able to afford without a grant, such as testing a product to BoP population. While some have increased funding from traditional partners to meet any funding gaps, others have taken a more market facing approach through more effective sales channels by focusing on the willingness and ability of BoP customers to buy their products or finding a bulk buyer for their solutions (often the government). Some enterprising companies have applied for and received multiple awards between Round 1 and 5 that has enabled them to see their solutions evolve into more successful, market-oriented products within the MA system. While financing was a concern stated by awardees regarding their long-term sustainability, the Evaluation Team did not find any evidence that any awardees ceased to operate after the end of their MA funding. Further, innovations tested within MA continued to be operational after the award period. However, it is notable that within the MA reporting system, awardees are not asked to submit a final or end-of-award report, and FICCI has not enacted a formal process23 to follow up with awardees after their grant terms have ended, which limits its ability to gauge sustainability or opportunity for future success. 22 MA Application Cycle http://www.millenniumalliance.in/application_cycle.aspx 23 FICCI has conducted informal follow up with awardees, though not systematically 33 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV INVESTMENT The likelihood that an awardee and their innovation funded under MA will continue to be successful in the long term depends on a myriad of factors. It was noted by both awardees and stakeholders that the current model of grantmaking though small grants may not be sufficient for entrepreneurs to scale up considerably, make an impact and ensure sustainability. One awardee felt that there are “a few business models which are scalable across different areas.” Another stated, “Millennium Alliance has a tendency to take a venture approach where far too many number of ideas are selected and given small chunks of money. For sustainable impact, a long-term strategy is required.” Even for the awardees who received the full amount of funding they requested, the amount may have allowed them to reach only short-term development goals; they would need multiple rounds of support (both financial and non￾financial) to see their innovation through to production or to scale and ensure success over the long term. An awardee noted the need for continuous “support for 5-6 years after which project can sustain itself.” Several awardees stated that their business would benefit from a deeper investment by MA and its partners at the outset coupled with MA’s support with networking to help ensure that their hopes for sustainability will come to fruition. Others emphasized the importance of “the right partners to ensure the product will still be cost effective and more stable,” and will ensure they have “consistency from product to product.” As discussed in the Leveraging section, 46 grants were reported as securing external financing amounting to a total of $49 million. 37 percent of the awardees were able to get a mix of grants from private and government sources, in addition to equity and debt financing to support their work. While equity-based funding amounted to 59 percent of the total, debt and private sector grants were 18 percent and 17 percent, respectively. Government grants amounted to only 7 percent. STRATEGIES FOR SUCCESS A significant aspect of sustainability lies in the success awardees feel when developing networks and new partnerships for investment, rolling out a product and receiving feedback from the market on how their innovation is received and utilized. In terms of scaling-up within India and beyond, awardees felt successful when they personally developed links to host country or state governments, and local partners and had both a clearly defined need and demand for their solution. Awardees highlighted the importance of building trust in communities they plan to work with (e.g. farmers), which takes time, but is a measure for likelihood of success in developing lasting business relationships. Understanding language and culture was important in working both outside of their host state and country along with assuring an affordable cost, and continuous pipeline of resources. Without those 34 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV assurances, awardees felt that not only would they not be able to roll out a product, but that the consumer interest and investment would not be sustainable over time. One awardee expressed that upholding a positive reputation and a business model focused on “sustainability and quality over quantity” led to increased trust and continued expansion of their innovation. Other awardees also noted that they felt successful when they were proactive in leveraging the existing relationships they had with FICCI, and in creating networks on their own instead of waiting for FICCI to make connections, as FICCI could not be expected to provide support after the grant. Adaptability was a key focus for several awardees who adjusted business operations to suit the conditions and needs of the market, i.e. changing their target audience, shifting price points, and using commercial marketing channels. Being flexible in an ever-changing market is a necessity for many awardees in remaining relevant and upholding a sustainable business. SUSTAINABILITY OF THE PLATFORM While the source of funds is partner-driven, respondents report that the sustainability of the MA platform lies in the commitment of key stakeholders and funding partners. FICCI has been able to orchestrate buy-in from other donors such as DfID and GoI. While some stakeholders in the inclusive innovation space expressed reservations about sustainability, other stakeholders felt that for both existing partners and potentially new partners, there was a commitment to “taking it through to the next stage.” One partner noted that although the Technology Development Board had pulled out of MA due to a shift in policy, the GoI’s Ministry of Science and Technology is committed to the continued support of MA, and saw MA as key in continuing the global expansion of Indian innovation. As discussed in the Leveraging section of this report, at its launch, MA announced an ambitious goal to become a USD 50 million fund, the progress towards which is unclear (unless awardee-leveraged funds are counted, however, as noted, attribution to MA is problematic), but MA has been successful in attracting additional funding partners and continues to expand through on￾boarding new partners. Millennium Alliance has laid out its sustainability strategy in its 2016 Vision Document to 2025, in which it continues commitment to “game changing innovations” and plans to increase its funding base in order to expand. The plan calls for the creation of a business development unit that will be charged with mobilization of funds and developing new networks but does not suggest how new resources will be mobilized, nor tentative targets. Developing the sustainability of the platform is an ongoing effort by all MA partners. According to FICCI, the Department of Science and Technology To address poor reading rates in Karnataka, Round 3 Education awardee Akshara Foundation started a 60 day rapid program in 1600 schools to help children relate more to books and initiate libraries in schools. Government praise came with a request to scale up; with MA funding, Akshara rolled out the school library program as a pilot in both urban and rural schools and ensured that there were sufficient resources needed for language-appropriate print materials available in schools. Akshara received positive feedback from teachers who were trained on storytelling and from schools implementing the library that saw an improvement in literacy and children’s confidence. As noted by Akshara staff, “the funding really allowed us to do the pilot in rural schools, especially to a backward block (education and economically). We were able to bring the joy of reading to 35,000 children.” While the grant ended in 2017, Akshara focused on providing all resource materials to the government to continue implementation and created a publicly available how-to guide on their website so that other NGOs could adapt and continue to utilize their successful model in the future. Box 5: Education Success Story 35 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV has sanctioned an additional $714,000 for one year with an indication of further extension. Similarly, existing partners, especially DFID, have shown interest in exploring the possibility of floating no-interest debt funding in phase II of MA, for which the Government of India has recognized FICCI LEAD as an incubator. Lastly, considering the legal complexity of using CSR funds for funding MA awardees, which are a mix of non-profit and for- profit entities, the Ministry of Corporate Affairs (GoI) has been approached to amend the CSR law to enable inflow of these funds into the alliance. CONCLUSIONS MA has no scope for variations in sustainability requirements based on the stage, geography, nature, sector and business maturity of the inclusive innovation. The singular “one size fits all” approach to sustainability and scale is throughout the program, limiting not just the scope of sustainability throughout the program lifecycle, but the nature of support expected by both MA partners and awardees. Past awardees interviewed by the Evaluation Team were still operational and continued to pursue the growth and development of the innovations that benefited from MA. The Evaluation Team could not perform a more complete a diagnostic due to the lack of systematic follow￾up from MA on awardees both during and after the completion of their grant. If funding is a proxy for the scope and quality of the innovation, reduced funding alters the sustainability of revised solution and product. The revisions in scope and quality are not considered in a re￾assessment of sustainability, thereby creating a mismatch between the proposal and award activities. The platform has not provided support for developing a revised sustainability strategy under these conditions. Most importantly, the MA award amounts do not allow entrepreneurs to reach the full breadth and depth of impact needed for an innovation to be successful over a longer term. Awardees are appreciative of more intimate technical assistance with regards to commercial sustainability especially help in budgeting, cost optimization strategy support, and business plan co-creation. The MA Program itself has growth ambitions that require additional resources for scale-up, but there is no concrete plan or targets for developing these resources. 36 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV EQ5: MANAGEMENT EQ5 How effective is the management of MA project by FICCI? The management section of the evaluation resulted in a wide array of findings arising from the breadth of evaluation questions asked under this heading. Therefore, Management findings and conclusions are broken down into smaller themes. The themes are: (1) management effectiveness; (2) selection and disbursement; (3) capacity building; and (4) monitoring and evaluation. MANAGEMENT EFFECTIVENESS FINDINGS FICCI has a staff team of five people assigned to MA, some of whom are assigned only part￾time. Awardees and FICCI staff reported that the limited personnel forces staff members to play multiple roles, rather than having designated portfolio managers. Two FICCI staff members represent the primary contacts for all awardees. An MA partner explained that they have a good working relationship with FICCI. “They have limited capacity, but the intent is there,” explained the respondent. Another MA partner commented that FICCI is “trying to do too much with too little staff,” and others concurred. Limited staffing means limited support for awardees. As one awardee said: “If the right team is put together with MA, they have access to great people. They can help you figure out who to connect with, when, and how. Many awardees are first time entrepreneurs and this engagement would help the speed at which these organizations are growing…. With FICCI overall there is a lack of ownership and their leaders are doing so many things so they cannot focus on the awardees or providing consistent support.” Respondents suggest that structure of the staff team is also not conducive to effective management, especially considering the support that awardees request, demonstrated later in this section. Awardees and key informants alike suggest a portfolio-based structure either by sector or Stage in which FICCI can provide specific support to the needs awardees, rather than the general approach currently used. Awardees suggest that the lack of sectoral or even start-up expertise is a missed opportunity. Stakeholders and awardees refer to FICCI as only a grant mechanism, not an innovation￾supporting platform; FICCI excels in event management and planning, but not systematic support for innovation. "Right now it’s a top down approach and the oversight should be coming from those who operate in this system and know the unique needs from the bottom up," said an awardee. One￾third of both awardees and stakeholder respondents reported that MA would be more effective if it could take innovators through the stages of development and support a more systematic trajectory toward financial, market and development outcome success. As discussed later in this section pertaining to monitoring and evaluation, MA management has so far underutilized results outcomes from awardees in managing the platform. 37 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV CONCLUSIONS In terms of MA staffing, respondents favorably acknowledged the effort that the small MA team applies in ensuring MA runs smoothly. At the same time, there are several areas of work that have been neglected. Feedback mechanisms (discussed in Knowledge Management and Communications) are one example, but others include a lack of sectoral in-house expertise or for early-stage start-ups. FICCI staff are over￾stretched and performing the minimum functions to ensure the grant-making facility is operating. The generalist structure of FICCI is not meeting the needs of awardees in terms of managing the platform effectively. Overall, the focus of MA management has been on grant-making, the effectiveness of which is discussed in the next session. There is little attention paid to the development of awardees as businesses or organizations for the award to support the long-term development of awardees. SELECTION AND DISBURSEMENT EFFECTIVENESS FINDINGS All awardees with whom the Evaluation Team spoke went through a similar application process. FICCI reports adapting and learning from the process in each round, which from Round 3 onward has featured: Partner deliberation of the applications, leading to the awards Pitching the application by the awardee to a panel of experts with a mix of sectoral, development, and commercial experience Due diligence by technical and sectoral experts (including site visits A second screening process by sectoral experts, also sourced from USAID and DfID specialists An initial screening process by FICCI-selected technical specialists A public call for applications in local and national newspapers with details clearly outlined on the MA website Generally, respondents believed that the application process was robust, and that technical, financial and management analyses are conducted during the selection process. Awardees reported having sufficient information on what was being evaluated in the call for proposals. An awardee reported, “in the RFP we could see exactly what they were looking for - timeline, relevancy, cost and so on.” Some smaller scale applicants struggled with the application process. At least five awardees mentioned that they had to hire a consultant to navigate the application process, partly because they lacked sufficient 38 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV capacity in English to respond to the call. Others in more remote locations suggested that the online process might put some applicants at a disadvantage because of poor internet connection and computer literacy. Another MA partner noted that the volume of applications is high in each round (nominally 1300 applications). Of that, estimated the respondent, about half would not be appropriate. The rest would be appropriate, but only about 200 would be good proposals. These are the proposals that would be competing for the awards. Generally, the screening processes result in about 10 applications per sector, or about 60 shortlisted applicants for each pitch round, of which about half get awards. In the process of filtering applications, technical experts, many of whom are from USAID, are consulted (Stage C above), and the shortlisted applications are inspected through site visits through a due diligence process by a combination of sectoral and development or business specialists to validate the claims made by the applicants in terms of stage of innovation, viability of the proposed activity. Although the jury members- selected based on development, commercialization and sector expertise￾have access to due diligence documents and the proposal, respondents believed that the pitch is the most important component to ultimately be selected for an award. The pitch is nominally 10 minutes long, three of which are spent with a video made in advance by the MA team that showcases the innovator displaying how their innovation works or will work, as well as their worksite and materials used. This leaves up to seven minutes for the innovator to present the case for funding, which jury members score on a scale of 1-10 (formerly 1-5). Some awardees felt the jury lacked technical sector expertise or did not ask appropriate questions to fully understand the innovation, while others felt the due diligence questioning and jury were well-versed in their sector and innovation. The Evaluation Team received a mix of responses indicating the extent to which jury members were appropriate to make recommendations. The following quotes from awardees indicate the diversity of perspectives: “[The jury] asked one or two intelligent questions but for a 2.5 crore project, they didn’t understand who we were and what we wanted to do… it was demeaning.” “... there were no obstetricians, which would have been the most relevant for us, but there were engineers and GoI pediatricians.” “There were 11 or 12 people in the pitch. They asked good questions.” “They asked 15 to 20 questions, which were relevant.” “The jury was tough… they asked good questions though.” The graded pitches are then sent along with the due diligence documents and proposal to the Apex Committee, comprised of funding partners, to make the final decisions on applications to award and award amounts. Apex Committee members interviewed for this evaluation suggest that the decisions are near unanimous. Most respondents also mentioned that the process “took too long,” especially compared to other grants they had applied for, but still respected it as a process. Awardees reported that the duration of time from 39 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV proposal submission to award receipt was upwards of eight months to one year, and several respondents suggested the time was in excess of one year. There was no measurable difference noted from awardee respondents across rounds; however, two awardees felt that Round 3 awardees faced greater delays due to a perceived changeover of staffing at MA. Awardee frustration stemmed partly from the MA stated timeline of four to five months for approvals and the lack of communication of delays by FICCI. Others noted that the environment in which they proposed to work changed considerably during the period it took to approve awards, to the extent they had to modify the proposal design. FICCI has been effective at disbursing funds to awardees. The start dates of each award by round have been consistent in June with the exceptions of 2014 (Round 2 in November) and 2015 when MA applications were put on hiatus while the management team adjusted the application system. One of the main areas of reconfiguration was related to the monitoring and evaluation system, which was identified as a weakness by the MA management team and partners after Round 2 after realizing that they had limited ability to understand impact. According to FICCI Management, FICCI contracted Thinkthrough Consulting (TTC) to design an improved monitoring and evaluation system, and then to manage results data for subsequent rounds. Awards have been consistently issued since then for Rounds 3 onward in June of each year.24 The values of awards in each round have fluctuated, as has the concentration on sectors. Figure 9 below shows the value of disbursements by round and sector. Round 3, which included DfID funds specifically for international application of interventions, shows a slight increase in clean energy, health, and water and sanitation, and a sharp decline in education, one of USAID’s priority sectors. In Round 4, there were no DfID funds, and USAID and FICCI agreed to better align with USAID development outcomes, resulting in a smaller overall disbursement envelope, which was corrected in Round 5, in which there was a marked increase in education, health and agriculture and the introduction of disabilities, a DfID priority area. Similarly, Figure 10 below shows the number of awards issued by sector and round. In Round 3, there was an increase in the number of awards issued, but as seen in Figure 9 and Figure 11, fewer funds were issued in Round 3 compared to Round 2. Round 5 shows more awards issued to health and education innovations. 24 Individual awards commenced on different dates, depending on the negotiations with MA. 40 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Figure 9: MA Disbursements by Round and Sector (Million INR) Figure 10: MA Disbursements by Round and Sector (Number of Awardees) As noted in the Sustainability and Development Outcomes sections, many respondents commented that the value of awards was less than what they asked for, and more than half of awardees (55%) received less than half of the value of their application. As Figure 11 shows, the average value of awards has declined after Round 3, when the number of awards increased to 34 from 21 in Round 2, showing a movement to issuing more smaller awards than in earlier rounds across all sectors. Source: FICCI records Source: FICCI records 41 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Figure 11: Average Award Size by Round and Sector (Million INR) Awards are decided by the Apex committee. As noted above, the average award amount was higher in Rounds 1 and 2 than subsequent rounds. The value of the awards is capped by the stage of development of the applicants. The MA website clearly lays out both the stages and the award caps for each stage, as shown in Table 6. Table 6: Stages of Development and Award Caps STAGE DESCRIPTION AWARD CAP 1 Piloting or testing an innovation 30 lakh (10 lakh for capacity building) USD 48,850 (USD14,280) 2 Scaling or replicating an innovation which has been successfully piloted 1 crore (10 lakh for capacity building) USD 142,850 (USD14,280) 3 Innovation for developing countries No cap (10 lakh for capacity building) (USD14,280) Source: FICCI interviews and http://www.millenniumalliance.in At least 17 respondents interviewed for this evaluation reported receiving a smaller award than they proposed in the application process. FICCI representatives explained this by suggesting that applicants tented to apply for more than that for which they were eligible and/or applied as a Stage 2 business but were assessed as Stage 1. An MA partner explained that the award amounts are products of negotiation, so sometimes an award may be given to appease an advocate on the committee, but to a lesser amount. The same MA partner said that there is a need for MA and its partners to justify its reach and claiming a greater number of awardees is one way to do that, which may explain some of the award reductions. While the ways in which MA communicates the differences between ask and award will be discussed in the Development Outcomes section of this report, some unpacking of the ask-award data is discussed here. Between Round 1 and 5, 33.9% of the awardees submitted applications that exceeded the cap for their stage of business. They were all awarded Stage 1 support. Applicants proposing more than the cap for Stage 1, but assessed as Stage 1 innovations, experienced an award of 43.6% of what they proposed on average. Applicants that requested within their assessed stage received 93% of the cap for their stage Source: FICCI records 42 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV on average. Stage 1 applicants who asked within the FICCI limits received an average of 106%25 of their proposed amount. Stage 2 and 3 were awarded 53% and 63%, respectively, of their proposed budgets. Successes and failures have already been discussed in Innovation and Development Outcomes. Only one award was prematurely stopped by FICCI for underperformance. Otherwise, failure has been elusive due to a lack of data and definition. CONCLUSIONS The application and award approval process is generally considered robust, but there are some issues that advantage some types of applicants over others. English-only and online constraints were brought up by several respondents as barriers to application. Remembering that the Evaluation Team only spoke with awardees, not rejected applicants, it is possible that this is a larger issue that privileges awardees more likely to possess these skills or have the resources to acquire them for the purpose of the application. In terms of disbursement of awards, FICCI has consistently issued awards within its priority sectors, many of which overlap with USAID development outcomes. Over time, MA has adjusted its disbursement strategies to better align with the interests of funding partners and external messaging to promote the platform’s wide scope. At the same time, the shift in Round 3 to 36 awardees but 44% less total funding than disbursed in Round 2 signals a move to more awardees of smaller value. Although several awardees whom the Evaluation Team interviewed reported receiving awards significantly smaller than their proposed amount, this can be attributed largely to either misalignments of self-assessment and MA-assessment of stage of innovation; or applicants attempting to frame their innovation as Stage 2 to receive a higher amount of award. The extreme variances in proposed budgets and awards is found in the Stage 1 awardees who applied as Stage 2 or for Stage 2 caps. For those who made proposals that were consistent with MA￾assessed staged, Stage 1 awardees received higher awards than requested on average and Stage 1 and 2 received smaller amounts than proposed, but not enough to completely negate development outcomes. CAPACITY BUILDING FINDINGS Capacity building funds are embedded within awards, with 10 lakh mandated by FICCI after Round 2 to be dedicated to capacity building. Many awardees were unaware of the 10-lakh fund allotment embedded within their budgets for capacity building and were only told that the funds would be reserved for capacity building after receiving the award. Current Rounds make this amount more explicit; however, there remains significant confusion among awardees around how capacity building funds can be spent. While some awardees reported being able to choose their own capacity building priorities and source out the best way to develop that capacity (whether through seeking out opportunities in India or even outside the country), others understood that capacity building was primarily intended for attending FICCI training events and expositions (referred to as “road shows”). Upon further probing with respondents, those that were able to allocate capacity building funds to non-FICCI hosted events did so on their own initiative, approaching FICCI with the request. FICCI suggests that they are 25 This figure is skewed by one Round 3 awardee that received 71% more than they asked for. 43 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV flexible to allow awardees to use capacity building funds for any reasonable activities and confirmed that the onus is on the awardee to make such requests to FICCI. The Evaluation Team identified three awardees in Round 4 who were Stage 1 and received a total award of 10 Lakh solely for capacity building, and not for product development or scale-up. Many awardees interviewed mentioned that FICCI did not discuss or assess capacity building priorities or needs with them. They expressed that they had outstanding capacity building needs that have not been met under MA (e.g. team building, management, internal training, marketing). In some cases, awardees reported giving feedback on the type of capacity building required (now part of the quarterly reporting template, but it remains unclear how the data will be collated and used), but no action was taken to provide such support. Several other awardees reported never being consulted about capacity building interests or needs. Awardees acknowledged that the new reporting templates ask for capacity building interests, but also commented that no action has been taken on the requests they have submitted, and they have received no feedback. FICCI respondents confirm that they conduct capacity building needs assessments regularly, with the most recent one conducted in October 2018. Some Stage 1 awardees specifically, stated an interest in developing the capacity to scale up, with awardees noting key areas of institutional growth necessary to invest in in order to scale-up, including partnership and network development, understanding market expectations and appropriate certifications, product development, human resource development and personnel skill building, and identifying additional financial resources. Though these skill areas were identified as those most critical for early-stage entrepreneurs seeking to advance their innovation stage, few of the respondents confirmed that the learning activities in MA focused on business and operational competency were offered to them under MA. Awardee respondents expressed an interest in capacity building on topics directly related to their business rather than compliance with MA, such as marketing, upgrading in value chains, global scaling up, and business and operational competencies. A key informant commented, “capacity building should be mirroring what impact investors need and what awardees need to market themselves.” Yet another awardee asked, “What is capacity in terms of a start-up? It is having the intellectual and physical bandwidth on the ground. I need improved field operations…I don’t need conferences.” Concurring perspectives were shared by several other awardees as well. These quotes suggest a misalignment between the offerings of MA and the interests of awardees. However, as Annex 11 shows, there have been several training events oriented towards developing business practice and institutions. Though, these events have been offered sporadically, such that not all awardees might have been offered the event during their funding cycle. Annex 11 also shows that nine capacity events have been offered by FICCI in six years of operation. Additionally, the majority of events tend to be centered in and around Delhi, requiring awardees to use personal or capacity building funds for travel and lodging. Awardees based outside of Delhi stated the distance, time and cost a distraction from participation. While workshops on monitoring, evaluation, and reporting to FICCI are regular (though also centered in Delhi), others are typically one-off. For example, an awardee requested training on the regulatory environment, which MA has provided once in the past. FICCI drafted a Capacity Development Plan after Round 2, which addresses a number of themes for capacity building that awardees mentioned to the Evaluation Team; however, as shown above, awardees still perceive a deficit in terms of addressing capacity building needs. The extent to which these focal themes were systematically derived from awardee input remains unclear. Several respondents explained that capacity-building needs vary based on stage of innovation, but under MA, all awardees received the same capacity-building support options, making some options 44 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV inappropriate for different types of awardees. One awardee explained, “I have communicated with other awardees and we all have similar experiences of feeling that the capacity building activities were not relevant to us…the relevance depends on your type of company and the stage of the innovation.” Another explained that the legal training “was very helpful and the fundraising one was not helpful - it was for early stage. We were already well-versed in that.” Other respondents echoed this sentiment. Some awardee respondents suggested that MA capacity building is more useful for smaller companies, and others suggest it was better suited to larger companies. Awardees interviewed provided mixed reviews of the quality of capacity-building activities that have been offered by FICCI, with some suggesting they have been useful and others suggesting that they attended one or two events and found them not worth the effort. Expositions are considered eligible for use of capacity-building funds, and awardees who attend appreciated that they can gain international exposure through this avenue. However, several awardees reported that training events offered by FICCI were too general to be useful for them. CONCLUSIONS Lack of clarity from FICCI on what the eligibility and criteria is for capacity building funds has created confusion among awardees with the effect of under-utilization of this feature of the platform. The general approach to delivery of trainings to both businesses and non-profit organizations in a range of stages of development and an array of sectors limits the impact that capacity building funds can be used to improve the skills and knowledge among awardees. There is also lack of clarity of how FICCI addresses expressed interests of awardees in its capacity building repertoire. Consistent offerings are around the core needs of MA to ensure awardees are compliant with conditions of contract and effectively communicate MA results to the public while the types of capacity building that awardees find useful are inconsistently offered, and the quality of capacity building events is contested. MONITORING AND EVALUATION FINDINGS The MA monitoring and evaluation system has been in development since Round 3, before which there was no centralized system for understanding development outcomes or project progress. The system has been led by TTC, however the expectations of the system were not sufficiently clear, which has led to what both TTC and FICCI refer to as an iterative process, or “learning by doing” as respondents described it. The growing pains have not been lost on awardees, some of whom noted in interviews; one awardee explained that the MA monitoring and evaluation system is “super weak and rudimentary.” Starting in Round 3, with TTC under contract, MA developed a list of 17 quantitative “common indicators,” which are grouped by sectors, with one to four indicators per sector.26 The common indicators are collected by TTC as part of award-level indicators that are agreed upon by TTC and the awardee at the start of the award period. The common indicators have been developed to be broad 26 https://web.archive.org/web/20190318044315/http://www.millenniumalliance.in/impact.aspx 45 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV enough to apply to a range of projects; however, according to respondents, they are sometimes an awkward fit on the ground. As one WASH awardee reflected, “[TTC] asked us for indicators on the number of toilets we were putting into place and we couldn’t get it through to them that we weren’t installing toilets- the setting of the indicators can be improved substantially- you need people who really understand what the awardees are trying to do. You can’t impose indicators that have nothing to do with what we are doing.” One awardee reported that they were encouraged by TTC to use indicators that they felt failed to reflect the impact they were making with the award. Some of the respondents explained that although there is some room for qualitative responses, such as lessons learned, in reporting formats, the indicators themselves fail to capture important results. Yet another claimed, “We have to assume that they are the right indicators, it wasn’t done according to our input.” The respondent clarified that the indicators were chosen by TTC with little opportunity for the awardee to revise. Other awardee respondents expressed similar sentiments. Awardees do not uniformly submit monitoring results to TTC. The TTC tracking sheet shows that for the first four quarters of Rounds 3 to 5, TTC managed to receive reports from about 70% of awardees. Figure 12, below, shows the result of an assessment conducted by TTC after Round 4 of the availability of data from awardees. This report varies slightly from the analysis above, but still shows overall that 31 of 88 (35%) awardees do not make their data available to MA. Further, that which is reported is often incomplete, with qualitative sections that could highlight unintended results, for example, often left blank. Figure 12: Data Availability from Awardees Round 1-4 by Sector Source: TTC While there has been capacity building on monitoring and evaluation for awardees, respondents report that the nature of the training is on completing the forms provided by FICCI rather than on how to formulate, verify, and interpret individual indicators. They also reported that the quality of the training was insufficient to develop a reliable monitoring system. Other awardees reported that the monitoring and evaluation capacity building was beneficial. 46 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV FICCI reports that the MA team plans to verify results for 10 awardees per year. However, the Evaluation Team was unable to obtain evidence that this spot-audit approach is taking place. No awardee respondents with whom the Evaluation Team spoke reported any contact with FICCI in the field after the due diligence process was completed. In addition to quarterly reporting of indicators, several respondents mentioned ad-hoc requests (and often last minute) for data by TTC or FICCI that they had not agreed to collect. This was a source of frustration for awardees who feel that unreasonable expectations have been put on them. For example, one awardee was asked for sex-disaggregated data when the awardee had been collecting household-level data. Another awardee called the monitoring “piecemeal.” There are some inefficiencies built into the outsourcing of monitoring and evaluation, as representatives of FICCI, TTC, and other organizations commented. As a key informant commented, monitoring and evaluation needs to “sit within the core [of MA] – there should be a consultant involved solely from the start – a solid amount of time monthly with the same people. String the learning together as opposed to piecemeal. It is in the interest of the funding agency to get the richest learning. It is right now only outsourced.” Awardees also reported limited to no feedback on monitoring frameworks or reporting. One awardee cited a rapid changeover in staffing at TTC for the lack of feedback. The feedback loop from TTC to awardee is discussed further in Knowledge Management and Communication. Further, some awardee respondents commented that the indicator-based monitoring and evaluation style of progress tracking is NGO-focused, missing the point for businesses. Two respondents suggested that start-ups have specific milestones that indicate progress and signal market and management readiness to advance to the next stage of business. CONCLUSIONS Although there has been advancement since Rounds 1 and 2, MA has had four years since then to develop its monitoring evaluation systems. MA and TTC have focused their efforts on working with awardees to establish indicators and to report on them regularly. Overall, these limited objectives have been met; however, the monitoring and evaluation system remains problematic. The common indicators are not clearly linked with MA outcomes and outputs in a cohesive theory of change or logical framework, making it impossible to assess the extent to which they are measuring the “right things” and putting into question the effectiveness of the existing monitoring and evaluation framework. At the same time, these common indicators are not a natural fit with all awards. Indicators often lack specificity and time-bound progression targets, jeopardizing the efficacy of the monitoring and evaluation system as a whole and limiting the ability for FICCI to manage, or do anything but report on, monitoring data. By subcontracting monitoring to TTC, FICCI has relinquished a significant opportunity to build up in-house capacity for monitoring and evaluation. There is a risk in this arrangement that the potentially rich dataset and relationships with awardees is at an arms-length. This arrangement also creates some communication delays, thus reducing the efficiency of management. 47 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV The monitoring and evaluation framework has not been documented well. Award-level monitoring plans comprise some results-statements with indicators, intermittently populated with time￾bound targets, and without follow up by TTC or FICCI. Monitoring indicators are therefore used only for rolled-up reporting that can be published to demonstrate MA results and falls short of a management tool or a system that could be used as a basis to improve the performance of awards. Other than the reporting of indicators and few qualitative fields such as lessons learned, there was no evidence of a broader monitoring strategy. As several awardees commented, there is a significant amount of ad-hoc data collection as well as mis-targeted results monitoring. Due to the lack of standardization and definition about what indicators mean and how they should be collected, as well as the lack of systematic verification by TTC, FICCI to a third-party monitor, the reliability data that are reported from the monitoring templates is unclear. FICCI and TTC have been focusing on strengthening monitoring but have paid little attention to evaluation. 48 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV EQ6: GENDER EQ6 To what extent did the MA platform consider gender issues in the awards in each sector? FINDINGS There are a number of awardee organizations that focus primarily on empowering women, both women-owned and men-owned. For example, by offering low-cost, high quality products for women, running production units employing women and employing women to distribute goods such as water. Some organizations reported running trainings and other initiatives to empower women in the community. At the same time, working with women can pose cultural and societal challenges for awardees. As one awardee responded, “We have faced opposition- no one was ready to let the women take charge. We have to be clear about the local religious and cultural context if we want to scale up the operational model,” which illustrates the challenges of integrating gender considerations into their award. Another awardee noted that at times he and his co-founder, who is a woman, needed to decide who ought to approach a potential business or distribution partner based on gender, as some male-dominated professions are not comfortable doing business with women. The Evaluation Team reviewed a FICCI-provided list that identified 23 out of the 124 awardees (18%) in the Round 1 to 5 award portfolio as “woman entrepreneurs.” However, there was no evidence that FICCI has a working definition of women entrepreneurship currently used under MA. The Evaluation Team applied the following definition to classify an organization as woman-owned: if either one-third or more of directors are women together with the top decision maker being a woman, or at least 51% of the organization is owned by women (a 50/50 division between men and women does not qualify). Using this criterion, it was determined that there were only seven awardee organizations that could be classified as women-owned and/ or controlled, or six percent in the entire portfolio. While this was almost one third less than originally estimated by FICCI, the difference between these figures is likely the stricter criteria the team applied, including ownership of at least 51%. Round 3 WASH awardee Hasiru Dala Innovations in Bangalore is a social enterprise that works with waste pickers and waste picker entrepreneurs from marginalized communities to formalize their work (through the introduction of a workers ID card and the introduction of a uniform), as well as offer a predictable and stable source of income through regular clients and garbage load, better working conditions with cleaner waste and safer waste handling practices. Many waste pickers are women and Hasiru Dala empowers them, not only through a stable income and livelihood, but also by, for example, help women get access to bank accounts and health insurance, as well as schooling and loans for low￾income private schools for their children. Waste picker entrepreneurs get access to loans to buy a vehicle which they own after four years. The main founder is a woman with two decades of experience in working with waste pickers. While the venture only received one third of the grant amount they applied to catalyze and scale their proven model, the award has opened doors for them and given them credibility and the process of applying for the award helped the team get clear on their strategy going forward. Nevertheless, while they have a gender balanced team, and promote improving women’s livelihoods, they did not feel that this improved their exposure or support from MA. Box 6: Gender Success Story 49 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV As FICCI lacks a definition to specifically identify women entrepreneurs, there is also limited focus on gender in the selection process. There is no explicit scoring or selection criteria related to gender inclusion, and no evidence of specific considerations to include or target women applicants or innovations targeted towards benefitting women. Furthermore, there is no gender expert on the jury or selection team, only 15% of jury are female, and of these, most are from MA funders. A sectoral breakdown of jury members further highlight that women jury members were primarily clustered in education and health. “On a year to year basis, about 50% of the jury remains the same. It is very male dominated,” said one key informant who has participated on the jury every round. Aside from inconsistent requests to collect sex-disaggregated data at the beneficiary level, there is no systematic reporting or collection of data on gender. As one Round 2 awardee respondent said, MA “only cared if the innovation worked. No one asked about gender or for gender data.” Another Round 3 awardee responded, “They wanted to know ratio of boys vs girls only, but we wouldn’t say it was a priority for them.” Awardees reported an overall lack of gender focus within the MA platform; they have not been offered training related to gender inclusion and reported that the MA program team does not stress gender aspects or raise awareness about gender inclusion or equality. When asked if FICCI had talked to them about gender-equal workplaces, gender balance (the number of women in their organization), or inclusion (policies and practices to include women), awardees stated that no such conversations had taken place. In the context of FICCIs external relations, there is no explicit mention of gender in the MA informational brochure27, nor is there a mention of gender in the publicity materials28 or social media feeds29 through which the MA promotes the platform. One awardee commented, “I don’t actually remember them sharing what is their vision but did not get the impression that gender was a priority for them.” Another said, “FICCI does not seem to favor innovation directed towards women” in the context of what FICCI’s core focus areas are. CONCLUSIONS There is no evidence that MA has a gender inclusive focus with respect to the selection process and criteria, or the composition of the jury. The low number of female-owned or run awardee organizations suggests that there has not been a concerted effort to create a more gender balanced portfolio or to bring on board more women owners and managers. There is no evidence of mechanisms that support woman applicants or provide support specifically for women awardees. MA does not engage with awardees on gender inclusion in any meaningful way. There is neither a focus on ensuring that the innovation MA supports follows gender inclusive principles, nor an emphasis on gender inclusive and equal work practices within the awardee organizations. 27 https://web.archive.org/web/20190409012413/http://www.millenniumalliance.in/imgs/MA_Brochure.pdf 28 E.g. https://web.archive.org/web/20190409012305/https://twitter.com/MADevImpact/status/1077088936361689088 29 https://twitter.com/MADevImpact and https://www.facebook.com/MADevImpact/ 50 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Gender equality is not systematically tracked or reported on. Without tracking data on gender inclusivity both at the level of the awardee organization and the level of the awardee innovation and how it is rolled out, there is not much one can say about the gender impact of the MA as a whole. Overall, the current design of the MA platform does not have gender at its core. While FICCI claims to engage on gender, there is no evidence of gendered considerations in how the FICCI team operates or in how the team manages the MA platform. This is misaligned with the values of core funders including DfID and USAID that have both taken steps to emphasize gender inclusion and equality across programs and other activities. 51 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV EQ7: KNOWLEDGE MANAGEMENT AND COMMUNICATIONS EQ7 To what extent has the MA project succeeded in serving as a knowledge platform for the innovations supported and established with the wider Indian social innovation and development ecosystem? FINDINGS The Evaluation Team identified three major channels for knowledge management and communications based on responses from both stakeholders and awardees: knowledge sharing among awardees, between awardees and FICCI, and between FICCI and its networks and the wider public. While awardees agreed that information sharing is important in their growth as entrepreneurs, there was limited evidence of knowledge sharing and communication reported by awardees among themselves or with other actors outside of MA as part of their involvement in the platform. Sharing through networks among awardees was often inconsistent or ad-hoc as a result of casual meetings or occasionally through FICCI-sponsored, MA branded events. The main outward facing activity from FICCI has been inviting awardees to take part in showcase events, where they could present their work in a stall for invited MA partners and investors; from Round 1 to Round 5, MA has conducted 210 roadshows across India. Awardees felt that these MA sponsored showcases as well as the workshops (i.e. capacity building) were important opportunities to bring awardees together but many felt they used a “top-down” approach to give information about MA expectations to awardees or enable partners to interact with awardees, rather than serving as collaborative environments and opportunities in which awardees could systematically learn from each other. One health awardee confirmed that “the roadshows are not useful...it is a nice opportunity to meet with other awardees and see what others are doing but that’s it.” Most of the awardee respondents stated that while it would be necessary and beneficial to make connections with other awardees working in their sector or geographic location to learn from each other’s success and challenges, there was no systematic platform in which to facilitate peer learning through MA. One awardee noted, “We don’t have a formal structure for learning from each other outside of these exhibitions.” Another awardee noted that it would be in MA’s best interest to facilitate these types of peer learning connections, stating, “If we had more cohort or sector-based meetings that would be best. If you have a great pool of people in the same sector it’s a great way to leverage their strengths.” Many awardees noted the presence of a WhatsApp group initiated by FICCI that served as an information platform for awardees in a given award round. While a select few deemed the group useful for making connections, the majority stated that this platform was not responsive to specific sectors and was used mainly to provide updates on upcoming events or highlight where successful awardees were showcased, and not to directly connect awardees with each other. An education awardee noted, “We had a WhatsApp group, but I didn’t see value in it...I did not facilitate any conversations with other awardees who work in education.” Another awardee confirmed the lack of opportunity for peer learning, stating, “There are no other opportunities to see what the other clean energy awardees are doing. It would be very important- that should be enabled because especially for those in other areas, other rounds, we can learn from each other.” 52 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Awardees reported that they shared updates to the public on their innovations through their personal websites, word of mouth or social media pages, and did not use the MA website for publicity. Awardees who have made connections with others both in and outside of their sectors to enable learning in the form of guidance or troubleshooting cite their own proactivity and not assistance from FICCI, even where there is an expectation that FICCI will or should facilitate. Another awardee confirmed this point, remarking, "We have a lot of things going on and want to share with others… this is not a very collaborative space- where there is a larger agency that is capable of making growth happen, they should be able to do this." Feedback loops are a key mechanism for management to provide oversight and support, as well as generate key information for learning. Between awardees and FICCI, there is an expectation to complete quarterly or milestone-based reports that will inform FICCI about activities conducted, progress made and fund utilization. The quarterly report template includes sections dedicated to recording information on lessons learned, capacity building needs, major successes, challenges and mitigation strategies. However, while this information is being collected by FICCI, awardees shared that there is no follow-up guidance, feedback or action taken by FICCI. Additionally, there is no evidence of a process in place by either TTC or FICCI to capture and share learnings from awardee reports disaggregated by sector or stage of innovation, which would prove helpful for awardees looking to scale up or make connections with others in their sector. Finally, there is no evidence of reported results being disaggregated by women-owned business of women-focused grants, whether it be success, challenges, risks or lessons learned that would enable MA to identify or track specific ways in which the platform could better support women entrepreneurs. Initiating the new reporting format was meant to capture key areas for systematic knowledge sharing, but as one health awardee stated, “[MA] told us there would be follow-up, but there hasn’t been anything significant. We are asked to show how we have utilized our funds but there is also no follow up on this.” Only in the last three months (as of November 2018) has FICCI tasked TTC with collating the information captured in individual reports, namely the lessons learned and challenges, but it is not yet evident how FICCI intends to use this information. TTC has not initiated an exercise to collect this information from past reports, but only from Round 5 onwards. To date, there is no systematic compilation or separate analysis of the risk mitigation strategies identified or success or failure of these strategies contained in awardee reports, by either TTC or FICCI, and this information is not being shared for learning purposes. To promote learning, as one awardee points out, “Sometimes you would like a feedback on what went wrong.” As noted in the Innovation and Development Outcomes sections of this report, both stakeholders and awardees share the critique that MA is not representative of entrepreneurs in the North East areas of India. Regarding knowledge management and communications, there has not been a concerted effort to recruit these innovators to the MA platform, or to conduct systematic outreach to areas outside of larger cities. While a little more than one-third of the 115 awardees in Rounds 1 to 5 are currently headquartered outside of major innovation hubs and capital cities of Delhi, Bangalore, Mumbai and Chennai, some awardees operating in these areas (some the sole awardee in their state) feel overlooked or “neglected,” and that their work does not receive fair recognition. As one WASH awardee noted, “Most of the researchers belonging to [Indian Institutes of Technology] IIT, Guwahati and in the Universities of the North East do not know about MA initiatives… there was no media publicity or special attention given that someone from the North East received an MA award.” 53 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV One agriculture awardee noted, “Being outside the metropolitan area does pose challenges related to access to the MA platform such as attending roadshows, events, and workshops.” Many awardees stated that there were no applicable regional activities for them to participate in to share information and that they were tasked with traveling to Delhi for MA facilitated events, for which they could draw on their capacity-building funds. Several awardees felt that the attention paid to awardees in their region was superficial, and that MA engagement in half-day programs or limited outreach activities in the region were “just to meet the formalities,” but failed to close the knowledge gaps between innovators in the North East and the MA promoters or supporters who conduct the activities. Another awardee claimed an “inferiority complex” when comparing themselves with entrepreneurs in larger, more collaborative markets in the major capital areas. A senior MA partner confirmed that while there was good work happening outside of the most popular areas, unfortunately entrepreneurs based in “low￾income states don’t have much knowledge or support” under the MA platform. There is both an opportunity and a need for MA to better engage with entrepreneurs outside of major cities in order to share information, make connections and promote business growth. One awardee stated, “There is a definite scope to create more awareness about the program among entrepreneurs in remote rural areas, even from states like Bihar, Jharkhand and Odisha. Many of them are doing excellent work in social sectors and MA can engage at the state level.” There are some indications of success coming from specific FICCI offices as discussed in the Leveraging section of this report. Between FICCI and its larger network of members, partners and the wider inclusive innovation ecosystem, evidence of MA knowledge sharing is minimal. Among its network of partners and members, monthly partner meetings are conducted to discuss updates in management, operations, and partnerships and are useful opportunities to present awardee work and discuss lessons learned among partners. These meetings have been highly beneficial, but only to partners and select invited awardees as discussed in the Program Design section. To the wider public, the MA website and social media pages post success stories from select awardees and provide updates on upcoming exhibitions or events, as well as allowing interested parties to express interest in innovations showcased on the site. MA also released a coffee table book in June 2018 in New Delhi; the book showcased stories and pictures of all awardees from Rounds 1-5. It was also noted that FICCI has encouraged the idea for a newsletter, which would serve as a means of sharing information gleaned from awardees and from MA management with its stakeholders; however, at the time of the evaluation, this initiative had not yet begun. However, some stakeholders feel that MA is not as cutting-edge as it was when it first began and needs more visibility in the wider ecosystem to better promote the platform, what it can provide for early stage entrepreneurs, and the awardees who can benefit from it. FICCI and MA have not published any work on lessons learned, either from awardees on lessons on becoming a social entrepreneur or regarding the MA platform itself such as building the portfolio, design strengths and weaknesses and supporting inclusive innovation. There are several stakeholders who engaged more deeply with the MA platform when it began and have since only peripherally been involved with the platform. One stakeholder who operates within the ecosystem but does not attend monthly meetings noted, “Within the network or partners, action learning suffers because there is no regular knowledge sharing.” Another remarked, “I am just not sure what is happening at MA.” These remarks emphasize the need for stronger communication of the platform across all stakeholders, and a need for the platform to better utilize and involve the innovation expertise that it sought to engage from MA’s inception. 54 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Similar sentiments were echoed by awardees, where one awardee felt that "MA should have a strong media partnership from day one… so much good work is happening, but no one knows of it." There is no evidence of FICCI’s participation in conferences or seminars to provide knowledge sharing across the ecosystem. To date, the most highly publicized activities are the awards ceremonies, exhibitions and road shows, which serve to showcase awardees, but do not promote a platform for systematic knowledge sharing with others in the ecosystem. While there are many success stories coming out of the MA platform, it is also expected to have awardees who are less successful or who could be considered as “failed.” FICCI provided a list of awards considered to be failures, however, while FICCI regularly shares success stories both externally with the public and internally with partners, there is no evidence of these lessons from “failed” awardees being captured, shared or acted upon by FICCI. CONCLUSIONS Despite being operational for seven years, MA and MA-sponsored activities and events have been insufficient to promote consistent, ongoing knowledge sharing among awardees and with key stakeholders both in and outside of MA, which could better promote the MA platform, enhance visibility of FICCI as a platform leader in innovation, and contribute meaningfully to the innovation ecosystem. MA involves awardees at multiple stages of innovation and growth. However, awardees do not currently have a platform to systematically share lessons learned with other awardees about technical, management and other business processes, which is a missed opportunity to motivate awardees, assist in troubleshooting and promote sustainability of their businesses. This lack of peer learning and connection in turn affects early-stage entrepreneur’s abilities to knowledgeably scale-up, within or outside of their state or even outside of India. Similarly, the lack of a regulated institutional knowledge base within MA to capture, respond to and share feedback and lessons learned from awardees and also from its own learnings prevents the wider ecosystem from generating valuable information about how best to support early stage innovators through their entrepreneurial life cycle. While successes are easily lauded and promoted, failure is recognized as a reality by FICCI, but not systematically addressed even though the lessons from failure can be critical to guide both other awardees and MA in the future. There is a missed opportunity for deeper engagement with the limited awardees in areas outside of heavily populated capital areas, namely the North East region. The lack of concrete connections between awardees in all states, especially between those awardees within the same sector and those looking to scale up to new states, and the minimal outreach conducted in these areas is detrimental to the promotion of ongoing learning about implementation challenges and success in resource poor areas. The lack of awareness among entrepreneurs in more remote areas about the platform also prevents MA from having a wider applicant base in which to select awardees. 55 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV EQ8: PROGRAM DESIGN EQ8 Is a Chamber of Commerce an appropriate mechanism for what USAID is trying to achieve via the MA platform? FINDINGS DESIGN EFFECTIVENESS There is a high level of concurrence among respondents that MA is “a good idea,” as one respondent phrased it. Particularly when MA began in 2012, it filled a perceived gap in the market (discussed in the Background section of this report), and continues to fill a critical gap in start-up grant-making today. Most respondents with whom the Evaluation Team spoke suggested that FICCI is the best-placed organization to manage MA, namely due to its extensive networks, perception of neutrality, and close ties with GoI. An MA partner representative asserted, “If I had to put my money on it, I would put it on FICCI.” Another suggested that in 2012, partners were not sure that FICCI was the right choice, but that they have been effective at grant-making. No respondents could identify a better alternative to FICCI for the implementation of the MA, but some suggested that there is space for more collaboration with organizations working in a similar space, specifically business incubators and venture capital investors. A key informant commented: “[FICCI does] a good job of bringing people together – especially the government and all these different partners. FICCI is so connected that not having them involved would mean that we would lose the connection. I wouldn’t remove FICCI but should bring in a few people better at partnership management who are relationship savvy.” However, an alternative perspective is that FICCI is the wrong type of organization because of its bureaucratic function, lack of agility and flexibility, and lack of understanding about the needs of start-up companies. According to one key informant, FICCI is more of a facilitator than a strategic thinker, suggesting that FICCI needs to adapt to changing environments of fast-paced innovators better. Another key informant expressed, “FICCI is driven by costs” rather than by a paradigm of investing in start-up companies, sharing a similar sentiment that stimulating innovation has to be more than holding events and giving grants. Some respondents cited that since MA started, a much broader range of support for early stage entrepreneurs has emerged in India, and that MA should respond to this changing environment by blending better with other types of financing models, such as equity investment, debt financing, and repayable awards. Respondents acknowledged that different financing mechanisms might be more suitable to different types of enterprises and therefore having a selection of mechanisms available may ensure the most appropriate fit with a greater number of innovators. This has been discussed already in response to evaluation question on Innovation. The design of MA is that it is overseen by a Strategic Advisory Committee, as defined in section B2b of the Public Private Alliance Collaboration Agreement between FICCI and USAID: 56 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV “FICCI will establish a Strategic Advisory Committee (the SAC), comprised of senior leadership of the Parties’ respective entities, including the USAID Mission Director or designee, the FICCI Secretary General, and the Secretary of TDB and/or the Secretary of the Department of Science and Technology in the Ministry of Science and Technology, as well as additional Alliance Partners as may be agreed. The SAC will meet on a regular and as-needed basis, as defined by the Parties. The SAC will be responsible for providing broad strategic guidance to the overall MA Program and will assist in resolving operational problems or disagreements.” Although no longer referred to by MA partners as the SAC, there are regular (mostly monthly) partner meetings where senior managers meet to discuss the management of MA. Partners with whom the Evaluation Team spoke suggested that the meetings were useful and that there is sufficient engagement of the funding partners (FICCI, USAID, DfID, and GoI). Several awardees who were invited to present at monthly partner meetings also suggested that the experience was one of the few times that they obtained feedback from MA, which was quite useful. Respondents raised some questions, however, related to issues with the partner committee. First, FICCI is represented by MA staff rather than the Secretary General, and these staff members are not empowered to make many decisions, which creates delays in decision-making. Second, there are divergent interests among the funding partners and other partners, which do not provide financial support but provide training or marketing services, such as the World Bank and Facebook. One of the main selling points of FICCI, attested to by FICCI itself and a range of stakeholders, is its network. However, there is a lack of evidence that FICCI has tapped its network to support MA either financially or in-kind. As one partner representative close to MA noted, “We have only seen one letter from FICCI asking their membership to engage.” As stated in the Management section, FICCI has engaged some members as MA partners or auxiliary funders; however, the engagement is small compared to the overall size of the network. Another element of the design of MA is the inclusion of technical experts from the partners to assist in the selection process. Respondents from within FICCI and among the partners concur that these experts from USAID, DfID and GoI are an important part of identifying sector-relevant initiatives. The Evaluation Team found no evidence of problems created by FICCI serving multiple stages of businesses or having multiple sectors. There are suggestions from respondents that different stages of business should have different requirements and that FICCI fails to deliver support that specifically addresses these phases, i.e. stage-oriented capacity building or peer-learning. USAID/India has adapted in several instances, within its own constraints, to ensure that MA is effective at addressing the needs of the Mission. This is despite USAID’s lack of contractual leverage to influence the operation of MA, as explained by multiple USAID respondents relating to the nature of the contract. FICCI and the other partners view key USAID personnel in high regard and look to them for leadership. As an example, USAID staff have been instrumental in addressing a communication gap evidenced after the first two years of operation by becoming a key organizer of the monthly partner meetings to facilitate dialogue among the partners, as well as facilitating introductions of new partners to the platform. On USAID’s lead, the SAC was remodeled as the partner committee, which featured monthly meetings of the partners. This level of engagement is recognized by several partners as important to the effective operation of the MA. 57 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV INDICATORS OF FAILURE Failure has also been addressed in the Innovation and Knowledge Management and Communication sections of this report. FICCI identified three reasons for project failures: 1) An awardee could not develop the prototype; 2) An awardee cannot obtain the appropriate permits or patents; and 3) The prototype did not perform well in the marketplace FICCI has not, however, categorically tracked these failures during or after the award implementation period. FICCI generated a list of five failed projects for the Evaluation Team. One failure, for example, failed to obtain a permit for a medical device during the award period; however, two years later received a valuation of USD 10 million in multiple acquisition offers. Another, according to FICCI, developed a product that proved too expensive for small farmers, though there was a strong market for larger farmers. In this instance, the innovation failed to produce intended development outcomes, but was a commercial success. One weakness that limits understanding of failure is an underdeveloped monitoring and evaluation system, which is explored in the Management section of this report. From a design perspective, there was little attention paid to monitoring and evaluation, as explained by MA partners and FICCI alike. There are not always time-bound targets for indicators, rendering assessment of success or failure impossible during the award period and sometimes even after the award has finished. Payments to awardees are not based on results, but on activity-based milestones, nominally at 45%, 45% and 10% intervals during the implementation period. CONCLUSIONS In many ways, FICCI has been a reactive organization, trying to keep pace with increasing numbers of awardees and putting effort into adapting and improving its management of MA. While there is support for FICCI as the implementer of MA, there are also ample critiques from MA partners, awardees, stakeholder and FICCI itself. No clear alternatives to FICCI emerged consistently from respondents. One of the missed opportunities in MA is linking the FICCI members with MA awardees. Without examples of this occurring systematically, it is difficult to determine how this might work, but MA outsiders have had good success with mentoring arrangements. Established businesses in the FICCI membership may present ample fund capitalization potential for MA awardees. USAID has proven useful to MA implementation where it has been involved. Of note, the USAID technical experts have proven useful in the selection process and USAID has been instrumental in the SAC/ Monthly Partner Meetings. Failure of awardees is difficult to measure in MA because there are very few awards that have ceased due to performance issues. Payments are made by rough activity completion milestones, not by results achieved, so there are few examples of awardees who are not able to deliver activities. Due to the weak monitoring system, MA is unable to identify failures in general, and when it does, there is no long term follow up to understand the extent to which failure was temporary or critical. The indicators of failure are therefore insufficient. 58 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV RECOMMENDATIONS The Evaluation Team identified 35 recommendations based on conclusions from the evaluation. The Team then assessed the priority for each recommendation for heuristic reasons. The 17 recommendations assessed by the Evaluation Team as high-priority are listed in Table 7 below and represent the recommendations most easily acted upon by MA. The other 20 recommendations are listed in Annex 8 and are still important. It is also entirely possible that MA partners might argue that some medium- or low-priority recommendations are high-priority. Overall, the Evaluation Team recommends that all recommendations are considered and discussed among MA management within the next year, focusing first on the recommendations in Table 7, and then on Annex 8. The evidence basis for each recommendation is noted by evaluation question and recommendations are grouped into themes, which were developed inductively from the recommendations themselves. These themes include: 1) Focus: programmatic specificity in terms of what MA aims to accomplish; 2) Management and Governance: the way that MA is run, including decision-making, sense of ownership and finances; 3) Monitoring, Evaluation and Learning: measurement of results and impacts, and how MA partners, awardees, and stakeholders can learn from results; 4) Service Delivery: the nature of service that MA provides awardees. The Evaluation Team then suggested a lead partner to act on the recommendation and the relative level of effort required (Low: usually simple action that can be acted upon with a single communication or policy; Medium: a more complex series of discussions or deliberations that may take several person-weeks to complete; and High: an involved process that will likely require several person-weeks of work, deliberations, meetings, or the development of a robust product or documentation). Finally, the Evaluation Team suggests a timeframe reflecting urgency of action (short-term: within the next six months; medium￾term: within the next 18 months; or long-term: within the next 18 months to three years). 59 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Table 7: High-Priority Recommendations REF RECOMMENDATION EQ THEME LEAD EFFORT TIME FRAME 1 If the aim of MA is to support innovation, the platform should introduce selection criteria that focus more clearly on innovation, drawing, for example, on innovation challenge competitions. These should be based on a clear understanding of what counts as innovation for the MA. FICCI and the MA partners need to agree on one clear, and more specific, definition of innovation that is then used systematically to inform the way the platform and program operates, as well as the kind of awardees it selects, the support it offers the awardees, and the outcomes expected. Innovation scorings should be more systematic and map to the definition on which MA has agreed. Likewise, MA should create and track indicators related to innovation of awardees in order to be able to deduce whether awardee solutions constitute innovation, and to be able to understand what works and what does not work with respect to enabling innovation among awardees. To understand what indicators can be tracked, review how partners such as Marico Innovation Foundation and UnLtd India track innovation. EQ1 Focus FICCI Medium Medium Term 2 Develop a leveraging plan that shows: 1) how and when FICCI will reach its USD 50 million funding target; 2) from whom FICCI intends to solicit funds; and 3) how FICCI intends to better engage both with its members and its larger network in fundraising and capacity-building to raise funds. EQ2 Management & Governance FICCI High Short Term 60 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 3 Contractually obligate future awardees to submit monitoring data in order to receive payments, particularly if payment milestones are shorter in duration. MA could stipulate that awardees must be in good standing in terms of financial and progress reporting in order to receive the next tranche of funds. The next stage would be to set thresholds for results performance under which progress must be reviewed by MA before disbursements are made. EQ5 / EQ8 Management & Governance FICCI Low Short Term 4 Check for and support the development of policies to create gender inclusive workplaces as part of the due diligence process. Such criteria should either be in place or awardees should commit to putting them in place with the support of an MA specialist. Fundamental criteria should include at a minimum: 1) development of a POSH committee and policy; 2) maternity leave policies; and 3) provision of training / workshops on gender inclusive work culture and unconscious bias training. EQ6 Management & Governance FICCI Low Short Term 5 Continue support for MA from MA Partners, with FICCI as the lead implementer. Continue engagement of the MA partners through monthly meetings, including the engagement of awardees as possible. FICCI representatives with decision-making authority need to be at MA partner meetings. This requires that either a) senior FICCI management attend MA partner meetings regularly, or b) MA management be empowered to make decisions. EQ8 Management & Governance USAID Medium Short Term 6 Develop a detailed monitoring and evaluation strategy for MA including: 1) rationale for monitoring; 2) using monitoring data for effective project management; 3) operating procedures for under performance; 4) an MA theory of change that clearly links to common EQ5 Monitoring, Evaluation and Learning FICCI High Medium Term 61 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV indicators and the role of innovation; 5) verification protocols, including award selection and division of responsibilities; 6) development impact evaluations; and 7) post-award impact assessment. This monitoring and evaluation strategy document should be developed jointly by TTC and FICCI, should monitoring data collection continue to be contracted to TTC. Specifically, such a strategy should verify the ways in which FICCI expects itself to use award monitoring for management purposes and risk mitigation. 7 Strengthen data collection to capture networking and leveraging across the MA at both the program and awardee levels. FICCI’s existing beta tested grant management platform can be extended to track outputs from business grade connection and conversations. The Management section of this evaluation also suggests mechanisms to ensure greater awardee participation in supplying data to MA. EQ2 Monitoring, Evaluation and Learning FICCI High Short Term 8 Develop core indicators that address gender issues, not simply sex disaggregated data. Such indicators could include changes in the social or financial status of women, but also number of decision-makers who are women in the company or organization, gender inclusive policies and other signifiers of a gender inclusive work culture, and policies that emphasize ways to engage with awardees or beneficiaries that are gender inclusive. To develop indicators, draw on the work that has been done under the remit of gender lens investing. EQ6 Monitoring, Evaluation and Learning FICCI Medium Short Term 62 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 9 FICCI’s monitoring and reporting system should ensure that collecting awardee information from reports is an iterative process for learning, not only capturing successes, key lessons learned and challenges for all awardees (and disaggregating by sector and stage of innovation), but in systematically feeding that information back to awardees and key stakeholders and coordinating a space in which to discuss feedback at all levels. This process will improve both award management and communication. In practice, this would include a regular summary and sorting of feedback collected from awardees (based on quarterly reports for example), to be assessed by a key person at FICCI and prioritized for discussion with management and possibly the MA partners. Each feedback item should be responded to in some way, even if non-actionable, in which case a response can explain why the suggestion is not actionable. progress on addressing feedback should be documented and key people who will address the issue should be made accountable through regular report-back on time￾bound mechanisms of addressing feedback. EQ7 Monitoring, Evaluation and Learning FICCI Medium Short Term 10 Develop a manual for awardees to: 1) develop their own theories of change and monitoring frameworks; 2) develop effective indicators; 3) use and measure common indicators where reasonably feasible; 4) project time-bound targets; 5) report against time￾bound targets; and 6) assess actual results against targets to improve project management. Sufficient explanation of MA’s theory of change would be helpful for awardees to align with it. EQ5 Service Delivery FICCI High Medium Term 63 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 11 Create mechanisms for women to access awards and maximise benefits of the award through: 1) women specific outreach - e.g. connect with women focused incubators / networks (“sheroes”) and other ecosystem stakeholders with large women entrepreneur networks; 2) bringing gender focused stakeholders in as partners - and engage actively with them; 3) considering creating capacity building and awareness training pre award to encourage more women applicants; and, 4) considering creating women specific capacity building or mentoring or peer learning groups. Consider either running gender issue and / or women entrepreneur focused rounds or dedicate a proportion of awards to women-owned businesses and/or awards that propose to primarily benefit women and address gender issues. Awardees that focus interventions / innovation solutions on women should be connected to local groups that specialize in gender equality to ensure that they have a well-aligned gender inclusion lens. EQ6 Service Delivery FICCI Medium Medium Term 12 FICCI should engage in transparent conversations with awardees regarding the amount of their expected funding and the consequences of a reduced budget and work with awardees to ensure they are able to make intended impact with budget limitations prior to award. EQ3 Service Delivery FICCI Low Short Term 13 FICCI and USAID must review applications with a realistic understanding of the varied sector-specific needs for entrepreneurs regarding timelines, especially for health awardees who require longer timelines for regulatory compliance and initiating lengthy clinical trials and provide support accordingly in order to ensure outcome expectations are met. EQ3 Service Delivery MA Partner s Medium Short Term 64 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV This can also be done in line with USAID’s sector￾specific priorities and interests. 14 Either develop in-house capacity to address the needs of start-ups (development milestones, expansion, etc.) through mentoring, or build better linkages with business incubators and investment networks that do this. Ideally, MA would do both: starting with reaching out to 1) its membership to identify mentors who could support the development of awardees, and 2) business incubators and investment networks in order to groom awardees to graduate into these additional support mechanisms. EQ5 Service Delivery FICCI High Short Term 15 Create clear terms and parameters around capacity building funds, including: 1) obligations of awardees to attend FICCI-sponsored capacity building events; 2) flexibility for awardees to use capacity building funds on non-FICCI capacity building, including the approval process as applicable; and 3) reporting obligations for awardees to summarize in what ways capacity building events benefited awardees. Clearly outline to awardees, in writing, criteria for capacity-building fund utilization. Clarify the process by which awardees may request using capacity building funds for non-FICCI sponsored events (e.g., who do they ask, how do they ask it, how long before the event do they need to request, and when will FICCI respond). EQ5 Service Delivery FICCI Low Short Term 65 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 16 Make the selection process gender inclusive by 1) recruiting women experts to the jury, 2) adding a gender specialist to the jury, and 3) including gender criteria in the application form - where awardees detail existing efforts on gender inclusion and planned efforts and basic metrics such as number of women and roles of women within organizations. EQ6 Service Delivery FICCI Medium Short Term 17 MA would be better to demonstrate how fewer inclusive innovations can become commercial successes through robust support than by providing financial support to a wide range of innovations which may not be able to scale effectively. This would mean pivoting the MA service provision to more holistic support system than it does now and deliver direct, robust non-financial support (mentoring, networking, coaching, etc.) to accompany financial support. EQ4 / EQ5 Service Delivery FICCI Medium Medium Term 66 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX 1: STATEMENT OF WORK (including revisions after in-brief) SECTION C-DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK Mid-Term Performance Evaluation of Millennium Alliance Program PROGRAM INFORMATION Program Title: Millennium Alliance Program Start-End Dates: May 29, 2012 - May 29, 2019 Budget: $7.7 million Program Description: The Collaboration Agreement for the Millennium Alliance: An lndia-U.S. Partnership for Global Development Program is attached. INTRODUCTION The Millennium Alliance (MA) is a multi-million dollar India-U.S. public-private partnership that leverages Indian creativity, expertise and capital to source and scale locally developed innovations that address Indian and global development challenges. USAID, in collaboration with partner organizations, links Indian innovators with intellectual and financial supporting five major development sectors: Health, Education, Agriculture, Water and Sanitation and Clean Energy. The fundamental objective of MA is to offer an inclusive platform to identify, support and scale innovative solutions - developed and tested in India that have the potential to provide comprehensive socio-economic, health and environmental benefits to the base of the pyramid population in India and globally. The MA aims to create significant developmental impact at the base of the pyramid population. The MA platform is a joint initiative of USAID, Federation of Indian Chambers of Commerce and Industry (FICCI) and the Government of India's Technology Development Board (TDB) under the Ministry of Science and Technology. MA has grown from three founding partners in 2012 to eight partners including UKAID, ICCO Cooperation, ICICI Foundation, Wadhwani Initiative for Sustainable Health (WISH) Foundation, the World Bank and Facebook. What started as a $7.7 million initiative is now about $25 million. The program duration is from May 2012-May 2019. The MA project has three components that enable FICCI, as the Executive Secretariat of the MA, to establish and operate the MA Platform to form partnerships and programs to identify, support, and scale innovations that will achieve concrete development impacts for the base of the pyramid population in India and globally. The three components are: 1) seeding and scaling innovations to achieve concrete development outcomes; 2) platform and partnership development; and 3) knowledge management and communications. MA provides multi-tier funding. Stage 1 funding is for projects with proof of concept (PoC). The stage 2 is for projects that have demonstrated success at pilot level or small scale stage. Stage 3 is for projects that have credible and rigorous evidence of development impacts at significant scale. So far five rounds of solicitations and 110 awards have been made for funding through the MA platform. 67 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Over the five calls for proposals by MA more than 3000 Letters of Interest were received, more than 110 innovators have been supported either through direct grants, capacity building and opportunities for networking with other investors, 11 partnerships have been negotiated, and $33 million has been leveraged in cash and kind. Twelve of these are global transfer programs that take proven innovations from India to countries in Africa and South Asia. The Millennium Alliance platform has grown over the last three years and bilateral donors like, UKAID, and other philanthropic and corporate foundations like ICCO, WISH Foundation, and the ICICI Foundation have joined the platform. The other partners that have joined as knowledge partners include World Bank, Intellicap and Ankur Capitals. The partnerships have helped MA establish diverse alliances between national and local government counterparts, local companies, and international donors, and helped build trust across sectors and countries. C.3 PURPOSE OF EVALUATION A. Purpose USAID/India seeks to evaluate the extent to which the MA platform has been effective at leveraging Indian creativity and resources to identify and scale innovative solutions that will address development challenges that are a focus of USAID/India and/or the Government of India. More specifically, the evaluation aims to determine whether the goal and objectives of the three components of the MA project are being met. The three components are: a) seeding and scaling innovations to achieve concrete development outcomes; b) platform and partnership development; and c) knowledge management and communications. As part of the evaluation process the Mission wants to establish how the innovations were selected; the success or failure rate of innovations and the extent to which innovations were adapted and lessons were learned from the process; for those innovations scaled up to assess the extent to which they had a positive impact on targeted beneficiaries; to what extent innovations leveraged funding from other sources and how the leveraging impacted the implementation and use of the innovation; and learn lessons from the overall MA project that will guide the design of future approaches. The evaluation will also assess whether MA is an efficient mechanism/approach to manage and achieve the desired results of leveraging public private partnership to support and scale innovative solutions. B. Evaluation Questions The evaluation will address questions in the following major areas: Innovation Leveraging Development outcome Sustainability Management Knowledge management and communications MA Program Design (Internal USAID) Under each major area, the Contractor is expected to assess the current status of the project relative to each question, identify challenges, opportunities and recommend improvements. Please note that the 68 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV evaluation questions will be streamlined and finalized upon consultation between the Contractor and USAID/India and will appear in the final work plan. The final set of questions will depend on information availability, resources, and USAID/India's priority information needs. Below are illustrative questions and sub- questions for each area. Innovations Innovation refers to novel business or organizational models; operational or production processes; or products or services that lead to substantial improvements in executing against development challenges. Innovations help produce development outcomes more effectively, more cheaply, that reach more beneficiaries, in a shorter period of time, and more sustainably. ● To what extent has the MA program succeeded in identifying and selecting innovative solutions for implementation that will support USAID's technical sectors and corresponding activities? ● How innovative are the solutions identified and implemented? Are the criteria used to identify innovations appropriate for the technical sector? ● How was the funding level for design and testing of the innovative solutions decided? ● What is the status of the activities that have failed and what is the cost of the failures? Could these failures have been turned into successes? If so, at what cost? ● How has MA support and/or funding been a catalyst for the organizations to grow and move from one stage to next? ● What types of innovations have been most impactful? (Such as products, processes or business models.) Leveraging Resources that a non-traditional USAID partner brings to a public-private partnership, i.e., the portion not being borne by USAID. These non-traditional resource partners are typically NOT receiving USAID funds. It can be a variety of forms - anything of value that is measured, financial contributions, third party contributions, donated services or property, or intellectual property. It can NOT be audited. ● To what extent has the MA project been able to leverage funding and non-financial value services from various sources like private sector, government, and other bilateral and multilateral donor agencies for additional development outcomes? ● How much funding has the program leveraged directly and indirectly, for what sectors/activities, and from whom? What is the value added? ● How is the leveraged funding being tracked? To what extent have the awardees leveraged FICCI's core strengths (partnerships, networking events) to forge partnerships with private sector corporations and other project awardees? 69 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Development outcome To what extent have the innovative solutions made an incremental contribution to USAID's development results? ● To what extent has the MA project succeeded in the creation and expansion of a platform for seeding and scaling innovations to achieve concrete development outcomes? ● To what extent have the innovative solutions contributed to the improved socio economic, health care, education and environmental benefits to the targeted bottom of pyramid population? ● To what extent has the intervention been cost effective? What is the return on investments (ROI) of the funds on the expected outcome? What impact evaluations have been conducted? ● To what extent has the MA project succeeded in additional fund raising (financial or non financial) to support and scale innovative solutions? Sustainability Sustainability refers to the scale up of the innovative solution with strong potential for commercialization. funded primarily by private sector entities, including social venture capital funds, and/or government resources. The sustainability of market-based innovative solutions will be based primarily on the willingness of BoP consumers to pay for the products and services generated by these solutions. ● To what extent have the innovative solutions become sustainable? ● What criteria have been used to determine that an innovative solution is demonstrating signs leading to sustainability? ● What are the aspects that have generated success where others have not succeeded? Management ● To what extent has the MA platform been effective in terms of management of awards and capacity building of the awardees in technical, managerial, business administration, and sustainability issues? ● To what extent have detailed technical, financial, and management analyses been undertaken prior to selecting each grant awarded under MA? ● How effective is the management of MA project by FICCI? ● To what extent has the input from FICCI played a fundamental role in the success of awardees and the results achieved? 70 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ● How much funding has been distributed by FICCI to awardees and how much has been lost due to unsuccessful innovations? Are there trends in the types of successes & failures? ● How effective is the Monitoring and Evaluation (M&E) plan that is being used by FICCI for the MA project? ● Is MA a good model to replicate to other Missions, and in other parts of India at the levels of the GOI? Elaborate if you agree or disagree. In absence of a FICCI type of organizations in other countries, what could be an alternate arrangement? Gender ● To what extent did the MA platform consider gender issues in the awards in each sector? ● Have gender issues been integrated into award design and implementation? Is gender integration being monitored and evaluated? ● How many awards were made to women-owned or women-led organizations? ● How many awards were to women impact businesses or women impact non-profits? Knowledge management and communications ● To what extent has the MA project succeeded in serving as a knowledge platform for the innovations supported and established with the wider Indian social innovation and development ecosystem? ● How has the MA promoted ongoing learning throughout the innovation life cycle? Have lessons learned fed into and contributed to activities in different sectors in order to capture linkages and synergies? ● Have lessons learned from successful and failed innovations, technical, management and other business processes been shared with new (and other) awardees in an open and transparent approach? ● To what extent has outreach (success stories, best practices, roadshows and other approaches) been used by MA and its awardees to share their successes or failures? What approaches have been used? What's been the impact? USAID/lndia's Design of the MA platform ● Is a Chamber of Commerce an appropriate mechanism for what USAID is trying to achieve via the MA platform? ● How effective has the USAID/India innovation and partnership model been to increasing the achievement of results under the Mission's development objectives? ● How effective has the Strategic Advisory Committee been in managing the MA mechanism? 71 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ● How effective has the USAID technical team's participation been in the technical evaluation process of potential awardees under the MA platform? ● Is the funding of a portfolio of innovations that are at different stages of development the most effective approach? ● Is multi-sectoral a better approach than a more focused approach on one or two sectors for selecting interventions and will one get better results over time and be more efficient to manage and generate partnerships and leveraging? ● How effectively has USAID/India been at adapting and learning continuously through the implementation of MA? ● What would be an effective approach for M&E and reporting for this mechanism? ● What would be the most appropriate indicators to use that link successes and their impact (or failures) to the relevant USAID technical sector or activity? C. Audience and Intended Uses The audience of the evaluation report will be the USAID/India Mission, and specifically each technical office but also the support offices; the Asia Bureau; and the Mission's implementing partners. USAID will use the report's lessons learned to inform the future of the MA platform, and the design of new projects and amendments to existing projects and /or activities. C.4 EVALUATION DESIGN AND METHODOLOGY The evaluation will primarily focus on agriculture, health, water and sanitation, education and energy activities for which awards were made up to phase 4 since 2013 (Phase 5 was awarded recently). Awards that fall under the category of "other" will be reviewed and considered based on their relevance to USAID's strategy and the lessons learned. Evaluation methods, including information on evaluation design and data collection, are presented below. The selected Evaluation Team will need to present a well-thought methodology for answering the evaluation questions while keeping in mind USAID's evaluation criteria. The Contractor is expected to propose various innovative methods to assess the partnership and innovation aspects in the evaluation. The evaluation will include qualitative and quantitative data collection. The work plan should give a description of specific data collection methods such as individual or focus groups and structured interviews, checklists and other tools. A SWOT analysis will be undertaken to give a perspective of the relative strengths, weaknesses, opportunities, and threats faced by the interventions to inform future recommendations. Sampling: The evaluation will conduct desktop review of documents to identify a sampling frame, purposively select interviewees, and create a stratified sample with accurate representation of each technical sector within the MA portfolio. The sample will include a minimum of 25% of the total activities. The priority sectors are the interventions on health and Water and Sanitation. In addition to the activities 72 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV implemented in India, the evaluation will also include four activities that were supported by MA in India and subsequently have been transferred and implemented in Afghanistan and Nepal. The following criteria will be used to select the sample for each of the sector. ● Dollar value of the award ● Stage of the award ● Geographical location of the award ● Difficulty or uniqueness of the proposed innovative approach ● Applicability of the innovative approach to support/enhance achievement of results under a specific technical sector ● On the qualitative methodology, the evaluation may include one or more of the following: theory of change, semi-structured interviews, key informant interviews (KII), ● Focus Group Discussions (FGDs), case studies and most significant change methodologies. The Evaluation Team will be provided relevant background documents, to include: ● USAID ● USAID/India Country Development and Cooperation Strategy ● USAID/India Development Objective 3 Project Appraisal Report ● Millennium Alliance Agreement ● MA Monitoring and Evaluation Plan ● MA Award ● Baseline data ● Project Work Plans ● Quarterly Reports ● Annual Reports ● Budget and financial reports ● Project performance data ● Other MA-related documents The questions included below are illustrative and not inclusive. The Evaluation Team will review documentation provided by USAID and the MA awardees, and any relevant secondary research they collect (especially on partnership approach). An instrument will be developed to codify and organize data from the document review for analysis according to the evaluation questions. For qualitative data specifically, the Evaluation Team will need to ensure key informant interviews and/or focus groups are recorded and detailed notes will be taken during interviews. Recordings will be used as a means to verify the notes if there is a discrepancy. The Evaluation Team will explore whether they are using any qualitative data analysis software like Dedoose or NVivo. REPORTING REQUIREMENTS AND DELIVERABLES The following are deliverables required of the Contractor under this Task Order: Initial meeting with USAID/Program Office Director, Senior M&E Specialist/COR of the evaluation 73 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV contract, and USAID/technical office directors and/or their COR/AORs to discuss the evaluation. In addition, logistical details such as number and location of interviews, interview and meeting scheduling, etc. will be discussed. Work Plan: The Evaluation Team will provide to USAID/India's technical and Program Office Director and the COR a detailed Work Plan including a projected timeline, a detailed description of the evaluation methodology and data collection and analysis methods which will be used (including draft interview questions and data collection tools). This Work Plan will be reviewed and approved by the COR within 5 business days of the receipt of the draft. Site Visits and Interviews: The Evaluation Team will conduct a thorough review of the Program through site visits and interviews. Site visits will be planned taking into consideration factors like geographical diversity, representation of various partners, and the scale of the interventions. Mid-term Briefings: The Evaluation Team Leader will meet with the COR and representatives from the technical offices and Program Offices to provide an update on status, key issues affecting project implementation, and any initial findings/tentative conclusions. In addition, the team will provide to the COR weekly email updates on status and key issues in the alternating weeks between the semi-monthly briefings. Debrief Presentation: The Evaluation Team will make a PowerPoint presentation of preliminary findings and conclusions to USAID/India and key stakeholders on the main findings of the evaluation. Draft Report: The contractor will submit a draft written report in English within forty days of the debriefing. The report should clearly describe findings, conclusions, and recommendations, and should incorporate comments and questions raised. Presentation: An electronic version of the report will be provided to the COR for dissemination among relevant Mission staff for review and comment. USAID will provide comments on the draft report within two weeks of submission. Submission of all raw and processed data, which becomes the property of USAID/India. Final Report (due within twenty-five calendar days after receiving written comments from USAID). The team will submit a Final Report in English that incorporates all Mission comments and feedback. The format will include an executive summary, table of contents, methodology, findings, and recommendations. See below for an outline of the final report. Both an electronic version and five copies of a written version will be provided to the COR for dissemination among relevant Mission staff and stakeholders. The evaluation COR will submit one electronic copy of the Final Report to the Development Experience Clearinghouse at http://dec.usaid.gov after final approval. The Final Report will have the following contents: ● Table of Contents (1 page); ● Executive Summary - concisely state the most salient findings and recommendations (2 pages); 74 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ● Introduction-Purpose, audience, and synopsis of task (1 page); ● Background - Brief overview of development context and problem, USAID strategy and activities implemented in response to the problem, purpose of the evaluation (2-3 pages); ● Methodology - Describe evaluation methods, including constraints and gaps (1 page); ● Findings/Conclusions/Recommendations - For eight main evaluation questions (up to 4 pages per evaluation question); ● Issues -Provide a list of key technical and/or administrative issues, if any (1-2 pages); ● Success Stories- Individual success stories which illustrate how USAID project activities have improved lives of people at the bottom of the pyramid ● Annexes - Document the evaluation methods; schedules; bibliography of documents reviewed; tables with all data, list of meetings, interviews and focus group discussions, and SOW - all materials should be succinct, relevant and readable. ● Recommendations should also state any challenges to being implemented and why, as well as how to overcome them. C.6 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not, and why. Evaluation reports shall address all evaluation questions included in the scope of work. The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, Evaluation Team composition, methodology, or timeline need to be agreed upon in writing by the COR. Evaluation methodology shall be explained in detail, and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. Evaluation findings will assess outcomes and impact on males and females. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people's opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. Sources of information need to be properly identified and listed in an annex. Recommendations need to be supported by a specific set of findings. Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. C.7 EVALUATION MANAGEMENT The Contractor shall be responsible for obtaining visas and country clearances for travel for the consultants if required. 75 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV The Contractor shall be responsible for coordinating and facilitating assessment-related team planning meetings, field trips, interviews, and other meetings in conjunction with USAID and the Program team. The Contractor shall be responsible for international and in-country logistics as applicable such as transportation, accommodations, communications, office support, etc. The Evaluation Team will receive support from USAID/India in selecting priority organizations and places to visit during the evaluation. The Evaluation Team is expected to schedule interviews or other modes of data collection with key stakeholders, though USAID/India can assist in providing contact information. Additionally, USAID/India will provide a letter of introduction to the Evaluation Team to assist in the scheduling of interviews. SECTION D - PACKAGING AND MARKING Reference Evaluation Services IDIQ Contract No. AID-486-1-14-00001, as modified. All Clauses from the referenced IDIQ apply to this Task Order. Monitoring and Compliance of Branding and Marking Plans The approved Branding Strategy and Marking Plan are part of the award. USAID is responsible for monitoring compliance based on the negotiated and approved Branding Strategy and Marking Plan. All Public communications and commodities financed by USAID should be branded with USAID identity. If the TOCOR or the monitoring team notes any cases of non-compliance, the issue should be reported in the report. Major or repeated non-compliance with the requirements might trigger the standard suspension and termination procedures. SECTION E - INSPECTION AND ACCEPTANCE E.1 Reference Evaluation Services Indefinite Quantity Contract No. AID-486-1-14-00001. All Clauses from the referenced IDIQ apply to this Task Order. SECTION F - DELIVERIES OR PERFORMANCE Reference Evaluation Services Indefinite Quantity Contract No. AID-486-1-14-00001. All Clauses from the referenced IDIQ apply to this Task Order. PERIOD OF PERFORMANCE The period of performance for this Task Order is 140 calendar days. The performance period for the evaluation starts on January 1, 2019 and concludes on May 21, 2019. F.3. DELIVERABLES Delivery Schedule: MANAGEMENT OF TASK ORDER 76 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV The Contractor shall provide Task Order management necessary to fulfill all the requirements of this Task Order. This includes cost and quality control under this Task Order. All work under this Task Order shall be completed within the period of performance. SUMMARY REPORTING MATRIX (see Section F.7 for submission requirements to Development Experience Clearinghouse) PLACE OF PERFORMANCE The place of performance under this Task Order is India as specified in the Statement of Work. AIDAR 752.7005 SUBMISSION REQUIREMENTS FOR DEVELOPMENT EXPERIENCE DOCUMENTS (SEP 2013) Contract Reports and Information/Intellectual Products. Within thirty (30) calendar days of obtaining the Contracting Officer Representative's approval, the Contractor must submit to USAID's Development Experience Clearinghouse (DEC) one copy each of reports and information products which describe, communicate or organize program/project development assistance activities, methods, technologies, management, research, results and experience. These reports include: assessments, evaluations, studies, technical and periodic reports, annual and final reports, and development experience documents (defined as documents that (1) describe the planning, design, implementation, evaluation, and results of development assistance; and (2) are generated during the life cycle of development assistance programs or activities.) The Contractor must also submit copies of information products including training materials, publications, databases, computer software programs, videos and other intellectual deliverable materials required under the Contract Schedule. The following information is not to be submitted: Time-sensitive materials such as newsletters, brochures or bulletins. The Contractor's information that is incidental to award administration, such as financial, administrative, cost or pricing, or management information. Within thirty (30) calendar days after completion of the contract, the Contractor must submit to the DEC any reports that have not been previously submitted and an index of all reports and information/intellectual products referenced in paragraph (a)(l) of this clause. (b) Submission requirements. The Contractor must review the DEC Web site for the most up-to- date submission instructions, including the DEC address for paper submissions, the document formatting and the types of documents to be submitted. The submission instructions can be found at: https://dec.usaid.gov. Standards. Material must not include financially sensitive information or personally identifiable information (PIT) such as social security numbers, home addresses and dates of birth. Such information must be removed 77 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV prior to submission. All submissions must conform to current USAID branding requirements. Contract reports and information/intellectual products can be submitted in either electronic (preferred) or paper form. Electronic documentation must comply with Section 508 of the Rehabilitation Act of 1973. The electronic submissions must consist of only one electronic file, which comprises the complete and final equivalent of the paper copy. In the case of databases and computer software the submissions must also include necessary descriptive information, e.g., special backup or data compression routines, software used for storing/retrieving submitted data, or program installation instructions. Electronic documents must be in one of the National Archives and Records Administration (NARA)- approved formats as described in NARA guidelines related to the transfer of permanent E-records. (See http://www.archives.gov/records-mgmt/initiatives/transfer-to-nara.html). (3) Essential bibliographic information. Descriptive information is required for all Contractor products submitted. The title page of all reports and information products must include the contract number(s), Contractor name(s), name of the USAID Contracting Officer's Representative, the publication or issuance date of the document, document title, (if non English, provide an English translation of the title), author name(s), and development objective or activity title (if non-English, provide a translation) and associated number, and language of the document (if non English). In addition, all hard copy materials submitted in accordance with this clause must have, attached as a separate cover sheet, the name, organization, address, telephone number, fax number, and Internet address of the submitting party. 78 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX 2: DATA COLLECTION INSTRUMENTS AND CONSENT KEY INFORMANT INTERVIEW INSTRUMENT Background Information: ● Name/Title ● Professional Affiliation 1. What has been your role with the Millennium Alliance Program? How long have you been involved? 2. Can you talk us through how you came to engage with the platform? What interested you in partnership? 3. How do you define (inclusive) innovation? How does this sit with the MA platform? What do you think of MA platform’s innovation focus? 4. What are your observations about how the MA platform is currently operating? (locations, sectors, expansive models, stages of innovation??) Opinions about their involvement in the BOP space. What works and what does not work so well? 5. Overall what have been some of the major contributions that resulted from the MA platform? How has it contributed to strengthening the ecosystem? 6. Has the role of the MA platform in India’s innovation ecosystem changed over time? If so, how? And what is its role now? What should its role be going forward? 7. Does the MA platform have a strong brand value? Has it changed over time? Does it promote and communicate what it does effectively? Is the platform well respected? 8. How would you describe the capacity of FICCI to lead/ run the program and build awardee capacity, and help awardees scale? What is their strength? Weakness? 9. What is your sense of how partners work together in the broader network? Does FICCI draw on the network of partners? What is working well? Is there anything you feel that could be strengthen here? Anything that you feel is not so useful?? 10. If you completely redesigning this MA platform, based on your experience, what would be your top priorities? 11. What are your thoughts on how MA can further engage the private sector? (partnership development, marketing, communications, private partners). 12. Is the MA platform in a position to become self-sustainable in future? If so, how? Should it continue to focus on grant finance or other forms of financial and non-financial support? 79 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 13. Is the MA platform actively encouraging women-led or women focused entities? What are the key considerations that you see for the engagement of women-led or women owned organizations in innovation? Should they do this differently? For Donor Reps - DfID: A. How well does the collaboration between donor partners function? Could you talk us through how you engage and the role of FICCI in donor engagements? B. We understand that each donor has different M&E requirements and data points with respect to awardees. Do you think it is possible to streamline these, or do you have stringent criteria you need to adhere too? C. What do you see as opportunities for donors to strengthen the platform? D. Do you see a need for other kinds of partners to be brought on board? If so, which ones and why? For (ex-)Government Officials: A. Do you see opportunities for the MA platform to engage more deeply with other parts of the GoI and / or State Governments that supports inclusive innovation and development? B. How well does the collaboration between donor partners, the MA platform and the GoI partner function? Could you talk us through how you engage and the role of FICCI in donor and GoI engagements? C. Do you see a need for other kinds of partners to be brought on board? If so, which ones and why? AWARDEE INSTRUMENT Background 1. For what purpose did your organization seek grant funding through the Millennium Alliance Program? Can you tell us more about your innovation? What was the key development problem that your innovation sought to address (health/education/WASH/ag/clean energy)? Has this ever been done before? 2. Who are the key users, and how will they access this innovation?*** 3. Why did you decide to pilot/market/ roll-out your innovation in the area you selected and to the population you identified? What kind of considerations did you have for selecting this site? What are the strengths and weaknesses of this location and approach? *** Application and selection process 1. Tell us how you decided to apply for a grant with MA? How did you hear about the platform? Had you applied before? Did you have any connection to FICCI before? 80 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 2. Please walk us through your application process and the different stages (probe due diligence, pitch). What kinds of questions were asked? What was the expertise of the panel like? 3. In total, how long did the application and selection process take? 4. How have you sought funding for your innovation before, through grants or other means? How would you describe the application process for this grant compared to other grantmaking platforms? Was there anything in FICCIs process that worked particularly well? Anything that didn’t work so well?*** Contracting, Award and Start up 1. Can you tell us how you learned of your innovation being successful and selected for award? How much did you request, and how much did you receive for your grant? 2. If you received less than you were expecting, was this communicated to you? Were you able to find additional funding to make up the difference? Did you have to make adjustments to your project based on the amount you received? Based on your knowledge, how was the amount of your grant determined? 3. In total, how much time passed between when you submitted the application, received the award, and received the first disbursement/tranche of funds? 4. How did you receive your disbursements, and under what conditions (reimbursements, lump sum, tied to deliverables or actions/milestones)?*** 5. Did you have taxes applied to your disbursements? 6. Do you have any ideas for improving FICCI’s application and award management process?*** Reporting and M&E 1. As an awardee, is your organization expected to provide regular activity reporting to MA Program? What about financial reporting? How often? Have you done that? 2. What is the format required for activity and financial reporting, and how was this communicated to you? Who is your main point of contract for reporting at FICCI? 3. Do the reporting expectations differ than those from other organizations under which you’ve received funding? What are the key differences, if any, between the MA reporting process and others you have experienced? 4. In terms of monitoring results, how did you decide on milestones? How you currently track progress of your activities? What are the dates of your award, and were they realistic to accomplish the milestones in your workplan? Why or why not?*** 5. Do you have any challenges in reporting on key indicators? Why or why not? 81 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 6. What are your overall feelings about how reporting and monitoring are done under MA? What is working well? Is there room for improvement?*** Growth and Capacity building 1. Besides the grant funding, can you tell us about other services that FICCI has provided your organization through the MA Program? (Probe capacity building for reporting, M&E, product development, marketing, branding etc) how would you describe this support? What was most useful? Was there anything not useful? *** 2. Has FICCI every asked awardees what their priorities are for capacity building and growth? 3. What are your organization’s priorities in terms of capacity building? How would you describe the capacity in your organization before receiving a MA grant? How would you describe it now? 4. (If under Round 3-4) How did you decide what you wanted to spend the capacity building funds on? 5. Has your organization participated in any of the Roadshows, expositions, or other events sponsored by MA? Was there any results from the event(s) for your organization? *** Utility, Sustainability and Scale of innovation 1. How would you describe the overall developmental benefits created by your innovation for people at the “bottom of the pyramid”? If you have rolled this product out to users, what kind of feedback have you received? If you have not rolled this product out to users, what are the reasons for this?*** 2. Have you applied for patents, trademarks, IP, copyrights or do you have the intent to do so? If so, what has this process been like for you? Has anyone helped you?*** 3. What other sources of financing have you explored to support your innovation’s development? Has FICCI or its partners assisted in your ability to obtain financing?*** 4. How is cost or affordability considered with marketing or producing your product? What about accessibility?*** 5. Do you have an interest in scaling up this innovation, and what does that mean to you? (more people, more sites, outside of India, etc). *** 6. Have you attempted to scale up? If yes, what has been the outcomes? If no, what do you feel will be necessary to scale up this innovation? How might you do that? Do you feel that your organization has what it would need to scale up? Why or why not? (probe materials, sales, financing, research)*** 7. What do you think the long-term effect of this project will be? At what levels do you think the effects will be most felt- community, institutional, etc.?*** 82 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Knowledge Mgmt & Communications and Relationship building 1. From your experience, compared to other such grant programs, overall how would you rate MA’s communications and outreach? Who was your main point of contact, and was it easy to reach them?*** 2. Can you tell us if your organization has developed useful relationships with other organizations or companies by participating in networking, roadshows, or other events sponsored by MA? *** 3. Were you provided any opportunities to meet other innovators? What about any of the partners or members?*** 4. How do you share information about your innovation? Who is your target audience? 5. Has being part of the MA platform contributed in any way to the exposure your innovation or organization has had? Why or why not? *** Gender 1. Is your organization female-owned? 2. If you are women owned/operated, have you have faced any challenges in being a woman entrepreneur? Did MA assist you in overcoming those challenges?*** 3. During the application and selection process, did MA provided any special considerations for women-owned or led businesses? 4. Do you think there were any issues or barriers in the MA grant process specific for women? If so, what are they? 5. Did MA state any particular interest in innovations that benefited women? Was gender integration a priority for you in developing this innovation? Do you think gender integration is a priority for FICCI for this platform? 6. Do you collect any data relating to male and female end-users? Why or why not? Is this of interest to your project? 7. What do you see as the challenges and opportunities facing women, ethnic minorities and other vulnerable groups in this sector (clean energy, health, agriculture, WASH, education)? Do you think your innovation has helped women to address some of these challenges? If so, how? If not, why do you think this? Overall 83 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 1. What, if any, are the most notable ways in which the MA grant has been useful for your innovation?*** 2. Are there aspects of the MA grant program that could be improved? If so, what are they? 3. Have there been any challenging or problematic aspects to managing or working with MA? What are those? How have they been addressed?*** 4. What recommendations would you make to improve the MA Program?*** 5. In general, what do you think can contribute to an innovation funded by MA being successful or unsuccessful? Anything in particular to an innovation in [your sector]? Is there anything that you think may make an innovation less successful under the MA platform? Why?** Thank you. *** Denotes questions to be probed more in-depth for awardees designated as failures. FICCI INSTRUMENT Introductory Questions 1. What has been your role in the MA Program? 2. Are there specific people who cover any specific sector(s) under this Program? Indicate if there is a specialist for Agriculture Clean Energy Education Health WASH Other All None 3. How long have you been involved with the Program? 4. Was geographical focus a criteria for selection of the awards? Why majority of the awards are specific to some states, particularly concentrated in the Southern and Western states of India? Q1 1. How would you describe the overall benefits that have been generated from FICCI’s support of this program? 2. How does FICCI keep track of the results and outcomes of this Program? 3. What criteria does FICCI use to select awardees? Is geographic location or focus a criterion for selection? 4. Please describe how the Evaluation Panel functions to select the awardees? 5. Can you describe at what stage most of the innovations are at when they are selected for an award? 6. How does FCCI report results to USAID? And how often? 84 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 7. In your opinion, which of your MA grants created the greatest developmental impact? How? Q2 1. Which other organizations, bilateral or multilateral donors, GOI agencies, and private sector, provided additional funding directly to FICCI that it has channeled the MA Program? 2. Have these donor funds been used for specific purposes within the MA Program? 3. Over the life of MA what has been the total amount of such funding? Rps/$ 4. Have awardees been able to obtain private sector financing to support the scaling up of their innovations? Q3 1. How has FICCI worked to provide scaling up services to awardees so that they have been able to develop business plans, including marketing strategies? 2. Does FICCI currently have the staff and funding needed to carry out these activities? 3. What has FICCI learned in carrying these activities? 4. In your opinion, how effective has FICCI been in helping awardees scale up? 5. What other services does FICCI provide to awardees? Have these been consistently successful? 6. In your opinion, what are the factors that have led to awardee success? Q4 1. How many of the awardees to date do you consider to be truly sustainable? 2. Can you name some specific awardees that have been particularly successful in terms of development impact in India? What have they achieved? 3. Can you estimate the percentage of the 86 current awardees that have been able to commercialize and/or distribute their product? ____% 4. How do you monitor awardees success? Do you do this even after the award period is over? Q5 1. In your opinion, how effective do you consider that FICCI has been as a manager of the MA awards process through the five rounds of competition? 2. Has FICCI recognized any challenges or problems with the awards process it has used? 85 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 3. At what point are the innovations generally funded? (Idea stage Protype Created Prototype Tested Prototype Product already being commercialized, but needing volume or marketing expansion) 4. In your opinion it is better to award the awardees at a certain stage of innovation or development? 5. What was the total amount of MA funding given out to awardees as of December 31, 2018? Rps 6. In each round, what was the total number of applicants and the percentage of the applicants that received awards? R1 ____% R2____% R3 ____% R4 ____% R5 ____% 7. How does FICCI determine the level of funding to provide to awardees? 8. What is the effect on FICCI’s awards process if the applicant has previous similar awards from other donors? 9. After the award, what challenges do the awardees typically face? 10. Do USAID Mission personnel participate in the awards process? Is that useful? 11. How does FICCI define awardee failure in the MA? 12. What is the failure rate of awardees financed by this Program? 13. How might the awards process be improved to select awardees that would have a greater probability of success? 14. Has FICCI already made some of changes to improve the process? What were they? Q6 1. To what extent did the MA selection process address gender considerations? How were gender considerations looked at in terms of innovation? 2. In your opinion, how effective has FICCI been in encouraging female innovators to participate in the grant process? How represented do you feel women-owned organizations have been in the MA program? Why? Are there specific criteria for woman owned businesses when it comes to MA applications? 3. Are there specific challenges for women-owned businesses in applying for grants under mechanisms such as MA? If so, what might those be and why might those challenges exist? If not, why do you think this is? (probe: cultural restrictions, financing, accessibility, communication) 4. Are awards expected to include specific gender-responsive indicators as part of their monitoring plan? How is progress towards meeting gender-specific targets measured by awardees? (probe: for women owned and non-women owned organizations)? Any specific gender considerations for reporting? 86 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 5. Which woman-owned or led awardees been successful in scaling up their activities? If so, in what ways? If no, why might this be? 6. Are there any awards and innovations that you feel have been particularly beneficial for women? Why do you think these were particularly successful? 7. For those awardees whose innovations were targeted to benefit women, did they face any challenges with implementation? Please explain. Q7 1. How has FICCI used its knowledge platform to encourage even more innovation in India? 2. How many networking events or workshops has FICCI sponsored for MA awardees? Have some awardees been able to form successful partnerships with other organizations? At these networking events or workshops? How many? Which ones? 3. How has FICCI promoted MA through its local chapters? Q8 1. In your opinion, what are the aspects of the MA program that continue to be challenging or problematic? 2. Do you see a strong future for MA? 3. In future rounds of MA awards, are there recommendations that you would make to improve the entire process? 87 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV GROUP WORKSHOP INSTRUMENT AGENDA: Supporting Inclusive Innovation in India Monday, March 11, 2019. 13:30 to 16:30 India International Centre 40, Max Mueller Marg, Lodhi Gardens, Lodhi Estate, New Delhi, Delhi 110003 INTRODUCTION SESSION 1: TODAY Introduction to questions ● What are the main trends in inclusive innovation landscape right now? ● To what extent does a focus on supporting inclusive innovation lead to the improvement of the lives of people living under the poverty line? What are the advantages and disadvantages of this focus? ● What is missing in the inclusive innovation ecosystem in India today? What are the main contemporary challenges for the inclusive innovation ecosystem? Small group discussion: identify your five best answers to each question Report back TEA BREAK SESSION 2: FUTURE Introduction to questions ● What would you like the inclusive innovation ecosystem to look like in India in 10 years? ● What types of support models and mechanisms are required to enable inclusive innovation in India? ● What are the obstacles to further strengthening the inclusive innovation ecosystem in India? Small group discussion: identify your five best answers to each question Report back PLENARY: GETTING FROM HERE TO THERE ● How should these support models and mechanisms relate with one another? ● Which are the main groups who could make this happen? What should their roles be? Which of them are already at the table, and which are not? Do they exist yet, or are now institutions required? If not, why are they not there? ● How inclusive is the inclusive innovation ecosystem? Are there any groups of people who are benefiting more or less than others from enhancing innovation in India? How to these new models and mechanisms improve the inclusivity of innovation in India? How to engage these groups in the greater innovation ecosystem? 88 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV INTRODUCTION TEXT AND RESPONDENT CONSENT 13/2/2019 Name, Title Name of Organization/Awardee Dear XXXXXX, In 2012, the United States Agency for International Development (USAID) Mission in India launched the Millennium Alliance (MA) Program, its flagship innovation platform that leverages Indian creativity, expertise, and resources to identify innovation, test promising solutions, and scale effective interventions and products. In the past five years, MA has supported over 100 innovators at all stages of innovation across health; education; agriculture; clean energy; and water, sanitation and hygiene (WASH). USAID/India has recently contracted Social Impact (SI), a U.S. company specializing in monitoring and evaluation studies, with an Indian subcontractor, Athena Infonomics (AI), to conduct a performance evaluation designed to examine project effectiveness and challenges, and document best practices and lessons learned during the course of this program. The evaluation will look at the systems and processes for application, selection and grant disbursement, along with capacity building, resources and support provided to awardees and any ground-level impact from your MA grant. This evaluation will consider how these lessons can be applied to future innovation grantmaking activities in USAID/India’s key priority sectors. This is not an evaluation of your organization, but of the MA program and its work towards innovation and development only. {Organization/Awardee Name} has been one of many recipients of this funding and has been randomly selected to take part in this evaluation. We would like to request your participation in an interview with our evaluation team If unable to meet in person, we can call you via Skype or over the phone. The interview is anticipated to take 60-90 minutes of your time. Please advise at which time you will be able to meet with the evaluation team or if you have an alternative preferred date. Additionally, if you are unable to meet with us it would be wonderful if you could provide the names and contact information of up to 2 colleagues within your organization also closely involved with MA. Your experience and observations of the MA program will be highly valuable to this learning process and contribute greatly to the evaluation effort. Sincerely, The Social Impact Evaluation Team 89 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV DRAFT INFORMED CONSENT FOR KII AND GROUP INTERVIEW Interview date and location:_________________________ Interviewer: ____________________ [Where appropriate] Agency/Company:__________________________ [Where appropriate] Title(s):____________________________ Name(s):______________________________ Sex: ______ Consent Statement: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is in India to conduct an independent evaluation about the Millennium Alliance (MA) program, which is funded by USAID. We would like to conduct a brief interview today to learn about your perspectives on the program’s [your award’s] performance, outcomes and lessons learned. This information will be used in a final report for USAID that will be publicly available. It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any questions if you do not feel comfortable answering. If you decide to take part, your responses will be kept strictly confidential. This means that your name will not be mentioned anywhere in the report, and will not be provided to anyone, including at your company or place of work. Any personal information we collect today will be stored in a secure computer file. The objective of this research is to improve the performance of projects like MA and to ensure their success in the future for innovators and partners. The information may be used by other organizations as well. It is important that you know that we are not evaluating you or your organization, only the MA program as a whole. The interview is expected to take up to 1.5 hours. Do you have any questions? You may ask questions at any time. If you have questions or concerns about the research after we leave today, you can contact [insert Evaluation Team name and contact information for each sub-team]. By saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation and voluntarily consent to participate. Will you participate in this interview? You may answer yes or no. ◻ Yes, I will participate ◻ No, I will not participate 90 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV GROUP WORKSHOP INVITATION 5/3/2019 Dear Innovation Thought Leader, In 2012, the United States Agency for International Development (USAID) Mission in India launched the Millennium Alliance (MA) Program, its flagship innovation platform that leverages Indian creativity, expertise, and resources to identify innovation, test promising solutions, and scale interventions and products. In the past five years, MA has supported over 100 innovators at all stages of innovation across health, education, agriculture, clean energy, and water, sanitation and hygiene. USAID/India has recently contracted Social Impact (SI), a U.S. company specializing in monitoring and evaluation with an Indian subcontractor, Athena Infonomics (AI), to conduct a performance evaluation designed to examine project effectiveness and challenges, and document best practices and lessons learned during the course of the MA program. This evaluation considers how these lessons can be applied to future innovation grantmaking activities in USAID/India’s key priority sectors. Social Impact would like to invite innovation thought leaders to share your perceptions regarding efficiency and effectiveness of innovation in India in an informal and participatory setting in order to generate ideas about what types of support innovators will need in the next few years. The workshop will aim to identify key learning outcomes, to build on successes, and identify areas for future investment and support for innovation in India. The workshop will be held on Monday, March 11 2019 from 1:30-4:30pm at the India International Centre 40, Max Mueller Marg New Delhi - 110003. We encourage you to arrive at 1:15pm. Your experience as a thought leader in enabling innovation is highly valuable to this learning process and contribute greatly to the evaluation effort. We hope to see you there. Please RSVP to r.myers@dala.institute by Friday, March 8, 2019 with the subject “INNOVATION WORKSHOP”. Sincerely, Rodd Myers, PhD Social Impact Evaluation Team Leader 91 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX 3: RETURN ON INVESTMENT METHOD The Evaluation Team, and specifically, the RoI specialist, supplied some rudimentary RoI indication in this report. In this Annex, the methods used are elaborated so that readers of this evaluation may further develop some tools to systematically measure RoI within MA. RoI is a concept borrowed from the corporate sector and, in this case, applied to a development initiative. It therefore fits better with some kinds of awardees than others and may not apply as well to non-profit awardees as to corporations. Further, although many of the awardees are for-profit entities, they are ostensibly pursuing development outcomes as a condition of participating in MA. Therefore, the Evaluation Team’s approach to RoI is more aligned with the concept of Social Return on Investment, which not only considers revenues and costs, but non-monetary returns. With this in mind, and given that the Evaluation Team was originally tasked with calculating RoI for MA as a whole, for which there was neither sufficient data nor time to collect sufficient data, this report provides some examples of RoI and elaborates the methods used. The RoI methods described here are built around the data availability, or possible data availability with minimal disruption, within MA. The Evaluation Team recognizes that in order to conduct a complete RoI analysis, especially given the retroactive application of data collection, several months of work and field visits would have to be completed. This is because (i) data are not readily available, (ii) awardees are not under obligation to produce said data, and even if they were, (iii) the ways in which data would be tabulated by awardees could vary widely. The Evaluation Team recognizes USAID’s interest in RoI, and also cautions USAID that comparisons among sectors and awardees is fraught with variables that may be significantly different in different contexts, as this report highlights in the Development Outcomes section, using some examples. Even simple metrics such as cost per person reached can be tempting to compare among projects. However, the quality of reach and development outcome of each reach can be highly variable and futile to compare. For example, how can agricultural production yield increases compare with an avoided neonatal death? USAID must also consider its comparative advantage in light of other donors’ activities in any given context, as well as myriad other factors (e.g. Agency strategies, Congressional earmarks, etc.). Finally, the Evaluation Team excludes future costing and returns assumptions from its analysis. This is because the great number of assumptions that must be made in such a projection, in most cases, render the projection unuseable. This annex, therefore, presents a method for how RoI can be used in the context of MA, with some degree of meaning and reliability if appropriate resources were applied to the intensive quantitative and qualitative data collection that would be required. The first section outlines some basic RoI metrics and assesses their availability, or prospective availability within MA. The second section addresses the complex notion of attribution. METRICS AND DATA AVAILABILITY IN MA The following table outlines some rudimentary variables that could be used for RoI calculation in the MA, suggesting the level of availability of each variable. 92 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV No. Variable Data Source and Availability 1 Direct Costs Total amount allocated to each beneficiary available from FICCI 2 Indirect Costs Aggregate overheads available, apportioned equally across Awardees 3 Reach (no. beneficiaries) Available for most Awardees (>90%). However, the depth of reach and engagement varies across the portfolio (e.g. access to finance vs. neonatal breathing support device). Also, the vulnerability of the beneficiaries is heterogenous. 4 Institutional partnerships Details of partnerships partially available from Quarterly reports. 5 Revenue generation 60% of awardees have a revenue model, however very few have reported revenue (needs to be examined if this is due to reporting or gestation issues) 6 Reserves and surpluses Not available. awardees do not report Profit and Loss 7 Leverage multiplier Available for some awardees (outside of the program contribution from various parties) 8 Savings in fund raising costs Terms of funding (for debt and equity) are not available from the portfolio monitoring platform 9 Direct jobs created Available 10 Indirect jobs created Available for ~ 70% of awardees, but no validation These variables can be tabulated to, for example, track the overall RoI for an award, or to compare changes in RoI for an award over time. The data are currently available to varying extent from awardees. However, tabulating RoI variables is not particularly telling for awardees whose revenues are from more than only MA. It cannot, therefore, be assumed that 100% of the returns from the innovation can be attributed to MA. Therefore, an attribution calculation must be applied. ATTRIBUTION Attribution is a significant challenge in calculating RoI for MA. Imagine an innovation that receives an MA award. If 100% of the innovation is funded by MA, MA could reasonably claim returns are attributable to the award (although this says little of whether or not the innovation could have generated more returns should it have received alternative funding). But most awardees receive funding by multiple sources, including the paid or sweat equity of the awardees. As the table below shows, there are several ways of understanding attribution each with its own advantages and disadvantages. 93 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV No. Aspect Description Advantages Disadvantages 1 Complete Attribution Assumes complete credit for outcomes (current approach in reporting) Simple to compute Does not consider additionality over and above autonomous outcomes 2 Qualitative Inputs Places the onus of attribution on the awardee, usually on a percentage or scale Includes awardee impressions of attribution Since it entirely relies on subjective inputs from awardees, could have biases (positive or negative) 3 Qualitative + Investment Prorata Treats award as Equity, and prorates outcomes to the award on a base of total equity value, adjusted to awardee inputs on relevance Objective, provides a proportional view of additionality over and above autonomous outcomes One size approach across portfolio might mask sectoral nuances 4 Qualitative inputs + prorata investment + control adjustment Considers difference in outcomes between similarly structured firms in the same business across treatment and control Fairly objective since it also incorporates sectoral nuances and makes very few assumptions (other than comparability of entities) A relevant and observable control group might not be always available Source: Evaluation Team The Evaluation Team considers Aspect 4 the most robust and therefore used these considerations in understanding RoI. The table below suggests how attribution can be factored into RoI calculations. The table provides some examples of how many returns from an innovation could reasonably be attributed to MA years on year. No. Head Year 1 (Col A) Year 2 (Col B) Year 3 (Col C) 1 Value of equity (from balance sheet) 60 100 160 2 MA Grant for the year (start of year) 0 20 10 94 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 3 Other Grants/ equity infusions in the year (start of the year) 0 10 30 4 Total additional funding (equity and grants) 0 30 40 5 Implied growth rate of equity 11% (A1+B4)->B1 14.3% (B1+C4)- >C1 6 Implied MA grant value (cumulative @ end of year) 0 22.2 (A6+B2)*(1+B5) 36.8 (B6+C2)*(1+C5) 7 Implied share of value 0% 22.2% (B6/B1) 23% (C6/C1) Attributable share of outcomes for the year 0% 22.2% 23% Source: Evaluation Team This annex provides a brief introduction to some methods that could reasonably be used to calculate RoI within the MA program. The Evaluation Team suggests that should USAID decide to collect RoI on all, or even a sample, of awardees, resources will be required to make a robust analysis, which will require several days of work with each awardee. 95 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX 4: DATA ANALYSIS VARIABLES The following table shows the codes used for coding detailed interview notes. The codes were developed from the evaluation questions for the first stage, and inductively from the data in the second stage. Coding was conducted in Dedoose, a qualitative data analysis subscription software package. ID Parent ID Title 1 Benefit (non-financial) 2 Character 3 2 Effectiveness 4 2 Efficiency 5 2 Negative / Weak 6 2 Positive / Strong 7 Evaluation Questions 8 7 Development outcome 9 8 Business operations 10 8 Clean Energy 11 8 Education 12 8 Environment 13 8 Health 14 8 Livelihoods 15 7 Innovation 16 15 Definition 17 15 Prototype development 18 7 Knowledge management and communications 19 18 Knowledge of MA 20 18 With awardees 21 18 With partners 22 7 Leveraging 23 7 MA program design 24 23 Learning 25 23 Model 26 23 Technical team 27 7 Management 28 7 Sustainability 29 28 Business model 30 Features 31 30 Application & Selection process / result 32 31 Previous application 33 30 Capacity building 96 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 34 30 Committee 35 30 Comparison to other grants 36 30 Contracting & Award 37 36 Disbursement & lump sum 38 30 Delay 39 30 FICCI relationship 40 30 Financial 41 40 Additional Resources 42 40 Cost 43 40 Grant as requested 44 40 Grant less than requested 45 44 adjustment to scope 46 40 Returns on Investment 47 40 Taxes not owing 48 40 Taxes owing 49 30 Gender 50 30 How heard about MA? 51 30 Impact evaluation 52 30 Implementation 53 52 project timelines 54 30 Lessons learned 55 30 M&E 56 30 Networking / Partnership 57 30 Patents & IP 58 30 Project Design 59 30 Publicity / Support /Reputation 60 30 Recommendations 61 30 Reporting 62 30 Risk 63 30 Scale-up 64 30 Success strategy 65 Feedback from MA to awardee 66 Follow-Up 67 Gender 68 Highlight 69 Input / Advice / Mentoring 70 Joint funding 71 Quote 97 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 72 SWOT 73 72 Challenge 74 72 Failure 75 72 Opportunity 76 72 Success 77 72 Threat 98 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX 5: SCHEDULE OF WORK 99 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX 6: LIST OF KEY INFORMANTS AND AWARDEES INTERVIEWED AWARDEE INTERVIEWS Awardee Organization Name Sector Round Dates Stage For Profit/ Non FP Sex DELHI 1 Kaivalya Education Foundation Education 4 (7/17-7/20) 2 Non Profit F 2 Boond Engineering and Development Clean Energy 3 (7/16-3/17) 2 For Profit M 3 Swajal Water WASH 3 (4/16-7/17) 1 For Profit M/F 4 CropConnect Enterprises Agriculture 4 (7/17-7/20) 2 For Profit M 5 Simpa Networks, Inc. Clean Energy 3 (5/16-2/17) 2 For Profit M 6 Self Reliant Initiatives through Joint Action (SRIJAN) Agriculture 3 (6/16-6/19) 1 For Profit F 7 Parvata Foods (operates in Sikkim) Agriculture 3 (5/16-6/19) 2 For Profit M 8 Desolenator/ eKutir Rural Management WASH 3 (7/16-6/17) 1 For Profit F MAHARASHTRA (MUMBAI/PUNE) 9 Gram Oorja Solutions Clean Energy 3 (6/16-7/19) 2 For Profit M 10 Aakar Innovations Health 3 (6/16-6/18) 3 For Profit M 11 Swayamsiddha Mahila Utkarsha Foundation WASH 2 (11/14-10/17) 2 For Profit M 12 Sanjeevani Disaster Equipment Agriculture 4 (4/17-3/20) 1 For Profit F 13 Science for Society (S4S) Agriculture 1, 3 Mumbai (7/13-3/14), Nepal (6/16- 10/18) Kenya (6/16-3/18) 2, 3 For Profit M KARNATAKA (BANGALORE) 14 Akshara Foundation Education 3 (11/14-11/16) 2 Non Profit F 100 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 15 Coeo Labs Health 3 (7/16-8/18) 1 For Profit M 16 Sattva MedTech Health 3 (6/15-7/18) 1 For Profit M 17 Leowin Solutions Health 4 (1/17-12/18) 1 For Profit M 19 Audicor Cardiometrics Health 5 (6/18-5/20) 1 For Profit M 18 Hasiru Dala Innovations WASH 3 (6/17-6/19) 1 For Profit M WEST BENGAL (KOLKATA) 20 Sundarban Coop Milk and Livestock Producers Union Agriculture 5 (4/18-3/21) 1 For Profit M ODISHA (BHUBANESWAR) 21 Sikshasnadhan Education 4 (7/17-7/20) 2 Non Profit M REMOTE 22 Farms n Farmers (Nepal) Agriculture 2 (11/14-3/18) 3 For Profit M 23 ZMQ Development (based in Delhi) (Afghanistan) Health 2, 5 (8/15-7/18) 3 For Profit M/M 24 Naireeta Services (Bangladesh) Agriculture 2 (2015-2019) 3 For Profit M 25 ENVIRON (Assam) WASH 4 (7/17-6/19) 1 Non Profit F 26 Excel Matrix Biological Devices (Hyderabad) Health 4 (7/17-6/19) 1 For Profit M 27 Pratimesh/MicroX Labs (Bangalore) Health 2 (11/14-11/15) 1 For Profit M 28 Nauguchiya Jan Vikas Lok Karyakarm (Bihar) Agriculture 4 (617-7/19) 1 For Profit M STAKEHOLDER INTERVIEWS USAID Location Sex 1 Sr. Manager Delhi F 2 Sr Advisor Delhi M 3 Senior Manager Delhi F 101 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV FICCI 4 Sr. Manager Delhi F 5 Manager Delhi F 6 Sr. Manager Delhi M 7 Sr. Manager Delhi M 8 Regional Consultant Kolkata M 9 Sr. Manager Kolkata F Think Through Consulting 10 Sr. Manager Delhi M 11 Manager Delhi M 12 Consultant Delhi M Key Stakeholders 13 Sr. Manager, IFC - International Finance Corp.-World Bank Group Delhi M 14 Sr. Manager, Ankur Capital Mumbai F 15 Adviser, National Science & Technology Entrepreneurship Development Board (NSTEDB) Delhi M 16 Sr. Manager, DfID India Global Team Delhi F 17 Sr. Manager, Unltd India Delhi F 18 Sr. Manager, Innovation Alchemy Consulting Pvt Ltd Bangalore F 19 Sr. Manager, Indian Angel Network Delhi F 20 Manager, IBAC Kolkata M 102 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX 7: LIST OF DOCUMENTS REVIEWED Author Title Date Purpose FICCI List of MA facilitated capacity building initiatives and workshops - Understand focus of capacity-building FICCI List of all MA awards - Assess awards by geography, stage, round, value FICCI Vision 2025 - Future direction of MA FICCI MA Awarded Enterprises Booklet, Round 1-5 - Background document of awards by innovation description, geography and round FICCI Millennium Alliance RFP for Monitoring & Evaluation 2015 Contextual information FICCI List of experts for Millennium Alliance - Contextual information FICCI Partners Meeting presentation 2018 Contextual information FICCI Sample documents and reporting templates: Fund utilization; Due Diligence; Quarterly reporting; workplan, monitoring plan, evaluation criteria - Understand M&E process FICCI Millennium Alliance Capacity Building Plan 2016 Understand management process FICCI Millennium Alliance Frequently Asked Questions - Contextual information FICCI India-Rwanda Innovation Growth Program - Contextual information TTC M&E Capacity Building workshop presentation 2018 Understand M&E process TTC Proposal for Millennium Alliance RFP for Monitoring & Evaluation 2015 Contextual Information USAID/India Country Development Cooperation Strategy (CDCS) - Understanding USAID/India’s strategy USAID/India Evaluation of USAID/India Development Objective 4 Mid-term Performance Evaluation 2017 Contextual information 103 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV USAID/India Evaluation of USAID/India Development Objective 3: Assessing the Relevance, Effectiveness, and Impact of Public Private Partnerships 2017 Contextual information USAID and FICCI Public Private Alliance Collaboration Agreement Between Federation of Indian Chambers Of Commerce And Industry and United States Agency For International Development Concerning the Millennium Alliance: An India-U.S. Partnership For Global Development 2012 Basis of agreement University of Greenwich for DfID Performance Evaluation of the Global Component of ‘Innovative Ventures & Technologies for Development’ (INVENT) 2018 Reference of mid-term evaluation that included MA 104 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX 8: MEDIUM AND LOW PRIORITY RECOMMENDATIONS MEDIUM PRIORITY RECOMMENDATIONS REF Recommendation EQ Theme Lead Effort Time frame 18 MA needs to find a clearer focus, perhaps narrowing down its focus to specific sectors, types of innovation or stages, in order to reduce risk and build holistic support that maximizes potential outcomes. The focus of the MA program could potentially be linked to the focus sectors of USAID and the other Partners. This would allow innovators to collaborate more around common interests and objectives. Differentiate between MA programmatic capacity building and business￾oriented capacity building. The former (e.g. monitoring and evaluation, reporting to MA, sharing successes on social media) should be required of all awardees to ensure proper conduct during the ward and maximize the validity of monitoring data. MA programmatic capacity building should be excluded from award funds. Business-oriented funds should be controlled by awardees with clear conditions on eligibility, with MA only confirming eligibility of non-FICCI events, not approving them. In addition to providing greater autonomy of awardees, this separation will result in a more transparent allocation of award funds and operational funds. EQ1 Focus FICCI Medium Medium Term 19 Develop a robust definition of sustainability within MA. This notion should also be expanded beyond financial sustainability to environmental and social sustainability. Apply the definition of sustainability throughout the application, grant management and innovation management lifecycle at the project and awardee level to encourage sustainable business. EQ4 Focus FICCI Medium Short Term 20 Enhance sectoral focus within MA Management. This focus could come in the form of focused sectoral calls in which FICCI and MA partners facilitate relationships among awardees, major EQ2 / EQ5 Management & Governance FICCI High Medium Term 105 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV investors, and potential industry collaborators (including from FICCI membership) in like￾sectors. Another means to this end would be to provide in-house sector-based portfolio managers who could also serve as technical advisors, as well as in-house expertise on start-up enterprise development and sectors. Increasing staffing levels, especially with staff additions who take on specific portfolios by sector or business development stage would enable FICCI to supply a fuller range of services to awardees and better prepare them for success. 21 Utilize partners who are thought leaders in the innovation ecosystem to both i) reinforce MA’s existing networks and ii) establish new connections to stakeholders who can share lessons learned and help MA better define strategies for operating in and navigating the inclusive innovation space. Solicit ecosystem partners that can enable that support, whether network access, peer learning, capacity building, refine the business plan, strengthen leadership or offer mentoring. EQ 1 / EQ7 Management & Governance FICCI Medium Long Term 22 Utilize partners who specialize in media and partnership development to develop a media strategy and improve the platforms visibility. This strategy will ensure MA’s engagement in high level conferences and public expositions that contribute to the innovation ecosystem and reposition FICCI as thought leaders. EQ7 Management & Governance FICCI Medium Long Term 23 Evaluate the opportunity costs to subcontracting monitoring to TTC. There would be a number advantages to FICCI directly recruiting experienced monitoring, evaluation and learning specialists, including (i) streamlined feedback mechanisms to awardees; (ii) better integration between quarterly reporting and with payment mechanisms by only advancing payment to awardees who are up-to-date on reporting, making use of instant information transfer within the same office; (iii) using monitoring results to manage awards rather than just as a reporting EQ5 Management & Governance FICCI High Medium Term 106 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV tool; and (iv) using qualitative feedback such as capacity building needs and lessons learned to create faster turn-around to meet awardee needs. 24 MA needs to follow up on awardees after the award period is completed. Developing a simple online survey could be sufficient for this if resources do not allow for at least a sampling for face-to face meetings. Follow-up annually asking about the sustainability of the innovation and some key indicators that would suggest financial and non-financial sustainability related viability, valuations, market expansion and so on. EQ4 Monitoring, Evaluation and Learning FICCI High Medium Term 25 MA should develop clear parameters for what constitutes a success and a failure. This would require a review of the portfolio. With clear parameters in place, MA would be able to adjust the selection process accordingly to avoid failure and track awardees to ensure that capacity building is adjusted as needed to reduce failure rates. Possible parameters for failure could include an awardee’s inability to (ii) spend the full award, (ii) develop the innovation as promised, or (iii) achieve development outcomes. Mechanisms of failure should have trigger points and monitoring flags, with clear protocols for correction such as an early warning with support from MA to alter the undesirable trajectory and the cessation of further disbursements for awardees who do not comply. Further understanding of what makes a failure in the MA program therefore requires further research, including a clearer working definition of what constitutes a failure. EQ1 Monitoring, Evaluation and Learning FICCI Medium Short Term 26 MA can better engage with entrepreneurs outside of Bangalore, Delhi, and Mumbai by promoting more structured outreach and learning opportunities and improving the entrepreneur’s exposure by working with key stakeholders in more remote areas. FICCI should aim to utilize regional centers as hubs for knowledge management for entrepreneurs and create linkages between the regional centers and FICCI headquarters to best ensure that the MA platform EQ3 / EQ5 / EQ7 Service Delivery FICCI Medium Medium Term 107 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV can wholly benefit from innovative solutions. Regional offices can assist in marketing MA and with the application process technically, linguistically, or technologically. At a minimum, this could include directing applicants to designated centers that have personnel who can explain the process and help with language and/or uploading the application. More substantially, this could include providing a funding mechanism that would enable small funds for applicants based in more remote jurisdictions to develop applications. Consider a targeted effort to engage more with entrepreneurial networks outside of larger cities to best encourage rural applicants and respond to dynamic development needs in hard to reach places. 27 Develop an engagement plan within FICCI to explore ways that its membership might be able to benefit MA awardees. Such benefits might take the form of mentoring or coaching, joint capacity￾building activities, and/or capitalization of the FICCI pool of award funds through repayable or non-repayable awards. EQ8 Service Delivery FICCI Medium Medium Term 28 MA should provide support to awardees to develop and manage sustainability plans as part of its Capacity Building. Since sustainability is more readily assessed for Stage 2 and 3 applicants, different criteria should apply to different stages of innovation. EQ4 Service Delivery FICCI High Short Term 29 Report back to awardees on a plan for annual FICCI-hosted capacity building events. Show how offerings relate to suggestions from awardees based on quarterly report feedback and annual needs assessments for capacity building of awardees. The current plan is an internal document and the extent to which its contents are driven by awardee input remains dubious. EQ5 Service Delivery FICCI Medium Short Term 30 MA staff should initiate a systematic process, whether through WhatsApp or another digital platform, to connect awardees with others in their sector and in their geographic area, regardless of EQ7 Service Delivery FICCI Low Short Term 108 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV round of award, will streamline knowledge sharing of both successes and challenges and contribute to the growth and success of early stage entrepreneurs as they grow their business. LOW PRIORITY RECOMMENDATIONS REF Recommendation EQ Theme Lead Effort Time frame 31 Develop some basic criteria to reduce the overall numbers of applicants who qualify to submit proposals. This could be done by including some pass/fail criteria in the application such as demonstration (not explanation) that (i) the innovation addresses at need for BoP; (ii) the innovation is commercially viable (for commercial applicants); and (iii) the proposed innovation is a fit within at least on MA focus sector. EQ5 Focus FICCI Low Medium Term 32 MA must alter indicators of success and development impact for early-stage innovators. If wants development impact, Stage I awards will not contribute substantially, however, as they grow, these innovations could have substantial development impacts and therefore should not be eliminated from MA. One solution would be to integrate objectives and indicators that show business growth on the one hand and development outcome on the other. To draw up indicative indicators, consult with partners that already use tried and tested indicators for financial returns and social impact including UnLtd India and Ankur Capital, and awardees Crop Connect (whose founders were previously involved with Development Marketplace). EQ3 Impact MA Partners Low Long Term 33 If FICCI and USAID are committed to tabulating RoI, direct resources to collecting the data required to do an RoI analysis on at least a sample of awards. This report contains some EQ3 Monitoring, Evaluation and Learning FICCI High Short Term 109 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV suggestions on methods and indications of the availability of data under current data collection systems, however collection of data that could be reliable would require an in-depth RoI study due to the lack of readily available data that must be purposefully collected. See ANNEX 9. Another step in calculating the RoI could include estimating the value of indicators, which must be context specific and time-bound, requiring detailed assumptions. 34 Develop better connections with other platforms and investors in the inclusive innovation space. MA does not have to meet all expectations of awardees directly. Forming better alliances with investment networks and business incubators into which MA awardees could ‘graduate’ could provide immense leveraging for awardees. Bringing some of these groups into the MA network, even as partners, could also increase program-wide leveraging. EQ2 Service Delivery FICCI Medium Medium Term 35 Provide capacity building activities specifically oriented different stages of awardee maturity. This will address the issue that awardees developing proof of concept have significantly different needs than awardees expanding internationally. Build in more capacity building to advance to the next stage of development through (i) helping awardees know what resources are available to them, (ii) building the capacity of awardees to build relationships and convincing proposals to entities that can assist in their advancement and (iii) making connections between investors and awardees, or incubators. EQ5 Service Delivery FICCI High Medium Term 110 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX 9: INNOVATION ANALYSIS CONSIDERATIONS This report uses the term ‘inclusive innovation’. However, there are many related and overlapping definitions that all emphasize innovation with a focus on development outcomes and low-income consumers. These include social innovation, market-based solutions to poverty, social entrepreneurship, impact ventures, products and services for the bottom of the pyramid (BoP). Inclusive innovation in India operates within the context of social change, and broadly encapsulates the idea of creating access to products and services for the underserved, or livelihood opportunities as producers, primarily from the low-income segment at the bottom of the pyramid. Innovation is taken to be something new in the local context, rather than ‘new to the world.’ Inclusive innovation has a long history in India, with a base in grassroots innovation and indigenous knowledge movements, and innovation therefore emerged from a range of sources and stakeholders. Therefore, the inclusive innovation and social entrepreneurship ecosystem in India is expansive and includes a wider range of different kinds of innovation actors than may be found elsewhere. These can broadly be classified into: ● Grassroots innovation – individual innovators solving real world problems they face – often in smaller towns, peri-urban or rural areas. Grassroots innovation is often spoken of in conjunction with frugal innovation (make more with less) and jugaad innovation (make do and mend innovation). A leading network advocating for grassroots innovation is the Honey Bee Network as well as the National Innovation Foundation, an autonomous body of the Department of Science and Technology, which runs an inclusive innovation award. ● BoP Solutions by conventional private sector are products and services that are affordable enough for low-income households (the BoP) to buy. There are different variations of BoP solutions. For example, Hindustan Unilever’s PureIt water filter was developed with the low￾income consumer in mind and the distribution and outreach is undertaken through NGOs, unlike for other categories of consumers. Other goods and services where BoP are considerable consumers, though not exclusively targeted through tailored distribution systems, include P&Gs shampoo sachets as well as RIL’s Jio phone network and handsets. ● NGOs working on poverty alleviation that may or may not be innovating, scaling or trying new things. Funded by private and government grants, and a major recipient of CSR funding in India. International donors, and international foundation such as BMGF and Michael & Susan Dell Foundation are prominent examples, while newer Indian foundations including Azim Premji Philanthropic Initiatives and Rohini Nilekani Foundation have emerged in recent years. There are over 3 million NGOs registered in India and there are concerns that not all undertake the work they claim, which has resulted in increased monitoring of NGOs as well as stricter rules for setting up an NGO as well as receiving funding, especially foreign funding. ● Social enterprises and entrepreneurs are broad terms to denote entities and individuals that use entrepreneurial methods to create social impact, in the case of India, primarily development and poverty alleviation. Unlike in countries such as the UK, there is no definition of what a social entrepreneur or social enterprise is in India, and the term is therefore wide open to 111 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV interpretation. The result is that a wide range of for-profit and non-profits call themselves social entrepreneurs, and there are obvious overlaps with the three types of organizations outlined above. Generally, in India, those that self-identify as social entrepreneurs tend to be operating out of urban centers, be well educated and engage in English. This is a reflection of the focus of the social enterprise ecosystem itself, which has to date been highly centralized in India’s largest cities, primarily using English to communicate and with close links to US, UK as well as Africa. Additionally, related to these overlapping types of innovators and entrepreneurs for social good, are three other movements in India. These have limited overlaps with the inclusive innovation ecosystem detailed below (although there is potential for more). ● Recent government policy and programs are focusing on creating entrepreneurs and innovators by allocating budgets to set up incubators in towns and district and running competitions for startups, as well as setting up tinkering labs in schools, and setting up a network of mentors. Prominent programs include Startup India and Atal Innovation Mission. ● Likewise, higher education institutions in India have jumped on the entrepreneurship bandwagon in India, setting up incubators, accelerator programs, and entrepreneurship centers on campus. Some of these incubators are excellent for conventional startups (e.g. IIMA’s CIIE and IITB’s SINE) while others focus on social entrepreneurs (OP Jindal’s JCSIE). However, most are poorly run since professors and other staff have limited to no experience in engaging with and supporting new companies and entrepreneurs. ● The large number of micro and small entrepreneurs, running small businesses, factories and production units, for example. Many of these are necessity entrepreneurs (by default rather than by choice) and part of the informal economy, with limited sustainability or pathways to growth and scale. However, there is increased interest in how to spur more entrepreneurship of this kind, which in turn contributes to job creation and development. One example is the Global Alliance for Mass Entrepreneurship (GAME). The ecosystem enabling inclusive innovation in India has evolved rapidly in the past decade, from a handful of incubators and impact investment funds, to a large number of entities offering specialized financial and non-financial support to organizations in specific sectors, issue areas, organizational type or venture growth stage. This ecosystem supports primarily self-identifying social entrepreneurs. As with the definition of social entrepreneurship, the ecosystem that works to enable social entrepreneurship each interpret the term differently, and to align with their internal vision. Therefore, different impact investors, or patient capital, that claim to support social entrepreneurs, can have wildly varied terms on which they operate. INDIA INCLUSIVE INNOVATION ECOSYSTEM Type About Examples (non-exhaustive) Impact investors Invests at the early stage by taking a stake in the company. Can only invest in for-profit entities. Aavishkaar, Menterra, Ankur Capital, Lok Capital, Acumen, Omnivore, Unitus Seed Fund, CIIE 112 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Bharat Innovation Fund, Asha Impact. Angel networks Individuals forming part of networks to invest together or individually at very early stage in for-profit companies. Most invest in both regular and social ventures (I3N only invests in social impact). India Angel Network (IAN), The Indus Entrepreneurs (TIE), Mumbai Angels, Intellecap Impact Investment Network (I3N) Debt finance Venture debt for growth stage social enterprises and regular debt for MSMEs (it remains difficult to access bank loans for social enterprises without collateral). Intellegrow, Kinara Capital, Northern Arc Capital Social Incubators & Accelerators Incubators focused on social enterprises, providing mentoring, capacity building, links to funders, and investment readiness training. Villgro, UnLtd India, Deshpande Foundation Sandbox, Dasra, Social Alpha. Incubators Conventional incubators that recently started programs for social enterprises Zone Startups, CIIE-IIMA Conferences Conferences to connect across ecosystem. Usually includes awards where social enterprises are showcased. Sankalp, Unconvention. Deshpande Development Dialogue Networks Networks of stakeholders India Impact Investment Network (investors); Aspen-ANDE (multi￾stakeholder); Asian Venture Philanthropy Network-India (multi￾stakeholder). Entrepreneur Journeys Journeys for nascent and budding entrepreneurs and innovators to learn, engage in peer learning and gain mentoring. Alumni networks from these journeys are later used by entrepreneurs. Jagritri Yatra (train journey) Shodh Yatra (journey on foot) Implementers & Consultants Diverse organizations that implement large scale donor funded programs, research and provide capacity building. Dasra (implementer); Intellecap (consultant); Ennovent (implementer & consultant) Competitions & Awards Programs that provide funding together with non-financial support. Marico Innovation Foundation program; Tata Social Enterprise Challenge; Arohan Social Innovation Award (Infosys) 113 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Government Initiatives Award and competitions by state and central government Maharashtra Startup Week Awards; Bengaluru Innovation Challenge; INVENT (with DfID) Funds of Funds organizations that fund investors and incubators in the ecosystem Rockefeller Foundation; Omidyar Network; World Bank, IFC. Likewise, some incubators may focus primarily on creating social impact, while others are narrowly focused on improving the incubatees’ business plan, for example. The below table summarizes the kinds of support that exist today. Characteristics of the Inclusive Innovation Ecosystem ● The dual (financial returns + social impact) or triple (financial returns + social impact + environmental sustainability) bottom line leads to tension for many entrepreneurs as the business side may have one set of prioritize while the social impact side requires a different set of priorities. In India this has led to new business models. At the same time, it means that entrepreneurs need a lot of support as they navigate the dual or triple demands on them. There is also a significant risk of ‘mission drift’ as entrepreneurs and innovators may be driven to take decisions to enable financial sustainability over social impact. ● Innovators need to be entrepreneurial to find a solution to the dual (Financial and Social) and triple bottom (Financial, Social and Environmental) line demands, as well as finding ways to overcome last mile challenges, Therefore, new business models are emerging and the support for innovators to either bring on board entrepreneurs or to become entrepreneurial is important in order to make ventures sustainable in the long term. ● The last mile challenges include creating new distribution or collection systems from scratch, train local talent to work, as well as create demand through awareness raising and marketing, and at times train consumers. Likewise, many new solutions have failed since there was no local maintenance available, so long-term strategies for maintaining products and services is important. At the same time, consumers are extremely price sensitive, and it is difficult to reach out with new information, or to decipher between true or false information which means new entrepreneurs need to overcome trust deficits, while reaching hard-to-reach consumers and create a business model with low margins. ● There is no one clear legal definition of social enterprise or innovator and regulatory choice of for profit or non-profit often comes down to what is the easiest way forward. Starting a non￾profit is usually a lengthy process, while it is easier to start a for-profit. On the other hand, for￾profits cannot access grant funding which makes it hard for them to access funds at early stages of development. ● There are issues surrounding regulations for setting up incubators and investors in India. For example, setting up foreign funds in India can be complicated. Strict regulations on bringing foreign funding in, especially for non-profit, makes it difficult for new entities to access foreign funding. 114 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Likewise, non-profit incubators cannot invest in entities using foreign funding. Recently there has been debates around the need to pay taxes on investments in startups, locally called Angel Tax. Likewise CSR funding cannot be used for social enterprise funding to any great extent and still comply with regulations under the two percent rule. ● Quantity of support does not equal quality. While there are a lot of entities in the ecosystem today, especially taking into consideration incubators and accelerators at universities, the quality of many of these remain limited. Likewise, many operate in silos and there has been limited effort to create a continuum between different forms of support. At the early stage there remains a significant gap in finance and non-financial support. ● Gender has in the past five years become an important dimension and gone from meaning women targeted as end beneficiaries to focusing on women social entrepreneurs, and the number of women in teams. A subcategory of investing called gender lens investing has emerged, in which gender factors are explicitly taken into account and integrated into investment decisions and management. Likewise, new incubator and accelerator programs as well as awards target women￾led enterprises specifically. There is an increased recognition that women need to be better included in the social enterprise ecosystem. ● Because of the varied ways inclusive innovation is understood and occurs, measuring innovation and developing appropriate indicators is difficult. This is further complicated by the need to track both financial metrics and social or developmental metrics. In India the last five years have seen an increased use in impact measurements as incubators, accelerators and investors have fine￾tuned their models through trial and error as well as using international impact measuring standards such as SROI and IRIS. However, international impact measurement frameworks generally have to be adapted to suit the Indian context. Innovation Guidelines ● The USAID definition of inclusive innovation is: “Innovation refers to novel business or organizational models; operational or production processes; or products or services that lead to substantial improvements in executing against development challenges. Innovations help produce development outcomes more effectively, more cheaply, that reach more beneficiaries, in a shorter period of time, and more sustainably.” (Source: USAID CDCS) ● Focus on the sectors and issue areas that USAID prioritizes currently and apply the definition above to consider what development challenges and outcomes mean within those sectors and issue areas. Given the lack of clear one-size-fits-all definition of inclusive innovation, look to establish your own definition that works for the purpose you have in mind. ● Once there is a clear understanding of outcomes and challenges that innovation should be working with or for (similar to mission-oriented programs or grand challenges), seek out innovation that targets these. ● Consider whether there are further criteria to apply – age of enterprise or organization; operating model: profit or not-for-profit; stated social focus or not; level of transparency and systems in place internally. 115 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ● By keeping the idea of ‘what is innovation’ as broadly something new in the local area, it allows for a focus on new ways of doing things for example by combining existing methods, or tweaking an operating model to suit a different kind of last mile challenge, emphasizing innovation in processes and ways of doing things more than new-to-the-world technologies. ● If, on the other hand, new-to-the-world technologies are sought, an innovation challenge or competition with strict criteria and a large award, may be more appropriate than an ongoing platform such as the MA. 116 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX 10: EVALUATION TEAM MEMBERS AND RESPONSIBILITIES EVALUATION TEAM ROLES AND RESPONSIBILITIES ROLE QUALIFICATIONS RESPONSIBILITIES Evaluation Team Sr. Evaluation Specialist/Team Leader (SI) Dr. Rodd Myers 22 years of development experience in 21 countries, 6 years overseeing monitoring and evaluation; and experience evaluating large, multisectoral programs Extensive experience with policy-makers, civil society and corporate stakeholders to design, operate and evaluate development programs Ph.D. from the University of East Anglia, and a Master’s of Science degree in Sustainable Agriculture and Rural Development from the Imperial College London. Led teams in complex environments, including a $25 million livelihoods program in 12 countries, Accomplished author with 18 peer reviewed publications and policy briefs to date. ● Lead the evaluation, including desk review, evaluation design, data collection and interviews, analysis of the findings, and report writing, working closely with SI technical specialists ● Manage the evaluation, including delegating responsibilities, training and guiding evaluation members, monitoring progress, and providing feedback on inputs ● Primary liaison with USAID for facilitating and leading briefings/presentations; 117 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Sr. Innovation and Public Private Partnership Specialist (AI) Dr. Lina Sonne Over 15 years of experience (eight in India) leading and designing innovation activities. Works with partners including academic institutions, multilateral agencies, impact investors, incubators, the public sector, corporations, startups, and social enterprises. Ph.D. in Economics and Policy Studies of Innovation and Technical Change. ● Provide Innovation and public￾private partnership expertise and context to evaluation design, tools and analysis ● Lead data collection in two sites while keeping communicating with TL ● Ensure analysis, planning, data collection, analysis, and reporting processes are technically sound ● Ensure that data collection adequately addresses the evaluation questions Evaluation Specialist Ms. Leah Ghoston Over ten years of experience in research, coordination, and management for global health and development projects. Skills in evaluation include qualitative methodological design and analysis, participatory approaches, formative research assessments, and management of global evaluations for USAID and local stakeholders. M.P.H with a focus on Health Education and Promotion from the University of the West Indies. ● Ensure analysis, planning, data collection, analysis, and reporting processes are technically sound ● Ensure that data collection adequately addresses the evaluation questions 118 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Evaluation Specialist Dr. Rajarshi Singh Monitoring, Learning, and Evaluation Technical Lead at Athena Infonomics. Has experience leading literature reviews and developing conceptual frameworks and research tools. Ph.D. and M.S. in Computational Mechanics from Carnegie Mellon University Logistician Ms. Aditi Gupta Current employee of Athena Infonomics and has experience providing logistical support services in India. ● Complete bookings for flights, hotels, meeting space, and transportation within India ● Schedule interviews with selected key informants ROI ANALYSIS Financial Analyst Mr. Praveen Ravi Specializes in Applied Decision Sciences and focuses on impact assessment, cost benefit assessment, governance, and reforms. Post Graduate Diploma in Management from IIM Lucknow and a B.Sc. from Chennai Mathematical Institute. ● Collect and review financial reports and monitoring indicator data for awardees ● Prepares estimate of programmatic ROI 119 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV TECHNICAL ADVISORY BOARD WASH Dr. Aravinda Satyavada Education Dr. Vir Narayan Agriculture Mr. Abhinav Gupta Cost Analysis Mr. Praveen Ravi Energy Dr. Sundersasan Srinivisan Health Ms. Kumkum Srivastava Recognized experts in their respective fields with a minimum of ten years of experience, some with as many as 35 years of experience. All have a deep understanding of India’s social, political and economic systems and ability to comment substantively on ongoing status of innovations in their respective fields. ● Provide in-depth technical expertise in the five focus sectors ● Provide sector input to tool development, data interpretation, and final report ensuring key sector specific issues are addressed and conclusions/recommendations are feasible and relevant ● Collect awardee financial data and awardee monitoring indicator reports ● Calculates ROI from MA Program HQ SUPPORT STAFF Program Director Ms. Paige Mason More than eight years’ experience managing and being a technical team member on USAID funded evaluations. ● Provide oversight to all evaluation activities ● Conduct quality assurance review on all deliverables and EQUI® checkpoints ● Troubleshoot issues that cannot be resolved internally among team members ● Ensure USAID satisfaction via routine check-ins ● Provide high-level guidance on methodology, major deliverables’, and any problems ● Oversee TL and HQ management staff 120 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV Program Manager Ms. Catherine Villada More than five years’ experience managing evaluations for USAID, DOS, MCC, and foundations. ● Responsible for contractual and financial management of the evaluation, including adherence to deadlines, requirements of the Statement of Work, and budget management ● Coordinates meetings and interactions among Social Impact, Athena Infomatics, and USAID, and within the Evaluation Team ● Ensures quality and compliance of deliverables with contractual requirements ● Manage the budget by tracking level of effort usage and other spending. Program Assistant Ms. Natalie Provost 1.5 years’ experience providing administrative support for USAID funded evaluations ● Provide administrative and logistical support—arrange travel and lodging, process expense reports, etc. ● Support Evaluation Team members with invoicing and setup in SharePoint ● Support deliverable development, including formatting, copyediting, and data visualization Gender Specialist Ms. Noshaba Zafar Extensive experience providing gender reviews for other ALME task orders ● Review all major deliverables to ensure appropriate addressing of gender considerations. 121 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX 11: LIST OF FICCI CAPACITY BUILDING EVENTS Title Date Themes First Process Documentation Workshop conducted by Social Impact Mumbai August 28-29, 2014 ● Development of Theory of Change and mapping of progress ● To document challenges and learnings of awardees ● To capture the impact of MA association and document short and long term changes U.S.-India Start-up Assist Workshop November 16, 2015 ● Commercialization, team building, branding, product promotion, government support, scaling up of a venture, regulatory compliance, and fundraising. ● “promoting women entrepreneurs in science and technology” Social Enterprise Innovations Seminar, Enhancing Service Delivery to the Underserved (World Bank) December 14- 15, 2015 ● Social enterprise sector in the Indian context that will allow the government, impact investors and other stakeholders to engage with the social enterprise to achieve specific development objectives ● Positive enabling environment, informing national and state level policies and tools for supporting social enterprises and enhancing effectiveness and results-at -scale; ● Key areas where the government could collaborate with the social enterprise sector in design and/or delivery of public services to advance social and economic development; ● Key market and government constraints for social enterprises Capacity building workshop for Millennium Alliance (MA) Awardees R-3 June 27-28, 2016 Monitoring and Evaluation 2nd Capacity building workshop for MA Awardees R-1, R-2 & R-3 Sept. 23-24, 2016 ● Monitoring & evaluation ● Regulation ● Taxation for Not for Profit organization, ● HR Practices ● Government schemes and aspects that can be leveraged for scaling-up 122 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV INVENT South-South Learning Program (DfID) January 30th, 2017 ● key challenges and opportunities in south￾south innovation transfer, ● cross learning and suggest means to improve the INVENT global program. ● global expansion Capacity Building through DIV Team (USAID) Three events: Mumbai, Bangalore and Delhi Criteria and the application review process that DIV follow for their calls Millennium Alliance (MA) Innovation Conclave (Monitoring & Evaluation Workshop and use of media for social impact for Round 5 Awardees) May 31 - 1 June 2018 ● Monitoring and Evaluation ● Media for Social Impact Millennium Alliance Investor Innovator meet and Capacity Building workshop October 9-10, 2018 ● Meet with investors and CSR representatives ● Sustainability Vs Scalability and Business Canvas Modeling for social enterprises ● Scale-up ● Storytelling Source: FICCI records 123 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV ANNEX 12: SIGNED DISCLOSURES OF CONFLICT OF INTEREST 124 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 125 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV 126 | USAID/INDIA MILLENNIUM ALLIANCE PROGRAM MID-TERM PERFORMANCE EVALUATION USAID.GOV