[March 2015] This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Devyani Srinivasan. andfill this text box. Mid-term Evaluation of the Prepaid Solar Energy Project, a for-profit approach to the twin challenges of technology and financing in Uttar Pradesh, India MID-TERM EVALUATION OF THE PREPAID SOLAR ENERGY PROJECT – A FOR-PROFIT APPROACH TO THE TWIN CHALLENGES OF TECHNOLOGY AND FINANCING IN UTTAR PRADESH, INDIA A PREPAID METERING AND MOBILE PAYMENTS TECHNOLOGY PLATFORM THAT MAKES CLEAN ENERGY SIMPLE, AFFORDABLE AND INVESTIBLE [March 10, 2015] DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Acronyms .............................................. 7 Executive Summary ............................. 8 Evaluation Purpose and Evaluation Questions ............................................ 12 Project Background ........................... 14 Evaluation Methods and Limitations 15 Findings, Conclusions and Recommendations ............................. 17 Annexes ............................................... 42 - Annex 1: Evaluation Statement of Work ..............................................43 - Annex II: Evaluation Methods and Limitations ..................................... 46 - Annex III: Data Collection Instruments ................................... 48 - Annex IV: Sources of Information ........................................................80 - Annex V: Disclosure of Any Conflicts of Interest ...................... 81 ACRONYMS LED UPS USAID Light-emitting Diode Uninterruptable Power Supply U.S. Agency for International Development EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS Simpa Networks is a for-profit enterprise with an innovative solar-as-a-service business model. After demonstrating its innovative business model with 200 customers, Simpa sought support from USAID’s Development Innovation Ventures to scale up to 12,000 systems by the end of the program in May 2015. Both during the USAID-supported period and beyond, Simpa will monitor the number of people it provides access to clean energy. However, this program also provides the opportunity to evaluate its impact: of the effect of increased access to clean, reliable electricity on customers’ well-being. The main indicators of customers’ well-being that Simpa has identified are their kerosene consumption, quality of health and satisfaction with energy. If after installing a Simpa system customers report that they have stopped or reduced their use of kerosene, and that they are consistently satisfied with their system, this would support Simpa’s plan to scale up. However if customers continue to use kerosene alongside their Simpa system, it may be that there is a demand for more energy and therefore larger Simpa systems. If customers report that they are not satisfied with their Simpa system, their feedback is expected to be used by Simpa to improve their offering. In addition to its impact, this evaluation is of Simpa’s cost-effectiveness for customers, and its value proposition to its customers and UrjaMitras. UrjaMitras are “Energy Friends”, local village level entrepreneurs who are independent sales agents for Simpa. The findings related to Simpa’s value proposition to its customers and UrjaMitras are expected to be used in its sales and operations. This evaluation was designed to be conducted between May 2013 and May 2015. Over this 2-year period, the evaluation questions are: 1. Who are Simpa’s customers? What is the value proposition of Simpa’s solar model and energy services for them? 2. What are the impacts of the program? To what extent are these impacts attributable to Simpa? 3. What motivates UrjaMitras to sell? What is the value proposition of Simpa for them? 4. How cost-effective are Simpa’s solar model and energy services for customers? The mid-term evaluation focuses to a greater extent on questions 1 and 3 than 2 and 4. PROJECT BACKGROUND In India at least 400 million people are without reliable access to electricity. Without access to electricity, consumers and small businesses rely on traditional fuels, kerosene, candles and batteries. It is estimated that the off-grid segment is already spending $50+ billion per year on these sub-optimal and highly unsustainable energy solutions. There are many promising technical solutions available to customers today, ranging from small-scale solar lanterns, to pico-solar home systems, to larger installed solar home systems, to community scale solar or biomass electricity microgrids, to solar-hybrid UPS/inverter solutions for the home and business. The problem is that these clean energy technologies almost always involve significant up-front costs and therefore must be financed.Simpa Networks sets itself apart from competitors by providing in￾house customer financing. Simpa’s in-house financing mechanism helps avoid a lengthy, complicated loan application process with financial institutions, for both Simpa and its customers, and, most important, makes its solutions affordable for a wider range of customers. Simpa’s Theory of Change is as follows: Inputs Processes Outputs Outcomes Impact People Product – solar Capital Product integration Product development Training Risk assessment Sales After-sales service Research Systems installed Systems used / energy paid for and consumed Improvements in well-being of consumer households There are two hypotheses behind its theory of change that Simpa seeks to refine. The first is its value hypothesis. This is a hypothesis of why people choose to “buy”. For Simpa, “buying” refers not only to installing a system but also continuing to pay for and consume the energy it generates. The second is its growth hypothesis. This is a hypothesis of how to find new customers. The evaluation of this project will be concerned with Simpa’s value hypothesis. However, because the evaluation will track the same customers over the grant period, it will not be able to significantly test Simpa’s growth hypothesis. EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS In order to address the evaluation questions, the evaluation consists of two parts: an evaluation of Simpa’s customers, and of its UrjaMitras. For the evaluation of Simpa’s customers it was important to establish causality. Since random selection and assignment to customer and non-customer groups was not possible, a quasi-experimental design was chosen in which the two groups were matched. The evaluation of UrjaMitras is through a pretest-posttest design. This report presents the results of the customer (midline) and UrjaMitra (baseline) surveys that began on the 13th of September, 2014. The evaluation was designed so that non-customers are included in the baseline and end-term evaluations, but not the midline evaluation. The chart below shows the relationship between the evaluation questions and methods used in the midline evaluation of Simpa customers, and baseline evaluation of UrjaMitras. It also includes the sample sizes for the treatment groups of customers and UrjaMitras. Evaluation Question Method Data Source Treatment Group Sample Size Who are Simpa’s customers? What is the value proposition of Simpa’s solar model and energy services for them? Quantitative Survey 215 What are the impacts of the program? To what extent are these impacts attributable to Simpa? Quantitative Survey 215 What motivates UrjaMitras to sell? What is the value proposition of Simpa for them? Qualitative and Interview and 323 Quantitative Survey How cost-effective are Simpa’s solar model and energy services for customers? Quantitative Survey 215 Based on these sample sizes, the margin of error for the customer survey is 6.62% and is 5.07% for the UrjaMitra survey (95% significance). There are limitations to how conclusive the evidence from the midline survey is, due to the methods themselves, data quality issues and bias. It is suspected that inaccuracies in respondents’ answers and data falsification made quality issues the most serious of the limitations of the midline evaluation. From a non-random sample of surveys, 11% of the midline data is estimated to be false (although not proven). FINDINGS AND CONCLUSIONS The questions that the midline evaluation addresses, that will improve Simpa’s implementation, are on its value proposition for UrjaMitras and customers. The questions that this evaluation addresses that will improve Simpa’s impact, are on customers’ reports of their satisfaction with and the quality of light from solar and non-solar sources, and their quality of health. While the evaluation of UrjaMitras does not address impact directly, as Simpa’s chosen distribution channel the ability to sell through UrjaMitras is a prerequisite in order for Simpa to create impact. The evaluation of UrjaMitras is of the value proposition of Simpa to them, and of their sales potential. Being an UrjaMitra is valued most for the financial benefits, but these are insufficient for it to be seen as a full-time activity. Instead, selling Simpa systems is viewed as a way to improve income and relationships with existing customers. This is evident from the responses to the questions on the benefits of being an UrjaMitra. For 27.6% of UrjaMitras commissions and prizes were the most important benefit. In addition to the benefits that they derive from being UrjaMitras, respondents’ current work patterns and their perceptions of customer demand are factors that may influence sales performance in the future. The element of respondents’ work patterns that Simpa was most interested in was the extent to which UrjaMitras are mobile. It was found that UrjaMitras are largely not mobile, as 70% of respondents spend most of their working hours at a fixed location. Given that 61.7% of respondents also identified “increasing the goodwill that they enjoy with existing customers” as a benefit of being an UrjaMitra, this suggests that the majority of UrjaMitras may spend their working hours at a shop or other place of business. Consequently, they may expect that they will sell Simpa systems largely to their existing customers. Sales to households who are not existing customers of UrjaMitras may therefore depend on either word of mouth or marketing support by Simpa. While the data on UrjaMitras’ perceptions of customer demand seems contradictory, it may be that their responses capture a general sense of optimism on their part rather than specific market insights. Approximately a year after they had had a system installed, the greatest number of customers (32%) chose “affordable installments” as the aspect that they were most satisfied with. 102 customers also said that they would recommend Simpa to others because it is affordable. The other aspects of the Simpa system that respondents said that they were most satisfied with were “no need to travel for purchase or service” and “free warranty and service”. These were chosen by 13% and 11.3% of respondents respectively. However, 13% said that they were not satisfied with any aspect of their Simpa system. When customers were asked specifically about the product and its installation, no major issues emerged. Overall, 68% of customers are willing to recommend Simpa to others. The two aspects of the Simpa offering that respondents were least satisfied with were customer service, and that the system cannot be used to run other appliances. However, overall, respondents were much more dissatisfied that the system cannot be used to run other appliances than they were with customer service. 45.2% of respondents said that the aspect of the Simpa offering that they were least satisfied with was that it cannot be used to run other appliances. (In contrast, only 7.6% of respondents chose, “customer service is not responsive” as the aspect of the Simpa offering that they were least satisfied with). Customers continue to use other options for lighting alongside their Simpa systems, often for the same activities. However, the use of “unclean” options (specifically candles / wax and kerosene lamps) dropped substantially. Correspondingly, the overall quality of health score improved substantially. On a scale of 0 to 6, respondents’ quality of health improved from 2.5 in the baseline survey to 5.3 in the midline. This difference is significant (T-test, p < 0.001). The score of 5.3 was also disaggregated for respondents who were classified as kerosene users (for lighting) at the time of the midline survey and those who were not. The quality of health score was 4.9 for kerosene users and 5.6 for non-users. This difference is also significant (T-test, p < 0.001). The other benefits that customers report as a result of their Simpa systems are greater satisfaction with and better quality light. On average customers rated their satisfaction with and the quality of light from their Simpa systems as higher than for most other solutions, for the same activities. However the average scores for satisfaction with and the quality of light from the Simpa system were on par with or lower than that from the inverter (again for the same activities). Customers’ preferences for their Simpa systems could also be because they were receiving fewer hours of electricity from the grid at the time of the midline survey than the baseline. The main findings of the midline survey of customers relate to how they value the Simpa offering and benefit from it, both in isolation and in relation to their experiences with other energy options. The main findings of the survey of UrjaMitras relate to how they value the Simpa offering, although they also contribute to a better understanding of the comparison between Simpa systems and other energy options. While the end term survey will further investigate what makes the Simpa offering affordable to customers, as long as payment in installments is a contributing factor it will be critical to make it easy for them to recharge and to make these commissions attractive for UrjaMitras. Accommodating the demand for systems that can run more appliances may lead to a trade-off with affordability for customers in the future. Similarly UrjaMitras who want to earn greater rewards will have to increase their investments of time and effort proportionately, which may eventually lead to a trade-off with their existing activities. The value customers’ perceive from the Simpa offering, and therefore the ability of UrjaMitras to sell to them, is also shaped by other energy options. While UrjaMitras believe that inverters are the main competitors to Simpa systems it would seem that the former are still too expensive for many customers to own. However, in considering Simpa’s value proposition to customers, the cost comparison to inverters would seem like an additional factor that it is important to include. EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE Simpa Networks is a for-profit enterprise with an innovative solar-as-a-service business model. Simpa customers take home a solar energy system by making a small initial down payment, then purchase prepaid energy credits using a mobile phone in amounts they choose – mimicking the highly successful pricing model for prepaid mobile airtime. Simpa’s cloud-based software tracks and processes payments, delivering proof of payment to the customer via SMS. After demonstrating its innovative business model with 200 customers, Simpa sought support from USAID’s Development Innovation Ventures to scale up to 12,000 systems by the end of the program in May 2015. Both during the USAID-supported period and beyond, Simpa will monitor the number of people it provides access to clean energy. However, this program also provides the opportunity to evaluate its impact: of the effect of increased access to clean, reliable electricity on customers’ well￾being. The main indicators of customers’ well-being that Simpa has identified are their kerosene consumption, quality of health and satisfaction with energy. If after installing a Simpa system customers report that they have stopped or reduced their use of kerosene, and that they are consistently satisfied with their system, this would support Simpa’s plan to scale up. However if customers continue to use kerosene alongside their Simpa system, it may be that there is a demand for more energy and therefore larger Simpa systems. If customers report that they are not satisfied with their Simpa system, their feedback is expected to be used by Simpato improve their offering. In addition to its impact, this evaluation is of Simpa’s cost-effectiveness for customers, and its value proposition to its customers and UrjaMitras. UrjaMitras are “Energy Friends”, local village level entrepreneurs who are independent sales agents for Simpa. The findings related to Simpa’s value proposition to its customers and UrjaMitras are expected to be used in its sales and operations. The specific decisions that the evaluation is directed towards informing are, “Which potential customers should we target? How should we sell to them? Who should we target to recruit as UrjaMitras? Once recruited, how do we retain them?” The main audiences for this evaluation report are Simpa’s leadership team, and USAID. However, once the end term evaluation is complete communication products based on the results will be disseminated to rural households and SMEs, UrjaMitras, and participants in “energy access” forums. The means of dissemination will depend on the stakeholder. To UrjaMitras, there is an opportunity to disseminate the relevant evaluation results through Simpa’s sales training, which is becoming increasingly more rigorous, and its active management of the field force. The evaluator will identify the main messages from the evaluation that are relevant to UrjaMitras, and engage in a discussion with Simpa on the potential for incorporating them into existing sales and operations processes. If this is not possible, the evaluator will create 2 dissemination products for UrjaMitras. The evaluation results will be disseminated through UrjaMitras to rural households and SMEs. A presentation will be created shortly after the end term evaluation to disseminate the results to participants in “energy access” forums. This presentation will be accompanied by an executive summary of the evaluation report, to provide participants further information. In addition, two articles will be written that draw on both the baseline and mid-term evaluation data, and that can be published in relevant forums. One article will be for an audience familiar with energy access issues, and the other will be for a general audience. EVALUATION QUESTIONS This evaluation was designed to be conducted between May 2013 and May 2015. Over this 2-year period, the evaluation questions are: 1. Who are Simpa’s customers? What is the value proposition of Simpa’s solar model and energy services for them? 2. What are the impacts of the program? To what extent are these impacts attributable to Simpa? 3. What motivates UrjaMitras to sell? What is the value proposition of Simpa for them? 4. How cost-effective are Simpa’s solar model and energy services for customers? The mid-term evaluation focuses to a greater extent on questions 1 and 3 than 2 and 4. PROJECT BACKGROUND In India at least 400 million people are without reliable access to electricity. Without access to electricity, consumers and small businesses rely on traditional fuels, kerosene, candles and batteries. It is estimated that the off-grid segment is already spending $50+ billion per year on these sub-optimal and highly unsustainable energy solutions. There are many promising technical solutions available to customers today, ranging from small-scale solar lanterns, to pico-solar home systems, to larger installed solar home systems, to community scale solar or biomass electricity micro-grids, to solar-hybrid UPS/inverter solutions for the home and business. The problem is that these clean energy technologies almost always involve significant up-front costs and therefore must be financed. Simpa Networks sets itself apart from competitors by providing in-house customer financing. Payment can be spread out over a period of 28 months. Simpa’s solar home systems incorporate proprietary prepaid metering technology, which regulates the amount of energy that is supplied to a customer based on whether he has purchased prepaid energy. In the payment process, customers make cash payments to Simpa branded agents in their home region. Then, the agent sends an SMS to Simpa’s server informing it of the payment. A special code is then sent via SMS from the server to the customer, who enters the code into the meter. This code unlocks the meter for a certain amount of energy consumption. This process is repeated until the customer has paid for the system in full, at which point the system is permanently unlocked. Simpa’s in-house financing mechanism helps avoid a lengthy, complicated loan application process with financial institutions, for both Simpa and its customers, and, most important, makes its solutions affordable for a wider range of customers. During the 1st year after installation (or 1st 2 years if the payment period is 2 years), free doorstep maintenance is provided, including 1 routine visit and 2 unplanned visits. After that, customers have the option of purchasing a low-cost annual maintenance contract, with the same level of service. This after￾sales service offer is extremely convenient for customers and also ensures that the systems are well￾maintained, which, in turn, minimizes the risk of customer non-payment. Simpa’s results chain is as follows: Inputs Processes Outputs Outcomes Impact People Product – solar Capital Product integration Product development Training Risk assessment Sales After-sales service Research Systems installed Systems used / energy paid for and consumed Improvements in well-being of consumer households EVALUATION METHODS & LIMITATIONS In order to address the evaluation questions, the evaluation consists of two parts: an evaluation of Simpa’s customers, and of its UrjaMitras. For the evaluation of Simpa’s customers it was important to establish causality.BecauseSimpa is a for-profit social enterprise that sells solar systems, it was not possible to choose an experimental design in which customers and non-customers would be randomly assigned. A quasi-experimental design was chosen for the evaluation of Simpa’s customers, in which they were matched with non-customers. The baseline evaluation of customers was intended to measure their usage of, and satisfaction with, other energy solutions prior to installing a Simpa system. Simpa’s installation process did not support the creation of a sampling frame of approved customers, from whom a random sample could be selected. A rolling baseline of customers was designed, with a sample that would consist of the first 400 customers acquired in Uttar Pradesh state. The evaluation of UrjaMitras is through a pretest-posttest design. The comparisons that Simpa is most interested in drawing in this part of the evaluation are amongst UrjaMitras rather than with a group not associated with Simpa. Therefore, there is no comparison group against which UrjaMitras will be measured. Structured interviews were conducted with a purposive sample of UrjaMitrasto develop the baseline survey of UrjaMitras. A stratified sampling technique was chosen to identify the UrjaMitras to be surveyed. This report presents the results of the customer (midline) and UrjaMitra (baseline) surveys that began on the 13th of September, 2014. The evaluation was designed so that non-customers are included in the baseline and end-term evaluations, but not the midline evaluation. The chart below shows the relationship between the evaluation questions and methods used in the midline evaluation of Simpa customers, and baseline evaluation of UrjaMitras. It also includes the sample sizes for the treatment groups of customers and UrjaMitras. Evaluation Question Method Data Source Treatment Group Sample Size Comparison Group Sample Size Who are Simpa’s customers? What is the value proposition of Simpa’s solar model and energy services for them? Quantitative Survey 215 N/A What are the impacts of the program? To what extent are these impacts attributable to Quantitative Survey 215 N/A Simpa? What motivates UrjaMitras to sell? What is the value proposition of Simpa for them? Qualitative and Quantitative Interview and Survey 323 N/A How cost-effective are Simpa’s solar model and energy services for customers? Quantitative Survey 215 N/A Based on these sample sizes, the margin of error for the customer survey is 6.62% and is 5.07% for the UrjaMitra survey (95% significance). There are limitations to how conclusive the evidence from the midline survey is, due to the methods themselves, data quality issues and bias. It is suspected that inaccuracies in respondents’ answers and data falsification made quality issues the most serious of the limitations of the midline evaluation. From a non-random sample of surveys, 11% of the midline data is estimated to be false (although not proven). FINDINGS, CONCLUSIONS & RECOMMENDATIONS FINDINGS The questions that this evaluation addresses, that will improve Simpa’s implementation, are on its value proposition for UrjaMitras and customers. The questions that this evaluation addresses that will improve Simpa’s impact, are on customers’ reports of their satisfaction with and the quality of light from solar and non-solar sources, and their quality of health. While the evaluation of UrjaMitras does not address impact directly, as Simpa’s chosen distribution channel the ability to sell through UrjaMitras is a prerequisite in order for Simpa to create impact. The evaluation of UrjaMitras is of the value proposition of Simpa to them, and of their sales potential. The evaluation asked UrjaMitras about their propensity to adopt new energy products, as the hypothesis was that a substantial percentage of UrjaMitras would be innovators (the segment of the target market that will adopt because they want to be among the first to have a new product). One of the most frequently used Domain Specific Innovativeness-scales was included in the survey to categorize each respondent as an innovator, early adopter, the early majority, the late majority or a laggard. However, contrary to expectations the greatest number of UrjaMitras fell into the laggard category. 45.5% of UrjaMitras were found to be laggards, and the percentage in each category decreased continuously culminating in only 1.2% of UrjaMitras in the innovator category. Adoption Category Count % Innovator 4 1.2% Early Adopter 49 15.2% The Early Majority 58 18.0% The Late Majority 65 20.1% Laggard 147 45.5% Total 323 100.0% This finding on technology adoption may be corroborated by the fact that only 2.9% of respondents said that the statement, “I have brought something new to my customers” describes the most important benefit that they derive from being UrjaMitras. However, it is possible that the statement “I have brought something new to my customers” did not describe the most important benefit to UrjaMitras because solar technology is not new to their catchment areas. While the survey included questions on the extent to which solar technology was new for UrjaMitras and their customers, the responses were ambiguous. For 27.6% of UrjaMitras commissions and prizes were the most important benefit. “Producing a good impression” was chosen by 13% of respondents as the most important benefit. It is related to commissions and prizes in that both are benefits that accrue to the individual, but are distinct in that they emphasize the financial and social aspects respectively. While only 7% of respondents chose the opportunity for social development or service as the most important benefit, for 42% it was an additional one. This suggests that UrjaMitras do recognize that selling Simpa systems results in benefits that accrue to the village community, but that they consider these secondary to the individual benefits that accrue to them. Being an UrjaMitra is valued most for the financial benefits, but these are insufficient for it to be seen as a full-time activity. Instead, selling Simpa systems is viewed as a way to improve income and relationships with existing customers. This is evident from the responses to the questions on the benefits of being an UrjaMitra. 16.9% of respondents said that the statement, “When I provide my customers with an additional product it increases the goodwill they have for me” describes the most important benefit that they derive from being UrjaMitras. That selling systems can be done alongside other activities was chosen as an additional benefit by 76%. Most Important Direct Benefit Count Percentage Rewards through commissions and prizes 85 27.6% Increase goodwill with additional product 52 16.9% Produce a good impression 40 13.0% More work = more rewards 29 9.4% Productive use of my time 28 9.1% Increase in village reputation 21 6.8% Serving the society 21 6.8% Sell Simpa with my other activities 19 6.2% New product to customers 9 2.9% Any other 2 0.6% No advantage 2 0.6% Total 308 100.0% Benefits Count Sell Simpa with my other activities 245 Increase goodwill with additional product 198 Rewards through commissions and prizes 130 More work - more rewards 107 Serving the society 136 Produce good impression 138 Productive use of my time 94 Increase in village reputation 38 New product to customers 17 Any other 0 No Advantage 24 Total 1103 Note: Multi-choice multi response question Respondents were asked about both the direct and indirect advantages of being UrjaMitras, as well as whether they felt that there were any disadvantages associated with the opportunity. 75% of UrjaMitras said that they valued their role in the company because they believe it is the first step towards other opportunities with Simpa. It is possible that the “other opportunities” that UrjaMitras are interested in are those that would provide a fixed income. This is suggested by the fact that only 9.4% of UrjaMitras said that they value performance based payment, and 27% saw the lack of a fixed income as a disadvantage. However, the majority of respondents (78%) saw no disadvantage to being an UrjaMitra. Advantages to Associating with Simpa Count 1st step towards other opportunities 240 Attend Simpa events 171 Receive visitors from Simpa 100 No advantages 38 Other 21 Recognition by Simpa’s senior management for performance 8 Total Unique Count 318 Note: Multi-choice multi response question Disadvantages of Being an UrjaMitra Count No disadvantage 248 No fixed income 86 Dissatisfied customers affect my reputation 75 Due to some changes made by Simpa, I have not received the prizes that I believed that I would 40 Don’t receive my commissions and/ prizes on time 33 Financially not attractive 26 Others 1 Total Unique Count 320 Note: Multi-choice multi response question Respondents were surveyed soon after they became UrjaMitras to collect baseline data on how they perceive the opportunity, and on other factors that may affect sales performance in the future. While this baseline survey was not intended to predict sales performance, the responses merit consideration by Simpa as it recruits, trains and manages UrjaMitras. In addition to the benefits that they derive from being UrjaMitras, respondents’ current work patterns and their perceptions of customer demand were identified as factors that may influence sales performance in the future. The element of respondents’ work patterns that Simpa was most interested in was the extent to which UrjaMitras are mobile. It was found that UrjaMitras are largely not mobile, as 70% of respondents spend most of their working hours at a fixed location. Given that 61.7% of respondents also identified “increasing the goodwill that they enjoy with existing customers” as a benefit of being an UrjaMitra, this suggests that the majority of UrjaMitras may spend their working hours at a shop or other place of business. Consequently, they may expect that they will sell Simpa systems largely to their existing customers. Sales to households who are not existing customers of UrjaMitras may therefore depend on either word of mouth or marketing support by Simpa. While the data on UrjaMitras’ perceptions of customer demand seems contradictory, it may be that their responses capture a general sense of optimism on their part rather than specific market insights. When asked if the supply of electricity from the grid in their catchment villages will improve in the future, 75% said yes. However, given that the question did not specify a time period for which UrjaMitras should make their predictions, their responses constitute weak evidence. It is striking that 99% of UrjaMitras said that the biggest competitor to Simpa systems is the inverter. If the supply of electricity from the grid does improve, and if the biggest competitor to Simpa systems is the inverter, then this could affect Simpa’s sales adversely. The results from the customer survey in this report also include further analysis of the comparison between Simpa systems and inverters. When UrjaMitras were asked about demand for Simpa systems over the next 6 months, they did not predict the adverse effects that their other responses would suggest. 82% of UrjaMitras believed that demand would increase continuously. Although UrjaMitras were asked whether they believed they could earn commissions through selling systems, recharges, or both, this question may have been misunderstood and the responses were not clear. However, based on the responses to the other survey questions it seems that most UrjaMitras may not recruit new customers actively, especially beyond their existing customer base. In addition, only 2% of UrjaMitras believe that their effort is critical in whether there is a demand to recharge systems or not. Therefore, if Simpa expects UrjaMitras to drive new sales and recharges, this is an area that merits attention. Demand Prediction Count Percentage Will increase continuously 260 82.3% Will increase initially and then reduce 48 15.2% Will decrease continuously 8 25.0% Total 316 100.0% While UrjaMitras can play a role in ensuring that systems are recharged, customers’ own desires and ability to do so are also critical factors. The customer survey asked respondents about the value they derive from their Simpa system, which will influence their recharge behavior. The survey also asked customers about how using their Simpa systems have affected their satisfaction with, and the quality of, lighting in their house and health. Beyond their influence on recharge behavior, this data on the effects of using solar systems can aid Simpa in its’ efforts to improve customer satisfaction and well-being. In order to better understand Simpa’s value proposition for customers, the survey included both the Domain Specific Innovativeness-scale and questions on the aspects of the offering that respondents were most and least satisfied with. Customers were both asked specifically about the product and its installation, and about other design parameters in Simpa’s value proposition including the distribution channel, the pricing and the recharge process / payment channel. Customers were also asked if they would be willing to recommend Simpa to others and if so, for what reason. The responses to this question corroborated the data on the aspects of the Simpa offering that customers were most satisfied with. The Domain Specific Innovativeness-scale was intended to measure whether a desire to adopt a solar system because it is a new kind of energy product was one of the reasons that respondents chose to buy. While again it was assumed that solar technology was new to respondents, in the case of customers we know at least that only 1person was using another solar product (at the time of both the baseline and midline). As the table below shows, the survey found that customers were heterogeneous in terms of their adoption of new energy products. Adoption Categories Count % Innovator 0 0.0% Early Adopter 42 21.5% The Early Majority 61 31.3% The Late Majority 60 30.8% Laggard 32 16.4% Total 195 100.0% The remaining questions on Simpa’s value proposition were intended to measure customers’ perceptions 10 to 12 months after they had had a system installed. Approximately a year after they had had a system installed, the greatest number of customers (32%) chose “affordable installments” as the aspect that they were most satisfied with. 102 customers also said that they would recommend Simpa to others because it is affordable. While these results make it clear that affordability is valued highly, it is difficult to use it to draw further conclusions on what affordability means to customers. More information is needed in order for Simpa to determine what to charge as a down payment and over time, and also to know whether customers would like the instalments to be more structured or more flexible. Given that only 15% of respondents said that any member of their household had held a salaried position in the last year, it is reasonable to hypothesize that customers would prefer more flexibility. However, these questions will be investigated further in the end-line evaluation. The other aspects of the Simpa system that respondents said that they were most satisfied with were “no need to travel for purchase or service” and “free warranty and service”. These were chosen by 13% and 11.3% of respondents respectively. However, 13% said that they were not satisfied with any aspect of their Simpa system. When customers were asked specifically about the product and its installation, no major issues emerged.64% of customers said that their systems run for 5 – 10 hours once the battery is charged. 79% said that the performance of the fan either meets or exceeds their expectations. 60% were satisfied or fully satisfied with the position of the lights and fan in their house. Overall, 68% of customers are willing to recommend Simpa to others. The two aspects of the Simpa offering that respondents were least satisfied with were customer service, and that the system cannot be used to run other appliances. Only 44% of respondents who had made a complaint were satisfied or fully satisfied with Simpa’s customer service. This finding is corroborated by the fact that only 6% of respondents rated service as the aspect of the Simpa offering that they are most satisfied with. However, overall, respondents were much more dissatisfied that the system cannot be used to run other appliances than they were with customer service. 45.2% of respondents said that the aspect of the Simpa offering that they were least satisfied with was that it cannot be used to run other appliances. (In contrast, only 7.6% of respondents chose, “customer service is not responsive” as the aspect of the Simpa offering that they were least satisfied with). Currently, 68 customers said that they use the Simpa system to charge their mobiles and 24 said they used it for other appliances. In most cases what these appliances were was not specified. However, from the baseline survey there is evidence that their homes are one of the significant assets that respondents own. All respondents have homes that are built either only of pucca(durable) materials or homes that are semi-pucca. Most respondents also have homes with either 3 or 4 rooms. Customers’ satisfaction with the affordability of their Simpa systems, and their desire to be able to use them to run more appliances, suggest that they may be willing to invest more in their homes. Aspect Most Satisfied With Count Percentage Affordable installments 67 31.6% No need to travel for purchase or service 27 12.7% Not satisfied 27 12.7% Free warranty & service 24 11.3% Separate part purchase not required 23 10.8% Immediate recharge upon payment 17 8.0% Other 14 6.6% Quick resolution of customer complaints 13 6.1% Total 212 100.0% Reason for Recommendation Count Affordable 102 Convenient to buy, recharge or use 84 Free warranty and servicing 74 Better quality than other solar companies 56 Other 1 Total Unique Count 199 Exhibit 9: Note: Multi-choice multi response question No. of Hours the System Runs Count Percentage < 5 hours 31 15.0% 5-10 hours 131 63.6% 11-15 hours 44 21.4% > 15 hours 0 0.0% Total 206 100.0% Fan Performance Count Percentage Exceeds expectations 57 26.8% Meets expectations 111 52.1% Below expectations 45 21.1% Total 213 100.0% Position of Lights and Fan Count Percentage Satisfied / fully satisfied 52 44.1% Neither satisfied nor dissatisfied 32 27.1% Dissatisfied / fully dissatisfied 34 28.8% Total 118 100.0% Aspect Least Satisfied With Count Percentage Cannot be used to run other appliances 95 45.2% Not dissatisfied 39 18.6% Cannot buy the parts separately 24 11.4% Long time to recharge after payment 22 10.5% Customer service is not responsive 16 7.6% Too expensive 9 4.3% Other 5 2.4% Total 210 100.0% Customers continue to use other options for lighting alongside their Simpa systems, often for the same activities.However, the use of “unclean” options (specifically candles / wax and kerosene lamps) dropped substantially. Correspondingly, the overall quality of health score improved substantially. In order to understand why respondents continue to use other options, the midline survey asked them to rate their satisfaction with and the quality of light from all the devices that they use. In order to ensure that the comparison was as accurate as possible, customers were asked for separate scores for each activity that they used the lighting device for. These activities were cooking, children’s studies, reading / writing by adults and eating dinner. In addition, customers were asked if they used their Simpa systems for any economic activities at home. The main option that respondents have access to for lighting is electricity from the grid. At the time of the midline survey 205 respondents reported having access to electricity from the grid, slightly more than the number of customers who said that they had used their Simpa system in the past 6 months (199). 15 more customers reported having access to electricity from the grid in the midline survey than in the baseline. However, the number of hours of electricity available decreased for 58% of customers. Perhaps not surprisingly, 8 more customers reported using inverters. There was an approximately 80% drop in the number of customers using candles / wax at the time of the midline compared to the baseline. There was an approximately 50% drop in the number of customers using kerosene for lighting. The average number of liters of kerosene used per month across all respondents was 5.5 at the time of the baseline survey and 2.62 at the time of the midline. This difference is significant (T- test, p < 0.001). However, those customers who continued to use kerosene did not use much less of it.If only respondents who reported using keroseneat the time of the midline are considered, the reduction in usage was only 0.15 liters(from 2.87 to 2.72) and is not significant (T-test, p > 0.6).The hypothesis that larger families continue to use kerosene was explored, but no pattern emerged. While not an “unclean” option, there was also an approximately 90% drop in customers using rechargeable LED torches (flashlights) from the time of the baseline survey to the midline. In addition, while at the time of the baseline customers were using LED torches for cooking, children’s studies, reading / writing by adults and eating dinner, at the time of the midline they were not using LED torches for any of these activities. It is likely that they were only using LED torches for activities that require a portable light. Given that the increases in the number of customers with access to electricity and inverters between the baseline survey and midline were only 15 and 8 respectively, and there are fewer customers using all of the other options, it is surprising thast 39% reported increases in expenditures on lighting (excluding payments for their Simpa system). Nevertheless, the average expenditure per month on lighting (excluding on the Simpa system) decreased by approximately INR 100 / USD 1.5, from the baseline figure of approximately INR 380 / USD 6. While analysis of the individual responses of the 39% of customers may reveal the sources of increased expenditure (and will be attempted to inform the end-line), it is also possible that the responses are inaccurate as expenditure data is difficult to collect. 190 15 107 151 168 205 23 20 17 82 199 0 50 100 150 200 250 Electricity Inverter Candles / Wax Rechargeable LED Torch Kerosene Lamp Simpa Solar System No. of Customers Energy Option Figure 1: Usage of Energy Options Baseline Midline Change in Expenditure on Lighting Count Percentage Increased 80 38.5% Decreased 126 60.6% No change 2 1.0% Total 208 100.0% While the survey attempted to estimate kerosene consumption from questions on the number and types of lamps used, their carrying capacities and how long they are lit for, it is difficult to collect accurate data on respondents’ usage of and expenditure on energy options other than Simpa. For this reason, and because it was beyond the scope of the survey, respondents were not asked about their use of “unclean” options for cooking. While we therefore cannot conclude that respondents’ overall usage of “unclean” options has reduced, a reduction in the use of “unclean” options for lighting does seem to correspond to improved health scores. On a scale of 0 to 6, respondents’ quality of health improved from 2.5 in the baseline survey to 5.3 in the midline. This difference is significant (T-test, p < 0.001). The score of 5.3 was also disaggregated for respondents who were classified as kerosene users at the time of the midline survey and those who were not. The quality of health score was 4.9 for kerosene users and 5.6 for non-users. This difference is also significant (T-test, p < 0.001). The other benefits that customers report as a result of their Simpa systems are greater satisfaction with and better quality light. On average customers rated their satisfaction with and the quality of light from their Simpa systems as higher than for most other solutions, for the same activities. However the average scores for satisfaction with and the quality of light from the Simpa system were on par with or lower than that from the inverter (again for the same activities). These activities were cooking, children’s studies, reading / writing by adults and eating dinner. While customers on average prefertheir Simpa systems for these activities, it does not seem that increased productivity is an additional benefit of the system (only 4 respondents said that they used their systems for economic activities at home). Customers’ preferences for their Simpa systems could also be because they were receiving fewer hours of electricity from the grid at the time of the midline survey than the baseline. This is corroborated by the fact that comparing the satisfaction and quality scores in the baseline and midline surveys reveals that customers’ perceptions of light from the grid have worsened in this time period. These differences are significant for both men and women, as the bar graphs below illustrate. 3.5 3.3 4.5 4.2 3.3 3.1 3.2 3.1 0 1 2 3 4 5 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Quality Score Activity Figure 2: Men's Average Quality Scores for Electricity Baseline Midline 3.8 4 3.8 4 3.2 3.1 3 3.1 0 1 2 3 4 5 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Satisfaction Score Activity Figure 3: Men's Average Satisfaction Scores for Electricity Baseline Midline This section of the report summarizes the impact data that was gathered throughthe midline evaluation and has been discussed above. In addition, it compares the baseline and midline values where relevant. All of the indicators are presented in table form, with the exception of “average score for satisfaction with lighting by situation (gender disaggregated)”. Bar graphs have been chosen to more effectively present the “average score for satisfaction with lighting by situation”, and have been included following the table. Customers’ experiences with using the Simpa system for cooling have been omitted from this table as it was expected that the values for this indicator would not be comparable due to 3.33 3.5 3.9 4.2 3.1 3.2 3.1 3.1 0 1 2 3 4 5 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Quality Score Activity Figure 4: Women's Average Quality Scores for Electricity Baseline Midline 2.9 3.9 4.4 4.1 3.1 3.1 3 3 0 1 2 3 4 5 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Satisfaction Score Activity Figure 5: Women's Average Satisfaction Scores for Electricity Baseline Midline seasonal variations. However, customers’ experiences with using the Simpa system for cooling were discussed earlier in this report. Indicator Year 1 Year 2 Average score for satisfaction with customer service N/A 4* Average quantity of kerosene purchased 5.5 liters 2.62 liters Average score for quality of health 2.5 5.3 Benefit most valued by UrjaMitras N/A Commissions and prizes Average expenditure on solar lanterns INR 1,268 (approximately USD 20)** N/A * 8.98% margin of error ** This figure reflects the cost of purchasing a solar lantern, and includes travel costs. It does not include the costs of repairing the lantern, which will be accounted for in the end term evaluation. 3.8 4 3.8 4 3.2 3.1 3 3.1 0 1 2 3 4 5 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Satisfaction Score Activity Figure 6: Men's Average Satisfaction Scores for Electricity Baseline Midline 4.2 4.3 4.2 4.4 4.1 4.2 3.9 0 1 2 3 4 5 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Satisfaction Score Activity Figure 7: Men's Average Satisfaction Scores for Inverters Baseline Midline 2.4 1.8 2.7 4 2.9 3.5 3 2.8 0 1 2 3 4 5 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Satisfaction Score Activity Figure 8: Men's Average Satisfaction Scores for Candles Baseline Midline 2.3 2.3 2.1 4 2.9 2.8 2.6 2.6 0 1 2 3 4 5 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Satisfaction Score Activity Figure 9: Men's Average Satisfaction Scores for Kerosene Baseline Midline 4.2 4.1 4.5 3.9 0 1 2 3 4 5 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Satisfaction Score Activity Figure 10: Men's Average Satisfaction Scores for Simpa Solar System Midline 3.06 4.2 3.04 3.3 2.72 4.08 0 1 2 3 4 5 Electricity Inverter Candles / Wax LPG/Petromax Kerosene Lamp Simpa Solar System Average Satisfaction Score Energy Option Figure 11: Men's Average Satisfaction Scores Across Activities Midline 2.9 3.9 4.4 4.1 3.1 3.1 3 3 0 1 2 3 4 5 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Satisfaction Score Activity Figure 12: Women's Average Satisfaction Scores for Electricity Baseline Midline 4.3 4.3 5 4.5 4.3 4.1 4 3.9 0 1 2 3 4 5 6 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Satisfaction Score Activity Figure 13: Women's Average Satisfaction Scores for Inverters Baseline Midline 2.1 2 4.4 3.1 4 2 3 0 1 2 3 4 5 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Satisfaction Score Activity Figure 14: Women's Average Satisfaction Scores for Candles Baseline Midline 2.4 2.1 2.9 3.9 2.9 2.8 2.5 2.8 0 1 2 3 4 5 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Satisfaction Score Activity Figure 15: Women's Average Satisfaction Scores for Kerosene Baseline Midline 4.1 4.1 4.5 4.1 0 1 2 3 4 5 Cooking Children's Studies Reading / Writing by Adults Eating Dinner Average Satisfaction Score Activity Figure 16: Women's Average Satisfaction Scores for Simpa Systems Midline CONCLUSIONS The main findings of the midline survey of customers relate to how they value the Simpa offering and benefit from it, both in isolation and in relation to their experiences with other energy options. The main findings of the survey of UrjaMitras relate to how they value the Simpa offering, although they also contribute to a better understanding of the comparison between Simpa systems and other energy options. In addition to the conclusions based on these findings, this section of the report also includes changes that will be made to the execution of the end term survey, to address the challenges that were faced in the midline. The sample of customers was planned to consist of the first 370 acquired in Uttar Pradesh state. Since these were the first customers, it was expected that they would be more likely to be innovators in technology adoption than later customers. It was also expected that UrjaMitras would be likely to be innovators in technology adoption, who would then convince customers of the benefits of solar technology. The surveys found that the first customers in Mathura were more likely to be innovators in technology adoption than UrjaMitras in the 5 sampled districts. The bar graph below compares the data that was presented earlier on the percentages of customers and UrjaMitras in each of the adoption categories. 3.05 4.075 3.025 2.333 2.75 4.2 0 1 2 3 4 5 Electricity Inverter Candles / Wax LPG/Petromax Kerosene Lamp Simpa Solar System Average Satisfaction Score Energy Option Figure 17: Women's Average Satisfaction Scores Across Activities Midline However, the technology element of Simpa’s value proposition is likely to be secondary to the financial for both customers and UrjaMitras. For customers it is the affordability of the system that is important, and for UrjaMitras it is the rewards that they receive. While the end term survey will further investigate what makes the Simpa offering affordable to customers, as long as payment in installments is a contributing factor it will be critical to make it easy for them to recharge and to make these commissions attractive for UrjaMitras. Yet, customers also stated that they want systems that can run more appliances. Accommodating the demand for systems that can run more appliances may lead to a trade-off with affordability for customers in the future. Similarly UrjaMitras who want to earn greater rewards will have to increase their investments of time and effort proportionately, which may eventually lead to a trade-off with their existing activities. The value customers’ perceive from the Simpa offering, and therefore the ability of UrjaMitras to sell to them, is also shaped by other energy options. In particular, the midline survey revealed from customers’ responses that there has been both a drop in the number of hours of electricity they receive and their satisfaction with this option. In the same time period, 8 customers have acquired inverters resulting in a total of 23 in the sample. For these 23 their average satisfaction scores with inverters were higher than the corresponding figures for all customers with Simpa systems. While UrjaMitras believe that inverters are the main competitors to Simpa systems it would seem that the former are still too expensive for many customers to own. However, in considering its value proposition to customers, the cost comparison to inverters would seem like an additional factor that it is important to include. It was important for the midline survey to investigate customers’ use of other energy options not only to understand how these influence Simpa’s value proposition but also because the potential health and environmental benefits of solar systems can only be realized when the 0.0% 21.5% 31.3% 30.8% 16.4% 1.2% 15.2% 18.0% 20.1% 45.50% -10.0% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% Innovator Early Adopter The Early Majority The Late Majority Laggard Adoption Category Figure 18: Customers and Urja Mitras by Adoption Category Customer UrjaMitra consumption of “unclean” fuels is reduced or stopped. Based on the available data, customers’ use of candles / wax and kerosene has dropped dramatically. This was corroborated by customers’ quality of health scores. However, these findings will be validated further through the end term evaluation. Validation will be very important in the end term evaluation because of the issues that arose during the midline. The midline had been designed so that phone surveys could be used to monitor the in-person data collection. 30% of customers and 20% of UrjaMitras were asked some of the same questions over the phone as they had been in person. The intention was that any falsification of the data collected in-person would be detected if it did not match the responses to the phone surveys. However through a detailed investigation process it emerged that the phone and in-person survey responses differed even when there was reliable evidence that the same person had been asked both sets of questions. The midline phone surveys were therefore not a reliable tool to detect whether the in-person data had been falsified. In order to address these issues, some of the questions in the end term evaluation will be modified to make them easier to answer. In addition, both the in-person and phone surveys are in the process of being pilot tested to check whether consistent answers can be expected from the same respondent. It is also planned to ask enumerators to provide the GPS co-ordinates of respondents and photograph the system IDs to make it very difficult to falsify data. ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK 01/05/2013 Terms of Reference for Evaluation of “Prepaid Energy” Program Intervention Background In India at least 400 million people are without reliable access to electricity. Without access to electricity, consumers and small businesses rely on traditional fuels, kerosene, candles and batteries. It is estimated that the off-grid segment is already spending $50+ billion per year on these sub-optimal and highly unsustainable energy solutions. There are many promising technical solutions available to customers today, ranging from small-scale solar lanterns, to pico-solar home systems, to larger installed solar home systems, to community scale solar or biomass electricity micro-grids, to solar-hybrid UPS/inverter solutions for the home and business. The problem is that these clean energy technologies almost always involve significant up-front costs and therefore must be financed. Simpa Networks is a for-profit enterprise with an innovative solar-as-a-service business model. Simpa customers take home a solar energy system by making a small initial down payment, then purchase prepaid energy credits using a mobile phone in amounts they choose – mimicking the highly successful pricing model for prepaid mobile airtime. Simpa’s cloud-based software tracks and processes payments, delivering proof of payment to the customer via SMS. Simpa’s results chain is as follows: Inputs Processes Outputs Outcomes Impact People Product – solar Capital Product integration Product development Training Risk assessment Sales After-sales service Research Systems installed Systems used / energy paid for and consumed Improvements in well￾being of consumer households Over the next two years Simpa plans to scale up sales of Solar Home Systems on a pay-as-you-go basis, and demonstrate the commercial viability of the model at sufficient scale to attract private market investment. Simpa plans to provide access to clean energy to 60,000 un- electrified people, mostly in Uttar Pradesh, in the next 2 years, and nearly 4 million people in 5 years. It is expected that the results of the evaluation will be used by Simpa to refine their value proposition, particularly in terms of the solar product, and their expansion plans. Evaluation Purpose and Indicators The purpose of the evaluation is to answer the following question: What are the impacts on customers, in terms of access to energy and its resulting benefits, which are attributable to Simpa? In addition to measuring the indicators below, the evaluation should also address financial/economic benefits, and whether health and education impacts are relevant. Indicator Data Source External Reporting Frequency Availability and/ quality of energy services Baseline and End-Term Evaluation At end of project Levels of access to energy services achieved Baseline and End-Term Evaluation At end of project Confidence in ability to seize opportunities provided by energy (gender disaggregated) Baseline and End-Term Evaluation At end of project Methodology The evaluation design will be quasi-experimental. The comparison group will be chosen from the districts in which Simpa plans to launch after May 1st, 2015. They will be consumers of kerosene, candles and/ D-cell batteries, who live in un-electrified homes. The evaluation design must include a mid-term and a final evaluation. The consulting firm is free to propose that a baseline study be conducted as well. However, it should keep in mind that unlike in the case of a non-profit organization, it is not possible for Simpa to select the treatment group. This is because as a for-profit, customers must choose Simpa (and not vice-versa). In addition, by 1st August, 2013, it is expected that only 220 Solar Home Systems will be sold in Uttar Pradesh. The evaluation should use a mixed method approach (i.e. both qualitative and quantitative tools). Collaboration and Communication The consulting firm will be accountable to an M&E consultant hired by Simpa. She will be involved in the design of the evaluation tools, the analysis of the data and co-authoring the evaluation reports. She may also choose to engage as a participant-observer in certain portions of the data collection. In addition to regular meetings with the M&E consultant, the consulting firm must also be willing to meet monthly with the Simpa team while the evaluations are being conducted. The purpose of these meetings will be to present progress and results, and receive feedback. The key deliverables and dates are below: Deliverable Date Evaluation Plan 27/05/13 Mid-term Evaluation 21/06/14 End-term Evaluation 20/04/15 However, if Simpa achieves its sales targets faster than planned, the dates for the mid-term and end-term evaluations will be brought forward. If this happens, the evaluation team will be given sufficient advance notice. Evaluation Team The ideal evaluation team would include thematic expertise in energy and social enterprise/market-based models, and functional expertise with both mixed method and integrated-cost approaches. Applications To be considered, please submit your proposed methodology, timeline and budget, as well as the qualifications of the evaluation team, to Devyani.oskarsson@gmail.com by the end of the day on Monday, May 6th. ANNEX II: EVALUATION METHODS AND LIMITATIONS A quasi-experimental design was chosen for the evaluation of Simpa’s customers, in which they were matched with non-customers. Non￾customers were asked to participate in an in-person survey if they resembled Simpa customers in terms of the education level of the chief earner, and either his / her occupation or the material used in the construction of the house. Once the surveys had been completed, customers and non-customers were matched using propensity score matching. Scores were generated using logistic regression. The qualifying variables considered for matching were age, gender, family members, education, earning members, marital status, school children, house type and rooms. The significant variables were age, family members, earning members and school children. A rolling baseline of customers was designed, with a sample that would consist of the first 400 customers acquired in Uttar Pradesh state. However, because of communication and coordination issues related to surveying each customer between approval and installation, the baseline survey was halted for a month before it was resumed. Customers acquired by Simpa in this month are therefore not part of the sample, and this may be a source of bias. The evaluation of UrjaMitras is through a pretest-posttest design. However, in this context in which UrjaMitras are acquired on an ongoing basis as Simpa expands into new districts, and will continue to sell once the grant period is over, the evaluation seeks to measure changes in motivations across time rather than before and after a discrete event. Therefore at the time of the baseline survey respondents had been UrjaMitras for varying lengths of time. Structured interviews were conducted with a purposive sample of UrjaMitras to develop the baseline survey of UrjaMitras. In order to construct the purposive sample, it was decided to compare a district Simpa had entered recently with one in which it had a longer presence. Hathras and Aligarh were chosen out of convenience because of their geographical proximity to each other. 5 UrjaMitras were initially selected in each district. UrjaMitras were chosen from 2 categories. The first category was those who have sold 2-4 systems in a month. The second category was those who have sold 5 or more systems in a month. The 2 categories were almost equally represented in each district. An additional UrjaMitra was interviewed in Aligarh. A stratified sampling technique was chosen to identify the UrjaMitras to be surveyed. The sample was selected so as to be representative of UrjaMitras in different districts and performance categories at the time of the end-term evaluation. The sampling frame was the 420 UrjaMitras who had made at least one sale in the past 4 months. This report presents the results of the customer and UrjaMitra surveys that began on the 13th of September, 2014. From an original sample size of 400 customers, 378 were matched and 215 were surveyed in the midline. The sample size of UrjaMitras was 323. Based on these sample sizes, the margin of error for the customer survey is 6.62% and is 5.07% for the UrjaMitra survey (95% significance). The drop from 378 at the time of the baseline to 215 at the time of the midline was because many customers were not easily reachable on the days on which the survey was scheduled. For the end line attempts will be made to reach the original 378 customers and improve the scheduling of the surveys. There are limitations to how conclusive the evidence from the midline survey is, due to the methods themselves, data quality issues and bias. For the evaluation of UrjaMitras a combination of qualitative and quantitative methods was the most appropriate as it would yield data on both the range of motivations for becoming an UrjaMitra, and the prevalence of each. For the evaluation of customers quantitative methods were considered the most appropriate as they would enable close-ended questions to be asked, and the gathering of quantitative data. Close-ended questions were used to ask about the specific parameters of the value proposition that Simpa had identified prior to the baseline and wanted feedback on. Quantitative data was required to compare customers’ satisfaction levels with a range of energy options, and to estimate their usage of kerosene and expenditure on energy (in order to determine impact and cost-effectiveness). However, given that much of the quantitative data that the evaluation sought to gather is difficult to recall, observational methods may have provided better quality data (but for a purposive sample). The midline had been designed so that phone surveys could be used to monitor the in-person data collection. 30% of customers and 20% of UrjaMitras were asked some of the same questions over the phone as they had been in person, after an interval of 2 months or less. The intention was that any falsification of the data collected in-person would be detected if it did not match the responses to the phone surveys. It is suspected that inaccuracies in respondents’ answers and data falsification made quality issues the most serious of the limitations of the midline evaluation.When through a detailed investigation process it emerged that the phone and in-person survey responses differed even when there was reliable evidence that the same person had been asked both sets of questions, the conclusion was reached that either the information being asked for was difficult to recall, or respondents did not appreciate the importance of answering accurately. This was the first issue that affected data quality. The second issue was falsification of the data. From a non-random sample of surveys, 11% of the midline data is estimated to be false (although not proven). This estimate is based on one common question in the in-person and phone customer surveys that provided consistent answers. There are four threats to internal validity of the evaluation of customers that were identified at the time of its design. The first is selection bias. Since Simpa customers must choose to buy a solar connection, they cannot be selected randomly. In addition, because customers will be acquired gradually, the first 400 customers were surveyed, rather than a random sample from the universe of customers. The second threat is mortality, or attrition. If customers who are dissatisfied with their Simpa system stop repaying, then it will be mostly those who are satisfied who are retained as customers after 2 years. This is likely to exaggerate the benefits of Simpa’s systems. The third threat that is related, but will have the opposite effect, is if customers minimize their usage of the system. They may then report the Simpa system to not be of much benefit to them, when in fact they haven’t used it enough. The fourth threat is that there is a lack of variability in some of the indicators such as quality of health. If customers do not perceive poor health due to poor lighting and kerosene usage to be a problem, they may downplay its effects. This will lead to a result that suggests the benefits of solar lighting are less significant than they are. ANNEX III: DATA COLLECTION INSTRUMENTS Customer Survey Good…………….! I am……………….. As you may recollect, we had surveyed you last winter on your experience with lighting solutions. This survey is being conducted across 2 years and we would like to speak to you again now. I would be grateful if you could spare some time to help us by answering a few questions. Please be assured that your name will not be associated in any way with the information you have provided, and that no one will contact you for selling anything to you based on this information.This is purely for our reference. Respondent agrees to be interviewed 1 CONTINUE Respondent does not agree to be interviewed 2 END Name of Simpa Customer Address Block Name Village Name Pincode Mobile No. 0 State Uttar Pradesh 1 District Mathura 1 NSCENER S Interview Details Date of Interview Interviewer’s Name Duration of Interview (In Mins) Starting Time of Interview Quality Check Accompaniment Back Checked Scrutinized Supervisor 1 1 1 Field Head 2 2 2 Instructions to Interviewer 1) The Head of Household (HoH) must be one of the participants. 2) For some questions responses from a male and female household member are required. These questions will be indicated with further instructions. Section 1: Lighting and Electricity No. Question Code Response Instruction Q1.1 Is the site electrified? Yes.......................................01 CONTINUE No........................................02 SKIP TO Q7.1 Q1.2 On average, how many hours of electricity are you receiving in a day at present? RECORD RESPONSE Q1.3 In the last month, what was the total household expenditure on electricity? RECORD RESPONSE Q1.4 Do you use a stabilizer to control power fluctuation? Yes......................................01 SKIP TO Q3 No........................................02 CONTINUE Instructions to Interviewer 1) Ask the following questions to the male respondent. No. Ask the following questions to the female respondent Questio n Q2 What are the activities that you do when you are using the grid supply / electricity / government supply for lighting? Where do these activities take place? How would you rate the quality of the grid supply / electricity / government supply to perform these activities? How satisfied are you with the light from the grid supply / electricity / government supply to perform these activities? How would you rate the quality of the grid supply / electricity / government supply to perform these activities? How satisfied are you with the light from the grid supply / electricity / government supply to perform these activities? Code Cooking..................01 Children’s studies.....................02 Reading / writing by adults................03 Eating dinner...............04 Kitchen...........01 Room..............02 In the open....03 Other..............04 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied..05 Very satisfied............04 Somewhat satisfied..03 Not very satisfied....02 Not at all satisfied....01 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied..05 Very satisfied.........04 Somewhat satisfied..03 Not very satisfied..02 Other (specify).....05 Not at all satisfied..01 Respons e 2.1.1 Cooking 2.1.2 2.1.3 2.1.4 2.1.5 2.1.6 2.2.1 Children’s studies 2.2.2 2.2.3 2.2.4 2.2.5 2.2.6 2.3.1 Reading / writing by adults 2.3.2 2.3.3 2.3.4 2.3.5 2.3.6 2.4.1 Eating dinner 2.4.2 2.4.3 2.4.4 2.4.5 2.4.6 2.5.1 Other - 2.5.2 2.5.3 2.5.4 2.5.5 2.5.6 No. Question Code Response Instruction Q3 In the past 6 months have you used an inverter for lighting? Yes...........................................................01 CONTINUE No............................................................02 SKIP TO Q5.1 Instructions to Interviewer 1) Ask the following questions to the male respondent. No. Ask the following questions to the female respondent Questio n Q4 What are the activities that you have done when you used the inverter for lighting? Where do these activities take place? How would you rate the quality of the inverter to perform these activities? How would you rate the quality of the inverter to perform these activities? How would you rate the quality of the inverter to perform these activities? How satisfied are you with the light from the Inverter to perform these activities? Code Cooking................01 Children’s studies...................02 Reading / writing by adults.....................03 Eating dinner...............04 Other (specify).............05 Kitchen...........01 Room..............02 In the open....03 Other..............04 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied....05 Very satisfied.............04 Somewhat satisfied...03 Not very satisfied......02 Not at all satisfied......01 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied...........05 Very satisfied...04 Somewhat satisfied..03 Not very satisfied..02 Not at all satisfied..01 Respons e 4.1.1 Cooking 4.1.2 4.1.3 4.1.4 4.1.5 4.1.6 4.2.1 Children’s studies 4.2.2 4.2.3 4.2.4 4.2.5 4.2.6 4.3.1 Reading / writing by adults 4.3.2 4.3.3 4.3.4 4.3.5 4.3.6 4.4.1 Eating dinner 4.4.2 4.4.3 4.4.4 4.4.5 4.4.6 4.5.1 Other - 4.5.2 4.5.3 4.5.4 4.5.5 4.5.6 No. Question Code Response Instruction Q5.1 In the past 6 months have you used a generator for lighting? Yes..........................................01 CONTINUE No...........................................02 SKIP TO Q7.1 Q5.2 What is the fuel used for running the generator? Only kerosene......................01 SKIP TO Q6 Only diesel............................02 CONTINUE Both kerosene and diesel..03 CONTINUE Petrol......................................04 SKIP TO Q6 Q5.3 In the last month, how much did you spend on diesel to run the generator? RECORD RESPONSE Instructions to Interviewer 1) Ask the following questions to the male respondent. No. Ask the following questions to the female respondent Questio n Q6 What are the activities that you have done when you used the generator for lighting? Where do these activities take place? How would you rate the quality of the generator to perform these activities? How satisfied are you with the light from the generator to perform these activities? How would you rate the quality of the generator to perform these activities? How satisfied are you with the light from the generator to perform these activities? Code Cooking..............01 Children’s studies.................02 Reading / writing by adults...................03 Eating dinner...............04 Other (specify).............05 Kitchen...........01 Room..............02 In the open....03 Other..............04 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied......05 Very satisfied...............04 Somewhat satisfied.....03 Not very satisfied........02 Not at all satisfied.......01 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied..05 Very satisfied........04 Somewhat satisfied..03 Not very satisfied..02 Not at all satisfied..01 Respons e 6.1.1 Cooking 6.1.2 6.1.3 6.1.4 6.1.5 6.1.6 6.2.1 Children’s studies 6.2.2 6.2.3 6.2.4 6.2.5 6.2.6 6.3.1 Reading / writing by adults 6.3.2 6.3.3 6.3.4 6.3.5 6.3.6 6.4.1 Eating dinner 6.4.2 6.4.3 6.4.4 6.4.5 6.4.6 6.5.1 Other - 6.5.2 6.5.3 6.5.4 6.5.5 6.5.6 No. Question Code Response Instruction Q7.1 In the past 6 months have you used a lead acid battery for lighting? Yes..............01 CONTINUE No..............02 SKIP TO Q9 Q7.2 In the past month, how much did your household spend on recharging your lead acid battery or batteries? RECORD RESPONSE Instructions to Interviewer 1) Ask the following questions to the male respondent. No. Ask the following questions to the female respondent Questio n Q8 What are the activities that you have done when you used a lead acid battery for lighting? Where do these activities take place? How would you rate the quality of the lead acid battery to perform these activities? How satisfied are you with the light from the lead acid battery to perform these activities? How would you rate the quality of the lead acid battery to perform these activities? How satisfied are you with the light from the lead acid battery to perform these activities? Code Cooking..............01 Children’s studies.................02 Reading / writing by adults............03 Eating dinner......04 Other (specify)..05 Kitchen...........01 Room..............02 In the open....03 Other..............04 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied......05 Very satisfied...............04 Somewhat satisfied.....03 Not very satisfied.........02 Not at all satisfied.........01 Very good..........05 Good...................04 Fair......................03 Bad......................02 Very bad.............01 Extremely satisfied...........05 Very satisfied.............04 Somewhat satisfied..........03 Not very satisfied............02 Not at all satisfied.............01 Respons e 8.1.1 Cooking 8.1.2 8.1.3 8.1.4 8.1.5 8.1.6 8.2.1 Children’s studies 8.2.2 8.2.3 8.2.4 8.2.5 8.2.6 8.3.1 Reading / writing by 8.3.2 8.3.3 8.3.4 8.3.5 8.3.6 adults 8.4.1 Eating dinner 8.4.2 8.4.3 8.4.4 8.4.5 8.4.6 8.5.1 Other - 8.5.2 8.5.3 8.5.4 8.5.5 8.5.6 No. Question Code Response Instruction Q9 In the past 6 months have you used a Simpa solar system for lighting? Yes................................................01 CONTINUE No................................................02 SKIP TO Q11.1 Instructions to Interviewer 1) Ask the following questions to the male respondent. No. Ask the following questions to the female respondent Questio n Q10 What are the activities that you have done when you used a Simpa solar system for lighting? Where do these activities take place? How would you rate the quality of the Simpa solar system to perform these activities? How satisfied are you with the light from the Simpa solar system to perform these activities? How would you rate the quality of the Simpa solar system to perform these activities? How satisfied are you with the light from the Simpa solar system to perform these activities? Code Cooking..............01 Children’s studies.................02 Reading / writing by adults............03 Eating dinner......04 Other (specify)...05 Kitchen...........01 Room..............02 In the open....03 Other..............04 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied......05 Very satisfied...............04 Somewhat satisfied.....03 Not very satisfied.........02 Not at all satisfied.........01 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied...........05 Very satisfied...........04 Somewhat satisfied..........03 Not very satisfied...........02 Not at all satisfied...........01 Respons e 10.1.1 Cooking 10.1.2 10.1.3 10.1.4 10.1.5 10.1.6 10.2.1 Children’s studies 10.2.2 10.2.3 10.2.4 10.2.5 10.2.6 10.3.1 Reading / writing by adults 10.3.2 10.3.3 10.3.4 10.3.5 10.3.6 10.4.1 Eating dinner 10.4.2 10.4.3 10.4.4 10.4.5 10.4.6 10.5.1 Other - 10.5.2 10.5.3 10.5.4 10.5.5 10.5.6 No. Question Code Response Instruction Q11.1 In the past 6 months have you used any solar product (other than Simpa) for lighting? Yes...............................01 CONTINUE No...............................02 SKIP TO Q13.1 Q11.2 How was this product bought? Cash up front..............01 CONTINUE Loan from bank...........02 Loan from MFI.............03 Instructions to Interviewer 1) Ask the following questions to the male respondent. No. Ask the following questions to the female respondent Questio n Q12 What are the activities that you have done when you used solar energy (other than a Simpa system) for lighting? Where do these activities take place? How would you rate the quality of solar energy to perform these activities? How satisfied are you with the light from solar energy to perform these activities? How would you rate the quality of the Simpa solar system to perform these activities? How satisfied are you with the light from solar energy to perform these activities? Cooking................01 Children’s studies...................02 Reading / writing by adults..............03 Eating dinner.......04 Other (specify)...05 Kitchen...........01 Room..............02 In the open....03 Other..............04 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied......05 Very satisfied...............04 Somewhat satisfied.....03 Not very satisfied.........02 Not at all satisfied.........01 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied...........05 Very satisfied...........04 Somewhat satisfied..........03 Not very satisfied...........02 Not at all satisfied...........01 Respons e 12.1.1 Cooking 12.1.2 12.1.3 12.1.4 12.1.5 12.1.6 12.2.1 Children’s studies 12.2.2 12.2.3 12.2.4 12.2.5 12.2.6 12.3.1 Reading / writing by adults 12.3.2 12.3.3 12.3.4 12.3.5 12.3.6 12.4.1 Eating dinner 12.4.2 12.4.3 12.4.4 12.4.5 12.4.6 12.5.1 Other - 12.5.2 12.5.3 12.5.4 12.5.5 12.5.6 No. Question Code Response Instruction Q13.1 In the past 6 months, have you used wax (candles) for lighting? Yes..................................................01 CONTINUE No...................................................02 SKIP TO Q15.1 Q13.2 In the past month, how much did your household spend on candles? RECORD RESPONSE Instructions to Interviewer 1) Ask the following questions to the male respondent. No. Ask the following questions to the female respondent Questio n Q14 What are the activities that you have done when you used wax (candles) for lighting? Where do these activities take place? How would you rate the quality of wax (candles) to perform these activities? How satisfied are you with the light from wax (candles) to perform these activities? How would you rate the quality of wax (candles) to perform these activities? How satisfied are you with the light from wax (candles) to perform these activities? Cooking................01 Children’s studies...................02 Reading / writing by adults..............03 Eating dinner.......04 Other (specify)...05 Kitchen...........01 Room..............02 In the open....03 Other..............04 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied......05 Very satisfied...............04 Somewhat satisfied.....03 Not very satisfied.........02 Not at all satisfied.........01 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied...........05 Very satisfied...............04 Somewhat satisfied..........03 Not very satisfied..............02 Not at all satisfied.............01 Respons e 14.1.1 Cooking 14.1.2 14.1.3 14.1.4 14.1.5 14.1.6 14.2.1 Children’s studies 14.2.2 14.2.3 14.2.4 14.2.5 14.2.6 14.3.1 Reading / writing by adults 14.3.2 14.3.3 14.3.4 14.3.5 14.3.6 14.4.1 Eating dinner 14.4.2 14.4.3 14.4.4 14.4.5 14.4.6 14.5.1 Other - 14.5.2 14.5.3 14.5.4 14.5.5 14.5.6 No. Question Code Response Instruction Q15.1 In the past 6 months have you used an LPG-based lamp (Petromax) for lighting? Yes....................................01 CONTINUE No.....................................02 SKIP TO Q17 Q15.2 How often do you fill gas in the lamp? RECORD RESPONSE Instructions to Interviewer 1) Ask the following questions to the male respondent. No. Ask the following questions to the female respondent Questio n Q16 What are the activities that you have done when you used an LPG￾based lamp for lighting? Where do these activities take place? How would you rate the quality of an LPG-based lamp to perform these activities? How satisfied are you with the light from an LPG-based lamp to perform these activities? How would you rate the quality of an LPG-based lamp to perform these activities? How satisfied are you with the light from an LPG-based lamp to perform these activities? Cooking................01 Children’s studies...................02 Reading / writing by adults..............03 Eating dinner.......04 Other (specify)...05 Kitchen...........01 Room..............02 In the open....03 Other..............04 Very good..........05 Good...................04 Fair......................03 Bad......................02 Very bad.............01 Extremely satisfied......05 Very satisfied...............04 Somewhat satisfied.....03 Not very satisfied........02 Not at all satisfied.......01 Very good..........05 Good...................04 Fair......................03 Bad......................02 Very bad.............01 Extremely satisfied...........05 Very satisfied...................04 Somewhat satisfied..........03 Not very satisfied..............02 Not at all satisfied.............01 Respons e 16.1.1 Cooking 16.1.2 16.1.3 16.1.4 16.1.5 16.1.6 16.2.1 Children’s 16.2.2 16.2.3 16.2.4 16.2.5 16.2.6 studies 16.3.1 Reading / writing by adults 16.3.2 16.3.3 16.3.4 16.3.5 16.3.6 16.4.1 Eating dinner 16.4.2 16.4.3 16.4.4 16.4.5 16.4.6 16.5.1 Other - 16.5.2 16.5.3 16.5.4 16.5.5 16.5.6 No. Question Code Response Instruction Q17 In the past 6 months have you used a rechargeable LED torch / light for lighting? Yes....................................01 CONTINUE No.....................................02 SKIP TO Q19 Instructions to Interviewer 2) Ask the following questions to the male respondent. No. Ask the following questions to the female respondent Questio n Q18 What are the activities that you have done when you used a rechargeable LED torch / light for lighting? Where do these activities take place? How would you rate the quality of a rechargeable LED torch / light to perform these activities? How satisfied are you with the light from a rechargeable LED torch / light to perform these activities? How would you rate the quality of a rechargeable LED torch / light to perform these activities? How satisfied are you with the light from a rechargeable LED torch / light to perform these activities? Cooking..................01 Children’s studies.....................02 Reading / writing by adults................03 Eating dinner.........04 Other (specify).....05 Kitchen...........01 Room..............02 In the open....03 Other..............04 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied......05 Very satisfied...............04 Somewhat satisfied.....03 Not very satisfied........02 Not at all satisfied.......01 Very good..........05 Good...................04 Fair......................03 Bad......................02 Very bad.............01 Extremely satisfied...........05 Very satisfied...............04 Somewhat satisfied..........03 Not very satisfied..............02 Not at all satisfied.............01 Respons e 18.1.1 Cooking 18.1.2 18.1.3 18.1.4 18.1.5 18.1.6 18.2.1 Children’s studies 18.2.2 18.2.3 18.2.4 18.2.5 18.2.6 18.3.1 Reading / writing by adults 18.3.2 18.3.3 18.3.4 18.3.5 18.3.6 18.4.1 Eating dinner 18.4.2 18.4.3 18.4.4 18.4.5 18.4.6 18.5.1 Other - 18.5.2 18.5.3 18.5.4 18.5.5 18.5.6 No. Question Code Response Instruction Q19 In the past 6 months have you used a kerosene lamp for lighting? Yes.............................................01 CONTINUE No..............................................02 SKIP TO Q22.1 No. Question 20 Which type of kerosene lamps do you use? What are the oil carrying capacities of your kerosene lamps? Once filled, how many days do they last? On average, how many kerosene lamps do your household light each day? On average, how many hours per day do you light kerosene lamps for? Code 5 days or less...........................01 6-10 days...................................02 11-15 days................................03 16-20 days................................04 21-30 days................................05 More than 30 days..................06 Response Wick 20.1.1 20.1.2 20.1.3 20.1.4 20.1.5 Hurricane 20.2.1 20.2.2 20.2.3 20.2.4 20.2.5 Instructions to Interviewer 1) Ask the following questions to the male respondent. No. Ask the following questions to the female respondent Questio n Q21 What are the activities that you have done when you used a kerosene wick / lamp for lighting? Where do these activities take place? How would you rate the quality of a kerosene wick / lamp to perform these activities? How satisfied are you with the light from a kerosene wick / lamp to perform these activities? How would you rate the quality of a kerosene wick / lamp to perform these activities? How satisfied are you with the light from a kerosene wick / lamp to perform these activities? Cooking................01 Children’s studies...................02 Reading / writing by adults..............03 Eating dinner...............04 Other (specify).............05 Kitchen...........01 Room..............02 In the open....03 Other..............04 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied......05 Very satisfied...............04 Somewhat satisfied.....03 Not very satisfied.........02 Not at all satisfied.........01 Very good...........05 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Extremely satisfied...........05 Very satisfied.............04 Somewhat satisfied..........03 Not very satisfied.............02 Not at all satisfied.............01 Respons e 21.1.1 Cooking 21.1.2 21.1.3 21.1.4 21.1.5 21.1.6 21.2.1 21.2.2 21.2.3 21.2.4 21.2.5 21.2.6 Children’s studies 21.3.1 Reading / writing by adults 21.3.2 21.3.3 21.3.4 21.3.5 21.3.6 21.4.1 Eating dinner 21.4.2 21.4.3 21.4.4 21.4.5 21.4.6 21.5.1 Other - 21.5.2 21.5.3 21.5.4 21.5.5 21.5.6 No. Question Code Response Instruction Q22.1 In the past 6 months have you used any other fuel for lighting? Yes (please specify)......................01 CONTINUE No...................................................02 SKIP TO Q24.1 Q22.2 In the past month, what was your expenditure on this fuel? RECORD RESPONSE Instructions to Interviewer 1) Ask the following questions to the male respondent. No. Ask the following questions to the female respondent Questio n Q23 What are the activities that you have done when you have used this fuel for lighting? Where do these activities take place? How would you rate the quality of this fuel to perform these activities? How satisfied are you with the light from this fuel to perform these activities? How would you rate the quality of this fuel to perform these activities? How satisfied are you with the light from this fuel to perform these activities? Cooking................01 Kitchen...........01 Very good...........05 Extremely satisfied......05 Very good...........05 Extremely Children’s studies..................02 Reading / writing by adults..............03 Eating dinner.......04 Other (specify)...05 Room..............02 In the open....03 Other..............04 Good...................04 Fair......................03 Bad......................02 Very bad..............01 Very satisfied...............04 Somewhat satisfied.....03 Not very satisfied.........02 Not at all satisfied.........01 Good...................04 Fair......................03 Bad......................02 Very bad..............01 satisfied.........05 Very satisfied..04 Somewhat satisfied........03 Not very satisfied.........02 Not at all satisfied........01 Respons e 23.1.1 Cooking 23.1.2 23.1.3 23.1.4 23.1.5 23.1.6 23.2.1 Children’s studies 23.2.2 23.2.3 23.2.4 23.2.5 23.2.6 23.3.1 Reading / writing by adults 23.3.2 23.3.3 23.3.4 23.3.5 23.3.6 23.4.1 Eating dinner 23.4.2 23.4.3 23.4.4 23.4.5 23.4.6 23.5.1 Other - 23.5.2 23.5.3 23.5.4 23.5.5 23.5.6 No. Question Code Response Instruction Q24.1 Have you used the lights powered by your Simpa system for any economic activity at home? Yes.........................01 CONTINUE No.........................02 SKIP TO Q24.3.1 Q24.2 What is the activity? RECORD RESPONSE Q24.3.1 24.3.2 24.3.3 24.3.4 24.3.5 24.3.6 Which of the following appliances do you use your Simpa system for? Mobile...................01 Other (specify)...02 MULTIPLE CODES POSSIBLE No.Q25 Question To what extent do you agree with the statement that [STATEMENT]? Code Fully Agree…………………………………………………………………………………01 Agree..................……………………………………………………………………………02 Neither Agree Nor Disagree……………………………………………………………..03 Disagree…...................………………………………………………………………………04 Fully Disagree………………………………………………………………………………05 Statements Q25.1 In general, I am among the first in my circle of friends and relatives to buy a new type of energy product when it appears. Q25.2 If I heard that a new type of energy product was available in the store, I would be interested enough to buy it. Q25.3 Compared to my friends and relatives, I own many types of energy products. Q25.4 In general I am the first in my circle of friends and relatives to know the brands of the latest energy technologies. Q25.5 I will buy a new energy product even if I haven’t heard of or tried it before. Q25.6 I like to buy new energy products before other people do. SECTION 2:Health Concerns No. Question Code Response Instruction Q26.1. 1 26.1.2 26.1.3 26.1.4 Which of the following problems have you experienced in the past 6 months? (Record aided) Respiratory Infection....01 MULTIPLE CODES POSSIBLE Eye irritation due to smoke...............................02 Breathing problems.......03 None of them................04 Q26.2 In the past 6 months have you experienced any accident caused by using lighting sources such as kerosene, candles or a generator? Yes..................................01 CONTINUE No....................................02 SKIP TO Q27.1 Q26.3. 1 26.3.2 26.3.3 26.3.4 What is the nature of these accidents? Fire....................................01 MULTIPLE CODES POSSIBLE Severe burn.....................02 Tripping over lighting device..............................03 Other (specify)...............04 Q26.4 Have any of these accidents happened more than once in the past 6 months? If yes, record accident code and the number of times the accident happened. SECTION 3:Household Information No. Question Code Response Instruction Q27.1 Have any members of your household worked as labourers in the past 12 months? (Include members not present at the time of the survey) Yes..................................01 CONTINUE No..................................02 SKIP TO Q29.1 Q27.2 How many of your household members have worked as labourers in the past 12 months? RECORD RESPONSE No. Question Q28 For how many months was this person resident in this house in the past 12 months? What is the type of labour this person performed? How was this person paid? How much was this person paid per day for this work? How many days in a month did this person do this work? How many months in a year did this person do this work? Code Use 1 row for each household member. The total number of rows should be equal to the response to Q 27.2 According to the criteria, is this person a member of the household? Yes...................01 No...................02 Response 28.1.1 28.1.2 28.1.3 28.1.4 28.1.5 28.1.6 28.1.7 28.2.1 28.2.2 28.2.3 28.2.4 28.2.5 28.2.6 28.2.7 28.3.1 28.3.2 28.3.3 28.3.4 28.3.5 28.3.6 28.3.7 28.4.1 28.4.2 28.4.3 28.4.4 28.4.5 28.4.6 28.4.7 28.5.1 28.5.2 28.5.3 28.5.4 28.5.5 28.5.6 28.5.7 28.6.1 28.6.2 28.6.3 28.6.4 28.6.5 28.6.6 28.6.7 28.7.1 28.7.2 28.7.3 28.7.4 28.7.5 28.7.6 28.7.7 28.8.1 28.8.2 28.8.3 28.8.4 28.8.5 28.8.6 28.8.7 No. Question Code Response Instruction Q29.1 Have any members of your household held a salaried position in the past 12 months? (Include members not present at the time of the survey) Yes..................................01 CONTINUE No..................................02 SKIP TO Q30 Q29.2 How many of your household members have held salaried positions in the past 12 months? RECORD RESPONSE No. Question Q29 For how many months was this person resident in this house in the past 12 months? What is this person’s salary per month? How long has this person been working for this employer? Code Use 1 row for each household member. The total number of rows should be equal to the response to Q29.2 According to the criteria, is this person a member of the household? Yes...................01 No...................02 Response 29.1.1 29.1.2 29.1.3 29.1.4 29.2.1 29.2.2 29.2.3 29.2.4 29.3.1 29.3.2 29.3.3 29.3.4 29.4.1 29.4.2 29.4.3 29.4.4 29.5.1 29.5.2 29.5.3 29.5.4 29.6.1 29.6.2 29.6.3 29.6.4 29.7.1 29.7.2 29.7.3 29.7.4 29.8.1 29.8.2 29.8.3 29.8.4 No. Question Code Response Instruction Q30 Does your household cultivate its own land? Yes..................................01 CONTINUE No..................................02 SKIP TO Q32.1 No. Questio n Q31 What crops did you cultivate in the past 12 months? How many times did you harvest this crop in the past 12 months? What was the yield in quintals per harvest? What price did you sell the crop at per quintal? What were your expenditures per quintal? Code Respons e 31.1.1 Wheat 31.1.2 31.1.3 31.1.4 31.1.5 31.2.1 Rice 31.2.2 31.2.3 31.2.4 31.2.5 31.3.1 Pulses 31.3.2 31.3.3 31.3.4 31.3.5 31.4.1 Oil seeds 31.4.2 31.4.3 31.4.4 31.4.5 31.5.1 Potatoes 31.5.2 31.5.3 31.5.4 31.5.5 31.6.1 Sugarcane 31.6.2 31.6.3 31.6.4 31.6.5 31.7.1 Other 31.7.2 31.7.3 31.7.4 31.7.5 No. Question Code Response Instruction Q32.1 Have any members of your household worked in their own business in the past 12 months? (Include members not present at the time of the survey) Yes..................................01 CONTINUE No..................................02 SKIP TO Q34 Q32.2 How many of your household members have worked in their own business in the past 12 months? RECORD RESPONSE No. Question Q33 For how many months was this person resident in this house in the past 12 months? Code Use 1 row for each household member. The total number of rows should be equal to the response to Q32.2 According to the criteria, is this person a member of the household? Yes..............................01 No...............................02 What is the business that your household member(s) work in? (Response) How much does the business earn per month (i.e. earnings net of expenses)? (Response) When was this business started? (Response) Response 33.1.1 33.1.2 33.1.3 33.1.4 33.1.5 33.2.1 33.2.2 33.2.3 33.2.4 33.2.5 33.3.1 33.3.2 33.3.3 33.3.4 33.3.5 33.4.1 33.4.2 33.4.3 33.4.4 33.4.5 33.5.1 33.5.2 33.5.3 33.5.4 33.5.5 33.6.1 33.6.2 33.6.3 33.6.4 33.6.5 33.7.1 33.7.2 33.7.3 33.7.4 33.7.5 33.8.1 33.8.2 33.8.3 33.8.4 33.8.5 No. Q34 Question In the last month, what were your 3 most important expenses? What was the expenditure on each? Code Response 34.1.1 34.1.2 34.2.1 34.2.2 34.3.1 34.3.2 No. Question Code Response Instruction Q35.1 In the last month, to what extent were you able to meet the expenses that you mentioned? Fully.................................01 Partially............................02 Not at all..........................03 SINGLE CODE ONLY Q35.2 In the last month, what was your total household expenditure on kerosene? RECORD RESPONSE Q35.3 What were your household savings in the last month? RECORD RESPONSE Q35.4 Does anyone in your household have a bank account? Yes...................................01 No....................................02 SINGLE CODE ONLY Q35.5 Is anyone in your household currently repaying a loan? Yes...................................01 CONTINU E No....................................02 SKIP TO Q37.1 No. Q36 Question What is the monthly repayment for each loan? What is the source of each loan? What is the purpose of each loan? Code Response 36.1.1 36.1.2 36.1.3 36.2.1 36.2.2 36.2.3 36.3.1 36.3.2 36.3.3 36.4.1 36.4.2 36.4.3 SECTION 4:Feedback on Simpa No. Question Code Response Instruction Q37.1 Which of the following aspects of your Simpa system are you most satisfied with? (Record aided) I am not satisfied with any aspect of my Simpa system..................................................................01 I pay in instalments so it is affordable.................02 I don’t have to buy all the parts separately.........03 I don’t have to travel to buy the system, recharge or service it........................................04 Once I pay, my system recharges SINGLE CODE ONLY immediately....05 There is a warranty and servicing is free......06 Customer service resolves my complaints quickly..................................................................07 Other (specify)....................................................08 Q37.2 Which of the following aspects of your Simpa system are you least satisfied with? (Record aided) I am not dissatisfied with any aspect of my Simpa system.......................................................01 It is too expensive..............................................02 I cannot buy the parts separately....................03 I cannot use it to run other appliances (e.g. TV, fridge, another fan)......................................04 It takes a long time for my system to recharge after I pay.............................................................05 Customer service is not responsive..............06 Other (specify)....................................................07 SINGLE CODE ONLY Q37.3 How many hours does your system run for? RECORD RESPONSE Q37.4 How does this compare to your expectations? Exceeds expectations........................................01 Meets expectations............................................02 Below expectations............................................03 SINGLE CODE ONLY Q37.5 How does the performance of the fan compare to your expectations? Exceeds expectations........................................01 Meets expectations............................................02 Below expectations............................................03 SINGLE CODE ONLY Q37.6 How satisfied are you with the position of the Simpa lights and fan in your house? Fully satisfied.......................................................05 Satisfied...............................................................04 Neither satisfied nor dissatisfied.....................03 Dissatisfied...........................................................02 Fully dissatisfied...................................................01 SINGLE CODE ONLY Q37.7 Have you ever made a complaint to customer service? Yes.......................................................................01 CONTINU E No........................................................................02 SKIP TO Q37.9 Q37.8 How satisfied are you with Simpa’s customer service? Fully satisfied.......................................................05 Satisfied...............................................................04 SINGLE CODE Neither satisfied nor dissatisfied.....................03 Dissatisfied...........................................................02 Fully dissatisfied...................................................01 ONLY Q37.9 Would you recommend Simpa to others? Yes.......................................................................01 CONTINU E No........................................................................02 TERMINAT E Q38.1 Q38.2 Q38.3 Q38.4 Q38.5 What would you tell others about Simpa? (Record aided) The instalment / credit model makes it affordable............................................................01 It is convenient to buy, recharge and get serviced................................................................02 There is a warranty and servicing is free.............03 Simpa provides higher quality than other solar companies...........................................................04 Other (specify)....................................................05 MULTIPLE CODES POSSIBLE Is there anything else that you want to tell us? USE BACK OF THE PAGE IF YOU NEED MORE SPACE Definition of household: A group of people who normally live and eat their meals together. For the purposes of this survey, “normally” is taken to mean that the person concerned has lived in the household for at least three of the past twelve months. 1. The only exceptions to be made to this rule should be for: (i) persons who are the main provider for the household; and (ii) newlyweds. 2. Servants, lodgers, farm-workers, and other such individuals who live and take meals with the household are to be counted as household members, even though they may have no blood relation to the household head. It is very important that you define the household membership strictly according to the criteria outlined above. These guidelines may not be the same as others you may be familiar with from other surveys, and at times they may not conform to the household’s own notion of who should be considered to be a household member. Please discuss any questions that arise in the field with your supervisor. UrjaMitra Survey We are conducting a study to understand your experience as an UM. We would be grateful if you could spare us some time. There are no right or wrong answers, and we would just like your honest responses. While we will be sharing some of this information with Simpa, please be assured that your name will not be associated in any way with the information you have provided. No-one will contact you to sell you anything based on this information. Respondent agrees to be interviewed 1 CONTINUE Respondent does not agree to be interviewed 2 END Name of UM Address Block Name Village Name Pincode Mobile No. 0 State Uttar Pradesh 1 District NSCENER S Interview Details Date of Interview Interviewer’s Name Duration of Interview (In Mins) Starting Time of Interview Quality Check Accompaniment Back Checked Scrutinized Supervisor 1 1 1 Field Head 2 2 2 No.Q1 Question To what extent do you agree with the statement that [STATEMENT]? Code Fully Agree………………………………………………………………………………01 Somewhat Agree…………………………………………………………………02 Neither Agree Nor Disagree……………………………………………………03 Somewhat Disagree………………………………………………………………04 Fully Disagree…………………………………………………………………05 Statements Q1.1 In general, I am among the first in my circle of friends and relatives to buy a new type of energy product when it appears. Q1.2 If I heard that a new type of energy product was available in the store, I would be interested enough to buy it. Q1.3 Compared to my friends and relatives, I own many types of energy products. Q1.4 In general I am the first in my circle of friends and relatives to know the brands of the latest energy technologies. Q1.5 I will buy a new energy product even if I haven’t heard of or tried it before. Q1.6 I like to buy new energy products before other people do. No. Question Code Response Instruction Q2 Before you first heard about Simpa, what was your exposure to solar energy? I had not heard of solar……………………………01 I had heard of solar but not seen any products…02 I had seen other solar products…………………03 I had used another solar product…………………04 SINGLE CODE ONLY Q3 In how many villages have you tried to sell a Simpa system? RECORD RESPONSE Q4 On average, how many hours of electricity per day do those villages get? RECORD RESPONSE Q5 Do you think their electricity supply will improve in the future? Yes…………………………………………………01 No…………………………………………………02 SINGLE CODE ONLY Q6 In how many of those villages had they heard of solar energy before? RECORD RESPONSE Q7 In how many of those villages had they seen a solar product before? RECORD RESPONSE Q8.1 Q8.2 As an UM, how can you earn commissions from Simpa? Through selling systems…………………………01 Through recharges………………………………02 MULTIPLE CODES POSSIBLE Q9 Which of the following factors will most influence whether your customers recharge their systems in the future? (Record Aided) How much money they have at the time…………01 How well their system has been working…………02 How convenient it is for them to recharge………03 Changes in the weather…………………………04 How insistent Simpa and the UM are……………05 Other (specify)……………………………………06 SINGLE CODE ONLY Q10 What do you think is the biggest competitor to Simpa systems (solar or non-solar)? RECORD RESPONSE Q11 - Q11.1 Q11.2 Q11.3 Q11.4 Q11.5 Q11.6 Q11.7 Q11.8 Q11.9 Q11.10 Which of the following are the advantages to you of being an UM? (Record Aided) There are no advantages to being an UM…………01 I can sell systems alongside my other activities…02 When I provide my customers with an additional product it increases the goodwill that they have for me…………….03 The more I work, the more I am rewarded through commissions and prizes……………………………04 I am rewarded in the form of commissions and prizes…..05 It makes me feel good to benefit my customers and serve society………………………………………06 More people in my village and neighboring villages have a good impression of me……………………07 I like making productive use of my time…………08 The reputation of my village has increased because we have solar………………………………………09 MULTIPLE CODES POSSIBLE Q11.11 I have brought something new to my customers…10 Other (specify)……………………………………11 Q12 Of the advantages that you mentioned, which is most important to you? SINGLE CODE ONLY Q13 - Q13.1 Q13.2 Q13.3 Q13.4 Q13.5 Q13.6 Which of the following are the advantages to you of being associated with the Simpa company? (Record Aided) There are no advantages to being associated with the Simpa company………………………………01 Being an UM is the first step towards other opportunities with Simpa (e.g. sevamitra, RSA)…02 I enjoy attending events Simpa organizes for its team…..03 I like receiving visitors from Simpa’s offices in Bangalore and other places………………………04 I want to be known by Simpa’s senior management for my performance………………………………05 Other (specify)……………………………………06 MULTIPLE CODES POSSIBLE Q14 - Q14.1 Q14.2 Q14.3 Q14.4 Q14.5 Q14.6 Q14.7 Which of the following are the disadvantages to you of being an UM? (Record Aided) There are no disadvantages to being an UM……01 I do not receive a fixed income……………………02 Being a recharge agent is not attractive financially.03 When my customers are dissatisfied it affects my reputation………………………………………04 I don’t receive my commissions and/ prizes on time……..05 Due to some changes made by Simpa, I have not received the prizes that I believed that I would…06 Other (specify)……………………………………07 MULTIPLE CODES POSSIBLE Q14.8 If the Simpa offering remains as it is, what changes do you expect in demand over the next 6 months? (Record Aided) It will increase continuously………………………01 It will increase initially and then reduce…………02 It will decrease continuously………………………03 SINGLE CODE ONLY Q15 - Q15.1 Q15.2 Q15.3 Q15.4 Over the past 12 months, what are the types of work that you have done? Casual labor (farm and non-farm)…………………01 Salaried employment (specify position)…………02 Own farm activities………………………………03 Own business (specify)……………………………04 MULTIPLE CODES POSSIBLE Q16 When you are working, do you spend Travelling…………………………………………01 SINGLE most of your time travelling or at a fixed location? At a fixed location…………………………………02 CODE ONLY Q17 Approximately what percentage of your working hours do you spend on Simpa? RECORD RESPONSE Q18 How much agricultural land does your household own? RECORD AMOUNT AND UNIT Q19 What is your level of education? Illiterate…………………………………………01 Literate but without formal schooling……………02 Less than primary…………………………………03 Primary……………………………………………04 Middle……………………………………………05 Matriculate…………………………………………06 Intermediate………………………………………07 B.A./B.Sc…………………………………………08 M.A./M.Sc…………………………………………09 Professional degree………………………………10 Diploma……………………………………………11 SINGLE CODE ONLY Q20 What is your approximate age? RECORD RESPONSE Q21 Record gender of UrjaMitra. Male………………………………………………01 Female……………………………………………02 SINGLE CODE ONLY Q22 Is there anything else that you want to tell us? USE REST OF THE PAGE IF YOU NEED MORE SPACE UrjaMitra Interview Guide We are conducting a study to understand your experience as an UM. We would be grateful if you could spare us some time. There are no right or wrong answers, and we would just like your honest responses. While we will be sharing some of this information with Simpa, please be assured that your name will not be associated in any way with the information you have provided. No-one will contact you to sell you anything based on this information. Can we talk to you? Can we record our conversation? 1. How did you become an UM? 2. Why did you become an UM? 3. What are the ways in which you earn commissions from Simpa? 4. How has your experience compared to your initial expectations? 5. Has it been easy or difficult to earn commissions? 6. Do you think this will change in the future? Why or why not? 7. Are you interested in continuing to be an UM? If yes, why? 8. Other than being an UM, what is the other work you do? 9. Do you work from a fixed location, or do you travel? 10. Is there anything else you want to tell us about being an UM? ANNEX IV: SOURCES OF INFORMATION Persons interviewed 5 UrjaMitras in Hathras district, and 6 UrjaMitras in Aligarh district Project documents 2012 Application for Prepaid Energy Project by Simpa Networks Milestone Reports (since 2013) Comparison Of Simpa To Other Products, Simpa Networks 2014 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523