Impact Evaluation Baseline Report: Totohealth SPRING Monitoring and Evaluation – May 2017 Department for International Development SPRING Monitoring and Evaluation PO 7117 List of Partners • Research Plus Africa May 2017 Coffey International Development Ltd The Malthouse 1 Northfield Road Reading Berkshire RG1 8AH United Kingdom T (+44) (0) 1189 566 066 F (+44) (0) 1189 576 066 www.coffey.com Registered Office: 1 Northfield Road Reading Berkshire RG1 8AH United Kingdom Registered in England No. 3799145 Vat Number: GB 724 5309 45 This document has been approved for submission by Coffey’s Project Director, based on a review of satisfactory adherence to our policies on: • Quality management • HSSE and risk management • Financial management and Value for Money (VfM) • Personnel recruitment and management • Performance Management and Monitoring and Evaluation (M&E) Heidi Ober, Programme Director Signature: Impact Evaluation Baseline Report: Totohealth CASE STUDY: TOTOHEALTH SPRING MONITORING AND EVALUATION Abbreviations and Acronyms BPE Business Performance Evaluation IP Implementing Partner KDHS Kenyan Demographic and Health survey KSh Kenyan Shillings KPI Key Performance Indicator NGO Non-Government Organisation NCD Non communicable diseases PPI Progress out of Poverty Index SMS Short Message Service ToR Terms of reference USAID United States Agency for International Development USD United States Dollar CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MARCH 2017 Contents Executive Summary 1 Introduction 1 1.1 Introduction to Totohealth 1 1.2 Selection for Impact Evaluation 2 2 Methodology 3 2.1 Overall approach to Totohealth Impact Evaluation 3 2.2 Quantitative Telephone Survey 3 3 Qualitative in-depth interviews with key informants 5 4 Limitations 6 4.1 Design limitations 6 4.2 Data limitations 6 4.3 Non sampling and sampling error 6 4.4 Respondent profile 7 5 Results and Analysis 8 5.1 Reaching Adolescent Girls 8 5.2 Effectiveness 10 5.3 Impact 16 Overall Experience 26 Annexes 27 Annex A: SPRING Impact Evaluation Methodology Annex B: Totohealth SPRING funding Annex C: Totohealth March 2017 revised approach and provisional timetable Annex D: Interim feedback to Totohealth Annex E: Demographic Profile: Telephone Survey Annex F: Completed depth interviews CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 i Executive Summary Introduction and overview Totohealth is a Kenyan based provider of advice and support to expectant and new mothers. They were selected as part of the first SPRING cohort of businesses, and are the second of six impact evaluations conducted by Coffey International Development. The core business is an SMS service providing health and child development advice to expectant and new mothers. Subscribers receive SMS content determined by the term of the mother or the age of the baby (up to 5 years of age). For example, pregnant women will receive messages on safe delivery or antenatal care, whilst new mothers receive messages on breastfeeding and child development. With SPRING funding, Totohealth developed SMS content and helpdesk support appropriate to adolescent girls and undertook targeted referral activity to increase the number of adolescent girl subscribers to the service. Through SPRING, Fuseproject supported their general business development, and provided technical assistance in design and marketing of a maternity product (the Totobag). The baseline research includes a telephone survey of all (104) adolescent girl subscribers to the service and 24 semi-structured interviews with key informants. Key findings from the baseline research are summarised below. What works in reaching adolescent girls with the service? Hospitals and clinic visits presented the best opportunity to target young expecting/new mothers for Totohealth services, with 68% of subscribers first becoming aware of the service here. Many respondents also subscribed to the service at the hospital clinic, frequently with the assistance of a Totohealth representative, or subscribed remotely via SMS (51% and 39%, respectively). Only 45% of respondents were pregnant when they first became aware of Totohealth, and fewer still (39%) were pregnant when they subscribed. Less than half of girls became aware of and subscribed to the SMS messaging support while pregnant. This suggests that Totohealth struggles to identify and target expectant adolescent mothers with pregnancy and safe delivery messaging support. Accessing Totohealth services Phone ownership influenced how girls interacted with Totohealth, with respondents indicating that those using a shared phone had difficultly receiving messages frequently, saving messages to refer back to, and interact with Totohealth compared with girls that had their own phone.1 88% of all respondents reported reading the most recent SMS, and 69% were able to describe its content. Sixty eight percent of respondents shared the SMS messages, with almost half sharing the messages with their partners (47%), a friend (39%), or a parent or relative (20%). Sharing of SMS content among friends in particular, will be instrumental in growing subscriptions. Respondents found messages easy to understand, timely and using clear and appropriate languages. Many respondents spoke of saving their messages and referring back to them later. Depending on the age of their baby, when prompted, mothers recalled receiving the following: • 43 % of all respondents recalled messages on safe delivery and 40% on pregnancy health. • 77% that had delivered their baby recalled messages on child development and 72% on nutrition and breastfeeding, but fewer recalled messages on reproductive health (46%) or prevention of infectious and non-communicable diseases (45%). • Only 27% of respondents with babies > 6months recalled receiving messages on First Aid. 1 We had not anticipated the use of shared phones, which became noticeable amongst 12 key informants that subscribed between June – December 2016, where five informants made use of a shared phone compared with just two of the twelve in May 2016. Totohealth does not collect details of which subscribers used a shared phone. We will gather data on shared phone usage amongst the full sample at endline. . CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 ii All informants spoke highly of Totohealth advice, accessing support alongside (and sometimes instead of) health care professionals and family. When asked how much they trusted different sources if information, respondents stated they placed greatest trust in information from Totohealth (90% stating they trusted Totohealth a lot) and other healthcare professionals (84%), with friends trusted the least (18% trusted a lot). Fourteen percent of respondents reported they had used the helpdesk and all users spoke highly of the service as providing them with a source of reassurance regarding their baby’s development. The Helpdesk was perceived as less useful in emergency situations where an immediate response was required. This may be an area where Totohealth could improve (or possibly better manage user expectations). Updated information related to the Totohelth SMS service Totohealth have confirmed that as of 26 March 2017, they will charge subscribers for use of the Heldpdesk (250Ksh for 6 months). When subscribers attempt to contact the Helpdesk, they will receive an automatic response informing them of the charge and requirement to pay before they receive an answer. Users were open to paying 200Ksh for the SMS service, but some felt that they would not have subscribed to the service if there had been a charge at the outset. As of December 2016, some new subscribers will be charged 200Ksh for the service. Totohealth have confirmed that this does not affect the majority of adolescent girls whose fee is covered by Kenyan County Government Health or NGO Donors. Health related behaviour amongst young and expectant mothers All key informants felt they had benefited from the information provided by Totohealth. When directly asked if their behaviour had changed as a result of receiving Totohealth SMSes, many informants felt it had, as stated by one: “Totohealth has told me how...to feed and take care of the baby [and my behaviour] is different now.” In the survey, we gathered behavioural data on the uptake of antenatal care; delivery; breastfeeding; immunization; family planning; and visiting health facilities. Where possible, this data has been triangulated against the Kenya Demographic and Health Survey. Antenatal care • 96% of all respondents and 97% of those that subscribed to Totohealth services whilst pregnant, received antenatal care from a skilled provider2 .This is comparable to the KDHS 2014 in which, 94% of 15-19 and 96% of 20-24 year olds received ANC from a skilled provider. • 65% of respondents that had subscribed to Totohealth services while pregnant felt that it had influenced their behaviour, including 26% who said it had influenced them to a great extent. Safe delivery • 96% of respondents that had given birth had delivered their baby in a health facility3 and 7% gave birth at home. This compared with 62% of 15-19 and 62.5% of 20-24 year olds who had delivered their baby in a health facility in the 2014 KDHS survey. • A third (33%) of respondents felt that Totohealth had influenced their decision on where to give birth during their pregnancy, and 51% felt it had not influenced them at all. Breastfeeding • All of the 81 respondents that had given birth had breastfed their baby. • 66% of respondents with babies over 6 months old had breastfed exclusively for 6 months. According to the KDHS 2014 findings, 61% of 15-49 year olds exclusively breastfed their baby for 6 months. 2 Doctor, nurse or midwife 3Note small sample size at baseline: n= 18 CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 iii • 64% of mothers who had delivered their babies felt Totohealth had influenced their breastfeeding, rising to 68% amongst those with babies > 6 months. At endline all mothers should have babies over 6 months old. Immunisation • Nearly all (96%) respondents that had delivered their baby had started taking their baby for vaccinations. The only respondents that had not taken their baby for vaccinations were mothers whose babies were days old. • 94% of respondents who had delivered had taken their baby for a BCG vaccine. All respondents whose babies were older than 2 weeks had taken their baby for the Pentavalent vaccine and nearly all had taken their baby for Pneumococcal (99%) and Polio (94%). • Most respondents (77%) had taken their baby for the Roto Virus and 81% for measles. According to the 2014 KDHS, coverage of children receiving the 1st dose of Pentavalent and polio stood at 98 and 97 percent, BCG (98%), measles (71%), while the third dose of Pentavalent and polio at 90% and 81%, respectively. • 68% of respondents stated their baby had been given every mandatory vaccination, 24% said their baby had not and 8% did not know. The KDHS survey measures how many babies aged 12-23 months have received basic or full vaccinations. Our evaluation will triangulate this with the uptake of Totohealth respondents’. At this stage, only 24% (n=25) of the Totohealth respondents’ babies were over 12 months. 69% of respondents that had delivered and 71% of those whose babies were >2 weeks olds felt that Totohealth SMS messages had influenced their decision to get their baby vaccinated. Family planning • 51 % of respondents that had delivered their baby were using some form of contraceptive. This compared with 14% of 15-20 year olds in the KDHS 2014 survey. • The most common contraceptive was the use of injectables (69%). Twelve percent of respondents were using implants and 12% were using the pill. Five percent were using a male condom and only 2% had been sterilised or were using an IUD. • Only forty one percent of respondents taking family planning felt Totohealth had influenced their decision and only 26% felt Totohealth had influenced them to a great extent. While respondents are more likely to use family planning than on average (as indicated by KDHS), fewer are likely to attribute their family planning decisions and behaviour to Totohealth SMSes than in other areas, e.g. immunization and breastfeeding. Visiting a healthcare facility Totohealth send messages encouraging users to visit healthcare facilities when they or their baby are showing signs of illness. In the last 12 months, respondents reported they visited a health care facility twice on average, 1.7 times for themselves and 3.2 times for their children. 30% of respondents thought they were visiting a healthcare facility more since receiving Totohealth messages, 37% thought it about the same and 26% thought it less. Perceptions of wellbeing Overall, few informants perceived their livelihoods overall had improved as a result of accessing the service. Insight: The impact of Totohealth is, by and large, limited to improved health care. Overall, while there were examples of how, through improving their childcare knowledge and practice, mothers had greater control over their lives, this was not perceived to be a key benefit of the service and few perceived their livelihoods in all had improved as a result of accessing the service. Wider Market None of the 24 key informants we spoke to were aware of any other similar assistance available for expecting and new mothers. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 1 1 Introduction This report presents the initial findings from the baseline research for Totohealth, the second of six grantee businesses selected for inclusion in the SPRING Impact Evaluation4 . The report is structured as follows: • Section 1: Introduces the Totohealth, outline its SPRING funded plans and explain its selection for impact evaluation. • Section 2: Provides and overview of the research methodology used to gather information to inform the report. • Section 3: Presents initial findings around: - How and why adolescent girls subscribe to the service; - The extent to which subscribers recall the information received; and - The influence of SMS content on the changed behaviour childcare, and the access, use and understanding of health services. Where appropriate and where possible we compare data from this survey with data from the Kenya Demographic Health Survey (KDHS). Where possible, we highlight differences at subgroup level (described in section 4.5) and report against logframe indicators. Annexes include the overarching SPRING Impact Evaluation framework; a tabular summary of the respondent profile; and research tools and templated used during data collection. 1.1 Introduction to Totohealth Totohealth is a Kenya-based start-up company servicing mothers and fathers during pregnancy and the first 5 years of their children's lives. The core business is an SMS and voice technology service for pregnant girls, women and young parents. The service sends SMS messages that provide health and child development advice based on the stage of the mother’s pregnancy or the age of the child. This includes a series of questions that aim to detect health conditions that the child or mother might have to help reduce maternal mortality, child mortality and detect developmental abnormalities in early stages. Before SPRING, approximately 15,000 users across six counties in Kenya were receiving the SMS service. With SPRING funding, Totohealth focused on targeting adolescent mothers and developing SMS content and helpdesk support appropriate to their needs. Examples include: For expecting mothers: “To prevent nausea never let your stomach get empty. Eat a high protein snacks (avocado, egg, cheese) and drink fruit juice, ginger/lemon tea, milk.” (healthy pregnancy) “You might go into labour before you reached 40weeks. Know where to deliver and who’s joining you. Have sanitary pads, clothes and diapers ready” (delivery) For mothers of new born babies: “Do not take any drugs, alcohol or cigarettes. It is unhealthy for your baby and through the breastmilk will end up in {your baby}’s body” (breastfeeding) “Next week {your child} must go for his 3rd vaccination. Ensure you take him because they save lives, but only when taken all and in time” (immunization) 4Details of the overarching framework of impact evaluation for the SPRING programme detailing: grantee selection, methodology are provided in Annex A. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 2 “Go to the doctor in case {your baby} is feeling hot/unusually cold, not suckling well, has sunken or swollen soft spot on head or yellow eyes” (non communicable diseases) “If {your son} is not yet creeping or crawling, reply with ‘no’ so we can give you some tips on how to stimulate him” (child development) And for mothers of older babies: “When {your son} plays he touches everything and unconsciously puts his hands in his mouth. Make hand￾washing a routine to prevent diseases” (hygiene) “People with illnesses such as cancer, diabetes and HIV deserve care. If you know someone who needs extra care, contact KEHPCA on 0722507219” (healthseeking) Prior to SPRING, Totohealth mainly received funding from NGOs for one-off SMS campaigns, but sought to introduce direct payment by parents at KSh 200/year ($2) and introduce a suite of maternity bags. The bags were intended to be sold on commission directly to women by Community Health Volunteers (CHVs) at prices ranging from KSh 1,400 ($14) to KSh 6,000 ($60). The core product (branded as Totobag) was intended to be sold at KSh 3,000 ($30) including 17 different items ranging from sanitary towels, to nappies and toys. CHVs are also doing commission-based outreach for the SMS service. 1.2 Selection for Impact Evaluation In consultation with the Implementing Partner Totohealth was selected for impact evaluation on the basis that it appeared to have good staying power, had a product that was proven to work in other geographies and could be easily scaled-up. On the basis that it already had c15,000 subscribers, it was deemed to have the biggest potential reach of the first cohort BPE grantees, and a good ability to generate data. Additionally, Totohealth’s digital component of SMS messaging was identified as a source of interesting lessons. Although consumer health and well-being are generally difficult to attribute, Totohealth’s model revolves around collecting health information for its customers, so its impact on girls’ health and well-being is evaluable and customers will be straightforward to identify and track. Totohealth also provides learning by enabling girls to understand more about their own health and their child’s development. Table 1: Areas of impact identified Areas of possible impact were identified: Effectiveness of reaching adolescent girls’: Attribution of SPRING support to Totohealth can be determined by the number of adolescent mothers and their children targeted by Totohealth before and after receipt of SPRING funding. Impact on adolescent girls’ learning: Totohealth provides learning by enabling girls to understand more about their own health and their child’s development. Attribution of Totohealth products and services to adolescent mothers can be evidenced by mapping subscribers’ knowledge of child development and self-confidence before and after receiving SPRING products and services, and by using a comparison group. Impact on adolescent girls’ wellbeing: Totohealth’s SMSes help adolescent mothers identify routine, serious, and potentially life-threatening ‘triggers’ of conditions impacting the development of their children. The most common conditions identified are motor difficulties, but triggers also cover malnutrition and rare. Impact on adolescent girls’ wellbeing: Totohealth SMSes may directly impact adolescent girls’ wellbeing by increasing their health literacy and knowledge of pregnancy, pre and post-natal health and infant nutrition and development. It may increase the confidence of young mothers. Impact on adolescent girls’ wellbeing: Totohealth SMSes encourage subscribers to take up antenatal care, safe delivery of their baby, immunization, family planning and regular visits to healthcare facilities; as well as advise on good breastfeeding practice and child nutrition. This behavior may impact the health of mother and child. The Kenyan Demographic and Health survey (KDHS) also monitors this behaviour. Attribution of SPRING support to changing behaviour can be determined by comparing the behaviour of subscribers to Totohealth services to that of the overall population of adolescent girls and mothers. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 3 Further details on Totohealth’s SPRING funded activities are available in Annex B. 2 Methodology The evaluation was designed in consultation with Totohealth, the IP and Research Plus, our local research partner. The primary method of data collection is through telephone interviews with all adolescent girl service subscribers and follow-up semi-structured interviews with key informants. Before we focus our attention on the baseline methodology and data collection, we describe our overall approach to the Totohealth impact evaluation, including recent revisions to our design: 2.1 Overall approach to Totohealth Impact Evaluation At the start of our impact evaluation design, Totohealth identified a group of adolescent girls that had subscribed prior to SPRING funding and who had been receiving the standard messaging content sent to all subscribers. Following SPRING funding, from June 2016 onwards, Totohealth communicated the intention to undertake further referral activity to grow the number adolescent girl subscribers. The SPRING subscribers would receive messages tailored to adolescent girls which would be introduced in January 2017. Our intention was that the girls that subscribed prior to SPRING and who received general messaging content would form a control group, while those subscribing after SPRING and who received the tailored messaging content would form our intervention group. In May 2016 we carried out baseline research among adolescent girls that had subscribed pre-SPRING funding (control group) and in December 2016 - February 2017 we carried out our research among adolescent girls that had subscribed post-SPRING funding (intervention group). The research among both groups included quantitative telephone surveys with all subscribers and a series of follow-up semi-structured interviews with key informants. It was our intention that we would adopt similar research techniques to conduct midterm and endline data collection to track the progress of both control and intervention groups. In February 2017, Totohealth informed us that there had been internal confusion regarding the introduction of the SPRING funded adolescent girl SMS message service and that the targeted SMS content had been introduced in December 2015 for all subscribers 20 years and under. This meant that all of our research participants (both intervention and control) had been receiving the same SPRING funded SMSes and we no longer had a counterfactual. On this basis, and in light of the small number of completed interviews in both rounds of baseline data collection, we have revised our approach to consolidate all responses into a single baseline report, which will be replicated for the endline data collection.5 .Our data analysis considers differences in the demographic profiles of the two sub￾samples. At both baseline and endline we triangulate findings against the Kenya Demographic Health Survey6 . At endline we will identify and interview a comparison group of adolescent mothers from similar backgrounds and locations who have not used Totohealth to form our counterfactual. By comparing the results between the two groups we will be able to assess the impact that using the adolescent-specific Totohealth has upon the girls compared with those who have never used the service 2.2 Quantitative Telephone Survey In May and December 2016 we conducted telephone interviews with all adolescent girls that had subscribed to the Totohealth services. Instrument Design We designed a quantitative survey to gather information from adolescent girls that had subscribed to the Totohealth service. The same survey was used for both the May and December 2016 fieldwork. 5 At endline we plan to identify and interview a suitable control group of girls who have not benefited from the service. This revised approach is detailed in Annex C 6 The 2014 Kenya Demographic Health Survey data was available for baseline reporting. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 4 The survey gathers baseline information on the following: • Questions about their experiences of subscribing to and using Totohealth to enable us to assess the effectiveness of Totohealth referral process, and • Specific health-related behavioural questions to enable us measure the impact of the Totohealth messaging: • Antenatal care and delivery • Breastfeeding and nutrition • Child development • Immunization • Family planning • Healthcare seeking The telephone survey was designed in consultation with and approved by Totohealth before we began the fieldwork process to ensure we understood the business and, as far as possible, gathered information that was useful to the business. Following approval, the survey was translated in Kiswahili and scripted for computer aided telephone interviewing (CATI). This ensured all questions were correctly routed. The full survey is available in Annex F Fieldwork Training The survey team was comprised of five female interviewers and a female supervisor. All were present for a one day training held on the 18th May at a community centre in Nairobi. The training day commenced with an outline of the project followed by a question by question review of the survey and the briefing notes. The interviewers then had the opportunity to familiarise themselves with the survey content and the tablets they would conduct the interviews on before conducting mock interviews with each other to test the survey script and routing. Piloting Following training, each of the interviewers then had the opportunity to pilot the interview with a respondent. A total 5 pilots were completed. Training and piloting highlighted the need for additional explanatory notes in the survey in the immunization section and the questionnaire was amended accordingly. Fieldwork Totohealth provided us with the names of 431 girls that had consented to be contacted. Of these we succeeded in interviewing 104 adolescent girl subscribers. Both the total number of girl beneficiaries and the number of completed interviews was less than we anticipated. Reasons for the low number of completed interviews included: • A large proportion (109) of the adolescent girls that had subscribed prior SPRING funding were over the age of 217 and so not eligible to be included. Following SPRING funding, Totohealth improved the accuracy of its database to address this issue. • A large proportion (135) of adolescent girls that subscribed post-SPRING funding were unavailable. This included instances where telephone was never switched on, or combinations of the phone being switched off, the number ringing out or a shared phone prevented us from reaching the right individual This is despite at least 6 repeat attempts to contact each number at different times of the day and different days. • Some respondents stated they were not subscribed to Totohealth services or did not recall having subscribed to Totohealth services but were receiving messages. 7As girls could have been 20 when they registered with Totohealth but had turned 21 since registration we decided to keep20 year old girls within the sample. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 5 The sample output report is detailed below: Table 2: Sample output summary Total May 2016 December 2016 Total initial sample provided 431 194 242 Aged 21 or over 119 109 18 Unavailable 139 20 1135 Wrong number/don’t recall registering 55 8 29 Refused 14 6 6 D 3 3 Toto baby > 5 years 1 1 0 Baby died 1 1 0 Completed interviews 104 49 55 3 Qualitative in-depth interviews with key informants Our qualitative baseline is comprised of 24 face to face key informant interviews with a selection of Totohealth beneficiary girls, 12 conducted in May –June 2016 and 12 in December 2016 – January 2017. We purposively selected the from the telephone survey sample to include a broad range of respondents according to the following criteria: • Income levels (using Progress out of Poverty Index). • Age of child subscribed with Totohealth to ensure we captured a range of child ages as targeted by the Totohealth service (SMSes content and messaging sequence is dependent on the age of the child.) • Use of Totohealth’s help desk to ensure we spoke to some of the minority of girls that had indicated in the survey they had used the helpdesk service. • Extent to which Totohealth had influenced behaviour to allow us insight into the factors that impacted on why girls might or might not feel influenced by Totohealth messaging. Instrument Design The qualitative discussion guide followed the structure of the quantitative survey but included detailed questions to enable us to explore the influence that the messages have had upon the girls in more depth including: • their impact on the girls’ knowledge, attitudes and behaviours; • reasons why the messages may not have had the desired impacts; and, • any unintended effects of the service. As per the telephone survey, the qualitative guide was designed in consultation with and approved by Totohealth before we began the fieldwork process to ensure we understood the business and, as far as possible, gathered information that was useful to the business. Following approval, the guide was translated in Kiswahili. The full guide is available in Annex G. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 6 Fieldwork Briefing and training A telephone brief with our local research partner was conducted on the 24th June 2016 and on the 30th June the agency undertook a half-day training with the two female interviewers who conducted all of the in-depth interviews. Fieldwork During July 2016 and January 2017, twenty-four key informant interviews were completed with a representative spread of respondents. The interviews were conducted in Kiswahili and fully transcribed into English. 4 Limitations 4.1 Design limitations The limitation of reliance on the businesses to provide information to inform the evaluation design and gain access to beneficiaries was identified at the outset of the SPRING programme. We attempted to address this by working closely with Totohealth, but as described in section 2.1, have had to revise our design in light of a recent change in information. The revised design has meant that we no longer have a control group at baseline and endline and are thus unable to measure attribution observed change in the intervention group to Totohealth. However, through a combination quantitative and qualitative research with the intervention group (at our baseline and endline) and a control group (at our endline) we will be able to assess the contribution of Totohealth to observed changes and to explain how and why intended changes have, or have not, happened. 4.2 Data limitations The impact evaluation collects its own data and so does not rely on Totohealth in this regard. However, data are largely self-reported by the girls themselves, subject to recall and positive response bias. To help identify any bias in our results we will triangulate some of this against data gathered by the grantees themselves. Robustness of the data is limited by the numbers of girls surveyed and final number of participants are still subject to refusal and attrition. We selected Totohealth for the evaluation anticipating that it would generate a large number of girl beneficiaries for the study. However, Totohealth has struggled to recruit the number of beneficiaries as planned. Using the contact details provided to us by Totohealth, we sought to maximise the number of participants by making multiple attempts to contact each girl and where necessary, arranging convenient times to call the girls back to take part in the interview if she was unable to at the time of the original call. However, as detailed in Section 2.2, due to poor quality of the contacts lists provided, we have not been able to successfully complete as many interviews as expected. 4.3 Non-sampling and sampling error The estimates from a sample survey are affected by two types of errors: non-sampling errors and sampling errors. Non-sampling errors are the results of mistakes made in implementing data collection and data processing, such as failure to contact adolescent girl subscribers, misunderstanding of the questions on the part of either the interviewer or the respondent, and data entry errors. Through comprehensive training, piloting, data cleaning we have tried to overcome non-sampling error in our data collection. However, it is to be expected that some non￾sampling error will be present which is difficult to estimate. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 7 Sampling errors, on the other hand, can be evaluated statistically. Using the population data Totohealth has provided us with, we are able to calculate the sampling, or margin of error8 for the survey. The margin of error for the total sample of adolescent girl subscribers is +/-7.4%. This means that if 50% of respondents responded to an SMS from Totohealth, there is a 95% probability that between 42.6% and 57.4% of all adolescent girl subscribers did respond to an SMS from Totohealth. The total sample will be surveyed again in further waves to create a longitudinal study. The total sample size of 104 individuals will allow us to detect a change over time of 0.3 standard deviations in any continuous variable with a confidence level of 95% and a statistical power of 90%.9 At baseline we also examine the differences between the two sub-samples (May 2016 / December 2016) to check for any significant differences across the main variables of interest. The current sizes will be able to detect a difference between the two sub-samples of 0.3 standard deviations across any continuous variable, or a change of 8 percentage points across any proportion, with a confidence level of 95% and a statistical power of 80%.10 4.4 Respondent profile Below we describe the respondent profile drawing on the subgroups used for data analysis11: Respondent age, education and household composition Forty one percent of subscribers were aged 16-18 and 59% were aged 19-20. Thirty eight percent of respondents had no schooling, 35% were educated up to lower secondary completed, 20% were educated to upper secondary completed and eight percent had higher education. Fifty three percent of respondents were married, 44% were single and 3% divorced. PPI Score The mean PPI score across all respondents is 55 and around half of the respondents live on less than $2.50 a day. According to the PPI score card, among respondents in Kenya with a PPI rating of 55 there is a 7% chance that the person lives on less than $1.25 a day and a 44%chance that the person lives on less than $2.50 a day. Length of registration Fifty five percent of respondents were subscribed with Totohealth for between 1 – 3 months, 26% were subscribed for 4-6 months and 19% subscribed for more than seven months. Age of child subscribed Nineteen percent of respondents were still pregnant, 28% had a baby aged 0-5months, 27% a baby aged 6-12 months and 24% a baby 12 months or more. 8 The margin of error is a statistical expression of the sampling error in a survey's results. It asserts a likelihood (not a certainty) that the result from a sample is close to the number one would get if the whole population had been queried. 9 Sample size calculations were made using the following Stata command: sampsi 0 0.25, n(104) sd1(1) method(change) pre(1) post(2) r1(.7). 10 The following Stata commands were used: sampsi 0 0.3, n1(55) n2(49) sd(1) alpha(0.5) power(0.8), and: sampsi 0 0.08, n1(55) n2(49) alpha(0.5) power(0.8). 11Detailed respondent profile is provided in Annex E CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 8 5 Results and Analysis In this section of the report we explore feedback from the surveys and in-depth interviews. We start by describing how respondents first heard of and subscribed for Totohealth services. We then explore respondent spontaneous and prompted recall of the Totohealth text messages, use of the Totohealth helpdesk, trust in Totohealth, as well as interest in and willingness to pay for services. 5.1 Reaching Adolescent Girls 5.1.1 Finding out about Totohealth More respondents (68%) found out about Totohealth while visiting a hospital than from any other source, as described during key informant interviews: “I went to hospital, [a Totohealth representative] approached me, we talked and I gave them my [details]... I registered that day... Since then we have been talking.” (informant 2) “When I attended antenatal check-up some people were at a front desk at the hospital and spoke to me about Totohealth, they explained what it was and took my details (informant 14) “I read a poster on the walls of the hospital, then I just activated on my phone.” (informant 7) Other ways of finding out about Totohealth included: through friends (20%), the internet or social media (16%), community referral (12%). Less frequently mentioned were home visits, where a Totohealth representatives or people known in the community, had actively sought subscribers door-to-door. Table 3: Finding out about Totohealth A.1 Where did you first hear about Totohealth? And where else? Total Base = all respondents 104 While visiting a hospital 68% From a friend 20% Information on the internet/social media 16% Through a Community Referral Centre/Kidogo Centre/meeting at Carolina for Kibera12 12% Marketing materials such as brochures 6% Home visit 7% Other 5% NB Multiple responses were permitted (only 20% of respondents provided more than 1 response) Respondents who were pregnant when they first heard about Totohealth totalled 45 per cent. 5.1.2 Registering for Totohealth Respondents were more likely to register while at hospital or by texting Totohealth’s number. Respondents that had subscribed before May 2016 were most likely to have subscribed via texting the Totohealth number (69%). Those that subscribed between June – December 2016 were most likely to subscribe during a hospital visit (76%). This is likely to reflect Totohealth’s change in recruitment strategy. With SPRING funding, Totohealth sought to increase the number of adolescent girl subscribers and actively targeted them, first through referrals and then through 12 Community referral centre = 16%; Kidogo centre = 8%;meeting at Carolina for Kibera = 10% CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 9 community health volunteers. Prior SPRING funding, Totohealth did not actively target adolescent girls to subscribe to the service. Table 4: Subscribing to Totohealth A.4 How did you register for Totohealth? Total (%) Base = all respondents 104 While visiting a hospital 51% Through text to the Totohealth number 39% Through a Community Referral Centre/Kidogo Centre/meeting at Carolina for Kibera13 5% Subscribed during home visit 5% Key informant interviews enabled us to explore the registration process in greater detail, with the following salient findings: Those registering in hospital or during a home visit spoke of assistance of a Totohealth representative (who some informants referred to as a doctor): “It was a doctor who told me about the Totohealth Program and the messages they send about raising a child... The doctor also took my number. That’s when I gave the doctor my number.” (informant 9) "Someone came around the camp, I asked him, and also other people, and they told me that it’s not a bad thing. I asked his wife... Her husband is the one that came here. And since the wife uses it, then I saw there was no problem" (informant 20) Assisted registration enables Totohealth to achieve immediate results. Similarly, speed and ease of registration is important in ensuring subscriber details can be captured on the move: “We were in a hurry because I was on the queue to get treatment so we didn’t talk much but they explained to me how they will be messaging me advices on bringing up my child.” (informant 2) However, there were some signs of a lack of clarity of what users were subscribing to at the outset14: "I did not know (that I had registered) ... I went to the hospital and the doctor asked me to write my name and my child’s name down ... he did not tell me why... and since then I have been receiving messages.” (informant 18) Encouragingly, despite some initial lack of clarity about the service, none of the respondents sought to unsubscribe. Indeed, many more recent subscribers sought more messages than they received. Recall and perceptions of SMSes is detailed in section 5.2.1. Insight: Hospital visits are the largest single source of information and registration to Totohealth services. Respondents who reported that they subscribed to Totohealth services after they had delivered their baby total 61%. In other words, only 39% of respondents were pregnant when they subscribed. Bearing in mind that 45% of respondents reported first hearing about Totohealth services when they were pregnant, this suggests a few respondents waited until they had given birth before subscribing. It also suggests that Totohealth has greater difficulty reaching women when they are still pregnant to provide them with antenatal and safe delivery support. Insight: Less than half of girls became aware of and subscribed to the SMS messaging support while pregnant. 13 Community referral centre = 16%; Kidogo centre = 8%;meeting at Carolina for Kibera = 10% 14 This was more notable amongst the SPRING subscribers, when Totohealth had introduced the Community Health Volunteers, who worked on a commission basis. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 10 Totohealth targets pregnant and new mothers, with SMS services available from mothers five months pregnant to children five years. Totohealth provides antenatal, nutrition and safe delivery messaging support for pregnant girls. However, of the girls we surveyed, only 46% became aware of Totohealth while pregnant, and only 40% were still pregnant when they subscribed. This suggests that Totohealth struggles to identify and target expectant adolescent mothers with antenatal, nutrition and safe delivery messaging support. None of the 24 key informants we spoke to were aware of any other similar assistance available for expecting and new mothers. Insight: Totohealth are the only providers of this type of service that respondents are aware of: None of the 24 mothers we spoke to in the key informant interviews were aware of any other similar assistance available for expecting and new mothers. 5.2 Effectiveness In this section of the report we explore respondent top of mind and prompted recall of the Totohealth SMSes, their trust in Totohealth, their use of the Totohealth helpdesk and their willingness to pay for Totohealth products and services. 5.2.1 Accessing Totohealth services: SMS recall Ninety three percent of respondents reported they were still receiving SMSes, and 88% had read the most recent SMS. Sixty nine percent of all respondents were able to describe the content of their most recent SMS. There was little difference in recall based on length of subscription (72% of those subscribed 4+ months recalling content compared with 66% of those subscribed 0-4 months). Insight: Sixty nine percent of all subscribers were able to recall the content of the last SMS they had received from Totohealth. We will assess whether this has improved between baseline and endline research, as a proxy to subscriber engagement in the service and adoption of the health advice. All of the 24 informants we spoke to perceived the SMSes were easy to understand, clear and relevant. Many informants cited that the use of Kiswahili was particularly helpful in enabling them to read and digest the messages. The use of simple words and timely messaging was also highlighted: “They send in the language that each person understands... the words used [and] messages are simple and easy to understand” (informant 12) “They capture the right language and they don’t complicate it... They are precise and they help you a lot” (informant 9) "They are easy to understand and straight forward and they tell me what to do when I need to" (informant 23) Insight: Adolescent girls perceive Totohealth messages as easy to understand, clear and relevant. All key informants were happy with the number of messages they received, no respondents sought fewer messages, though some sought more. This was particularly evident amongst informants that were more recent subscribers to the Totohealth service and those that had younger babies, as explained by an informant with a 12 month + old baby: “When the baby is young you have a lot of question but after one year the baby is okay.” (informant 1) Top of mind and most useful message recall During in-depth interviews, we asked informants to recall which messages had been most useful15 . . 15Message recall can be expected to reflect both the age of the registered baby and how long the mother has been receiving messages. Recalling that more respondents subscribed to the service after they had given birth, we can expect more feedback on messaging support relevant to young mothers (e.g. Messages on breastfeeding, immunisation, child development, nutrition, healthcare seeking and family planning and not those on antenatal care and safe delivery). CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 11 Overall, more informants spontaneously mentioned messages concerning their baby’s development and nutrition as most important to them. The most useful messages were perceived to be those that were about the baby’s stages of development and those that encouraged better care: “The [most useful messages are those] about how the baby grows and what he should be doing at certain stages, so when it reaches a certain time, I know what my baby should be doing, how he should be growing” (informant 12) “[The messages] give me details that I do not get at the hospital.... they make you be always on the alert before your baby moves on to the next stage of his life [so that] you already know what will happen” (informant 23) One informant alluded to the fact that these messages were most meaningful as they were about developments she as a mother could see: “They told me that at 10 months now, the baby should start making movements, should start responding to the things around her... I remember messages about the external development that I can see” (informant 6) Other messages helped guide expecting and new mothers on how to promote development, for example how to help the child when she is learning to walk: “Totohealth told me to hold my child’s hand so that she learns how to walk. After some two or three days she will learn to walk on her own.... [I did that] and she started walking on her own’” (informant 16) ....or how to handle teething: “I did not know anything about [teething], so I used to exert pressure on his teeth when my boy had pains not knowing what I am supposed to do. But now I know about it and I follow instructions according to the message when that problem arises I know what to do....” (informant21) ...and feed their baby a varied diet “I remember the ones advising me the baby is supposed to be eating these kinds of food, mash the foods, the baby needs to eat variety... so I decided to keep changing her die” (informant 9) Some informants that had subscribed when they were pregnant spontaneously recalled messages about pregnancy and safe delivery: "When my time came to deliver, they had sent me a message telling me that like a week before. My water broke when I was in the house at night and because of that message they had sent I knew exactly what to do" (informant 22) Insight: Messages on child development and nutrition support are particularly highly valued, though this is likely to be influenced by when expectant and new mothers subscribe to the service. Prompted recall Respondents were read a list16 of the topics of information that Totohealth sent SMSes about and asked which they remembered receiving. When prompted in this way, less than half of respondents recalled messages on safe delivery of their baby or on pregnancy health (43% and 40%, respectively). This is likely to be affected by the low number of mothers subscribing when they are pregnant, but provides an indication of the reach of messaging. Of those that had delivered their baby, more recalled receiving messages on child development & stimulation (77%); nutrition & breastfeeding (72%), hygiene (70%) and immunization (68%) than on reproductive health or prevention of NCDs (46% and 45% respectively). Few mothers with babies younger than six months recalled receiving messages on First Aid (27%). Message recall will be revisited at endline to identify what advice Totohealth is able to provide to its subscribers over a 12 month period. 16 As in the table below. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 12 Table 5: Totohealth SMSes - types of information received A5.8: What kinds of information have you received from Totohealth.? Total (%) Base =currently receiving SMSes 97 Safe delivery of baby 43% Pregnancy Health 40% Base = all receiving SMSes and delivered* 78 Child development & stimulation 77% Nutrition & Breastfeeding 72% Advice about hygiene 70% Reminders about immunization 68% Advice about parenting 58% Reproductive Health 46% Prevention of infectious and non-communicable diseases 45% Base = all receiving SMSes with babies > 6 months old 62 First Aid 27% Feedback from key informant interviews suggested that information about breastfeeding and immunisation was also provided by healthcare professionals. However, timely information about the baby’s development was less frequently accessed through other avenues. 5.2.2 Access to and use of SMSes Key informant interviews reflected different ways of accessing and using the SMS services, with some users receiving messages to their own phones while other users accessed the messages through shared phones. Many informants who had their own phones spoke of keeping messages to refer back to, both for themselves and to help others: “Sometimes when I have forgotten something or I want to confirm something for a friend, I will go through it [the messages] until I get that message" (informant 23) However, where informants shared a phone this impacted on their ability to access the SMSes as and when they wanted to and when the messages were sent. Those sharing phones tended to share it with their partner or family and recounted having to read and remember sometimes infrequently accessed messages: “Sometimes I may just have to depend on my memory because this phone is used by everyone in the house, it’s the only one we have. So maybe it’s within my reach or one of the others has it” (informant 7) "I do not read [some messages]... [for example, when] I leave the phone at the house the messages get in at that time [when I am not there].... [and] when I read the message... I forget easily". (informant15) On informant shared her phone with her friends and described how twice a week her friends would visit and access their Totohealth messages on her phone: “I have friends who also receive messages from Toto health but they do not have phones. What I do, every Monday and Thursday I give them my phone so that they can put their SIM cards and read their messages.” (informant14) CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 13 The use of shared phones is not something that Totohealth has a record of or was previously aware of. However key informant interviews afforded us an opportunity to gather information on this behaviour. Based on the small number of key informant interviews we undertook (24), our research suggests that the SPRING girls surveyed in December 2016 show a markedly higher use of shared phones than the pre-SPRING girls surveyed in May 201617 .. In December 2016, two out of twelve girls used a shared phone, compared with five out of twelve in May 2016. Insight: Phone sharing is higher amongst girls with lower PPI scores and may go some way in explaining the difficulty in contacting girls to take part in the baseline survey. Sixty eight percent of respondents shared the SMS messages with other people. Most shared the SMSes with their partners (47%), a friend (39%), or a parent or relative (20%). Sharing of SMSes among friends in particular, is important in growing subscriptions. “They help me to know a lot of things that I did not know. I get good advice and share with my friends that have children or are pregnant.”(informant 7) Recalling that only a fifth of respondents first heard about Totohealth through their friends, it suggests an opportunity to actively use word of mouth to further grow numbers of adolescent girl subscribers. Insight: Four in ten respondents shared messages with their friends. Actively targeting this message sharing may offer a route to growing adolescent girl subscribers. 5.2.3 Trust in Totohealth and other sources of information Respondents to the survey stated they placed greatest trust in information from Totohealth (90% stating they trusted Totohealth a lot) and other healthcare professionals (84%). The trust in Totohealth may be overstated as respondents seek to a halo/ interviewer effect, but suggests that Totohealth information is as highly trusted as that of health professionals. Table 6: Trust in different sources of information C1.5: How much do you trust the following sources of information Total “Trust a lot” (%) Base = all respondents 104 Totothealth 90% Other healthcare professionals 84% Mother 68% Media (radio, TV, website) 45% Other family members (Grandmother, sister) 36% NGO’s 32% Friends 18% Key informants reiterated a high level of trust in Totohealth information, citing the following reasons: • Totohealth was a reliable source of free, accurate, regular and relevant information: 17 Our findings coincide with independent research Totohealth undertook in November/December 2016. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 14 “I trust Totohealth because they are always there for me. I do not need to use my [data] bundles to Google information because they will offer it to me for free..... They are very informed and provide all information.” (informant 4) • Totohealth was a reliable prompt on how to care for their baby and source of reassurance on their baby’s development: Totohealth the most because the messages they send me, alert me on what to do and for sure when I go to hospital, I find that they are right... they advise me and help me, that’s why they are my number one.” (informant 9) “[Totohealth] give me real and practical advice very often and I even wonder how they know what my baby will do” (informant 15) • Totohealth was a source of impartial, up-to-date advice: "in the case of family members, the mothers may tell you some advice about taking care of the child but the advice is an outdated one unlike the one for Totohealth Program which offers different and trusting advice" (informant 19) 5.2.4 Helpdesk During our telephone survey, 27% of respondents indicated they were aware of the helpdesk function and half of these (54% (n=14) of all respondents) had gone on to use the helpdesk. Half of helpdesk users (n=8) recalled asking questions about nutrition or development of their baby, four recalled questions about their own or their babies health, and two could not recall. Most helpdesk users were satisfied with the service, 10 were very satisfied, three satisfied and only one slightly dissatisfied. In the 24 key informant interviews we explored helpdesk user experiences in greater detail. Our first observation is that more informants used the helpdesk than indicated in our survey. This comes despite the effort to improve respondent understanding of the term ‘helpdesk’ by introducing a more detailed explanation in the second tranche of telephone surveys. Insight: Helpdesk use may be higher than the survey suggests18 Key informant interviews indicated that helpdesk use might be higher than the survey reflected. Fifteen of the 24 key informants stated in telephone survey they had not used the helpdesk. However, during key informant interviews, five of the nine non-users went on to speak of SMS’ing Totohealth about concerns regarding their baby, some of them doing so on a regular basis. Overall, informants that had accessed the helpdesk were very satisfied with the support they had received. Some longstanding service users spoke of a longstanding relationship with Totohealth, and multiple occasions where the helpdesk and provided support, as illustrated in the following comments: “I asked them what kind of foods is good for calcium, and they told me.... There was a time she had diarrhoea and she was vomiting, I asked them and they told me that she is teething... I sent a message when my baby was not crawling and they told me to put stuff out of her reach and it worked...she had rashes and they said it was heat rash... “(informant 11) Another informant described an occasion where she had engaged in dialogue with the helpdesk when she was concerned her baby was not developing as it should, and the helpdesk had been able to advise her: “I was worried because [my baby’s] legs could not stretch at the knee. So I wrote [Toto] a message and explained.... they gave me advice on how [to massage] his legs slowly and after three day of doing this, his legs were okay. I was happy with the service... it took long to reply, but they finally sent responses that helped me.”(informant 8) And another described an instance where they had sought nutritional advice from the helpdesk: 18Totohealth themselves estimate helpdesk use at about 30% of all subscribed service users. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 15 “[Toto] told me... I can give her food...at first she did not eat ...[I SMSed and] they said it is because it is something new it is going to take a long time but she will eat eventually and to give her water at least twice or three times in a day.....the SMS made it easier for me..... (informant 1) On one occasion, accessing the helpdesk for a minor enquiry had also replaced a need to take their baby to hospital or clinic. It was evident that the helpdesk was best suited to non-emergency guidance, support and advice as service response is not immediate, as recounted by one informant: “If you text now they [Toto] respond after two days or day and maybe it is an emergency you want the answer immediately. So they should be a bit quick to respond to the messages (informant 3) This was the only criticism of the helpdesk – that response times were not always fast enough. One informant suggested Totohealth provide more explicit information about expected helpdesk lead times: I would say that they should…they should tell their members the specific days they should consult.... (informant 6) Insight: Users need to understand that the Helpdesk is for non-emergency situations.19 Where informants used their partner’s or family’s phone to receive messages from Totohealth (more typical of those lower down the poverty index) they might be less able to respond to messages or ask questions as illustrated in the following comment: “I don’t normally carry the phone .... But if I had the phone, I would probably have replied the messages or asked a question.” (informant 21) Update: March 2017 update from Totohealth Totohealth have confirmed that as of 26 March 2017, they will charge subscribers for use of the Helpdesk (250Ksh for six months). When subscribers attempt to contact the Helpdesk, they will receive an automatic response informing them of the charge and requirement to pay before they receive an answer. All Totohealth messaging has been revised to remove trigger questions and direct subscribers to relevant health professionals. 5.2.5 Future Sustainability - Paying for services The current revenue model for the SMS service is based on payments from development agencies and some county governments. All contracts are in their first year, which means in particular long term sustainability is unclear. Totohealth needs to find ways to scale up work with government counties or even national government eventually and/or implement end-user payments for SMS services. Paying for services During key informant interviews we explored respondent’s willingness to pay for the service. We first asked informants if they would be willing to pay 200Ksh per annum for Totohealth services now they had been using the service, and then we asked them if they would have paid 200Ksh to register for the service. Nearly all informants (20/24) indicated they would be happy to pay 200Ksh per year for Totohealth services, and felt this was an acceptable and affordable fee: “Yes, I would because they have helped me a lot and the money is no issue; according to the information they offer 200 is a little compared to what I gain” (informant 3) 19 This insight was reported back to Totohealth in an interim key findings feedback session. Full details of the summary note shared and discussed with Totohealth is provided in Annex D. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 16 "Because the service that you get there is not like any other since you get so much important information" (informant 22) However, some felt that they would not have subscribed to the service at the outset if they had to pay for it, suggesting that Totohealth should consider providing the service for free initially so that users understand what they are paying for before asking for a fee. Four informants we spoke to said they would not pay 200Ksh per year for Totohealth services largely due to financial constraints. However, one informant felt that as her baby grew older she felt she no longer needed their services. Insight: Users are happy to pay for the SMS service once they have experienced it. Key informant interviews suggested that most subscribers would pay for the service now that they had experienced the benefit of it. However, payment from the outset might be harder to achieve. Update: March 2017 update from Totohealth As of December 2016, new ‘referral’ subscribers (that contact Totohealth directly as opposed to through a community health worker) are charged 200Ksh to subscribe to the service. Totohealth has confirmed that most of the adolescent girl subscribers do not pay as they are funded by from existing Government (County) and NGO support. 5.3 Impact 5.3.1 Perceived behaviour change All informants felt they had benefited from the information provided by Totohealth. When asked directly if their behaviour had changed as a result of receiving Totohealth SMSes, many informants felt it had, as expressed by one: “Yes my behaviour has changed as Totohealth has told me how I am supposed to feed and take care of the baby [and it] is different now.” (Group 1; informant 4) “ Other areas where informants volunteered examples of behaviour change included: • Safe delivery – responding to SMSes on how to respond when their contractions were 10 seconds apart (but not as often advice on where to give birth, as some informants had already decided on this) • Breastfeeding and nutrition - responding to SMSes advising them to breastfeed exclusively for six months before introducing other foods, and to feed their baby a varied diet. • Development and growth – recounting SMSes that helped them understand what to expect at each stage of their babies growth, for example SMSes on what to expect during teething so that when their baby was crying they understood the cause. • Immunisation – recounting SMSes that reminded and prompted them to take their baby to clinic for various vaccinations that they otherwise would not have taken them for. • Hygiene – responding to SMSes teaching them the importance of washing their hands and keeping the house clean for their baby. However, not all informants felt Totohealth had influenced their behaviour, as illustrated in the following comment: “[Totohealth] has taught me things that I’ve never known as I look after my baby. It’s like a mirror to me where I look after myself what I have attained and what I have not...It was of importance to me as it enabled me to see whether the child has added weight as stated....but there is no difference. I would have raised him the same.” (informant 20) CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 17 5.3.2 Behaviour change In the survey, we gathered data on the uptake of: antenatal care, delivery, breastfeeding, immunisation, family planning and visiting health facilities as data points we could triangulate against the Kenya Demographic & Health Survey where possible to assess how Totohealth services improve girls’ wellbeing during pregnancy, birth and early child rearing. Antenatal care We asked respondents a series of questions about the Antenatal care they received. Our analysis consider all respondents (as all are theoretically eligible to receive messages on antenatal care) as well as singling those respondents that were pregnant when they subscribed to Totohealth services (as we the latter will have received messages about antenatal care.) Ninety six percent of all respondents and 97% of those that subscribed to Totohealth services whilst pregnant, received antenatal care from a skilled provider. According to the KDHS 2014 findings, 94% of 15-19 and 96% of 20- 24 year olds received ANC from a skilled provider. Table 7: Antenatal care B1: Where did you receive antenatal care?* Total (%) All pregnant when subscribed (%) Base = all respondents as stated 104 41 Government hospital 47% 44% Government health centre 22% 22% Government dispensary 14% 15% Private hospital / clinic 17% 22% Faith-based, church, hospital / clinic 5% 5% Your home 2% 2% Other home 1% 2% Nursing / maternity home 1% 2% Other 2% 2% *Multiple responses allowed Sixty five percent of respondents that had subscribed to Totohealth services while pregnant felt that it had influenced their behaviour. Of these, 65%, again, felt it had influenced them to a great extent. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 18 Table 8: Totohealth influence on place, frequency and timing of antenatal care B1.4a: Did the SMSes you received from Totohealth influence the place, frequency, and/or timing of ante-natal care for this pregnancy? Total All pregnant when subscribed (%) Base = all respondents as stated 104 41 Yes 36% 65% B1.4b: How much did SMSes you received from Totohealth influence the place, frequency, and/or timing of antenatal care for this pregnancy? Total All pregnant when subscribed (%) Base = those influenced by Totohealth SMSes (B1.4a) 37 26 To a great extent 73% 65% To some extent 22% 23% To a slight extent 5% 12% Delivery We asked respondents a series of questions about the where they delivered their baby. Our analysis considers all respondents that have given birth and only those respondents that were pregnant when they subscribed to Totohealth services and who have since delivered their baby20 . Ninety three percent of respondents that had given birth had delivered their baby in a health facility and 7% gave birth at home. This compared with 62% of 15-19 and 62.5% of 20-24 year olds delivered their baby in a health facility in the 2014 KDHS Survey. Table 9: Delivery B2.1 Where did you give birth ….? All who have given birth (%) All pregnant when subscribed + now given birth Base = all respondents as stated 81 18 Government hospital 55% 55% Private hospital / clinic 18% 22% Government health centre 9% 6% Your home 7% 6% Government dispensary 9% 11% Faith based church hospital 1% - Nursing/ maternity home 1% - 20At baseline, only 44% of those of respondents that had subscribed when pregnant had delivered their baby. This is a small sample size and indicative only. We expect that this sample size to increase by endline as more respondents that subscribed when pregnant will have delivered their baby. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 19 A third (33%) of all respondents felt that Totohealth had influenced their decision on where to give birth during their pregnancy, rising to 51% amongst those that had subscribed when pregnant and had now delivered their baby. Table 10: Totohealth influence on place of delivery B2.2 How much did the SMSes from Totohealth influence where you gave birth during this pregnancy? All who have given birth (%) All pregnant when subscribed + now given birth Base = all respondents as stated 81 18 To a great extent 21% 10% To some extent 6% 10% To a slight extent 6% 10% Not at all 51% 35% DK 16% 20% Feedback during key informant interviews suggested that whilst Totohealth helped expectant mothers deal with their pregnancy and identify when to visit the hospital, the decision on where to give birth had already been made or was influenced by other sources (family or other healthcare professionals). This is illustrated in the following extract of the conversation between interviewer and key informant (a mother of a week-old child): “I: Which SMS can you remember you read and responded to it? R: The message said that if I feel pain after every 10 minutes, I go to hospital because those are signs of labour. I: And how did you respond to it? R: when I started feeling the pains, I remembered the message and I knew I was in labour so my grandmother took me to hospital. I: How did the SMSes influence your decision about the delivery of your child? For example if they influenced you to deliver your baby in a hospital or clinic? R: They did not influence me. I: Why did it not influence you? R: I already knew I would go to hospital for delivery.” (key informant 24) Thus while Totohealth provided important information to help the expectant mother identify when she needed to go to hospital, it did not influence the decision to give birth in hospital. Breastfeeding Our evaluation considers how many Totohealth subscribed mothers exclusively breastfeed their baby for six months, which is the recommended length of time. At the time of baseline research, 51% of respondents had a baby over six months old (n=53). This is expected to increase by endline. Sixty six percent of respondents with babies over 6 months old had breastfed exclusively for 6 months. According to the KDHS 2014 findings, 61% of 15-49 year olds exclusively breastfed their baby for 6 months. A quarter of mothers with babies over six months old (n=13) had stopped breastfeeding their baby and had done so after an average 12 months. At endline we will revisit this measure. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 20 Table 12: Breastfeeding and weaning Total Base = all those with babies > 6 months 53 B3.2 % exclusively breastfeeding for first 6 months (%) 89% B3.2 mean length exclusive breastfeeding in first 6 months (mean) 5.6 B3.4 % Still breastfeeding (%) 75% Base = all mothers who have stopped breastfeeding 13 B3.4 At how many months did you stop breastfeeding? (mean) 12 Sixty four percent of mothers who had delivered their babies felt Totohealth had influenced their breastfeeding practice, rising to 68% amongst those with babies > six months. This was consistent across the subgroups. Table 13: Totohealth influence on breastfeeding B2.2 How much did the SMSes from Totohealth influence your breastfeeding practice? All who have given birth (%) All those with babies > 6 months (%) Base = all respondents as stated 81 53 To a great extent 47% 49% To some extent 11% 13% To a slight extent 6% 6% Not at all 28% 28% Don’t Know 8% 4% When discussing top of mind recall of any messages from Totohealth, key informants regularly recalled breastfeeding SMSes, both spontaneously and when prompted. They recalled how Totohealth had influenced how they weaned their baby and introduced other food into their baby’s diet, including: • How long to breastfeed exclusively for before starting their baby on other foods: “I would have given [my baby] water and milk had [Totohealth] not sent me that message about exclusive breastfeeding” (informant 14) • How long to continue breastfeeding for: “If I did not have Totohealth, I would have stopped breastfeeding my child at one year. But they told me to ...breastfeed exclusively for six months, and then continue breastfeeding up to two years. I can say that those messages made me continue to breastfeed, so I can say that they improved the situation.” (informant 9) • How to balance their baby’s diet during breastfeeding and weaning: “Yes the one that told me what foods to give my baby when weaning and breastfeeding was very useful, it helped me not to be giving the baby the same kind of food all the time and also to breastfeed her to compliment the foods” (informant 6) Health professionals at hospitals, clinics, and mothers were also regularly mentioned as informing and influencing their decisions on breastfeeding and immunisation. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 21 Insight: Respondents were no more likely to breastfeed exclusively for six months than the average of 15-49 year old mothers in the 2014 KDHS survey but recognise Totohealth as influencing how they wean their babies. Immunisation Our evaluation will consider how many of the Totohealth registered babies have been vaccinated, both basic and full vaccinations. Particular attention should be given to the age of the Totohealth registered baby. Nearly all respondents that had delivered their Totohealth subscribed baby had taken their baby for vaccinations. The only respondents that had not taken their baby for vaccinations were three mothers whose babies were just days old and who explained their reasons for not having started to immunise their baby: • No BCG in the hospital. • Gave birth in the house, so attending clinic the next day. • Told to go back to clinic to be immunised. Ninety four percent of respondents who had delivered had taken their baby for a BCG vaccine. All respondents whose babies were older than 2 weeks had taken their baby for the Pentavalent vaccine and nearly all had taken their baby for Pneumococcal (99%) and Polio (94%). Most respondents (77%) had taken their baby for the Roto Virus and 81% for measles. According to the 2014 KDHS, coverage of children receiving the 1st dose of Pentavalent and polio stood at 98 and 97 percent, BCG (98%), measles (71%), while the third dose of Pentavalent and polio at 90% and 81% respectively. Sixty eight percent of respondents stated their baby had been given every mandatory vaccination, 24% said their baby had not, and 8% did not know. The KDHS survey measures how many babies aged 12 – 23 months have received basic or full21 vaccinations and our evaluation will look to triangulate Totohealth respondents’ uptake of immunisation against this. At this stage, only 24% (n=25) of the Totohealth respondents’ babies were over 12 months. We will revisit this at endline when more Totohealth registered babies will have had their basic and full vaccinations. Table 14: Immunisation All who have given birth (%) Base = all respondents as stated 81 B4.3 A BCG vaccination against tuberculosis that is, an injection in the arm or shoulder that usually causes a scar? 94% Base = mothers with baby’s aged 2 weeks and over 68 B4.4 Polio vaccine, that is, drops in the mouth? 94% B4.6 A Pentavalent vaccination, that is, an injection given in the left outer thigh? 100% B4.7: A Pneumococcal vaccination, that is, an injection given in the right outer? 99% B4.8 A Rota virus vaccination given orally? 77% Base = mothers with baby’s aged 9 months and older 47 B 4.9 A measles injection at 9 months or older 81% 21 The Kenya Division of Vaccine and Immunisation (DVI) recommends that, by 12 months of age, children receive bacillus Calmette–Guerin (BCG), three doses of polio vaccine, three doses of a pentavalent vaccine (a combination vaccine comprising five vaccines, namely diphtheria, pertussis, tetanus, Hemophilusinfluenzae type b [Hib], and hepatitis B), and one measles vaccine. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 22 Sixty nine percent of respondents that had delivered and 71% of those whose babies were >two weeks olds felt that Totohealth SMSes had influenced them to get their baby vaccinated. Respondents whose baby was six months and older were more likely to feel that Totohealth had influenced them to get their baby vaccinated than those with babies under six months (72% v 39%22). Table 15: Totohealth influence on immunization B4.10 How much did the SMSes from Totohealth influence you to get [baby name] vaccinated? All who havegiven birth(%) All mothers with babies aged > 2 weeks (%) Base = all respondents as stated 81 68 To a great extent 52% 58% To some extent 13% 9% To a slight extent 4% 4% Not at all 28% 28% N/A 4% - Informants regularly recalled Immunisation SMSes as top of mind of any messages from Totohealth, both spontaneously and when prompted. Some informants recalled SMSes which prompted them to attend the clinic, as expressed by one: “It is so easy to forget sometimes and you find that you remember after the date has already passed. After you receive the text it just prompts you to take the baby for it.” (informant 4) Another recounted how the SMSes had alerted her to vaccinations that she otherwise would not have known about: “When the rubella vaccine was being given and when there are polio campaigns, they remind me... there are those times that I do not even know that there are vaccinations being given, so Totohealth lets me know and it helps.” (informant 11) Healthcare professionals and family members were also mentioned as helping to encourage users to immunise their children. This was particularly notable amongst Group Two informants. “My mum... and my big sister... told me that if I don’t take my child to be immunized it’s she that will be hurt... [Totohealth] only stated that I should not miss any appointments for immunization. When a certain date for immunization reaches, I am supposed to take my child to be immunized” (informant 19) Insight: Respondents were more likely to immunize their babies than the average of mothers in the 2014 KDHS survey and many respondents felt Totohealth has influenced them. Family planning All respondents that had delivered their baby were asked if they were currently doing something or using any method to avoid getting pregnant. Fifty one percent of respondents that had delivered their baby were using some form of contraceptive. This compared with 14% of 15-20 year olds in the KDHS 2014 survey23 . The most common contraceptive was the use of injectables (69%). Twelve percent of respondents were using implants and 12% were using the pill. Five percent were using a male condom and only 2% had been sterilised or were using an IUD. 22 Note small sample size n=26 23 Ibid CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 23 The most common reason for not practicing family planning were: • Too soon after giving birth (29%) • Not married (22%); • Don’t know/don’t know where to go (16%) • They/husband didn’t like it (7%) Only 41% of respondents taking family planning felt Totohealth had influenced their decision and only 26% felt Totohealth had influenced them to a great extent. Those with a lower PPI score were slightly more likely to feel that Totohealth had influenced their choice of birth control than those with a higher PPI score (47% v 36%). Table 16: Totohealth influence on family planning B5.3 How much did the SMSes from Totohealth influence you in using this type of birth control? All who have given birth (%) All practising family planning (%) Base = all respondents as stated 81 42 To a great extent 13% 26% To some extent 2% 5% To a slight extent 5% 10% Not at all 28% 57% Don’t know 1% 2% Key informant interviews gave insight into the lower perceived influence of Totohealth on respondents family planning choices, with a lower recall of family planning SMSes, both top of mind and when prompted. On prompting by reading out some of the SMSes, some recalled the messages but highlighted whilst the SMSes had informed them, they had not changed or influenced their behaviour: “[The SMSes] did not influence me in any way but they help me add more knowledge.”(informant 8) Others indicated that they had their own family planning in place and did not identify this as an area Totohealth should be addressing, perceiving Totohealth’s role as advising them on how to care for their baby. Insight: Respondents are more likely to take up family planning than 15-20 year old girls in the 2014 KDHS. However, they do not attribute this decision to Totohealth Visiting a healthcare facility Totohealth also send messages encouraging users to visit healthcare facilities when they or their baby are showing signs of illness. Our survey included a few questions about respondent use of healthcare facilities. In the last 12 months, respondents visited a health care facility twice on average 1.7 times for themselves and 3.2 times for their children. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 24 Table 17: Visiting a healthcare facility Total Base = all respondents 104 B6.1 In the last 12 months, about how many times have you visited a healthcare facility for care for yourself? 1.7 B6.2 In the last 12 months, about how many times have you visited a healthcare facility for care for your children? 3.2 Thirty percent of respondents thought they were visiting a healthcare facility more since receiving Totohealth messages, 37% thought it about the same and 26% thought it less. Table 18: Totohealth influence on healthcare seeking B6.3: Since receiving Totohealth SMSes, do you feel like you are visiting a healthcare facility more or less, or about the same?? Total Base = all respondents 104 More 30% About the same 37% Less 26% DK 8% Key informant interviews reflected a lower recall of health seeking behaviour SMSes, both top of mind and when prompted. Some informant s felt these messages were something they already knew so did not see the value of them: “They said if something is wrong, I should see a doctor. Isn’t that something I already know? (Laughing) It is just common sense. So I don’t think the ones I have been gotten are useful.”(informant 6) Friends and family were also identified as playing a role in health seeking. Some respondents also sited that the Totohealth advice and helpdesk support meant they required less support from other health professionals. Summary of behaviour change Below we have compiled a summary of Totohealth’s perceived influence on behaviour at baseline. Recognising the varying and sometimes small base sizes, the highest levels of perceived influence were recorded for immunisation, breastfeeding and antenatal care. Table 19: Summary of Totohealth SMSes stated influence their behaviour How much did the SMSes from Totohealth influence… To a great / some/ slight extent(Group 1) Base = varies B1.2 The place, frequency and timing of antenatal care (all those subscribed when pregnant n=41) 65% B2.2 Where you gave birth (all those subscribed when pregnant and delivered n=18) 52% B3.4 Your breastfeeding practice (all delivered n=81) 64% CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 25 B4.10 You to get [your baby name] vaccinated/immunized(all delivered n=81) 71% B5.3 How much did the SMSes from Totohealth influence you in using this type of birth control? (all doing something to avoid getting pregnant n= 42) 41% More Base = all respondents 104 B6.3:Since receiving Totohealth SMSes, do you feel like you are visiting a health care facility [more] 39% 5.3.3 Perceptions of wellbeing, life and employability skills Whilst Totohealth services target the health and development of pregnant/new mothers and their children and not their wider wellbeing, life and employability skills, we sought to establish what wider benefits might be achieved. This was consistent across the subgroups and will be revisited at endline. Table 20: Perceived wellbeing, life and employability skills How would you rate how well you are able to…. (very able, fairly able) Since subscribing to Totohealth SMS service, has your ability to ... increased Very able/Fairly able Base = all respondents 104 C3.1 Control my life(and make decisions about my life) 82% C3.2 Stay in or go back to school 67% C3.3 Care for yourself and your children /others in my household 87% C3.4 Manage new situations/ people I do not know? 76% For a few young mothers, Totohealth was able to provide exceptional and wide-reaching support: “I was wondering how to raise my child, but they came in and helped, they gave me courage to even go back to school... [Totohealth] really helped me pick myself up (informant 1) “[Totohealth] has grown my knowledge on how am supposed to nurture my baby... I feel like a fully grown woman now because of the information I get from them, you know they say information is power.” (laughing). (informant 15) Overall, however, few perceived their livelihoods in all had improved as a result of accessing the service. However, some identified the following areas of change: • Improving their knowledge, confidence and control in how to look after their baby • Helping them keep a cleaner home, and, • Being a better role model and parent to their child. On improving knowledge, informants cited being able to better control their lives through being able to care for their child: “No,[Totohealth does not help in any other areas of my life] ... only childcare....It helps in controlling your life because… you take care of your baby; it helps you take better care of [the baby] “(informant 10) "I can now make my own individual decisions with regard to bringing up [my baby] in the right manner." (informant 23) Some informants cited improved hygiene practice in their households: CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 26 “No, nothing has changed overall, but I can now make my own individual decisions with regard to bringing up [my baby]; and they have helped me maintain a clean environment in my home.” (informant 5) “{Totohealth] only talk about the baby and things related to the child....so… no it has not [changed my life in any other areas than my child]... It has changed the way I care for the household because they encourage hygiene to prevent the child from getting diseases. So now I am cleaner and I keep the house clean.”(informant 6) And some recounted where the SMSes had influenced how they behaved around their child, in being a good role model and providing a safe and calm environment: “There was a time they text me and tell me that she is a stage that she understands a lot so whatever behaviour she will imitate from me, it should be good. So you see if I have…bad behaviour, she will learn (informant 11) Insight: The impact of Totohealth is mostly limited to improved health care. Overall, while there were examples of how, through improving their childcare knowledge and practice, mothers had greater control over their lives, this was not perceived to be a key benefit of the service and few perceived their livelihoods in all had improved as a result of accessing the service. Overall experience Overall, all key informants recounted high levels of satisfaction, highlighting the following in particular: • A supportive and reassuring service and making frequent reference to the timeliness of messages: “When I got my child, I was wondering what to do. My mother is not always around. But when I started receiving SMS I realized some things normal in a child e.g. rashes. Right now I would be doing crazy things.” (informant 1) “It is a good program that is helping a lot people with information on how best to raise up their children and I pray to God that they may continue like that for a long time until the babies are big” (informant 23) • Tailored support when interacting with the helpdesk: “My overall experience has been good ...if [Totohealth] tell you that at that stage the child needs to be doing something or developing in a certain way you can ... and if your baby is not doing or developing as [Toto] are saying, you can send them a message and they will help you. It has been very good” (informant 12) CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 27 Annex A: SPRING Impact Evaluation Methodology Overview of the Impact Evaluation The Impact Evaluations are the core contribution to assessing SPRING’s impact on the lives of adolescent girls and the wider market for products, services and business models benefitting adolescent girls. They also play an important role in assessing SPRING’s additionality and value for money. Over the lifetime of SPRING, we will conduct a total six grantee impact evaluations. Each impact evaluation will be comprised of three phases: baseline, mid-term and endline. Business Selection Process The selection of the right businesses to include in the Impact Evaluation is crucial to ensuring that we can measure impacts of the programme. To help ensure that we selected the most appropriate business for inclusion in the Impact Evaluation we asked key members of the IP team to provide their opinions of the six Business Performance Evaluation (BPE) businesses against a number of criteria. The IP team members who provided their opinions included the SPRING CEO, the Programme Director, Technical Director, Fuse Project and the SPRING Investment Advisor. The criteria we asked the IP team to consider included: chances of business survival (so there is a good chance that the business will be operational at the endline stage); depth of impact (the impact upon girls that is measureable); breadth of impact (to ensure we can collect a good sample of beneficiary girls) and potential for learning (to help inform future programming). After reviewing the feedback of the IP team and our own assessment of the evaluability of the businesses, we decided, in agreement with the IP and with donor approval, to select Shekina and Totohealth for the Impact Evaluations. In this report we present the findings from the midline research for Shekina. The baseline findings for Shekina (and Totohealth) are held under separate cover. Approach to SPRING Impact Evaluation Methodology The SPRING impact evaluations are designed and tailored to the individual circumstances and business models of the selected grantee. However, all impact evaluation methodologies share several common features: • Focus on adolescent girls. Whereas the Business Performance Evaluation focuses on SPRING businesses as the unit of evaluation, the focus on the Impact Evaluation is adolescent girls, who, depending on the business approach, benefit as users of the product or service, by providing services, or working in the supply chain. • Use of local female interviewers. Adolescent girls may be hesitant to open up to male interviewers or people they don’t know. We use local female interviewers to build trust between interviewer and respondent and thereby improve the quality of response. • Mixed methods. We use both qualitative and quantitative methods to triangulate the impact of the SPRING business on adolescent girls. • Common impact indicators. We have developed a suite of common indicators for the impact areas of learning, earning, saving, safety, and well-being that will be asked of respondents in all cohorts so that we can compare results both within and between cohorts. We also use a suite of common impact indicators for women’s empowerment to measure impact across all enterprises. After each cohort we will review these common indicators to ensure they are still relevant. • Design in consultation with grantee, IP and local partners. We will design research instruments to answer the evaluation questions concerning the effectiveness and impact of the businesses. The instruments will be designed in consultation with the grantee, the IP and, where relevant, our local research partners and both instruments will be approved by the grantee before fieldwork. This ensures the instruments elicit information that is useful to the business as well as informing the evaluation and that the questions are culturally appropriate. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 28 Annex B: Totohealth SPRING funding SPRING support24 SPRING support has been focused on two main areas summarised below. A $80,000 grant supported the implementation of activities: • Development and integration of content for adolescent girls and voice content – With SPRING support Totohealth has developed teen content for SMS services. The newly developed messages have been integrated into the existing platform ready to be sent to the subscribed adolescent mothers. In order to reach adolescent mothers who are illiterate and those visually impaired, all messages have also been translated to Kiswahili and Somali. Totohealth then recorded the messages as audio splitting the audio messages based on when they need to be sent out. Totohealth now have a total of 382 audio messages in both Kiswahili and Somali. • Totobag content development and design – fuse project helped redesign the Totobox into a Totobag including the prototyping of various different bag designs and content. The newly designed bag is about to be piloted. • Business development and marketing for Totobags • General business development– Jonathan Ridley (Head, M-KOPA Labs) provided business advice through a mentor relationship. • Legal advice – Totohealth made a strategic expansion to Tanzania in September 2015. Through SPRING legal mentorship, they subscribed and entity in Tanzania in partnership with 2 local business partners in Tanzania. 24 Business Performance Evaluation – Totohealth Business Model Write Up CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 29 Annex C: Totohealth March 2017 revised approach and provisional timetable Totohealth have recently provided us with additional information that means we will need to revise our approach to the Impact Evaluation. This document presents the issue and our proposed solution. The Intervention Totohealth provides an SMS message service to young mothers and fathers in Kenya to monitor the health of their child from month five of pregnancy to age 5. Automated messages are sent up to twice weekly and include a series of yes/no ‘trigger’ questions based on child developmental data to determine whether there are any problems with the child’s development. Subscribers can follow up with Totohealth’s helpdesk to ask questions. With SPRING funding Totohealth proposed to focus on: • targeting adolescent mothers; and, • developing SMS content and helpdesk support appropriate to clients’ needs. Current Approach At the start of our baseline research, Totohealth communicated that the SPRING funded message service targeted at adolescent girls would be introduced in January 2017 and that over the remainder of 2016, through SPRING funding, it would undertake referral activity to grow adolescent girl subscribers and these later subscribers would receive SPRING funded messaging. Based upon this information we decided to measure of the additionality of the SPRING funded support by assessing the extent to which these SMSes improve the effectiveness of the service provided to adolescent mothers. In May 2016 we carried out baseline research among Pre-SPRING Totohealth subscribers who receive the standard Totohealth SMSes (our control group). We then carried out our research among adolescent girls receiving the SPRING funded SMSes in December / January 2016/17 (our intervention group). The research among both groups included quantitative telephone surveys with all subscribers and 12 follow-up semi-structured interviews. We then planned to interview these same girls again 12 months after their first interview to assess the effect of the different service: the standard SMSes (control group) Vs SPRING-funded SMSes (intervention) on a range of indicators. The Problem Totohealth recently informed us that there had been internal confusion regarding the introduction of the SPRING funded adolescent girl SMS message service and that the targeted SMS content was introduced in December 2015 and that subscribers 20 years and under have received this content since then. This means that all of our research participants (both intervention and control) have been receiving the same SPRING funded SMSes and we no longer have a counterfactual. Proposed Solution To overcome this problem we recommend the following steps: • To merge the intervention and control group samples into one intervention group sample and revise our baseline report accordingly (as we now know both groups receive the same service) • To carry out the endline research with our intervention group 12 months after the baseline research to measure change against the indicators and carry out the semi-structured interviews to understand in more detail how girls have interacted with the service and how and why change has happened. • At the endline stage to identify and interview a comparison group of adolescent mothers from similar backgrounds and locations who have not used Totohealth to form our counterfactual. We propose to use the following approaches: • Snowballing during endline data collection amongst Totohealth service users: in the May/June 2017 and December 2017/January 2018 endline surveys and KIIs we will ask respondents and key informants if they CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 30 know of, can obtain consent from, and provide details of other adolescent mothers who have not subscribed to Totohealth (either by choice or lack of awareness). • Working with Totohealth, the IP and our local research partner to identify suitable locations, for example hospitals and clinics where Totohealth subscribers registered for the services as well as other geographically relevant day clinics and hospitals, to conduct face to face surveys with adolescent mothers who have not subscribed to Totohealth. We will survey the comparison group to establish their health behaviour, sources of information and influence, confidence and empowerment and compare this to the Totohealth subscribers. By comparing the results between the two groups we will be able to assess the impact that using the adolescent-specific Totohealth has upon the girls compared with those who have never used the service25 . Provisional Timetable Table 1: Provisional Research Schedule - Totohealth All subscribers to May 2016 (Pre-Spring) All subscribers June 2016 – December 2016 (SPRING) All non-subscribers (Comparison) Baseline Survey (Telephone) May - June 2016 December 2016 – January 2017 Baseline semi-structured interviews May - June 2016 December 2016 – January 2017 M M Endline Survey (Telephone) May- June 2017 December 2017 – January 2018 May/June 2017 &Dec 2017/January 2018 Endline semi-structured interviews May- June 2017 December 2017 – January 2018 May/June 2017 &Dec 2017/January 2018 Limitations Without being able to interview a control group at baseline and endline we are unable to measure attribution observed change in the intervention group to Totohealth. However, through a combination quantitative and qualitative research with the intervention group (at our baseline and endline) and a control group (at our endline) we will be able to assess the contribution of Totohealth to observed changes and to explain how and why intended changes have, or have not, happened. Summary Given the unforeseen circumstances and after considering the strengths and limitations of the approach we would like to propose this solution to the SPRING donors for their consideration. 25 From a methodology perspective it would be better to interview the control group at baseline and endline to measure the difference in difference but for ethical reasons this will not be possible (as we would need tell the girls that the baseline interview was about Totohealth then require them to refrain from subscribing from a service that has the potential to improve the health and wellbeing of both the girls and their babies) CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 31 Annex D: Interim feedback to Totohealth TOTOHEALTH KEY TAKE OUTS MAY 2016 Survey outline • In May 2016, Coffey undertook a survey with all Totohealth subscribers who were all under 20 years of age (49 in total). Coffey went on to undertake a series of face to face, semi-structured interviews with 12 of the respondents, where we had the opportunity to gather more detailed information. • The following high level feedback includes feedback from both the telephone survey and face to face interview. Where we refer to Key Informants, we are referring to the 12 face to face interviews. Survey Sample Profile: • The average age of respondents was 19 years. 12% of respondents were still pregnant. • Most respondents were first time mothers and do not look after other children with only 14% who report having other children to look after. • Half (51%) were married and forty five percent were single. Eighty eight percent were Christian and six percent were Muslim. A third (29%) spoke Kikuyu as their first language, 16% Lou, 14% Lua, 10% Kamba and 10% Nandi. • Twenty two percent had completed upper secondary, 20% completed lower secondary and 24% had completed primary education. Two percent had no school and 8% had attended madrassa. • The mean PPI score across the 49 Phase 1 respondents was 61. According to the PPI score card, among respondents in Kenya with a PPI rating of 61 there is only a 1% chance that the person lives on less than $1.25 a day and a 29%chance that the person lives on less than $2.50 a day. This score will be monitored as Totohealth implements its SPRING programme. Key Findings: Awareness and subscribing • Hospitals and clinic visits presented the best opportunity to target young expecting/new mothers for Totohealth services, with 55% of respondents becoming aware of Totohealth services here. Friends (33%) were also an important source of information. • Through key informant interviews, respondents recounted a variety of ways they had become aware of Totohealth during their visit to the hospital: - Most frequently mentioned was personal contact with a Totohealth representatives: “I went to hospital, [a Totohealth representative] approached me, we talked and I gave them my [details]... I subscribed that day... Since then we have been talking.” (20 year old; baby 17 months; 14 months subscribed) - Word of mouth from health professionals at the hospital; and, - Posters and marketing material at the hospital. • Respondents were more likely to register through SMS (69%) than any other route. A fast and easy process facilitated registration. Insight: Girls are more likely to subscribe after they have given birth to their baby and not while pregnant Totohealth targets pregnant and new mothers, with SMS services available from mothers five months pregnant to children 5 years. However, of the girls we surveyed in Phase 1, 67% subscribed to Totohealth services after they had given birth to their baby and 20% of these subscribers had become aware of Totohealth while they were still pregnant. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 32 • When discussing how respondents had subscribed it was evident that speed and ease of registration was also an important feature in ensuring people could quickly be added to the service, as captured in the following recollection: “We were in a hurry because I was on the queue to get treatment so we didn’t talk much but they explained to me how they will be messaging me advices on bringing up my child.” (20 year old; baby 17 months; 14 months subscribed) • None of the 12 key informants we spoke to in person were aware of any other similar assistance available for expectant and new mothers. SMS recall • A high recall of Totohealth SMSes was evident, with 83% of respondents able to recall the theme of their most recent SMS. Sixty percent of respondents shared their SMSes with either family or friends. • Key informants spoke highly of the Totohealth SMSes they had received, describing them as easy to understand, timely and using clear and appropriate language. Some key informants who had their own phones and were able to save their messages and reported referring back to them. Other key informants spoke of sharing a phone and having to read messages which might then be deleted. Insight: SMSes are easily understood and relevant: Key informant interviews reflected that users perceived the SMSes as easy to understand, clear and relevant: “They send in the language that each person understands... the words used [and] messages are simple and easy to understand” 20 year old; baby 17 months; 14 months subscribed) “They capture the right language and they don’t complicate it... They are precise and they help you a lot”(20 year old stay at home mother, baby 18 months;16 months subscribed) • Key informants most strongly recalled messages on their baby’s growth and stages of development, though messages on breastfeeding, immunization, parenting and family planning were also recalled. When prompted (with a showcard listing the topic areas), 83% of respondents recalled receiving messages on nutrition & breastfeeding and immunization, 80% recalled messages on child development and 76% recalled messages on hygiene. Key informant interviews demonstrated a high level of recall of message content within each of these topic areas. • Respondents reported a high level of trust in Totohealth advice, accessing support alongside (and sometimes instead of) health care professionals and family. Helpdesk • During the telephone survey, 30% of Phase 1 respondents indicated they were aware of the helpdesk function and 57% of these had gone on to use the helpdesk. However, key informant interviews suggested that subscribers were not always aware they were using the helpdesk when they engaged with Totohealth services. • Users spoke highly of the Helpdesk as providing them with a source of reassurance regarding their baby’s development. The Helpdesk was perceived as less useful in emergency situations where an immediate response was required and it was suggested that this was an area where Totohealth could improve (or possibly better manage user expectations). Most helpdesk users were very satisfied with the service. Insight: Helpdesk use may be higher than the survey suggests 26 Key informant interviews indicated that helpdesk use might be higher than the survey reflected. Six of the twelve key informants had stated in telephone survey they had not used the helpdesk. However, during key informant interviews, three of the six non-users went on to speak of SMS’ing Totohealth about concerns regarding their baby, some of them doing so on a regular basis. 26Totohealth themselves estimate helpdesk use at about 30% of all subscribed service users. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 33 • In general it was evident that the helpdesk was best suited to non-emergency guidance, support and advice as service response is not immediate, as recounted by one informant: “If you text [Totohealth] now they respond after two days or day and maybe it is an emergency you want the answer immediately. So they should be a bit quick to respond to the messages (19 year old, still in school, living at home, baby 11 months, subscribed 10 months). • The girl that made this comment, went on to describe two instances when had consulted the helpdesk – one where she had consulted the helpdesk when her baby was constipated and she felt she had been given advice which worked and she was happy and another where her baby had fallen, it was an emergency and she was advised to go to the hospital, which she felt she was insufficient advice. • It was suggested that Totohealth should improve their communication of the services they could offer through the helpdesk and the response times they could commit to: I would say that they should…they should tell their members the specific days they should consult.... (19 year old, single baby 10 months, 6 months subscribed) Insight: Users need to understand that the Helpdesk is for non-emergency situations: Feedback on the Helpdesk suggests that user expectations need to be better managed so that they understand when and how it is able to assist. Paying for services • Eleven of the twelve participants in the key informant interviews indicated they would be happy to now pay 200Ksh per year for Totohealth services, feeling this was an acceptable and affordable fee: “Yes, I would because they have helped me a lot and the money is no issue; according to the information they offer 200 is a little compared to what I gain” (19 year old, baby 12 months, 10 months subscribed). • However, some admitted that they would not have subscribed to the service at the outset if they had to pay for it, suggesting that Totohealth should consider providing the service for free initially so that users understood what they were paying for before asking for a fee. • Only one of the twelve key informants we spoke to would not now pay 200Ksh per year for Totohealth services. This was due to financial constraint and the feeling that as the baby was older she no longer needed the services. Insight: Users are happy to pay for the SMS service once they have experienced it: Key informant interviews suggested that most subscribers would pay for the service now that they had experienced the benefit of it. However, payment from the outset might be harder to achieve. The Totobag During key informant interviews we explored respondent’s interest in buying a bag and what might be able to afford on a single purchase. Our findings augment Totohealth’s existing research conducted amongst 300 pregnant women at hospitals in Nairobi. All key informants were interested in the Totobag. The five items in the bag that participants viewed as most important were mosquito nets; thermometers; soap & antiseptic; baby grow; and blankets. When designing their own Totobags, participants most frequently included baby clothes; blankets; antiseptic; thermometer; mosquito nets; diapers; toys; soap; and petroleum jelly. Insight: Young mothers are interested in the Totobag but 2,500Ksh is beyond the reach of many Price was evidently higher than many would be able to spend on a single purchase, with most indicating a ceiling of 1000 – 1500 K. CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 34 Annex E: Demographic Profile: Telephone Survey Adolescent girls surveyed in May 2016 included all those that had subscribed to the Totohealth service up to May 2016. Those surveyed in December 2016 include all adolescent girls that subscribed between May – December 2016 through SPRING funded referral/recruitment activity. Through consolidating the two datasets and analysing the findings by the variables where significant differences were noted, we are able to observe where these variables have influenced the results. Below we detail key differences between the two sub-groups. Statistically significant differences between the sub￾groups (May 2016 v December 2016) are denoted with an asterix (*). Table 1: Age Profile Total (%) May 2016 (%) December 2016 (%) Base = all respondents 104 49 55 16 9% 4% 15%* 17 12% 4% 20%* 18 19% 14% 24% 19 33% 43%* 25% 20 26% 35%* 16% Table 2: Marital Status Total (%) May 2016 (%) December 2016 (%) Base = all respondents 104 49 55 Single 53% 45% 44% Married 44% 51% 55% Widowed - - - Divorced 3% 4% 2% Table 3: Household Numbers Total (%) May 2016 (%) December 2016 (%) Base = all respondents 104 49 55 CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 35 One or two 9% 8% 9% Three 38% 49%* 27% Four 9% 4% 13% Five or six 18% 18% 16% Seven or eight 13% 12% 15% Nine or more 14% 8% 20% Table 4: Mother Tongue Total (%) May 2016 (%) December 2016 (%) Base = all respondents 104 49 55 Luo 12% 16% 9% Luyhia 10% 14% 5% Kikuyu 21% 29% 15% Kalenjin 22% 2% 40%* Other 35% 39% 31% Table 5: Religion Total (%) May 2016 (%) December 2016 (%) Base = all respondents 104 49 55 Christian 90% 88% 93% Muslim 6% 6% 5% Other 4% 6% 2% Table 6: Education Total (%) May 2016 (%) December 2016 (%) Base = all respondents 104 49 55 No school 2% 2% 2% Some-primary 8% 4% 11% CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 36 Primary completed 28% 24% 31% Some years of Lower secondary 12% 4% 20%* Lower secondary completed 22% 20% 24% Total up to lower secondary complete 72% 54% 88%* Some years of upper secondary 7% 14%* - Upper secondary completed 13% 22%* 5% College/Technical training 4% 2% 5% Some years of University 3% 4% 2% Adult Education / Adult Literacy 1% 2% - Attended madrassa 8% 18% Table 7: Age of baby Total (%) Group 1 (%) Group 2 (%) Base (n= all respondents) 104 49 55 Still pregnant 19% 12% 22% 0-5 months 28% 10% 42%* 6-12 months 27% 33% 20% 13- 24 months 19% 35%* 4% Over 24 months 5% 6% 5% Table 8: Length of Totohealth registration Total (%) Group 1 (%) Group 2 (%) Base (n= all respondents) 104 49 55 1 -3 months 55% 26% 80%* 4 – 6 months 26% 32% 20% 7– 11 months 5% 12%* - 12 months + 14% 29%* - CASE STUDY: TOTOHEALTH SPRING IMPACT EVALUATION – MAY 2017 37 Annex F: Completed depth interviews Table1: Completed key informant in-depth interviews Key informant Age of key informant Baby’s age Length subscribing to Totohealth (months) PPI Score 1 20 13-24 months 12+ 67 2 20 13-24 months 2 69 3 19 6-12 months 8 75 4 18 13-24 months 12+ 72 5 19 6-12 months 3 40 6 19 6-12 months 6 47 7 18 6-12 months 2 77 8 19 6-12 months 6 76 9 20 13-24 months 12+ 61 10 20 13-24 months 12+ 78 11 16 13-24 months 8 70 12 20 6-12 months 8 64 13 17 6-12 months 4 60 14 18 6-12 months 3 54 15 17 6-12 months 3 26 16 19 0-3 months 3 65 17 17 0-3 months 5 47 18 17 0-3 months 1 17 19 16 0-3 months 5 60 20 19 0-3 months 5 29 21 18 6-12 months 1 44 22 18 0-3 months 5 55 23 20 0-3 months 1 49 24 18 still pregnant 2 60