i Performance Evaluation of USAID/Uganda AFFORD – Health Marketing Initiative Project Evaluation Improving the Lives of Ugandans By: Augustine Ankomah, Joanne Spicehandler, Festus Kibuuka, Steven Mobley Submitted to: Zdenek Suda, Program Officer, USAID/Uganda September, 2013 ii ACKNOWLEDGEMENT This work was supported by funding from the USAID Mission Uganda. The consultants appreciate the technical advice and support from the USAID Mission Uganda staff towards the implementation and preparation of this report. The input from the USAID Mission Uganda Leadership and the AFFORD Final Evaluation Task Managers Ms. Charmaine Matovu and Ms. Rhobbinah Ssempebwa is particularly appreciated. The evaluation team also appreciates the Management and Staff of the AFFORD Project and UHMG for their input and support during the field visits. The input from all respondents including the project beneficiaries and their families, staff of the USG funded partners, staff from local governments consulted, Ministry of Health, Uganda AIDS Commission is very much appreciated. iii USAID | AFFORD – Health Marketing Initiative The USAID | AFFORD PROJECT is funded by the U.S. Agency for International Development and began as a five-year (2005-2010) health marketing initiative led by John Hopkins University Center for Communication Programs (JHUCCP) in partnership with Futures Group International, the Malaria Consortium, Pulse Communication, ACLAIM Africa and Communication for Development Foundation of Uganda. In 2010, USAID /Uganda awarded a three year US$ 18.5 million follow-on Assistance agreement (AFFORD II) and an additional one year no cost extension (2011-2014), to Johns Hopkins University (JHU). This second phase has been led by Johns Hopkins University Center for Communication Programs (JHUCCP) in partnership with Uganda Health Marketing Group (UHMG) with a mission to integrate health communication and social marketing techniques to address a variety of health issues and behaviors in Uganda, and an overarching aim of strengthening the institution of UHMG from an organization primarily dependent upon USAID funding to a private sector institution that is self￾financing, self-governing and meeting its goal of contributing to improved health status of Ugandans through Social Marketing. Cover photo: Uganda. UHMG personnel displaying social marketing products at AFFORD I close-out presentation at Sheraton-Kampala, 2011. From left to right: Juliet Rumanyika, UHMG Brands Officer, Family Planning; Esther Atuuse, UHMG Brands Officer, HIV & Child Health; Barbara Akatukunda, UHMG Sales Commission Agent; Rosemary Anyanga, UHMG Sales Representative, Kampala & Central Region Photographer: Unknown This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Augustine Ankomah, Joanne Spicehandler, Festus Kibuuka, Steven Mobley iv The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government SEPTEMBER, 2013 USAID/Uganda AFFORD – Health Marketing Initiative Project Evaluation Improving the Lives of Ugandans Submitted to: Zdenek Suda, Program Officer, USAID/Uganda By: Augustine Ankomah Joanne Spicehandler Festus Kibuuka Steven Mobley v Table of Contents EXECUTIVE SUMMARY .................................................................................................................................. x INTRODUCTION/BACKGROUND and PURPOSE ............................................................................................. 1 METHODOLOGY ............................................................................................................................................ 1 FINDINGS ....................................................................................................................................................... 3 1. Did AFFORD increase accessibility and affordability of HIVIAIDS, reproductive health, child survival and malaria products and services for Ugandans? .............................................................................................. 3 Why are sales poor? .................................................................................................................................. 5 What is the difference in affordability before AFFORD and now? ........................................................... 8 2. How relevant and effective have AFFORD’s social marketing approaches been in increasing awareness and uptake of health services? ..................................................................................................................... 9 What interventions are most successful? ................................................................................................. 9 Which ones were less successful? ..........................................................................................................10 Were AFFORD’s social marketing approaches more successful in achieving this result in some situations or locations or among some population groups than others? ..............................................12 3. What specifically is AFFORD’S contribution to the delivery of health and HIV/AIDS services in Uganda? ....................................................................................................................................................................14 How does AFFORD rank among condom and contraceptive distribution/sales agencies in Uganda? ..14 Can the “Good Life Clinic” approach be classified as true Social Marketing? ........................................16 What are the missed opportunities/untapped potentials, if any? .........................................................17 4.To what extent did AFFORD strengthen UHMG systems and capacity for technical, financial, institutional, and market sustainability? ....................................................................................................17 Technical Capacity: ................................................................................................................................. 18 Management of Subgrantees and Community Level Work with MARPS: ..............................................19 Institutional Capacity ..................................................................................................................................19 Sustainability: ..........................................................................................................................................19 Finance and Administration: ...................................................................................................................20 Governance .............................................................................................................................................20 HR and Procurement:..............................................................................................................................23 Who owns UHMG? ................................................................................................................................. 23 Stages of Organization Development .....................................................................................................24 CONCLUSIONS .............................................................................................................................................25 Accessibility and Affordability:................................................................................................................25 vi AFFORD’s Relevant and Effective Social Marketing Campaigns: ............................................................25 AFFORD’s contribution to delivery of HIV services:................................................................................26 How AFFORD has strengthened UHMG’s capacity .................................................................................26 RECOMMENDATIONS .................................................................................................................................28 Accessibility and Affordability:................................................................................................................28 AFFORD’s Relevant and Effective Social Marketing Campaigns: ............................................................28 AFFORDS contribution to delivery of HIV services: ................................................................................28 UHMG’s Capacity Strengthening by AFFORD..........................................................................................28 LESSONS LEARNED ......................................................................................................................................29 Annex 1 Statement of Work....................................................................................................................... A Annex 2 Resources.................................................................................................................................... G Annex 3 Persons Met ................................................................................................................................... I Annex 4 Interview guides........................................................................................................................... K Annex 5 Methodology ..............................................................................................................................UU Annex 6 Challenges faced by some GLCs ..................................................................................................XX Annex 7 Management of Subgrantees ...................................................................................................... ZZ Annex 8 Staffing Growth and Composition ...........................................................................................AAA Annex 9 Social Marketing Operation (SMO) – an Overview of considerations for startup ................... BBB vii List of Tables Table 1 AFFORD product sales (2009-2013) Table 2 HIV prevention and support services Table 3 Antenatal care services Table 4 Retail audit average prices Table 5 HIV prevention activities by region and HIV prevalence Table 6 Outlets stocking contraceptives by Region, TFR, CPR and Unmet Need Table 7 Stages of Organizational Development Table 8 UHMG Stages of Development List of Figures Figure 1 Overall presence of UHMG products Figure 2 Number tested and received results Figure 3 Number tested and received results as couples Figure 4 Health seeking behavior by exposure to Power of Day One Campaign Figure 5 Competitors market Share (condoms) Figure 6 Market share (female contraceptives) Figure 7 Number of condoms distributed in Uganda (2007/08-2010/11) viii List of Abbreviations and Acronyms AIDS Acquired Immune Deficiency Syndrome AFFORD African Foundation for Development BCC Behavior Change Communication BOD Board of Directors CARISMA Caribbean Social Marketing Program CPR Contraceptive Prevalence Rate CSM Condom l Social Marketing CSO Civil Society Organization CSW Commercial Sex Worker DFID Department for International Development PMTCT Prevention of Mother to Child Transmission FAITH Fighting Aids in the Home (Project) FF Fisher Folks FGD Focus Group Discussions FSW Female Sex Worker GOU Government of Uganda GLC Good Life Clinic HCT HIV/AIDS Counseling and Testing HCU Humanitarian Care Uganda IDI In-depth Interviews IEC Information Education and Communication IP Implementing Partner JHU Johns Hopkins University KPIs Key Performance Indicators KRAs Key Result Areas LQAS Lot Quality Assurance Sampling MNCH Maternal and Newly Born Child Health MARP Most-at-risk-population MD Medical Doctor M&E Monitoring and Evaluation MOH Ministry of Health MOST Management and Organizational Sustainability tool MSU Marie Stopes Uganda NAYODE National Youth for Development NGO Non-Governmental organization NICRA Negotiated Indirect Cost Agreement OCAT Organizational Capacity Assessment Tool PE Peer Educator PACE Program for Accessible Health, Communication and Education PHA People infected with HIV/AIDS PEPFAR President's Emergency Plan for AIDS Relief PF Product Facility PMP Performance Monitoring Plan PPP Purchasing Power Parity PMTCT Prevention of Mother to Child Transmission QI Quality Improvement RWI Research World International SMO Social Marketing Organization ix SOW Statement of Work SSECODA Ssese Community Development Association TFR Total Fertility Rate UDHS Uganda Demographic Health Survey UHMG Uganda Health Marketing Group UKAID United Kingdom Aid Agency UNFPA United Nations Fund for Population Activities USAID United States Agency for International Development WHO World Health Organization x EXECUTIVE SUMMARY USAID/Uganda, as a key player within the USG’s response to Africa’s HIV/AIDS challenges, and with considerable PEPFAR support funding, has supported Uganda’s social marketing efforts since 1994. In August/September 2013, the USAID | Uganda health team contracted with four consultants to carry out a performance evaluation of the AFFORD project to determine the extent to which UHMG’s capacity, as Uganda’s premiere, indigenous organization promoting social marketing, has been built and how it has played a key role in improving the uptake of healthy behaviors and increasing demand for delivery of health and HIV/AIDS services in Uganda. The evaluation team was composed of members with skills and background in social marketing, design, implementation and evaluation of health programs, business approach to service delivery, organizational development, and organizational capacity assessment. The team reviewed literature, project documents, data sets and reports; traveled to ten districts in the East, West, North and Central Regions and conducted 62 personal interviews with Good Life Clinics (GLC), NGO subgrantees, UHMG Board members, AFFORD staff, UHMG Founder members and key staff of the organization. The team also had interviews with government officials from the Ministry of Health, donors, implementing partners and other stakeholders. The evaluation team also conducted 10 focus group discussions with peer educators, health educators and beneficiaries- Fisher folks, truckers, female sex workers, youth and married couples. The evaluation, including report writing, lasted six weeks. The evaluation team addressed four areas: Accessibility and Affordability of products and services; Relevance and Effectiveness of AFFORD’s Social Marketing Campaigns; AFFORD’s Contribution to Health and HIV/AIDS Services; and Capacity Strengthening of UHMG. Key Findings Accessibility and Affordability: To measure accessibility of products, AFFORD undertakes periodic retail audits to measure the proportion of outlets (e.g. clinics, pharmacies, drug shops) stocking their products. From the data, it is evident that a high proportion of outlets in Uganda stock the three USAID donated products. The latest June 2013 retail audit report shows that 85 percent of outlets stock Pilplanplus, while eight out of ten stock Protector Condoms (79%) and Injectaplan (78%). However, product placement levels are much lower for other social marketing products. Only half of the outlets have Zinkid, and 44 percent have Restors. Evaluating product accessibility through placement levels in retail outlets is not enough. Indicators that measure the ease of clients’ and potential clients’ actual access to products should be considered. Although products are available at retail outlets, when accessibility is measured at the GLC clinic level, it was observed that the number who access services at clinic levels have decreased since 2009. For example, the number of women who received ANC services for the first time decreased by 47% between 2009 and 2012. Our discussions and interviews showed that accessibility and availability of condoms in and around hotspots, such as trucker spots, was patchy and depended upon the zeal and commitment of the community volunteer. On the whole, pricing, or affordability, does not seem to be an issue, especially for Protector condoms which are generally one half to one third the price of competitive brands. Accessibility and affordability is basically unchanged during the last four years. Relevance and Effectiveness of AFFORD’s Social Marketing Campaigns: AFFORD, through UHMG, is currently the leader in mass media health communication in Uganda. UHMG as a sub xi grantee to AFFORD, and with technical support from JHU has been spearheading the health communication efforts of AFFORD. Donors, partners and the data confirm UHMG’s leadership position. There is a useful blend of TV, billboards, radio spots, talk shows, and social media aimed at increasing demand. All partners, including the Ministry of Health, are pleased with their performance. Data show that about 78 percent of adult Ugandans are reached by at least one AFFORD message in a quarter. AFFORD’s analysis shows there were statistically significant differences between those who were exposed to their communication messages and those who were not. Even after taking account of differences of socio-demographic characteristics, women exposed to communication messages were 71 percent more likely to report current use of a modern method and twice as likely to have discussed family planning with their partner. At the community level, among MARPs and other groups however, there was no hard evidence, as AFFORD was unable to collect data on behavior change among high-risk populations and other groups as a result of the protracted delay in the ethical review process. However, one gets the impression from talking to sex workers that they mostly maintain consistent and correct condom use, although in our interviews, one hears of money having greater influence over behavior change than BCC, meaning the more they are paid the less they adhere to their commitment to use condoms. There is evidence that AFFORD’s HIV prevention efforts (measured by stocking of outlets with condoms) were highest in the regions with highest HIV prevalence. The highest proportion of outlets with Protector Condoms (88%) was recorded in Central 1, the region with the highest HIV prevalence (10.6%). With respect to meeting family planning demands, however, AFFORD through UHMG has not targeted regions of highest need for family planning. On the whole, the regions with the highest unmet need - (e.g. Mid north, West Nile, Mid-Eastern/Eastern) have the fewest percent of outlets stocking PilPlanplus, Injectaplan and the other contraceptives. AFFORD’s Contribution to Health and HIV/AIDS Services: AFFORD’s success through UHMG in mass media health communication is very impressive and underscores the organization’s strength using mass media. AFFORD’s data show that the messages helped to increase knowledge and enhance spousal communication about family planning and contraceptive use. Notwithstanding this achievement, the overall sales figure for AFFORD social marketing products can be rated as poor. At a highly subsidized price, the Protector condom is the cheapest condom brand in Uganda. The number sold could be a crude indication of HIV prevention efforts. On the whole, there has been no increase in the sales volume of Protector condoms since 2009. Although UHMG’s commercial brand Condom ‘O’ shows an appreciable increase in the early years, it has also since plateaued, even if targets are met in 2013. AFFORD relies on a few regional distributors and institutional buyers. This alone is ineffective in predominately rural Africa. It is important to note however that AFFORD, through UHMG, sells more condoms than any social marketing organization in Uganda. In 2012 UHMG sold half of all social marketing condoms in Uganda. Capacity Strengthening: AFFORD has strengthened UHMG’s internal financial systems and controls, monitoring of subgrantee financial compliance, and HR systems and performance management. UHMG has also succeeded in establishing a physical infrastructure for the office and warehouse, though recent growth requires additional space. Policies, procedures and systems for HR, Finance, Procurement, Subgrants Management, the Board of Directors and other areas of the organization are in place and functioning. UHMG has passed its A133 audit of USAID compliance, and has done reasonably well on its most recent external audit. Marketing staff are experienced at media purchasing, and their campaigns reach a substantial number of people in Uganda. UHMG staff turnover has been reduced, but is still contract-dependent. The Product Facility (PF), the commercial arm of UHMG, has covered its operating costs including salaries. xii However, while structures and systems have been established, at the 8-year mark, UHMG is still struggling to achieve sustainability. USAID experience in social marketing shows that it takes up to ten years to establish a truly sustainable local entity like a social marketing organization or indigenous organization (O’Sullivan et al, 2007). Internally, the organization does not appear to have a clear strategy for how each division works together to support one Mission and Vision. Based on the evaluation, the organization has received uneven strengthening of capacity. PF for example, has a growing commercial warehousing and haulage business, but has lost sight of product sales. Similarly, while the Management and Strategic Information department has forged ahead in external recognition for its success in media and communications, it, too has not focused on appropriate strategies to address key social marketing challenges. The Programs and Services division has succeeded in establishing a nationwide franchise of 200 Good Life Clinics which has the potential of providing needed services particularly to women. However, AFFORD is still grappling with how to improve and measure quality in the clinics due to inadequate expertise in clinic quality management of the GLC clinics. The UHMG research team, excluding AFFORD, still lack expertise in critical areas, particularly data analysis. Over the past three years, AFFORD has succeeded in strengthening UHMG’s relative financial capacity but at a price of neglecting technical capacity in other key areas of social marketing. Currently, there is an institutional tension between UHMG’s commercial and social marketing arms as to which way to go. UHMG and their Product Facility (PF) division operate as if they are independent organizations. The strength of UHMG lies in a strong, synergistic relationship between the two in a manner that makes the organization a strong competitor in the social marketing arena in Africa. The Board and Management are also in conflict about their respective roles and responsibilities to each other and the organization. The addition of a new layer of decision-making via the Founders’ Forum will only complicate and further delay the already complex decision-making process, and further erode the authority of both the Board and Senior Management. Recommendations: Accessibility, Affordability and Sales  Identify and grow the number of distributors and non-traditional outlets (NTOs) and ensure that they are easy to reach within 5-10 minutes (to the last mile)  Conduct more comprehensive willingness-to-pay studies and consumer insight analysis Health Service Delivery  Suspend further GLC accreditations until quality is standardized and monitored in GLCs  Ensure extensive clinical experience in Family Planning, HIV and Maternal Health at UHMG  Increase demand creation activities around GLCs to get more clients to use services Institutional Capacity  Embark upon a systematic capacity building exercise for research and BCC staff  Revisit the Board’s composition to improve the synergy among UHMG departments and clarify the roles and responsibilities of the Board and Management  Continue training supervisors in performance management and provide training in feedback  Continue allocating staff to multiple project funding sources, wherever possible, to ensure their continuity in the organization  Continue diversifying funding sources, particularly soliciting more potential private sector support, to reduce dependency on USAID. 1 INTRODUCTION/BACKGROUND and PURPOSE The USAID AFFORD Project is funded by the U.S. Agency for International Development and began as a five-year (2005-2010) health marketing initiative led by John Hopkins University Center for Communications in partnership with Futures Group International, the Malaria Consortium, Pulse Communication, ACLA Africa and Communication for Development Foundation of Uganda. In 2010, USAID/Uganda awarded a three year US$ 18.5 million follow-on Assistance agreement (AFFORD) and an additional one year extension (2011-2014), to Johns Hopkins University (JHU). This second phase has been led by Johns Hopkins University Center for Communications Programs (JHUCCP) in partnership with Uganda Health Marketing Group (UHMG) with a mission to integrate health communication and social marketing techniques to address a variety of health issues and behaviors in Uganda, and an overarching aim of strengthening the institution of UHMG from an organization primarily dependent upon USAID funding to a private sector institution that is self-financing, self￾governing and meeting its goal of contributing to the improved health status of Ugandans through Social Marketing. AFFORD objectives have been to: 1) Further develop UHMG systems and capacity to support technical, financial, institutional and market sustainability; 2) Increase accessibility and affordability of HIV/AIDS, Reproductive Health, Child Survival and malaria products and services for Ugandans using innovative private sector approaches; 3) Increase knowledge and promote healthy behaviors and lifestyles among the general population and among targeted hard-to-reach and high-risk population segments. In August, 2013 USAID/Uganda contracted with four consultants to evaluate AFFORD. From the SOW (Annex 1), the primary purpose of the evaluation is to determine the extent to which UHMG’s capacity as an indigenous organization promoting social marketing has been built and how it has played a key role in improving the uptake of healthy behaviors and increasing demand for delivery of health and HIV/AIDS services in Uganda. The evaluation was designed to inform ongoing and future social marketing activities by UHMG and other Social Marketing organizations in Uganda The evaluation report is intended to provide “lessons learned” on strengthening local capacity as presented under the 2010 USAID Forward reforms. Questions the evaluation addresses are: 1. Did AFFORD increase accessibility and affordability of HIV/AIDS, Reproductive Health, Child Survival and malaria products and services for Ugandans? 2. How relevant and effective have AFFORD’s social marketing campaigns been in increasing awareness and uptake of health services? 3. What specifically is AFFORD’s contribution to the delivery of health and HIV/AIDS services in Uganda? 4. To what extent did AFFORD strengthen UHMG systems and capacity for technical, financial, institutional and market sustainability? METHODOLOGY Data collection was primarily qualitative in nature although quantitative secondary project data were also examined closely to evaluate performance. The evaluation team employed three main data collection methods: (1) Document Review of Project Monitoring Data and existing literature (List of documents reviewed is included in Annex 2). (2) In-Depth Interviews; (3) Focus Group Discussions. 2 The time devoted to the evaluation, 42 days, was not enough to embark on quantitative survey, as was done in the mid-tem evaluation of AFFORD I. Other reports, financial documentation, monitoring reports and accompanying databases, research reports and databases, training reports, curricula, protocols, as well as BCC materials were also examined. To evaluate the status of UHMG’s organizational capacity development, the team used the Pact Organizational Capacity Assessment Tool developed in South Africa. The team also examined UHMG’s organizational systems and processes, through interviews with relevant staff. Key informant interviews were held early in the evaluation process with the technical and management staff of AFFORD and UHMG. The evaluation team held interviews with a total of 62 persons. Those interviewed included Founding Members, the Board of Directors, senior management and staff of AFFORD and UHMG. Stakeholders who were interviewed included USAID staff, the Ministry of Health, Implementing Partners, Donors, and the Uganda AIDS Commission. (A list of all people met and interviewed is attached to this report as Annex 3). Seven Good Life Clinics (GLCs) out of 200 were included in the site visits from the North, Central, Eastern and Western parts of the country. In addition the evaluation team reviewed GLC service statistics for the period 2011-2013, and the GLC QI Summary Report from March 2013, to support our findings from the site visits. The evaluation team also visited the intervention sites of all four UHMG subgrantees. The team developed interview guides to assess capacity and quality of service of the four subgrantee organizations implementing interventions at the community level as well as the GLCs (Annex 4). The team also conducted 10 focus group discussions with peer educators, health educators and beneficiaries - fisher folks, truckers, female sex workers, youth and married couples. Each group was composed of 8 to 12 participants and separated into the population group. The team used a discussion guide prepared specifically for that group (see Annex 4). Since no surveys were included, the evaluation team did not conduct interviews with the general population, although it held in-depth interviews with community leaders, government officials, and implementing partners. The evaluation team used AFFORD-generated quantitative data to evaluate AFFORD’s interventions among the general population. Detailed Evaluation methodology in Annex 5). Limitations Given the diversity and varied nature of UHMG’s interventions, the evaluation’s sampling was not representative as team members could only visit a few intervention sites. For logistic and time-related reasons our choice of sites and the number of facilities visited may result in selection bias which could affect the extent to which some findings could be generalized. Furthermore, no quantitative primary data were collected, again because of time limitations. The evaluation team relied on AFFORD￾generated quantitative data. Furthermore, our team was not fully constituted as per the SOW at the start of the assignment, as one member joined two weeks after inception. This meant the team could not split into two sub-teams, which would have allowed the team to visit more field sites. The team, however, took advantage of the varied expertise and vast experience that already existed among the three team members on-site. Ethical considerations The evaluation team made every effort to address ethical considerations during the planning and implementation of the evaluation. Verbal consent was obtained from each respondent before interview. No financial or other incentives were provided to participants. Only evaluation team members had access to the raw data. 3 FINDINGS 1. Did AFFORD increase accessibility and affordability of HIVIAIDS, reproductive health, child survival and malaria products and services for Ugandans? AFFORD’s Performance Monitoring Plan (PMP) uses several indicators to measure accessibility. The overall indicator used is the percentage of sampled retail outlets carrying at least one AFFORD product. Based on this measurement, accessibility has increased substantially from 50 percent in 2010 to 85 percent in 2013 (AFFORD II PMP 2011-2013) Figure 1 To measure accessibility of products, AFFORD subcontracts periodic retail audits to measure the proportion of outlets nationwide (e.g. clinics, pharmacies, drug shops) stocking their products. Their data shows that a high proportion of outlets in Uganda stock the three USAID donated products. The latest retail audit report for Q2, reported in June 2013, (Figure 1) shows that 85 percent of outlets stock Pilplanplus while eight out of ten stock Protector condoms (79%) and Injectaplan (78%). However, product placement levels are low for other social marketing products. Half of the outlets have Zinkid, and 44 percent have Restors. For four USAID supported products, product placement levels are below 10 percent: Softsure, nine percent, Contramox, six percent, Newfem, four percent and Moonbeads two percent. While products are available at over 90 percent of pharmacies and around 85 percent of drug shops, less than one-half of bars (47%) stock condoms. This seems to suggest that for some users, or potential users, condoms are not easily available at high-risk spots such as bars where they are needed most. Evaluating product accessibility through product placement levels in retail outlets is not enough. In a country where about 80 percent of the population resides in rural areas, Tier 1 outlets represent little more than authentication for the product while doing little to contribute to accessibility. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Apr-13 May-13 Jun-13 Source: Retail Audit Report (Q2,2013) 4 High presence in outlets does not necessarily mean that products are accessible as confirmed by AFFORD’s own baseline data in 2009 which showed that only 35 percent of the target population found at least three AFFORD promoted products easy to obtain. Unfortunately there was no end-line survey in 2013 to measure current status of availability. One of the most important indicators of accessibility of products and services, however, but not used by AFFORD/UHMG, is the distance or travel time needed to reach the nearest service facility using the most popular means of transport in the community. Accessibility of products measured by product sales Another measure of accessibility is the number of products sold or distributed at the national level. The team therefore evaluated how well AFFORD has marketed USAID supported brands in addition to products promoted by USAID like Protector, Pilplan and Injectaplan. AFFORD has three categories of health products. Two are social marketing products: (1) USAID donated products (Protector, Pilplanplus, and Injectaplan); (2) USAID supported brands of products (Condom ‘O’, SoftSure, NewFem, MoonBeads, Acquasafe, Restors, Zinkid, Cotramox); (3) for-profit products which AFFORD procures and sells commercially. Table 1 shows the growth of sales of the social marketing products since 2009. The overall picture of sales is ‘flat to declining’. Only three products increased in sales between 2009 and 2012. The overall finding on sales of products for nearly all products, is that AFFORD sales have not increased since 2009. Only one of the three USAID donated products (Pilplanplus), saw an increase in sales between 2009 and 2012. For Protector, the cheapest and most heavily subsidized condom in Uganda, the number of pieces sold in 2012 was lower than 2009 (a decline of five percent). Only two out of eight of AFFORD’s own products (Condom ‘O’ and Zinkid) increased in sales between 2009 and 2012. For 10 out of 11 products, AFFORD may be unable to achieve their 2013 targets, unless there are sales to institutional buyers for products such as Protector and Pilplanplus. Table 1. AFFORD Product Sales (2009-2013) Actual 2009 Actual 2012 Growth 2009- 2012 Target 2013 Actual 2013 (9 months) Likely to achieve target? Protector pieces 16,050,674 15,237,492 -5.15 22,249,960 12, 522,780 No Pilplanplus cycles 1,412,650 2,123,190 50.1% 2,549,995 1,375,290 No Injectaplan vials 1,678,457 1,640,980 -2.2% 4,500,000 1,770,070 No Condom ‘O’ 320,241 1,283,703 300.9% 1,500,000 1,157,904 Yes? SoftSure cycles 55,090 33,980 -38.3% 72,000 29,323 No NewFem 55,573 15,880 -71.4% 40,000 21,460 No MoonBeads pieces 3,502 3,435 -1.9% 5,950 1,699 No Acquasafe tablets 3,971,746 3,270,720 -17.7% 5,400,000 2,880,400 No Restors sachets 1,672,973 770,945 -53.9% 2,000,004 1,109,050 No Zinkid tablets 1,358,102 7,990,300 488.3% 10,000,000 3,914,450 No Cotramox tablets 11,552,694 8,775,360 -24.0% 7,900,000 3,798,000 No 5 Protector Condom sales figures rising and falling imply that the distribution pipeline is being filled and usage lags behind. The overall trend is flat. Pilplanplus has had a modest increase since 2007 and AFFORD’s condom brand “O” has almost tripled since its launch. There is no evidence that the condom market is shrinking. There appears to be concentration of sales in urban areas in a country where over 80 percent live in rural areas. If there is a condom glut, then it is an urban phenomenon. There are no studies on the extent of market and availability of condoms in rural areas apart from retail audits. Regarding family planning products, there has been considerable decline in the sale of AFFORD’s own brand - NewFem. In 2009, the sales figure was 55,573; in 2012, it was 15,880 - a decline of 71%. MNCH products had mixed results. Zinkid’s sales increased nearly five times between 2009 and 2012, although it is unlikely that 2013 sales will match that of 2012. The sales for Restor and Acquasafe both fell between 2009 and 2012 by 53.9 percent and 17.7 percent respectively. Why are sales poor? Product sales are not increasing in the marketplace and staff is reticent to acknowledge the data. Almost all believe that the percentage of outlets stocking their products and AFFORD’s market share, indicate they are doing well. Stock at the distributorship is an inefficient proxy indicator for sales. Marketing and Strategic Information (MSI) and the Product Facility (PF) are working off of different figures. Whereas sales and distribution are handled by the Product Facility, social marketing and demand creation is done by MSI. This, in itself, is not necessarily a problem. The challenge however is lack of coordination and communication between PF and MSI. Although individual departments may be addressing poor sales, there was no documented organization-wide strategy to address flat and declining sales. Rather the evaluation team observed that each department has its own ad hoc tactics to increase sales. For example, one staff member mentioned addressing the dropping sales by taking out a van and conveying more product to the different locations. Another said they are in the process of forming a relationship with a community organization to distribute condoms. The evaluation team did not come across any systematic documented analysis of what is wrong with sales or any consumer insight research. According to AFFORD, during the last two years, they have shifted decisively toward reliance on a small number of carefully selected distributors, known as “Trade 1” distributors, located in urban areas who sell roughly 85-90% of all AFFORD products through their market-based, commercial channels. There is a need to widen significantly this approach by penetrating Uganda’s still neglected rural areas. The Northern region needs greater penetration and competition is weak there. Irrespective of results from retail audits, the AFFORD brands are primarily urban brands (Retail audit report, June 2012). AFFORD appears to only drop products at its established outlets, and not make attempts to establish new outlets, depicted by near-same product placement levels in all the monthly retail audits. The wholesale price of AFFORD socially marketed products is low with high mark-ups for AFFORD wholesalers and distributors. There should be aggressive efforts to recruit a lot more distributors. At the moment, AFFORD has its own wholesale pharmacy shops in two towns. They will need further support to improve sales in their regions. While it is true that sales of USAID supported condoms are flat, it is important to put this in perspective. Overall condom use in Uganda has declined over the past decade - from 16.1% to 13.3% between 2004/05 and 2011. More specifically, there has been a substantial decline in the proportion of people using condoms during the last sexual intercourse between 2004/05 and 2011, according to the AIDS Indicator Survey. This decline was more pronounced in the age group 25-49 (from 47.5% to 9.7%) 6 compared to those in the age group 20-24 (56% to 25.2%). At the national level the number of condoms distributed has dropped substantially from 119 million in 2007/08 to 69 million in 2010/11 (a decline of about 42%). Availability of HCT services The number of persons who have tested for HIV and have received their results (another AFFORD indicator for availability) has not increased substantially from the 2011 figure of 123,245. The target for 2013 is 338,145, but less than half (144,346) has been achieved in nine months in 2013 (Figure 2) . Similar low figures were observed for couple testing (Figure 3). This may be due to issues with stock-out of HCT test kits. Figure 2. Number Tested and Received Results Figure 3. Number Tested and Received Results as Couples 120000 132000 338145 123245 117304 144348 0 50000 100000 150000 200000 250000 300000 350000 400000 2010/2011 2011/2012 2012/2013(9months) Target Actual 11300 12400 45000 12306 5729 14341 0 5000 10000 15000 20000 25000 30000 35000 40000 45000 50000 2010/2011 2011/2012 2012/2013(9months) Target Actual 7 Malaria Malaria control through AFFORD II supported by the US President’s Malaria Initiative (PMI) focuses on behavior change and capacity building for malaria case management in the private health sector. The BCC activities aim at increasing timely malaria diagnosis and treatment within the community. The Power of Day One campaign is a mass-media campaign launched in June 2011 : to create awareness among caretakers of children and pregnant women on the importance of testing and treating malaria within 24 hours of fever onset; and to enhance the capacity of the private health sector in malaria prevention and case management. The Power of Day One campaign targeted pregnant women and caretakers of children in the campaign districts via radio, community meetings, billboards, posters, and other point of service materials. To support the campaign, a hotline and SMS platform provided callers with resources on malaria prevention and information about where to get testing and treatment for malaria. Figure 4. Health seeking behaviour by exposure to Power of Day One Campaign The evaluation team observed that the campaign has been successful and has now been scaled up nationally with the Stop Malaria Project (SMP). In the three (out of six) districts where the campaign took place, 64% of the respondents reported having seen or heard campaign messages during the 12 months preceding the survey. Exposure levels were about similar for both males (64%) and females (63%). There is evidence, as shown in Figure 4, that persons exposed to the Power of Day One were more likely to have a household member treated for fever within 24 hours (78%) than those not exposed (69%). Similarly, 74 percent of those exposed (compared with 46% of those not exposed) had a household member take a blood test when they reported having fever. Antenatal Care Table 2 on antenatal care - provides an example of some of the conflicting data the evaluation team has received from AFFORD. The figures from AFFORD’s PMP for 2011-2013, in bold, are contrasted with figures from AFFORDs service statistics data for the same time period. The actual numbers are 78 74 69 46 0 10 20 30 40 50 60 70 80 90 HH member treated within 24 hours for fever in past two weeks** HH member received blood test for fever in past two weeks** Exposed Unexposed Source: USAID Joint BCC Survey, 2012 8 completely different, suggesting that one is more updated than the other, even though it is long past 2012 and therefore, figures should match by now for that year. In addition, the evaluation team observed that targets are not being met. The PMP data for number of clients who actually attended their first antenatal visit indicates that 24,090 attended in 2012, in comparison with 45,547 in 2011, a drop of about 46%. In 2013, as shown in Table 2, at 9 months, the actual number is only 50% of the targeted amount, and is therefore likely to only match the 2012 figure rather than show an increase. While the actuals came close to meeting targets for numbers receiving IPT1 in 2012, they are lower than 50%, at 9 months in 2013. The number receiving IPT2 has only reached 50% of the target in 9 months. AFFORD explains that the MCH and FP services are relatively new and expects that the overall trend in utilization will be upwards in the future as demand creation activities are increased in GLC communities. Table 2 Antenatal care services received from GLCs (2011-2013) 2011 (Actual) 2012 (actual) Target 2013 Actual 2013 No. ANC clients first Visits 45,547 24,090 40,000 20,840 28,588 40,000 24,489 No. of clients for IPT 1 19,465 17,725 35,000 16,499 21,235 35,000 19,555 No. of clients for IPT 2 12,677 12,626 30,000 12,353 14,690 30,000 14,255 What is the difference in affordability before AFFORD and now? Condoms: Pricing, or affordability, does not seem to be an issue, especially for condoms, which are generally one half to one third the price of competitive brands. Indeed, affordability and pricing is basically unchanged in the last four years. A Retail Audit was undertaken in 2012, and pricing information for all brands observed in outlets was documented. The only popular brand that was priced within the parameters of Social Marketing pricing strategy was Protector. An often used formula for Condom Social Marketing is .01 x per capita GNP ÷ 100 = per unit retail price. The per capita GNP is approximately US$215. However the Purchasing Power Parity (PPP) is perhaps a more reliable indicator of cost of living and relative income level. The PPP for Uganda is US$1,817. Therefore, an appropriate price for a single condom at the retail level is between US$ 0.022 up to US$ 0.182. At 2500 UGX to the Dollar, it equates to 55 UGX to 455 UGX, or for a package of 3, 165 UGX to 1365 UGX. The 2013 Retail Audit of condom prices has Protector’s average cost in the marketplace to be 700UGX (see Table 3), more than double AFFORD’s suggested price but still about what the market should be able to support. Nonetheless, Protector is a competitive product and affordable to the lower income segment of society. However, there is no demographic evidence on users. 9 Table 3. Retail Audit Average Prices 2. How relevant and effective have AFFORD’s social marketing approaches been in increasing awareness and uptake of health services? What interventions are most successful? One of the key components of social marketing is promotion. This consists of the integrated use of communication strategies including mass communication, advertising and media advocacy. Consequently, Mass media is frequently used as one of the major channels of communication in social marketing. Mass media can enhance social marketing activities through reinforcing the key messages regarding desired behavior. AFFORD through UHMG is currently successful as the indigenous leader in mass media health communication in Uganda. Donors, implementation partners and the data available confirm UHMG’s leadership position. They employ a useful blend of TV, billboards, radio spots, talk shows, and social media aimed at increasing demand. All partners, including the Ministry of Health, are pleased with their performance. Data show that about 78 percent of adult Ugandans are reached by at least one AFFORD message in a quarter. AFFORD has been successful in building the capacity of UHMG staff in mass media communication. On family planning, for example, 70 percent of respondents surveyed in 2013 (72% males, and 69% females) had been exposed to an AFFORD family planning message during the past year. Sixty-eight percent were exposed to a message on eliminating mother to child transmission of HIV. AFFORD’s analysis shows there were statistically significant differences between those who were exposed to their communication messages and those who were not. For example, 38.6 percent of females exposed to Product Price at Retail (UGX) Protector 700 Life Guard 1,150 Condom O 1,800 Rough Rider 4,600 PilPlanPlus 1,100 Injectplan 1,800 Depo Provera 1,500 New Fem 2,400 Softsure 2,000 Microgynon 3,400 MoonBeads 7,800 AquaSafe 120 Water Guard 150 Water Guard Liquid 1500 Restors 500 Zinkid 300 Note: AFFORD products shaded 10 family planning messages were more likely to have used modern contraceptives compared with 27.0 percent of those who were not exposed. The corresponding figures for males were 30.5 percent for those exposed and 21.7 percent of those not exposed. Even after taking account of differences of socio￾demographic characteristics, AFFORD’s analysis concluded that, for females, those exposed were 71 percent more likely to report current use of a modern method and twice as likely to have discussed family planning with their partner (Uganda Joint BCC Survey, 2012). Regarding HIV, similar analysis shows that, on the whole, better outcomes were observed among exposed respondents than those not exposed. Nineteen percent of respondents exposed to HIV messages reported to have used condoms at last sex compared with 13% of those not exposed (Uganda Joint BCC Survey, 2012). AFFORD’s UHMG individual campaigns have all been shown to have high exposure levels. These include 52.9% (males, 54.1%; female 49.3%) exposure for the ‘Sexual Network Campaign’ (partner reduction campaign); Smart Choices (family planning campaign) [35.7%; males 31.6 %, females 36.4%]; and GENEXT Campaign (advocacy campaign for small family size norm)[33.8%; males 34.7%, females 29.9%]. All have been shown to have positive association with health outcomes. Two factors have contributed to AFFORD projects’ recent success in mass media health communication. Firstly, there are several talented in-house mass media and media-buying experts who are able to produce trendy messages. Secondly, AFFORD is supported by a substantial budget to acquire media time on a scale many other social marketing organizations in Uganda are unable to afford. There is enough evidence to conclude that AFFORD’s mass media messages have had a wide exposure and have increased awareness of millions of adult Ugandans about key health issues. In terms of outcomes in behavior change, data show that those exposed to AFFORD/UHMG messages are more likely to have embarked on healthy behaviors. However since there are several messages and campaigns by other implementing partners as well, one is unable to attribute behavior change solely to AFFORD/UHMG interventions. AFFORD’s analysis was unable to partial out the effects of other messages from other partners since data was not collected on exposure to non- AFFORD messages. Which ones were less successful? Compared with mass media, two of AFFORD’s social marketing strategies were found to be less successful. These are: (1) Working with most-at-risk populations MARPs, and (2) Good Life Clinics ‘Social Franchising’. Working with MARPs and other community level groups Several HIV prevention projects have demonstrated the feasibility of social marketing to promote sexual health among MARPs. CARISMA program in the Caribbean and USAID-funded ‘Social Marketing and Targeted Communication for HIV and AIDS prevention among most-at-risk Populations in Burma, China, Lao, and Thailand region are good examples of social marketing projects aimed at promoting preventive behavior among MARPs . CARISMA has generated lessons learned about how to design and implement social marketing interventions tailored to the needs of MARPs. In sub-Saharan Africa, social marketing approach is currently being used for HIV prevention among MARPs in Nigeria through USAID-funded ‘Strengthening HIV Prevention Services for MARPs’ project. One of the sub-questions the evaluation team was asked to address is the extent to which AFFORD was successful in increasing uptake of products and services among hard-to-reach and high risk populations. AFFORD, through four subgrantees, implements interventions at the community level for most-at-risk populations (fisher folks, 11 female sex workers and truckers) as well as for youths and married couples. UHMG subgrantees have several community volunteers and peer educators involved in community level behavior change. In terms of numbers, there has been an increase in the persons reached with messages. Based on AFFORD data the number of truckers reached has increased from 2,667 in 2011 to 3,229 in 2013. Similar increases were observed among the two other MARPs (FSW and Fisher Folk). The community level interventions also targets couples with messages on fidelity, HIV counseling and testing and family planning. The number of couples reached has doubled from 74,376 in 2011 to 149,818 in 2112. Although the number reached has increased, there are no survey data to evaluate the extent to which behavior has changed as a result of AFFORD interventions. The evaluation team therefore based its findings on field interviews, observations and project data. If the implementation of MARPs interventions at the community level continues in its current form, it is unlikely to achieve behavior change particularly among fisher-folks, youth and married couples. Among the various subgrantees, the evaluation team observed different levels of enthusiasm for the work. Only one subgrantee showed some success in its approach to reach MARPs. The NGO is NAYODE in Kasese district. They benefit greatly from a dynamic leader who is committed and effective. Sex worker interventions in Kasese look promising, because of the ingenuity of this subgrantee. The fidelity intervention the evaluation team saw in Mbale among married couples was very patchy and uncoordinated. As mentioned earlier, it appears sex workers are more likely than the other risk groups to insist on condom use. Our discussion with MARPs suggests that men, particularly fisher-folks in Kalangala, have not yet absorbed the message of risk aversion. While the fisher-folks maintained that they use condoms, sex workers in the community said this was not usually the case. In our focus group interviews, the FSWs were among the most motivated to share information and support their colleagues to get diagnosis and treatment. In Kasese they have formed a support group for FSWs, including those who are HIV positive. They support pregnant FSWs in getting PMTCT to ensure the birth of a healthy, HIV negative baby. They appear to have some model activities that should be further explored and documented such as the potential for collective efficacy to ensure condom use by all FSWs. One of the weaknesses the evaluation team observed is the lack of a behavior change theory that underpins the interventions among MARPs and other groups. Although AFFORD has documentation of the theories they reported using in their interventions, the evaluation team saw no evidence that the theories have been operationalized to inform the design and implementation of community level interventions among MARPS and other groups. The evaluation team also noted other implementation challenges also impact on the success of interventions. These included inadequate record-keeping, lack of connection with the GLCs in the community (except for NAYODE), and different levels of support from AFFORD. Each subgrantee has been receiving 3-month to one-year contracts, renewable at the end of each year, through competitive bidding. Technical support for the subgrantees, that would enable them to perform at a high level, is inadequate, mainly because UHMG has limited experience in community level interventions, and AFFORD, over the past three years, has not considered this as a priority. 12 Were AFFORD’s social marketing approaches more successful in achieving this result in some situations or locations or among some population groups than others? The evaluation team sought to find out the extent to which AFFORD interventions were being implemented in most needed areas. HIV Social marketing of condoms is an important component in a range of HIV prevention strategies. HIV prevalence varies by region, from 4.1% in Mid-eastern region to 10.6% in Central Region. As mentioned earlier, AFFORD/UHMG has implemented successful nationwide HIV prevention mass media campaigns The evaluation sought to ascertain whether HIV prevention efforts were implemented in areas of high prevalence. Since AFFORD/UHMG’S data on condom sales were not neatly disaggregated by regions, the evaluation team used the proportion of outlets stocking condoms as a proxy indicator of HIV prevention efforts. On the whole, stocking of Protector condoms is high in nearly all regions, with high prevalence regions having the highest stock. This suggests that AFFORD/UHMG is making condoms available in outlets situated in regions where HIV prevalence is highest. For example, as shown in Table 4 the highest proportions of outlets with Protector condoms (88%) was recorded in Central 1, the region with the highest prevalence (10.6%). However, one region needs special attention, as it appears to be underserved in terms of placement levels at outlets. Mid north has one of the highest prevalence figures (8.3%) but only 56 percent of outlets stocked condoms in July Table 4. HIV prevention activities by region and HIV prevalence Region HIV prevalence % stocking Protector (June 13) % stocking Condom O (June 13) No. of districts with AFFORD interventions Existence of MARP interventions Kampala 7.1 81 43 1 Central 1 10.6 88 17 11 Kalangala Central 2 9.0 84 6 11 East Central 5.8 81 8 8 Mid Eastern 4.1 57 19 14 North Eastern 5.3 56 10 15 Mid Northern 8.3 56 10 14 Oyam West Nile 4.9 61 17 8 Mid Western 8.2 75 16 10 Kasese South West 8.0 81 20 4 National 7.3 79 18 96 13 2013. AFFORD must be commended for siting its youth intervention in the region, but more outlets are needed to satisfy the demand that may be created. It is also important to mention that AFFORD community level HIV prevention efforts among most-at-risk populations (fisher folks, female sex workers, and truckers) are in regions of high prevalence. There is an intervention among fisher-folks and sex workers in Kalangala island (Central 1 Region). There are similar community level HIV prevention interventions among key populations in Kasese (Midwestern Region), Oyam (Mid-Northern Region), and among married couples in Mbale (East). Table 5. Outlets stocking contraceptives by Region, TFR, CPR and Unmet Need Family planning With a total fertility rate of 6.2 children, women in Uganda have one of the world’s highest fertility rates. Although Uganda’s fertility rates are generally high, there are substantial regional variations, which should be taken into account in family planning programming. Total fertility rates (TFRs) range from 3.3 children in Kampala to 7.5 children in the North Eastern (Karamoja) region. It is worth mentioning that with the possible exception of Kampala, all regions need family planning interventions. However, given that the regions differ regarding their contraceptive needs, the evaluation team sought to find out whether AFFORDs family planning interventions target regions or locations with the highest need in order to obtain the maximum impact. The evaluation team used three indicators - total fertility rate (TFR), contraceptive prevalence rate (CPR), and unmet need for family planning to identify regions of highest need for family planning intervention. As shown in Table 5, the regions of highest need include Mideastern/Eastern (TFR=7.5; unmet need 38.3); North eastern (TFR=7.5; unmet need 20.5). Others are East Central and West Nile. There is no evidence to suggest that AFFORD is targeting regions of highest need for family planning. The starting point of successful family planning campaigns is to address unmet need of family planning. Areas of high unmet need, in some way, reflect potential demand for family planning. On the whole, as shown in Table 5, the regions with the highest unmet need - (e.g. Mid north, West Region TFR CPR Unmet Need % of outlets with PilPlan Plus % of outlets with Injecta Plan % of outlets with Newfem % of outlets with Moon Beans % of outlets with SoftSure No. of GLC Kampala 3.3 40.2 16.6 92 81 10 4 19 22 Central 1 5.6 30.7 26.5 92 81 4 2 11 29 Central 2 6.3 30.7 35.4 81 64 3 1 3 24 East Central 6.9 27.7 41.9 72 65 1 0 5 13 Mid Eastern/Eastern 7.5 23.2 38.3 63 63 7 2 11 13 North Eastern/Karamoja 6.4 7.4 20.5 51 68 1 1 4 20 Mid North/North 6.3 23.4 42.5 51 68 1 1 4 12 West Nile 6.8 13.6 42.9 50 39 0 0 4 11 Mid Western/Western 6.4 26.8 30.4 86 74 2 2 6 27 South West 6.2 25.1 36.9 90 87 5 1 10 19 National 6.2 26.0 34.3 85 78 4 2 9 200 14 Nile, Mid-Eastern/Eastern) have the lowest placement of PilPlan Plus, Injectaplan and the other contraceptives. AFFORD needs region-specific interventions to address family planning. Interventions must reflect regional variations that explain the unmet need. 3. What specifically is AFFORD’S contribution to the delivery of health and HIV/AIDS services in Uganda? How does AFFORD rank among condom and contraceptive distribution/sales agencies in Uganda? Discussions with key stakeholders, including government officials, donors, and NGOs, confirmed what the evaluation team had observed: that AFFORD/UHMG is a key player in the area of distribution of health products in Uganda. The Uganda Demographic and Health Survey 2011 records that one in four pill users (25%) use PilPlan (the earlier version of AFFORD’S Pilplanplus while three in ten (29%) use Protector condoms. There is evidence that AFFORD currently dominates market share in many of its product and price-point categories (see Figure 5). Figure5 Competitor Market Share 48 41 42 18 18 20 17 16 16 6 8 8 9 15 13 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Apr-13 May-13 Jun-13 Other RoughRider Condom o Trust Lifegaurd Protector Source: Retail Audit (Q2,2013) 15 Figure 6 Market Share (Female Contraceptives) Although Protector’s market share has been declining over the past few months, according to retail audit reports, it still had 42 percent of the market in June 2013, compared to 48 percent in April 2013. Its nearest competitor- Life Guard - holds about 20 percent of the market. AFFORD products are even more dominant in the market share of female contraceptives. The Q2 2013 retail audit in June 2013 (Figure 6) shows that Injectaplan had 88 percent of market share among injectable contraceptives, and Pilplanplus had 95 percent among oral contraceptives. However, in the sale of water purifiers, the AFFORD product – Acquasafe - lags behind its competitor Waterguard (42 % and 54 % respectively). In oral rehydration salt (ORS), the AFFORD product has 39 percent of market share compared to Oralyte (53%). Cotramox is the market leader in its category – holding 33 percent of the market share. At the national level, AFFORD, through UHMG, is a leading distributor of condoms. From data available, and shown in Figure 7, MSI was for three years (2007-2010), the largest distributor of condoms, but was overtaken by UHMG is 2012. In 2011-2012, AFFORD through UHMG distributed more condoms than PACE and MSI put together. With respect to contraceptives, including condoms, UHMG has the largest share of Uganda’s market. 88% 12% 95% 4%1% 98% 2% 83% 17% 100% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Moon Beads Microgynon Bayer Newfem Ovulet Softsure Microlut Pills Microgynon Schering Pilplan Plus DepoProvera Injectaplan Retail Audit 2013(Q2) 16 Figure 7 Number of Condoms Distributed in Uganda (2007/08 to 2010/11) Can the “Good Life Clinic” approach be classified as true Social Marketing? One approach by AFFORD aimed at improving access to health services nationwide is through social franchising of Good Life clinics. Under AFFORD I, AFFORD was providing HCT, and care for HIV positive clients. AFFORD through UHMG has, since 2009, established the Good Life social franchising to apply social marketing values and expertise to clinical services. Two hundred clinics have been accredited so far and offer tiered service delivery outlets, with about 300 still to be considered. Given that a major cause of maternal mortality is lack of access to facilities, GLCs were designed to bring services closer to women. GLCs therefore have the potential for health market expansion by providing services to those who would not otherwise be covered by the existing health system. Nearly all persons who have utilized their services are women. According to AFFORD/UHMG’s official position (GLC Strategy 2013), the clinics are chosen as follows: “The selection process targets interested parties to be enrolled into the GLC Network. This is a purposeful activity carried out on a regular basis by UHMG technical team…” Under AFFORD II, as part of the private sector strategy, there has been an increasing emphasis from donors for AFFORD to expand to clinic-based service delivery for long-term contraceptive methods like implants and IUDs, and for maternal and child health services. The concept and theory behind the Good Life Clinics can be classified as Social Marketing, although it has been poorly planned and implemented. Social Marketing addresses the lower income segment of society with products and/or services that are both accessible and affordable. GLCs do this to some extent. The Good Life Clinic is an attractive idea that needs to be properly developed to live up to its potential. Most of the family planning and MNCH services offered primarily focus on women. Although AFFORD should be commended for establishing and scaling up the franchised network of clinics, many opportunities were missed in the formation of the GLC franchise. There is no evidence an operations manual was included in the franchise agreement. An operations manual is the “franchise bible,” and varies in length and scope depending on the business, but normally includes quality standards, 0 10,000,000 20,000,000 30,000,000 40,000,000 50,000,000 60,000,000 70,000,000 80,000,000 90,000,000 2007/08 2008/09 2009/10 2010/11 MoH MSI PACE UHMG Source: Uganda Global AIDS Response Progress Report (2012) 17 brand identity, management policies and procedures, and delivery systems. AFFORD could have proposed robust data collection procedure based on metrics to measure the success of Good Life Clinics in terms of access, quality, cost-effectiveness and equity. There has also been fierce competition among family planning agencies to add services to the same clinics. Some GLCs have been branded by other organizations. This diffuses the impact and importance of the brand and is counter to the principles of Social Franchising. It is not uncommon, for example, to find one agency (franchisor) providing long-term methods and another one providing short-term methods in the same clinic. Now the competition has expanded beyond family planning to other services such as MNCH. For example, a GLC interviewed by the team mentioned their interest in seeking out another implementing partner, NUHITES, because free mama kits were available through them, and not through AFFORD. The evaluation team saw that competitor organizations are also poaching volunteer outreach staff by attracting them with enhanced incentives. What are the missed opportunities/untapped potentials, if any? At the GLC level, the missed opportunities include adequate planning for quality, appropriate QI instruments prior to accrediting GLCs, standardized packaging of services to the different tiers of GLCs, data collection to identify clinic quality problems and to find timely ways to address them, and learning tools for best practices in health service delivery including how to improve clinic supervision and management. Another opportunity missed early on was the design and implementation of Operations Research (OR). It’s unlikely that key lessons from the project will be evidence-based, as several useful aspects of the project are not systematically documented. AFFORD/UHMG could have used the opportunity to build into the project design specific OR interventions at little or no cost. For example, are GLCs with active Good Life promoters doing better than those without? Are districts with Mother’s clubs/or fidelity clubs having better outcomes in HCT among couples? Are the interventions among fisher folks in Kalangala which is less integrated doing as well as those among fisher folks in Kasese where integrated services including drop-in centers and youth centers are included? How do intervention outcomes differ in communities where sex workers are well-organized (as in Kasese) compared to those in Kalangala? How can we assess the effect of the apparent existence of collective efficacy to condom use among sex workers in Kasese versus the efforts at individual efficacy, as practiced in Kalangala? 4.To what extent did AFFORD strengthen UHMG systems and capacity for technical, financial, institutional, and market sustainability? UHMG has had a good start. It is one of the few social marketing organizations to have its own official building and warehouses with an excellent haulage and bulk distribution system. UHMG also has the beginnings of a Social Enterprise in place with the warehouse or Product Facility (PF) becoming a profit center in its own right. This new income stream would complement other income streams such as their retail pharmacies and branded product distributions. To achieve sustainability in a social marketing program, the social programming side of the equation needs supplemental support from additional income streams. UHMG has benefitted from JHU through AFFORD. UHMG, through its relationship with JHU, should have had access to global best practices in IEC and BCC materials, which could have been adapted to the local markets. The evaluation team observed that while capacity has been 18 strengthened by AFFORD in certain areas, there are still gaps in critical others including research. Technical Capacity: Appropriate technical capacity to carry out the organization’s social marketing mandate is critical to institutional sustainability. AFFORD has succeeded in building the technical capacity of a segment of the organization notably in mass media health communication. They are now able to design and deliver campaigns of national reach. In addition, AFFORD has succeeded in building the capacity of UHMG in certain core areas including international procurement, warehousing and haulage that facilitate distribution of products to key towns (although not to the last-mile). There are still some core areas of social marketing where success in capacity building has been limited. These include research, marketing, programs, social franchising through and interventions among most-at-risk and other groups. The evaluation team observed that efforts were made under AFFORD I to build the capacity of staff in the other arms of social marketing, but AFFORD has pulled back in the last three years apparently to concentrate on financial sustainability. MSI and Programs As described earlier, the team believes that MSI is weak and must be strengthened in its ability to provide leadership in the social marketing of AFFORD products. There is an apparent lack of skills in social marketing approaches and AFFORD has been unable to develop strategies to improve UHMG’s social marketing arm. The MSI team is not abreast of sales, and has not been able to work with other teams to conduct the appropriate research to understand strengths and weaknesses in its product lines and distribution systems. It must apply an entrepreneurial mindset to get product to the last mile and ensure that AFFORD maintains its leadership in the marketplace. MSI must improve its data collection and analysis, and develop new strategies accordingly. MSI must also work with the PF to ensure that appropriate incentives are built in so that PF meets its goals in the sale of AFFORDS’s own brands. Programs and Services, the division responsible for AFFORD’s outreach of quality products and services at the community level must also be strengthened to meet the demands of its current agenda. As mentioned earlier, there has been a considerable lack of foresight, planning and strategy to address the needs and requirements of a growing social franchise. There is also a lack of significant experience within the division to adequately support quality clinical activity. There is a need to have a dedicated Ob/Gyn or nurse midwife with extensive expertise in short and long-term family planning as well as MNCH to work on GLCs. Research: JHU through AFFORD has strong and experienced staff who undertake detailed data analysis. Over the years, efforts have been made to strengthen the capacity of UHMG. AFFORD has not been successful in building the capacity of research staff. Currently UHMG lacks adequate research skills. Critical research skills needed include consumer insight studies, formative research, segmentation analysis, data analysis, evaluation skills to access program effects, distilling key findings for programs, and operationalization of key theoretical concepts into questionnaires. Use of evidence and theory to inform behavior change As stated earlier, the evaluation team observed that AFFORD has a list of behavior change theories but AFFORD has not been very successful in providing UHMG with skills in the operationalization of theories to inform the design and implementation of community level interventions being implemented by the UHMG subgrantees. The use of theories is an essential part of social marketing. 19 Weak management information systems Social marketing demands very robust and sensitive data gathering, collation, and dissemination for use within and outside the organization. However, in UHMG’s case, there is only a weak system to collect and use data to track progress. Management of Social Franchising GLCs have become an integral part of AFFORD’s health service delivery. The GLC franchise network is the key mechanism for provision of quality products and services at the community level. It is therefore an essential ingredient for a successful social marketing effort. There are technical skill gaps in AFFORD to successfully manage clinical social franchising. In the team’s observations and through secondary data provided by AFFORD, the team discovered weaknesses in a number of areas including record keeping and data collection, quality improvement, counseling, consistency of supplies and services and IEC. Detailed findings on the evaluation team’s observations at GLCs are presented in Annex 6. Management of Subgrantees and Community Level Work with MARPS: As UHMG grows, its subgrant portfolio will increase. However, unlike many social marketing organizations that engage subgrantees after long years of direct implementation of community level activities, UHMG has embarked upon the scheme without themselves having had sufficient, practical experience of community level interventions. Consequently, there are weaknesses in their management of subgrantees, which need to be addressed. Areas requiring further strengthening, at subgrantee level, include behavior change and related data collection, improved counseling and informational materials tailored to the target audience, and consistent levels of support for peer and health educators across subgrantees and youth-friendly services. One subgrantee, NAYODE in Kasese, mentioned outreach activities in local schools as well as for out of school youth. They operate a youth center where we were able to conduct interviews with young adults. Participants in this interview session were asked about their experiences with GLCs. They responded that GLCs are not youth friendly, and they were often turned away. (details are presented in Annex 7). Institutional Capacity Sustainability: There has been tremendous progress in achieving both institutional and financial sustainability, particularly in the past three years. UHMG was incorporated as a company limited by guarantee in 2006, and in 2011 was registered as an NGO after providing extensive justification that PF profits were fed back into social marketing activities. Staff grew from 31 positions in 2007 to 118 positions in 2013. The Product Facility has broken even for the first time this year. As of August 2013 they have covered all their payroll and operating expenses. The warehouse facility has exceeded the limits of current space and additional space is under construction. During the past three years, significant effort has also been invested in strengthening systems in governance, finance (including internal audit), human resources, procurement, subgrantee management, PF quality, ICT policy and vehicle policy. As of 2013, manuals exist, in completed or draft form, for each of these areas and are posted on the UHMG K4Health website. In 2012, UHMG was awarded as the prime contractor of a USAID project, the Uganda Good Life Integrated HIV Counseling and Testing Kampala Project. In addition, they are subcontractors on several other grants. 20 However, there remains a heavy reliance on donor funding, particularly USAID. Various management staff and Board members pointed out that 80% of UHMG’s budget comes from USAID and that USAID owns 90% of product facility assets (primarily trucks for distribution and a packaging machine for condoms). As of 2013, 72% of UHMG staff are still paid by AFFORD (Annex 9). Without continued support for the AFFORD project, UHMG risks losing most of its staff. The team’s interviews with Product Facility staff indicated that 80% of their customers for warehousing and distribution are donors (USAID, DFID and UNFPA) and the Ministry of Health. Only 20% of their current business is with other commercial entities. USAID also continues to support heavily the Product Facility’s media, communications and promotions efforts as an integral part of social marketing. UHMG does not yet have an established NICRA. This would help the organization cover its overhead costs and provide support for core institutional needs and activities not covered under project-specific budgets. The NICRA matter is still pending with USAID, and requires regular follow up. Finance and Administration: Through interviews with Board and Staff, as well as review of audit reports from Ernst and Young, and technical assistance reports from KPMG (a local branch of a Swiss management consulting firm) in 2012 and 2013, there has been major progress in reducing financial risk and in tightening financial controls. UHMG has also passed the A133 audit, which measures finance, assets and compliance in accordance with USAID requirements. All finance areas are now on the Tally software system, which facilitates preparation of reports and analyses for all areas of the institution’s income and expenditures. In addition, the Administration wing of UHMG was reassigned to Finance in 2013, after originally reporting to the HR division. The consolidated Finance and Administration unit is now responsible for guiding budgeting, consistent tracking and monitoring of expenditures; ensuring compliance with donor requirements; internal controls, risk management, financial reporting across the organization, coordinating external audits, monitoring revenue, costing proposals for UHMG, management of subgrantee financial compliance, managing UHMG investments (identifying idle funds and looking for the best opportunities to invest), managing mortgage and property, managing vendors and contracts, ICT, security, vehicle logistics, assets for UHMG, and general administration and office management. The evaluation team saw samples of reports on subgrantee financial compliance from the Project Finance Manager and is satisfied that they conduct “mini-audits” of subgrantees. Both through interviews, the Ernst and Young audit report and KPMG documents, the evaluation team learned that PF’s level of reporting within the overall accounts still needs to be improved; their data management and stock management continue to be an issue. The Finance and Risk Director now provides financial oversight to the Product Facility, and a staff member came on board as Finance Manager for PF. These steps should continue to promote integration of PF’s systems with the rest of UHMG. Governance Governance has many layers at UHMG. According to the Board Manual of June 2013, there is a Founder’s Forum, a Board of Directors, and Senior Management. The Founders’ Forum is a newly constituted body and consists of the Founders, former Board members and an individual who provided technical assistance to UHMG at its inception. New members can be elected to the Forum if they can demonstrate a commitment to the Mission and a history of contribution to the organization’s agenda. Through interviews with Board, Founders, and Management, the evaluation learned that the terms of reference for the Founders’ Forum have not yet been developed. 21 The team was also told that the Founders’ Forum might actually be only an advisory body to the Founders. The final form this will take has yet to be determined and the team was led to understand this should be resolved at the Annual General Meeting on September 6, 2013. Based on our interviews with Founders and Board members, the purpose of the Founders Forum is “to ensure that the Board is on Mission and sticking to the mission/vision/values of the organization…and to ensure that the Board plays its role properly.” The Founders’ Forum, according to the Board manual, has the power to appoint Board members and to approve the appointment of the Managing Director. The evaluation team was shown a new diagram of the governance structure on September 11, the day after the evaluation assignment ended. According to this diagram, the Board now reports to a body called the “AGM,” or Annual General Meeting, and the Founders Forum reports on a dotted line into both the Board of Directors and the AGM. Thus there is still a group to whom the Board reports: the AGM. To the knowledge of the evaluation team, Founders, Board members and Senior Management attend the AGM, so it is still not clear which of these parties sits on top of the Board. Since the TOR for the new Founders Forum and AGM have yet to be determined, the power and decisionmaking dynamics remain the same. There is still no clear distinction in roles and responsibilities among the four bodies: senior management, the Board of Directors, The Founders, and the AGM. Without clear guidance, there is bound to be confusion in decisionmaking in UHMG. UHMG was incorporated as a company limited by guarantee in 2006. The subscribers to the organization at that time were the Founding members. In March 2011, UHMG was registered as a Non￾profit organization after significant documentation was provided to justify that all profits from the PF were plowed back into supporting the sustainability of UHMG and its social marketing mandate. Of the 7 founder members, only 3 were part of the first board. The first board comprised 7 members, i.e. 3 founder members, 3 board members, and the Managing Director. There are still two founder members serving on the Board of nine Directors. Board composition: The Board composition is as follows: a Public Health specialist with a pharmaceutical background, a medical doctor (area of medicine not specified), a marketer, a finance professional, a Public Health Specialist from the MOH, a Law & Marketing expert, a Business Development expert, an Auditor, a Finance person and a Lawyer. The Board is fairly gender-balanced with four females and five males. While this shows a diverse board in terms of range of areas covered, it does not necessarily reflect a balance between Programs Product Facility arms of the organization. Only one person on the current board represents the public sector and reproductive health programming. The other public health specialist has a background in pharmaceuticals. Six members represent the private sector in a variety of forms including law, audit and finance. BoardSource, in its guidelines concerning Board composition recommends representation of those from the community being served as well as those with expertise that the organization needs. A better balance of Program-related expertise to PF-related expertise is therefore desirable. Directors with social marketing experience, BCC/research background, health services delivery, program monitoring and evaluation background, or community outreach background in HIV, would be appropriate additions to provide greater equilibrium on decision-making related to the broad range of UHMG activities. Board Manual: A Board Manual was first drafted in 2006, and was revised in 2010, 2012 and 2013. Review of the manual suggests some areas of potential overlap between Board and Management, some weaknesses regarding conflict of interest, and too many decisionmaking layers for effective operation. After the Founder members evolved into a Board of primarily non-Founders, the current Board manual describes the creation of a Founders’ Forum to 22 oversee the Board. The manual states that, Founders “watch over directions and conduct of the Board to ensure that the Organization continues to move in the right direction and maintain the agreed values.” This should be the role of the Board of Directors. Founders also have the power to approve any candidate for the Board of Directors, as well as the appointment of the Managing Director, another role generally reserved for the Board in nonprofit organizations. The Founders have lifetime appointments; Board Members change every 3 years, but can be renewed once for a maximum of up to six years before a break would be required. New people can be invited to join the Founders Forum, if they have a demonstrated record of service to the organization (e.g., former Board of Director members). The Founders’ Forum, with lifetime appointments, has greater power than those with limited terms of office, thus shifting the balance of power from the Board to the Founders’ Forum. The manual states that the Board “shall be responsible for the day-to-day running of the organization…” as well as “…making policies and decisions on behalf of the organization.” Although our interviews with Board members indicate that some see day-to-day operations as part of their role, others do not. There has been conflict noted between Management and Board, both through this team’s interviews and in KPMG’s technical assistance report. Boards delegate the day-to-day responsibilities to Managing or Executive Directors. Involvement in day-to-day activities oversteps the Board’s role, and confuses the lines of authority within the organization. According to the Board manual, the Board (1) “may appoint Senior Staff,” (2) has the power to “employ and dismiss employees,” and (3) has the role of appointing the right people with the right qualifications and skills for all jobs.” Organizations that focus on strengthening nonprofit leadership and governance, such as BoardSource, recommend clear demarcation of Board and management responsibilities and delegation of these management roles to the Managing or Executive Director. The manual further mentions the role of “entering into contracts on behalf of the organization.” Organizational contracts with vendors, donors, and subgrantees are generally part of the day-to-day operations of the organization. If there is agreement between Board and Management on overall mission and strategy, Management should be able to proceed with these contracts, and inform Board members of all contract activities in line with Mission and Strategy. Some Board members expressed the need for the Board to review all proposals and their key staff prior to submission to donors. This, too, is an example of Board conflict with the management role. With the tight deadlines and extremely intense work environment required to produce proposals, time for Board review would cause enormous delays and threaten the competitiveness of the organization. Other Board interviewees did not feel a need to review proposals, and were interested in only those documents affecting overall policy. Conflict of Interest: The Conflict of Interest clause in the Board manual leaves open the possibility of Board members conducting personal business with the organization. It indicates that such members must declare their conflict of interest and absent themselves from decisions related to their business. This poses a serious risk to the organization and its reputation. The manual should state that no Board member or family of a Board member should be doing business with the organization, unless they are clearly documented as the sole source of the service or product in the country. The evaluation team inquired about this but there was no reported knowledge of Board or family members carrying on business with UHMG. Terms of Service: Board members have varied understandings of the length of their terms of service. The evaluation team was told that Board members are appointed to Committees for one year renewable, which is too short a period to serve the Committee well. Some Board members believe their overall appointment to the Board is for two years plus a renewable third year. After that they must take a break from the Board before further service. However the Board chair clarified that terms are 3 years, 23 renewable for another 3 years, at which time a break must be taken before further service. A good orientation system for new Board members would serve to clarify this. HR and Procurement: There has been considerable progress in the area of HR systems and implementation. A new HR manual was revised and approved in 2013. It now includes policies on training and staff development, a new USAID-funded internship program, rates of reimbursement for expenses, rates for consultants, conflict of interest, whistleblowing, and intimate relationships in the workplace. The Performance Management system is in place on paper, and was implemented with 18% of staff – mostly those in senior positions (see KPMG Capacity Assessment Report, January 2013). Staff assessments were conducted with these 21 managers and directors, and feedback was provided on strengths and weaknesses as well as plans for capacity development. As a result of this exercise, some job descriptions were also rewritten. There are now new appraisal forms that require measurement of staff both by KRAs and KPIs. KPMG observed that there had not been a history of solid performance management in the organization. In their assessment, they noted a “lack of definition and guidance on performance,” a “communication gap between managers and supervisees,” and “deliverables not well defined.” Therefore, supervisors will need mentoring and feedback training to ensure appropriate oversight of and dialogue with staff toward achieving work objectives, and developing the skills necessary to fill gaps in knowledge. Low salaries have been a persistent problem at UHMG. This year, they lost some staff members to competitor, Marie Stopes Uganda. The evaluation team was told that, in recruitments, low salaries are the reason they do not get their top two candidates for a position. A salary survey was conducted in 2010, but limited comparison to only Ugandan organizations. However, UHMG’s competitors are primarily local branches of international NGOs, like PACE and Marie Stopes Uganda. In addition, benefits packages are extremely limited in comparison with organizations like PACE and MSU. While health benefits are offered, there is currently no pension plan and no educational or transport allowances as often provided by these other competitors. Data from UHMG show remarkably high turnover rates in 2009, 2010 and 2011. However, it is noted that out of 120 current positions in 2013, only 8 have turned over this year. Therefore turnover has dropped considerably. The evaluation team learned that the high rates of turnover were due to the ending of contracts for specific projects. UHMG does not have a strategy for transitioning staff from AFFORD to other positions, nor does it currently have a mechanism to ensure that core technical staff are not lost to UHMG when a contract finishes. Efforts have apparently begun to assign percentages of senior management staff to different project budgets. However, most contracts that UHMG bids on do not allow for key project staff to provide only a percentage of their time. There is gender equality in management. Several women serve in key leadership and management positions at UHMG. The Managing Director and half of the top management staff are women. Who owns UHMG? In our interviews with Board and Founder members, the evaluation learned that most think that the Founders own UHMG. Only two Board members interviewed believe the Founders are not the owners. One responded: “No one. The Founders do not own it; in forming a company with limited guarantee they subscribe to it, they set it up; nobody owns UHMG; society owns it; it is set up on behalf of Ugandan citizens.” Another said, “There are no shares, nothing to be distributed. We are the ‘custodians of the vision.’ ” 24 The Memorandum of Association and Articles of Association dated 2006, state that there are no shares, and upon dissolution of the organization, should assets or property remain after payment of debts and liabilities, “…the same shall not be distributed among the Members of the Organization, but shall be given or transferred to some other Institution or Institutions having objects similar to the objects of the Organization, to be determined by the Founding members.” So, while, on paper, no one owns UHMG, Founders and Board tend to believe the Founders own UHMG. Stages of Organization Development In the Pact South Africa, Organizational and Capacity Assessment Tool (OCAT), “Sustainability refers to the long term continuation of an organization, program or project.” It entails both establishing mechanisms for the smooth functioning of governance, management, finances, human resources, and technical services, as well as the stage of an organization with respect to each category. The evaluation team applied this OCAT and the identifiers of the different stages of Organization Development are shown in Table 6. It should be noted that the team has not had sufficient time to develop and undertake a scoring system for the multiple facets of each UHMG division and their related activities. In general these OCATs are Table 6. Stages of Organizational Development Nascent The NGO is in the earliest stage of development. All the components measured by OCAT are in rudimentary form or nonexistent Emerging The NGO is developing some capacity. Structures for governance, management practices, human resources, financial resources, technical services and other key areas of the organization are in place and functioning Expanding The NGO has a track record of achievement; its work is recognized by its constituency, the government, the private sector, and the other NGOs active in the same sector Mature The organization is fully functioning and sustainable, with a diversified resource base and partnership relationships with national and international networks Table 7. UHMG Stages of Development Area Nascent Emerging Expanding Mature Governance X Overall Sustainability X Product Facility Commercial X Social Marketing X MSI Mass Media and Comm. X Social Marketing X Programs and Services X Finance and Admin. X Human Resources X 25 used for self-assessment and participatory analysis by an organization and its stakeholders. The evaluation team therefore applied this framework as a visual tool to suggest where UHMG stands, based on our observations at selected GLC and subgrantee sites, interviews with stakeholders, and the documents and data that have been provided by UHMG (Table 7). The team concludes that UHMG is predominantly in the “emerging” phase of development. Systems and structures are in place; however, in some areas, they require significant strengthening, testing and further application of best practices, particularly in social marketing and programs. Three areas of UHMG are in the “Expanding” phase: the Commercial wing of the PF, the Mass Media and Communications wing of MSI, and Finance and Administration. CONCLUSIONS UHMG is the kind of organization that has a niche in Uganda, and could be playing an important role - not only in expanding sales and reach for its products, but in providing quality services and reducing risk among MARPs. Accessibility and Affordability: One key finding was that, based on retail audits alone, from 78% to 85% of outlets stock the three USAID-donated products, the proportions stocking UHMG’s own brands are still low (2% to 52%). But the use of this measure masks the fact that rural areas are still badly underserved. It is important that accessibility is measured from the client’s perspective in terms of the ease, in distance or time, for them to access products. Notwithstanding the product placement levels, the overall sales figure for UHMG social marketing products can be rated as poor. On the whole, there has been no increase in the sales volume of Protector condoms since 2009. Although AFFORD’s commercial brand Condom ‘O’ showed appreciable increase in the early years, it has since plateaued, even if targets are met in 2013. However, the fact that Condom ‘O’ is competing favorably in its category of mid-priced condoms is an indication that it can have a good future in the market. When population is growing, more people become sexually active and one expects a social marketing condom would grow accordingly. If this does not happen, it can be a signal that more and more people in the lower income brackets may not be adequately embarking on risk reduction behavior (condom use). It is also possible that the growth of the free product user segment has reduced demand for economy-priced products, like Protector. There is a need for a comprehensive study of the condom market in Uganda using the total market approach to distill the market dynamics and how the various types of markets (social marketing, public and commercial sectors) are performing. The evaluation team believes that AFFORD’s aggressive mass media campaigns are not supported by a nation-wide strategy to push products through several distribution channels. AFFORD relies only on a few regional distributors and occasional institutional buyers. It is believed that when the distribution pipelines are filled, the product will reach the consumer. This does not work in most African countries. There are no UHMG strategies to reach the consumer in rural areas. UHMG relies mostly on the percentage of outlets observed that carry their product. Quantity of stock on hand is not an indicator. AFFORD’s Relevant and Effective Social Marketing Campaigns: AFFORD’S success in mass media health communication is very impressive and underscores the organization’s strength in using mass media. UHMG’s data show that the messages helped to increase knowledge, enhance spousal communication about family planning and contraceptive use. Persons exposed to the messages were more likely to have increased knowledge, communicate with spouse about family planning or use contraceptives. This was true for both men and women. 26 The findings show that AFFORD interventions in HIV prevention is targeting regions of highest prevalence, but this is not so with family planning interventions. This suggests that, if not addressed, AFFORD’s overall contribution to fertility reduction at the national level through family planning may be minimal. While the concentration of program efforts in Kampala and Central regions may enhance the financial sustainability of UHMG, as a key player in national level, UHMG must endeavor selecting family planning sites, particularly GLCs, to reflect the national family planning needs. AFFORD interventions at the community level are not as successful as their mass media campaigns, although one of the subgrantees is doing well. At the community level, the team’s discussions showed that accessibility of AFFORD marketed condoms in and around hotspots, such as trucker spots, was patchy and depended upon the zeal and commitment of the community volunteer. AFFORD through JHU has resources and expertise that should have refined the community interventions. Behavior change is the end result of social marketing but this does not appear to be so with many field workers of the subgrantees interested only in counting numbers reached. If this is not rectified, and critical skills developed before AFFORD ends, UHMG runs the risk of becoming deficient in one critical approach to social marketing. This may undermine their competitiveness in community level programming. AFFORD’s contribution to delivery of HIV services: AFFORD through UHMG has succeeded in establishing an indigenous organization which today, provides about half of social marketing condoms and even higher proportions of family planning products. Given that UHMG is a young organization, its ability to distribute more products than its competitors in 2012 is a mark of success for AFFORD. AFFORD has also introduced social franchising of GLCs with the potential of providing access to women who otherwise could not access clinic services for family planning, ANC, HCT and malaria treatment. However, in spite of JHU’s worldwide experience and the existence of global lessons on social franchising, several opportunities were missed at the early stage. The involvement of GLCs in long-term family planning methods is particularly important considering the impact of long-term methods on fertility reduction. But without quality service improvement, GLCs can potentially undermine the resurgence of long-term family planning in Uganda. At the GLC level, Quality Improvement tools were designed late in the life of UHMG – in 2013, four years after the GLC concept was initiated. Manuals and standards should have been designed before accrediting GLCs. AFFORD and UHMG are now trying to catch up on a system that is now as large as 200 GLCs around the country, with 300 yet to be accredited. The current focus for GLCs should be on quality rather than expansion of clinics. The Evaluation Team has not seen any data from UHMG staff or medical consultants to indicate that clinical activity or counseling have been observed on a routine basis, or that they have discussed with GLC staff any problems they are facing in service delivery. How AFFORD has strengthened UHMG’s capacity A major objective is to evaluate the extent to which the institutional capacity of UHMG has been strengthened by AFFORD through JHU-CCP. UHMG has been successful in strengthening its internal financial systems and controls, monitoring subgrantee compliance, and developing HR systems and procedures. In addition, a NICRA proposal has been submitted to USAID. Documented policies, procedures, and operational guidelines are in place for finance and accounting, human resources, procurement, sub-grants, governance, and internal audit. The challenge now is how to apply the rules and procedures since several of these have been adopted only recently. AFFORD should use the remaining period to institutionalize their application and document lessons learned. However, based on this evaluation, UHMG has experienced a segmented strengthening of capacity. While structures and systems have been established, at the 8-year mark, UHMG is still struggling to achieve technical sustainability. The concentration on the part of AFFORD on UHMG’s 27 financial sustainability, interpreted narrowly to mean profitability of the PF division, has led to the near neglect of programs and social marketing. PF for example, has a growing commercial warehousing and haulage business, but has lost sight of UHMG product sales. MSI has forged ahead in external recognition for its success, while the Programs and Services division - at the heart of UHMG’s social marketing activities - is at risk due to lack of expertise in quality management of GLCs and subgrantees, and the need for a better theoretical and evidence-based grounding in behavior change and strategies to address MARPs. UHMG, in its current state, is weak in key technical areas in social marketing, notably research, and the application of theory to design and implement interventions. This can eventually undermine its competitiveness. In addition, financially, UHMG is still heavily dependent on USAID, and is at risk of losing the bulk of its staff once AFFORD ends. The evaluation team believes that UHMG should have been much further ahead by the end of eight years and much closer to becoming a fully mature organization by this point in time. Furthermore, internally, the organization does not have a clear strategy for how each division works together to support one Mission and Vision. UHMG and their Product Facility (PF) division operate as if they are independent organizations. It has even been proposed that the PF split off from UHM, as a possible subsidiary organization focusing more on for-profit activities. There is strong support for PF to go independent by a select few of senior management, a view the evaluation team does not share. The strength of UHMG lies in a strong synergistic relationship between the two in a manner that makes the organization a strong competitor in the social marketing arena in Africa. AFFORD has strengthened governance, to some extent, through the establishment of an operating Board with a procedures manual. However, UHMG is still in the early stages of developing strong governance practices with clear demarcation of roles and responsibilities among the various bodies at the top echelon of UHMG. Board and Management are in conflict about their respective roles and responsibilities to each other and the organization. The addition of a new layer of decision-making via the Founders’ Forum - or the AGM - will only complicate and further delay the already complex decision-making process, and further erode the authority of both the Board and Senior Management. This poses a serious hindrance to organizational decision-making, and likely has contributed in some way to losing sight of critical activities in the development and sustainability of UHMG as a social marketing initiative. Governance structures, including Board composition, a better understanding of the vision and mission of the organization, definition of roles and responsibilities of Board and Senior Management, as well as various policies stated in the 2013 Board manual must be addressed. Ownership of UHMG remains an issue. Most Founders and Board members interviewed believe the Founders own UHMG, while others believe they are stewards for the community-at-large. The original articles of the company of limited guarantee, registered in 2006, indicate that no one owns UHMG, in accordance with best practices in governance. It is important to stress the distinction between ownership and accountability. While no individual or group has ownership of UHMG, all facets of the organization must be held accountable to upholding the Mission and Vision. The team concludes that UHMG is predominately in the second or “Emerging” stage of the four stages of development of an organization (PACT, South Africa, OCAT). This is the stage where some capacity has been developed and structures and systems are in place for all of the major technical, organizational and governance areas. If efforts are put in place to test and run the procedures, improve governance and substantially build technical capacity, it is expected that within four years UHMG should be primarily in 28 the third or “Expanding” phase and that, at least, some areas should have reached the fourth stage of “Maturity.” RECOMMENDATIONS Accessibility and Affordability: 1. In order to have a clearer picture of product accessibility, include and measure indicators that assess the distance/time it takes to access a product or service. AFFORD needs a rural access strategy on how to reach rural dwellers and increase uptake. 2. In addition to the current distribution mechanism, there is the need for AFFORD to identify and grow the number of non-traditional outlets (NTOs) within hotspots and encourage them to stock condoms and monitor nighttime accessibility at hotspots. 3. There is a need for better understanding and use of available data, updated by regularly seeking consumer feedback and documenting lessons learned. 4. Performance indicators for senior management should be created that address incentives for sales and for maintaining quality in Programs and Services. 5. Having only two regional pharmacies may not add value. UHMG needs to rethink this approach. Either well-supported pharmacies are opened as part of an overall regionalization drive of the organization, or close the two offices, which at the moment are mere outposts, starved of support, from the head office. AFFORD’s Relevant and Effective Social Marketing Campaigns: 1. AFFORD needs to design specific interventions that address unmet need for family planning in different parts of the country to reflect regional variations in unmet need. 2. Records on reasons for discontinuation should be kept at each clinic, presented to UHMG through regular reports and used to design appropriate strategies. 3. Counseling needs to focus on a balanced presentation of the advantages and disadvantages of available methods and women’s contraceptive needs. AFFORDS contribution to delivery of HIV services: 1. Design theory-driven interventions based on lessons learned and global best practices. Otherwise, the village model may be redesigned as a structural intervention approach. (Lancet 2008). 2. Suspend further accreditation of new GLCs until quality is improved in the existing GLCs. 3. Develop an operations manual to incorporate all issues related to recruitment, accreditation, quality, supervision, compliance, brand loyalty, and all key aspects of social franchising. 4. Hire a dedicated GLC medical advisor on staff who has had significant experience with family planning and MNCH clinic-based procedures. 5. Improve data quality, particularly GLC service statistics and measurement of ‘reach’ at the community level. 6. Improve target setting. 7. Establish youth-friendly GLCs and/or make existing ones more youth-friendly. UHMG’s Capacity Strengthening by AFFORD 1. Support the creation of a system to collect, compile, synthesize, and present usable dashboard evidence for effective decision-making. 2. Continue the successful tightening of financial controls and oversight of all financial operations, and the strong support in monitoring the finances of subgrantees. 3. Continue to provide stronger oversight of the PF to ensure that stock and debtor management are brought under control. 29 4. Establish mechanisms to ensure continuity of both core management staff and key technical staff. Mechanisms to retain competent staff must be built into the funding arrangements, until self-sustainability is attained. 5. Ensure that the organization converts to the new appraisal system developed by KPMG and UHMG and that all supervisors are trained to develop the appropriate KRA and KPIs with all staff. 6. Contract an experienced organization development group to conduct a series of participatory Board workshops focused on: redefining the Management-Board relationship and their respective roles responsibilities, understanding the role of governance in the nonprofit context, seeking an appropriate emeritus (non-decisionmaking) role for Founders, reviewing appropriate board composition in order to maintain alignment with Mission and Vision, and reviewing best practices in board governance. Then revise the current Board manual to incorporate the changes determined through participatory process and diversify the Board to balance private sector and social marketing. Reconsider the need for a Founder’s Forum – or any governing body to oversee the Board of Directors - as it imposes yet another decision-making layer and further confuses already conflicting lines of authority between Board and Management. For USAID 1. Given the concentration of franchises in urban areas USAID may consider funding social franchises towards specific diseases or/and in only specific areas (rural or poor urban). 2. There is the need to investigate whether social franchising does not ‘crowd out’ other private health providers in the community by merely drawing clients from non-franchised clinics to franchised ones without expanding the market. 3. The PMP is generally an unwieldy document for decision-making. A dashboard of the trends of PMP indicators could be summarized and presented to management and donors to track progress. LESSONS LEARNED 1. Nascent, indigenous organizations should first focus on the areas of competence that led to their original vision and mission. This should be accompanied by adequate support at the outset, from an experienced Implementing Partner or combination of IPs, or local groups, in the broad range of participatory capacity development strategies that will simultaneously strengthen governance, finance, human resources and other administrative areas, as well as the areas required to build and sustain technical competence. Organizations and personnel selected should have a creative, problem-solving, entrepreneurial mindset. A guideline for forming a social marketing operation is included in Annex 10. 2. Global best practices must be the foundation of the operation in both private and social sector undertakings. Operations Research must be used to learn from and upgrade strategies continually to complete the cycle of observation, planning, implementation, evaluation, and re￾visiting the effectiveness of incorporated strategies on an annual basis. It was absent in this case. 3. Mechanisms must be established to ensure continuity of both core management staff and key technical staff from the inception of the organization. This may mean, for example, setting up some form of transitional funding so that expertise gained is not lost at the end of a contract. Mechanisms to retain competent staff must be built into the funding arrangements, until self￾sustainability is attained. 30 4. International NGOs often have a pool of technical experts to draw on from headquarters and regional offices when a specific expertise is required (e.g., to draft a proposal to compete for a bid). UHMG has no recourse to such facilities, and may not be competitive in responding to bids at short notice. They should be encouraged, and fully funded from the outset, to develop a database of local experts who can provide technical expertise in certain key areas such as data analysis, designing theory-based interventions, etc. 5. In the future, when determining the appropriate technical support consortium, for an indigenous organization of this nature, the following should be sought: demonstrated core experience, on staff, in social marketing, health service delivery programming, monitoring and evaluation, community level behavior change, and organizational capacity strengthening. The ACE Project, a capacity building project in Uganda for organizations working on HIV/AIDS, provides a model applicable to UHMG. While JHU may have the expertise, the team is concerned about whether UHMG received the appropriate levels of support in all needed areas from AFFORD particularly in technical capacity building. USAID should explore to what extent budgetary constraints or lack of FTEs for JHU AFFORD staff may have contributed to this situation. A Annex 1 Statement of Work USAID/UGANDA STATEMENT OF WORK EVALUATION OF AFFORDII PROJECT A. Background Implemented by Uganda Health Marketing Group (UHMG), the AFFORD I project (2005-2010) was a key actor in contributing to USAID’s strategy to build lasting capacity in the private health sector. USAID strategy recognizes that the country’s development and health goals cannot be met unless the private-for-profit and private-not-for-profit institutions are strengthened and work effectively alongside the public sector to tackle the priorities of high fertility, increasing HIV incidence, and high child mortality from malaria and other preventable diseases, such as diarrhea. In 2010, USAID/Uganda awarded an$ 18.5 million follow-on Assistance agreement to Johns Hopkins University (JHU) for three years, with the aim of strengthening the institution of UHMG to become an independent, technically and financially sustainable social marketing organization. The follow-on assistance built upon the achievements of the original five-year AFFORD project which developed and implemented a range of successful communication programs, products and services and established a local entity, UHMG, to sustain these activities. This new agreement was to provide the necessary technical resources and time to transition UHMG from an organization primarily dependent upon USAID funding to a private sector institution that is self-financing, self-governing and meeting its goal of contributing to improved health status of Ugandans through social marketing. Program objectives: 1. Further develop UHMG systems and capacity to support technical, financial, institutional, and market sustainability 2. Increase accessibility and affordability of HIVIAIDS, reproductive health, child survival and malaria products and services for Ugandans using innovative private sector approaches 3. Increase knowledge and promote healthy behaviors and lifestyles among the general population and among targeted hard-to-reach and high-risk populations B. Purpose The primary purpose of the evaluation is to determine whether UHMG’s capacity as an indigenous organization promoting social marketing has been built and how it has played a key role in improving the uptake of healthy behaviors and increasing demand for delivery of health and HIV/AIDS services in Uganda. The evaluation should provide evidence on how social marketing approaches can be applied to expand the delivery and use of quality and affordable health services in Uganda. This evaluation will inform ongoing and future social marketing activities by UHMG and other social marketing organizations in Uganda. For USAID Uganda, an evaluation of support to UHMG will provide initial lessons on strengthening local capacity as desired under the 2010 USAID Forward reforms. Therefore, this evaluation will mainly focus on program and capacity building areas. B C. Key Evaluation Questions The four questions to be addressed by the evaluation are: 1. Did AFFORD increase accessibility and affordability of HIVIAIDS, reproductive health, child survival and malaria products and services for Ugandans? 2. How relevant and effective have AFFORD’s social marketing campaigns been in increasing awareness and uptake of health services? 3. What specifically is AFFORD’s contribution to the delivery of health and HIV/AIDS services in Uganda? 4. To what extent did AFFORD strengthen UHMG systems and capacity for technical, financial, institutional, and market sustainability? The following sub-questions may prove useful in helping to answer each of these evaluation questions respectively: 1. Did AFFORD increase accessibility and affordability of HIVIAIDS, reproductive health, child survival and malaria products and services for Ugandans? a) What is the difference in access and affordability before AFFORD and now? b) Was social marketing more successful in some situations or locations or among some population groups than others? Which ones c) Which strategies were most successful? Which ones were least successful? d) What are the major factors associated with success or lack thereof? 2. How relevant and effective have AFFORD’s social marketing approaches been in increasing awareness and uptake of health services? a) Were AFFORD’s social marketing approaches more successful in achieving this result in some situations or locations or among some population groups than others? Where were their highest sales, and among who? Compare that with national disease burden/need. For example, were contraceptives successfully promoted more in regions with higher TFR, or condoms marketed among high risk groups? b) Among which hard-to-reach and high-risk populations was AFFORD most successful in increasing uptake? c) Did UHMG have potentials, strengths and comparative advantages that could have been exploited better to achieve this result? 3. What specifically is AFFORD’s contribution to the delivery of health and HIV/AIDS services in Uganda? a) How does AFFORD rank among condom and contraceptives distribution/sales agencies in Uganda? b) What are the distinctive successes of UHMG’s social marketing portfolio? How well has AFFORD successfully marketed UHMG own brands in addition to products promoted by USAID like Protector, Pill Plan and Injector Plan? c) Can ‘Good Life Clinic’ approach be classified as true social marketing? d) What are the missed opportunities/untapped potentials, if any? 4. To what extent did AFFORD strengthen UHMG systems and capacity for technical, financial, institutional, and market sustainability? C a) Is UHMG a stronger, more competitive and more sustainable organization now than it was three years ago? b) What governance structures are in place to manage UHMG and how do they operate? Who owns UHMG and who oversees activities of the board? c) To what extent has UHMG successfully transitioned into a private sector institution that is self-financing and self-governing? Analyze UHMG business plan. d) Does the business arm (e.g. Product facility, Warehousing) operate under common private sector and market principles? How are their customers? Do they have a significant group of customers outside of the donor funded organizations? Is UHMG competitive? e) What stage of organizational development would you place UHMG? f) What are the main lessons for USAID in building and nurturing capacity of Ugandan indigenous organizations? D. Methodology The evaluation will utilize a methodology that includes the use of qualitative and quantitative data collection methods, to answer the key evaluation questions. USAID/Uganda strongly encourages the use of a design matrix such as the one provided in Annex B. The evaluation team will be required to use an n established methodology to assess the capacity of UHMG. The team may adopt or adapt existing organizational capacity assessment tools as appropriate. At a minimum, USAID will require a thorough assessment of UHMG’s technical, organizational, adaptive and influencing capacity. Involvement and participation of key stakeholders has a potential in increasing use of evaluation results, and increased ownership of recommendations resulting from the evaluation; involvement of UHMG structures and AFFORD networks, facilities and clients is critical. USAID/Uganda HHE and PPD Offices, Uganda AIDS Commission, DFID, Ministry of Health and District Health staff are examples of key stakeholders. The team is encouraged to employ as many of the following data collection methods as they deem applicable: (1) Review of relevant documents that provide background information that is useful for assessing Social Marketing in Uganda, AFFORD and UHMG; (2) Review of Project Monitoring Data; (3) In-Depth Interviews; and (4) Focus Group Discussions 1. Project Monitoring Data The analysis of project monitoring data reported for 2010 to 2012, for indicators in the PMP, will be reviewed and analyzed by the evaluation team. Output and outcome indicators will allow the team to assess uptake and coverage of services and progress towards the achievement of agreed upon targets over time. Impact indicators will be used to assess UHMG’s and AFFORD’s contribution to long-term effects. Data for the indicators in the PMP is readily available from UHMG and AFFORD M&E Systems; the team will identify other specific reports that UHMG and AFFORD can provide for analysis. b) Review of other literature: mention other relevant literature that needs to be reviewed to understand better project performance and the context within which the project is operating. This includes the RFP, project work plans and reports, relevant activity reports, the 2012 BCC Evaluation, health sector strategic plan and annual sector performance reports, COPs, AFFORD I End of project evaluation report, etc. D 2. In-Depth Interviews The evaluation team will conduct qualitative in-depth interviews with key stakeholders and partners. The team will develop a structured interview guide that will be used to guide the interview process. The interviews should be loosely structured, but following the list of questions in the guide. A list of potential respondents will be developed by the evaluation team in collaboration with UHMG and AFFORD and eventually presented for approval to USAID/Uganda. Among the respondents that will participate in the in-depth interviews are: (1) selected members of district health teams; (2) selected personnel of health facilities supported by UHMG and AFFORD; (3) selected UHMG staff and project staff of AFFORD; (4) selected staff of USAID/Uganda; (5) Senior staff of UHMG and AFFORD; (6) Officials at the MoH and Uganda AIDS Commission; and (7) selected UHMG and AFFORD program clients/beneficiaries (8) UHMG founder members ; (9) UHMG Board members. The interview guides will be drafted by the evaluation team, before meeting the USAID team. This will give the USAID/Uganda team opportunity to review and make input to the interview guide, making sure it captures the questions of interest, as well as questions in which the implementing partners might be interested for learning purposes. 3. Focus Group Discussions (FGD) Focus group discussions will be conducted with respondents selected from the project intervention communities to solicit their views on the extent to which the project is contributing to access and quality of services that they provide or receive. Among respondents that will be included in a focus group discussion, are UHMG and AFFORD beneficiaries and target groups (including Most at Risk Populations), health workers; and other relevant groups. Each focus group will be composed of 8 to 12 respondents. The focus group discussions will be conducted using a structured discussion guide with open ended questions, and opportunities for probing and exploring issues as they arise during the process. The discussion guides will be drafted by the evaluation team, and the USAID/Uganda team will have opportunities to make input, through the inclusion of questions they would like answered by respondents. Other qualitative tools like case studies and most significant change especially at the UHMG level to document better what are some of the fundamental changes that they see in their capacity, what they are able to do now than before and how the capacity building efforts contributed to that. The team is encouraged to use an evaluation design matrix such as the one provided in identifying where and how they will apply each of the methodologies. E. Evaluation Team The evaluation team of four consultants will consist of three international consultants including a team leader who will also be the evaluation expert in the team, a social marketing expert and an organizational development expert; and a local consultant. Between them the team will have skills and experience in:  Evaluation of health (including HIV/AIDS) interventions  Health and HIV/AIDS programing  Organizational Development and Organizational Capacity Assessment  Social Marketing E  Business Approach to service delivery (understanding the role of public and private sectors in delivering services in an increasingly competitive market) The Team Leader will be responsible for managing the team in conducting the assessment and in preparing and finalizing all deliverables. This individual will be responsible for achieving assignment objectives as well as briefings and presentations, and will be the key liaison with USAID/Uganda. The Team Leader needs to be an innovative thinker with a strong understanding of USAID systems and procedures. Other consultants will bring specific and complementary expertise and submit to the leadership of the team leader in performing assignments in evaluation design, data collection, data analysis, report writing, briefings and presentations. 1. Team Leader The Team Leader and evaluation expert will be the holder of an advanced degree with a minimum of fifteen years of experience working with, assessing and/or evaluating organizations that manage health and HIV service delivery. S/he will be knowledgeable in evaluation methodologies and be skilled in using and analyzing data from rapid appraisal methods. The Team Leader will have played significant roles in designing and implementing a minimum of ten health program evaluations, three of which must have included use of a social marketing approach to service delivery. This individual will be a good writer and an innovative thinker with a strong understanding of USAID systems, procedures and information needs. 2. Social Marketing/Private Sector Expert The Social Marketing Expert will have a Master’s degree in a field relevant to delivery of health services using private sector approaches, such as MPH or health economics. In addition, s/he will have over ten years’ experience in management and/or assessment of social marketing approach to service delivery, and good knowledge of private sector operations. Must demonstrate mastery in analyzing performance of social marketing of health commodities and/or services 3. Organizational Development Expert The Organizational Development Expert will have good knowledge of Organization Development and extensive experience in assessing organizational capacity. Knowledge and experience in delivery of social services or public health practices as well as in applied program monitoring and evaluation is desired. Experience in health sector in Africa is preferable; s/he will provide advice on assessment/evaluation methodology of a local organization. 4. Local Consultant The Local consultant in the team will be a holder of a Master’s degree related to public health, have advanced knowledgeable in public and private sector delivery of health services in Uganda and the health systems and policies at national and decentralized levels. As the expert in local behavior and factors affecting adoption of health practices, knowledge of the culture, norms, decision making and health spending patterns of Ugandans is required. F. Duration The task is estimated to begin towards the end of July 2013 and be completed in not more than three months of commencement. F G. Deliverables 1. In Briefing: Introduction of the evaluation team, discussion of the SOW and initial presentation of the proposed evaluation work plan. 2. An Inception report detailing the Team’s interpretations of the assignment, a final evaluation design and methodology, sampling, analytical plans tools and work schedule. 3. Weekly Progress Reports: Brief informal reports summarizing progress, challenges and constraints and describing evaluation team’s response 4. Oral Presentation: Power Point presentation (including handouts). It should cover, at a minimum, the major findings, conclusions, recommendations, and key lessons. The evaluation team will liaise with the mission to agree on the dates, audience, venue and other logistical arrangements for briefings. 5. Draft Evaluation Report: The report must include an introduction, background of the project, main evaluation questions, evaluation design, data collection methods, findings, conclusions, recommendations and lessons learned). The report should comply with the USAID’s Evaluation Report standards set out in Annex 2. Other guidance on how to write evaluation reports can be found at the following link http://transition.usaid.gov/evaluation/HowtoNote￾PreparingEvaluationReports.pdf 6. Final Draft Report: Complete report incorporating comments from USAID and other stakeholders. The final report is expected to be 20-30 pages in length, excluding annexes. 7. Final Report: The contractor will submit a final report incorporating final edits for wider sharing. The final report must be approved by USAID prior to wider distribution. The final report must include the same minimum content as the Final Draft Report, as well as an executive summary with the final report. Once approved by USAID, the final report must be submitted to the USAID’s Development Experience Clearing House (DEC) within three months of USAID approval.*All reports should be provided in four (4) hard copies and one (1) electronic copy. 8. Cleaned labeled and ready to use electronic copies of datasets: The Evaluation Contractor will also submit on a compact disk (CD) all data records from the evaluation, i.e. full data sets from the quantitative data; in-depth interview and focus group discussions transcripts; and other documented survey responses. H. Consultant Selection Criteria 1. Education, skills and competencies 2. Experience and Past Performance G Annex 2 Resources List of documents reviewed No Document reviewed 1 A Report of Qualitative Research in Kampala, Mpigi, Mbale, Gulu and Mbarara Districts – Uganda May, 2011 2 AFFORD End Of Project survey Report-REEV CONSULT INTERNATIONAL (un-dated) 3 AFFORD II Capacity Building Plan August 2011 4 AFFORD Performance Monitoring Plan , April 1, 2011 – September 30, 2013 5 AFFORD/UHMG, Achieving and documenting impact at scale ( 2013) Steven Lake 6 CARISMA, Reaching Most at Risk Populations through Social Marketing: Caribbean Experiences,2011 7 Case study sustainability Continuum for social marketing UHMG - August 13 8 The Deliver Project, Uganda: Mapping the distribution of commercial goods to the last mile, February 2009 9 Distribution of UHMG products as per AIS survey regions August 2003, Research World International 10 EMTCT Commission plan to support the roll out of option B+ for EMTCT in Uganda, Dec 2012 11 Field Assessment Report for NAYODE and New EDEN, October 2013 12 Final HBCS Report, 2009_Final 13 KPMG Final Report on the Human Resource Capacity Assessment 14 Management and Organizational Sustainability tool ( MOST) Management Science for Health (MSH) 15 Memorandum and Articles of Association 13 October 2006 16 O’Sullivan, G., C. Cisek, J. Barnes, and S. Netzer. May 2007. Moving Toward Sustainability: Transition Strategies for Social Marketing Programs. Bethesda, MD: Private Sector Partnerships- One project, Abt, Associates Inc 17 Organizational Capacity Assessment Tool ( OCAT) PACT, 2002 18 Quarterly Progress Report – FY2013 Quarter 2 19 Rapid Situational analysis of HIV/AIDS among the long distance truck drivers and their sexual networks along the major transport routes in Western Nile Region- Final Survey Report- May 2011 20 Report on technical assistance provided to Uganda Health Marketing Group KPMG, July 2013 21 Report On Situational analysis of HIV/STI among Border Riders in Arua and Jinja District 22 Republic of Uganda (2009) HIV prevention and models transmission analysis 23 Summary GLC mentorship reports 24 Technical Assistance to UHMG, KPMG, October 2012 and July 2013 25 The AFFORD Health Marketing Initiative in Uganda- Mid-term Evaluation Final Report by KsanderRohozynsky, Christine Billingway and AnneteBongovanni 26 Training Peer Educators in Fishing Communities on HIV prevention : Participant's Handbook 27 Uganda AIDS Indicator Survey (AIS) 2011 28 Uganda Behavior Change Communication Survey Preliminary Campaign Results, UHMG June 2013 29 Uganda Demographic Survey 2011-Uganda Bureau of Statistics Kampala, Uganda August 2012 30 Uganda HIV Prevention Response and Modes of Transmission Analysis Final Report, March 2009 31 UHMG Board Manual, June 2013 H 32 UHMG Campaign Ready Analysis Financial Year 2012 Q 3 32 UHMG Capacity Building Plan October 2010 –September 2013, August 2011 33 UHMG Exposure to UHMG Campaigns Survey: 2013 Behavioral change Survey 34 UHMG Family Planning campaign Strategy (Genext Small Family Campaign) 35 UHMG Grant Award SUB Agreements with NAYODE, SSECODA and HCU 36 UHMG Product Prices over time ,UHMG, 2013 37 UHMG Management Letter, Year Ended 30 September 2012, 26 April 2013 38 UHMG Case study: Sustainability continuum for social marketing, August 2013 39 UHMG Willingness to pay for UHMG Brands: 2013 Behavioral change survey 42 UHMG Service Statistics 2011-2013 43 UHMG QI Summary Report, March 19, 2013 44 UHMG GLC Network Strategy for 2013-2015 45 UHMG Fishermen Training Manual – Participant Handbook 46 UHMG Model Village Plan, undated 40 UHMG HIV Prevention Project for Boda-Boda cyclists (Youth out of school) 41 USAID End of Project Review of “Social Marketing and Targeted Communications for HIV and AIDS Prevention Among Most-At-Risk Populations in Burma, China,Lao and Thailand 2007-2010 42 USAID-funded Strengthening HIV Prevention Services for Most-At-Risk Populations (SHiPS for MARPs) I Annex 3 Persons Met Name Designation /Organization Alex Mugasa UHMG Regional Pharmacy Manager-Eastern Region Alexander Mugume Deputy Chief of Party Anena Pauline Accounts Assistant, Health Alert Aryan Evelyn Registered Mid\wife, Iceme Health Center (UHMG -GLC) Oyam District Beatrice Kakiiza Malaria Program officer, UHMG Charmaine Matovu M&E Specialist, USAID Uganda Dan Wamanya Program Management Specialist, USAID Uganda Dr. Esther Kaggwa Research M&E Advisor, JHUCCP AFFORD Dr. Zainabu Akol Commissioner, Reproductive Health MOH Dr EspilidonTumukurate Director Program and Services, UHMG Dr KihumuroApuuli Director General, Uganda AIDS Commission Dr Myers Lugemwa Deputy Program Manager Malaria, Ministry of Health Dr. Laverte Wafula In Charge, Health Centre Mbale (UHMG GLC) Ekachelan Esau Chief Administrative Officer, Kalangala District Elizabeth Ikoju HR and Administration Manager. UHMG Emily Katarikawe Managing Director, UHMG Erasmus Tanga Chief Of Party, SPEAR Project Evelyn Babumba Mwasa Director Marketing & Strategic Information, UHMG Florence Ajok Odoch HIV/AIDS Specialist, STAR EC Project Francis Obutu Director, Health Alert - Gulu Francis Nsanga Knowledge Manager, UHMG Frederick Kawuma Founder Member UHMG George Inholo Country Manager, Unilever Uganda Limited Grace Namata Sagi Deputy Program Manager, Department of International Development Hannington Syahuka Director Resource Mobilization, UHMG Henry Semwanga Lule Deputy Executive Director, PACE Jackie Calnan Programme Management Specialist, USAID Uganda James Duworko Family Planning/Reproductive Health Advisor, USAID Uganda Janex Kabarangira Programme Management Specialist/Deputy Health Team Leader, USAID Uganda James Kato Coordinator, Humanitarian Care Uganda Joachim Kabaisera Finance Manager, UHMG John Bikala Wanzala Senior Internal Auditor, UHMG Joseph Mwoga Chairman UHMG, Board of Directors J Name Designation /Organization Joseph Mwangi Strategic Information Advisor, USAID Uganda Joshua Tusingwiire Brands Officer, Good Life Clinics Joyce N. Tamale Director Finance and Administration, UHMG Juliet Afica HIV/AIDS consultant, UHMG Julius Lukwago Director Sales and Marketing, PACE Richard Kitonsa Pharmacist Paul Lemi Brands officer, UHMG Moses Kafeero Finance Manager, Product Facility Christine Namulalo Pharmacy Assistant, UHMG Pharmacy Mbale Nankunda Babihuga Allen Executive Director, Communication for Development Foundation Uganda/UHMG Founder Member Olive Birungi Lumonya UHMG Board Member Onenchan Walter Administrative Assistant, Fitzan Medical and Dental Clinic Paul Kagumire Sales and Distribution Manager Phillip Okello Apira Director, Product Facility Rachel Apio Logistics & Procurement Manager UHMG Rebecca Nsamba Finance Manager (UHMG- HQ) Rhobbinah Sempebwa USAID Uganda, Agreements Officer Representative (AOR)/AFFORD Project Richard Sewajje Director Contracts/Grants STRIDES Project Richard Guma Finance Manager, AFFORD/JHU Robert Busuulwa UHMG Board Member Robinah Nganwa Lukwago Health Advisor, Department of International Development Ron Hess Resident Management Advisor ( RMA) AFFORD Samuel Zirimenya Monitoring and Evaluation Manager, UHMG Seyoum Dejene Clinical Care Specialist/USAID Sheila Kyobutungi Program Management Specialist/USAID Sister Babirye Mary Nursing officer/ In charge, Iceme Health Center Sister Karen Inyangati Clinical Officer, Sarem Hospital Mbale Valerie Komwaka Mitala Director, Human Resource, Procurement, and Governance UHMG Walter December Coordinator, Health Alert K Annex 4 Interview guides Interview and Focus Group Discussion Guides 1. Key stakeholder Interview Guide (NGO and Government Depts/Ministries) As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As key stakeholders, you play an important role in bringing products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you want to be part of this evaluation by answering a few questions. Let me also know if you have any questions before we start. 1. What AFFORD strategies, interventions and tools were used (facility assessment tool, curricula, etc.)? Please list. (Probe; Good Life Clinic approach) 2. Which of these strategies, interventions and tools would you consider to be key program elements? Please explain. 3. To what extent did participation in the other stakeholders like USAID, JHU etc. partnership advance or hinder project implementation? Please explain. 4. What worked well? Please elaborate. 5. What would you want done differently next time? Please explain why. 6. What strategies, interventions, tools, etc. would you recommend be sustained and/or scaled up? Please provide a justification for your response. 7. Do you think there has been a change at the national (regional/district) level in the past two years in any of the following? a. Condom availability/ affordability b. Health communication c. Other health products and services in both urban and rural areas d. Health behavior change (e.g.) e. Bed net use, f. Condom use g. HIV testing h. Contraceptive use i. ANC attendance j. Sales and distribution 9. Has AFFORD contributed in some way to any of these (repeat all the above areas)? 10. On the whole how can you comment on the performance of UHMG in Uganda? Are there any successes at the national level you can remember? L 11. What AFFORD strategies, interventions, tools should be discontinued? Why? 12. What were some barriers, if any, that you encountered? -Staff turnover? Lack of key support? Lack of technical assistance? 13. How did you overcome the barrier(s)? 14. In your opinion, has there been a change in marketing or volume of sales/distribution of the healthcare products in Uganda over the past two years? If so, what products? What specifically is AFFORD’s contribution to the delivery of health and HIV/AIDS services in Uganda?” 15. What are some of the other social marketing organisations in Uganda? a. In your own assessment regarding their contribution to health improvement in Uganda, how will you compare UHMG with others? b. What are some of the reasons for your answer? 16. What recommendations do you have for future efforts such as these? 17. Is there anything more you would like to add? Thank you - - M 2. Interview Guide: Founders Introduction As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As Founder members, you play an important role at UHMG. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you have any questions before we start. Please tell me little about how you became a Founder member. Tell me about how you assess the performance of UHMG? What are the signals that alert you to a key issue to be addressed? If they are not moving in the direction that the Founders feel is appropriate, based on the current strategy how is that handled? How long have you served? When does your term as a Founder member end? Do you have a manual or do you generally use the Board manual as a reference for the Founders Forum’s work? Please describe for me your role as a Founder member. What are your key responsibilities? Please explain the Founder’s Forum’s relationship to the Board of Directors. What decisions are made by the Founders? What decisions are left to the Board of Directors? Does one group report to the other (how does that work)? Please explain the Founder’s relationship, if any, with the Managing Director (MD) and Senior Staff. Do the Founders have a role in UHMG’s daily operations? If so, please explain, and provide examples. Is UHMG legally registered as a business entity or an NGO? Who owns UHMG? What role, if any, do the Founders play in resource development for UHMG? Tell me a little about general Founder operations. For example, how frequently does it meet, and what constitutes a quorum? How are Founder members recruited or appointed? What criteria are used for nominating Founder members, selecting them, and approving them? How are conflicts of interest handled? Are any Founders or Board members currently doing business with UHMG (earning money for products or services provided to UHMG)? What about family members of Founders or the Board? - - N What kind of expertise is on the Founders Forum? ? (Seek, in particular, to find out if there are legal and accounting/finance experts.) What areas of expertise, if any, do you feel should be on the Founders Forum, but are not currently present on the Founders. What kind of expertise if on the Board? (Seek, in particular, to find out if there are legal and accounting/finance experts.) What areas of expertise, if any, do you feel should be on the Board, but are not currently present on the Board Have there ever been any complaints or lawsuits filed against UHMG for any reason (e.g., by an employee/former employee, on a contract related issue with a business contractor or vendor).? Have there ever been any problems of these natures brought to the Founders’ attention (even a grievance filed by an employee), even if they did not require legal action? What was the situation and how did the Founders handle it? Tell me more about the rationale for a Founder members group and a separate Board of Directors? On what areas, if any, do you interact directly with the staff? How often are you at the UHMG office outside of Board meetings? Are you, or is anyone on the Board playing a specific management role? Do the Founders have a self-evaluation process? If so, what tool has been used to guide the evaluation? How frequently has this been done? What were some of the lessons learned? Who supervises the Board Chairperson? Is there anything else you would like to tell us about that you think will be useful to the work of the evaluation team? Thank you very much for your time. - - O 3. Interview Guide: Board of Directors Introduction As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As Board members, you play an important role in bringing products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you have any questions before we start. Please tell me a little about how you became a Board membe, and the skills you bring to the Board. How long have you served on the Board? When does your term on the Board end? Do you have a copy of the Board manual? Please describe for me your role as a Board member. What are your key responsibilities? What decisions is the Board responsible for? Please explain the Board’s relationship with the Founder’s Forum. Do they make decisions or are they purely a consultative body? Do they supervise the Board? Please explain the Board’s relationship with the Managing Director (MD) and Senior Staff. What is the Board’s role in their supervision? Does one Board member supervise the MD, or does the whole Board get involved with supervising the MD? In staff recruitment, which staff require Board approval? Which staff can be hired directly by the MD? Does the Board have a role in UHMG’s daily operations? Please explain, and provide examples. How are Board decisions made? On what areas of UHMG’s operations do Board decisions focus? Is UHMG legally registered as a business entity, as an NGO, or both? If both, what is the relationship between the two entities? Are any Board members (or Founders) currently doing business with the Board (earning money for products or services provided to UHMG)? What about family members of Board or Founders? Who owns UHMG? What role does, if any, does the Board play in resource development? Tell me a little about general Board operations. For example, how frequently does it meet, and what constitutes a quorum? How are Board members recruited? What criteria are used for nominating Board members, selecting them, and approving them? How are conflicts of interest handled? - - P What kind of expertise is on the Board? (Seek, in particular, to find out if there are legal and accounting/finance experts.) What areas of expertise, if any, do you feel should be on the Board, but are not currently present on the Board Have there ever been any complaints or lawsuits filed against UHMG for any reason (e.g., by an employee/former employee, on a contract related issue with a business contractor or vendor).? Have there ever been any problems of these natures brought to the Board’s attention (even a grievance filed by an employee), even if they did not require legal action? What was the situation and how did the Board handle it? What committees does the Board have? Do non-Board members serve on any of these committees? Are these outside people staff of UHMG or outside experts? What role do they play in Committee decisions? On what areas, if any, do you interact directly with the staff? How often are you at the UHMG office outside of Board meetings? Are you, or is anyone on the Board playing a specific management role? Tell me about the Board’s self-evaluation? Has the Board ever conducted a self-evaluation using the tool in the Board manual? How frequently has the Board conducted a self-evaluation? What were some of the lessons learned? Who supervises the Board Chairperson? Is there anything else you would like to tell us about that you thing will be useful to the work of the evaluation team? Thank you very much for your time. - - Q 4. Interview Guide: Management practices Introduction As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As Senior Management, you play an important role making UHMG a thriving and sustainable organization. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you have any questions before we start. Organizational structure responsibilities of all individuals? Planning ng short and long-term work and implementation plans? for achieving the mission of the organization? Program development responsible for program development? Administrative procedures -office communications? ative manuals reviewed and updated? Risk management Has the UHMG taken any steps to protect itself against staff abuse of resources? Information systems the responsibilities of the person/s? collection, analysis and dissemination of information organized in the UHMG? - - R How does the UHMG use the information generated by the monitoring, evaluation and reporting system? Program reporting ts program activities? program activities? - - S 5. Interview Guide: Finance Introduction As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As staff members, you play an important role in the work of UHMG to bring quality and affordable products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you have any questions before we start. Accounting chart of accounts, approval authority for financial transactions, and guidelines for controlling expenditures? Can you show us a copy? Internal Controls ey been conducted? Budgeting the annual operating plan? timely manner? penditure of funds in excess of approved, budgeted amounts? Financial reporting r the insufficiency or tardiness of financial reports? outcome? Financial sustainability - - T What is the current status of resource development plans for needed financial resources using the key markers in the business plan? registration of the organization, e.g., non profit and limited guarantee company? funding, and the tax status challenges? Is there anything else you would like to tell us about that you feel would be useful to the evaluation team’s work? Thank you for your time. - - U 6. Interview Guide: Human Resources Introduction As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As employees, you play an important role in the management of staffing, quality recruitment and staff development at UHMG. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you have any questions before we start. Please tell me, in general, the areas that the HR and Administration division handles. How many staff are in HR, how many in administration? What does administration cover? Tell me about the development of the HR handbook. Who prepared it and how is it used? Do all staff have a copy? If so, any feedback on its usefulness? Is the online version final or under revision? What are some of the changes being explored, if undergoing revision? I understand that UHMG contracted KPMG to conduct an HR capacity assessment. Can you tell me about that process? (what was KPMG’s role, what tools were used) Are those results available? lessons learned in its work? Let’s talk about the performance management process. Please tell me more about the systems that have been instituted. e copies of the new job description formats and sample descriptions? How are the Key result areas (KRAs) and Key Performance Indicators (KPIs) assessed? [Note: some kind of scoring or what? –perhaps evident on the form] ning? management process? What can you tell me about the rate of turnover in the past few years at UHMG? Is there any documentation on this? applied to future recruitments? Can you describe how the recruitment process is handled (including how you work with the hiring manager) (e.g., posting, managing candidate selection, interviewing, approval, and onboarding? he candidate? Any data you can share on this? I understand that UHMG has grown in size from 60 to 90 staff members (stated I the current work plan). - - V What are the current projections on the absorptive capacity of UHMG to take on the other AFFORD-paid staff after the project ends? How is the termination process handled? Can you describe a situation and tell me what the process was? Has there ever been a case of wrongful termination brought to the legal authorities against UHMG? If so, what happened, and how was that handled? Is there anything else you would like to tell us about that you feel would be useful to the evaluation team? Thank you for your time. - - W 7. Interview Guide: Product Facility Introduction As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As staff members, you play an important role in the work of UHMG to bring quality and affordable products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you have any questions before we start. 1. How do you do your distribution? How do you select your regional distributors? 2. What is the linkage between the 2 business units: PF and Health Consulting? What is linkage between both business units and UHMG. 3. What factors go into setting UHMG’s prices? Are prices regulated wherever products are distributed? Does UHMC get any percentage from franchisee or vice versa? 4. Product Facilities; TWO objectives a. increase access to quality affordable products b. financial sustainability 5. They do two types of distribution: 1) social distribution; 2) commercial distribution a. What are the differences between these two? 6. How do you measure access to products? a. What is level of access to MARPs? 7. 15 regional distributors: how are they appointed? a. Is there a list of these distributors? What are the criteria for selecting a distributor? b. Who are the distributors employed by, and what is their commission structure? c. What are the terms of the relationship (under what circumstances can they be dropped?) d. Are these distributors covering the most at risk communities where HIV is high prevalence (they say they have data or they can analyze available data to do this) e. Are they reaching the most vulnerable groups (MARPS)? (Concern they are not reaching sufficient MARPS.) 8. There is data on distribution but these are not indicators of user access; how does UHMG get data on user access, including data on the access of MARPS to their products 9. They say their surveys include questions about brand use; can we see that data? 10. How do you set prices of your products? How has price increase over the years? Have you ever done Willingness to Pay Study in UHMG? How do measure ‘Affordability’? How do you whether your products are reaching the poor? 11. Why does your warehousing and haulage services seem to be so popular among MOH, and donor community. - - X 12. Do they charge MOH for warehousing or distribution? If no why (why would a profit-making organization not charge anything?) MOH has recommended UHMG on its website, which has also promoted their popularity with the donor community. 13. They say they charge their donor clients 10% (includes insurance and professional fees) a. 10% of what? And for how long? 14. Do they also have other private sector customers and what are they charged? 15. Is the warehouse insured? What does the insurance cover? 16. Cost recoverable products - how well they are doing? How many do they have? 17. What are the existing sources of the organization’s financial resources? 18. What strategies does the UHMG have to diversify its income streams? 19. Tell us a little more about what “insider buying” is and how it applies to the 2 business units. 20. Is there a longer-term business/funding/resource development plan for the needed financial resources? 21. What is the UHMG’s future funding strategy? 22. Does the UHMG have cost recovery/income generation plans? 23. What awareness does top management show of the sources and mechanisms available for securing funding? 24. Is there a realizable plan for long-term support of the programs? 25. Do you have plans to register as a private commercial entity? - - Y 8. Interview Guide: Subgrant Management Introduction As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As staff members, you play an important role in the work of UHMG to bring quality and affordable products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you have any questions before we start. Please tell me about the subgrant process. How are potential subgrantees identified? How are they assessed? Tell me about the proposal process. Tell me about what takes place at the post-award workshop. Who monitors the subgrantee? How is that monitoring conducted? (site visits – how frequent?; phone calls?) What alerts you to an issue or concern with a subgrantee? Can you provide an example of an issue and how it was resolved? [Note: this question should be used with both financial and technical staff.] Tell me about your working relationship with the Program staff/Financial staff (ask about the other, if with Program staff, ask about Financial staff and vice versa). What are the markers for issuing future payments? What is the burn rate for the subcontractors? Tell me about the reporting process. How frequently are they submitted? What feedback, if any is provided on progress reports? Tell us about the Four subcontractors to UHMG. How have they been executing their financial/substantive reporting responsibilities? How do they manage field workers and volunteers? What are some of the specific challenges that your subgrantees face? Is there anything else you would like to add that would help us understand your work managing subgrantees? Thank you very much for your time. - - Z 9. Interview Guide: GLC Program Managers Introduction As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As staff members, you play an important role in the work of UHMG to bring quality and affordable products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you have any questions before we start. Tell me how GLC works- a typical one. How many? How many are fully accredited and how many are in the process of accreditation. (Ask for copies of accreditation guide/standards – the process of getting accredited? One documents says 560: They 200 are fully accredited; confirm. How many levels of GLCs are in operation? How does one move up the scale? On what basis did they scale up the number of GLCs? What research or evidence was this decision based on? And have they targeted the high risk areas in scale-up? What data are available? What is the accreditation process? What is the process for preparing a GLC for accreditation – how is the accreditation process is it monitored and what TA is provided? Tell me about the main set-up costs; Equipment; Products; Training? Can you quantify in cash? Tell me the things/resources UHMG provides. Does the GLC provider contribute anything? What are the main challenges of setting up a GLC apart from cost? Do you have a memorandum of understanding or contract with all GLCs? (Ask for a copy). Who signs it on behalf of UHMG? And at the GLC level? What is the charging policy at GLCs? Is UHMG involved in setting up prices at GLCs? How are prices regulated? Does UHMC get any percentage from franchisee or vice versa? How does UHMG monitor quality of services? (NB. We have been told there have been no mystery clients or client exit surveys) How do they supervise the quality of the clinics? What instruments to they use? How frequently do they conduct site visits/monitoring visits? Do they have an Operation guiding service provision at GLCs? (Ask for a copy). Does every GLC have a copy? What is the procedure if a clinic does not meet quality standards? Any example of this since the project started? Has anybody been sanctioned? How was that handled? What services are provided at each GLC clinic level? How are they referred on to different levels? Are there linkages between these levels? Is there a Supervision checklist - - AA Who implements the motor bodaboda project and the mother’s clubs? How are these tied to the GLCs? Do they look at adverse effects on the neighboring service providers who do not become franchisees and lose business to the franchised group? (Crowding out) What is the impact on other community businesses? Data – do the GLCs collect data? How do they do that? What information is collected? Is it disaggregated by male/female/youth/marital status? To whom do they send it? How do you create demand for GLC apart from the media? How do they continually motivate (“keep the fire burning?) providers to maintain their level of interest in the network? How do GLCs counsel clients (if each package has different counseling needs, we need to know the counseling requirements – i.e., what consumers need to know about the services provided in each of those areas.) If there is feedback from GLCS concerning specific challenges to their products and services, is there a reporting system to bring this feedback to the CSOs and UHMG (e.g., they mentioned a problem with rumors affecting IUD use)? What support do they need from UHMG? What is the connection between reporting these challenges and addressing them through media, informational and BCC campaigns? Do this link into message creation? How are they trained? And retrained? - - BB 10. Interview Guide: BCC/Mass Media Message Design – National Level As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As staff members, you play an important role in the work of UHMG to bring quality and affordable products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you have any questions before we start. 1. Can you tell me about yourself and your role here in UHMG? Your training? Experience? 2. Who supervises the communication of health messages in UHMG? 3. How many mass media campaigns are currently being aired? 4. How do you test your messages? 5. Who are target/audience and how were they selected? 6. Do you undertake market segmentation? (Ask for copies of segmentation analysis) 7. How do you reach rural areas with your messages? 8. How do you track exposure? 9. Are your exposure data disaggregated by consumer characteristics? 10. What are the main campaigns/interventions you have for young persons? 11. How do you know which messages to focus on? Do you do any surveys/analyses on behavior determinants? 12. For each campaign what are the actual behaviors you want promoted? 13. How do you track behavior change (general population, women, men, MARPs, youth) 14. Is BCC message development in UHMG based on any theory or model? 15. What skills in BCC interventions are being transferred from AFFORD to UHMG? 16. What skills do you think you lack? Thank you very much for your time - - CC 11. Interview Guide: UHMG Staff Responsible for BCC at the Community Level As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As staff members, you play an important role in the work of UHMG to bring quality and affordable products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you have any questions before we start 1. What types of community level BCC interventions do you have here in UHMG and where are they implemented? (IPC? Peer education? 2. Where your MARPs interventions? Who at the HQ is in charge of MARP interventions? 3. How experienced are you in implementing MARPs interventions at the community level? 4. How experienced are your sub-grantees in community level MARPs interventions? 5. Who designs the interventions for the sub-grantees? Are there any intervention manuals you can share with us? (e.g. Peer education guide; community mobilization). 6. How are the sub-grantees supervised in the field? When was the last time someone from Kampala went on support supervisory visit? Which sub-grantee was visited? Where? 7. Does UHMG have a supervisory checklist for sub-grantee regarding BCC implementation? 8. Whose responsibility is it to create demand for GLCs at the community level? 9. What is the link if any between the sub-grantees and the GLC? 10. Tell me more about the Village Model Approach. How many of such do you have? 11. Who are the ‘community volunteers’? Who recruits them? To whom do they report? 12. How do you assess whether MARP interventions are working? What are the behaviors of interest in your MARP interventions? 13. What interventions have UHMG got for young persons in the communities? 14. Do you have FSW interventions here in Kampala? Do you work with their clients? Thank you very much for your time - - DD 12. Interview Guide: Research, Monitoring and Evaluation Staff Introduction As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As staff members, you play an important role in the work of UHMG to bring quality and affordable products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you have any questions before we start. 1. Can you tell me about yourself and your role here in UHMG? Your training? Experience? How long have you worked in UHMG, and in the Research Dept./ 2. Tell me about the key research activities your team do wholly here in UHMG? Which ones do you contract out? Why? Work overload? Lack of specific skills? 3. What are the deliverables from research firms? Power points only? Detailed report with methodology? (Ask for sample copies of report). 4. How do you ensure that research firms provide you with quality data? Who designs the sampling? Do you check if it’s followed as planned? Field supervision? 5. What specific skills do you think you lack? (Proposal/protocol development; sampling and fieldwork? Data analysis (multivariate?); Evaluation analysis (exposed vs. not exposed, attribution analysis; Dosage analysis; Propensity Score matching). SPSS/STATA? 6. What aspect of the research do you outsource? The entire process? Data Collection? Analysis/Report writing? How many research outfits do normally work for you? How are they selected for a project? Competitive bidding? Sole sourcing? 7. Has DQA been undertaken in UHMG? When was the last one? Why and how was it done? Any report? 8. Do you have any link with the communication/mass media or BCC department? Are you skilled in evidence-based programming? 9. Do you skills in qualitative research in your team? Qualitative Software? E.g. NVIVO 10. Do you have a copy of the latest PMP for AFFORD project? How was the PMP developed? Who is in charge of its completion? Were you involved? Do you have information on condom use for most at risk groups? 11. How do you monitor the quality of GLCs? Exit Interviews? Mystery Clients? 12. How many in your team have had training in some aspect of research in the past two years? 13. Have any of you presented AFFORD’s achievements at any conference? What about journal articles? 14. What skills in Research, and M&E are being transferred from AFFORD to UHMG? Thank you very much for your time. - - EE 13. Interview Guide: Subgrantees Introduction As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As Subgrantees, you play an important role in the work of UHMG to bring quality and affordable products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you have any questions before we start. Please tell me about your subgrant with UHMG? What activities and staff are covered? What is the size of the budget to manage this scope of work? Tell me about the process of receiving your subgrant from UHMG? What did you have to do to qualify for the grant? Tell me about your participation in the post-award workshop. What did you learn there? What are the financial reporting requirements for your subgrant? Please describe the payment process. What do you have to submit in order to get paid? Have you ever had a delayed payment? If so, what happened? What kind of technical support does UHMG provide? – how frequent What kinds of things do you do when there is a technical support visit? What kind of financial reporting support does UHMG provide? What kinds of technical challenges with the subgrant have you faced? What kind of support did UHMG provide? (If no support, inquire if they brought the challenge to UHMG’s attention). What feedback, if any, has been provided on progress reports? Financial reports? What kinds of staff conduct the activities in the subgrant? (seeking type of job – peer educator, promoter, etc., - - FF ask if they are paid or volunteers? Do they have job descriptions? What kind of supervision do you provide (e.g., staff meetings, site visits, phone calls, etc.)? Can you describe a challenge with a field worker and how it was addressed? Is there anything else you would like to add that would help us understand your work as a subgrantee? Thank you very much for your time. - - GG 14. Interview Guide: Service Delivery Introduction As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As service providers, you play an important role in bringing products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you have any questions before we start. General and Accreditation Questions Please tell me about your role and activities here at the clinic. What products and packages of services are available here at this clinic? What products are available through referrals to a different level Good Life Clinic (GLC)? What steps did you have to take to become a Good Life Clinic? What characteristics did they look for in your clinic that made them feel you would be a good choice? Can you tell us about the accreditation process? How does that work? What training is required to become accredited? Tell me about your own experience with on-the-job training. What was the last course you attended? How frequently do you attend trainings? Where do you go for training? If you are already accredited, what is required, to your knowledge to maintain accreditation? What follow up is provided by UHMG? How frequently do they visit you? When was their last visit? What is the routine the supervisor goes through with you during the visit –what do they check for, what do they ask about? How long do they spend with you? If you have a question about your work, whom do you consult? What are some of the challenges you have faced as a GLC provider? How were these challenges addressed? Were they discussed with the UHMG supervisor? What support did you receive from them, if any? What support would you like to have in order to do your job better? Where Family Planning Services are provided, ask: When a woman comes into the clinic for family planning, what steps does she go through before she gets a contraceptive method? Whom does she see first? What informational materials are given.- - HH Tell me what you tell a client about the different methods available when she comes in. Describe it as if I were your client. What methods are most popular? Why? What methods are least popular? Why? What can you tell me about the availability of supplies? Are all methods always in stock? What happens if the method the woman wants is not in stock? How is a woman counseled if she complains of side effects? Have you ever had to tell a woman that she could not use a particular method? If so, what was the reason? Describe the situation. Specifically where Long term methods are provided Tell me about your experience with implants and IUDs? [Note: this question is for the appropriate level GLC where provided]. What happens when a woman comes for removal of the device? Tell me about your training in these methods. What kind of follow up on site has there been after the training? What kind of outreach or follow up, if any, is carried out on family planning in the community? Is there anything else you would like to share about your experience as a family planning provider in this clinic? Where MNCH Services are Provided, ask: [Note: 50 GLCs provide MNCH – are any on our schedule? Mostly level 4?] What services do you provide? [Note: the comprehensive UHMG list state: __ antenatal care __ delivery __ post natal care __ newborn and child health services __ integrated management of child illnesses __ Emergency Obstetric Care Please tell me how families know to come to your GLC for these services? Please tell me about what referral networks exist between your GLC and the community, including linkages from other GLCs or GLG health promoters? - - II Is your GLC equipped to handle Emergency Obstetric Care? If so, what special supplies were you furnished with to manage this? What are the challenges women face getting to your GLC in a timely way in an emergency situation? How is your GLC or UHMG contributing to addressing these challenges? What are some of the safe delivery practices your GLC provides? What services does your GLC provide in post natal care, IMCI and newborn health services? What counseling is given to mothers post-partum on breastfeeding? Are formula samples provided to mothers when they leave the facility? What counseling does a new mother receive post-partum on family planning options available to her? If she wants a pill, injectable, or other short-term method, where does she need to go to get access? If she wants a long-term method, how is that handled? Is there any data on how many women, if any, leave the GLC after delivery with a contraceptive method? If so, which methods do mothers most frequently leave with? What critical challenges, if any, has your GLC faced in the provision of any of these MNCH services? Please describe the situation and how it was handled. What specific technical support was received from UHMG? Where HIV services are provided, ask: Individuals specifically trained for the task perform counselling. b. Counselling takes place in an environment that ensures privacy. c. Counselling includes maintaining a positive health attitude. d. Counselling includes reducing the risk of HIV transmission. e. Counselling includes preventing mother-to-child transmission of HIV among reproductive-age women. f. Counselling includes verifying that the counselling is understood. Does this site have written procedures/ protocols for counseling? Have staff received formal orientation or training on the protocols? Please describe the methods and systems in place for quality assurance of counseling. How many people have presented at the site for HIV counseling and testing in past 30 days - - JJ b. What percentage of the total was female? Male? e. What percentage of those tested received their result? Please describe the testing protocol and algorithm used. (For example, brand names of first test, second test, tiebreaker test.) Do you have internal quality assurance for HIV testing? Please describe. Do you have external quality assurance for HIV testing? How many staff members in the following categories provide service to the counseling and testing unit? Is there a copy of the national counselling and testing or HIV testing guidelines on site? Is the counseling and testing service advertised or promoted in any way? How do you think counseling and testing can be improved in your institution? Are counseling and training staff supervised from outside? By whom? What specific technical support was received from UHMG Is there anything else you would like to add that you think would be useful to the evaluation team in its work? Thank you for your time. - - KK 15. Focus Group Discussion Guide: MARPS: Female Sex Workers As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As key stakeholders, you play an important role in bringing products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This discussion should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you want to be part of this evaluation by answering a few questions. Let me also know if you have any questions before we start. 1. What do you understand by Safe Sex? (Probe for how methods and reasons) 2. Risk Perception a. In your opinion, how does sex work expose you to health related risk? b. Do you think you are at risk of HIV infection? i. Probe for Level of self-assessed risk c. How can you protect yourself from HIV infection? 3. Condom Use a. How often do you use condoms with your clients? b. Some sex workers use condoms with some clients but not with others. What is your experience? i. Probe condom use with paying and non-paying clients – Reasons and how to address challenges. ii. What brand of condom do you usually use? c. Why is it difficult to suggest condom use in some situations? Any experiences to share? d. Do you think there are some friends of yours in this area/community who do not use condoms? What are the reasons? e. From what source do you obtain information on HIV? What about services such as HIV counselling and testing? 4. Afford interventions a. Talking about this community, what organizations help you to protect yourself from HIV infection? What do they do with you or for you? b. What do you know about AFFORD/UHMG? (OR ANY LOCAL IDENTIFIER OF AFFORD INTERVENTIONS) c. Tell me about your involvement with AFFORD/UHMG? - - LL d. How was it that you first learned about the program? Think back to when you first became involved with the program. e. What were your first impressions? i. Probe for ii. IPC iii. Condom promotion iv. Counselling and Testing v. Care and support f. In what way is your life different because of your participation in the program? g. Tell us something specific that has changed. How has the program changed the way you use condoms with your clients? h. What is the most important message you remember about the project? i. What need do you have that isn’t being met? How do you think what you are suggesting will improve your wellbeing? Thank you very much - - MM 16.Focus Group Discussion Guide: MARPS: Truckers and Fisher folks As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As key stakeholders, you play an important role in bringing products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This discussion should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you want to be part of this evaluation by answering a few questions. Let me also know if you have any questions before we start. 1 What do you understand by Safe Sex? (Probe for how methods and reasons) 2. Risk Perception a. In your opinion, how does your job expose you to health related risk? b. Do you think you are at risk of HIV infection? i. Probe for Level of self-assessed risk c. How can you protect yourself from HIV infection? 3. Condom Use d. How often do you use condoms with your sexual partners? e. Some truckers use condoms with some partners but not with others. What is your experience? i. Probe condom use with paying (sex workers) and non-paying partners (girlfriends) – Reasons and how to address challenges. What about your wife? f. Why is it difficult to suggest condom use in some situations? Any experiences to share? g. Do you think there are some friends of yours in this area/community who do not use condoms? What are the reasons? h. From what source do you obtain information on HIV? What about services such as HIV counselling and testing? 4. Afford interventions i. Talking about this community, what organizations help you to protect yourself from HIV infection? What do they do with you or for you? j. What do you know about AFFORD/UHMG? (OR ANY LOCAL IDENTIFIER OF AFFORD INTERVENTIONS) k. Tell me about your involvement with AFFORD/UHMG? l. How was it that you first learned about the program? Think back to when you first became involved with the program. - - NN m. What were your first impressions? i. Probe for ii. IPC iii. Condom promotion iv. Counselling and Testing v. Care and support vi. STI n. In what way is your life different because of your participation in the program? o. Tell us something specific that has changed. How has the program changed the way you use condoms with your clients? p. What is the most important message you remember about the project? q. What need do you have that isn’t being met? How do you think what you are suggesting will improve your wellbeing? Thank you - - OO 17. Focus Group Discussion Guide: Community members As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As key stakeholders, you play an important role in bringing products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This discussion should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you want to be part of this evaluation by answering a few questions. Let me also know if you have any questions before we start. 1. What is your position /role in this community? a. What do you do for a living? 2. Can we talk about the health situation in this community? What are the main health concerns? 3. In addition to the government are there any groups/associations that help in improving health care in this community? 4. Can you mention some of these groups and what they do? a. Do you play any role for any of these bodies? 5. Afford interventions a. What do you know about AFFORD/UHMG? (OR ANY LOCAL IDENTIFIER OF AFFORD INTERVENTIONS) b. Tell us more about your involvement (Probe PP i. Training ii. Service delivery iii. IPC c. What were your first impressions about UHMG interventions? What is your opinion of the program? d. In what way is this community (mention name) different because of your participation in the program? e. Tell us something specific that has changed. Probe for the following HIV testing GLC Condoms Malaria - - QQ Specific groups (e.g. youth). Availability of health products and services (e.g. child health, safe water, contraceptives, STI management, services, child immunization f. Are there any changes you have observed in terms of change in people’s health behavior? Probe for the following i. HIV testing ii. condom use iii. Bed net iv. IPT v. Family planning vi. Child immunization vii. STIs g. What is the most important message you remember about the project? h. What about the weaknesses? Any weakness you can talk about? i. What challenges did people in the community face in participating in AFFORD interventions? Are the challenges still barriers to participating/ j. What need do you have that isn’t being met? How do you think what you are suggesting will improve your wellbeing? - - RR 18. Focus Group Discussion Guide: Youth As you know, we are here to help take a look at the AFFORD project and learn about how it has contributed to the effort to bring health products and services, through social marketing, to communities. As key stakeholders, you play an important role in bringing products and services to the community. So your inputs are very valuable. Your comments are confidential, and your name will not be associated with anything stated in the report. This interview should take no more than an hour. Thank you for taking the time to answer our questions. Please let me know if you want to be part of this evaluation by answering a few questions. Let me also know if you have any questions before we start. Can we talk about the health situation in this community? What are the main health concerns? Can you mention some of these groups and what they do? Do you play any role for any of these bodies? Risk Perception In your opinion, how does your job expose you to health related risk? Do you think you are at risk of HIV infection? Probe for Level of self-assessed risk How can you protect yourself from HIV infection? Probe Condom use? Partner reduction; Abstinence What are you doing to prevent yourself from HIV infection? Condom Use How often do you use condoms with your sexual partners? Some young persons use condoms with some partners but not with others. What is your experience? Why is it difficult to suggest condom use in some situations? Any experiences to share? Do you think there are some friends of yours in this area/community who do not use condoms? What are the reasons? From what source do you obtain information on HIV? What about services such as HIV counselling and testing? What do you know about AFFORD/UHMG? (OR ANY LOCAL IDENTIFIER OF AFFORD INTERVENTIONS) Tell us more about your involvement (Probe - - SS i. Training; Service delivery; IPC; Community volunteers What were your first impressions about UHMG interventions? What is your opinion of the program? k. In what way is this community (mention name) different because of your participation in the program? TT l. Tell us something specific that has changed. Probe for the following HIV testing GLC Condoms Malaria Specific groups Availability of health products and services (e.g. child health, safe water, contraceptives, STI management, services, child immunization m. Are there any changes you have observed in terms of change in people’s health behavior? Probe for the following i. HIV testing ii. condom use iii. Bed net iv. IPT v. Family planning vi. Child immunization vii. STIs n. What is the most important message you remember about the project? o. What about the weaknesses? Any weakness you can talk about? p. How involved are young persons in this community in AFFORD interventions? q. Do young people visit GLC? If yes/no. What are the reasons? r. What need do you have that isn’t being met? How do you think what you are suggesting will improve your wellbeing? Thank you. UU Annex 5 Methodology An Overview The evaluation utilized a methodology that included qualitative data generated by the evaluation team, and secondary project data provided by project staff, to answer the key evaluation questions. The primary means of data collection involved document and systems review, individual interviews, group interviews, and observations. The stakeholders interviewed included staff of USAID, the Ministry of Health, UHMG, Implementing Partners, Donors, the Uganda AIDS Commission, Subgrantees, GLC staff, and beneficiaries of products and services. Instruments, including interview and discussion guides as well as checklists, were developed in Kampala by the Evaluation Team before the start of data collection field trips to guide interviews and observations. For organizational capacity assessment, the team developed interview questions, and confirmed through interviews, progress according to KPMG’s technical assistance report for 2013, as well as audit reports. We used the Pact Organizational Capacity Assessment Tool (OCAT) for South Africa to evaluate the stages of organizational development for each key area of UHMG. Data collection methods The data collection was primarily qualitative in nature, although quantitative secondary project data were examined closely to answer the key evaluation questions. The evaluation team employed three main data collection methods: (1) Document Review of Project Monitoring Data and existing literature; (2) In-Depth Interviews; and (3) Focus Group Discussions. Document review of project data and existing literature This encompassed review of AFFORD/UHMG’s work plan and progress reports, financial documentation, their Performance Management Plan, and baseline data, as available. The Evaluation Team also analyzed the results achieved against the targets and benchmarks set. Monitoring reports, research reports, related databases and protocols, as well as BCC materials, were examined. Several other publications were consulted and reviewed in order to obtain a comprehensive understanding of the background and context useful for evaluating Social Marketing in Uganda, AFFORD and UHMG. They included the 2011 Uganda Demographic and Health Survey and the State of Uganda Population Report 2012. In-Depth Interviews Key informant interviews were held early in the evaluation process with the technical, financial and management staff of AFFORD/UHMG. Those interviewed included some members of the Founders’ Forum, some Board of Directors, senior management and staff of AFFORD/UHMG project. The evaluation team then conduct in-depth interviews with key stakeholders and partners in Uganda. The team developed a semi-structured interview tool that was used to guide the interview process. VV Focus Group Discussions (FGDs) Altogether eleven focus group discussions were conducted with respondents selected from the project intervention communities. There were two with sex workers, three with youth, two with fisher-folks, two with truckers, two with peer educators and one with married couples. Each FGD group composed of 8 to 12 participants. Sampling plan Although purposive sampling was used, efforts were made to include as many diverse groups and individuals as possible. In-depth interviews with stakeholders were among government officials, donors, implementing partners and service providers. An initial list of interviewees was compiled by USAID/Uganda, UHMG and the evaluation team to include government, donors, Implementing Partners, and Subgrantees. Altogether, about 54 individuals were interviewed across these different stakeholder categories. The list of UHMG staff to be interviewed was selected by the evaluation team and UHMG to reflect the key divisions of the organization: Finance, Human Resources and Administration, Knowledge Management and Research, Product Facilities, Programs, and AFFORD staff. In addition, seven Founding members and Board members were interviewed. USAID/Uganda and the evaluation team together selected field sites, taking into account the variety of interventions offered and geographic diversity. Altogether, ten districts were visited. The sites for the evaluation were districts satisfying the following selection criteria: (1) One in each of the four regions of Uganda; (2) With presence of a UHMG subgrantee; existence of Good Life Clinics (GLCs) and/or the presence of interventions for most-at-risk-populations. Under the guidance of the evaluation team, UHMG staff and the sub-grantees assisted in the selection of focus group participants. C. Field Work The evaluation team spent a week (August 5 to 9) organizing the evaluation, meeting with USAID/Uganda and UHMG/AFFORD and preparing an inception report as well as designing tools and procedures for the evaluation. It spent three days (August 12 to 14) interviewing AFFORD/UHMG managers and staff as well as visiting implementing partners in Kampala area. After initial interviews in Kampala, the team spent about 10 days (August 15-24) visiting program sites and interviewing implementing partners, subgrantees, and program beneficiaries in ten districts of Kampala, Jinja, Oyam,GuluMbale, Masaka, Kalangala, Mbarara, Kasese, and Kabarole. The team returned to Kampala and spent one week (August 26-30) interviewing USAID staff, and concluded any key outstanding interviews with the Board of Directors, the Founders, and other UHMG or AFFORD staff. The team spent the last two weeks (September 2-14) reviewing and analyzing project data, reports and records,preparing the draft report to USAID, and revising and submitting the draft final report to USAID. The team also made a presentation to USAID staff on the preliminary results and recommendations. D. Data Analysis Plan WW Where project data existed, this evaluation compared baseline and end-line results. Given the amount of time involved in the translation and transcription of information, the analysis of qualitative data was notes-based. Using the thematic approach, with themes derived mainly from the evaluation questions; data were analyzed to address appropriate evaluation questions. Since analysis was done manually, we adopted a simple approach of putting all the information addressing a similar sub-question or issue together with a heading that reflected the category they presented, using bullets to separate the different units. Data were then summarized according to themes identified in the data. F. Limitations Given the diversity and varied nature of UHMG’s interventions, our sampling was not representative as only a few intervention sites could be visited. For logistic and time-related reasons our choice of sites may result in selection bias. Our team was not fully constituted as per the SOW at the start of the assignment. This meant we could not split into two teams, which would have allowed the team to visit more field sites. G. Quality Control Plan i: Introduction At the first meeting of the team, members will agreed on the roles, based on skills and value addition to the assignment. Each team member committed themselves to timeliness and quality of deliverables and agreed to hold each other mutually accountable for the success of the assignment ii: Data Collection The consulting team ensured quality in the data collection process. This was achieved by the following:  At the end of each day, the team met for debrief, reviewed achievements and challenges of the day. Team members continually compared notes and took note of emerging issues to have proper record of the challenges, solutions and trends in the data collected. iii.: Preparation, Check and Approval of Project Documents Reports produced by the consulting team included the inception report, draft report; draft final report and the final report. iv: Project Review During the project execution, the Team Leader appraised the performance of the team on a daily basis. The team leader ensured that the team focussed on answering the key evaluation questions, and the overall purpose of the assignment. XX Annex 6 Challenges faced by some GLCs Recordkeeping and Data Collection: As in many other areas already addressed, data collection and recordkeeping are not systematic. The information we received was often contradictory or incomplete. We have seen various documents with different numbers of accredited GLCs. One, for example, said 200, and another said 169. Figures we received on service statistics were different than those presented in the PMP. The AFFORD summary report indicates that of 169 accredited GLCs, only 50 have been accredited to provide MNCH services. We were later informed that of these 50 GLCs, 22 had been trained in a pilot effort to provide PMTCT starting from this financial year. We have seen no data or lessons learned on how effective these services are in the current clinics, and have not seen a strategy for expansion of services after these initial pilots. Quality Improvement: As stated previously, an Operations Manual is provided to all franchisees to ensure standardized quality of services. It also states consequences for noncompliance with these standards. Although many GLCs joined the network within the past few years, AFFORD has not developed a Quality Operations Manual. In fact, use of a Quality Improvement tool did not begin at AFFORD until March 2013, four years after the first GLCs were accredited. None of the GLCs we visited had been through a thorough QI assessment, before providing long-term contraceptive methods and performing deliveries of newborns. The QI summary report on the 50 GLCs offering MNCH services shows 15 out of 50 clinics, or 30%, to have scored less than satisfactory (between 24 and 65) on infection prevention. Although AFFORD has engaged 15 QI consultant physicians to monitor GLCs, the reports we have seen thus far focused more on spot checking the clinic environment and supplies/equipment available. We have not seen any reports on corrective follow up measures taken based on the QI summary report. Counseling: Another GLC activity requiring further attention is counseling. We asked GLC providers to counsel us on long-term methods. With regard to implants, providers told us about convenience of use for those seeking 3-5 years of protection. Side effects were not mentioned until probed. When asked, clinicians only mentioned not to worry about headache and weakness. Bleeding pattern side effects were not mentioned without further probing. Clinical and introductory trial data on implants, however, indicate that the major side effect is bleeding pattern disruption. Headache occurs more with combined oral contraceptives than with progestin-only methods like implantsFor implants, the cumulative removal rate is 21% over 2 years. The relationship between the high removal rate and the possible lack of counseling on bleeding side effects needs to be explored further. Consistency of Supplies: Fortunately, Implanon implants are preloaded into a sterilized, disposable trocar. However, for Jadelle, one clinic had been given enough trocars to dispose of them after insertion, while others had not and were re-sterilizing them. Not all clinics had autoclaves so some were cleaning trocars with disinfectant (Jik) before re-sterilizing with simple boiling technique for 15 minutes. One laboratory technician described washing and rinsing utensils with Nomi soap and tap water after they had already been boiled Providers were also unhappy that promises to provide equipment like mama kits and HCT tests were not kept. It is unclear why such promises were made since mama kits are not part of the package AFFORD/UHMG was giving to pregnant women. Informational Materials for GLCs:During our clinic visits, we asked GLCs, to show us the manuals they used for working with clients or target groups. In each case we were shown a stack of documents, YY mostly from the Ministry of Health, but also including technical documents, and training manuals from AFFORD that had redundant information. At the GLC level, we noticed numerous flipchart books, some of which were outdated, but many contained similar overlapping information. In one GLC, a clinician showed a flipchart book on implants that discussed Jadelle and Implanon. The duration of efficacy for the latter two was stated as 3-5 years, although one is recommended for three years (Implanon) and the other for 5 years (Jadelle). ZZ Annex 7 Management of Subgrantees Peer and Health Educator motivation, role and challenges: Most peer and health educators were proud of the role they play in the community. One commented, “I’m happy because I am known, they ask for me, I get respect.” Peer and Health educators earn some money from the sale of Protector condoms and generally they admitted to reinvesting the profits in buying more condoms, although one confessed to using it for other purposes, like purchasing flour. They see their role as mobilizing MARPs to seek and use services, in particular promoting condom use, HCT, and male circumcision. FSWs in activities supported by both NAYODE and SSECODA learned to organize and provide peer support. Behavior Change and Data Collection: Peer educators working through subgrantees primarily provide information to MARPs, sell condoms, and refer clients to GLCs for HCT and male circumcision. Occasionally they organize outreach events to attract as many people as possible to provide them with services at the event. Peer educators keep records of only names, sex and age. There is no record to indicate they have followed up with any clients or that their clients have actually attended a government facility or a GLC after referral. The only data collected is the number of people with whom they have had contact to share information or sell condoms. The subgrantees total all these contacts and sales and submit them in their progress reports as evidence of achievement. However, this data is not an indicator for behavior change. In addition, the formula used for calculating targets should be revisited. For example, in Kalangala, we were told that the subgrantee targets 19,500 fisher folk on an island with a total population of 70,000, where 50% are children, and of the remaining 35,000 more than half are women. This target represents a gross overestimate. Informational Materials: Peer and health educators did not have manuals that were explicitly aimed at providing messages for MARPs. They also showed us stacks of documents provided by UHMG from the MOH and other sources that were far too complex for the intended audience. Peer educators need a simple, user friendly guide that has been field tested. It should be an easy to carry manual, organized with tabs to take them quickly to the messages and information they need for counseling on HCTs, dual protection, family planning, PMTCT, condom use, and other issues of concern to the groups they counsel. We saw no low literacy materials for peer educators. Some of the flipcharts peer educators were using were hand painted, and the messages came from UHMG posters (which are advertisements and not messages for the target MARP audience). Youth Services:UHMG does not have a targeted coordinated youth intervention initiative. There is evidence in the literature that clinical franchisors are generally not youth friendly (YouthNet: Applying Social Franchising Techniques to Youth Reproductive Health/HIV Services). In UHMG documents, youth are defined as the age range 18-30. MSI messages were developed using this age range. In Kasese we found that despite this UHMG definition, they are addressing the 15-24 age group. It is therefore urgent that a younger age range be considered in all UHMG activities. One subgrantee, NAYODE in Kasese, mentioned outreach activities in local schools as well as for out of school youth. They operate a youth center where we were able to conduct interviews with young adults. Participants in this interview session were asked about their experiences with GLCs. They responded that GLCs are not youth friendly, and they were often turned away. AAA Annex 8 Staffing Growth and Composition PROJECT COMPOSITION FY07 FY08 FY09 FY10 FY11 FY12 FY13 Totals - 31 37 38 44 71 93 118 AFFORD JHU/CCP Numbers 31 37 37 39 51 66 85 Percentage % 100 100 97.37 88.64 71.83 70.97 72.03 Other projects Numbers 0 0 1 5 20 27 33 Percentage % 0 0 2.632 11.36 28.17 29.03 27.97 Other projects composition KORSHIP 0 0 1 2 1 0 0 STRIDES 0 0 0 1 2 2 0 STAR-SW 0 0 0 2 2 2 2 SCIPHA 0 0 0 0 10 10 10 Sub Projects 0 0 0 0 5 2 0 KGLIS 0 0 0 0 0 7 15 KAP 0 0 0 0 0 1 1 NUHITES 0 0 0 0 0 3 5 Totals 0 0 1 5 20 27 33 These figures show increasing numbers of staff assigned to AFFORD budget in 2011-2013 BBB Annex 9 Social Marketing Operation (SMO) – an Overview of considerations for startup There are operational issues that will be ongoing throughout the lifecycle of a Social Marketing enterprise and others that will be one time and others that will be periodic. Below is as concise an explanation of an effective protocol, as is possible in the available space. Before establishing a Social Marketing operation, research and incorporate “Best Practices” for every functional area of your organization. It is assumed that your product has been identified before you get started. But, if not, identify a problem in the community you plan to serve and provide products and services to that community to mitigate the problem at a price the lower income segment of the community can afford. There are many kinds of products in every category you might utilize, each with its own perceived benefit to your operation and to the perception of the consumer. All must meet established quality control standards but there are many varieties in the same category to choose from and in many cases, multiple varieties could be made available to segment your market, strengthen your bottom line and provide the consumer with choices. The 5 Ps are an integral part of any marketing effort. They consist of:  Product – ISO/WHO standard quality  Price – Affordable to your target consumer  Packaging* – Protects the product and attracts trial/continued use  Placement – Accessible at a time and place needed by your target consumer  Promotion – Targeted to your principle consumer but effective for all *Packaging is often placed under “Product” by private sector marketers. It is used as an additional P to emphasize the importance of packaging in contributing to the perception of quality/value when offered to consumers at more accessible pricing than private sector supply/demand formulas. 1st. You need to define your target audience and all niches within the population you hope to impact with your product and communications. Social Marketing isn’t just private sector product distribution at an accessible price. It is an integrated process of product, quality assurance, packaging, IEC, BCC, large and small media campaigns, peer networks, promotions, journalist PR, research, and public and social sector liaison and coordination. 2nd: Known benchmarks or best practices must be identified and general background research must be carried out to establish a baseline for future evaluation of success in meeting your objectives. 3rd. You must consult with all stakeholders and carry out a comprehensive strategic planning exercise for the following 3 to 5 years. 4th. An organization must be created, employees hired for both office and field work who are of a Social Marketing mindset, and you must both sell and promote your product. CCC 5th. During the strategic planning process an M&E strategy would have also been devised and after a set period of implementing your activities, as stated in step 4, you must evaluate your efforts, including management practices and sales activities. This activity audit must provide you useful information to then incorporate in step 1 which will be updated annually. This is the circle of life for a business. There are a number of influences that impact the success of your business that you can not control, but they must be known to you and you must take these things into consideration as you plan your operations and activities. They consist of:  Middlemen (collaborators and facilitators) – Distributors, Retail agents, independent sales force, NGOs, peer network, local public officials, and anyone who is between your operation and the consumer.  Suppliers and Providers – There are two dimensions to this group. There is the one you need and the one that needs you. You need the supplier of funds to keep your business alive until your income exceeds your expenses (donor organization, or LOC guarantor). Next is the manufacturer who you purchase product from to sell and all the product or service providers you require to maintain a thriving organization. These represent the lifeline of your business: suppliers and providers.  Competition – In Social Marketing, competition is not those who sell your product in the marketplace as much as it is those products or activities that distract the potential consumer of you product from accessing your product and instead use their disposable income to satisfy the perceived need of the competition’s product or service. Example: a customer doesn’t think it is “cool” to use condoms so doesn’t buy them but instead buys a “smartphone”; Or, they think the Sex Worker is too young and cute and can’t possibly have an STI and avoids using one. There are many distractions/competition to consistently and correctly using condoms. You must know and understand the competition.  Key Behavior Influencers – There are various individuals within your market and outside of your market that influence your target consumer. Have these behavior influencers on your side and your consumer will come to you. As you establish your operations, you will necessarily need to take into consideration factors generally beyond your influence or control. There may be modest exceptions to your level of influence over some. These uncontrollable factors that impact your business are:  Demographics and Economics – As an economy grows, so does the populations disposable income grow and the relative cost of your CSM product declines, making it more and more affordable and allows greater and greater sustainability of your operation.  Laws and Politics – Each place you sell your product is likely to have different rules and regulations applicable to pricing, placing, and promoting your product. You must know these and avoid upsetting the local and regional powers that be. DDD  Physical Environment – How do you get your product to your consumer and keep it fresh? Are their physical constraints such as bad roads, rivers with bridges out, remote villages, great distances between major markets, too hot to store safely, no fax, no computer, no electricity to complete reporting requirements? Consider these issues and resolve them before you start.  Local Customs – The local, regional, and national cultural practices must be known to the marketer before an appropriate plan can be devised to reach your target consumer. Know them.