Report of the 
Final Evaluation of Umoyo Network Malawi 
USAID Grant # HRN-A-00-98-00011-00 
Hugh Goyder 
March 2003 
For further information contact 
Carrie Osborne 
Program Manager 
Umoyo Network 
Save the Children US 
Private Bag 254 
Blantyre 
Malawi 
Tel: 265-621348 
Email: umoyo@malawi.net 
Prepared under 
Cooperative Agreement 
HRN-A-00-98-00011-00 
by NGO Networks for 
Health, a global initiative 
supported by USAID and 
managed by Save the 
Children in partnership 
with the Adventist 
Development and Relief 
Agency (ADRA), CARE, 
PLAN International, and 
Program for Appropriate 
Technology in Health 
(PATH)
Copyright © 2003 by NGO Networks for Health. All rights reserved. 
Additional copies may be obtained from: 
NGO Networks for Health 
2000 M Street NW, Suite 500 
Washington, DC 20036 
Tel. 202-955-0070 
Fax 202-955-1105 
www.ngonetworks.org 
info@ngonetworks.org
NGO Networks Umoyo Network Final Evaluation Report Page i 
Table of Contents 
Acronyms and Abbreviations 2 
1. Introduction 3 
2. Methodology 3 
3. Project Background 4 
4. Findings 7 
 4.1 Grants Management 7 
 4.2 Umoyo Network’s Impact at Community Level 7 
 4.3 The Impact of Umoyo Network’s Capacity Building Activities 13
 4.4 Monitoring and Reporting on Impact 14
 4.5 Benefits of a Network 15
 4.6 Umoyo Network’s Impact on Quality of Service 16
5. Conclusions of the Final Evaluation 16
6. Recommendations for Umoyo Network 18
 6.1 Capacity Building 18
 6.2 Advocacy 18
 6.3 Monitoring and Evaluation 18
 6.4 Grants Management 19
 6.5 Quality of Services 19
7. Recommendations for USAID - Malawi 20
ATTACHMENTS 
A. Evaluation of Umoyo Network’s Capacity Building 21
B. Evaluation of Quality of Services offered by Umoyo Network Partners 25
APPENDICES 
1. List of documents consulted and people met 28
2. Scope of Work 29
NGO Networks Umoyo ID Evaluation Final Report Page ii 
Acronyms and Abbreviations 
ADRA: Adventist Development and Relief Agency 
AHS: Adventist Health Services, Malawi 
AIDS: Acquired Immune-Deficiency Syndrome 
BCI: Behavior Change Initiative 
CBDAs: Community Based Distribution Agents 
CBO: Community Based Organization 
CHAM: Churches Health Association of Malawi 
CHBC: Community Home Based Care 
CS: Child Survival 
DAPP: Development Aid from People to People 
DFID: Department for International Development 
FP: Family Planning 
GOM: Government of Malawi 
HBC: Home Based Care 
HIV: Human Immune-Deficiency Virus 
ID Institutional Development 
LQAS: Lot Quality Assurance Sampling 
MACRO: Malawi AIDS Counseling and Resource Organization 
MANASO: Malawi Association of National AIDS Service Organizations 
M&E: Monitoring and Evaluation 
MANET+: Malawi Network of People living with HIV/AIDS 
MOHP: Ministry of Health and Population, Malawi 
NAC: National AIDS Commission, Malawi 
NAPHAM: National Association of People Living with HIV/AIDS in 
 Malawi 
NGO: Non-Governmental Organization 
NSO: National Statistics Office 
PLWHA Persons Living with HIV/AIDS 
PSI: Population Services International 
PVO: Private Voluntary Organization 
QOS Quality of Service 
RFA: Request for Applications (by USAID) 
RH: Reproductive Health (includes family planning, safe motherhood 
and the prevention and treatment of sexually transmitted diseases) 
SOW: Scope of Work 
STAFH: Support to AIDS and Family Health 
STD: Sexually Transmitted Disease 
STI: Sexually Transmitted Infection 
USAID: United States Agency for International Development 
VCT: Voluntary Counseling and Testing 
NGO Networks Umoyo ID Evaluation Final Report Page 1 
Final Evaluation of Umoyo Network 
1. Introduction 
This report documents the summative evaluation of Umoyo Network, Malawi, a 
USAID-funded project aimed at providing funding and technical assistance to 
Malawian NGOs working in reproductive health and HIV/AIDS. According to the 
consultant’s SOW, the purpose of the evaluation is ‘to document the 
accomplishments of Umoyo Network against its original objectives and to 
establish: 
What are the strategies that Umoyo Network utilized with the Sub-Grantees that 
have worked to improve and expand services and strengthened the organizations 
overall? Looking at these strategies – if they worked, why did they work? If not, 
why? 
What kind of unintentional events have helped or hindered the achievements of 
the goal?’ 
The evaluation was conducted in two short inputs, one in early December 2002 
and the second in March 2003. The full SOW is given in Appendix 2. 
2. Methodology 
This evaluation is based on the following: 
• discussions with Umoyo Network staff, USAID Mission HIV/AIDS Team 
Leader and Program Director of National AIDS Commission (NAC) 
• informal discussions with staff and volunteers at DAPP, TSA, and Malamulo 
Hospital 
• meetings and one phone interviews with managers of MANASO, AHS, and 
NAPHAM, and the Program Director of the National Aids Commission 
• an analysis of project documentation – including selected consultant reports, 
MIS data in quarterly and annual reports, the 2001 Mid Term Evaluations - a 
review of the reports of, and discussions with, the evaluators of the Quality of 
Service (March 2003) and the Institutional Development Program (November 
2002), and in the case of the ID Evaluation, participation in his debriefing 
session with Umoyo Network staff. 
The word ‘sub-grantee’ suggests that Umoyo Network is primarily a disbursement 
mechanism for USAID funds. Over the last 3 years the ‘sub-grantees’ have come 
to see themselves more as ‘local partners’ of the Network, (and receive much 
Page 2 NGO Networks Umoyo ID Evaluation Final Report 
more than funding from it): we therefore refer to them as local partners 
throughout. 
This report was commissioned as a ‘Final Evaluation’ of Umoyo Network for the 
period of the first contract awarded by USAID to Save the Children US (SC) from 
1999 to June 2003. During the research it became clear that every stakeholder 
involved sees the Network continuing in some kind of form well beyond that date. 
This evaluation report is therefore written more as a kind of ‘mid-term evaluation 
of a continuing project, and it makes recommendations for any further phase of 
funding from USAID. 
3. Project Background 
UNICEF estimates that more than 300,000 people are estimated to have died of 
AIDS-related illnesses since the first case in Malawi was reported in 1985.1
Today around 9 per cent of the 10.6 million population, or about one million 
people, are believed to be infected with HIV. But efforts to prevent and treat 
HIV/AIDS are hampered by severe poverty, increasingly frequent food shortages, 
and a severely under-resourced and under-staffed government health sector. 
USAID has therefore been active in finding ways of meeting this considerable 
challenge for over 11 years. 
Umoyo Network emerged out of the Support to AIDS and Family Health or 
STAFH Project, which was funded by USAID from 1992-2001 and included 
grants for 24 local NGOs. As a result of this project USAID/Malawi felt it needed 
to make a far greater investment in technical and institutional capacity building if 
funding to local NGOs in the health sector was to be really effective. In 1999 it 
therefore invited NGO Networks for Health to make a proposal that would 
promote sustainability through a phased marriage of grants management, 
capacity building, and network development. This proposal was consistent with 
the key Strategic Objectives of USAID - the ‘increased adoption of measures that 
reduce fertility & risk of HIV transmission, including improved child health 
services’ and ‘Behaviors adopted that reduce fertility & HIV/AIDS & improve child 
health’. Umoyo Network’s own goal was initially defined in the initial SOW in May 
1990 as ‘to promote & support the development of networks in order to improve 
the scope & quality of service delivery mechanisms’ 
Given this very brief history, three issues require comment. First Umoyo 
Network’s three activities (grants management, capacity building, and network 
development) are complementary, but each activity requires a specific set of 
skills. Initially there were difficulties in making this ‘phased marriage’ work well, 
and the annual report for 2000 said that ‘grants management & network 
development are separate activities requiring different types of staffing, phasing 
of activities, & support systems’ While the ID Evaluation suggests that, from the 
perspective of local partners, these different activities were in the end able to 
 
1
 UNICEF State of the World’s Children (2002) 
NGO Networks Umoyo ID Evaluation Final Report Page 3 
combine more successfully, there were definitely initial uncertainties, which are 
referred to below. 
Secondly, Umoyo Network is remarkable for its large and diverse number of 
stakeholders. NGO Networks for Health was a global health partnership program 
supported by USAID through a consortium of 5 US-based PVOs, and its major 
focus was to build the capacity of these PVOs to provide quality FP/RH/CS and 
HIV services. Save the Children US (SC) became the lead agency, and both its 
US and Malawi offices support the management of Umoyo Network. USAID/M as 
the sole donor has always had an important monitoring role. Within Malawi, the 
government (in particular the MOHP and the National Aids Commission) and 
local partners have a strong interest. This multiplicity of stakeholders has made 
the management of Umoyo Network an extremely complex and multi-faceted 
task, and lends support to a recent finding from USAID’s Bureau of Global Health 
that any new activity with PVOS/NGOs should have a fairly simple, 
straightforward design and avoid overly complex mandates.2
 
Thirdly, at least initially, there were discernible differences in the priorities of 
some of these stakeholders, with NGO Networks tending to emphasise 
networking as a priority, including an initial emphasis on setting up a Resource 
Centre in Blantyre, while USAID and SC have always given the highest priority to 
effective capacity building. The Strategic Objective in the initial Results 
Framework quoted above reflects these mixed agendas as it tries to link the 
anticipated improvement in services with the enhanced capacities of PVO/NGO 
Networks. This appears to refer to the expansion of NGO Networks as a 
Washington-based global project rather than to the capacity of local NGO 
networks. 
As a result of these overlapping agendas the first year was difficult, with changes 
in staff both in Umoyo Network and NGO Networks for Health. Some staff who 
worked for Umoyo Network at this time felt that the involvement of NGO 
Networks for Health and their frequent consultancy inputs were a ‘distraction’. In 
particular local partners initially had to cope with far more demanding systems for 
financial disbursement and M & E than they were used to, and these partners felt 
they were spending too much time collecting data, and too little on 
implementation: one ex-member of staff said that in the first year 60% of our time 
seemed to go into data collection and evaluation. 
The absence of an overall Log Frame or clear Results Framework for Umoyo 
Network itself has probably complicated the overall management of the project. 
For instance Darcy Ashman noted an internal sense of confusion (about project 
goals) but added that it has not appeared to affect the external performance of 
the project3
. Both Darcy Ashman and the ID Evaluation conclude that the 
 
2
 USAID –Bureau of Global Health/Office of Population & Reproductive Health Enhancing PVO & NGO 
Partnerships in Family Planning & Reproductive Health (Sept 2002) 3
 Network Development Report – Darcy Ashman, NGO Networks for Health Dec 2001. 
Page 4 NGO Networks Umoyo ID Evaluation Final Report 
underlying uncertainties about project goals have not seriously hampered the 
work, but they have probably made it a harder project to manage, monitor, report 
on, and evaluate. 
One early challenge for the project was to reach its own definition of a ‘Network’. 
The word ‘Network’ is itself a little ambiguous as the term is now widely used to 
cover a range of technical and institutional relationships. A consultant from NGO 
Networks, Darcy Ashman reviewed this issue in December 2001 and posed the 
question: Is Umoyo Network itself a network that promotes networking – or is 
Umoyo Network basically a USAID project that promotes networking, among 
other goals? She concluded that it is more of the latter – a project that promotes 
service delivery, capacity building, and networking in HIV/AIDS and RH. 
From these early uncertainties Umoyo Network has consolidated its activities 
round its initial assumption that improved programme delivery by the NGOs 
would follow after the organizational capacity building, and that networking or 
networking development function …was intended to follow after the grant 
management mechanism was established. 4
The original hypothesis on which the Umoyo Network was based was that 
‘enhancing PVO/NGO capacity to provide FP/RH/CS/HIV services and 
strengthening PVOs/NGOs will result in a significant and sustainable increase in 
the quality, access, and use of health information and services.’ This hypothesis 
contained some implicit assumptions about how effectively even strong NGOs 
can operate without government support, especially in relation to clear 
government policies and standards, an improving (rather than declining) 
government health service, a sufficient supply of test kits, drugs and FP supplies, 
and an availability of appropriately trained staff. All of these are frequent 
constraints mentioned in the mid-term evaluations, and remain challenges 
requiring advocacy at the time of this evaluation. (See recommendations on 
advocacy below) 
 
In accordance with national policies, Umoyo Network has always aimed at the 
integration of FP, RH, treatment of STIs, with HIV/AIDS related services, and has 
employed staff with specialist knowledge of these sectors. Though this integrated 
approach made sense to the established mission hospitals, those local partners 
that had been founded specifically in response to the HIV/AIDS epidemic saw it 
as far less relevant.5
 This commitment has meant that the Network is aiming to 
offer its local partners a wide range of both institutional and technical support – 
including grants management, inputs on governance and management, as well 
as technical advice on FP/RH/STI’s, HIV/AIDS, and VCT. Rick James has 
evaluated the impact of this capacity-building work from the local partners’ 
 
4
 Darcy Ashman (2001) 
5
 Needs Assessment of the Status of Integration of HIV/AIDS, STI, & Reproductive Health Services 
achieved by Umoyo sub-grantees, Malawi. (Ruminjo & Wohlfarht for ENGENDERHEALTH April 2002) 
NGO Networks Umoyo ID Evaluation Final Report Page 5 
perspective, and the Executive Summary of his report is included in Attachment 
2. 
4. Findings 
4.1 Grants management 
While capacity building has grown to be the major focus of the work, grants 
management remains of vital importance. In the whole period of funding (4 years 
from June 1999) the Network has spent $7.7million in total, of which almost $4.5 
million is in the form of sub-grants to the 11 different Network members. In 
addition $102,372 has been distributed in 43 small grants to local organizations. 
This is a fairly labor intensive scheme as 316 proposals were received in total, 
and there has never been an evaluation of its impact. (See recommendations). 
However there is strong anecdotal evidence that this kind of small scale funding 
is very helpful, and the NAC is now building on the Network’s experience in 
setting up its own small grant’s scheme. An example of this is seen in the case 
of Ubale (or ‘Friendship’) Network described in the box below: 
Ubale Network, Blantyre: an example of how networks can work with 
minimal funding 
The Ubale Network is a group of small CBOs, all run by volunteers, who look 
after orphans in and around Blantyre to enable their carers to go to work. This 
network has grown from 6 CBOs in 2001 to 34 in March 2003.Umoyo Network 
has helped with a total of only $5,836. This money has been used to raise the 
profile of the Network with the Dept of Soc Welfare and other Government 
Departments, and it has also paid for the Network to undertake a Needs 
Assessment of CBOs involved in the care and support for Orphans in. Following 
a recent Fundraising Workshop run by Umoyo Network they mobilized more than 
60 volunteers to clean the grounds of offices, factories and shops in return for 
food and cash donations. 
 
4.2 Umoyo Network’s impact at community level 
4.2.1 LQAS as an Evaluation tool 
The major way in which Umoyo Network’s impact at community level has been 
assessed has been by means of LQAS surveys. Before reviewing the information 
provided by these surveys it is necessary first to assess the strengths and 
weaknesses of the LQAS approach itself. This is especially important in that 
LQAS ‘came with the contract’: over the course of the project the LQAS 
methodology has been successfully transferred to Umoyo Network’s partners 
and in the long term it may well represent the most significant contribution made 
by NGO Networks to the whole project. One local partner told the ID Evaluation 
Page 6 NGO Networks Umoyo ID Evaluation Final Report 
that at first the LQAS was ‘derailing the operations of NGOs and people were 
very unhappy. But the beautiful baseline data produced was very illuminating, 
and we changed our program design on the basis of this data’. The strengths and 
weaknesses of the LQAS appear as follows: 
LQAS strengths: 
- The data generated has been tested and has been found to be reliable 
- The use of small samples makes it relatively cost effective 
- Local partner organization staff are able to undertake the data collection, 
analysis and report writing 
- It is now popular with local partners, who after up to 3 survey ‘rounds’ are now 
able to conduct their own surveys, and (according to the CB evaluation) are 
increasingly using it for their own decision-making. The sub-division of areas 
into different ‘supervision areas’ is especially useful for disaggregating overall 
performance. 
- It offers useful data for comparison of impact across different local partners 
LQAS weaknesses 
- It is a quantitative approach: there is a need to supplement it with more 
qualitative analysis – a gap addressed in the participatory Mid Term 
Evaluations 
- It puts more focus on attitudes and behaviour at the community than on 
changes in the capacity of local partners, so is only a partial answer to either 
the local partners’ or Umoyo Network’s overall evaluation needs (see below) 
- It has absorbed a relatively large amount of the time of M & E staff at the 
expense of other M & E methods, especially in the first 2 years. As the Mid￾Term Evaluation Report for AHS (July 2001) said a shortfall of the focus on 
LQAS is that no attention has been given to AHS own monitoring system and 
evaluation systems at headquarters and at the health centers. 
- There have been some problems with the analysis of the consolidated data 
which has been sent to Washington for analysis with delays in production of 
reports and there has to date been far too little feedback of the results of this 
analysis either to the local partners or to Umoyo Network itself. This whole 
arrangement appears to Umoyo Network staff as ‘extractive’ and hardly 
seems designed to build local analytical capacity. 
The conclusion of this evaluation is that the LQAS methodology should be 
maintained, since it represents an investment in time and effort, and the 
approach has been successfully transferred to local partners. There would be 
very high training costs and a risk of losing valuable data if a new contractor were 
to come in after June 2003 with a different approach. Consistency and continuity 
are key principles in this kind of capacity building work, and whatever the 
outcome of the current RFA exercise it would be useful for as many partners as 
NGO Networks Umoyo ID Evaluation Final Report Page 7 
possible to undertake further LQAS surveys, using those questions relevant to 
their particular programmes. 
However in any future project an annual LQAS survey should be optional, and 
the next survey should be done by most partners after a further 2 years of 
funding as part of a broader mid-term review process. The analysis of the results 
should be done as far as possible by the local partners themselves, with 
assistance where required from Umoyo Network’s M & E Department. 
While the LQAS should be one important component of any future programme, 
there is also a need for more outcome and impact indicators, as noted by the MIS 
report that saw ‘a need for better communication & definition of core indicators 
between Umoyo Network and Sub-grantees to give more comparability from one 
sub-grantee to another’.6
 
4.2.2 Summary findings of LQAS surveys for August/September 2002 
As requested in the SOW, a preliminary study was made of the recent LQAS 
surveys. It should be stressed that no statistical analysis was undertaken. In all 
cases the local partners have done some initial analysis of their own, and though 
it will be interesting to see the results analysed using statistical packages7
, the 
reports and more recent discussions with Rick James in the Institutional 
Evaluation make it clear that in most cases partners have already drawn useful 
conclusions from the data. These conclusions are summarized in the following 
table: 
Agency Key findings 
NAPHAM Overall: positive results except in respect of condom use…… 
Level of HIV Awareness high, but “there is no behaviour change 
among Malawians; 
Increase in use of condoms minimal amongst youth & men but 
apparently higher amongst women. Both availability & knowledge 
of how to use condoms has increased; 
Few people going for HIV Tests; but big increase in no. of youth 
who report going for an HIV/AIDS test 
No change by either men or women about awareness of support 
groups for home based care; 
Increase in awareness of both youth & women (but not men) re 
signs of STIs 
‘Big & Professional People’ found to be more resistant to taking 
HIV tests, acknowledging their HIV status, and more reluctant to 
join NAPHAM 
 
6
Sub-grantee NGOs Management Information Systems – Documentation Report – September 2002 
7
 Final Report due on 30th May 2003
Page 8 NGO Networks Umoyo ID Evaluation Final Report 
DAPP –1 
Blantyre/ 
Chiradzulu 
Dists 
Overall: men appear to have better knowledge of HIV/AIDS issues 
than women & youth 
Lack of knowledge about specific ways in which parents can 
reduce or prevent mother to child transmission of HIV. 
Fall in condom use from 35% in BS to 24% overall, and from 46% 
in BS to 33% now for youth (but 41% of respondents claim to use 
condoms in extra-marital sex) 
Only 40% aware of support groups for home based care (cf 
NAPHAM) 
Need to expand VCT services 
Decrease in stigma and increase in knowledge of positive living 
for those with HIV. 
DAPP-2 
Lilongwe 
Big increases in HIV/AIDS awareness; high awareness of 
mother/child transmission of HIV/AIDS; No increase in use of FP 
by either men or women. Low rate of condom use (max 15% all 
men; 25% youth, 10% women) 
Only 20% of women & 30% of men aware of support groups for 
home based care 
Low knowledge of STI’s other than AIDS 
Low % of respondents going for VCT 
Malamulo 
Hospital 
Major improvements found in quality of post-natal care & female 
awareness both of STIs, HIV & FP issues. 
“100%” of both males & females (15-49)…knew how they can 
avoid contracting HIV/ AIDS & STI’s” 
Trends in condom use ambiguous but remain at about 20% for 
men, have fallen from 15-10% for women. 
Low awareness (27%) amongst men of HIV mother-child 
transmission 
Major declines in many key indicators between 2nd & 3rd LQAS 
surveys explained by loss of key staff in 2001-2. 
Ekwendeni 
Hospital 
High awareness & use of FP (70% of women), with 60% of 
women seeking antenatal care 
Over 70% of both men & women know signs/symptoms of STI’s. 
Over 90% of women aware of HIV/AIDS & HIV test, but only 25% 
use condoms. 
60% of men (claim to) use condoms – a high figure that may need 
checking! 
Adventist 
Health 
Services 
National programme run from 19 clinics: differences in 
perfomance between different areas noted; 
50% of men (much higher rate than other NGOs) use condoms 
but 99% know where to obtain them; 
% of women using modern FP methods has fallen from 69% in 
May 2001 to 38% now, while % of men using same methods has 
remained stable at 54%. Big increase (from 45 – 76%) in mothers 
of babies under 11 months prepared to go for immunization 
NGO Networks Umoyo ID Evaluation Final Report Page 9 
MACRO Programme covers Lilongwe, Blantyre, Mzuzu. (Results not yet 
analysed on a % basis & no comparison made with previous 
surveys) – but survey shows low awareness of support groups; low 
awareness of STIs; low condom use 
World 
Alive 
Ministries 
ICOCA 
Project 
Strong awareness of VCT; only 50% knowledge of support groups; 
amongst men: high awareness of HIV/AIDS, but lower awareness 
of STIs as they affect women; condom use at 45-50%; women: 
increasing awareness of HIV/AIDS; big variation between areas in 
relation to use of FP; increasing awareness of STIs,VCT & HIV 
tests, & but only 20% use condoms compared with 25% in 2000 
Youth: (not covered in previous surveys; 2002 LQAS being used as 
baseline) generally high awareness both of STIs & HIV/AIDS; 
condom use 40-50%; 
Follow up Quarterly Qualitative Assessment planned 
Salvation 
Army 
High awareness of HIV/AIDS by both youth, men & women; but 
more women than men aware of existence of support groups; ‘FP & 
condom indicators show that both men & women can demonstrate 
knowledge of appropriate family planning methods, but they are not 
being used’ 
Over 90% of youth do not take part in traditional, high risk, cultural 
practices, still only 20-35% of youth use condoms; lack of knowledge 
re. Mother-child transmission of HIV. 
4.2.3. Comments on LQAS findings 
The above results show the familiar gap between knowledge and attitudes, which 
in general appear to be improving, and people’s behaviour in relation to avoiding 
the risk of HIV/AIDS infection, which has shown less progress over the last two 
years. This gap between reported changes in knowledge/ attitudes and actual 
changes in practice is a well known phenomenon: educated people in all 
countries are aware of the health hazards involved in excessive drinking or 
smoking, but there is often a very long time lag before (if at all) they alter their 
practice. 
How useful is condom use as a proxy indicator for behavioural change? First it is 
unrealistic to expect people to be entirely truthful about their own sexual 
practices, and over time respondents are likely to adjust their answers to meet 
what they define as the interviewers’ expectations. Even so the answers given in 
relation to condom use which suggest condoms are used on average only about 
35% of the time are remarkably consistent across all the agencies.8
 On the 
optimistic side there is a far higher rate of condom use in areas covered by 
Umoyo Network’s partners than national MDHS figures which suggest that only 
14% of men and 5% of women used a condom during their last sexual 
 
8
 The exception is the figure of 60% quoted in the LQAS survey undertaken by Ekwendeni Hospital 
Page 10 NGO Networks Umoyo ID Evaluation Final Report 
encounter.9
 Aggregate condom use will always be a less useful indicator than 
condom use amongst those groups most at risk –especially youth, sex workers, 
and other people in temporary and extra-marital relationships. It might therefore 
by useful if future LQAS surveys were to focus less on general condom use, but 
focus more on the key behavioural indicators identified by USAID – condom use 
with non-regular partners, the increased age of sexual debut, and a reduction in 
the number of sexual partners. 
From the perspective of an overall impact assessment, many of the factors that 
inhibit behavioural change are outside the control of Umoyo’s local partners. 
These include inadequate investment in education, chronic poverty, increasing 
food insecurity, high levels of migration to towns, and a lack of job opportunities, 
especially for women. Against this difficult background even the modest 
improvements recorded in this LQAS should be seen as major achievements. 
These LQAS findings need to be compared with the summary results of the 
participatory mid-term evaluations conducted between July and November 2001, 
which reviewed more qualitative issues. These showed some improvements in 
services, especially by the CBDA’s, and the introduction of VCT services, but 
they also documented some common challenges – especially weak integration of 
RH at the level of service delivery; too limited a focus on youth, insufficient 
transfer of training knowledge within partner organizations, and inadequate 
behavioural change strategies and IEC materials to support them. 
Since the MTEs a needs assessment of integration has been carried out, with 
training for service provider managers in integration and in facilitative 
supervision, and a follow up of action plans will take place in April 2003. There 
has been a TOT in youth friendly services and the NGOs included this in their 
workplans this last year; all have been awaiting the national sexual and RH 
behaviour change strategy and many more IEC materials have been supplied. 
Rick James has raised transfer of training knowledge within the partner 
organizations and Umoyo staff plan to address this by the development of 
organization capacity building plans in the next funding. 
One consistent finding of both the LQAS surveys and the mid-term reviews is that 
the integration of RH, FP, and HIV/AIDS services (one of Umoyo Network’s 
objectives) will only assist those people who are reached by these integrated 
services. These evaluations point up the need for more outreach work to at-risk 
groups like youth who do not often come to RH or FP clinics; and this in turn 
suggests a need for any future project to have a larger budget to support local 
partners with transport (especially motor bikes) to support this kind of outreach 
work. 
 
9
 Quoted in USAID’s Request for Applications – increased availability & quality of HIV/AIDS related 
support services through NGOs-November 2002 
NGO Networks Umoyo ID Evaluation Final Report Page 11 
4.3. The Impact of Umoyo Network’s Capacity Building Activities 
‘The programmes we are implementing are community-based. We have tried our 
best to get the communities to own these programmes, but if we want to make 
significant changes at community level we need more time.’ 
‘I don’t see any reason why USAID should bring in another contractor (to replace 
Umoyo Network). Its not just about money – its all the support they have given 
us’ 
(Views expressed by Umoyo Network partners) 
As the Executive Summary of the ID Evaluation is attached in Part 2, this section 
highlights a few of the major findings of this evaluation. First it is useful to track 
how Umoyo Network’s capacity building strategy has involved a strong 
combination of quantitative and qualitative investments, as illustrated below: 
 
Type of input Grants disbursement Wide range of Training inputs; 
specific management & OD 
support 
Output Expansion of staff Improvement in numbers and 
quality of staff at different 
levels: e.g. 
(Boards, management, 
supervision, service delivery) 
Improvement in financial 
systems 
Result Expansion of FP/STI/RH/ 
HIV/AIDS/VCT services 
Overall improvement in quality 
of service. 
 
It is difficult in a summative evaluation of this type to document fully the 
expansion of services made possible by Umoyo Network’s grants. However they 
include both an expansion in terms of geographical coverage as well as 
expansion in the range of services offered, especially in relation to HIV/AIDS 
services and VCT. A key point made by local partners in relation to this 
expansion of services has been Umoyo Network’s willingness to fund core costs 
like rent, vehicles, and staff: even well-established local partners often find it 
difficult to mobilize resources for such core costs from other donors. 
Secondly one successful component of the Network’s approach has been its 
pragmatic and flexible approach in responding to the management needs of 
different organizations: examples are the secondment of a VSO volunteer to help 
introduce management systems at MANASO and the recent deployment of a 
local consultant to act as Executive Director of MANET to help resolve some long 
standing management problems. The capacity building work has also been quite 
Page 12 NGO Networks Umoyo ID Evaluation Final Report 
comprehensive, with training not just for staff but also for Board members, 
including, in the case of Ekwendeni Hospital, training for the whole Synod of the 
church. This is in contrast to many similar projects, which focus more efforts on 
training a narrower range of staff. 
However sound this strategy, the current gains must be seen as relatively fragile, 
as the ID Evaluation emphasizes. There are still major governance problems in 
many Boards; and there is both a shortage of people in Malawi with the expertise 
and commitment to serve on a Board and a lasting expectation that Board 
members should be paid ‘sitting allowances’. Also the training and exposure 
received by senior staff involved in the Network have made them more attractive 
to other employers, and by March 2003, with only three months more to go with 
USAID funding, at least one manager had decided to move on. Similar problems 
of staff retention are noted in the Quality of Service Evaluation, and were 
specifically reported in relation to VCT Counsellors working for MACRO and 
amongst CBDA’s at Malamulo. In addition all the local partners lose tragically 
large numbers of staff every year as a result of HIV/AIDS. These factors can be 
seen as external ‘threats’ to the capacity building work largely outside Umoyo 
Network’s control: while staff salaries in the mission organizations are in many 
cases pegged to government rates, the recent flow of funds into HIV/AIDS￾related projects has increased the demand for staff with appropriate skills. 
A further problem discussed in the ID Evaluation is that some local partners felt 
their staff were spending almost too much time in workshops and training 
courses. The implication for Umoyo Network is that in future it should spread 
out formal training over a longer period, but back it up with more frequent 
‘mentoring and coaching’ visits to partners. 
4.4 Monitoring and Reporting on Impact 
Umoyo Network has taken many positive steps to ensure that its impact can be 
documented. There were baseline institutional assessments, baseline LQAS 
studies, mid-term evaluations of all local partners, mid-term LQAS studies, end￾of-project LQAS studies and this final evaluation. In addition there have been 
many consultancies on a wide range of topics. What more could it have done? 
One major finding is that in the first phase much senior management time that 
has gone into reporting to USAID could have been used for more creative 
purposes – especially more support to local partners. All reports need to either 
add to the learning by staff and local partners, or to meet USAID’s accountability 
requirements as a donor. Long reports rarely achieve either purpose. The MIS 
report notes the problems of data utilization within Umoyo Network’s local 
partners, and probably more data has been generated across the whole network 
than can be effectively utilized, especially if the project is closed down in mid￾2003. There is a need to design reporting systems based on the principle of 
NGO Networks Umoyo ID Evaluation Final Report Page 13 
“optimal ignorance”; and to continue the excellent work done in the MIS review to 
map out exactly who needs what kind of information at each level. As argued in 
the ID Evaluation, the Network needs to monitor its impact against a smaller 
number of critical indicators, and it needs to focus more on the impact it is having 
on its local partners. 
These “core indicators” should be worked out with the Umoyo Network’s local 
partners and agreed with USAID at the start of any future project. They are likely 
to involve a combination of capacity building indicators, indicators for improved 
service and coverage, and a few behavioural change indicators which should be 
used for evaluation rather than quarterly monitoring purposes. Quarterly reports 
should report briefly on progress as perceived by the Network’s staff and local 
partners, and changes in the quality and quantity of services. 
4.5. Benefits of a Network 
It is challenging for an evaluation to document the benefits of a Network. There 
are some clear costs – the time that Umoyo Network staff put in to maintaining 
the network, and the time that local partners spend in coming to meetings. And 
many of the benefits appear as ‘subjective’ and hard to measure with 
conventional indicators. However both the ID Evaluation and this overall 
evaluation traced considerable gains achieved by partners as a result of 
networking, many of them with minimal costs. 
One important benefit of a network is psychological. Staff who were previously 
isolated in their own organizations now have an opportunity to discuss common 
problems, find solutions, and hence improve their morale. This is a significant 
factor for organizations involved in what is often difficult and depressing work, 
and a number of partners said they now feel ‘part of a family’. Secondly Umoyo 
Network has helped give individual partners a greater self-confidence and 
credibility in Malawi, with MANASO able to run the World Aids Campaign for 
2002/3, and others able to access more drugs from the MOHP. Further indicators 
worth considering for any future programme in relation to network development 
might include: 
-the development of a common advocacy plan, worked out jointly by Umoyo 
Network staff and its local partners. (See recommendations below) 
-the total amount of non-USAID resources (both in terms of funds and other 
supplies and equipment) which local partners are able to access. 
As regards the overall sustainability of the Network it is too early to expect the 
Network to continue to flourish as it is at present without outside funding. The 
evaluation of any further phase of funding of Umoyo Network will need to review 
Page 14 NGO Networks Umoyo ID Evaluation Final Report 
the probability of the Network being able to survive with significantly less external 
funding. 
4.6. Umoyo Network’s impact on the Quality of Service 
The extent to which the benefits of capacity building and network development 
have translated into improved services on the ground have been explored in the 
Evaluation of the Quality of Services (QOS) undertaken by Lennie Kamwendo 
(see Attachment 2). Overall this evaluation found that the local partners 
‘…have made tremendous progress towards achieving the objectives of raising 
awareness, increasing access and providing quality care in areas of family 
planning, sexually transmitted infections and voluntary counselling for HIV/AIDS. 
Their strengths are in the proposals, which give clear indication of what needs to 
be achieved; good progress made in meeting targets which have been exceeded 
in some cases; evidence of commitment to achieving the objectives; widespread 
rapport with clients; some measure of integration of FP/STI/VCT services; 
commendable impact on individuals, families and communities as evidence by 
rising numbers of clients. The honesty with which negative aspects of care were 
shared, is also commendable.’
This report shows a major improvement in quality of care despite some gaps in 
relation to documenting what QOC means for each member of staff, and further 
problems in relation to infection prevention. In discussions in relation to this 
evaluation it turned out that one reason why infection prevention is sometimes 
not up to standard is due to a lack of water in some facilities. 
The overall conclusion we can draw from the QOC Evaluation is that Umoyo 
Network’s capacity building and network development activities have resulted in 
a considerable improvement in the quality of care offered. Separate from this 
evaluation, the Network has also produced data showing that its partners have 
also been able to increase the numbers of people they reach through their 
programmes.10 
5. Conclusions of the Final Evaluation 
It is important to acknowledge both the strategic importance of what has been 
achieved so far and the potential for the future. Umoyo Network’s local partners 
have potential access to about 2.7 million people, or about one quarter of 
Malawi’s population. If they can continue to provide high quality services in RH, 
FP, and HIV/AIDS in the long term then they should indeed be able to help bring 
about healthier Malawian families. 
 
10 from Umoyo Network report ‘Expansion of Sub-grantee Services’ 
NGO Networks Umoyo ID Evaluation Final Report Page 15 
The SOW for this evaluation posed two specific questions. The first question was 
‘What are the strategies that Umoyo Network utilized with the Sub-Grantees that 
have worked to improve and expand services and strengthened the organizations 
overall? Looking at these strategies – if they worked, why did they work? If not, 
why?’ 
From the analysis above it will be seen that the key reason for the relatively high 
impact of Umoyo Network has been its use of a combination of strategies. The 
impact of grants management, capacity building, and network development 
activities have all been far greater than if any one of these activities had been 
implemented in isolation. The findings of this evaluation show an improvement in 
both the quantity and quality of services offered by Network partners, an 
expansion of their capacity, and the emergence of a strong network. 
However the relatively short time scales have put pressure on the project to show 
immediate results in three broad areas where they are notoriously hard to 
achieve in a short timescale – the capacity development of local NGOs, the 
formation of a sustainable network, and key behavioural changes which will slow 
down the spread of HIV/AIDS. The ID Evaluation notes that capacity building and 
network development are long term programs requiring in USAID’s own judgment 
a consistent and continuous investment over 5-10 years. While Umoyo started in 
July 1999, the current Programme Manager only arrived in November 2000, and 
there has been considerable uncertainty in the last 6 months due to the expected 
ending of the contract in early 2003 (now extended until June 2003). Given the 
brief (two-year period) between her arrival and the start of uncertainties relating 
to the closure of the current contract the present PM and her staff have done an 
extraordinary job, the impact of which on the local partners is well documented in 
the ID Evaluation. The challenge now is to increase the local ownership of 
Umoyo Network itself, and in the next phase there will be a significant 
management job to be undertaken to make the Umoyo Network staff a more 
cohesive team. 
In relation to strategies, the option of making Umoyo Network a local NGO in its 
own right was reviewed and rejected at an early stage. This seems the correct 
decision in that there is already a surplus of donor-created local NGOs in Malawi, 
but many fewer successful networks. 
The second question asked in the SOW was What kind of unintentional events 
have helped or hindered the achievements of the goal?’ The first point to make 
is that Umoyo Network’s local partners have been implementing their 
programmes against a background of drought, food insecurity, and increasing 
rural poverty, all of which make behavioural change more difficult. The second 
contextual issue noted in this evaluation (especially in section 4.3 above) is 
Malawi’s overall shortage of skilled health workers of all levels, especially in rural 
areas, which makes it difficult for Umoyo Network’s local partners to retain key 
staff. 
Page 16 NGO Networks Umoyo ID Evaluation Final Report 
 
6. Recommendations for Umoyo Network
6.1.Capacity building 
6.1.1 Reduce reporting burdens but increase internal monitoring of capacity 
building 
6.1.2 Bring the same rigour to the M & E of capacity building has been 
introduced by the use of the LQAS method for M & E at the community 
level. This requires a more consistent approach to organizational 
assessments, followed by a process of defining and monitoring 
appropriate indicators for capacity building in close consultation with local 
partners. The regular review of these indicators then needs to become a 
more central part of discussions with local partners 
6.1.3 In view of the pressures on local partners’ current staff training courses 
should be carefully prioritized and spread out over a longer period, 
allowing more time for partners to ‘internalise’ the training. Formal training
should be backed up with more frequent ‘mentoring and coaching’ visits to 
partners. 
6.1.4 Umoyo Network should develop an electronic database of information on 
its local partners which can be accessed by all its staff. This should 
provide all key data on the NGOs, and the different capacity building 
activities including notes on all visits made. 
6.2 Advocacy 
6.2.1 One logical response to the various important external constraints has 
been the increasing amount of work that Umoyo Network is now 
undertaking at national level with the NAC and MOHP in relation to these 
issues. This activity is likely to be of increasing importance in the future, 
but there is a risk that all the effort committed will not in fact result either in 
improvements in the capacity of partners or in improved services on the 
ground. To avoid this we propose that the Network should develop and 
implement an advocacy strategy in close consultation with its local 
partners. As the Network develops, more of this advocacy work should be 
undertaken by the local partners, rather than by Umoyo Network staff.
6.2.2 This advocacy strategy needs to spell out the policy and practice changes 
that Umoyo Network is aiming for as a result of its interaction with different 
government agencies (e.g. MoHP, NAC, and other donor agencies), and 
should propose an indicative time schedule. 
6.3 Monitoring and Evaluation 
NGO Networks Umoyo ID Evaluation Final Report Page 17 
6.3.1 For any future programme make a clearer distinction between the M & E 
requirements of local partners and those of Umoyo Network. In order to 
meet the information needs of Umoyo Network, local partners should be 
encouraged to collect consistent, and comparable data at different points 
in time on a small number of indicators, rather than data on such a wide 
range of indicators. 
6.3.2. In any future programme it would be desirable to divide up the M & E 
functions between: 
-Monitoring of Umoyo Network’s own work, making more use of capacity 
building indicators; and commissioning, implementing, and following up 
Mid-term Evaluations. 
-Supporting the M & E efforts of partners, including both LQAS and 
helping them design appropriate M & E systems. 
6.4 Grants management 
6.4.1. Evaluate a sample of small grants distributed 6 months after 
disbursement, to see the impact of the funding. Use the results of this 
internal evaluation to fine-tune the policy in relation to small grants. 
6.4.2 Encourage longer periods of funding for well-established local 
partners - probably for two years at a time, culminating in an LQAS & mid￾term evaluation. The mid-term evaluations should review the amount of 
funds which local partners have been able to access from non-USAID 
sources. 
6.5 Quality of services 
• All facilities need to have ready copies of all protocols and standards 
related to FP, STI and VCT management and use these to measure 
quality of care. 
• Infection prevention practices need to be strengthened in all areas of 
health care, with adequate water supply for all clinics. 
• Facilities lacking the necessary transportation to implement outreach 
clinics should receive assistance with transport to enable access to the 
poorest clients 
• Potential FP clients should receive a full medical assessment before 
CBDAs provide them with family planning methods 
Page 18 NGO Networks Umoyo ID Evaluation Final Report 
• Facilities should strive for full integration of services so that maximum 
compliance can be achieved. This can also reduce stigma, which deters 
many clients from seeking care, especially VCT. 
7. Recommendations for USAID-Malawi: 
7.1 Work to more realistic timescales 
Capacity building by contract is unlikely to work. Continuity of funding is vital for 
any kind of long-term impact, including the retention of key staff both within 
Umoyo Network itself and within the partners. USAID in Malawi should consider 
moving towards a longer-term ‘partnership’ approach with agencies like Umoyo 
Network in which funding in principle is agreed for a 4-5 year period, with a 2-
year mid-term review and an annual review if required. The problem may be less 
the time-scales themselves than the uncertainties that result from delays in 
decisions about future contracts. 
7.2 Try to increase flexibility
Effective work in HIV/AIDS prevention requires a highly flexible and multi-sectoral 
approach. The more restrictions put on the type of activities that can be funded, 
the less effective capacity building will be. Some of the key needs identified 
during these evaluations of Umoyo Network include increased budgets for drugs, 
food for PLWHAs, their carers and families, and more transport to support 
community level work, especially in rural areas. 
7.3 Reduce reporting requirements 
As discussed in the report too much scarce management time, both of Umoyo 
staff and of its local partners, has gone into reporting to USAID, and the resulting 
reports have become rather long. At the start of any future contract USAID 
should agree with the contractor its minimum reporting requirements, and clearly 
specify the maximum length for reports. Normally a donor’s overall accountability 
requirements are best met by a combination of formal reporting, informal 
monitoring by donor staff, and evaluations. 
7.4 Agree core impact indicators for both behavioural change and capacity 
building. 
Both USAID and Umoyo Network would find monitoring easier if at the outset 
they were able to agree a small number of key indicators to be monitored 
throughout the contract. These indicators should include both capacity building 
indicators and indicators of behaviour change. 
NGO Networks Umoyo ID Evaluation Final Report Page 19 
ATTACHMENT A 
Evaluation of Umoyo Network’s Capacity Building - Rick James 
EXECUTIVE SUMMARY 
Umoyo Network has been supporting 11 Malawian NGOs with funding and 
capacity-building in order to improve the scope and quality of their work in 
reproductive health and HIV/AIDS. Umoyo Network has invested heavily in 
attempting to build the capacity of its sub-grantees, indeed this is one of the main 
differences of approach between Umoyo Network and other USAID-funded 
projects in the past. This is an innovative approach, but one that requires 
considerable analysis to find out whether such investment was worthwhile. 
An evaluation of Umoyo Network’s capacity-building work with sub-grantees was 
undertaken in October and November 2002 by INTRAC in order to find out: 
1. the perceived impact on the Sub-Grantees of the Umoyo Network investment 
in Institutional Development over the last three years (questions of 
accountability) 
2. what learning can be gained from this experience in order for Umoyo Network 
/ USAID improve its institutional development support to Malawian NGOs in 
the future. 
It is important to point out that this evaluation took place at a time of USAID 
funding insecurity and this may have affected the responses. 
In order to answer questions of impact, it was necessary to answer three 
related questions: 
• To what extent has capacity of sub-grantees increased over the period of 
Umoyo Network support? 
• How has this affected beneficiaries and other stakeholders? 
• To what extent can these changes be plausibly associated with Umoyo 
capacity-building inputs? 
The findings from the different data gathering methodologies point to a very 
significant increase in sub-grantee capacity in the last three years. Sub-grantees 
estimate their overall capacity has increased by almost 50%. Although the 
starting base was very low and the sub-grantees may have overestimated the 
extent of their change, there is a clear improvement in capacity, particularly in 
organisational areas. The main areas of change are: 
Page 20 NGO Networks Umoyo ID Evaluation Final Report 
• Organisational and Programme Strategy 
• Governance 
• Leadership 
• Management skills 
• Team development 
• Structural changes 
• Planning, Monitoring and Evaluation (PME) systems 
• Financial Systems 
• Human Resource (HR) Policies 
• Staff technical skills 
• Funding diversification 
• Profile 
• Networking and Collaboration 
This evaluation largely confined itself to analysing the impact of capacity-building 
on sub-grantees as other evaluation exercises are going to look at the all￾important question of whether these organisational changes actually resulted in 
benefits at community level. The anecdotal evidence gained from this work, 
would point out very real benefits at community level both in terms of programme 
continuity (avoiding the interruption of community programmes through internal 
problems) as well as a shift towards more participatory and empowering 
approaches in service provision to communities. There were also significant 
benefits from Umoyo Network working closely with government departments 
(NAC and RHU), which brought a practical NGO perspective into policy 
development work, as well as linking NGOs with government much more closely. 
The extent to which these changes in capacity can be directly attributed to 
Umoyo Network is impossible to ascertain as the variable of Umoyo Network 
capacity-building cannot be separated from the many other factors that also 
influenced NGO capacity over that same period. It is possible, however, to say 
that these changes in sub-grantee capacity can be ‘plausibly associated’ with 
Umoyo capacity-building inputs. Not only did sub-grantees often attribute the 
change to Umoyo inputs, but also Umoyo capacity-building inputs featured as 
key events when sub-grantees presented a history of their organisation. 
As well as providing some accountability for resources expended, the evaluation 
also sought to systematically reflect on the capacity-building undertaken and 
highlight key learnings for Umoyo Network. 
The main methods used by Umoyo to build capacity in sub-grantees were: 
• Staff Advice 
• Training (both technical and management) 
• OD Consultancy 
• Monitoring and evaluation processes 
• Networking meetings 
NGO Networks Umoyo ID Evaluation Final Report Page 21 
The key variables that determined whether or not the inputs actually built 
capacity were: 
1. Sub-Grantees Taking Responsibility (owning the need for change) 
2. Sub-Grantees Making a Willed Response (in particular leadership openness 
to change) 
3. Sub-Grantees Having the Resources to Respond (financial and time) 
4. Umoyo-Sub-Grantee Relationship 
5. Overall Umoyo Network Capacity-Building Strategy 
6. Quality of Specific Capacity-Building Support (by Umoyo staff and outside 
consultants/trainers) 
7. The Context (social, economic and political) 
Umoyo Network is clearly only able to influence some of these, while others 
depend on the sub-grantees and others the wider context. 
Conclusions 
Overall Umoyo Network is a very successful capacity-building programme in a 
very difficult context of Malawi. Umoyo Network has invested heavily in capacity￾building, (particularly in the areas of management and organisation) and, even in 
the short-term, there have been considerable changes in sub-grantee capacity 
that can be plausibly associated with Umoyo inputs. Programme performance of 
sub-grantees in communities appears to have improved noticeably too. As 
USAID own policy guidelines state, ‘PVO institutional ventures are not ‘cheap’ 
investments…, but if they engender even partially successful local institutions, 
they are cost effective in the long-run, given the continued flow of benefits as 
costs gradually decrease’11. 
Recommendations 
The recommendations below are shared by sub-grantees, staff and senior 
management. There is remarkable coherence and overlap. The major message 
is that Umoyo should keep going in the same direction as it is an extremely good 
quality programme in very difficult circumstances. The recommendations of this 
evaluation are that Umoyo should: 
Maintain: 
• Umoyo Network’s Overall Approach 
• Management and Organisational Change as a key capacity-building area 
• Holistic Approach 
• Mix of Methods and Resource Support 
 
11 Accelerating Institutional Development, USAID PVO ID Evaluation Series, p 3 
Page 22 NGO Networks Umoyo ID Evaluation Final Report 
Improve by developing a more coherent and focused capacity-building 
strategy: 
• Clarify Conceptually 
• Individualise through a Participatory Capacity-Building Planning Process 
• Prioritise and Sequence 
• Follow-Through – More visits 
• Fine-Tune Specific Capacity-Building Methods 
• Shift the Relationship from a Sub-grantee to a partner 
While the external programme recommendations can be seen as fine-tuning, 
many of the suggested improvements have been around for some time. An 
important question is why they have not been implemented before. The answer 
to this question probably lies in the OD issues of Umoyo itself. It is these internal 
issues that represent the greatest threat to Umoyo’s sustained performance in 
the future. 
Umoyo should address the internal implications and tensions: 
• Re-structure to fit better with strategy 
• Develop systems to monitor and evaluate the capacity-building 
• Improve staff motivation and turnover through greater ownership and 
responsibility 
• Review management culture and style 
USAID also should recognise that their demands for quick results in a very short￾time frame is an important factor in exacerbating these issues. 
NGO Networks Umoyo ID Evaluation Final Report Page 23 
Attachment B. 
Evaluation of Quality of Services offered by Umoyo Network Partners 
(Lennie Kamwendo – March 2003) 
EXECUTIVE SUMMARY 
1. Introduction 
Four Umoyo Network Sub-Grantees were evaluated to determine the quality of 
care that these facilities provide. Field visits were made to Ekwendeni hospital, 5 
Adventist Health Service facilities, Malamulo hospital and the MACRO centres in 
the three regions of Malawi. 
2. Methodology 
Data were collected using separate focus group discussions with service 
managers and providers; structured observation of health care provision and 
random exit interviews of family planning clients, patients with sexually 
transmitted infections and clients who had come for voluntary counselling and 
testing. Data were collected over a period of twelve days and the major areas of 
focus were availability of project proposals and knowledge of such proposals by 
service providers as this would form the basis for service provision. Information 
regarding strategies used for achieving the targets, quality of care, staff 
performance appraisal and training was also sourced. 
3. Findings 
 
• All facilities had proposals that clearly stated the program objectives and 
targets to be achieved. However, not all service providers had working 
knowledge of the objectives 
• Strategies in place include some level of service integration, use of IEC 
with involvement of community leaders to raise awareness, increase 
access and provide care in the areas of family planning, sexually 
transmitted infection management and voluntary counselling and testing. 
Some facilities have well established youth clubs. Drama, the electronic 
and print media have also been used with commendable impact on 
individuals, families and communities. 
• Quality of care was evaluated at program and operational levels. Sub￾grantees have made tremendous progress towards achieving program 
targets and these have been exceeded in some cases. Structured 
observation and exit interviews indicated satisfactory provision of care as 
evidenced by short waiting time by clients, appropriate attitudes of service 
providers as well as the ability to attract large numbers of clients. 
Page 24 NGO Networks Umoyo ID Evaluation Final Report 
Voluntary counselling was exceptionally good and appreciated by clients. 
However, it was also clear that clients do not really know the difference 
between good and bad quality care because even those who had never 
been physically examined in the family planning units indicated that they 
were satisfied with the quality of care. 
However, infection prevention practices were below standard as were the 
procedures for family planning. Lack of adequate space as well as 
laboratory facilities in some centres compromise quality. Shortage of staff 
and means of transportation also prevent meaningful implementation of 
outreach clinics. 
 
• Staff performance appraisals are only done at the end of the year. There 
are no structured supervisory activities and checklists are not available 
except at MACRO Mzuzu centre. Operational level determination of quality 
of care is therefore compromised. 
• Many service providers have undergone relevant training to enhance 
performance and improve quality of care. 
4. Recommendations 
• It is necessary that all service providers have a working knowledge of the 
program objectives so that they can have consistent guidance. 
• All facilities need to have ready copies of all protocols and standards 
related to FP,STI and VCT management and use these to measure quality 
of care. The Reproductive Health Guidelines should be read and followed 
for client management. The protocols should be displayed on the walls for 
easy reference. 
• Infection prevention practices need to be strengthened in all areas of 
health care 
• Facilities lacking the necessary transportation to implement outreach 
clinics should be considered for such assistance to enable access to the 
poorest clients. 
• Opportunity needs to be taken to assess all FP clients before CBDAs can 
provide family planning methods 
• Facilities should strive for full integration of services so that maximum 
compliance can be achieved. This can also reduce stigma, which deters 
many clients from seeking care, especially VCT. 
NGO Networks Umoyo ID Evaluation Final Report Page 25 
• Supportive supervision needs to be implemented at the operational level in 
order to improve the quality of care given as well as staff morale. 
• Training programs need to be planned and implanted more often in order 
to give opportunities to more service providers to be trained. 
Page 26 NGO Networks Umoyo ID Evaluation Final Report 
Appendix 1: List of documents consulted and people met 
Documentation: 
• Umoyo Network Partners: Mid Term Evaluation Reports - 2001 
• Sub-grantee NGOs Management Information Systems, Documentation 
Report Sept 2002. 
• LQAS Reports September 2002 
• Network Development Report – Darcy Ashman (NGO Networks Washington,) 
Dec 2001 
People met: 
• Umoyo Network Staff 
• Rick James, INTRAC, Blantyre 
• Lennie Kamwendo, Consultant 
• DAPP, TSA, and Malamulo Hospital staff 
• Francina Nyirenda, MANASO 
• Kumbukani Black, NAPHAM 
• Florence Chipungu, AHS 
• William Vargas, NGO Networks for Health, Washington 
• Elise Jensen – USAID-Malawi 
• Roy Hauya – NAC, Lilongwe 
• Justin Opoku – Field Office Director (SC-US) 
NGO Networks Umoyo ID Evaluation Final Report Page 27 
Appendix 2 – Scope of Work 
Final Evaluation of Umoyo Network 
I. GENERAL INFORMATION 
Project Name: Umoyo Network 
Project Number: HRN-A-00-98-00011-00 
Field Officer Director: Justin Opoku 
Program Manager: Carrie Osborne 
Donor: USAID 
Consultant: Hugh Goyder 
Contractor/Grantee: Save the Children, USA 
Life of Project Funding: 1999-2003 
Nature of work: Lead the Final Evaluation of Umoyo 
Network. 
Time frame: Up to 10 working days (Wednesday 27th
November to Saturday 8th
December2002) and 14 further working 
days from 3-14 March 2003 
2. INTRODUCTION 
Since 1999 Umoyo Network has been working with 11 Malawian NGOs to 
provide grants and technical assistance to improve the scope and quality of their 
work in reproductive health and HIV/AIDS. All of the NGOs have had the 
opportunity for institutional development and capacity building in governance, 
management and technical skills. 
3. PURPOSE 
The Final Evaluation of the USAID funding to Umoyo Network seeks to: 
1. Document the accomplishments of Umoyo Network against its original 
objectives; 
and to establish: 
2. What are the strategies that Umoyo Network utilized with the Sub￾Grantees that have worked to improve and expand services and 
strengthened the organizations overall? Looking at these strategies – 
if they worked, why did they work? If not, why? 
Page 28 NGO Networks Umoyo ID Evaluation Final Report 
3. What kind of unintentional events have helped or hindered the 
achievements of the goal? 
Umoyo Network now requires a consultant to draw together the findings of 
various studies being undertaken towards the Final Evaluation of Umoyo 
Network. These studies include: 
• An Assessment of the Institutional Development Program being 
undertaken by Rickk James of INTRAC; 
• An Assessment of the changes since the mid-term evaluation in the quality 
of clinical services provided by four of the Sub-Grantees (MACRO, AHS, 
Malamulo, Ekwendeni); 
• A Quantitative Survey, using LQAS, of the changes in knowledge, 
attitudes and behaviors at community level to measure the outcomes and 
impact of the work of the Sub-Grantees compared to the baseline survey 
(draft report complete); 
• A Report on the Results achieved (from the Management Information 
Systems of the Sub-Grantees, reported in Quarterly and Annual Reports); 
4. METHODOLOGY 
The Consultant will: 
• Review the indicators set to measure the project; 
• Review assessment and evaluation reports and other documentation; 
• Meet with the other external evaluators to discuss their findings, or plan 
their input; 
• If required interview the Sub-Grantee Project Managers/ Directors and 
other senior staff; 
• If required interview Umoyo Network and PVO Partner staff; 
• Review the Reports from the other evaluations 
• Produce a first draft Final Evaluation Report by 12th March 2003, of no 
more than 20 pages plus appendices and to produce a Final Report within 
5 days of receipt of feedback on the Draft.