MID-TERM ASSESSMENT PROGRAMME ON IMPLEMENTING INTERNATIONAL STANDARDS FOR SUPREME AUDIT INSTITUTIONS AMONG DEVELOPING COUNTRIES IN MIDDLE EAST AND NORTH AFRICA REGION FOR INTOSAI DEVELOPMENT INITIATIVE FINAL REPORT | 28 JULY 2017 2 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Table of Contents Abbreviations ............................................................................................................................................... 4 Quality Assurance ........................................................................................................................................ 5 Executive summary ...................................................................................................................................... 6 1. Introduction .......................................................................................................................................... 7 1.1 Purpose of the mid-term review ...................................................................................................... 7 1.2 Methodology of the mid-term review ............................................................................................... 8 1.2.1 Scope .............................................................................................................................................. 8 1.2.2 Review criteria ................................................................................................................................. 8 1.2.3 Data-collection methods .................................................................................................................. 9 1.3 Structure of the Report .................................................................................................................... 9 2. Relevance.......................................................................................................................................... 10 2.1 Review questions .......................................................................................................................... 10 2.2 Continuing relevance of the objectives .......................................................................................... 10 2.3 Logic of the Design and its Results framework ............................................................................. 11 2.4 Complementarity and overlap with other development assistance to the beneficiary SAIs ........... 12 2.5 Realism of the assumptions underlying the Programme ............................................................... 13 2.5.1 Assumptions at the level of the participating SAIs ......................................................................... 13 2.5.2 Assumptions at the level of IDI and other stakeholders ................................................................ 14 2.6 Relevance of adjustments to the initial design of the Programme ................................................. 15 2.7 Risk assessment and mitigation measures ................................................................................... 16 2.8 Conclusions on relevance ............................................................................................................. 16 3. Efficiency ........................................................................................................................................... 18 3.1 Review questions .......................................................................................................................... 18 3.2 Cost efficiency of the delivery ........................................................................................................ 18 3.2.1 Project management ..................................................................................................................... 18 3.2.2 Workshop costs ............................................................................................................................. 19 3.3 Comparative assessment against other implementation methods ................................................ 19 3.4 Timeliness of output delivery ......................................................................................................... 20 3.5 Quality of the output delivery ......................................................................................................... 22 3.5.1 Feedback from participants ........................................................................................................... 22 3.5.2 Feedback from resource persons .................................................................................................. 22 3.5.3 Feedback from the case studies ................................................................................................... 23 3.6 Conclusions on efficiency .............................................................................................................. 24 4. Effectiveness ..................................................................................................................................... 25 4.1 Review questions .......................................................................................................................... 25 4.2 Achievement of the objectives ....................................................................................................... 25 4.2.1 Intermediate outcomes .................................................................................................................. 25 4.2.2 Final outcomes .............................................................................................................................. 27 4.3 Value added of 3i-Programme in meeting the ISSAIs ................................................................... 28 3 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD 4.4 Conclusions on effectiveness ........................................................................................................ 29 5. Conclusions and recommendations ................................................................................................... 30 Annex 1 Intervention logic .......................................................................................................................... 33 Annex 2 List of documents ......................................................................................................................... 34 Annex 3 List of Interviews .......................................................................................................................... 35 Annex 4 Case study results ........................................................................................................................ 36 Annex 5 Questionnaire results ................................................................................................................... 49 4 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Abbreviations ARABOSAI : Arab Organization of Supreme Audit Institutions CA : Compliance audit CBC : Capacity Building Committee DAC : Development Assistance Committee FA : Financial audit 3i : ISSAI Implementation Initiative IDI : INTOSAI Development Initiative INTOSAI : International Organisation for Supreme Audit Institutions ISSAI : International Standards for Supreme Audit Institutions PA : Performance audit PSC : Professional Standards Committee SAI : Supreme Audit Institution SAI PMF Supreme Audit Institution Performance Measurement Framework USAID : United States Agency for International Development 5 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Quality Assurance 1. FPO World has been commissioned by IDI to carry out the “Mid‐term Assessment of  Programme on Implementing International Standards for Supreme Audit Institutions  (ISSAIs) among developing countries in Middle East and North Africa (MENA) region”  2. On behalf of FPO World, this final report was prepared by the following team:  Name   Title  Contact  Ferdinand Pot (team leader)  Director FPO World  Ferdinand@FPO.World Sura Khuzai  Independent consultant  surakhuzai@hotmail.com 3. The Terms of Reference requires the following for the Final Report: “On completion of  the mid‐term assessment, the assessor will prepare a draft mid‐term assessment report  containing details of the mid‐term assessment methodology and process, mid‐term  assessment findings substantiated by evidence, lessons learned, conclusions and  recommendations for further advancing the programme objectives and grant  objectives. After discussions and feedback from the IDI (in consultation with US AID),  the final mid‐term assessment report will be prepared.  The deliverables are to be:  ‐ written in clear and concise English;   ‐ the mid‐term assessment report should include an executive summary;  ‐ besides conclusions, the mid‐term assessment should provide recommendations for  the way forward;  ‐ the mid‐term assessment report should also include a list of consultations /  interviews conducted, documents/literature reviewed, surveys conducted , etc. (as  annexes)  ‐ all deliverables should be submitted via e‐mail to the IDI contact person.”  4. The final report has been quality assured externally by Mr. Martin Sinclair. The assessor  is familiar with IDI and the 3i‐programme as he represented  the UK National Audit  Office on the IDI Board for several years until 2015. However, he has had no  involvement in the approval or design of this Programme in either his previous or  current occupation.   5. The objective of the quality assurance review was to ensure that:   The final report is consistent with the approach described in the Inception Report;   There is a valid link between the findings/evidence, conclusions and  recommendations;   The report is written clearly and with correct use of English language;   The comments of the Client are adequately reflected in the report.  6. All significant matters raised during the quality assurance process have been addressed  in this version of the Final Report  Prepared by: Martin Sinclair, independent expert  Date: 25 July 2017  6 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Executive summary This mid-term review has assessed the performance of the ISSAI Implementation Initiative (3i) Programme in Arabic against three of the OECD-DAC criteria including relevance, efficiency and effectiveness. The 3i Programme in Arabic is focused on Supreme Audit Institutions (SAIs) belonging to the ARABOSAI region of INTOSAI covering countries from the Middle East and North Africa (MENA) and aims to support these SAIs in moving from current audit practices to ISSAI-based audit practices by strengthening their professionalism and understanding of the ISSAIs. This Programme followed from INTOSAI’s adoption of the ISSAIs in 2010 and the consequent mandate given to IDI to support ISSAI implementation in the SAIs of developing countries. The 3i-programme for Arabic-speaking countries started in 2014 and will continue until December 2019 (including the ISSAI Based Performance Audit of Institutional Framework for Fighting Corruption under the SAI Fighting Corruption Programme). The objective of the mid-term review is to give independent assurance that the programme is on track to achieve its objectives and to provide recommendations on any bottlenecks in the further implementation. The reviews shows that the work is largely progressing in line with the schedule as defined in the Results Framework of the 3i Programme. Based on a sound logical framework that has been successfully tested before in Anglophone countries, IDI has rolled out the 3i-Programme to Arabic speaking in an efficient manner. Efficiency is observed in the cost effective use of the Programme’s resources. The review has not identified room to save on the costs of project management and the organization of workshops. We conclude also that an alternative delivery method based on long term in-country presence of external experts would be significantly more costly. The 3i-delivery method makes ample use of SAIs own resources through resource persons and support in kind and is able to serve a large number of countries within a relatively tight budget. Furthermore, the Programme is being implemented efficiently evidenced by the timeliness and the quality of the output delivery. Both the beneficiaries and the professional cadre are highly appreciative of the quality of the workshop sessions and the supporting guidance materials. The participation of SAIs from ARABOSAI’s developing countries is higher than anticipated. This is reflected in the number of SAIs that have signed the Statement of Commitment, the use of iCAT tool and the ISSAI Implementation Handbooks. Furthermore, more staff than anticipated have completed the ISSAI Certification Programme. Based on the current progress, it is expected that also the delivery of ISSAI-based pilot audits in the domain of financial audit will be delivered with some delay in 2017. The pilot audit in compliance and performance audit are still on schedule and expected to be completed in 2017 and/or 2018. Our overall conclusion is that IDI has implemented the project successfully to the mid-term review point and is on track to deliver the majority of the agreed outputs. These outputs have raised awareness about the ISSAIs and the implications for existing practices among the participants of the programme. As the project moves to the next phase, a key challenge will be to ensure that the 3i-programme participants can flourish within their home SAIs and can effect change by actually conducting ISSAI-based audits. Here, the Programme is still facing some challenges. The relatively low number of SAIs participating in pilot financial audits is an issue of concern. To successfully achieve the final objective, the report recommends a number of improvements relating to the formulation of the logical framework, the measurement of the effectiveness of the Programme and the quality assurance of the key outputs. ISSAI implementation is a long term process. It is unrealistic to expect that by the end of the programme period the SAIs in the region will have fully converted their audit practices to ISSAI-compliant methods. For this reason, it is suggested that the cooperating parties IDI and USAID, together with the SAIs in the region, give early thought to what more can be done on ISSAI implementation into the 2020s. In this regard, the review team noted that IDI has started the 3i Phase-II programme to maintain the momentum on ISSAI adoption and have defined ISSAI implementation as a long term work stream in its strategic plan. 7 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD 1. Introduction 1.1 Purpose of the mid-term review The focus of this mid-term review is the ‘Programme on implementing International Standards for Supreme Audit Institutions (ISSAIs) among developing countries in Middle East and North Africa (MENA) region (further referred to as the 3i programme in Arabic in the remainder of this report). This Programme followed from the International Organisation of Supreme Audit Institutions (INTOSAI) adoption of International Standards for Supreme Audit Institutions (ISSAI) in 2010 and the consequent mandate given to IDI to support ISSAI implementation in the SAIs of developing countries. After the launch of the 3i-programme in 2012 in English-speaking regions, IDI initiated the 3i-programme for Arabic-speaking countries in 2014. The main objective of the Programme is to support SAIs in the ARABOSAI region in moving from current audit practices to ISSAI-based audit practices by strengthening their professionalism and understanding of the ISSAIs. Within this aim, the Programme is composed of five elements:  Development of public goods;  ISSAI certification programme;  ISSAI knowledge network / community of practice;  ISSAI-based cooperative pilot audits;  SAI level ISSAI implementation start-up. A logical framework of the Programme is included in Annex 1. The 3i-Programme in Arabic is funded by USAID with a grant of USD 500,000 and a similar co-share contributions from IDI as per the agreement signed on 27 September 2013.1 The Programme currently covers 16 SAIs in the ARABOSAI region. The countries can be divided in three categories based on the source of funding for the Programme activities (see Table 1). Participation of the SAIs of Algeria, Egypt, Iraq, Libya and Morocco are funded by the grant from USAID. The SAIs in Gulf countries fund their own participation and the rest of the SAIs are supported by IDI from its basket fund. Table 1 Participating SAIs USAID-funded Self-funded (Gulf-countries) Other (IDI basket fund) Algeria Bahrain Mauritania Egypt Kuwait Sudan Iraq Saudi Arabia Palestine Libya Qatar Lebanon Morocco Oman Syria Jordan2 Tunisia3 Yemen4 The purpose of this independent mid-term review is to assess the achievement as to date of the agreed programme targets and to identify lessons learned for future programme implementation towards attaining its objective. The mid-term assessment will be used to inform the work plan design for the remainder of the 1 In 2015, the grant agreement was amended. USAID funding was increased to USD 2,000,000 and the scope of the project was broadened to include support for SAI’s on engagement with stakeholders and fighting corruption in addition to the original focus on ISSAI implementation. Early in 2017, however, USAID revised the budget to $1 million following a review of their priorities. 2 Participation of Jordan was initiated in 2016. 3 The participation of Tunisia was funded by USAID in the years 2014 and 2015. As per 2016, funding is provided from the IDI-basket funds. 4 Yemen has signed the cooperation agreement, but did not actively participate in the programme. 8 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD activity, and to improve management and communications between IDI, USAID and other donors and development partners. 1.2 Methodology of the mid-term review 1.2.1 Scope Object of the review The mid-term review has covered the implementation of the 3i-programme only and not the activities related to programmes on SAI Engagement with Stakeholders and SAI Fighting Corruption. Time period The mid-term review has covered the 3i-Programme in ARABOSAI region from 2014 to end of the year 2016. During this period, the Programme carried out the following activities: Table 2 3i-Programme’s activities covered by the mid-term review Activity Date Venue 3i Management Workshop 24-26 June 2014 Casablanca Morocco 3i Product Adaptation Meeting 4-20 March 2015 Dubai, UAE 3i Product Adaptation Meeting (Add on for Financial Audit) 7-18 April 2015 Morocco 3i Product Adaptation Meeting (Add on for Compliance Audit) 8-11 May 2015 Tunisia Online selection test 29 April 2015 Community of practice October 2015 Translation of iCATs into Arabic 2015 3i ARABOSAI ISSAI Implementation Workshops 26 May-12 June 2015, Morocco iCAT review Meetings & Workshops on facilitating ISSAI Implementation for financial, performance and compliance audits 16-28 November 2015, Kuwait 3i ARABOSAI Product Adaptation Meeting on ISSAI based CA of procurements 8 to 18 May 2016, Khartoum, Sudan IDI-ARABOSAI Workshop on ISSAI based Compliance Audit of procurement 18 to 19 September 2016 Amman, Jordan FA Planning Workshop in November ? 21-25 November 2016 Rabat, Morocco 1.2.2 Review criteria The review is based on three out of the five OECD-DAC standards for development evaluations: relevance, efficiency and effectiveness. The two other DAC criteria ‘impact’ and ‘sustainability’ are recognised in the assessment of the relevance of the programme. We will focus on whether the project design has sufficiently considered ‘impact’ and ‘sustainability’. The review will not provide, however, a judgment on impact and sustainability as programme implementation is still ongoing and a full assessment of these criteria cannot be supported by adequate evidence at this stage. For each of the evaluation criteria, specific review questions have been formulated. These questions are referred to in Chapters two, three and four. 9 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD 1.2.3 Data-collection methods The mid-term review made use of document analysis and stakeholder interviews. This work was in two parts. We reviewed documents issued by IDI and interviewed members of their staff.. Second, the review examined the experience of seven participating SAIs - Morocco, Iraq, Sudan, Palestine, Mauritania, Libya and Tunisia.5 More details on the interviewees and documents are included in Annex 1 and Annex 2. The results of the seven case study SAIs are reported in Annex 3. In addition, we conducted a survey of the Programme’s ‘resource persons’ to grasp their appreciation on various aspects of the efficiency and effectiveness of the interventions. Resource persons were regional ISSAI experts from the subcommittees and ISSAI mentors from the region. Mentors were selected based on their experience with capacity development, ISSAIs and audit practices in their SAIs. For the 3i￾Programme for SAIs in the ARABOSAI region, IDI has selected 19 resource persons mostly employed by ARABOSAI’s member SAIs. The survey was sent to 13 resource persons who have participated in all four events of the programme including the Product Adaptation Meeting, the ISSAI Implementation Workshops, the iCAT Review Meeting and the Workshop on Facilitating ISSAI Implementation. The response rate was 100 per cent. The results of the questionnaire responses are included in Annex 4. 1.3 Structure of the Report This report is structured in three main chapters, covering ‘relevance‘, ‘efficiency’ and ‘effectiveness’. In each chapter we describe in detail the evaluation criteria used, our conclusions and the evidence base used to support these conclusions. The final chapter takes the conclusions together and suggests recommendations for the consideration of IDI in the further implementation of the programme. 5 The SAIs of Lebanon and Algeria were also approached to participate. The review team was not able to get into contact with the SAI of Lebanon; the SAI of Algeria was available, but only after the review team closed data collection. 10 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD 2. Relevance 2.1 Review questions The relevance criterion is concerned with the project design. It seeks to establish whether the intervention addresses the priorities and policies of the beneficiaries (the participating SAIs). It also inquires whether the design of the intervention is consistent with the overall goal, the intended impacts and effects that are pursued by the Programme sponsors, IDI and USAID. We used the following review questions: i. Do the objectives of the Programme continue to be relevant today? ii. Is the initial design of the Programme logical and coherent (including whether the results framework is consistent in terms of inputs, outputs and outcomes) and has it learned lessons from the 3i￾intiative in English-speaking countries? iii. Does the design ensure complementarity with other providers of assistance to the beneficiary SAIs and avoid/limit overlap? iv. To what extent have the assumptions underlying the Programme proven to be realistic? v. To what extent have adjustments of the initial design of the Programme supported its continuous relevance? vi. Has the Programme recognised risks towards efficiency, effectiveness, impact and sustainability and provided adequate arrangements to mitigate them? 2.2 Continuing relevance of the objectives Supreme Audit Institutions carry out a crucial role in the governance and public financial management of developing countries. The UN General Assembly has recognised the important role of SAIs in “promoting the efficiency, accountability, effectiveness and transparency of public administration, which is conducive to the achievement of national development objectives and priorities as well as the internationally agreed development goals, including the Millennium Development Goals.". 6 These sentiments are continued and reinforced in the Sustainable Development Goals (SDGs) approved by the General Assembly in 2015. Until 2010 there were no universally recognised standards to guide the work of the SAIs. In 2010, INTOSAI adopted such standards now referred to as the International Standards for Supreme Audit Institutions (ISSAIs). Key stakeholders such as the international financial institutions and major donors all encourage the development of strong SAI’s. The credibility of SAIs will be enhanced where the quality of the audit work undertaken is underpinned by adherence to the internationally recognised standards, the ISSAIs. In practice, there is a wide variation in working methods between SAIs across the world and some SAIs will work more in line with the ISSAIs than others. The IDI launched the 3i Programme Phase-I in 2012 to support SAIs in developing countries in implementing ISSAIs focusing initially on English speaking regions. It delivered iCATs and ISSAI Implementation Handbooks, ISSAI certification programmes for financial, performance and compliance audit in English. The 2014 global survey demonstrated a need for support for ISSAI implementation in non-english speaking regions.7 ARABOSAI, the regional body representing SAIs in MENA countries, has emphasized the need for support to its members in adopting the ISSAIs by providing support materials tailored to the needs of the SAIs in the region. 6 Resolution A/66/209 adopted by the 66th United Nations General Assembly on 22 December 2011. 7 The Global Survey is a triennial survey of IDI among SAIs ac to infer the development needs of the SAIs. 11 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Demonstrating continuing relevance would require observation of the current audit practices of the participating SAI and to compare them with the ISSAIs. INTOSAI bodies have developed two different tools to assist in ISSAI implementation: (1) SAI Performance Measurement Framework (PMF), which seeks to measure compliance against the ISSAIs, and is designed to provide (some) assurance to external users of the reports. (2) ISSAI Compliance Assessment Tools (iCAT), which facilitate SAIs to undertake a detailed mapping of their practices to inform ISSAI implementation strategies, but which do not measure compliance or performance against the ISSAIs, and are not intended to provide assurance to external users of the reports. The 3i-Programme has not made use of the SAI-PMF tool which focuses on assessing SAIs compliance with the ISSAIs. IDI considers, given the focus of the project on ISSAI implementation, that the iCAT is the more appropriate tool for understanding and developing the audit practice of the participating SAIs in each of the three relevant audit disciplines: financial, compliance and performance audit. Therefore, as part of the 3i-Cooperation Agreement and the ISSAI certification programme, each participating SAI has used the iCAT to assess its own implementation needs. Indeed, in all of our seven case studies we found confirmation that this work has been completed. The iCAT process is based on self-assessment by the participating SAIs which has inherent limitations as noted above. The 3i-programme addressed these limitations of self-assessment by holding ‘review workshops’ that allowed IDI resource persons to review and comment on the iCATs as they were developed. As the SAIs own the results of the individual iCAT assessments, they are entitled to keep the results confidential and the results of the iCATs were not shared with the review team. IDI has not carried out their own assessment of the iCAT reports. In the absence of objective evidence we cannot give a conclusive answer on the review question. However, based on a professional judgement, there is no reason to doubt the continued relevance of the Programme given that (1) the SAIs in the MENA region do not have a long tradition of conducting audits on the basis of ISSAIs; (2) all eligible SAIs have signed the Statement of Commitment and (3) the case for the Programme was robust and it has only run for two years since inception. 2.3 Logic of the Design and its Results framework The design of the Programme has been described in the Programme Plan “ISSAI Implementation Initiative (3i Programme) in ARABOSAI”. The logical framework was evaluated previously by the Independent Evaluation of the “Programme on Global Partnerships to Strengthen Capacities of Supreme Audit Institutions” in 2015.8 The conclusions of this evaluation was largely positive: “The Program as designed focused on the production of tools (iCATs and handbooks) and encouraging SAIs to make use of these tools by training auditors; and has been a great success.” However, the evaluation noted that the successes are mainly related to the outputs and that achievement of the final outcomes or impact is questionable. Accordingly, it recommended the inclusion of a new outcome objective for a future Program: “the hoped for impact of the Program is not to have a suite of well-designed tools that SAIs use regularly; these are only outputs. For a future or continuation of the Program, the IDI should consider a new objective (outcome): Improvement in the quality of audit work produced by SAIs as measured by their levels of compliance with the ISSAIs. The impact will then be greater public accountability and better use of public funds.” Our review of the Results Framework of the 3i-Programme in ARABOSAI found that this recommendation of the previous evaluation was implemented. The framework states ‘improved governance and financial 8 This Programme started in 2011 and was co-funded by the World Bank. The 3i-Programme for SAIs in the ARABOSAI region follows the design of this Programme including the use of the ISSAI Certification Program, management workshops for SAI top management, knowledge sharing (development of tools (iCAT) and Handbooks and making them available on a platform and additional workshops (iCAT review workshop and projects whereby SAIs are supported in audit planning and review meetings regarding cooperative audits). 12 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD management in the public sector’ as its purpose/impact and ‘SAIs in ARABOSAI move from current audit practices to ISSAI-based audit practices’ as its objective (final outcome). In addition, the evaluation also recommended that measurement of the achievement of the objective is required: “Any future project should ensure that initial benchmarked levels of ISSAI compliance are measured for SAIs worldwide (using the excellent iCAT and SAI PMF tools already produced). Changes in compliance levels can then be tracked with repeated use of the tools over a number of years.” As observed in the previous section, this recommendation is implemented by including iCAT assessments as an obligatory requirement of the 3i-Cooperation Agreement. A potential weakness is that the method of the iCAT assessment is done on the basis of self-assessment and no separate quality assurance process is included in its methodology. In order to ensure the quality of the iCAT assessments and, related ISSAI implementation strategies, the 3i-pProgramme has incorporated review workshops as an integral part of the 3i-Programme. This review process incorporated the following steps: During the iCAT review workshop, the IDI capacity development manager ARABOSAI (IDI CDM) developed a checklist for evaluating the quality of the iCAT reports and the ISSAI implementation strategies. The checklist was discussed and agreed with the resource persons. IDI CDM distributed the reports and strategies among the resource persons prior to the workshop (each report and strategy had to be reviewed by 2 different resource persons based on the agreed checklist). Non￾Arabic speaking experts participated in the iCAT-review workshop and interpreters were put at their disposal to help them check the quality of the iCAT reports and provide feedback. During the workshop IDI CDM had daily meetings with resource persons to discuss their feedback on SAI iCAT reports and strategies. In addition to that IDI CDM distributed the iCAT reports and ISSAI strategies between peer SAIs. Each participating SAI had to provide feedback on the iCAT report and ISSAI strategy of another SAI using the same QA checklist. During the workshop each SAI had to present their iCAT reports and implementation strategy. Each presentation was followed by a feedback from a peer SAI and then the feedback of the resource persons. After the iCAT review workshop IDI CDM followed up with the revision of the iCAT reports and ISSAI strategies. When he received the 43 revised iCAT reports and 15 strategies he read through them to check improvement before certifying participants. The workshop’s extensive review process is likely to contribute to improved quality of the iCATs and the ISSAI implementation strategy. However, the rigour of the process cannot be determined as the iCAT assessments and the ISSAI-implementation strategies are not shared by IDI for external review without the consent of the participating SAIs. Although the review process is a recommendable element in the 3i-programme, it does not have the same rigour as a quality assurance process incorporating formal review procedures and in which the quality assurer provides a formal statement on the quality of the document. The review process relies to a large extent on the experience of the participating resource persons and the willingness of participating SAIs to amend their documents. The role of IDI is largely limited to coordination and finalised outputs are not given an IDI ‘quality stamp’. 2.4 Complementarity and overlap with other development assistance to the beneficiary SAIs Many SAIs participating in the Programme also receive support from other international donors. The responses to question 8 in our case studies (see Annex 4) show that each of the seven selected SAIs have received some kind of other external support during the period in which they were provided with assistance through the 3i-programme. Table 3 provides a brief overview. 13 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Table 3 Support received by participating SAIs from other international development partners Country Other support from international donors Iraq  World Bank project to develop a risk based audit manual based on ISSAIs;  Project with the Netherlands SAI on performance audit and on communication and stakeholder management;  Governance Strengthening Project with USAID with participation of the SAI. Libya  World Bank Performance Measurement Framework programme. Morocco  Joint pilot performance audits based on ISSAIs in cooperation with the French Court of Accounts;  INTOSAINT programme to improve the level of integrity at SAI in cooperation with the Netherlands Court of Accounts and the French Court of Accounts; Palestine  Cooperation with Sweden in performance audit.  EU project for capacity building that includes a component for audit according to ISSAIs.  World Bank Performance Measurement Framework  Partnership with Netherlands SAI – sustainable development programme. Mauritania  World Bank project with a component tackle the implementation of international standards. Sudan  AFROSAI and AFROSAI-e in training on performance audit;  World Bank through financing a Roll Out financial audit manual. Tunisia  Support from Netherlands Court of audit, World Bank, UNDP, European Union in the implementation of the Strategic Plan 2016 – 2020. The risk for the 3i-Programme arising from other support projects is the potential overlap between the efforts of different donors and missed opportunities for complementarity. Such risks for overlap are increased as there is no formal coordination mechanism between IDI and other international donors at country level. The participating SAIs indicate that they do not coordinate with IDI prior to accepting other international support. However, the coordination mechanisms at global and regional level are quite strong. Through the work of INTOSAI Donor Cooperation Secretariat, all major donors are aware of the importance of the ISSAIs. At the country level, it is the responsibility of the participating SAI to coordinate its own development efforts whether internally resourced or supported by international donors. The formulation of the ISSAI implementation strategies and SAI development strategies should provide the basis for a coordinated approach. 2.5 Realism of the assumptions underlying the Programme 2.5.1 Assumptions at the level of the participating SAIs To allow successful implementation of the Programme, the Programme document states four assumptions about the participating SAIs. Table 4 states these assumptions and includes our assessment whether these assumptions were realistic. 14 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Table 4 Assessment of the realism of the assumptions relating to the participating SAIs Assumption Observation Judgement SAIs want to implement ISSAIs  All 16 member SAIs of ARABOSAI, with the exception of the SAI of Jordan, have signed the cooperation agreement; Met SAIs and SAI leadership are willing to change systems and behavior  Except for individual cases of specific SAIs, there is no evidence that SAIs are not committed. Vice versa, this mid-term review also comes too early to confirm this assumption. The country case studies are too limited in scope to make observations on system and behavioral change. Unclear SAIs have necessary mandate  Except for limitations in the mandate of the SAI of Mauretania and Morocco none of the six case study SAIs indicate limitations in the mandate to carry out financial, compliance and performance audit in line with the ISSAIs (see Annex 3, question 2). In the case of Morocco, the Government does not yet prepare financial statements (inhibiting financial audit as per the ISSAIs), but it is expected to do as from 2018. Largely met SAIs will keep commitments made  Although progress is not fully in line with the Cooperation Agreement, evidence from the six case study SAIs demonstrate progress in the implementation of the requirements of the cooperation agreement (see Annex 4, question 1). Met 2.5.2 Assumptions at the level of IDI and other stakeholders To facilitate successful implementation of the Programme, the Programme document states three assumptions about the abilities of IDI and other stakeholders. Table 5 states our assessment whether these assumptions were realistic. Table 5 Assessment of the realism of the assumptions relating to IDI and other stakeholders Assumption Observation Judgement IDI has sufficient resources (funding and staff) to manage this programme  Two staff members of IDI are involved in the programme: IDI Capacity Development Manager ARABOSAI and the IDI Programme Coordinator for ARABOSAI region. IDI has been able to fund these positions from their IDI basket fund and the co-funding from USAID. Met IDI will get in-kind contribution from SAIs in terms of required resource persons and hosting facilities.  The Programme receives ample in-kind support from several participating SAIs. For example, the SAI of the UAE has provided in-kind support to the translation of the ISSAI handbooks and the iCATs in Arabic.  SAIs of hosting countries have sponsored various aspects of the hosting facilities including airport transfer, meeting venues, etc. Met Partners agree to release experts to the programme.  Participating SAIs and partner SAIs have released experts in the ISSAIs to act as resource persons without charging fees. In most cases, the Programme only has borne travel and accommodation costs.  The Arabic-speaking members of the Financial Audit Subcommittee of INTOSAI were not released for three of the four events. In the absence of resource person from Financial Audit subcommittee, the quality control requirements were addressed by some of the regional resource persons with professional qualifications Partially met 15 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD 2.6 Relevance of adjustments to the initial design of the Programme The contract amendment for the 3i-Prorgamme, agreed between IDI and USAID (see section 1.1 above) includes the extension of the implementation period for the 3i-programme component until the end of December 2019 and increases the USAID portion of the budget from US$ 500k to US$ 930k.9 Table 6 provides a detailed breakdown of the increase in the USAID grant. It shows that the additional funding is mainly allocated to cover the project management costs for three more years and to contribute to ISSAI-based pilot audits. Table 6 USAID grant (2013-original versus 2015-amended) Original USAID budget allocation (in US $) Amended USAID budget allocation (in US $) Create awareness 30,000 30,000 Create capacity 358,000 203,066 ISSAI implementation 154,028 ISSAI knowledge network 32,000 78,475 ISSAI-based pilot audits 169,950 Project management costs 80,000 295,405 Total USAID 500,000 930,924 (Total cost-share IDI basket fund and in-kind contribution) 528,500 831,91610 Given that the transformation towards ISSAI-based audits is a process that requires a longer time span the extension of the duration till December 2019 is considered to be a beneficial amendment to the original contract. The incorporation of funding for ISSAI-based pilot audits is also considered to have value added for the successful delivery on the programme results. The addition of two further components to the Programme can potentially reinforce the transformation towards ISSAI implementation. The ‘SAIs engaging with stakeholders programme’ follows a model that builds stakeholder engagement on the backbone of the audit process. IDI believes that in order to have a robust audit process, convey key messages to stakeholders to facilitate change and ensure higher impact of audit, stakeholder engagement is critical. Facilitating 3i cooperative audits is an essential part of the programme results framework. For the IDI programme on SAI fighting corruption, an ISSAI based performance audit of institutional framework for fighting corruption will be facilitated. As such while the PA audit methodology will be covered, the subject of the audit is one which is highly prioritised by SAIs in the region.11 We understand the complementarity of these two components in building the impact of SAIs and their close relationship to developing audit processes through the implementation of ISSAIs. Nevertheless their addition after project implementation has begun runs the risk that the Programme will acquire too many objectives which can result in a loss of impact and focus. It will require very close management by IDI and this participating SAIs to mitigate this risk over the remaining two years of the Programme. 9 As noted in Section 1.1, during the implementation of this mid-term review in March 2017, the agreement with the USAID for USD 2 million was unexpectedly limited to USD 1 million. This development has occurred outside the time scope of this mid-term evaluation and is disregarded in the further description of this section. 10 Excluding the cost share component of SAI Fighting Corruption and SAI Engaging with Stakeholders. 11 As evidenced by the priorities indicated by SAIs on the new IDI programme portfolio presented at a SAI stakeholder workshop held in Kuwait in February 2017. 16 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD 2.7 Risk assessment and mitigation measures This review question addresses the adequacy by which the Programme has recognised risks towards relevance, efficiency, effectiveness, impact and sustainability and whether it has provided sufficient arrangements to mitigate those. The Programme document includes a risk analysis that identifies the following factors and mitigation measures. Table 7 Risk assessment and implementation of mitigation measurest Criteria** Observed Risks* Mitigation measures* Effectiveness of risk mitigation** Risk for relevance Added value  IDI service delivery model followed  SAI commitment  Programme-based on SAI needs Risk has been mitigated (see section 2.2) Risks for efficiency Quality of deliverables  Goods developed with the involvement of standard setters  Material developed and delivered following IDI service delivery model Risk has been mitigated. Quality of deliverables are well appreciated (see Chapter 3) Availability of resource persons  Database of resource persons  Use of resource persons from other regions and organisations Risk has been mitigated. Availability and quality of resource persons are well appreciated (see Chapter 3) Availability of safe venues  List of safe venues  Dialogue with SAIs  More blended solutions Risk has been mitigated. Logistical arrangements are generally well appreciated (see Chapter 3) Risks for effectiveness SAI willingness to share status of ISSAI implementation  Confidentiality of shared information Too early to assess whether risk mitigation has been effective Risks for sustainability Sustainability  Programme based on SAI needs  Involvement of SAIs in programme planning  Lessons learned  Programme monitoring  3i Phase II, support for pilot audits Too early to assess whether risk mitigation has been effective *Source: programme document **Source: review team We conclude that the risk assessment and the implementation of mitigation measures is well developed for risks relating to the relevance and the efficiency of the programme. Less attention is given to the risks relating to the effectiveness, impact and sustainability of the programme. The risk to effectiveness is key mitigated by the willingness of SAIs to report the status of their ISSAI implementation to IDI albeit on a confidential basis. We found that none of our case study SAIs has a monitoring system in place to report on the implementation of the ISSAIs. However, it may be too early to assess this aspect in the mid-term review. 2.8 Conclusions on relevance This chapter has reviewed the ongoing relevance of the design of the 3i-Programme in Arabic on the basis of six review questions. Overall, the conclusion is that the Programme is still highly relevant. Compliance with the ISSAIs is a key requirement for SAIs worldwide to contribute more effectively to better public governance and financial management. The countries in the MENA region do not have a long tradition of 17 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD conducting audits on the basis of ISSAIs and they could not participate in the initial phase of ISSAI implementation as this was delivered in English. Given the challenges of transforming the audit methods of the SAIs in this region at both the institutional level and at the level of the auditors, the reforms will require a longer timespan than the current two years of programme implementation. The increase of the USAID grant and the extension of the time scope to December 2019 recognises these challenges. This extension will allow IDI to support the SAIs in the conduct of ISSAI-based pilot audits. On the other elements of the relevance of the project design we have reached a positive conclusion. Both at the level of the participating SAIs and at the level of IDI and other stakeholders, the assumptions built into the project design have largely been met. The SAIs have demonstrated willingness and interest to implement the ISSAIs and are by and large not constrained by their national legislation. IDI has sufficient resources to manage the program and regional SAIs provide in-kind support. The main concern in the project design concerns the measurement of effectiveness. The project design has not ensured that baseline levels of ISSAI compliance were measured for the participating SAIs at the beginning of the project in an objective manner. Neither is such follow up measurement being incorporated in the project design. The project does require the use of the iCAT tools and such iCAT reports have indeed been prepared as part of the ISSAI Certification Programme and discussed in review workshops. However, these reports are not used as an initial benchmark to monitor progress over time. The quality of the reports is open to question as no external quality assurance has been conducted. The reports remain internal to the participating SAIs and are not available to donors and other key stakeholders. The risk management framework of the 3i-Programme has not addressed this issue. The Cooperation Agreement between IDI and the SAIs requires that SAIs prepare a monitoring report on ISSAI implementation on an annual basis, but most of the participating SAIs still need to act on this requirement. This omission is related to the delineation of the responsibility of the participating SAIs and IDI in the Results Framework. In this framework, IDI’s responsibility is confined to the delivery of the 3i-Programme and the individual SAIs are responsible for compliance with the ISSAIs and the application of the tools to measure compliance with ISSAIs. Although this division of responsibility reflects the relationship between INTOSAI and individual SAIs, it enhances the risk that the outputs of the programme have been delivered successfully, but that actual audit practice in participating SAIs has not moved significantly towards full compliance with the ISSAIs. IDI recognises the issue but has limited scope to address it directly. It is looking for alternative ways to address programme effectiveness but ultimately this requires wider discussion and agreement within INTOSAI. Currently there is no INTOSAI wide framework for regulating the use of ISSAIs or for monitoring their implementation or for assessing ongoing compliance with their requirements. Greater clarity on this issue within INTOSAI would greatly facilitate the measurement of the effective implementation of the ISSAIs by SAIs. 18 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD 3. Efficiency 3.1 Review questions The criterion ‘efficiency’ is concerned with the link between the resources and the outputs of the 3i￾Programme. Based on this criterion, this chapter addresses whether the outputs were delivered in good quality and within the planned timescales. The following review questions have been used: i. Could the outputs have been delivered more cheaply by making more efficient use of the inputs? ii. Compared to alternative options, does the Programme have a comparative (cost) advantage in furthering ISSAIs in MENA countries? iii. Did the Programme deliver the output according to the (time) schedule and the targets? iv. Did the Programme deliver the outputs in good quality? 3.2 Cost efficiency of the delivery Table 9 shows the expenses of the Programme up to 31 December 2016. Table 8 Program expenses (in US$) Funded from USAID funds Funded from IDI basket funds Item Expenses Item Expenses Project management 161,911 Project management 167,000 Workshops 488,945 Workshops 491,058 Miscellaneous 3,343 Translation 61,566 Total 654,199 Total 719,624 Source: FPO calculations based on Q4 2016 IDI-ARABOSAI report The table shows that the two most important cost components are project management and the organisation of workshops. Our assessment of the cost efficiency thus focuses on these two items. 3.2.1 Project management The total project management expenses amount US$ 328,911 and is about 24 per cent of the total project cost to December 2016. For this amount, the programme has benefitted from 1.5 full time equivalent (fte) of project management support (including the IDI Capacity Development Manager ARABOSAI and 50 per cent of the costs of the IDI Programme Coordinator for ARABOSAI region) over a period of 39 months.12 Average project management costs amount US$ 5,622 per month.13 It is not likely that qualified staff could be recruited for lower costs.14 Neither do we consider it likely that the programme could be managed by less than 1.5 fte staff capacity given the size and complexity of the programme involving 16 SAIs and the large number of project events and related logistical efforts. 12 Although prior to 2016, the CDM has not worked full time on the 3i-Program, IDI is not able to quantify the amount exactly as it has not used a time writing system. However, given that additional work is being done by IDI senior management in the IDI Headquarters and these costs are not allocated to the 3i-programme, the review team considers 1.5 fte a good approximation on average. 13 Calculated by dividing the total project management costs (US$ 328,911) over 54.5 working months (39 months of 1.5 full time equivalent). 14 As the Capacity development Manager is based In Tunesia, IDI makes savings on the labour costs while reaping the benefits of proximity to the region. 19 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD 3.2.2 Workshop costs A total amount of US$ 980,003 has been spent on workshops. Table 10 shows costs for each workshop and the number of days and number of participants. Table 9 Composition of expenses to workshops and meetings Item Total Nr of days Nr of participants Cost per day per participant IDI-ARABOSAI 3i Management Workshop 46,026 3 34 451 3i Product Adaptation Meeting 102,565 31 34 237 3i ISSAI Implementation Workshop 403,976 17 68 349 3i Workshop on Facilitating ISSAIs 257,441 12 82 262 IDI-ARABOSAI Meeting with SAI Management 49,838 2 31 804 3i ARABOSAI Product Adaptation Meeting CA 32,764 10 5 655 Workshop on CA Topic 71,143 12 25 237 Financial Audit Planning Workshop 16,250 4 9 451 Total 980,003 81 288 341 Source: Own calculations based on Q4 2016 IDI-ARABOSAI report Table 10 shows that the average costs per participant for each workshop day is US$ 341. The costs also include the expenses for international travel and visa costs. As a consequence, the table also shows that the average costs are higher for workshops with a relative short duration as travel expenses make a greater share of the total. Corrected for travel expenses, the average costs per participant is in the range of the standardised United Nations per diem rates for accommodation and daily subsistence for the workshop locations used (Casablanca US$ 230; Amman US$ 360 and Khartoum US $219) and no significant savings on these expenses can be expected. 3.3 Comparative assessment against other implementation methods The key methodological approach of the 3i Programme is rooted in the IDI service delivery model which follows a facilitation approach to empower SAIs to be able to manage their own capacity development. That model implies the use of international workshops for staff of the participating SAIs facilitated by more experienced colleagues (resource persons). During these workshops, the participants are expected to learn from the experiences from their peers from other SAIs and from the ISSAI facilitators / resource persons invited by IDI. Based on these discussions and learning experiences, the participants return to their home SAI and are expected to apply the lessons learned to institutional reform (strategy and manuals), audit work and training of their colleagues. The programme makes almost no use of “in-country support from external IDI-funded experts”. Although, a small part of the budget is targeted for such support in a few countries, it has not yet been used. This review question asks whether an alternative delivery method could be considered more cost-effective.15 To answer this question, we hypothesise as an alternative delivery method a programme that is based on IDI-funded ISSAI experts who work on-site to support the participating SAIs on ISSAI-based strategies, manuals, in-house training and pilot audits.. Table 11 provides an estimate of such a delivery method. The calculation is based on the following assumptions: 15 Cost effective is defined as the balance between outputs and costs. A more cost effective delivery method could thus either deliver the same outputs for lower costs or deliver more outputs for the same costs. 20 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD  A team of three ISSAI expert (for FA, CA and PA) are working on-site;  Each expert is present on-site for three years for 100 working days per year;  Savings on project management: reduction of 50 per cent;  Savings on workshop costs: workshops and interpretation costs are borne by the participating SAIs. Table 10 Costs of an alternative delivery method Item Nr of working days Cost per working day (lump sum including travel and per diem) Cost per day per participant Costs of FA expert 300 1000 300,000 Costs of CA expert 300 1000 300,000 Costs of PA expert 300 1000 300,000 Total costs per SAI per year 900,000 Number of participating SAIs (excluding SAIs from the Gulf countries) 11 9,900,000 Project management 200,000 Total costs of the programme 10.100,000 From Table 11, an initial conclusion is that the 3-Programme is a cost-effective delivery method. Due to the costs of hiring ISSAI experts, the costs of the alternative delivery method are significantly higher than the 3i￾Programme. However, the cost comparison has to be set against the a judgement on how effective the alternative deliver models are in delivering the desired outcomes. A final conclusion is conditional upon the ability of the 3i-Programme to deliver on the outputs (see Section 3.4) and to achieve the outcome objectives (see Chapter 4). 3.4 Timeliness of output delivery Table 12 compares the delivery schedule of the outputs up to 31 December 2016 envisaged in the original programme document of 2013 with the actual implementation up to 31 December 2016. 21 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Table 11 Actual output delivery compared to schedule Outputs Scheduled up to the end of 2016 Actual delivery16 3i Products (iCATs, ISSAI Implementation Handbooks) available in Arabic By 2015 Completed in 2015 ISSAI Certification Programme for Financial, Performance and Compliance Audit designed, developed and delivered to SAI staff as per IDI’s systematic approach to training (in Arabic) By 2015 Completed in 2015 Number of SAIs supported in ISSAI Implementation by reviewing iCATs By 2016: ‐ 6 FA ‐ 8 PA ‐ 8 CA Completed 11/2015: ‐ 14 FA ‐ 14 PA ‐ 15 CA Number of SAIs supported in ISSAI Implementation in conducting ISSAI based pilot audits ‐ 2 FA (2016) ‐ 9 PA (2018) ‐ 9 CA (2017). ‐ FA: support to three SAIs have started;17 ‐ PA: planned for 2017/18; ‐ CA: Support to 9 SAI in procurement audit has started and is expected to be completed in 201718 No. of SAI provided onsite support ‐ 2 FA (2016) ‐ 3 PA (2018) ‐ 3 CA (2017). ‐ No activities reported up to 31/12/201619 3i Community of practice available in Arabic By 2015 Completed October 2015 Cumulative number of trained SAI PMF assessors from SAIs in ARABOSAI (Male / Female) Male;female ‐ 55;7 (2014) ‐ 70;10 (2015) ‐ 85;15 (2016) ‐ the target up to 2014 has been met, but no activities reported up to 31/12/2016.20 The Table shows that IDI has been largely successful in meeting the output deadlines and the targets. In some cases, such as the number iCAT reports, the actuals surpass the targets. However, in other areas, the support has not been delivered or has been delayed. First, the support in ISSAI￾based financial audit was scheduled to be completed in 2016, but the preparatory stages were only picked 16 Q4 2016 IDI-ARABOSAI report 17 A 3i Financial Audit Planning Workshop was held in Morocco in November 2016. Further support will be provided to three SAIs of Iraq, Palestine and Sudan. The participating SAIs will undertake the audit in 2017. They will be provided online support in the initial part of 2017 and the quality assurance of the reports will be conducted towards the end of 2017. 18 In May 2016, course material for ISSAI Based Compliance Audit on Procurements were developed in a workshop in Khartoum. In September 2016, 25 participants from nine SAIs attended a workshop in Amman. After the workshop, audit teams from the participating SAIs (except Libya) posted their draft audit plans in the community and got feedback on the draft audit plans from resource persons. 19 IDI considers the physical workshops held in Morocco for Iraq and Palestine to be on-site support. The review team considers this not clearly defined in the programme documents and considers that ‘on-site support’ raises the expectation of engaging in expert missions to SAI offices in-country. 20 Complementary support was provided by ARABOSAI in the form of a workshop on SAI PMF held from 20-24 November 2016 in Saudi Arabia in which 20 participants were trained. 22 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD up towards the end of 2016. The planned on-site support in financial audit in 2016 was not delivered. Instead, the planned support to financial audit will be carried out using online communication. Finally, the cumulative training of SAI-PMF assessors has not been implemented as planned.21 3.5 Quality of the output delivery Three sources provide evidence on the quality of the output delivery: ‐ Feedback from participants via evaluation forms distributed and collected by IDI after the close of each workshop; ‐ Feedback from resource persons via a questionnaire distributed as part of this mid-term review; ‐ Interviews with the case study SAIs. 3.5.1 Feedback from participants IDI has requested participants to grade their satisfaction with the workshops on a scale from 1 (inadequate) – 5 (excellent). The questions included various aspects of the workshops including: (-) value added; (-) organization; (-) workshop materials; (-) exercises/case studies; (-) training venue; (-) time schedule; and (- ) instructors. The detailed scores are shared with the review team. Table 13 shows the average rate across the various aspects. Table 12 Overall levels of satisfaction of the 3i-Programmes’ workshops rated by participants Workshop FA CA PA Comments 3I ISSAI Implementation Workshop 3,7 3.9 3.8 The average rating is negatively influenced by low rates of satisfaction on the quality of the accommodation iCAT Review Workshop 4.1 4.0 3.7 On average, PA scored lower satisfaction rates. The lowest score (2.9) was awarded to time schedule which was considered too tight Workshop on facilitating ISSAI Implementation 4.1 4.1 3.9 Overall high rates of satisfaction. Lowest rates of satisfaction was awarded to the time schedule (considered too intensive) Workshop on Compliance Audit of Procurement n.a. 4 n.a. Overall, a high rate of satisfaction. Lowest rate was awarded to accommodation conditions (2.8) The results show high levels of satisfaction with the workshops. Especially, the content, workshop materials and quality of resource persons were highly appreciated. The tightness of the time schedule and the quality of the accommodation received occasionally lower rates. 3.5.2 Feedback from resource persons As part of this mid-term review, a questionnaire was distributed to 13 resource persons that have participated in the workshops. The findings in the three important areas ‘communication’, ‘logistics’ and ‘workshop content’ are reported below. The full details on the questionnaire responses are reported at Annex 5. 21 IDI refers to the recent formal adoption of the SAI-PMF tool as per the INTOSAI Congress in December 2016 as a reason underlying the delay in this deliverable. The 2016 Q4 performance report states that two SAI PMF training courses have been factored in the programme for 2017. However, as ARABOSAI has expressed an interest in a more extensive support programme, IDI is currently reconsidering the approach towards SAI-PMF support for the ARABOSAI region. 23 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Table 13 Responses of resource persons to the question ‘what is your appreciation of the quality of the communication about the events by the organisers?’ 1 Inade￾quate 2 Partially inade￾quate 3 Neutral 4 Satis￾factory 5 Excellent No opinion Total Clarity 1 1 11 13 Timeliness 1 5 6 1 13 Responsiveness 1 2 10 13 Table 14 Responses of resource persons to the question “What is your appreciation of the quality of the logistics of the events” 1 Inadequate 2 Partially inadequate 3 Neutral 4 Satisfactory 5 Excellent Total Visa 1 5 7 13 Travel 2 1 5 5 13 Accommodation 2 11 13 Table 15 Appreciation of the resource persons to the following statements Statement 1 Fully disagree 2 Largely disagree 3 Neutral 4 Largely agree 5 Fully agree Total The event(s) were well organized and allowed for a ‘effective’ learning experience of the participants on the application of the ISSAIs 6 7 13 The events were led by resource persons which had enough experience on the ISSAIs to instruct the participants with authority 6 7 13 The findings from the questionnaire responses are consistent in their appreciation of the way the 3i￾programme has applied the workshop methodology. Both in terms of communication, logistics and workshop contents, the resource persons rate the efforts of IDI with either’4’ or ‘5’. 3.5.3 Feedback from the case studies The feedback from interviewees in our six case study SAI’s is consistent with the observations from the participants’ workshop evaluation forms and the survey among the resource persons. The feedback from the representatives of the six case study SAIs can be found at Annex 4 (question 6). 24 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD The representative of the Board of Supreme Audit of Iraq states: “We learned a lot from 3i programme. The material was very well prepared and tailored to suit our SAIs, experts were highly qualified and were able to explain any unclear areas. In general I would say this programme is very well organised.” The representative of the Cour des Comptes from Mauritania is similarly positive: “We find it good and organised. The content was relevant and objective, it was very useful and it covered the three types of audit. Very beneficial and rich with information.” Only one dissenting comment was noted by the Libyan Audit Bureau relating to financial audit: “We think 3i programme needs to be more detailed and to enable communication and entering into partnerships with IFAC in preparing advanced training programmes related to financial audit specifically for institutions with special activities (read: parastatal entities)” 3.6 Conclusions on efficiency This chapter has reviewed the efficiency of the implementation of the 3i-programme in Arabic on the basis of four review questions. Overall, the conclusion is that the delivery by IDI of the 3i-Programme is highly efficient. We have not identified room to save on the costs of project management and the organisation of workshops. We conclude also that an alternative delivery method based on on-site presence of external experts would significantly be more costly. In addition, we conclude that IDI has been able to deliver the majority of the outputs in a timely manner and to good quality. Both the beneficiaries and the professional cadre are highly appreciative of the quality of the workshop sessions. However, some activities have been delayed. The delay in the training on SAI PMF is justified by reference to the recent adoption of the tool by INTOSAI. More crucial is the delay in the support to ISSAI-based pilot audit which has only started up by the end of 2016. Also, no on-site support to selected SAIs have yet been initiated. As financial audit is the most challenging type of audit for SAIs in the ARABOSAI region, the experience gained from this activity is likely to be crucial for further progress in the region. 25 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD 4. Effectiveness 4.1 Review questions The review of the effectiveness aims to establish to what extent the Programme is on schedule to attain its outcome-objectives. The Programme’s Result Framework has distinguished between IDI outcomes and SAI outcomes. As noted in Section 2.3, we will use the terms intermediate and final outcomes. The following review questions have been used: i. To what extent is the Programme on track to achieve its outcome-objectives? ii. How do participating SAI perceive the value added of the Programme to implement the ISSAIs? 4.2 Achievement of the objectives 4.2.1 Intermediate outcomes Table 17 shows the intermediate outcomes achieved up to the end of 31 December 2016 against the targets envisaged in the original programme document. Table 16 Intermediate outcomes: actual achievement versus planned targets Intermediate outcomes Target Actual achievement22 % of SAIs from developing countries in the region signing the cooperation agreement. 80 % in 2015 Met. ARABOSAI counts 22 members of which 5 Gulf countries and 17 developing countries. Two of the developing countries, Djibouti and Comores, are not entitled to participate in the programme as they are participating in the same programme for Francophone African countries. Somalia has not signed the agreement as a special IDI programme is focused on Somalia. All remaining 14 development countries have (100%) have signed the Statement of Commitment. Four countries, however, have not been active consistently during the programme (Lebanon, Jordan, Syria and Yemen) reducing the active participation of developing countries to 10 SAIs (62 %).23 Percentage of participating SAIs reporting use of. iCATs and ISSAI Implementation Handbooks iCATs 80% (2016) Met. 14 SAIs have submitted iCATs for review for all three types of audit (88 % out of 16 participating SAIs).24 Percentage participating SAI Staff who complete ISSAI Certification Programme 60% (2016) Met. 96 Participants were selected to take part in the Audit certification Programme and 79 participants (82%) completed the Programme. 22 Derived from 2016 Q4 Performance Report, IDI powerpoint presentation on 3i Programme- ARABOSAI and own calculations. Note that the reviewers conclusions data 23 SAI Jordan stopped active participation one year after the start, but it has continued participation by the end of 2016. The participation of Syria is not continuous due to problems of getting visas and flight tickets due to political unrest. For similar reasons, SAI Yemen could not continue with the program. The participation of SAI Lebanon was discontinued as they did not conduct the iCATs. 24 In addition to the 14 SAIs, SAI Qatar submitted the CA iCAT. SAI Qatar did not submit the other 2 iCATs because the SAI participated only in the CA certification programme. 26 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Table 17 shows that more countries than expected have signed the cooperation agreement. However, the active participation of the ARABOSAI member SAIs is slightly less than expected. Political instability in the region, which is outside the control of IDI, is the main causal factor. The 3i-Programme has made efforts to enhance the participation. The SAI of Jordan rejoined the 3i programme again in September 2016 by participating in the ISSAI based compliance audits of procurement. The SAI of Somalia is supported by IDI by means of a bilateral programme. The intermediate outcomes relating to the degree of participation of the individual SAIs is exceeding expectations in terms of the number of SAIs making use of the iCAT tool and the certification programme. This is consistent with the high levels of appreciation by participating SAIs of the IDI support provided via the Arabic language guidance and the workshops. It is noted that the distribution of certified participants is not even among the participating SAIs. Table 17 shows that while 10 participants from the Tunisian Court des Comptes were certified, for the Mauritanian SAI only one participant was. This uneven distribution follows from various factors. First, not all SAIs nominated the maximum number of 15 candidates. For instance, Mauritania nominated 8 participants and Lebanon only 4. Second, the 3i-programme applied a selection process for the certification programme based primarily on merit through an online test. Accordingly, SAIs who nominated good candidates got higher number of participants. A third reason is that not all participants completed all programme components successfully, so they were not certified. Table 17 Number of ISSAI certified participants per participating SAI and audit discipline Country CA FA PA Total Algeria 1 1 1 3 Bahrain 1 2 2 5 Egypt 2 3 3 8 Iraq 5 2 1 8 Kuwait 3 2 1 6 Jordan 0 0 0 0 Lebanon 0 0 0 0 Libya 1 1 1 3 Mauritania 1 0 0 1 Morocco 2 3 3 8 Oman 4 1 2 7 Palestine 2 3 3 8 Qatar 1 0 0 1 Saudi Arabia 1 1 1 3 Sudan 2 2 3 7 Syria 1 1 1 3 Tunisia 2 3 5 10 Total 29 25 27 81 A further note concerns the quantitative nature of the 3i-Programme targets. The use of iCAT’s and ISSAI implementation handbooks is reported, but the review team was unable to find substantive evidence on the quality of the iCAT’s and the ISSAI implementation strategies and action plans. 27 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD 4.2.2 Final outcomes Table 18 shows the targets for the final outcomes. Given the timing of this mid-term review, we cannot report on the realization of the targets as their achievement is scheduled for 2017 and beyond. However, based on the analysis of the performance up to 31 December and the audit plans for 2017, we can offer an assessment of the probability that the targets for ISSAI-based FA, CA and PA audits can be achieved. Table 18 Final outcomes: planned targets and probability of achieving them Final outcomes Target Probability % participating SAIs which start ISSAI implementation as per their ISSAI implementation strategy and action plan 25% (2019) Unclear, most SAIs have formulated an Action Plan for their ISSAI implementation strategy, but annual monitoring of the Action Plan / ISSAI implementation is not yet established (see Annex 4, question 1, item 4 and 9). % participating SAIs which issue audit reports of ISSAI based pilot audits within the established legal time frame FA: 50% (2017) Likely, currently 3 SAIs have started ISSAI￾based financial audit including Palestine, Sudan and Iraq. Target will be met if one of the countries publish the pilot audit report. CA: 40% (2018) Likely, currently 9 SAIs have started ISSAI￾based procurement audit PA: 40% (2019) Likely, a crosscut with SAI Fighting corruption programme is programmed. % ISSAI based pilot audits which generally meet best practice requirements as per applicable ISSAIs FA: 50 %(2017) Likely, currently 3 SAIs have started ISSAI￾based financial audit including Palestine, Sudan and Iraq. Target will be met if one of the countries prepares a report meeting nest practice requirements CA: 40% (2018) Likely, currently 9 SAIs have started ISSAI￾based procurement audit PA: 40% (2019) Likely, a crosscut with SAI Fighting corruption programme is programmed. The table shows that the expectations for meeting the targets for all three audit disciplines are in reach. However, the positive assessment should be seen in relation to the formulation of the target. As the target is formulated in relation to the number of participating SAIs in pilot audits, the target does not relate to the total number of SAIs participating in the 3i-programme. Especially for financial audit in which only three SAIs participate in the pilot audits, this formulation results in a low level of overall effectiveness. For compliance audit, the number of SAIs participating in the pilot audit is larger. However, in our view the target lacks ambition as the 3i programme reaches its objective if only 4 SAIs have published an ISSAI-based audit report out of 16 participating 16 SAIs. Asked about the impediments to conducting ISSAI-based audits, the common element in the feedback from the representatives from the various case study SAI was the ‘lack of awareness’, ‘resistance to change’ and ‘absence of manuals’: “Some auditors lack of awareness of the importance of implementing standards (and) There are no detailed manuals for the three types of audit” (Libya Audit Bureau); “We believe that 3i helps in overcoming these difficulties through strategic planning and managing time it also address the difficulty of change resistance which we do face” (Cour des Comptes, Morocco); “The 3i Program had theoretically provided us of means of overcoming these difficulties, we as participants in 3i programme were individually aware of these difficulties but the institution itself and other SAI auditors need to understand the ISSAIs. A methodology needs to be in place” (Cour des Comptes, Tunisia). 28 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD The representative of the Cour des Comptes of Morocco demonstrates a further understanding of the impediments and refers to the costs of conducting compliance and performance audit according to the ISSAI and the absence of financial statements for financial audit: “For Compliance and Performance audit we have no difficulties except the financial difficulty as the cost of conducting audit according to ISSAIs is relatively higher therefore if we do not have sufficient resources we can only conduct up to two audits per year based on ISSAIs” “As for financial audit; a temporary difficulty that we are working on overcoming that is the iCATs requirement for the SAI to provide and opinion on financial statements which we are unable to do but have requested government institutions to do so by the year 2020 and are now working on it and believe time is needed as it is already stated in CDC law”. 4.3 Value added of 3i-Programme in meeting the ISSAIs Table 19 provides evidence from the survey among the resource persons regarding the effectiveness of the 3i-Programme. As it is too early to evidence the value added of the programme by reference to ISSAI-based audits, the table demonstrates the Programme’s contribution via raising the awareness and the ability of the SAIs to carry out ISSAI-based audits. Table 19 Value added of 3i Programme Statement 1 Fully disagree 2 Largely disagree 3 Neutral 4 Largely agree 5 Fully agree Total The participants were well aware of the ISSAIs and the implications for their SAI before they joined the workshops 1 2 4 4 2 13 The participants were well aware of the ISSAIs and the implications of the ISSAIs for their SAI after they joined the workshops 2 11 13 The workshops have significantly improved the ability of the SAIs to implement the ISSAIs 5 8 13 The findings from the survey are confirmed by the interviews with the case study SAIs. The case studies also show that the perceived benefits of the Programme may vary among the SAIs . The representative of the National Audit Chamber of Sudan emphasises the increased awareness raised by the 3i-Programme: “The 3i Programmes and events had benefited us in assessing our SAI capacities on institutional level and at audit task level and it identified gaps for improvement.” The representative of the more advanced Court des Comptes (CDC) of Morocco SAI emphasised the efficiency of having the support of IDI: “Although CDC enjoys a very good reputation in Morocco yet we need to stay updated on latest development accordingly implement best international standards (…) The 3i Programme had helped us a lot and saved us time if done by ourselves than with the support of IDI”. The Board of Supreme Audit (BSA) of Iraq notes the self-learning and the opportunity for peer learning provided by the Programme: 29 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD “3i programme is gradual, we were introduced on how to prepare a plan for implementation of ISSAIs then given time to draft ours and return to do the second stage improving the quality of our work, the community of practice was useful as between participants we were able to discuss the difficulties we are facing. Discussing problems and obstacles gave us a clearer understanding and opportunity to learn from other experiences.” 4.4 Conclusions on effectiveness Under the criterion ‘effectiveness’, this chapter has reviewed whether the 3i-Programme is on track to achieve its outcome objectives. The review makes a distinction between intermediate and final outcome objectives. For the intermediate outcome objectives, the review draws a positive conclusion. Although due to regional instability, the active participation in the 3i-Programme has been somewhat lower than expected, more SAIs than anticipated have signed the cooperation agreement. The participating SAIs have also made greater use of iCATs and the ISSAI Implementation Handbooks than anticipated in the programme document. And, more SAI staff than anticipated have completed the ISSAI Certification Programme. For the review of the progress on the final outcome objectives, the timing of this mid-term review comes too early as the outcomes are targeted for 2017, 2018 and 2019. Nevertheless, based on the observations in the delivery of the programme’s output in Chapter 3, it can be concluded that the final outcome targets in each audit discipline are likely to be achieved. However, given that the final outcome targets are formulated as a percentage of the number SAI that participate in the pilot audits, meeting the final outcome targets do not imply that the 3i-programme is effectively achieving its objective to transform the SAI’s audit practices towards the ISSAIs. This is most obvious for financial audit in which only three SAIs participate in the pilot audit and the target is met if only one or two SAIs prepare and publish an ISSAI-based financial audit report. The achievement of the outcome objective relating to the implementation of the ISSAI implementation strategies is not clear. So far, few participating SAIs have set up a monitoring system to assess the implementation of the strategy and the corresponding action plans. In reviewing the evidence, in particular the view sign posted by interviewees in SAIs that resistance to change is major impediment, we consider there is a significant risk that the outcome objective for the Programme will not be fully met. 30 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD 5. Conclusions and recommendations This mid-term review has assessed the performance of the 3i-Programme in Arabic against three of the OECD-DAC criteria including relevance, efficiency and effectiveness. Based on a logical framework that has been successfully applied in anglophone countries, more SAIs than anticipated have signed the Agreement of Commitment and 16 SAIs are participating in the 3i-programme. The reviews shows that the implementation of the 3i Programme in Arabic is largely progressing in line with the schedule and that IDI has rolled out the 3i-Programme to Arabic speaking in an efficient manner. Efficiency is observed in the cost effective use of the Programme’s resources. The review has not identified room to save on the costs of project management and the organisation of workshops. We also conclude that an alternative delivery method based on on-site presence of external experts will be significantly more costly. The 3i-delivery method which makes ample use of SAIs own resources through resource persons and support in kind and is able to serve a large number of countries within a relative tight budget. Furthermore, efficiency is observed in the timeliness and the quality of the output delivery. Both the beneficiaries and the professional cadre are highly appreciative of the quality of the workshop sessions and the supporting guidance materials. Not surprisingly then, the achievement of the outcome targets are also largely on schedule. Although fewer SAIs from Arab developing countries are actively participating than originally anticipated, those that participate have so far demonstrated their commitment to the programme. This is reflected in the high use of iCATs and the ISSAI Implementation Handbooks. More staff than anticipated have completed the ISSAI Certification Programme. Based on the current progress, it is expected that also the delivery of ISSAI-based pilot audits in the domain of compliance and performance audit will be delivered on schedule in 2017 and 2018. Our overall conclusion is that IDI has implemented the project successfully to the mid-term review point and is on track to deliver the majority of the agreed outputs and intermediate outcomes. However, ultimately the success of the programme will depend on embedding the use of ISSAI’s within the beneficiary SAI’s in ways which are sustainable. Some points for improvement have also been observed and we suggest the following conclusions and recommendations as the project moves to the next phase. 1. The Results Framework within the Programme Document makes use of the terms IDI objectives and SAI objectives. This terminology is not common as a logical frameworks and it gives, unintentionally, the impression that IDI is not pursuing the SAI outcomes via the 3i-Programme. We recommend using the terminology of intermediate outcome objectives and final outcome objectives’. In this way, it is clear that IDI also feels committed to achieve the final objectives while recognising that IDI is not in full control given the sovereignty of the SAIs. Annex 1 includes a revised Results Framework using intermediate outcomes and final outcomes’ rather than ‘IDI Outcomes’ and ‘SAI Outcomes’. 2. The result indicators used for the final outcomes have shortcomings in measuring the achievement of the objective of the 3i-programme. The indicators do not measure how many of the participating SAIs are successfully transforming their audit practice towards the ISSAIs. Instead, the indicator is measuring the effectiveness of the pilot audits and the achievement of the targets are conditional on the number of SAIs that are participating in the pilot audits. This is most obvious for financial audit in which only three SAIs participate in the pilot audit and the target is met if only one or two SAIs prepare and publish an ISSAI-based financial audit report. We would recommend to apply these indicators and targets for the monitoring of the intermediate outcomes and not for the final outcomes. 31 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Furthermore, we would recommend to add indicators and targets on the number of SAIs that are participating in the pilot ISSAI-based audits. Especially for financial audit, this may imply that more effort is needed to stimulate SAIs to participate in ISSAI-based pilot audits. 3. A further concern in the project design concerns the objective measurement of the final outcome objectives which is the transformation of audit methods towards ISSAI-based methods. Although the project design is sound overall, it lacks agreed and objective baselines and targets for measuring improvement in the degree of ISSAI compliance by the participating SAIs The project does require the use of the iCAT tools but the rigour and quality of the reports are unclear as no external quality assurance has been conducted and the reports remain internal to the participating SAIs. The most appropriate tool for such a measurement is SAI PMF. Although the 3i-programme has faced a delay with the implementation of SAI-PMF training against the expectations set out in the Programme Document, it is now planning an extensive SAI-PMF programme in the ARABOSAI region to catch up in 2017. We endorse strongly this initiative. We recommend that these assessments are carried out in an objective manner and are available for future evaluators of the 3i-Programme. These assessments should be repeated at appropriate intervals over the life of the project . For this purpose, an amendment on the Cooperation Agreement should be considered. 4. A further critical issue in the project concerns quality assurance. Although IDI has established an extensive process of quality review by resource persons via workshops, it is not clear whether these controls are effective in assuring the quality of key outputs such as the iCAT assessments. Although the iCATs are owned by the SAIs and they are responsible at all times for quality assurance, the outputs are prepared as part of an IDI driven programme which provides them INTOSAI legitimacy. To assure that the quality review procedures via resource persons are adequate, we recommend that IDI carries out quality assurance on a limited and random sample of iCATs by ISSAI experts that are not involved in the 3i-Programme in Arabic. Establishing a quality assurance process as a basis for assessing the ultimate success of the 3i￾Programme is even more crucial for the ISSAI-based audits that will be prepared in the remaining period. Again, the participating SAIs are the first responsible for quality assurance. However, as the pilot audit bear the name of IDI, we recommend that IDI carries out an independent quality assurance. 5. Prominent bottlenecks in the transforming the audit methodology from the current ones into ISSAI￾based methods are the lack of awareness and the resistance to change. The 3i-programme aims to increase the awareness and to overcome the resistance to change by training staff of the participating SAI in the ISSAIs and that these staff serve as agents for change in their home SAI. The feedback from the participants is that the workshops successfully achieve this purpose among the workshop participants. However, in order to reduce the resistance to change, participants emphasise the need to deepen (more participants) and to extend (longer duration) the 3i-Programme. Two illustrative quotations are: “It is important though to continue this programme to cover more participants from SAIs. From our side we are trying to transfer knowledge through training courses we conduct internally to raise awareness on ISSAIs at our SAI but it is not as good as the resources available at IDI and the qualification of experts. We recommend to increase the number of participants and the duration of the events” (Board of Supreme Audit, Iraq) “We believe that there is a need to assist in the implementation of standards, it is not enough to only help us to draft a strategic plan and work plans. We believe that long term partnerships and long term projects more benefits will be gained. Continuation of support is vital to the success of ISSAI implementation.” (Court des Comptes, Morocco) Following our review, we share these concerns. The cooperation between IDI with USAID has proven to be successful and both parties have expressed satisfaction with the partnership. We recommend that the parties give early thought to what more can be done to maintain momentum on ISSAI 32 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD implementation into the 2020s and to ensure that a critical mass of change agents are trained for each SAI that has demonstrated commitment to ISSAI implementation. Annex 1 Intervention logic Based on the programme document, the following intervention logic is derived. Improved governance and finance management in the public sector in ARABOSAI-countries SAIs in ARABOSAI move from current audit practices to ISSAI-based audit practices Purpose / impact Final objective / outcome Activities / outputs Development of global public good Resources ISSAI certification programmes ISSAI knowledge network / community of practice ISSAI-based pilot audits SAI-level ISSAI implementation startup % of SAIs from the ARABOSAI region signing the cooperation agreeement USAID funds IDI-basket funds Own funds SAIs from Gulf-countries Support in-kind INTOSAI member states % of participating SAIs which start ISSAI implementation as per their ISSAI implementation  strategy and action plan % of participating SAIs reporting use of ICATs and ISSAI implementation handbooks % of participating SAIs which issue audit reports of ISSAI‐based pilot audits within the  established legal timeframe % participating SAI staff who complete ISSAI certification programme SAIs in ARABOSAI participate actively in the Programme Intermediate  objective / outcome % of ISSAI‐based pilot audits which generally meet best practice requirements as per  applicable ISSAIs Products available in Arabaic ISSAI certification programmes designed, developed and delivered (in Arabic) to SAI staff as  per IDI’s systematic approach to training Number of SAIs supported in ISSAI implementation by reviewing ICATs Number of SAIs supported in ISSAI implementation in conducting ISSAI‐based pilot audits Number of SAIs provided onsite support Communi ty of practice available in Arabic Cumulative number of trained SAI‐PMF assessors in from SAIs in ARABOSAI Annex 2 List of documents The following documents at Programme level have been reviewed: - Programme document; - Quarterly performance reports; - Quarterly financial reports; - Workshop materials; - Evaluation of the 3i-programme in English-speaking countries; - iCAT assessments subject to agreement by the SAIs concerned; - SAI specific documents (see case studies) - IDI Performance and Accountability Reports. The following documents (in Arabic) were received from the case study SAIs SAI – Country Received data Iraq  Common strategy Libya  Strategy including implementation matrix  ISSAI strategic plan  Individual plans  Reports related to procurement pilot audit  Annual audit reports  Laws and regulations  Proposed amendments on the law  Code of ethics Mauritania  ISSAI strategic plan Palestine  Strategic plan for implantation of ISSAI Feb 2016  SAACB Law and code of ethics  Participants individual work plans Morocco  Copy of statement of commitment – the agreement  Programme participants’ nomination letter 1st phase  Programme participants’ nomination letter 2nd phase  3i outcomes  iCAT letter  Strategy  Work plan signed by facilitators  Letter of assigning ISSAI committee at CDC  CDC Law  Financial regulation law shoeing starting of financial audit  Morocco constitution Sudan  letter by Auditor General related to assigning a coordinator for standards with a duty of implementation of AFROSAI manual,  letter by Auditor General related to nomination of trainers to participate in INTOSAI and ARABOSAI workshop that took place in Kuwait Tunis - 35 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Annex 3 List of Interviews The following programme stakeholders interviewed: Interviewee Position Einar Gørrissen Director General IDI Archana Shirsat Deputy Director General IDI Ben Lazreg Abdelhakim IDI Capacity Development Manager ARABOSAI Fatima Doghman IDI Programme Coordinator ARABOSAI Shourjo Chatterjee IDI Strategy & Knowledge Manager Tyler Holt, Regional Economist, Director, US AID Middle East bureau The following representatives of the case study SAIs were interviewed: Organisation Name Court of Accounts of Tunisia Amel Elloumi. Member of INTOSAI committee of compliance audit Iraq Board of Supreme Audit BSA Dr Shayma Abbas Hwerif, Compliance audit expert and member at INTOSAI committee of compliance audit Fadia Fahmi, International coordinator Ms Libyan Audit Bureau Ala Al Masallati: Deputy head of SAI Dr. Tarek elhatab: Head of studies and strategic research center Dr. Yousef Khlifa : Head of Planning Department AbdAl Naser BISHA: Facilitator of Performance Audit Khald Samadi: Facilitator of Financial Audit Mahmoud Hamoudeh: Facilitator of Compliance audit Mauritania Court of Accounts, Cour Des Comptes CDC Ahmed Al Baddad, Consultant Moroccan Court of Accounts, Cour Des Comptes CDC Saeda Al Ifriki, Prosecutor and ISSAI Facilitator Palestine, State Audit and Administrative Control Bureau Yousef Hantash: facilitator for financial audit Sudan National Audit Chamber NAC Ikhlas Saad, Facilitator 36 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Annex 4 Case study results 1. Has the participating SAI been able to meet the commitments made in the 3i - cooperation agreement? Requirement in Statement of Commitment Country Situation in SAI 1. Nominate participants for ISSAI Certification Programmes as per required criteria Iraq 15 Participants attended the online examination that was conducted before joining the programme and only 8 passed and were certified. Libya Yes, 15 participants attended the online exam and 3 were selected and completed the programme and were certified on October 2016 Morocco Yes, CDC nominated 9 participants for ISSAI certification of only 8 were able to complete the programme. IDI were very accurate on attendance therefore the 9h participant was excluded due to absence. Palestine Yes, 7 participants attended the online exam, completed the porgramme and received certificates. Mauritania Yes three participants were nominated but only 1 finished the program, the reason is that one participant became the president of CDC and the other one had to drop off. Certificates had not been received. Sudan Yes, 7 participants Tunesia Yes, 10 participants but and they were all certified. 2. Use the iCATs to assess implementation needs with the help of ISSAI Facilitators (before July 2016) Iraq Yes, before November 2015 as we received support from 3 ISSAI facilitators from Arab countries including Brazil and Norway. We assessed the needs through developing a list questions to compare current processes with ISSAIs Libya Yes. We used iCATs tools with the help of facilitators through a central committee headed by technical managers and sub committees and results were sent to IDI on time before June 2016 Morocco Yes, Ms Saeda is an ISSAI facilitator and provided support mainly in producing iCATs tables 2 & 4 Palestine Yes, we received assistance from ISSAI facilitators. I am myself a financial audit facilitator and I provided support in assessing implementation needs. We were four Arab facilitators assisting SAIs according to type of audit, Shaymaa from Iraq and Mouna Mseiddi from Tunis – Compliance, Shafeeq from Morocco – Performance audit and myself for financial audit. Mauritania Yes, during the programme but not after we return to our SAI, for the audit I was responsible for only – compliance audit Sudan Yes, attached Auditor General decision no 42 year 2015. 3 iCat-reports were sent to IDI and based on them the participants were granted their certification Tunesia Not clear. 3. Formulate ISSAI Implementation Strategy Iraq Yes, iCATs helped us to know what ISSAIs are implemented at BSA and what are not, we tried to figure 37 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD in light of iCAT findings and with the help of ISSAI Facilitators (before July 2016). out why some ISSAIs were not implemented accordingly we incorporated these unimplemented ISSAIs as options within:  Audit plan 3 types of audit;  ISSAI implementation plan for the 3 tupes of audits them merged into one plan that was endorsed by BSA management;  BSA Strategic plan. Libya Yes based on the results using compliance assessment an ISSAI implementation strategy was drafted which its goals were in line with LAB strategic plan for the years 2015-2019 that included a special chapter for standards implementation and was indorsed in Feb 2015 Morocco Yes, In the beginning CDC had three strategies for each type of audit (compliance, financial and performance) then were merged into one strategy Palestine Yes, we finalized our ISSAI strategic plan on 1st of March 2016. Mauritania Yes we formulated ISSAI implementation strategy in light with iCAT for the three audits during the 3i programme then it was merged into one strategy Sudan Yes, an unified ISSAI implementation strategy was prepared by NAC facilitators of each standard between the periods of 2016-2018, this strategy was sent to IDI alongside reports. Tunesia I do not know but will check and confirm 4. Formulate Action Plan based on Implementation strategy and start implementation (before July 2016). Iraq Yes, a work plan before November 2015, after the procurement audit workshop on September 2016 actual implementation of compliance audit tasks started followed by financial audit. Libya Yes, through the LAB strategic plan that included within its strategic goals the enforcing of audit work through implementation of ISSAIs based on a work plan and execution programmes within a timeframe with clear identification and distribution of responsibilities and follow up mechanisms. In addition to individual plans and unifies strategic plans for facilitators. Morocco Yes, an action plan was formulated on Feb 2016. Palestine Yes. Mauritania Yes, it was a condition for the programme and we formulated an action plan based on implementation strategyfor the year 2016 directly after we returned to our SAI. Sudan Yes, a work plan was drafted according to strategy on 17 March 2016 Tunesia I do not know, possibly top management at CDC would know 5. Conduct at least one ISSAI based audit using ISSAI Facilitators (before July 2016) Iraq No, We started with compliance audit as we conducted a procurement audit as continuation of the 3i programme but after July 2016 and last meeting was in Jordan. Libya This phase was delayed by the initiative for an advanced stage. However LAB is in the process of preparing pilot tasks with contribution of facilitators whom had put this within their priorities at their individual plans regarding the pilot government procurement compliance audit 38 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD tasks but LAB participation was declined by IDI in this programme which took place in August 2016 instead an invitation was sent LAB branch at Baydhah City and their participation was accepted despite the noncompliance of participation conditions on them such as the number of employees at this branch around 30% of LAB staff, this branch is one of seven other branches of LAB that has 1900 staff within 36 branches. Even though we have continued the work despite of this interruption with ARABOSAI and an execution work plan was prepared for the procurement compliance audit. Morocco No, CDC are currently working on a pilot audit which started after July 2016 Palestine Yes, in Amman but in September 2016. It was on compliance audit – procurement. As reports produced by SAACB after learning about ISSAIs are now in line with these standards Mauritania Not yet. Work in progress but not yet done, however we have prepared a procurement audit during the programme and it is now about to be finalized Sudan We conducted 5 audits under the supervision of AFROSAI and one facilitator from ISSAI. We started a financial audit according to ISSAIs April 2016 with the help of 2 ISSAI facilitators Tunesia I heard that there has been a pilot on compliance audit but is not completed yet. 6. Table ISSAI based audit reports. Iraq Not yet. Possibly by the end of April 2017 Libya This question is based on previous answers as the execution of the pilot audit task was postponed then our participation was declined although LAB had finished more than 70% of the programme execution stages. Morocco No CDC are still at planning stage Palestine Work in progress and are planning to meet July 2017 to present our reports. Mauritania Not yet. We will discuss the report during next workshop Sudan We submitted two reports in the area of performance audit Tunesia No not yet, still at a preliminary stage – planning stage with a possibility to table reports by June 2017 7. Customize 3i products for use in the SAI Iraq BSA has developed many manuals based on ISSAIs therefore as did not have to start from scratch, compliance audit went faster as we are also working with Netherlands SAI we also worked with World Bank on developing a Risk Based Audit Manual based on ISSAIs. Libya LAB is in the process of preparing three audit manuals (performance, financial and compliance) and they were incorporated within LAB strategic plan Morocco CDC formed a standards committee of which min outcomes are guidelines and procedures manual for ISSAIs, this committee updated CDC manuals and guidelines based on ISSAIs and they are still drafts awaiting to be endorsed by top management. Palestine Yes, in fact before joining 3i programme we started preparing 3 manuals before attending the programme: 39 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD financial, performance and compliance, therefore after attending the programme we redrafted the manuals according to ISSAIs until we reached the final draft. We also started internal training programme for our auditors on using these manuals. Mauritania Not yet. We prepared a work plan to prepare manuals and guidelines. We hires a consultant to assist us in preparing a compliance audit manual but it was not a good experience as consultant was not committed. Sudan We have translated and endorsed AFROSAI-E manual for financial audit and stated using it April 2016, in addition we have translated and endorsed AFROSAI-E performance audit manual Tunesia No we did not however we might have already have in place a performance audit manual that was previously developed based on ISSAIs but not according to 3i 8. Actively use the 3i community portal. Iraq Yes, I have been active on the 3i community portal, it is very useful, user-friendly and easy to use. It is keeping me posted on progress made by other SAIs, for example we did not start working on performance audit and we expect to start May 2017 and we post this information on the portal which gives us an indication if other SAIs started or not, we prepared our plan based on ISSAIs and we will post it on the portal next week, it helps us communicate between other teams and assist each￾others in solving problems and learning from each other. We will publish our report on the portal next May. Libya LAB facilitators are actively participating at the community of practice portal and publish their activities and stages of execution of their individual plans however we have few comments on this portal as it clearly lacks supervised communication and there is no publication of latest updates on ISSAIs and amendments that take place on them in addition to the technical complications in using this platform. Morocco Yes, Ms Saeda uses the 3i community portal and she thinks it is a very important knowledge sharing tool however she finds it very complicated and not user friendly CDC considers creating such portal for the use of 3i community for auditors to exchange knowledge at the level of their institution but lack the needed resources. Palestine I think it is useful and can be developed to become a distance learning tool. The problem is with the seriousness of users in dealing with such community of practice. The need to improve their skills in using it. I would like to mention here an initiative by Queen Rania of Jordan named IDRAK where they have developed a good online distance learning tool. In addition; this portal is focusing on communication it should be improved to include ISSAI updates, scientific training materials and informative videos. The portal might be a little complicated for someone with limited IT knowledge and it lacks color and interaction, I think there’s room for improvement. Mauritania Yes I have actively used 3i Community portal and found it very useful but not all participants from my SAI use it. 40 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Sudan Yes, for the sake of participation at the online test by nominated participants Tunesia I do not know, personally did not use it. 9. Monitor, evaluate and report on ISSAI implementation annually. Iraq Not yet we expect to do that after we finish audit work based on our audit plan however we have reported on our progress regarding iCATs, follow up and reporting is a request by our strategic planning at BSA. Libya The ISSAI implementation stages are being followed up and evaluated by planning administration as they follow up the LAB strategic plan implementation matrix. Since the year 2015; LAB publishes within its annual report a special chapter related to internal activities that includes stages of implementation of strategic plan of which includes ISSAI implementation goals We are in the process of preparing annual report for the year 2016 and once it is endorsed it will be published. Morocco No as CDC started in less than one year ago therefore nor reports yet possibly by end of 2017, however they are very optimistic as head of CDC called the year 2017 year of standards and considered the modernization of audit as the highest objective of the year. Palestine Yes, we have created at SAACB a follow up committee responsible for ensuring the implementation of ISSAIs at not only our pilot compliance audit on procurement but all audit reports published by our SAI. To ensure monitoring and follow up we have incorporated at SAACB strategic plan 2017-2021 a complete strategic goal on 3i. Mauritania Not yet Sudan No Tunesia Not yet 10. Use ISSAI facilitators to implement the Action Plan. Iraq Yes, me as a facilitator have assisted in the preparation of training materials and assisted in training and transfer of knowledge on ISSAIs, more than 800 members had been trained at BSA. Libya It was taken into consideration during the preparation of individual plans for facilitators that it should be in line with LAB strategic plan in relation to ISSAI implementation therefore the facilitators within assigned team to execute the work in addition to dedicating facilitators’ time to conduct tasks. Morocco Yes and facilitators signed on the action plans Palestine Yes, As I am one of the facilitators; I have assisted my SAI on the following: Individual projects, I was responsible to follow up with auditors that took part in the 3i programme. Each one played three roles in implementing ISSAIs as a supporter, facilitator and project manager based on an individual work plan. Mauritania Not yet Sudan 6 Facilitators from NAC participated in the training on financial audit manual and one facilitator from NAC participated in the preparation of training material for performance audit and trained on it. 41 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Tunesia I do not know 2. Is the SAIs impeded in the application of the ISSAIs by an inadequate legal mandate? Element of the SAIs mandate Country Situation in SAI Does the legal mandate impede application of the ISSAIs? Iraq No. Law is ISSAI compliant, except for financial independence standards. Libya No, LAB legal mandate within article 28 for constitutional declaration issued in August 2011 and articles 2 and 14 of the Law no 19 for the year 2013 regarding the reorganization of LAB and articles 2 and 13 from by law for the same law and article 41 that ensured that audit operations must be in line with ISSAIs. Morocco No, not at all. Palestine First law for SAACB was back in the year 1994, latest amendment was law no 15 for the year 2004. Article 23 mentioned the responsibilities of SAACB in safeguarding the public money. Within the new amendments that are awaiting cabinet approval we have clearly added financial audit according to ISSAIs. Mauritania No, amended Mauritania CDC law was drafted based on ISSAIs. Sudan No Tunesia No not at all, on the contrary the legal mandate supports the application of International standards Has the SAI legal mandate to carry out financial audit according to ISSAIs (i.e. give an opinion on the Annual Financial Statement)? Does the government prepare an annual financial statement Iraq Yes we carry out financial audit and give opinion on the annual financial statements, BSA in addition to few more Arab SAIs does that (Sudan, Egypt, Palestine and Bahrain). BSA Law no, 31 for the year 2011 amendment on articles 6 and 10. Yes, Financial statements are submitted on 31/ of the following year Libya  government is obliged according to article 11 of LAB law to provide LAB with final accounts by a maximum of 4 months from the end of fiscal year.  Article 14 from the law had obliged institutions subject to LAB audit to provide their accounts to LAB within allocated dates.  Article 14 from by law had obliged independent institutions with independent budgets to provide LAB with their financial reports to audit them within a duration of not more than 4 months from the end of the fiscal year. Morocco CDC does not conduct financial audit as government does not prepare annual financial statements therefore CDC gave the government time until 2018 in coordination and support of the parliament - Work in progress. Palestine Within the new amendments that are awaiting cabinet approval we have clearly added financial audit according to ISSAIs. In reality and in our daily practice we do conduct financial audit and do give an opinion on final accounts we even publish it at our website www.SAACB.PS Mauritania CDC amended law no. 3 for the year 2007 is derived from international standards mainly related to the assignment of the head of CDC. It is in based on the French financial system. As for the financial statements, CDC does not give an opinion on annual statements however provided reports on the execution of budget and table it within annual report to parliament. Currently CDC 42 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD management are developing legal framework to enable CDC give opinion on Annual Financial Statements. Yes government provide annual financial statements. http://www.cdcmr.mr/ar/tegarir/noussouse/mahkema/ Sudan Yes, Article 3 point 2 from NAC law year 2015 audit paragraph: it clearly states to give an opinion on the financial statements according to laws, regulations and Sudanese standards. Article 13 point 1 obliges the institutions subject to NAC audit to provide their final accounts within six months period from end of fiscal year Tunesia Yes, the constitution and CDC law does not disallow and new amendments to take place on the legislation will govern further support to the implementation of financial audit Yes Ministry of Finance prepares annual financial statements based on public accounting law. Has the SAI legal mandate to carry out compliance audit according to ISSAIs? Iraq Yes, BSA law For the year 2011 article 6 Libya Yes, LAB legal mandate is in line with ISSAI practice of compliance audit within article 11 of the LAB law and article 41 of by law that clearly states the importance of being in line with ISSAIs. Chapter five from by law provides an explanation Morocco Yes, Article 25 of CDC law. Palestine Yes, Article 23 of SAACB law number 15 for the year 2004 Mauritania Yes article 4 from CDC law no 93 amended on the year 2007 Sudan Yes, Article 6 point 2 from NAC law year 2015 and previous laws. Examination of contracts and loans agreements and grants to ensure their compliance with laws and regulations and the extent of follow up on its implementation. Tunesia The original law regulating CDC work for the year 1968 and its amendments year 2008 clearly mentioned compliance audit. Chapter 44 (amendments to original law no 3 year 2008 dated 29 January 2008) Has the SAI legal mandate to carry out performance audit according to ISSAIs? (and report on them in separate publications?) Iraq BSA amended law gave BSA the delegation to conduct performance audit according to article 6:2 and 6:4. It included:  Compressive performance audit and partial performance audit  Specialized performance audit  Programmes and policies audit BSA prepares up to 6000 reports including the annual reports. Annual reports, quarterly and specialized performance audit reports, policies reports and they are published at BSA website. Libya LAB law articles 21,22 and 23 in addition to chapter 4 from by law explains how legal mandate is in line with performance audit according to ISSAIs. Currently LAB prepares performance audit reports and they were published within annual reports 2014 and 2015, it tackled government performance audit in a number of programmes related to energy sector and electricity development projects. Morocco Yes, Article 75 of CDC law. CDC issue in addition to its annual report other reports related to public opinion for example a report on pension which was published on CDC website, annual report and official gazette. This report was also discussed at the parliament. 43 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Palestine Article 23 of SAACB law number 15 for the year 2004, also our reports are published at our website. There’s good environment for transparency Mauritania No, the CDC law does not clearly state performance audit nor request it. However it would be a possibility to add it. All information is only published through CDC annual report, which is available on CDC website any other special reports are not allowed to be published according to the law. Sudan Yes, Article 6 point 2 section D states that performance audit examines the system to ensure objectivity and management of resources in economy, efficiency and effectiveness to achieve goals. Article 6 point 2 section (j) states that pre audit, continuous audit and post audit in addition to performance audit environment audit and IT audit, forensic audit for the public money crimes Tunesia The annual report is published de facto on the CDC website according to the law since 2011. And according to the New constitution of 2014 the publication of the report is a right and an obligation. There’s no article to prohibit publication of other reports for example we have published a report years 2011 and 2014 related to control of elections campaigns. Is the SAI independent in reporting on its audits? Iraq Yes, BSA is independent in selecting audit topics, preparing audit plans and endorsing audit plans. No interference, parliament may suggest audit topics and prime ministry and we may add them to our plan as long as it is relevant to safeguarding public money and preventing it from waste. In addition to collaboration with transparency and accountability committees whom suggest focus on specific areas. Libya Yes LAB enjoys full independence in deciding the content of its reports, timing, publication and distribution according to article 53 from LAB law and chapter 8 from by law related to preparation of reports, communicating the, types of them with no restrictions of its contents. Morocco Yes, According to CDC law they issue an annual report based on audit tasks planned by CDC in addition to 2 to 3 audit tasks as per recommendation of parliament. The auditor is independent and enjoys an immunity therefore no pressure by any means on audit report drafting. Palestine Yes, SAACB enjoys independence in reporting on audits, publishing reports, selecting audit topics we even have a manual for selection of audit topics. Parliament may suggest topics too according to the law and we have the right to accept or decline the request. Mauritania Yes, CDC is independent in selecting audit topics and institutions to be audited, it should be listed within annual audit plan. Therefore CDC is independent in reporting on its audit. Sudan Yes, Article 4 point 1 and article 6 section (g) from NAC law year 2015 Tunesia Yes the CDC is independent in selecting audit topics, preparing audit plans and reporting. The new Constitution promulgated on 27 January 2014 specifically spelled out the CoA’ independence and provides provisions concerning the independance of the Judiciary to which belongs the CoA. The Tunisian Court of Accounts has not restrictions to report the results of its audit work and is required to report at least once a year. According to the new constitution, The Court of Accounts reports annually to the President of the Republic, the Head of Government and the Head of Parliament about its findings and recommendations. 44 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Are there any other bottlenecks in the SAIs independence that can inhibit compliance to the ISSAIs? Iraq Yes Financial independence, we audit ministry of finance that is responsible for endorsing our budget. BSA enjoys administrative independence and reports to legislative authority. Libya The non-endorsement of the Libyan constitutions which allows for the independence of LAB according to level 1 of ISSAIs related to establishment of SAI, its independence, authorities and conditions related to assigning and terminating its head. Morocco There is not except for the independent budget standard, status in Morocco is that after the endorsement of CDC budget by parliament a committee is formed to oversight the expenditure but is not by Ministry of Finance MoF, any oversight by MoF has been cancelled. Palestine Yes financial and administrative independence is a bottleneck although it is clearly stated in the law but in reality we are following Ministry of Finance and Civil Service Bureau. Mauritania No as law is iterated from ISSAIs. Sudan Yes, based on NAC law article 21 it is prohibited to terminate auditor general except by the agreement of the country president and the agreement of 2/3 of national council members Tunesia Yes, the follow up on the report is not by CDC, the executive authority does the follow up and through a Supreme Commission. In addition there’s no administrative independence nor financial. CDC is part of the executive authority. Needs financial resources. 3. How is the communication on Programme events by IDI and/or ARABOSAI viewed by staff of the participating SAIs? Country Situation in SAI Iraq With IDI we have an excellent experience with them as they bring qualified experts, high quality training materials and give us examples from experiences of other SAIs to learn from them to enable us develop and continue progressing. It adds value to us and uses latest means of communication such as community of practice and drop box. IDI has more programme and it is easy to enroll in them. As for ARABOSAI; BSA had been elected as the head of the executive board for the coming three years. The procedures of acquiring visa had been an issue and several times we were unable to attend events due to visa. We think that IDI are more organized. Libya LAB is in the process of restructuring according to scientific bases that took into consideration available human resources capacities accordingly most employees look at the participation of LAB in programmes related to ISSAIs as an opportunity to develop and build the HR capacities in addition to LAB vision within its strategic plan “an excellent SAI that works in efficiency and effectiveness and economy based on ISSAIs ” Morocco We have been working IDI as a donor and we are very comfortable with the collaboration. Programmes are very important and relevant to capacity building. As for ARABOSAI we do not benefit much from their website, for example we find resources at ARFOSAI and CREFIAF more useful. IDI training methodology is very useful and we have benefited much from it mainly at standards implementation as we would not have been able to move forward without our commitment with IDI. The signature of the agreement was a good idea to ensure follow up and implementation. The other factor is related to language, we lack resources in Arabic and we seek most of what we need from AFROSAI and CREFIAF mostly in French. We think there is a need to have manuals and guidelines in Arabic for ISSAIs implementation for the three types of audit. Palestine In addition to being a facilitator; I do also assist in the capacity building at Civil Societies Organizations and worked with many donors, truly the programmes prepared by IDI are one of the best and very useful, the way they introduce topics, skills of experts, content of training material has always been outstanding and this also applies on 3i programme. 45 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD As for ARABOSAI; it is not as good, they only have five training programmes and are implemented according to donations received, not much benefit. Mauritania IDI programmes are very useful and relevant to SAIs, we have benefited a lot from them, however in the past more projects were available and apparently less financing is available as less programmes are taking place. The reduction of per diems had reduced the number of interested participants from our side as joining such programmes is costly on them specially if a weekend occur and no full coverage was provided. Arabosai, CDC is active and would like to take part in all events and I personally as committee member at the strategic planning committee but again and due to lack of resources procurement committee were unable to participate due to high costs. I recommend to coordinate with donors to provide funding for such activities. Sudan We found our participation very useful as it widened participants’ knowledge about the programmes causing to improve the capacities of participants’ and our SAI too. Tunesia All participations available events by IDI to CDC were beneficial and added value, it increased my knowledge in ISSAIs, as for ARABOSAI it has been hosted by Tunis CDC 4. How is the quality of the programme events of the 3i-programme perceived by the SAI? Country Situation in SAI Iraq Excellent and we learned a lot from 3i programme. The material was very well prepared and tailored to suit our SAIs, experts were highly qualified and were able to explain any unclear areas. In general I would say this programme is very well organized. Libya We think 3i programme needs to be more detailed and to enable communication and entering into partnerships with IFAC in preparing advanced training programmes realted to financial audit specifically for institutions with special activities. Morocco Although CDC enjoys a very good reputation in Morocco yet we need to stay updated on latest development accordingly implement best international standards. Any organization will not sustain if it stopped improving and for CDC perspective development and improvements will only take place if we are in line with ISSAIs and 3i programme had helped us a lot and saved us time if done by ourselves than with the support of IDI. Palestine Excellent Mauritania We find it good and organized, I personally attend this programme. The content was relevant and objective, it was very useful and it covered the three types of audit. Very beneficial and rich with information. Sudan The 3i programmes and events had benefited us in assessing our SAI capacities on institutional level and at audit task level and it identified gaps for improvement. Tunesia Positive to some extent as the impact is not yet foreseen, we are still at implementation stage. Continuous training and raising awareness will add to improving standards culture. 5. What are the primary difficulties in meeting the ISSAIs (as per financial, compliance and performance audit)? And how does the 3i-programme address these difficulties? Country Situation in SAI Iraq 3i programme is gradual, we were introduced on how to prepare a plan for implementation of ISSAIs then given time to draft ours and return to do the second stage improving the quality of our work, the community of practice was useful as between participants we were able to discuss the difficulties we are facing. Discussing problems and obstacles gave us a clearer understanding and opportunity to learn from other experiences. 46 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Libya  Some auditors lack of awareness of the importance of implementing standards (change resistance)  There are no detailed manuals for the three types of audit  No communication with peer SAIs with experience in the field of implementing standards to learn from them about their experiences in this area.  LAB is still at preliminary stages of ISSAIs implementation baring the fact that it had never been guided in international standards before.  The programme has no communication mechanisms to overcome difficulties Morocco For Compliance and Performance audit we have no difficulties except the financial difficulty as the cost of conducting audit according to ISSAIs is relatively higher therefore if we do not have sufficient resources we can only conduct up to two audits per year based on ISSAIs As for financial audit; a temporary difficulty that we are working on overcoming that is the iCATs requirement for the SAI to provide and opinion on financial statements which we are unable to do but have requested government institutions to do so by the year 2020 and are now working on it and believe time is needed as it is already stated in CDC law. We believe that 3i helps in overcoming these difficulties through strategic planning and managing time it also address the difficulty of change resistance which we do face. Palestine The programme took us through difficulties in meeting ISSAIs during all stages of audit and all types through analyzing the risk. As for SAACB our biggest challenge is performance audit as we are relatively new to it. The financial and compliance audit I can say we are in line with ISSAIs. Mauritania 3i Programme covered it well. Training and good implementation is the way to address these difficulties. The SAI has to be adequate to comply with ISSAIs. Sudan There were no difficulties due to the availability of the community of practice portal that provided the solutions. Tunesia 3i Program had theoretically provided us of means of overcoming these difficulties, we as participants in 3i programme were individually aware of these difficulties but the institution itself and other SAI auditors need to understand the ISSAIs, a methodology needs to be in place . 6. What could be improved in the 3i-Programme? Country Response by SAI Iraq It is important though to continue this programme to cover more participants from SAIs and from our side we are trying to transfer knowledge through training courses we conduct internally to raise awareness on ISSAIs at our SAI but it is not as good as the resources available at IDI and the qualification of experts. We recommend to increase the number of participants and the duration of the events. I would say that we were able to implement up to 80% of what we learned at this first attempt therefore continuation of such programme will definitely improve our level of implementation of ISSAIs. Libya Programme can be improved by incorporating the following: - Conducting condensed training programmes in cooperation with IFAC, entering into partnership with IFAC specifically - Benefiting from advanced SAIs experiences in the field of implementing ISSAId through conducting joint audit operations. Morocco We believe that there is a need to assist in the implementation of standards, it is not enough to only help us to draft a strategic plan and work plans. We believe that long term partnerships and long term projects more benefits will be gained. Continuation of support is vital to the success of ISSAI implementation. Palestine Quality assurance on reports prepared according to ISSAIs for all types of audit, ensuring its proper application of standards which I believe IDI is taking into consideration. 47 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Mauritania To sustain this improvement a continuous repeat training programme for the young auditors must be in place to ensure proper transfer of knowledge. More equality in the number of participants from SAIs. Sudan Increasing the programmes that aim at improving the managerial capacities to manage implementation of work plans. Practical training to implement standards to be done as on the job training through conducting a pilot audit task supervised during the audit task phases (stages). Tunesia More follow up after the events and for longer periods of time as implementation takes time. SAIs should be motivated to implement ISSAIs 7. Did the extension of the scope of the Programme with ‘stakeholder management’ and ‘anti-corruption add value to the 3i-programme? Country Response by SAI Iraq Yes, it is very important mainly due to the very bad ranking of Iraq at International transparency indicators, we need to understand what must be done to improve our scores therefore we have worked on an anti-corruption performance audit report in coordination with integrity commission. We need to learn from other countries’ experiences in this regard. Libya From our perspective; we think that the extension of the scope of the programme to include other institutions such as agencies working at anti-corruption will hinder the progress of the programme as these agencies work according to laws and strategies where SAIs work according to ISSAIs. Morocco For the situation in Morocco there are no stakeholders involved in oversight work therefore no anticorruption institution available. Being a Court of Accounts our prosecutors are within CDC. At this stage we believe there is no added value of this extension. Palestine I do not recommend that, only at the selection of audit topics that might be useful. Mauritania No, I think it is not useful to extend the scope of the programme to include more stakeholders as in the case of Mauritania there’s a supreme transparency committee of which the head of CDC is the deputy therefore the coordination is direct and such committee does not enjoy the same level of independence the CDC has. It is preferable that and coordination to take place through CDC. Such stakeholders lack the knowledge of international standards. Sudan I cannot answer as I have no available information Tunesia I think this is a very important topic, as it tackles ISSAI 12 and being a member at communication committee at CDC would be in place to inform you that we are drafting our communication strategy, we fully understand this extension in the programme scope and support it. We are currently working with World Bank, Netherlands SAI and conducting a SAI PMF 8. Does the SAIs also benefiting from other donor-funded projects? If yes, are the projects coordinated with IDI’s 3i-programme? Country Situation in SAI Iraq Yes, we had a project with the World Bank and developed a risk based audit manual based on ISSAIs, another cooperation with the Netherlands SAI of performance audit and on communication and stakeholders in addition to sustainable development goals. Another project with USAID that we partially participate at. No, we are not coordinating with IDI regarding other donors. Libya Yes, LAB participates in a number of programmes that are organized by international organizations for example World Bank Performance Measurement Framework programme which LAB had participated in two rounds of it. 48 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Morocco CDC had selected peers to improve the implementation of ISSAIs accordingly are cooperating with Netherlands Court of Accounts, jointly conducting pilot audits based on ISSAIs in addition to the French Court of Accounts. During these peer reviews we have selected SAIs that had longer experience in implementing ISSAIs to learn from their experiences and to introduce us to the problems faced during audit according to ISSAIs. Dutch team suggested a performance audit pilot and we choose to improve our performance audit manual accordingly. The programme is called INTOSAINT to improve the level of integrity at SAI through leading by example and the evaluation will take place in May based on ISAAI 12,30 and 40. There is no coordination with IDI however CDC takes into consideration previous programmes so that no duplication takes place. Palestine Yes, we have been coordinating with Sweden SAI mainly in performance audit as they have assisted us during three years in building the capacities of 8 auditors who became experts in performance audit. Another EU project for capacity building that includes a component for audit according to ISSAIs. World Bank Performance Measurement Framework Partnership with Netherlands SAI – sustainable development programme We ensure and according to the agreement signed to notify IDI of our cooperation with other donors. Back in Feb 2016 in Kuwait we have presented training plan and selected from 8 projects 5 that we found not duplicating or overlapping with other donors. Mauritania Yes. CDC is very recently partially and on a very minor level benefit from a World Bank project at the level of the country of which a component tackle the implementation of international standards, this project works on improving training, development of manuals and supports the implementation of standards. No there is no coordination with IDI on this project as it is still new and the budget is very limited. Sudan During the years 2015-2016; NAC benefited from AFROSAI and during the same years from AFROSAI-e in training on performance audit as it won 3 trophies for the 3 participating teams. In addition; NAC benefited from World Bank through financing a Roll Out financial audit manual that was conducted by NAC trainers and NAC funded the printing of the manual. Tunesia Our main focus is on our SAI strategy, we are trying to be in line with ISSAIs mainly at level 2 (independence and quality assurance), the main goal is to make a difference to citizen’s life. We are improving our capacity in communication through donors and try to coordinate to reduce overlaps and duplication therefore we are in continuous contact with IDI on any means of cooperation with donors. We have a clear vision and well-defined strategy This vision is outlined in the CoA strategic plan for the period 2016-2020 : « To lay sound foundations for good governance, consolidate its independence, and modernise and enhance the quality and value added of its working methods » these are the highlights of the Court of Accounts› Strategic Plan 2016 – 2020. Efforts are intensified to enhance the impact and the prominence of the Court of Accounts’ work. Partners who believe in the capacities of the CoA (Netherland Court of audit, World Bank, UNDP, European Union…) 49 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Annex 5 Questionnaire results 1 Background 1.1 In which 3i event did you participate as resource person? Event Product Adaptation Meeting ISSAI Implementation Workshops iCAT Review Meeting Workshop on Facilitating ISSAI Implementation yes 10 12 12 1 no 3 1 1 1 Total 13 13 13 13 1.2 For what type of audit have you acted as resource person? Total Financial audit Compliance audit Performance audit 13 4 5 4 2. On communication 2.1 What is your appreciation of the quality of the communication about the events by the organisers? 1 Inade￾quate 2 Partially inade￾quate 3 Neutral 4 Satis￾factory 5 Excellent No opinion Total Clarity 1 1 11 13 Timeliness 1 5 6 1 13 Responsiveness 1 2 10 13 2.2 Please provide a clarification or examples to support your appreciation:  Sometimes we receive official letters related to our nomination and selection as trainers in a very short notice and directly before the event however we have many internal procedures at our SAIs to approve our participation which is causing us uneasiness and we have already informed organizers of this concern.  Notifying us of the timing of meetings before a sufficient time.  Involving facilitators in the preparation of plans and taking their opinion at all stages.  As I am not speaking Arabic, we had some challenges during the workshop with translators. Hakim who were responsible for all did an excellent job, and tried to the tree resource persons that did not speak Arabic. However, the translators where not familiar with the vocabulary in auditing. 50 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD  During the product adaptation workshop at UAE, the programme objectives were clearly presented and expected results in addition to steps to be taken to reach these objectives.  We worked in UAE for two weeks on modifying and altering scientific material as we have identified programme agenda, distribution of sessions between trainers.  The coordination was continuous with ISSAI programme manager and between training experts regarding programme content until was endorsed in Morocco at its final version.  The same methodology was used at ICAT review and standards implementation facilitation workshop.  3i Programme was very clear in regards to the goals and the outcomes as at the beginning of the first sessions the programme allocated these sessions to introduce and explain benefits reflecting on upgrading credibility of SAIs and improving efficiency of SAI workers and ensuring high audit reports quality.  In addition we always find prompt response from IDI in relation to pending questions, challenges and difficulties we face.  The activities we accomplish with IDI are based on individual and group work within a team of facilitators and usually responsibilities are identified and there is cooperation in preparing scientific material according to INTOSAI methodology under the supervision of programme manager.  The time of selecting me to participate at the events comes directly before the event takes place very short notice which causes me to leave all my other work and stay late at night to finish the tasks requested from me.  The correspondences that took place between organizers and myself during the preparation for the programmes.  The experience of programme supervisor and continues coordination efforts between facilitators in all means (Skype, emails and meetings).  Whenever we face any problem and had inquires; there’s always prompt response and direct reply. In Addition, there’s positive acceptance to any improvement and development suggestions provided by us.  As in all previous experiences with organizer, during this program the objective were explained and all steps, circumstances, phases planned to be implemented by expert organizers, trainers and participants of programme. Explaining the importance of achieving programme goals and the positive impact of it on the performance of our SAIs gave motivation and created an environment of enthusiasm at high level on everyone specially through team work that organizers made sure to maintain. In addition to ensuring maximum benefit from knowledge and scientific material available from accumulated experiences by everyone and respect to this experience with good utilization of it to ensure success of programmes at all phases.  A great deal of trials to harmonize between the volume of work requested and available time with prompt remedial action to any emergency or request in a manner that serves achieving objectives and furnish comfort atmosphere to all members of the group enabling incorporation in work and participation with effectiveness by everyone based on their roles and specialization.  As for timing, organizers were very cautious and all the time to communicate with everybody within a sufficient time frame to reply and to move forward towards accomplishing any task requested in line with the programme time frame that had been addressed within the primary agreement signed by SAIs. 3. On logistics 3.1 What is your appreciation of the quality of the logistics of the events. 1 Inadequate 2 Partially inadequate 3 Neutral 4 Satisfactory 5 Excellent Total Visa 1 5 7 13 51 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD Travel 2 1 5 5 13 Accommodation 2 11 13 3.2 Please provide a clarification or examples to support your appreciation:  Visas are always delayed and late, the reason is the delay in sending invitation letters and official correspondences to our SAIs, it is exhausting although they assist us in communication with embassies and notify them to facilitate issuance of visa.  The travel I fixed myself. We lived at nice hotels, central in town.  Accommodation at hotels was excellent and we received great welcome and care by the Moroccan and Kuwaiti SAI.  As for travel, the long travels and connected flights needs to be reconsidered to become more comfortable.  Visa was an issue to enter UAE in the right time, we arrived very late from workshop start date, 4 days late!  Travel conditions are very uncomfortable as we were flying for more than 8 hours at economy class and after a day of two of complete travel exhaustion we do not take any resting time before programme starts (this only applies on long distances)  As for travelling: the tickets of facilitators are always economy class even if exceeding 6 hours flight and this is not motivating at all.  Trainers meeting to prepare scientific material in Abu Dhabi (March 2015): accommodation acceptable.  Morocco workshop (June 2015): accommodation not acceptable.  iCAT reports review meeting and facilitators workshop (November 2015): accommodation acceptable.  The delay in getting the visa to Kuwait was an example, I only received my visa 24 hours before travelling although I sent all requested papers on time. Travel is on economy class.  Visa procedures and timing of flights.  The visa from UAE was not granted to some of the experts requested to modify the scientific material.  There’s great attention to travel details such as tickets, welcoming us at airport and providing good accommodation. The only exception is the visa as we always face problems and complications in getting it. Few of us were unable to attend first meeting in UAE and few arrived very late after meetings started.  Prompt responsiveness was also sensed here as organizers and directly after I have notified them had amended my travel dates to be adequate to my city’s status and my residency venue that is far from the capital taking into consideration the programme time frame not to effect achieving requested tasks at a high level of quality and accuracy. In addition; there was continuous follow up on us to ensure that our passports were valid and we receive visa’s in adequate time also organizers provided suitable accommodation that was comfortable for everyone. 4. Effectiveness 4.1 To measure the effectiveness of the events, please give your appreciation on the following statements: Statement 1 Fully disagree 2 Largely disagree 3 Neutral 4 Largely agree 5 Fully agree Total The participants came well￾prepared to the events 2 2 6 3 13 52 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD The participants were well aware of the ISSAIs and the implications for their SAI before they joined the workshops 1 2 4 4 2 13 The event(s) were well organized and allowed for a ‘effective’ learning experience of the participants on the application of the ISSAIs 6 7 13 The events were led by resource persons which had enough experience on the ISSAIs to instruct the participants with authority 6 7 13 The participants were well aware of the ISSAIs and the implications of the ISSAIs for their SAI after they joined the workshops 2 11 13 The workshops have significantly improved the ability of the SAIs to implement the ISSAIs 5 8 13 4.2 Please provide a clarification or examples to support your appreciation:  When we start the process of training; we find that participants are always ready for the workshop, they read about ISSAIs and they know that they exist and from our side we explain more for them. They implement what we teach them when they return to their SAI and some of them even train their colleagues and conduct internal ISSAIs courses at their SAIs.  Based on iCATs report; a group of procedures were taken of which most important was creating a quality assurance unit at the level of the Moroccan Court of Accounts.  Participants were introduced to the importance of ISSAIs before programme started.  Selection of participants by IDI through a beforehand test to ensure selecting the suitable segment.  Participants were requested to conduct individual projects ensuring their full / clear understanding of standards.  Not all were well prepared when they came, and I think it is because it is difficult to grasp the ISSAIs before you have discussed them. Some of the concepts are not well known, and maybe a little theoretical.  At the beginning; participants did not show great enthusiasm regarding the importance of ISSAIs and its implementation but with time and specially at the stage of presenting steps of audit and best practices they return really eager to learn and participate.  At the end of the 2 workshops, it was clear that participants were convinced about the importance of ISSAIs and its implementation at their SAIs, they were excited about preparing an implementation strategy and work plan for each strategic option and the role that they should play at this stage.  The design and good preparation of the programme and the sessions contributed to the success of the two workshops in Kuwait and Morocco.  Both the implementation strategy and work plan are considered the road map for ISSAI implementation. However and to ensure success of this programme continuation is important, staying updated on ISSAI and follow up from those in charge of 3i programme. 53 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD  At the beginning of the programme most participants were not paying attention to the standards and some of them thought that implementing them is a waste of time. But after the programme we sensed great enthusiasm and willingness from participants to implement standards caused by their knowledge of its importance to them and to their SAIs.  Such programme is good but not enough to implement standards, we should take the hands of SAIs to ensure that the implementation had started. This can be done by providing them with assistance in implementing strategies.  As for financial audit, there had been some concepts that were already implemented at public sector, these were not very clear within standards, I guess this area of standards needs review and possibly reduce its number and further explain some of them.  Variances between participants.  Morocco workshop; we touched a general prior knowledge from participants’ specially those who attended online test to qualify them to participate at the programme, their skills and knowledge was more concrete specially that they prepared iCATs report, strategic plans for the implementation of ISSAI at their SAIs.  As for the good organization of events, it allows well for an effective learning experience for participants and implementing it at their SAIs: for ARABOSAI, attending the workshops is very important for this type of programmes.  Some participants’ level of knowledge and skills in the area of financial audit is very limited, they actually have not practiced it and I specify here participants from countries like Moroccan, Tunis and Palestine. This caused the increase of time needed to clarify financial audit works and the presentation of requirements of ISSAIs  National Audit Chamber of Sudan experience as we started auditing according to ISSAIs.  Good organization + seriousness of facilitators = reasons for programme success  SAIs actually started implementing compliance audit and financial audit according to ISSAIs. In addition it prepared ISSAIs awareness booklets (guidelines) to educate both those working at top management within SAI and auditors in the field executing the tasks. Such guidelines educate workers about the content of ISSAIs, there importance in improving the quality of audit reports therefore its impact on safeguarding public money from waste.  The picture was not clear to some participants in regards to what will be presented to them about what the implementation of ISSAIs at their SAIs will add value to their work that some of them think that what they already have is complete and that their local standards stated within their accredited procedures before participating did cover what is needed from their perspective. But the programme and through their interaction and exchange of knowledge and opinion between participants, experts, trainers and organizers many concepts were explained that were missed by some participants regarding the benefit from ISSAIs implementation specifically related to the believes of SAIs such as the proof how ISSAIs will assist in the high performance of work and the good quality in addition to creation of a common language between all SAIs contributing significantly in understanding subjects addressed by everyone facilitating the transfer and exchange of knowledge and experiences at highly technical manner between all SAIs. 4.3 What was in your view the main value added of the workshops?  Develop SAIs capacity in audit.  Training and qualifying many auditors at SAIs on ISSAIs and how to implement them, which caused the transfer of knowledge about ISSAIs to the rest of the auditors at their SAIs through conducting ISSAI training at their SAIs.  Awareness on the importance of ISSAIs at the level of SAIs.  Enabling a big number of participants to become ISSAI proficient in one type of audits (financial, performance and compliance).  Unifying working methods at the level of participating SAIs.  Raining awareness about ISSAI inside SAI.  Exchanging experiences between participants.  Getting acquainted about successful experiences at SAIs. 54 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD  The participants got a mixture of theoretical lecture, and practical tasks based on their own material. This made it easier to compare and discuss. When they had one workshop to introduce the ISSAIs and discuss the difficulties first, and then in the next workshop they had made a report on the mapping of their practice against the ISSAIs for discussion. This made them learn faster. In addition, they presented a strategy for how they were going to implement the standards in their SAI.  Realizing the importance of ISSAIs and its implementation at performance audit adding quality, reliability and professionalism to work of SAIs. This requires strengthening and maintaining knowledge earned by assisting SAIs and following up on their implementation of their ISSAI strategies.  Drawing attention to the importance of standards so that SAIs adopt this concept, this will benefit SAIs.  Morocco workshop: it had added value for the development of participants’ knowledge and skills specially that it combined theoretical and applicable perspectives through different types of exercises (individual exercises, group, role play …) Practical case study throughout preparing templates to be accredited and endorsed to be used at the stage of implementation on the level of SAIs  Kuwait meeting: at this stage; all participants were brought together after they have finished the requested tasks at the level of their SAIs to discuss outputs of the work that was presented in the iCATs reports, strategic projects to implement standards, getting feedback from facilitators and other participants. This meeting was an opportunity to ensure the participants capacity to assist their SAIs in implementing ISSAI.  In addition; during this meeting participants’ were trained to become ISSAIs facilitators.  Providing participants with the knowledge and skills needed to implement the ISSAIs at their SAIs participants to know each other, continuous communication with trainers to reply on their inquiries.  Improve the quality of participants.  SAIs recognition for weaknesses in audit procedures that were seen while using iCATs for assessing SAI performance and preparing strategic plans and work plans to fulfill gaps.  The implantation and application dimension that assisted different participants to understand the requirements of standards framework.  Exchange of knowledge between countries and being exposed to the reality of audit work at peer SAIs, also self-evaluate our BSA to understand where it stands in comparison with other SAIs.  Gain new knowledge from scientific materials provided during the events that is always excellent and greatly include practical cases through the exercises and the case studies delivered during the workshops.  Change incentives are created during the knowledge provided throughout the workshops that is transferred to our SAIs from attending the workshops.  Improve the efficiency and skills of the participating teams and empower them with new skills.  Introducing participants to new communication techniques such as Drop Pox and Community of practice – portal.  The most important benefit from this programme is that it provided the opportunity for participants and their SAI to practically apply the information and available scientific material introduced to them. Accordingly it was not just theoretical exchange of information but it also included some discussions from real work of SAI that participant had faced, some participants had previous knowledge about ISSAIs but did not have practical experience of implementing them and what standards stated in regards to conducting audit work requested by them. Furthermore how the link between these standards and their legislations and accredited procedures within their SAIs, in addition to standards statement of good rationalization and harmonization between these legislations and SAI procedures in comparison with standard they plan to consider from ISSAIs.  In addition to the work requested by participants to complete the iCATs from their Sais and to draft ISSAIs implementation plans, and how to facilitate implanting ISSAIs did allow for all of them the opportunity to explore the difficulties that they may face and 55 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD discuss means to finding adequate solutions to them which made the work of each SAI a pilot of which a participant sharing the experience with the rest of participant who can benefit to the maximum in solving problems they face them within their SAIs enabling to achieve the optimal goal that is the up scaling of audit work to the highest quality levels reflecting positively on citizens’ life. 4.4 What was in your view the main weakness of the workshops?  Preparation and doing travel arrangements before a sufficient time from the event.  The problem of English language being the communication language between experts and a group of participants who lack good knowledge of English language.  The problem of translation of interventions from English to Arabic.  The problem of translation of scientific material from English to Arabic.  Variances in the level of experiences between participants.  For me it was to understand the translators. Therefor it is difficult to tell what the weaknesses were. Maybe the participants should have been better prepared. They had not seen the standards before. All three streams, financial, compliance and performance audit could have had a meeting together before they went to the first workshop. Than the discussions would have been even better.  Weakness is basically represented from trainers not being trained before the beginning of the programme. This would have developed the capacities of the trainers and improve the unified understanding of the concepts and more agreement on common terminology because as you know there are differences in some definitions and terms between North Africa and Middle East countries.  Condense of scientific material content / suitable time.  Need for more time, selection of participants based on their practice of audit type.  Despite of the big success achieved from attending these workshops; it would have been better if these workshops were designed for targeting top management at Arab SAIs as thy have the authority to adopt the implementation of the ISSAIs at the filed.  No continuation in implementing ideal – role model audit tasks using ISSAIs.  The difference and variance in the level of participants.  Limitation of time.  Limited number of participants allowed from each Country.  Some assistance provided through foreign experts caused a basic weak point in relation to the workshop I participated at. The language factor and the weak experience in the field of modern training methodology and in the fields of audit work itself in addition to the lack of awareness of circumstances of Arab SAIs did reduce the percentage of gained benefit of what have been expected from this expert. At a certain time it was clear to all of us that there had been a multiplied cost caused by recruiting translators for a long time causing the waste of programme time that is very precious and scarce in comparison with the programme high volume of requirements specially that SAIs did not agree to allocate longer time for the workshops and freeing participants and experts from their daily work for long periods. 5. Recommendations 5.1 Based on my experience, I would like to make the following suggestions for improvement for any further events.  Early preparing for workshops to facilitate full organization of scientific material.  It is necessary to always have an expert to look at the scientific material that is being prepared by trainers to ensure that it is in line with ISSAs and that it is clear enough enabling to reach good results and goals of programme.  The importance of reviewing the translation of scientific material from English to Arabic 56 | Page Mid-term review of the 3i-Programme for SAIs in the ARABOSAI-region WWW.FPO.WORLD  To conduct quality assurance reviews on the reports prepared by SAIs to ensure that they were prepared according to ISSAI.  To conduce a group audit by SAIs.  Repeating the programme another time.  To make all participants better prepared. They should maybe pass a test before they join.  Enabling trainer and giving them the opportunity to benefit from training skills workshops to update their knowledge and strengthen their capacities in regards of training and keep them up to date with ISSAIs and other topics, this will cause more success to training programmes that are continuously developing.  Increasing the duration of the programme.  Organising workshops bringing together experts and trainers.  Improving the Arabic tools at 3i product.  Strong points that I suggest to maintain: the programme had covered the three types of audit (performance, compliance, financial) although that was a burden on programme management but it was positive in unifying methodology for each type of audit especially at the second level of standards. In addition to requesting participants’ to prepare a unified ISSAI implementation strategy.  Allowing sufficient time for trainers to prepare scientific material (Abu Dhabi allocated time to tailor / adapt scientific material was not sufficient) and improving the quality of translation for materials provided in Arabic and English  Clarifying the roles of experts at the programme.  Updating the (everyone available material) such as iCAT manual and implementation ISSAIs booklets so that it is in line with ISSAI 4000 that was recently endorsed during INCOSAI – Abu Dhabi.  Updating scientific material to be in line with ISSAI 4000 that was recently endorsed during INCOSAI – Abu Dhabi.  Allocate sufficient time for workshops based on participants’ experiences and select them according to wok they are trained to conduct specifically that the training on the implementation of ISSAIs not on ISSAIs themselves.  Designing a workshop in the field of ISAAIs targeting Arab SAIs top management.  To implement model audit tasks for each type of audit under the supervision of experts at all stages or phases of audit based on standardized audit manuals.  Continuation of support to SAIs in the same mechanism with a possibility to condense workshops related to this programme allowing an increase in number of participants giving chance for more auditors to benefit.  The maximum benefit from available experiences within Arab SAIs as they are aware of the status at SAIs more than someone lese even if the level of knowledge and skills that such experts have that may be available if not even exceed some foreign experiences participated in this programme. Our experience in this programme was a good proof as organizers, the rest of experts and trainers from Arab SAI were able to fulfill the shortage that had happened to achieve the objectives at the expected level of good performance.