` FINAL Report Assistance for AOCJ Activity Support for USAID/Egypt Cairo TDY Report USAID/M/CIO/BCCS Mr. Freddy Blunt 202-567-4169 fblunt@usaid.gov U.S. Agency for International Development (USAID) Federal Center Plaza (SA-44), Office No. 516G (5th Floor) 400 C Street, S. W. Washington D.C. 20024 This document is a deliverable in fulfillment of services as requested in: BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 CLIN 3: Technology Transfer Program (TTP) Submitted electronically by: AINS, Inc. 806 W. Diamond Avenue, Suite 400 Gaithersburg, MD 20878 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft April 7, 2010 BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report i AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft DOCUMENT CHANGE HISTORY The table below identifies all changes that have been incorporated into this document. Change Control Date Ver. Comments Document Control No. (DCN)* 31 December, 2009 1.0 Draft AID BCCS EG AOCJ TDY RPT 123109 V1.0 5 January, 2010 1.1 Approved Draft AID BCCS EG AOCJ TDY RPT 010510 V1.1 5 January, 2010 1.2 Draft to Egypt Mission AID BCCS EG AOCJ TDY RPT 010510 V1.2 12 January, 2010 Comments from Egypt Mission 2 February, 2010 Comments Received from Egypt Mission 10 February, 2010 1.3 Draft to Egypt Mission AID BCCS EG AOCJ TDY RPT 021010 V1.3 16 February 2010 Comments from Egypt Mission 10 March, 2010 2.0 Prioritized Version [Not sent to Egypt] AID BCCS EG AOCJ TDY RPT 031010 V2.0 15 March, 2010 Request for Report from Egypt Mission 15 March, 2010 2.1 Reordered Version AID BCCS EG AOCJ TDY RPT 031010 V2.1 *DCN – as shown on document cover Actions Date Action Responsibility 31 December, 2009 Initial Draft Submitted AINS VP 5 January, 2010 Government’s comments received (Version 1.1) COTR 5 January, 2010 Version 1.2 submitted AINS PM 12 January, 2010 Government’s comments received V 1.2 COTR 2 February, 2010 Government’s comments received COTR 10 February, 2010 Version 1.3 submitted AINS VP 16 February, 2010 Government’s comments received and Request for Prioritization of Recommendations and a GOE Summary COTR BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report ii AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft Report 10 March, 2010 Submission [to whom? This was not sent to USAID Egypt] of prioritized draft and GOE Summary Report AINS VP 15 March, 2010 AID BCCS COTR’s Comments AID BCCS EG AOCJ TDY RPT 031010 V2.1 15 March, 2010 Revised Final Draft Report AINS VP COTR’s Acceptance BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report iii AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft TABLE OF CONTENTS 1.0 Executive Summary ............................................................................................................. 1 2.0 Glossary of Terms ................................................................................................................ 2 3.0 Background .......................................................................................................................... 3 4.0 Objective & Methodology ................................................................................................... 5 5.0 Findings and Recommendations .......................................................................................... 5 5.1 Infrastructure ................................................................................................................... 7 5.1.1 Security Components ................................................................................................ 7 5.1.2 Physical Security ....................................................................................................... 7 5.1.3 LAN and Server Infrastructure Plan ......................................................................... 8 5.1.4 SAN Configuration ................................................................................................... 9 5.1.5 Power Infrastructure.................................................................................................. 9 5.1.6 WAN Capacity ........................................................................................................ 10 5.1.7 Disaster Recovery Policy ........................................................................................ 10 5.1.8 Security Policy ........................................................................................................ 11 5.1.9 Backup Policy ......................................................................................................... 11 5.1.10 Scanning Process ..................................................................................................... 12 5.1.11 Oracle Components .................................................................................................. 12 5.2 CMS Development........................................................................................................ 13 5.2.1 CMS Design Documents......................................................................................... 13 5.2.2 CMS System Development Process........................................................................ 13 5.2.3 CMS Testing and Deployment Model .................................................................... 13 5.2.4 CMS Workflow ....................................................................................................... 14 5.2.5 CMS Project Documents and Traceability ............................................................. 14 5.2.6 CMS System Reporting and Workflow Management ............................................ 15 5.2.7 CMS Rollout Procedures ........................................................................................ 15 5.2.8 CMS Security and Integrity Features .................................................................. 1516 5.2.9 CMS User Training ................................................................................................. 16 5.2.10 CMS User Feedback ........................................................................................... 1617 5.3 PIC Sustainability ......................................................................................................... 17 5.3.1 PIC Staff Hiring ...................................................................................................... 17 5.3.2 PIC Training Procedures ......................................................................................... 17 5.3.3 Help Desk Planning ................................................................................................ 18 5.3.4 Technical Training .................................................................................................. 18 5.3.5 Maintenance Policy ................................................................................................. 19 5.3.6 Metrics Usage ......................................................................................................... 19 5.3.7 Models and Analytics ............................................................................................. 20 5.3.8 Project Staffing Utilization ..................................................................................... 20 5.3.9 Sustainability Budget .............................................................................................. 21 5.3.10 Project Reporting ..................................................................................................... 21 5.3.11 Quality Assurance/Quality Management (QA/QM) ................................................ 22 BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report iv AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft 6.0 Lessons Learned................................................................................................................ 2223 7.0 Conclusions ........................................................................................................................... 24 Appendix A: Participants and Schedule ...................................................................................... 25 Appendix B: Table of Recommendations ..................................................................................... 29 Appendix C: Standards References .............................................................................................. 32 BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 1 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft 1.0 EXECUTIVE SUMMARY The United States Agency for International Development (USAID) funded the Administration of Criminal Justice (AOCJ) program in Cairo, Egypt in 2006. The USAID Mission in Cairo requested USAID/M/OCIO/BCCS Division to conduct an independent, mid-term evaluation of the operational sustainability of the overall program under the charter of USAID Automated Directives System (ADS) Chapter 548. Under tasking by the USAID/M/OCIO/BCCS Division, AINS responded to this request and evaluated activities in three broad areas: infrastructure, Case Management System (CMS) development and Prosecution Information Center (PIC) sustainability. The team and its on-site schedule are shown in Appendix A. The team’s review methodology included a combination of desk reviews, visit to a CONUS reference site (to set terms of reference), on-site Temporary Duty (TDY), interviews, and independent verification and validation (IV&V), briefings and this TDY Report. The IV&V methodology and report together cover the following: a. Purpose of the program and the intervention b. Information needs and organizational background of host country institution (i.e., Prosecutor General’s Office) c. Review of Information and Communications Technology (ICT) components in the AOCJ program including resources and systems d. Analysis of functional system requirements including critical information flows, systems and products e. Review of the technical architecture including software and hardware f. Analysis of ICT infrastructure - both in-country and AOCJ g. Recommendations for the project management team h. Recommended action plan for the project management team The team’s findings and recommendations to assist the AOCJ in improving the operational sustainability of the overall program are discussed in the report and summarized in Appendix B. The purpose of the AOCJ program is to enhance the capability of Egypt's Prosecutor General’s Office (PGO) through a series of activities, including the automation of nine public prosecution offices and the development of a PIC. The purpose of the intervention was to conduct a high level review of the AOCJ initiative comprising the design and acquisition of a complete infrastructure system, including hardware and CMS, for the nine prosecution offices and the PIC. The assessment included the plans for the CMS pilot in Alexandria, Egypt prior to being rolled out to other PGO offices. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 2 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft Overall, the AINS team found the AOCJ and the USAID in-country project teams to be well integrated and productive with a number of measurable accomplishments to date, and open to identifying ways to optimize operations in the future. The growing number of staff of the PGO in particular was found to be enthusiastic about their work which is critical to future sustainability. The AINS team’s other major findings can be summarized as follows: • The AOCJ project has experienced great success to date through the strong partnership of the PGO, USAID, and in-county implementing partners and contractors; • The AOCJ project effort has achieved foundational capabilities in all three of the evaluated categories; • There are identified gaps in organization, program management support and analysis; • There are potential gaps in system testing, quality assurance, security, help desk, and metrics as well; these can potentially be addressed within the current program resources with active planning; • There are infrastructure and personnel gaps which will need to be addressed as the number of sites expand; and • The program should encourage further development of expertise in architecture, analysis (network and business), and security in order to facilitate and sustain future AOCJ efforts. Our overall recommendation is for USAID/Egypt to review the recommendations provided herein and make a determination of the value and level of effort to pursue each item in order to improve operations and sustainability within the program’s funding, schedule and programmatic constraints. We also recommend a follow up Critical Project Review in the next three months (in a pre-production phase) to provide further independent assessment and corrective input. Additional specific in-depth reviews, e.g., security reviews, are also recommended below. The AINS team’s recommendations along with suggested priorities are listed in Appendix B. Web links to referenced information technology (IT) standards are provided in Appendix C. 2.0 GLOSSARY OF TERMS No. Acronym Full Title 1. ADS Automated Systems Directive of the USAID 2. AES Advanced Encryption Standard 3. AIIM Association for Information and Image Management 4. AOCJ Administration of Criminal Justice Project 5. ARMA Association of Records Managers & Administrators 6. BCCS Business Consulting and Client Services 7. BPM Business Process Management 8. CIO Chief Information Officer BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 3 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft No. Acronym Full Title 9. CISSP Certified Information Systems Security Professionals 10. CMS Case Management System 11. CO Contract Officer 12. COBIT Control Objectives for Information and Related Technology standards 13. COTR Contracting Officer Technical Representative 14. EA Enterprise Architecture 15. GOE Government of Egypt 16. HIPS Host Intrusion Prevention System 17. HP Hewlett Packard 18. ICT Information and Communications Technology 19. ISO International Standards Organization 20. ISS Information Systems Security or Internet Security System 21. IT Information Technology 22. ITIL IT Infrastructure Library standards 23. IV&V Independent Verification & Validation 24. LAN Local Area Network 25. M Management 26. MOJ Ministry of Justice 27. MSAD Ministry of State for Administrative Development 28. NCJS National Center for Judicial Studies 29. NCT Network Transmission Control 30. PIC Prosecution Information Center 31. PGO Prosecutor General Office 32. QA Quality Assurance 33. SAN Storage Area Network 34. SHA Secure Hash Algorithm 35. SLA Service Level Agreements 36. TDY Temporary Duty 37. WAN Wide Area Network 38. XML Extensible Markup Language 3.0 BACKGROUND The subject AOCJ initiative began in March 2006 and comprised the design and acquisition of a complete infrastructure system, including hardware and the CMS, for nine prosecution offices and the PIC. The CMS is being piloted first in Alexandria, prior to being rolled out to other offices. The AOCJ project is currently in Alpha testing with deployment planned to complete in 2010. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 4 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 5 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft 4.0 OBJECTIVE & METHODOLOGY As tasked by the USAID/M/OCIO/BCCS Division and the Cairo Mission, the objective of this IV&V assessment was to conduct an independent, mid-term review of the AOCJ project in Egypt in three broad program areas: infrastructure, CMS development and PIC sustainability. Our methodology included a desk review of documentation made available by the program; a visit to a comparable reference site in State’s Attorney’s Office of Montgomery County, Maryland in order to set the terms of reference for the review; and an on-site TDY to the PGO and other offices in Egypt to make an independent assessment. Certain documentation was not available until arrival in Cairo. One of these was the Requirements Document which was in Arabic and its translation was not justified due to its volume. Prior to the team’s arrival, the documents table of contents were translated and clarifications were provided on several of the network diagrams and datagram. While the team was in-country, the USAID/AOCJ team was very supportive of all information requests. During the TDY, the AINS team performed a series of interviews with USAID/Egypt management and program staff, PGO staff, and the AOCJ in-country implementing partners and contractors. During the course of the intervention, the team documented its understanding of the following in order to complete the assessment and arrive at recommendations:  Infrastructure: Included a review of the ICT infrastructure supporting the AOCJ system. This includes the computers, networks, and legacy systems currently in place or currently under procurement;  CMS Development: Included a review of the CMS’ ability to support the intended business requirements identified within the program effort.  Sustainability: Included a review of the current processes and procedures in place and supporting the AOCJ system. In the context of the above, the outcome of this analysis provides USAID/Egypt with a documented needs assessment, including an initial set of findings and recommendations. It is our understanding that USAID/Egypt and the PGO will analyze these findings and recommendations and identify and prioritize those activities it wishes to undertake. They will request a follow on intervention as and when they see fit. 5.0 FINDINGS AND RECOMMENDATIONS Based on a series of interviews with the representatives from the PGO, the USAID/Egypt team, AOCJ, and contractors, the TDY Assessment Team was able to determine an initial set of findings regarding the current AOCJ project environment within the Egyptian PGO. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 6 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft The team then analyzed the findings to develop an appropriate set of recommendations which USAID/Egypt and the PGO could apply in order to proceed with further implementation under AOCJ. Although these recommendations center on the provision of an Information Technology (IT) solution(s) in three functional areas, their scope is broader in nature, focusing on the IT aspects of these additional four areas: people, processes, infrastructure, and policy. This approach reflects the fact that functional IT solutions are often linked across organizations in broader considerations including the human network and program governance. The findings and recommendations are provided below accordingly. Combined, the set of recommendations provides an integrated approach to the AOCJ implementation necessary for success in both the near and long-term environment. The recommendations vary in the degree of complexity and level of effort required to execute them and have been classified accordingly. The following provides a generalized description of the varying levels of effort:  Low: Relatively easy to implement in the short-term; require a low level of effort; require a low level of financial or staff resources; has little or no dependency on other recommendations.  Medium: Relatively more complex to begin implementation in the short-term; require a medium-term timeframe in which to complete; may require the completion of precedent recommendations/tasks; require some level of financial or staff resources; tend to require more than one interrelated set of activities or tasks.  High: Most complex to implement; require a long-term approach, commitment, and timeframe to complete; may require the completion of precedent recommendations/tasks; require major level of financial or staff resources; requires several interrelated sets of activities or tasks. To assist with scheduling, tasks have also been prioritized by the following levels:  Critical: These tasks should be scheduled within the next 60 – 120 days. They may result in major schedule, cost or performance impacts if not accomplished.  Operational: These tasks should be scheduled prior to completing development and testing. They may result in minor schedule, cost or performance impacts if not accomplished.  Sustainment: These tasks should be completed prior to completion of the project. They may result in minor sustainment issues if not completed. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 7 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft 5.1 INFRASTRUCTURE 5.1.1 SECURITY COMPONENTS Finding: From responses received to questions to the PGO, Amiral (the primary CMS implementing partner) and AOCJ, it was determined that there are no established security standards in Egypt. Security for the system is configured based on the Confidentiality, Integrity, and Availability (CIA) model. It is primarily focused on network, not application level, security. Identity management is provided from the Active Directory. Biometrics are planned but the enrollment and operations process has not been finalized. Encryption uses SHA hashing and AES encryption. NTC (Network Transmission Control) is in place to ensure customers cannot bring their laptop to work and log in to the network with it. The architecture also incorporates HIPS (Host Intrusion Prevention System) to prevent local login. ISS (Internet Security System) is used to correlate logs. Recommendation: The lack of adopted standards by the Egyptian government should be formally documented with one of the external agencies such as MSAD. In the absence of guiding standards, the project should review and adopt as a baseline from the ISO standards of security such as ISO 15408 and ISO 17799 (see Appendix C for links to standards). While the system is still in development, it is clear that effort needs to be focused on remaining details of the system. In particular, target policies and procedures need to be rapidly developed as guidelines for the remaining development. Particular consideration should be given to identity and privilege management as well as to application security. Information risk management should be evaluated as well as operational security of the system. Security training for all levels of users should be incorporated into the training plan based on the developed policies. Recommended Deliverable: Develop security policies and procedures which have been vetted against accepted international standards. Conduct a security audit of the system to include intrusion detection efforts. Continue to audit security and privilege use cases in application development and testing. Examine stronger common logging ties between application and network security. Level of Effort: Medium Priority: Operational 5.1.2 PHYSICAL SECURITY Finding: Because the PGO is a critical national organization, physical security needs to be carefully examined. Basic door access controls are implemented in all server rooms. Environmental security such as temperature and power monitoring is present in varying levels at each facility but is not centrally integrated. All server rooms have windows which are not controlled. In the most exposed HCB facility, bars were placed over the window to prevent access. Cameras are used only in the PIC. The PIC also has an automated shutdown process in place in the event of a power or temperature failure. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 8 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft All other server rooms shutdown in the event of power failure. No prioritization of servers/components shutdown is implemented. No layering of physical security systems or communications is implemented. Recommendation: The AOCJ team should update the security planning for the operation prior to cutover. Ideally, all server rooms should feature integrated environmental and security monitoring. Alarms or common events generated by these systems should be commonly logged and appropriately routed to the local staff and a central monitoring capability such as the help desk. All windows need to have closure alarms. The server rooms themselves should incorporate motion detection as well as video. Preferably, video should be captured both at the entry and within the server room itself. Tamper detection for the door and pass/key lock needs to be implemented if it is not currently done. The team should consider communications redundancy, such as the use of cell systems, when routing alarm notifications. Finally, firm cutover and detailed security procedures for the PIC staff should be implemented as soon as installation is completed. Violations of security procedures should be reported immediately to PIC management. A best practice of security integration is to provide for redundancy in detection. Another is to provide for centralized sensor integration. It is also critical to provide for redundancy in communications. All of these practices are far more affordable with current tools on the market and should be considered for the PIC. Recommended Deliverable: AOCJ should assist in making minor improvements to site physical security and should assist the PIC in setting up logging and routing of security events. Level of Effort: Low Priority: Sustainment 5.1.3 LAN AND SERVER INFRASTRUCTURE PLAN Finding: Seemingly, significant care was taken in developing a robust networking infrastructure in the PIC facilities. As part of the installation process, the infrastructure was validated by a third party; however, the requirement to implement the system within often aging facilities has forced some compromises in placement which are unavoidable in these conditions. This includes placing equipment in various offices unsuitable for housing equipment, or in uncontrolled closets. Minor details in wiring, patching and rack configuration remain to be completed. Recommendation1: Prior to cutover, all final details of completion for each site need to be identified, listed, tracked and formally resolved. Completion of the punch lists needs to be tracked by the project manager. Recommendation2: As procedures are developed for operation of the site, these should include scheduled, documented inspections by the site team of all uncontrolled areas within the facility. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 9 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft These should be conducted weekly to prevent system compromise or degradation. Any issues should be remedied and reported immediately to PIC management. Recommended Deliverable: Appropriate punch lists and operational procedures must be instituted. Level of Effort: Low Priority: Sustainment 5.1.4 SAN CONFIGURATION Finding: SANs are fully-powered IBM units and SAN architecture is as expected. SAN traffic is segmented from all other network traffic. Though all parties have reported adequate performance, no load testing has been performed. Updated data-sizing estimates based on new sizing information have not been completed. Validating information segmentation for appropriate use was out of scope for this short visit. Recommendation: SAN performance during load testing must be monitored, measured and logged to validate that throughput capabilities are appropriate and sustainable. Update data sizing and validate that SAN capacity will sustain the solution for projected levels of use. Validate information segmentation on SAN solution to ensure smooth performance. Recommended Deliverable: Perform detailed SAN architecture review, capacity and performance evaluations. Create and implement procedures to review and report on performance and capacity. Level of Effort: Low Priority: Operational 5.1.5 POWER INFRASTRUCTURE Finding: It was reported during the TDY that some facilities, primarily the HCB, experience power outages and other fluctuations. PIC management also reported at the close of the study that planning had already begun on a future move of the HCB to more modern facilities. Recommendation: While the implementation is ongoing, the AOCJ team should implement a means of monitoring and recording power issues at all of the sites. These outages should include those which occur off hours necessitating some level of automation. This monitoring effort should be continued throughout the project cycle to determine what level of issues, if any, exist. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 10 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft Recommended Deliverable: Monitoring effort and any required equipment. Level of Effort: Low Priority: Sustainment 5.1.6 WAN CAPACITY Finding: Basic sizing estimates of the WAN channel requirements were developed by the Amiral team in the initial design. An approach has been taken in the architecture to minimize WAN requirements. As would be expected, resultant capacity is also limited by cost considerations. According the implementation team, all communication lines utilize an alternate route, hot spare approach for failover where possible. According to our discussions with the team, failover and capacity have not been currently tested. Similarly, no detailed communications channel modeling has been completed to account for channel loading variations, response or latency. Further, analysis has been limited to the initial architecture and does not address any potential expansion. While the TDY team has no current reason to believe the system has any compromise as a result of the planned WAN capability, it will be prudent to conduct more detailed analysis in the this area to avoid any risk. Furthermore, specific loading tests should be conducted within the test bed and on the live network prior to cutover to validate expected performance. The needed skills to conduct this analysis are certainly within the current Amiral/AOCJ team. Recommendation: A follow on WAN analysis should be conducted based on an updated model. This analysis should account for different scenarios of channel usage and should be validated by specific testing efforts. Recommended Deliverable: An updated analysis of anticipated WAN usage. Level of Effort: Medium Priority: Operational 5.1.7 DISASTER RECOVERY POLICY Finding: Basic backup procedures are in place and executed in a regular fashion. The PIC center has local equipment redundancy as well as storage and communications redundancy. In the event of a total communications failure, each site is intended to operate in a standalone mode until WAN communications is restored. There are redundant WAN connections to each hub and the Alexandria site has redundant disaster recovery hardware in the event of an outage at the PIC. While physical provisions are largely in place, planning needs to be completed to fully realize available capabilities. Amiral, the system integrator has been tasked with providing a disaster plan. This plan must clearly address implementation, testing and updating to protect critical PIC processes in the event of failure. Recommendation: A complete disaster recovery policy with accompanying procedures needs to be developed and implemented in the near term. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 11 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft Likely scenarios and appropriate responses should be developed. Standards of recovery need to be determined based on the organization’s business requirements. The plans must include disaster recovery for the local offices as well as the PIC operation. Depending on the criticality and resources, the AOCJ program may consider a reduced disaster recovery capability where only critical capabilities are recovered and performance metrics are relaxed. Once implemented, disaster recovery test plans need to be executed on an annual basis. Recommended Deliverable: Develop, measure, and monitor disaster recovery policy. Level of Effort: Medium Priority: Sustainment 5.1.8 SECURITY POLICY Finding: Basic security provisions are implemented and infrastructure tools such as firewalls and virus scanning security applications at the server and desktop are in place. No organization￾wide security policy or procedures exist. Basic password standards are implemented. Biometric security is expected to be incorporated. Recommendation: Develop and implement a security policy including application, physical security, logon credentialing, network security, file encryption, integrated monitoring, and auditing. Updates of security systems should be automated. In parallel with development of a security policy, an audit should be conducted of the AOCJ system based on one of the common criteria models. PIC should play a role in the periodic test of security procedures put in place. Recommended Deliverable: Develop, measure, and monitor security policy. Appoint a security lead. Level of Effort: Medium Priority: Operational 5.1.9 BACKUP POLICY Finding: Basic backup procedures are in place and executed in a regular fashion. Only databases are currently backed-up remotely. Recommendation: A complete backup policy with accompanying procedures needs to be developed and implemented in the near term. Standards of recovery need to be determined based on the organization’s business requirements. A plan for offsite backup of electronic archives needs to be developed. Synchronization features between the electronic archive and the database backup must be implemented. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 12 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft Recommended Deliverable: Develop, measure, and monitor backup policy. Level of Effort: Medium Priority: Operational 5.1.10 SCANNING PROCESS Finding: The currently planned CMS document scanning process closely follows current manual processing. Standards of performance such as acceptable processing time have been discussed but have not been finalized. Establishment of dedicated scanning areas is a potential problem which must be addressed. Operational policies such as a clean desk operation have not been established. Recommendation: The operational workflow may have bottlenecks depending on document arrival time but is likely adequate given the volumes and high quality of labor. Alternatively, a ‘follow the work’ model could also be considered where labor is cross trained and used to optimize capacity. As mentioned in recommendation 5.3.11, a QA process should be introduced and can be used to validate this. Consideration should be given to planning future scanning areas. A proper preparation area should be centralized, easily accessible for all operators, and equipped with all materials needed such as staple removers, tape, staplers, etc. It should be collocated with a copier, so operators can use the copier instead of the scanning process. A clean desk policy where all scanning jobs are closed out by the end of the day should be a primary consideration. All work sub-processes should feature counts and comparisons to ensure that no work is lost prior to scanning. Acceptable SLAs for the scanning area should be clearly agreed to and incorporated into the operational procedures. Recommended Deliverable: Identify operational SLAs and detailed workflow processes for the scanning operation. Level of Effort: Low to Medium Priority: Sustainment 5.1.11 ORACLE COMPONENTS Finding: Oracle is an underlying component of the CMS architecture. Recommendation: While not a specific recommendation, the AOCJ team should understand that the Oracle system is a high-end solution not fully utilized in the current CMS solution. It is expected that the project has already given consideration to the vendor support and life cycle costs of this technology selection. Availability of in-country resources and support should be understood. As with other major partners, the PIC should engage Oracle on an ongoing basis in service and product planning for the operation. Along with other service vendors, Oracle professional services should be evaluated where appropriate for use in short term, high criticality technology tasks. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 13 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft Recommended Deliverable: Establish a continuing education process for the PIC team in Oracle applications, particularly in the Government sector. This can be accomplished without additional cost through open source IT media and Oracle sources. Additionally, the Egyptian Government subsidizes IT education through its Information Technology Institute (ITI). The Oracle core tool set represents a tremendous capability which can be widely applied to multiple Government applications and supported by the PIC. Level of Effort: Low Priority: Sustainment 5.2 CMS DEVELOPMENT 5.2.1 CMS DESIGN DOCUMENTS Finding: CMS Design Documentation is fairly robust within the Amiral development environment. Design documents within the documentation library do not reflect all updates to the system. Recommendation: Maintain documentation library to reflect updates to the system as implemented by the software vendor. Recommended Deliverable: Develop policies and procedures for regular documentation updates. Level of Effort: Medium Priority: Sustainment 5.2.2 CMS SYSTEM DEVELOPMENT PROCESS Finding: Amiral incorporates adequate separation of code between development, testing, staging and production builds. Recommendation: None Recommended Deliverable: None Level of Effort: None 5.2.3 CMS TESTING AND DEPLOYMENT MODEL Finding: CMS testing is performed primarily by Amiral. Subsequent testing is completed by AOCJ and PGO personnel. Industry best-practices dictate that an independent testing team performs testing instead of the developers doing the testing. While this is in place to a degree, significant quality gains could be realized through the use of an independent test team to review all changes at both system and functional levels. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 14 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft Recommendation: A dedicated QA process and team should be established and implemented to verify scope and requirements deliverables. Recommended Deliverable: Dedicated, independent QA processes with stated constant improvement goals. Level of Effort: Medium Priority: Operational 5.2.4 CMS WORKFLOW Finding: CMS workflow is incorporated at the object level within the PCMS application instead of using an existing and mature rules engine such as the workflow engine built into Oracle Universal Content Management. Recommendation: A detailed analysis of CMS workflow was outside of the scope of this IV&V effort and the team did not have detailed time to view the CMS system. Improvements in all of these areas will help the sustainability and operational effectiveness of the CMS system. Recommended Deliverable: Identify CMS product roadmap and cost benefit of possible automation process steps. Level of Effort: Medium Priority: Operational 5.2.5 CMS PROJECT DOCUMENTS AND TRACEABILITY Finding: Basic project artifacts are incorporated in the CMS project but are not timely enough in their development to have maximum impact. Some artifacts detailed elsewhere in this report such as policies or load analysis should be completed as early as possible to provide maximum value to the project prior to design and development activities. Recommendation: The CMS project needs to have timely production of core artifacts at the proper time in the project lifecycle. These artifacts include functional requirements, technical requirements, design documents, use cases, and test plans. Traceability through these documents will become more critical to success as the complexity of the environment grows. The discipline in the production and enforcement of these will be valuable to the PGO not only with current efforts but all future projects. Recommended Deliverable: Review current project documents for completeness with project management standards. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 15 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft Level of Effort: Medium Priority: Operational 5.2.6 CMS SYSTEM REPORTING AND WORKFLOW MANAGEMENT Finding: Though we asked, we did not see examples of CMS System Reports as the reports have yet to be designed and developed. The systems side of the application is a little light and should be significantly enhanced. Systems and workflow features within the application are built-in to the application objects and will require vigilant oversight to ensure that they are maintained and updated regularly to deal with business changes that may arise. Recommendation: Develop robust systems tools. Develop, implement and measure workflow management policies. Recommended Deliverable: Develop systems and Workflow Management policies, procedures, and reports. Level of Effort: Medium to High Priority: Operational 5.2.7 CMS ROLLOUT PROCEDURES Finding: Amiral incorporates adequate separation of code between development, testing, staging and production builds. Recommendation: As the product moves into beta and production release versions, consider regular builds on a weekly or monthly basis to incorporate requested updates and systems enhancements. It is critical to maintain regular maintenance windows for systems updates and to alert users as appropriate regarding potential systems outages. Guidance on new updates to the application should be provided to all affected users and included as updates to online help files, which do not currently exist. A dedicated QA team, as described in 5.2.9, would ensure that regular builds are tested adequately before moving the build to production. Recommended Deliverable: Updated rollout policies and procedures including user alerts and guidance on new functionality. Level of Effort: Medium Priority: Sustainment 5.2.8 CMS SECURITY AND INTEGRITY FEATURES Finding: CMS security and integrity are addressed through two major approaches: the underlying technology set and iterative use case development. A number of privilege levels have been developed to account for different business requirements. As was mentioned in section 5.1.1, identity management is provided from Active Directory but the enrollment and operations process has not been finalized. As with the use cases and system requirements, it was not BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 16 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft possible for the team to review in detail all potential security scenarios. However questioning of the design team and test team around sample use cases identified some scenarios where security and integrity processes must continue to be refined. It must be noted that these issues are to be expected in any development as complex as the AOCJ project - particularly around areas of exception handling. The solution for this is create a focus around security and integrity throughout the development/testing cycles so that issues are identified, prioritized and solved prior to operation. Recommendation: A project focus must be made for the identification of security and integrity issues within both the application and overall system. All use cases should be audited for potential security and integrity issues. These should be also evaluated in testing phases. Potential exploits of the system should be indentified in a security audit. Recommended Deliverable: Conduct security/integrity audit of CMS system and on-going evaluation of use cases prior to completing development. Level of Effort: Medium Priority: Operational 5.2.9 CMS USER TRAINING Finding: There is a level of training in place, though the timeliness of the training was not planned to match the software release schedule. Many users who have been trained on the system will need updated training to accommodate a lack of exposure between initial training and system rollout. There is no online help document, which would mitigate some risk associated with the timing of training sessions, providing users the ability to self-train on concepts not freshly committed to memory. Consideration should also be given to how any resultant on-line help capability is related to Help Desk capability being currently planned. Recommendation: Coordinate training with systems releases. Plan for rolling out training sessions for new users and skills refreshment. Incorporate online Help documentation to assist users after training. Recommended Deliverables: Updated training policies and procedures. Online Help documentation. Level of Effort: Low Priority: Sustainment 5.2.10 CMS USER FEEDBACK Finding: All user feedback is currently manual and ad hoc. During our interviews, users reported general acceptance of this format and seem happy with the results. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 17 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft Recommendation: Include an automated feedback mechanism within the application. This should be incorporated into an overall Help documentation module, which is currently missing from the application. Recommended Deliverables: Implement User Feedback and online Help mechanisms. Report on user feedback findings regularly. Level of Effort: Medium Priority: Sustainment 5.3 PIC SUSTAINABILITY This includes a review of the organizational and human resource structure in place to support the AOCJ system including PGO, PIC, and contractor staff. 5.3.1 PIC STAFF HIRING Finding: A final PIC staff structure has not been determined. Hiring has taken place for lower level technical positions only. Recommendation: Complete PIC organizational planning and identify needed positions. Recommended Deliverable: A final approved PIC organizational structure with associated budget and schedule planning. Level of Effort: Medium Priority: Critical 5.3.2 PIC TRAINING PROCEDURES Finding: A detailed training plan has been developed for the project. Training for PIC staff to date has largely focused on underlying technical applications with limited training by the Amiral team. Such training as is provided by Amiral is usually observation based. Opportunities to work with the system are limited as Amiral is still responsible for operation until handover, however, a risk is present that the PIC staff will be underprepared to take operational responsibility for the system after cutover. This may necessitate Amiral’s continuing support. Recommendation: Development of operational technical procedures needs to be focused on. This should be an early added responsibility for the expanded AOCJ role identified in 5.3.8. As these are finalized and vetted, these activities are an excellent place for PIC technical resources to begin to meaningfully engage in the project without added risk. Recommended Deliverable: PIC technical procedure development. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 18 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft Level of Effort: Low Priority: Operational 5.3.3 HELP DESK PLANNING Finding: No current help desk capability exists to review. The planned HP Openview solution has not been implemented and planning is minimal. Detailed planning is set to begin during December 2009. In discussions with project personnel, there were valid reasons to postpone this effort but the appropriate staff is in place to begin this effort. Recommendation: Incorporation of a help desk is significant to the success of the AOCJ program. The implementation must incorporate the Level 1 staff within the prosecution offices as well as the supporting vendors. Strong local operations may complement higher levels of support but must operate in an integrated model. It should serve as the single point of entry for issues in the system. The help desk must encompass an implemented desktop management solution. The help desk should incorporate troubleshooting trees for each of the supported applications/systems. Information gathered in the help desk system in supporting users should feed back into the training plans. A traffic model for the help desk must be updated with a proper set of requirements. Staffing plans, funding, resources, and training must be identified in the planning effort. Recommended Deliverable: Help Desk Management Plan with SLAs, and designated lead (from the AOCJ as responsible for the effort). Level of Effort: Medium Priority: Operational 5.3.4 TECHNICAL TRAINING Finding: Significant investment has been made in the technical and support staff at both the PIC and in the prosecution offices. All facilities feature capable training areas adequate for initial and refresher training. However, while vendor training will be provided in the cutover of these operations, skills must be preserved and extended to enhance the technical effectiveness of the PGO staff. An on-going technical training plan would broaden and preserve the technical base. It is also concerning that none of the staff interviewed has progressed towards higher forms of technical certification. While certifications themselves are not a final guarantee of technical capability, they 1) create a more rewarding and challenging environment, 2) continue to build a strong reputation for the PGO and PIC, 3) reduce reliance on outside contractors for core skills and 4) can reduce attrition of technical staff if properly structured. Recommendation: Resources should be directed towards the establishment of an ongoing technical training program for PGO staff. A body of self study materials should be available at BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 19 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft each facility. This should include training for the common Microsoft, ARMA, AIIM, and CISSP certifications. Staff should be encouraged and rewarded for the completion of desirable certifications. The program could consider reimbursement of fees for successful completion of testing. This would serve to provide the government with a consistent supply of technical expertise and would also serve to attract professionals into government service. If possible, support for training and testing could be tied to an additional window of commitment to employment at PGO. It cannot be over emphasized that the success of all IT related activities in this report depends on an adequate supply of trained technical staff to support the PGO IT infrastructure. Recommended Deliverable: Identify ongoing training activities for key operational and support resources. Level of Effort: Low Priority: Sustainment 5.3.5 MAINTENANCE POLICY Recommendation: A complete maintenance policy with accompanying procedures needs to be developed and implemented in the near term. The policy must include all elements of policy from daily support, contracts, and end-of-life planning. Standards of support need to be determined based on the organization’s business requirements. Any updated agreements with vendors reflecting maintenance needs should be executed. Recommended Deliverable: Develop, measure, and monitor maintenance policy as well as vendor agreements. Level of Effort: Medium Priority: Operational 5.3.6 METRICS USAGE Finding: Metrics or business process measurements are now only being considered for the PGO automation at the business level. Comprehensive metrics do not exist at all at the technology level. Recommendation: In parallel with the effort to introduce metrics for PGO business operations, metrics should be defined for all of the supporting technology services. These metrics must span business processes, infrastructure, help desk, and vendor SLAs. Some metrics will also be derived from the analysis efforts recommended in sections 5.1.6 and 5.3.7. Metrics must be supported by system generated data and should be accessible through dashboards or traditional reports. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 20 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft Recommended Deliverables: Complete business metrics plan and optimal system metrics to support that plan; examples of bench mark metrics; and periodic performance reviews. Level of Effort: Medium Priority: Sustainment 5.3.7 MODELS AND ANALYTICS Finding: There is little use of analytic tools for tracking and prediction of AOCJ project efforts. Most planning was conducted in the system design phase and has not been updated. Workload volumes have not been updated since 2007. Recommendation: At a minimum, models should be developed and/or updated for scanning, end-of-life backups, storage, and network and server utilizations. Current work volumes from all pilot sites should be gathered on a monthly basis and models should be updated. System performance metrics should be compared to these models on a monthly basis. These results should also be used to support the PGO’s hardware management policies. Testing should occur to validate performance expectations. Recommended Deliverables: Updated analytic models for recommended areas of scanning, backups, storage, network and server utilization. Periodic reports comparing system metrics during test, implementation and operation. Specific test cases constructed around system performance to validate model findings. Level of Effort: Medium Priority: Operational 5.3.8 PROJECT STAFFING UTILIZATION Finding: The PIC has strong top leadership and has filled some entry level technical positions; however, with the project rapidly moving to completion, there are no mid level counterparts within the PIC to optimally guide vendor development in critical areas such as system security and quality. Several comments from the AOCJ team and vendors indicated that major staff gaps will impact the project in the coming year. Recommendation: The AOCJ has capable midlevel staff which can act in support of the PIC in the near term to ensure optimal project development. We recommend that the AOCJ identify responsible leads in security, quality, project management, business planning and training/communications. With guidance from PIC leadership, these team members should have identified deliverables and activities in support of the PIC while the organization is being built. Two primary goals will be to review vendor deliverables in their assigned areas as well as to develop processes and procedures to be used by the PIC organization after cutover. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 21 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft Recommended Deliverables: An AOCJ Transition Plan to utilize AOCJ staff members to compliment the PIC team while they are building their organization. Level of Effort: Medium Priority: Operational 5.3.9 SUSTAINABILITY BUDGET Finding: An effort was completed to develop a five (5) year budget for the PIC system. It does not provide a clear, consistent, logical breakout of budget items. These are variously presented as item composites across the network, per site or around individual sites. No backup data exists for certain costs such as training, seminars, workshops or study tours. No basis is provided for escalators which are shown standard at 10 percent. Fleet refreshment is not consistent across similar hardware such as scanners and printers. Critical positions, such as the QA and security leads, identified in workshop planning are not accounted for. Support staff such as scanners is also not accounted for. Future costs for items such as furniture and facilities work are not accounted for. No provision is made for power or facilities costs. Future year maintenance costs for server and communications equipment are not included. No budget is included for emergency items. Recommendation: The budget should be consistently structured around site/item for project clarity as well as future flexibility. It should be updated with all missing items and a clear basis for assumptions documented. For planning purposes, items such as support staff or facilities which may be carried under other non-PIC budgets should still be accounted for. Recommended Deliverable: Updated budget for submission to the PIC. Level of Effort: Medium Priority: Critical 5.3.10 PROJECT REPORTING Finding: Project reporting to USAID has not been centrally managed in the past and suffers from myriad inconsistencies. Project tasks are often of a general nature and are not measurable. The current project plan is largely focused on equipment deliveries and installation. While it includes software development milestones, these are not all inclusive. Largely missing from the central reporting structure are larger project issues such as risk mitigation tasks or PIC activities. Linkages to actual project deliverables or payment milestones are incomplete; the project plan should always have strong linkages to other significant project documents such as the budget and risk register. Required resources for task completion have not been clearly identified. Everywhere possible, task completion should be measurable. Many project documents such as the risk register have not been regularly updated and must be better structured to be usable. Recommendation: Project management deliverables from the AOCJ team to USAID should be restructured to conform to PMI standards. Execution of this should be the responsibility of the BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 22 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft AOCJ Chief of Party (COP). The project plan and other major project documents such as the risk register must be reworked. Financial reporting should be consolidated into one updated document for USAID. A tight schedule should be developed to revise project reporting to USAID and should culminate in a critical project review. Prior to finalizing new deliverables, the AOCJ COP should have formats validated by USAID. Staff placed in transitional roles for the AOCJ as identified in 5.3.8 should present their different areas in this review as well. Recommended Deliverable: Revised reports, procedures and a critical project progress review. Level of Effort: Medium Priority: Operational 5.3.11 QUALITY ASSURANCE/QUALITY MANAGEMENT (QA/QM) Finding: Quality assurance is not formalized in the project and is highly dependent on human capabilities. Audit tools are available but quality as a discrete process is not built into the current CMS system. Recommendation: Quality should be introduced into the CMS workflow at several steps in the process, in particular the front desk and electronic archiving steps. This could be introduced as a discrete part of the workflow or done post-capture through viewing tools currently in the system. In general, QA is best done systematically prior to the user visibility to prevent erosion of confidence in the system. The amount of QA done in the system should be statistically significant but for the volumes of the CMS system can be from 1% to 5% of the transactions. QA can be used to audit individuals or groups. It can also be used as part of a training transition to monitor the work of newly hired employees or as part of specialized audits. The QA step itself can be executed by a departmental supervisor, a discrete QA team, or by the IT department. Recommended Deliverable: Quality Assurance/Quality Management Plan Level of Effort: Medium Priority: Operational 6.0 LESSONS LEARNED The following summarize the lessons learned from this engagement with regard to the evaluation itself: 1. Having an active sponsor in Ms. Laura Gonzales was imperative to the success of the intervention. 2. The leadership of the USAID/M/OCIO/BCCS Division Chief was key to the preparation and successful execution of this intervention. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 23 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft 3. While the decision was made to wait until the automation was further along in its development before seeking a review of the project, intervention in the earlier phases of the AOCJ initiative would have provided a valuable roadmap for this large program funded technology initiative. The benefits would have been as follows: a. Program governance and artifact reuse (per Clinger Cohen Act and ADS 548, see Appendix C) guidelines would have been enunciated and included in planning and acquisition strategies; b. Baseline IV&V and subsequent Critical Project Review milestones would have been identified and planned for 4. It helps to maintain continuity of IV&V resources that are familiar with previous assessments and expectations; this reduces IV&V and overall program costs by independently identifying required corrective actions. 5. Visit to reference site(s) by the IV&V Team prior to undertaking the TDY were key to the development of terms of reference for the AOCJ intervention thereby increasing prospects for technology transfer. 6. All significant decisions and agreements should be documented between the Project and USAID. 7. Applicable IT standards (e.g., Project Management Institute (PMI), ITIL, ISO 20000, COBIT – see Appendix C) should be adapted and adopted at the start of the Project; initial planning activities (such as the IT Workshop) should be followed up immediately and consistently throughout the key phases of the Project. BCCS TTP services should be requested, as applicable, to assist. USAID’s Development Experience Warehouse (DEC) should be utilized for leveraging available experiential information from other relevant USAID programs. Please see Appendix C for all standards references and links. Other applicable sound practices and lessons learned include the following:  To the extent possible all significant decisions and buy-ins by government counterparts and stakeholders must be documented; this includes sustainability requirements of Projects;  Risk should be actively and objectively anticipated and avoided not just managed, e.g., the risk of over promising stakeholders and government counter parts should be avoided by not making recommendations that are not adequately covered in the Project budget or may risk the sustainability of the Project over its potential extensions;  Project budgets should include a Management Reserve that may be required to pay for unbudgeted expenses (e.g., improvements listed in this report) in the later phases of the project;  Requirements gathering and analysis phases and activities should be staffed jointly by technical staff, as well as, stakeholder representatives possessing business knowledge;  Multiple, independent interventions should be planned to assist the Project in staying on track;  Project staff should actively manage outside resources (e.g., through active governance and application of standards); this can be achieved with the help of BCCS TTP services; BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 24 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft  The business case for outsourcing should be carefully examined across available resources to leverage in-house staff, host country sources and out-of-country resources. 7.0 CONCLUSIONS The AOCJ program has experienced significant success to date through the partnership of the PGO/PIC, USAID, and their in-country implementing partners. The teams appear to be well integrated both at the operational and the executive levels. While at different states of deployment, the three evaluated areas are capable of meeting the organizational requirements. Looking to the future, the PGO plans to significantly broaden functionality as well as complete deployment. The project is also at a critical point of transition and we strongly recommend that the PIC should implement capabilities for sustainability. These capabilities must include a deeper staff structure, detailed planning and procedures. The project organization itself, including the AOCJ, must also focus on the work needed to successfully transition the PIC from development into an operational status. The PIC leadership very clearly recognizes the importance of this effort. The project organization needs to both complete the development/deployment cycle as well as organize itself to support the needs of the PIC transition to operations and maintenance phases. Recommendations in this report have identified areas where the AOCJ team can focus itself to complete deployment and optimally support this transition. Success for AOCJ has been achieved so far through strong leadership and key individuals as well as contractors. To complete deployment and move into operation, the AOCJ program and PIC will need to continue to grow the organization, broaden the skills, and mature the processes/technologies that have brought it success thus far. Several specific plans and reviews are noted in the report. Recommendations and suggested priorities are summarized in Appendix B. Links for referenced standards are provided in Appendix C. A critical program review is recommended during the pre-deployment phase (approximately three to six months from the initial intervention). A Summary Report is provided for the GOE. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 25 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft APPENDIX A: PARTICIPANTS AND SCHEDULE The AINS Team comprised the following staff: Mr. Ron Mizrahi (AINS) Mr. John Doby (AINS) – Chief of Party Ms. Yasmine Zaki (Independent Translation Services) Their in-country sponsor is Ms. Laura Gonzales, USAID/Cairo Contracting Officer’s Technical Representative (COTR). The AINS Team’s other in-country contacts are listed in the chart below. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 26 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft TDY Schedule Egypt ADS 548 IV&V Review- Onsite Mission Review 29 November, 2009, Sunday, Cairo, Egypt  Meeting with Laura Gonzales/USAID and Gary DiNoia AOCJ Chief of Party  In-Brief discussion of schedule and logistics  Agreement on follow up actions (data collection, documentation) 30 November, 2009, Monday, Cairo, Egypt  Meeting with Laura Gonzales/USAID and Gary DiNoia AOCJ Chief of Party  In-Brief discussion of schedule and logistics  Meeting with David Selman and IT Team discussing project management, architecture, network, databases, security, storage  Agreement on follow up actions (data collection, documentation) 1 December, 2009, Tuesday, Cairo, Egypt  Meeting with Counselors Mahmoud Youssef and Tarek Kamel at HCB  Tour of facility  Meetings with technical staff and users 2 December, 2009, Wednesday, Cairo, Egypt  Travel to Quattameya  Introductions  Tour of facility  Meetings with Technical staff and users 3 December, 2009, Thursday, Cairo, Egypt  Site visit to TEA  Conduct any follow-on meetings, revisits  Preparation of presentation (English, Arabic if needed)  Coordination with Translators for report 4 December, 2009, Friday, Cairo, Egypt  Review CMS and PIC/Regional IT Documentation  Work on Report Findings  Interim conference with US team BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 27 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft 5 December, 2009, Saturday, Cairo, Egypt  Work on Report and Findings  Travel to Alexandria 6 December, 2009, Sunday, Alexandria, Egypt  Introductions  Tour of facility  Meetings with technical staff and users  Travel to Cairo 7 December, 2009, Monday, Cairo, Egypt (Mizrahi)  Travel to Smart Village  Site visit to Oracle Egypt Support Center  Introductions and meetings  Meetings with Amiral project management and technical staff 7 December, 2009, Tuesday, Cairo, Egypt (Doby) 8 December, 2009, Tuesday, Cairo, Egypt (Doby & Mizrahi)  Meetings with Amiral project management and technical staff  Individual discussions with each area lead  Meeting with CMS Application Development Team  Review of CMS system (architecture and functional)  Demonstration of CMS system  Discussions with CMS development, project management, support and users  Agreement on follow-up actions (data collection, documentation) 9 December, 2009, Wednesday, Cairo, Egypt  Sustainability discussions at PIC (training, hiring, finance, strategy, risk)  Follow-on meetings with project management, architecture, network, database, security, storage teams)  Prepare draft out brief presentation BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 28 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft 10 December, 2009, Thursday, Cairo, Egypt  Final meeting with AOCJ COP and Sr. IT Advisor  Debriefing with USAID (DG Office, Mission Director, Interested USAID parties) BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 29 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft APPENDIX B: TABLE OF RECOMMENDATIONS Section Reference Recommendations and Deliverables Level of Effort Priority 5.1.1 Security Components Develop security policies and procedures which have been vetted against accepted international standards. Conduct a security audit of the system to include intrusion detection efforts. Continue to audit security and privilege use cases in application development and testing. Examine stronger common logging ties between application and network security. Medium Operational 5.1.2 Physical Security Minor physical security improvements. Low Sustainment 5.1.3 LAN and Server Infrastructure Plan Develop installation punchlists and procedures. Low Sustainment 5.1.4 SAN Configuration Perform detailed SAN architecture review, capacity and performance evaluations. Create and implement procedures to review and report on performance and capacity. Low Operational 5.1.5 Power Infrastructure Monitoring effort and any required equipment. Low Sustainment 5.1.6 WAN Capacity An updated analysis of anticipated WAN usage. Medium Operational 5.1.7 Disaster Recovery Policy Develop, measure, and monitor disaster recovery policy. Medium Sustainment 5.1.8 Security Policy Develop, measure, and monitor security policy. Appoint a security lead. Medium Operational 5.1.9 Backup Policy Develop, measure, and monitor backup policy. Medium Operational 5.1.10 Scanning Process Identify operational SLAs and detailed workflow processes for the scanning operation. Low to Medium Sustainment 5.1.11 Oracle Components Continued education process. Low Sustainment 5.2.1 CMS Design Documents Develop policies and procedures for regular documentation updates. Medium Sustainment 5.2.2 CMS None. None None BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 30 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft System Development Process 5.2.3 CMS Testing and Deployment Model Dedicated, independent QA process with stated constant improvement goals. Medium Operational 5.2.4 CMS Workflow Identify CMS product roadmap and cost benefit of possible automation process steps. Medium Operational 5.2.5 CMS Project Documents and Traceability Review current project documents for completeness with project management standards. Medium Operational 5.2.6 CMS System Reporting and Workflow Management Systems and Workflow Management policies, procedures, and reports. Medium to High Operational 5.2.7 CMS Rollout Procedures Updated rollout policies and procedures including user alerts and guidance on new functionality. Medium Sustainment 5.2.8 CMS Security and Integrity Features Conduct security/integrity audit of CMS system and on-going evaluation of use cases prior to completing development. Medium Operational 5.2.9 CMS User Training Updated training policies and procedures. Online Help documentation. Low Sustainment 5.2.10 CMS User Feedback Implement User Feedback and online Help mechanisms. Report on user feedback findings regularly. Medium Sustainment 5.3.1 PIC Staff Hiring A final approved PIC organizational structure with associated budget and schedule planning. Medium Critical 5.3.2 PIC Training Procedures PIC technical procedure development. Low Operational 5.3.3 Help Desk Planning Help Desk Management Plan with SLAs and designated lead (from the AOCJ as responsible for the effort). Medium Operational 5.3.4 Technical Training Identify ongoing training activities for key operational and support resources. Low Sustainment 5.3.5 Maintenance Policy Develop, measure, and monitor maintenance policy as well as vendor agreements. Medium Operational 5.3.6 Metrics Usage Complete business metrics plan and optimal system metrics to support that plan; examples of Medium Sustainment BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 31 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft bench mark metrics; and periodic performance reviews. 5.3.7 Models and Analytics Update analytic models. Medium Operational 5.3.8 Project Staffing Utilization Transitional plan to utilize AOCJ staff members to compliment the PIC team. Medium Operational 5.3.9 Sustainability Budget Updated budget for submission to the PIC. Medium Critical 5.3.10 Project Reporting Revised reports, procedures and a critical project progress review. Medium Operational 5.3.11 Quality Assurance/Quality Management (QA/QM) Quality Assurance/Quality Management Plan. Medium Operational The following provides a generalized description of the varying levels of effort:  Low: Relatively easy to implement in the short-term; require a low level of effort; have little or no dependency on other recommendations.  Medium: Relatively more complex to begin implementation in the short-term; require a medium-term timeframe in which to complete; may require the completion of precedent recommendations/tasks; tend to require more than one interrelated set of activities or tasks.  High: Most complex to implement; require a long-term approach, commitment and timeframe to complete; may require the completion of precedent recommendations/tasks; requires several interrelated sets of activities or tasks. The following provides guidelines for task priority:  Critical: These tasks should be scheduled within the next 60 – 120 days. They may result in major schedule, cost or performance impacts if not accomplished.  Operational: These tasks should be scheduled prior to completing development and testing. They may result in minor schedule, cost or performance impacts if not accomplished.  Sustainment: These tasks should be completed prior to completion of the project. They may result in minor sustainment issues if not completed. BPA Contract Number: CIO-E-00-09-00006-00 Task Order Number: CIO-E-01-09-00006-00 USAID/Egypt AOCJ Program IV&V TDY Report 32 AINS DCN: AID BCCS EG AOCJ TDY RPT; AID Cairo 7 April 2010; Final Draft APPENDIX C: STANDARDS REFERENCES The following IT standards are listed in the order that they are referenced in the report: 1. USAID Automated Directives Systems (ADS) Chapter 548: http://www.usaid.gov/policy/ads/500/548.pdf 2. IT Security Standards, e.g., ISO 15408: http://www.iso.org/iso/catalogue_detail.htm?csnumber=50341 3. IT Security Standards ISO 17799: http://www.iso.org/iso/catalogue_detail?csnumber=39612 4. Clinger Cohen Act the Clinger-Cohen Act (also known as "Information Technology Management Reform Act of 1996") (Pub. L. 104-106, Division E): http://www.cio.gov/Documents/it_management_reform_act_Feb_1996.html 5. Project Management Institute (PMI) project management standards: http://www.pmi.org/Pages/default.aspx 6. PMI Global Standards for IT Projects: http://www.pmi.org/Resources/Pages/Library￾of-PMI-Global-Standards-Projects.aspx#Translations 7. Information Technology Infrastructure Library (ITIL) standards: http://www.itil.org/en/vomkennen/itil/index.php 8. ISO 20000 standards for service management: http://www.itil.org/en/vomkennen/iso20000/planungundumsetzung/index.php 9. Control Objectives for Information and Related Technology (COBIT) standards: http://www.itil.org/en/vomkennen/cobit/index.php 10. USAID’s Development Experience Clearinghouse: http://dec.usaid.gov/