ASW Component 3 Technical and Financial Evaluation Final Report Version No. 1.00 29 February 2016 This publication was produced for review by the United States Agency for International Development. It was prepared by consultants for the USAID-funded ASEAN Single Window Project. ASW Technical and Financial Evaluation – Final Report Page 2 of 80 ASW Component 3 Technical and Financial Evaluation Final Report Version No. 1.00 ASEAN SINGLE WINDOW PILOT PROJECT CONTRACT NUMBER: AID-486-C-13-00004, PROJECT TASK NO. 30012-14 NATHAN ASSOCIATES INC. USAID/REGIONAL DEVELOPMENT MISSION FOR ASIA AUTHORS: IAN HOGG DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ASW Technical and Financial Evaluation – Final Report Page 3 of 80 Table of Contents Acronyms and Definitions.......................................................................................................................5 1 Executive Summary.........................................................................................................................7 1.1 Has the Contractor fulfilled the terms of the Request for Proposal? .....................................7 1.2 Is the Contractor’s software secure, scalable and suitable? ................................................13 1.3 Are there any technical gaps remaining? .............................................................................14 1.4 Are there any further funding requirements needed?.........................................................14 1.5 Is there a need for a contingency plan?................................................................................15 1.6 Next Steps.............................................................................................................................15 1.6.1 Recommended actions prior to the cut-over to live operation....................................15 1.6.2 Recommended actions post cut-over to live operation ...............................................16 2 Introduction ..................................................................................................................................18 3 Terms of Reference.......................................................................................................................20 3.1 Activity Background ..............................................................................................................20 3.2 Objective ...............................................................................................................................20 3.3 Tasks......................................................................................................................................21 3.4 Deliverables...........................................................................................................................22 3.5 References ............................................................................................................................22 3.6 Management and Governance .............................................................................................23 4 Methodology Used........................................................................................................................24 4.1 References ............................................................................................................................24 4.2 Questionnaires......................................................................................................................24 4.2.1 AMS Questionnaire (Part 1)..........................................................................................24 4.2.2 AMS Questionnaire (Part 2)..........................................................................................25 4.3 Where the documents were used ........................................................................................25 5 Detailed Findings...........................................................................................................................27 5.1 Contractor’s performance in resolving Member States’ problems......................................27 5.1.1 Findings.........................................................................................................................27 5.1.2 Assessment ...................................................................................................................29 5.2 Contractor’s compliance with the Request for Proposal......................................................29 5.2.1 Findings.........................................................................................................................29 5.2.2 Assessment ...................................................................................................................38 5.3 User acceptance and success indicators of the ASW messaging system..............................41 5.3.1 Findings.........................................................................................................................41 ASW Technical and Financial Evaluation – Final Report Page 4 of 80 5.3.2 Evaluation .....................................................................................................................60 5.4 Network security, scalability and suitability of the Contractor’s software...........................61 5.4.1 Findings.........................................................................................................................61 5.4.2 Evaluation .....................................................................................................................63 5.5 Draft transition and migration plan for the cutover to live operation .................................64 5.5.1 Findings.........................................................................................................................64 5.5.2 Assessment ...................................................................................................................66 5.6 Funding requirements needed to support the operation and expansion of the ASW.........66 5.6.1 Findings.........................................................................................................................66 5.6.2 Funding Requirements..................................................................................................69 5.7 Adequacy of the plans for technical training for the Project Management Office (PMO)...74 5.7.1 Findings.........................................................................................................................74 5.7.2 Assessment ...................................................................................................................75 5.8 Quality assurance procedures for the ASW Regional Services.............................................76 5.8.1 Findings.........................................................................................................................76 5.8.2 Evaluation .....................................................................................................................77 6 Recommendations........................................................................................................................78 Appendix A – AMS Questionnaire (Part 1)............................................................................................80 Appendix B – AMS Questionnaire (Part 2)............................................................................................80 ASW Technical and Financial Evaluation – Final Report Page 5 of 80 Acronyms and Definitions ACDD ASEAN Customs Declaration Document ACTI ASEAN Connectivity through Trade and Investment AMS ASEAN Member State ASEAN Association of Southeast Asian Nations ASEC ASEAN Secretariat ASW ASEAN Single Window ASW PT ASW Project Team ASWSC ASEAN Single Window Steering Committee AS-1 (or AS1) Response generated by the ASW Gateway at the exporting country AS-2 (or AS2) Response generated by the ASW Gateway at the importing country AS-3 (or AS3) Response generated by the NSW system of the importing country ATIGA ASEAN Trade In Goods Agreement B2B Business-to-Business B2G Business-to-Government Cargo-IMP Cargo Interchange Message Procedures Cargo-XML XML equivalent of Cargo-IMP CEPT Common Effective Preferential Tariff CIA Certificate Issuing Agency CO (or COO) Certificate of Origin Contractor The company engaged to carry out ASW Pilot Project Component 2 Consultant The person engaged to carry out ASW Pilot Project Component 3 COTS Common Off-The-Shelf CSV Comma Separated Values e-ATIGA Electronic ASEAN Trade In Goods Agreement e-Cert Electronic Sanitary and Phytosanitary Certificate (XML) – by UN/CEFACT e-Phyto Electronic Phytosanitary Certificate (XML) – published by IPPC EDIFACT Electronic Data Interchange For Administration, Commerce and Transport EFT Electronic Funds Transfer EU European Union FTP File Transfer Protocol G2G Government-to-Government IATA International Air Transport Association IFCSUM UN/EDIFACT Forwarding and Consolidation Summary Message IFTMBC UN/EDIFACT Booking Confirmation Message IPM Information Processing Modeling IPPC International Plant Protection Convention IT Information Technology JMS Java Message Service LOE Level of Effort MIS Management Information System NPPO National Plant Protection Organization NSW National Single Window NVOCC Non-Vessel Operating Common Carrier ASW Technical and Financial Evaluation – Final Report Page 6 of 80 OCP Operational Certification Procedures PKI Public Key Infrastructure PLF Protocol on the Legal Framework PMO Project Management Office RES Customs Response message RFP Request for Proposal SAD Single Administrative Document SEOM Senior Economic Officials Meeting sFTP Secure File Transfer Protocol SOAP Simple Object Access Protocol SSL Secure Sockets Layer STA Strategic Trade Act TM Transition and Migration TOR Terms of Reference TWG Technical Working Group UNECE United Nations Economic Commission for Europe UN/CEFACT United Nations Centre for Trade Facilitation and Electronic Business UN/EDIFACT United Nations Electronic Data Interchange For Administration, Commerce and Transport USAID United States Agency for International Development VPN Virtual Private Network VPT-IT Virtual Project Team on IT XML Extensible Markup Language XSD XML Schema Definition 864 Electronic ATIGA Form D (e-ATIGA Form D) ASW Technical and Financial Evaluation – Final Report Page 7 of 80 1 Executive Summary This report provides a final technical and financial evaluation of the ASW Pilot Project Component 2 (Full-Fledged). It has been written by the Consultant engaged to carry out ASW Pilot Project Component 3. The findings in the report are based on various documentation, which was made available to the Consultant. The main reference for the report was “Request for Proposal for the Development and Implementation of the ASEAN Single Window Software”, published and distributed in November of 2014. Several other documents have also been used as references, such as the plans to set up a Project Management Office (PMO), technical documentation, test plans, test reports etc. Details of the methodology used, and the complete list of references used, are provided in the main body of the report. The Terms of Reference (TOR) for ASW Pilot Project Component 3 highlight five specific goals of the technical and financial evaluation. The Consultant’s key findings for each of these five goals, and the recommendations made in the report, have been summarized below. 1.1 Has the Contractor fulfilled the terms of the Request for Proposal? “Review whether the contractor has fulfilled the terms of the Request for Proposal for the Development and Implementation of the ASEAN Single Window Software published and distributed in November of 2014, including but not limited to the implementation of the ASW enabling infrastructure and its ability to support the cross border exchange of data in a secured network environment” In order to meet the objectives of this goal, the Consultant devised a questionnaire to be returned by the participating AMS following the completion of the end-to-end testing (Phase 1). Questions were based on the 12 assumptions and the scope of work (SOW) that were outlined in the Request for Proposal (RFP). For ease of reference, the Consultant’s findings have been summarized in Table 1 below, which provides a simple “YES” or “NO” as to whether the Contractor has complied with the RFP. Additional comments have also been provided where further improvement / clarification may be necessary. Table 1: Contractor’s compliance against assumptions and scope of work stated in the RFP Ref. Requirement Compliance with RFP? 1.1 To provide the necessary mechanism to enable exchange￾ready AMS to exchange the ATIGA Form D, electronically on an end-to-end basis from one NSW to other NSWs in an integrated, secured ASW environment adhering to the technical design of the ASW Pilot Project Component 1 YES. The Contractor has provided ASW Gateway / Software for the five exchange-ready AMS, namely; Indonesia, Malaysia, Singapore, Thailand and Viet Nam. 1.2 The ASW software shall implement functionalities that enable robust, consistent and secure application level information exchange (consisting of correct routing, 100% message delivery including processing of acknowledgement responses pushed by the receiving NSW, one time only YES. The participating AMS have completed their end￾to-end testing and have accepted the “End-to-End Test Plan” and “End-to-End ASW Technical and Financial Evaluation – Final Report Page 8 of 80 Ref. Requirement Compliance with RFP? delivery and security as per ASW Pilot Project Component 1 deliverables) Test Report” provided by the Contractor. 1.3 Software for the ASW must be hosted in each AMS YES. Based on feedback from the AMS, they are all hosting their own instances of the ASW Gateway / Software, except Thailand who are using their own NSW gateway. 1.4 The ASW software must employ queuing and message delivery re-try mechanisms to ensure delivery of messages and provide error response for unsuccessful delivery YES. Based on feedback from the AMS, the Contractor’s software does have this functionality, but the documentation as to how this can be used / configured for the ASW should be explained in the “ASW Gateway Technical Operations Guide”. Please also see Recommendation No. 1 1.5 The ASW software must not initiate connections to the NSWs. Messages received from other ASW software must be put into queues for retrieval. The system should ensure that an alert message is generated after a specific time if the message is not retrieved by the NSW YES. The alert feature is described as optional in the “Interoperability Specifications” (see section 4.7.5), but how this feature can be used / configured should be explained in the “ASW Gateway Technical Operations Guide”. Please also see Recommendation No. 1 1.6 NSWs must employ a periodic calling mechanism to retrieve data from ASW software. NSWs must be able to acknowledge receipt of the message retrieved from the ASW software (the Contractor should revise the four interface scripts developed in the ASW Pilot Project Component 1 to include message reference number and a response back to the ASW software i.e. web services, JMS, file system and secure File Transfer Protocol) YES. Based on feedback from the AMS, they are able to acknowledge the receipt of the message using Web Services, JMS, FTP or sFTP. 1.7 The ASW software will not perform data encoding format and/or message standards conversions. The NSW of respective AMS is expected to perform this function by using the data encoding format and message standard tested during the scaled-down pilot YES. The Contractor’s software has the ability to do this, but it is currently not being utilized for the ASW. ASW Technical and Financial Evaluation – Final Report Page 9 of 80 Ref. Requirement Compliance with RFP? 1.8 The full ASW Regional Services functionality as indicated in the Component 1 design architecture must be implemented to cover the updating of the reference data (national reference data and common reference data) and Management Information Systems (MIS), including automated monitoring of message exchanges between AMS, and network activity and performance monitoring to ensure continuous capacity, statistical logs, network visibility YES. The updating of the reference data has been documented in the ASW Regional Services Portal’s “ASEAN Administration Manual” and “User Manual”. The ASW Gateway also provides an automated mechanism for information on message exchanges to be uploaded to ASW Regional Services. However, it should be noted that Singapore and Thailand are not using this automated mechanism and will be using alternative mechanisms to upload this information. Please also see Recommendation No. 2 1.9 All information exchanges must be at least authenticated and authorized at the NSW level in accordance to the security standard defined per ASW Pilot Project Component 1 deliverables YES. The communications protocol that is being used to exchange documents, ebMS v2.0, provides the necessary authentication at the NSW level. 2 Implementation of Component 2 must adhere to the core characteristics of the Component 1 technical design architecture as agreed upon by AMS and the message exchange protocol that was used during the ASW pilot component 2 scaled-down. TO BE CLARIFIED. Based on feedback received from the AMS, there are some concerns about whether the Contractor’s software has adhered to ASW Pilot Project Component 1. 3 Commercial Off-the-Shelf Software (COTS) is neither required nor prohibited YES. This requirement has been met by Axway’s B2Bi software. 4 Contractor shall develop and maintain the ASW Regional Services (RS) Portal during phases 1 and 2 to support AMS risk management systems by providing synchronized reference data sets for validation and control. The contractor will be provided the code for the RS Portal that was used in implementing the Component 2 (Scaled Down) Pilot for review and possible use in implementing this contract. YES. Based on feedback from the AMS, the ASW Regional Services Portal has been installed as part of the ASW Regional Services. ASW Technical and Financial Evaluation – Final Report Page 10 of 80 Ref. Requirement Compliance with RFP? 5 Local NSW perimeter networks may be implemented and the proposed solution should be able to operate in such an environment. Member States have the discretion to determine its perimeter network for the ASW software. YES. Based on the feedback from the AMS, the ASW Gateway / Software has been successfully installed onto their local NSW networks. 6 The ASW software, which is hosted by each individual Member State that has the full control of its security, must provide the capability to configure what information is stored permanently, temporarily or not stored at all on ASW software (it includes messages that are being exchanged through the ASW software). The use of such capability is optional and should be configurable by the Member States. YES. The Contractor’s software provides comprehensive security features for the exchange of electronic documents and data archiving / purging has been partially documented in the “ASW Gateway Technical Operations Guide” (see section 14). Please also see Recommendation No. 1 7 The ASW Regional Services server shall be hosted at the ASEAN Secretariat (ASEC) during phase 1 and 2. Whether the server remains at ASEC or is moved to a data center thereafter will be determined at a later date. YES. The ASW Regional Services server is hosted temporarily in Jakarta and it is understood that there are plans to move it to a data center in in future. 8 The development and testing in this Project will include a full/turn-around end to end testing of information/message exchange from one NSW to other NSWs i.e. activities in addition to the ASW software connectivity testing. A full/turn-around includes the utilization of the ATIGA Form D by the government agency and actors involved. The contractor should be able to provide an end-to-end business process flow of the electronic ATIGA Form D including validation schemas for replacement, utilization, and cancelation procedures that must be implemented by an AMS NSW. YES. End-to-end testing was carried out from 24 th August 2015 to the 16th October 2015, and parallel testing started on the 26th October 2015. The Contractor has also developed a document called the “e-ATIGA Form D Process Specification and Message Implementation Guideline” and has made some amendments to the XML schemas during the end-to-end testing. ASW Technical and Financial Evaluation – Final Report Page 11 of 80 Ref. Requirement Compliance with RFP? 9 The contractor will provide a post live operation efficient and effective transition management (TM) plan that includes the possibility of an eventual upgrade of the current network via https to a more secure virtual private network such as IPVPN if AMS decide to implement such a system. Moreover, this TM plan will include recommendations of public awareness activities, recommendations to be provided by ACTI, to be implemented by AMS to prepare users for the implementation of other cross-border documents, including the export information, agreed by Member States. It recognizes that the implementation of the ASW will have an impact on other stakeholders involved. Paramount in the implementation of the ASW beyond 2015 is its acceptance by all actors involved and the easy transition to new roles and tasks. The contractor is not tasked with implementing the TM plan. YES. The Contractor has provided a document called the “ASW Pilot – e-ATIGA Form D, Plan for Transition to Live Operation” and a more detailed document called the “Schedule for Transition to e-ATIGA Form D Live Operation”. 10 AMS will need to setup a dedicated team to manage both the electronic and paper-based environments during the parallel testing phase to ensure a smooth testing phase prior to cutover to live implementation. YES. Each AMS have set up their own teams to carry out both the end-to-end and parallel testing, under the guidance of the Contractor. 11 AMS agreed to establish a Virtual Project Team on IT (VPT￾IT) comprised of national technical experts from Member States to facilitate the implementation of this contract with regard to task execution, coordination challenges, and response time to address implementation issues. YES. The VPT-IT team has been established and they have been working closely with the Contractor. 12 The contractor shall develop the capability within the RS Portal to support the optional use of PKI and digital certificates in the exchange of data by NSWs in accordance with the agreed technical design architecture (Component 1 Deliverables). YES. Based on feedback from the AMS, the Contractor’s software does have this functionality, but the documentation as to how it can be used / configured for the ASW is limited. It has not been documented in either the “ASEAN Administration Manual” or the “ASW Gateway Technical Operations Guide”, although there is some mention of this in the “Interoperability Specifications (section 6)”. Please also see Recommendation No. 1 ASW Technical and Financial Evaluation – Final Report Page 12 of 80 Ref. Requirement Compliance with RFP? SOW Apart from coordinating all activities/parties to ensure effective and successful pilot/parallel testing as part of project management, support from the Contractor will also cover troubleshooting any messaging system problems identified during the transition to live operations in the exchange of ATIGA YES. Based on the feedback from the AMS, they have rated the Contractor’s performance “Average” to “Excellent”, in terms of their project management, effectiveness and efficiency in resolving problems during the end-to-end testing. SOW The contractor will provide the NSW interface scripts installation manual that describes the step-by-step installation covered for the NSW integration module at each of the AMS. Moreover, an administration manual will also be provided that describes the administration functions for the NSW integration module installation at each of the AMS. YES. The interface specifications have been documented in the “Interoperability Specifications” (see section 4) and bridge scripts have been developed for FTP, JMS and Web Services. An administration manual for the ASW Gateway has also been provided in the form of the “ASW Gateway Technical Operations Guide”. Please also see Recommendation No. 1 SOW The Contractor shall operate the Regional Services and assist AMS in operation of their ASW software during the testing phase (Phase 1 and phase 2). The Contractor will provide a written manual on the operation and maintenance of the regional services software and any reporting systems. YES. Two manuals have been written for ASW Regional Services Portal; the “ASEAN Administration Manual” and the “User Manual”. SOW The Contractor shall provide a written set of specifications to assist any AMS in the development of national software to exchange information through the ASW system YES. The Contractor appears to meet this requirement through a document called the “Interoperability Specifications”. SOW The Contractor shall submit comprehensive monthly reports on activities during the duration of the project YES. The Contractor has maintained a document called a “test matrix” on a weekly basis during both the end-to-end and parallel testing. SOW The Contractor shall submit a fortnightly statistics report with detailed analysis of issues encountered and proposed solutions during the testing process YES. The Contractor has maintained a document called a “weekly log” during both the end-to-end and parallel testing. ASW Technical and Financial Evaluation – Final Report Page 13 of 80 Ref. Requirement Compliance with RFP? SOW The contractor must provide a version control procedure to track and merge changes in the operation of the ASW software and other software applications over time. This would provide a convenient and effective way of monitoring orderly changes or upgrades to minimize disruptions. Further, automate management of revision history in effecting upgrades and software patches of the ASW software or any other required software YES. Version control procedures have been provided in a document called the “Change and Configuration Management Guide” (see section 2.3). Please also see Recommendation No. 3 SOW The Contractor will provide input to the Regional Operations team, which will succeed the VPT-IT Team mentioned in Assumption 11 once the latter is in place, in linking the ASW web portal (asw.asean.org) with the ASW RS portal NO. The Contractor has not yet been able to provide this input, because the Regional Operations team has not been established. Based on feedback received from the AMS, the VPT-IT team have not been involved in the set￾up of the ASW Regional Services Portal and the Contractor is not obliged to provide training under the RFP. Please also see Recommendation No. 12 SOW The contractor shall provide an information security incident management plan. This will allow the effective handling of information security incidents. Lack of clearly defined organizational structure can create confusion, resulting in each phase of a response taking longer than necessary YES. Security incident management has been provided in a document called the “Operations Handbook” (see section 3.6). Please also see Recommendation No. 3 1.2 Is the Contractor’s software secure, scalable and suitable? “Assess the network security (in terms of security vulnerabilities), scalability and suitability of the Contractor’s software (a) for the exchange of the ASEAN Trade In Goods Agreement (ATIGA) Certificate of Origin (CO) Form D; and (b) to accommodate additional cross-border documents” The Contractor’s software is a Common-Off-The-Shelf (COTS) solution designed to meet the common requirements of any B2B gateway. Like many other similar COTS solutions, it supports well-known industry standard message formats, including UN/EDIFACT and XML which are the most widely used for the exchange of electronic trade documents. It also supports international security standards, including those defined in ASW Pilot Project Component 1. It is important to point out, however, that each AMS is responsible for their own network security and the Contractor’s software has been customized to meet the specific requirements of the ASW ASW Technical and Financial Evaluation – Final Report Page 14 of 80 Gateway, as defined in ASW Pilot Project Component 1. It will be almost impossible to just simply replace the Contractor’s software when their contract ends in December 2016, as it will require another round of end-to-end and/or parallel testing to be planned and executed well in advance. The Consultant finds no reason to be concerned about the Contractor’s software being secure and suitable for the ASW, because the main selling point of this kind of B2B solution is that they support all well-known international standards. However, scalability and network performance is more of an unknown factor, because the requirements outlined in ASW Pilot Project Component 1 can easily be met, perhaps too easily. The requirements are based on an assumption that there will be a total of 300,000 messages per day exchanged through the ASW network with an average size of 16KB per message. Although this may be an average size of an ATIGA Form D, which is a relatively small type of document, the average size for all types of documents to be exchanged in future would be expected to be much greater. For example, the average size of a Pre-departure Cargo Report could potentially be hundreds, or even thousands, times larger and ATIGA Form D. 1.3 Are there any technical gaps remaining? “Identify remaining technical gaps (including gateway connectivity issues, different schema implementation by Member States, etc., if any) that must be resolved prior to the cutover to live operation and provide recommendations on how to address any remaining gaps” During this technical and financial analysis, the Consultant was not made aware of any technical gaps, in terms of connectivity issues, schema issues etc. However, at the time of writing this Final Report, there were still a few technical issues to be resolved before the acceptance criteria for the parallel testing can be considered to be met: 1. ATIGA Form D – all categories should be fully tested. 2. Customs responses – should be fully tested by all participating AMS, including the different statuses as stated in the Parallel Test Plan’s acceptance criteria (i.e. “Received”, “Not Processed”, “Preferential Treatment Given” and “Preferential Treatment Not Given”). 3. Harmonization of ATIGA Form D and Customs response – Malaysia, Singapore and Thailand have all expressed their concerns about this issue. For example, Thailand continue to have difficulty in matching ATIGA Form Ds with the Customs responses. 4. MIS reporting through ASW Regional Services – Malaysia have reported that the statistics are inaccurate, Singapore have been uploading information from their ASW Gateway manually and Thailand have not been uploading any information during the testing. A summary of the recommendations identified in this report can be found in Section 1.5. 1.4 Are there any further funding requirements needed? “Identify any other funding requirements needed to support the operation and expansion of the ASW (including the electronic exchange of the export data and the four possible additional documents covered by the existing Information Process Modeling study), not included in the proposed funding requirements agreed by the 15th ASWSC Meeting and to be presented to SEOM. This will include recommendations to Member States to sustain the operation and maintenance of the ASW enabling infrastructure beyond 2015 either by entering into a maintenance contract with Axway or hiring a new contractor” ASW Technical and Financial Evaluation – Final Report Page 15 of 80 Based on the information available, the Consultant has identified a number of areas where funding may be required, particularly for the operation and expansion of the ASW. Firstly, there was some initial confusion about the Project Management Office (PMO). The document entitled “Design of Project Management and Planning Team (Project Management) Office” mentions that two staff will be recruited from January 2017. However, the document did not state who would take on these responsibilities from January 2016. After clarification, the Consultant was informed that the Virtual Project Team on IT (VPT-IT) will take on these responsibilities in the short term. This would suggest that costs identified in the document entitled “ASW Regional IT Operational and Staffing Costs”, which only cover the costs from January 2017, would need to be reviewed to cover the staffing and training costs expected in 2016. There may also be hardware and software costs. Another area of concern are the plans for the expansion of the ASW, because the final report for the “Information Process Modeling” study appears to be more theoretical, as it does not address some of the practical issues. For example, what are the incentives and benefits for Shipping Agents to send a Freight Booking Confirmation the NSW, unless the government mandates it? 1.5 Is there a need for a contingency plan? “Propose contingency plan to Member States for consideration if the end-to-end testing and parallel testing were assessed to be less than successful to proceed with the live operation on 30 Dec 2015” A contingency plan was discussed at the 33rd TWG Meeting held in Singapore between the 17th and 20th November 2015. During the meeting, the AMS agreed that the live operation would only proceed if two AMS successfully concluded the parallel testing. They also agreed that the transition to live operation would be carried out in stages and that the participation by Malaysia, Singapore and Thailand would be subject to the consent of the traders. The parallel testing was subsequently extended to the 8th January 2015 and the Contractor has released their “Parallel Test Report” (Release 2), which covers the parallel testing carried out between the 26th October 2015 and the 18th December 2015. The Contractor’s report concludes that Indonesia and Viet Nam are the first two AMS ready to proceed to live operation with a suggested target “go live” date of the 8th February 2016. At this stage, however, it is not certain which AMS will follow, as there appears to be a small number of issues that still need to be addressed. However, based on the information provided, it seems likely that Malaysia will be the next AMS to proceed, followed by Thailand, then Singapore. 1.6 Next Steps In total, this report contains 16 recommendations, which may require further action by the AMS. Of these, only 4 recommendations are considered to be essential prior to cut-over to live operation and the remaining can be considered post cut-over. 1.6.1 Recommended actions prior to the cut-over to live operation The following recommendations are considered essential actions to be carried out prior to the cut￾over to live operation: ASW Technical and Financial Evaluation – Final Report Page 16 of 80  Recommendation No. 41 : To make the Customs response mandatory and to ensure that it is fully tested during the parallel testing prior to the cutover to live operation.  Recommendation No. 5: To ensure that the alternative reporting mechanisms of Singapore and Thailand have been fully tested, so that the number of ATIGA Form Ds sent by the sending AMS match the number received by the receiving AMS. Similarly, the number of responses sent by the sending AMS should match the number received by the receiving AMS.  Recommendation No. 7: To ensure that the acceptance criteria in the Parallel Test Plan is met by resolving all outstanding issues, including; the testing of all categories of ATIGA Form D, the testing of all Customs response statuses, the harmonization of ATIGA Form D and Customs responses, and the accuracy of the ASW Regional Services reporting.  Recommendation No. 15: To reach an agreement with the Contractor to provide operational training to the VPT-IT team and to ensure that the “ASEAN Administration Manual” is comprehensive enough to be used as a reference guide for new PMO staff in the future. 1.6.2 Recommended actions post cut-over to live operation The following recommendations are actions that can be considered post cut-over to live operation:  Recommendation No. 1: To ensure that all of the features of the ASW Gateway / Software, as stated in the RFP, are thoroughly documented in the “ASW Gateway Technical Operations Guide”. At a minimum, this should cover queuing and re-try mechanisms, error responses for unsuccessful delivery, alert messages, users and roles, archiving / purging of data, use of PKI and digital certificates, installation manuals and interface scripts.  Recommendation No. 2: To ensure that all of the features of the ASW Regional Services, as stated in the RFP, are thoroughly documented in the “ASEAN Administration Manual” and “User Manual”. At a minimum, this should include; the updating of reference data, automated monitoring of message exchanges between AMS, MIS reporting and performance monitoring.  Recommendation No. 3: To ensure that documentation is generic and relevant for all documents, not just the e-ATIGA Form D. For example, the support procedures and security incident management plan is relevant for all documents and should be documented separately from the “ASW Pilot e-ATIGA Form D - Operations Handbook”. It is also suggested that the “ASW Pilot e-ATIGA Form D - ASW Gateway Technical Operations Guide” should be renamed, so that it is clear that it is relevant for all documents, and that the specific ATIGA Form D related functionality should be consolidated into “ASW Pilot e-ATIGA Form D - Operations Handbook”. The title of the “Development and Live Implementation of the ASEAN Single Window (ASW) Software for the exchange of the ATIGA CO Form D - Change and Configuration Management Guide” may just need to be renamed to a more generic title.  Recommendation No. 6: To ensure that all of the ASW technical specifications are platform neutral and can be easily implemented in any NSW environment without being dependent on any special software. This should include details of how the standard communications 1 Please note that the Recommendations are numbered in the order that they are identified in the main body of the report and are also summarized in a table at the end of the report. ASW Technical and Financial Evaluation – Final Report Page 17 of 80 protocol (i.e. ebMS) is being used and all interfaces between the NSW and the ASW Regional Services Portal.  Recommendation No. 8: To plan for and carry out more comprehensive testing of the security features (i.e. use of PKI and digital certificates) and scalability of the Contractor’s software, including stress testing, different document sizes / types (e.g. Pre-departure Cargo Report) and different formats (e.g. UN/EDIFACT).  Recommendation No. 9: To define three scenarios of small, medium and large transaction volumes and to ask the Contractor to provide the appropriate hardware / software requirements, together with the associated costs.  Recommendation No. 10: To review the ASW Regional IT Operational and Staffing Costs to cover the operation of the ASW Regional Services during 2016.  Recommendation No. 11: To carry out a comprehensive feasibility study into the readiness and willingness of Shipping Agents, Freight Forwarders, Port Operators and Ground Handling Agents to send their electronic transport documents to the NSWs on a voluntarily basis, including the Freight Booking Confirmation and Freight Loading Confirmation.  Recommendation No. 12: To carry out a comprehensive feasibility study into the readiness and willingness of national Customs administrations to request the Shipping Agents to send Pre-departure Cargo Reports to the NSW, which should also take into account any legal implications and any duplication of data being submitted.  Recommendation No. 13: To carry out a comprehensive feasibility study into the readiness and willingness of the certificate issuing authorities to exchange electronic Sanitary and Phytosanitary (e-SPS) certificates with their overseas counterparts via the ASW.  Recommendation No. 14: To carry out an analysis of the current international trends and industry initiatives to assess their applicability within ASEAN, and also to assess their potential impact on ASEAN in the future.  Recommendation No. 16: To reach an agreement with the Contractor to include quality assurance procedures in the ASW Regional Services Portal’s ASEAN Administration Manual and User Manual and, if possible, cover it in the operational training to the VPT-IT team. ASW Technical and Financial Evaluation – Final Report Page 18 of 80 2 Introduction In November 2014, a Request for Proposal (RFP) was issued for the ASW Pilot Project Component 2 (Full-Fledged), entitled the “Development and Live Implementation of ASEAN Single Window (ASW) Software for the exchange of the ATIGA CO Form D”. The RFP described the scope of work as; “Phase 1: Development of the necessary messaging software and end-to-end testing of the exchange of ATIGA Form D data between exchange-ready AMS for up to a two month period using a test environment, preceded by preparatory activities between contractor and participating AMS, that incorporates not only the functionalities of Component 2 (scaled down) but also the following features from Component 1; • Acknowledgement receipts; • Automated monitoring of message exchanges and configuration; • MIS statistics; • Message exchange and validation; • Audit trail; • PKI management; • Use of ASW regional services reference data systems (at minimum economic operator ID) The messaging software should be readily configured in the future to transfer additional data packages in XML format representing other trade documents. Phase 2: Transitioning to live environment (parallel phase) for two months in the exchange of ATIGA Form D, where both hard and electronic versions of the form are exchanged between exchange-ready AMS after a brief review of deliverables and a decision by participating AMS; Phase 3: Live exchange of data, including a disaster recovery / standby environment, among exchange￾ready AMS using a production environment after a brief review of deliverables and a decision by participating AMS. If additional AMS beyond the current exchange-ready AMS are prepared to join the cross-border exchange within the first six months of live operation, the contractor will assist each AMS to connect its NSW to the ASW environment through ASW software (costs to be estimated separately for additional AMS); The development and implementation of an integrated secured network adhering to the technical design of the ASW Pilot Project Component 1 for the implementation of ASEAN Single Window should achieve the following:  It should provide AMS the required capacity and resilience to enable connectivity between NSWs of AMS to facilitate exchange of information;  It should allow connectivity and interoperability between heterogeneous platforms through a secure network with high reliability performance. The architecture should support various international protocols standards and exchange mechanism (e.g. synchronous, asynchronous) within a secure operating environment; ASW Technical and Financial Evaluation – Final Report Page 19 of 80  It should ensure compatibility with international open communication standards that each of those Member States can then exchange data securely and reliably with any trading partners that use international open standards;” The RFP also refers to the ASW Pilot Project Component 3 as “Conducting the evaluation of outcomes of the Pilot Project and formulation of recommendations for the eventual ASW”. The purpose of this report is to carry out ASW Pilot Project Component 3 based on the Terms of Reference (TOR) laid out in the following section. The report describes the methodology used, the Consultant’s detailed findings for each task and a number of recommendations. ASW Technical and Financial Evaluation – Final Report Page 20 of 80 3 Terms of Reference While the scope of ASW Pilot Project Component 2 are outlined in the previous section, the Terms of Reference (TOR) for ASW Pilot Project Component 3 are outlined below. 3.1 Activity Background 1. ASEAN Member States (AMS) have noted that the ASW Pilot Project Component 3 would be conducted by a third party to evaluate the outcomes of the ASW Pilot Project Component 2 (Full￾Fledged) and formulate recommendations for the eventual ASW, which would cover the end-to￾end testing from 24 August to 16 October 2015, and the parallel testing from 26 October to 11 December 2015. The targeted live operation schedule is 30 December 2015. The final report, which would include both technical and financial analysis, would be presented to AMS prior to the cutover to live operation. 3.2 Objective 2. The objective of this activity is to evaluate the outcomes of the ASW Pilot Project Component 2 (Full-Fledged) and formulate recommendations for the eventual ASW, including the technical and financial analysis prior to the cutover to live operation by 30 December 2015. 3. The specific goals in the technical and financial evaluation of this ASW Pilot Project Component 3 are to: • Review whether the contractor has fulfilled the terms of the Request for Proposal for the Development and Implementation of the ASEAN Single Window Software published and distributed in November of 2014, including but not limited to the implementation of the ASW enabling infrastructure and its ability to support the cross border exchange of data in a secured network environment; • Assess the network security (in terms of security vulnerabilities), scalability and suitability of the Contractor’s software (a) for the exchange of the ASEAN Trade In Goods Agreement (ATIGA) Certificate of Origin (CO) Form D; and (b) to accommodate additional cross-border documents • Identify remaining technical gaps (including gateway connectivity issues, different schema implementation by Member States, etc., if any) that must be resolved prior to the cutover to live operation and provide recommendations on how to address any remaining gaps; and • Identify any other funding requirements needed to support the operation and expansion of the ASW (including the electronic exchange of the export data and the four possible additional documents covered by the existing Information Process Modeling study), not included in the proposed funding requirements agreed by the 15th ASWSC Meeting and to be presented to SEOM. This will include recommendations to Member States to sustain the operation and maintenance of the ASW enabling infrastructure beyond 2015 either by entering into a maintenance contract with Axway or hiring a new contractor. ASW Technical and Financial Evaluation – Final Report Page 21 of 80 • Propose contingency plan to Member States for consideration if the end-to-end testing and parallel testing were assessed to be less than successful to proceed with the live operation on 30 Dec 2015. 3.3 Tasks 4. Broadly speaking the tasks to be performed, prior to the cutover to live operation, are to: 1) assess the viability of the ASW components (software, hardware, applications, network infrastructure, and network security) prior to the cutover to live operation; 2) assess the demonstrated performance of the participating National Single Windows (NSWs) in relation to the end-to-end processes, the agreed user acceptance process and the success indicators as contained in the test results provided by the AMS designated technical team and from VPT-IT reports during the testing phase; and 3) identify any other funding requirements needed to support the operation and expansion of the ASW (including the electronic exchange of the export data and the four possible additional documents covered in the Information Process Modeling study) that were not included in the proposed funding requirements agreed by the 15th ASWSC Meeting. This will ensure the resilience of the ASW live operation and that each AMS has a functioning NSW in place for sending and receiving electronic cross-border messages. 5. The Consultant will prepare a comprehensive written evaluation report including: • Task 1: A description of the methodology to be used; • Task 2: An assessment of the Contractor’s performance in terms of project management, efficiency and effectiveness in resolving Member States’ problems encountered during the testing; • Task 3: An assessment of whether the implemented ASW messaging system complied with the agreed Request for Proposal for the Development and Implementation of the ASEAN Single Window Software published and distributed in November of 2014 in preparation for the agreed cutover to live operation by 30 December 2015. • Task 4: An evaluation of the user acceptance process and success indicators of the ASW messaging system to ensure that all stated requirements are met; • Task 5: An evaluation of the network security (in terms of security vulnerabilities), scalability and suitability of the Contractor’s software (a) for the exchange of the ATIGA CO Form D; and (b) to accommodate additional cross-border documents. • Task 6: An assessment of whether the draft transition and migration (TM) plan for the cutover to live operation can be successfully implemented; • Task 7: To identify any other funding requirements needed to support the operation and expansion of the ASW (including the electronic exchange of the export data and the four possible additional documents covered in the Information Process Modeling study) not included in the proposed funding requirements agreed by the 15th ASWSC Meeting. • Task 8: An assessment of the adequacy of the plans for technical training for the Project Management Office (PMO) Staff in the operation of the ASW Regional Services; and • Task 9: An evaluation of the quality assurance procedures for the ASW Regional Operations Services covering: o Security quality – effectiveness of the security features implemented. ASW Technical and Financial Evaluation – Final Report Page 22 of 80 o Information quality – accuracy, meaningfulness, and timeliness of the information produced. o Process quality – effect on information management process quality. 6. This activity will be carried out in collaboration with the participating AMS and the contractor. 7. The Consultant shall conduct the evaluation for a total of twenty (25) Level of Effort (LOE) days starting from 27 October to 31 December 2015, and submit the final report by 21 December 2015. 3.4 Deliverables 8. The following deliverables are required for this activity: • Deliverable 1: A preliminary technical and financial evaluation report for review and comment by AMS to be submitted by the Consultant on 11 November 2015, which should contain the detailed evaluation, preliminary findings, and recommendations, to be revised into the draft final report based on AMS comments and further testing; • Deliverable 2: Presentation of the preliminary technical and financial evaluation report (i.e. Deliverable 1) during the 33rd Meeting of the Working Group on Technical Matters for the ASW (TWG) in Singapore; • Deliverable 3: A draft ASW technical and financial evaluation final report will be submitted by the Consultant to AMS by 10 December 2015, which will incorporate the comments provided by AMS during the preliminary report presentation; • Deliverable 4: A final technical and financial evaluation report will be submitted by 21 December 2015; 3.5 References 9. The following documents can be used as references by the Consultant. They shall be provided to the Consultant upon contract award: • The ASW transition to live operation scope of work • ASW Protocol • ASW Agreement • Business processes for CEPT/ATIGA Form D (ASEAN preferential certificate of origin form) • Relevant deliverables of component 1 of ASW Pilot Project (Deliverables #4, #5, #6, #7, #11 and Sections 6, 7, 8 ,9 and 10 of deliverable #10) • ATIGA Form D application materials for purposes of this activity only • Information Process Modeling (IPM) study • ASW Messaging System Inception Report/Project Plan • ASW Messaging System Enhancement Specifications document • e-ATIGA Form D Process Specifications • e-ATIGA Form D Message Implementation Guideline • XML Schema Definition (XSD) document • Interoperability Specifications (including the Bridge Script) • Study on the assessment of the viability of ASEC to host the ASW servers ASW Technical and Financial Evaluation – Final Report Page 23 of 80 • Project Management Office (PMO) Design paper • ASW Transition Management (TM) Plan • Other relevant documents as determined by the Consultant. 3.6 Management and Governance 10. The evaluation will be conducted in collaboration with AMS through their representatives at the TWG and ASW Steering Committee (ASWSC) and with the contractor. AMS are requested to revert with their feedback / comments on the draft final report to be provided by the Consultant by 17 December, in order to provide for a smooth transition to cutover to live operation by 30 December 2015. 11. To ensure the success of this activity, AMS and their representatives, including the experts from the Contractor are required to provide support for the conduct of the activities by the Consultant (e.g. facilitation of contacts with technicians, policy makers, and those in charge of operating the NSWs) for the purpose of this evaluation study. 12. Throughout the duration of the project, the Consultant shall prepare and submit brief progress reports to the TWG and ASWSC every other week for monitoring purposes. ASW Technical and Financial Evaluation – Final Report Page 24 of 80 4 Methodology Used The purpose of this section is to outline the approach of Task 1 of the TOR; “A description of the methodology to be used”. 4.1 References Under item 9 of the TOR (see section 3.5), a number of documents were made available to the Consultant for the purpose of this exercise. However, the following documents have been used as the main references for this report. • Component 1 Deliverables • Request for Proposal • Contractor’s Technical Proposal in response to the Request for Proposal • Partnership Matrix for the Exchange of the ASEAN Customs Declaration Document (ACDD) • Information Process Modeling Study • Design of Project Management and Planning Team (Project Management) Office • ASW Regional IT Operational and Staffing Costs • End-to-End Test Plan • End-to-End Test Report • Parallel Test Plan • Parallel Test Report • Plan for Transition to Live Operation of the e-ATIGA Form D • Schedule for Transition to e-ATIGA Form D Live Operation • Report of the 33rd TWG Meeting • ASW Gateway Technical Operations Guide • Operations Handbook • Change and Configuration Management Guide • Interoperability Specifications • ASEAN Administration Manual (for the ASW Regional Services Portal) • User Manual (for the ASW Regional Services Portal) • Scope of Work for the PMO Regional Services Training 4.2 Questionnaires Before commencing the report, the Consultant devised a set of questionnaires to complement the references in 4.1. In particular, questionnaires were considered essential for Tasks 2, 3, 4 and 7. Two questionnaires were devised for the participating AMS. The first to follow the completion of the end-to-testing (Phase 1) and the second to follow the completion of the parallel testing (Phase 2). 4.2.1 AMS Questionnaire (Part 1) The first questionnaire, attached as Appendix A, consisted of 40 questions, divided into 5 sections. This questionnaire was to be returned by the participating AMS by Monday, 9 th November 2015. As there were no completed questionnaires returned by this date, the Consultant agreed to extend the submission date to Friday, 27th November 2015 allowing sufficient time to complete the Draft Final Report. Subsequently, Malaysia and Singapore submitted their completed questionnaires. ASW Technical and Financial Evaluation – Final Report Page 25 of 80 The final date for the submission of the first questionnaire was later extended to Wednesday, 16th December 2015 to allow sufficient time to complete this Final Report. Completed questionnaires were received from Malaysia, Singapore and Thailand. Indonesia and Viet Nam submitted their completed questionnaires during the 34th TWG Meeting held in Manila, Philippines from the 1st to 4th February 2016. 4.2.2 AMS Questionnaire (Part 2) The second questionnaire, attached as Appendix B, consisted 13 questions, divided into 2 sections. This questionnaire was to be returned by the participating AMS by Monday, 7th December 2015. As the completion date for parallel testing was extended to the 8th January 2016, the Consultant had agreed to the final date for the submission of the second questionnaire to be extended to Friday, 15th January 2016, so that this Final Report could be revised accordingly. Completed questionnaires were received from Malaysia, Singapore and Thailand. Indonesia and Viet Nam submitted their completed questionnaires during the 34th TWG Meeting held in Manila, Philippines from the 1st to 4th February 2016. 4.3 Where the documents were used Table 2 below shows where each document, reference or questionnaire, was used to carry out Tasks 2-9 of the TOR. The detailed findings of each task are provided in the subsequent sections. Table 2: How the reference documents where used for each task of the TOR Task Description References 2 An assessment of the Contractor’s performance in terms of project management, efficiency and effectiveness in resolving Member States’ problems encountered during the testing  Request for Proposal  Questionnaires 3 An assessment of whether the implemented ASW messaging system complied with the agreed Request for Proposal for the Development and Implementation of the ASEAN Single Window Software published and distributed in November of 2014 in preparation for the agreed cutover to live operation by 30 December 2015  Request for Proposal  Contractor’s Technical Proposal in response to the Request for Proposal  Report of the 33rd TWG Meeting  Questionnaires  ASW Gateway Technical Operations Guide  Operations Handbook  Change and Configuration Management Guide  Interoperability Specifications  ASEAN Administration Manual  User Manual ASW Technical and Financial Evaluation – Final Report Page 26 of 80 Task Description References 4 An evaluation of the user acceptance process and success indicators of the ASW messaging system to ensure that all stated requirements are met  Request for Proposal  End-to-End Test Plan / Report  Parallel Test Plan / Report  Report of the 33rd TWG Meeting  Questionnaires 5 An evaluation of the network security (in terms of security vulnerabilities), scalability and suitability of the Contractor’s software (a) for the exchange of the ATIGA CO Form D; and (b) to accommodate additional cross-border documents  Component 1 Deliverables  Contractor’s Technical Proposal in response to the Request for Proposal 6 An assessment of whether the draft transition and migration (TM) plan for the cutover to live operation can be successfully implemented  Request for Proposal  Plan for Transition to Live Operation of the e-ATIGA Form D  Schedule for Transition to e￾ATIGA Form D Live Operation  Report of the 33rd TWG Meeting 7 To identify any other funding requirements needed to support the operation and expansion of the ASW (including the electronic exchange of the export data and the four possible additional documents covered in the Information Process Modeling study) not included in the proposed funding requirements agreed by the 15th ASWSC Meeting  ASW Regional IT Operational and Staffing Costs  Partnership Matrix for the Exchange of the ASEAN Customs Declaration Document (ACDD)  Information Process Modeling Study  Questionnaires 8 An assessment of the adequacy of the plans for technical training for the Project Management Office (PMO) Staff in the operation of the ASW Regional Services  Request for Proposal  Design of Project Management and Planning Team (Project Management) Office  ASEAN Administration Manual  User Manual  Scope of Work for the PMO Regional Services Training 9 An evaluation of the quality assurance procedures for the ASW Regional Operations Services covering:  Security quality – effectiveness of the security features implemented  Information quality – accuracy, meaningfulness, and timeliness of the information produced  Process quality – effect on information management process quality  Request for Proposal  Contractor’s Technical Proposal in response to the Request for Proposal ASW Technical and Financial Evaluation – Final Report Page 27 of 80 5 Detailed Findings 5.1 Contractor’s performance in resolving Member States’ problems The purpose of this section is to detail the findings of Task 2 of the TOR; “An assessment of the Contractor’s performance in terms of project management, efficiency and effectiveness in resolving Member States’ problems encountered during the testing”. 5.1.1 Findings The findings below are based on the RFP and the AMS Questionnaires (Parts 1 and 2) submitted by Indonesia, Malaysia, Singapore, Thailand and Viet Nam. 5.1.1.1 Project Management 5.1.1.1.1 Request for Proposal The following are extracts from the RFP relating to this section. “Apart from coordinating all activities/parties to ensure effective and successful pilot/parallel testing as part of project management, support from the Contractor will also cover troubleshooting any messaging system problems identified during the transition to live operations in the exchange of ATIGA;” 5.1.1.1.2 Feedback from AMS Questionnaire (Part 1) The feedback from AMS Questionnaire (Part 1) relating to this section is provided below. Q11. How do you rate the Contractor’s performance in Project Management? Please note that the options were; “Poor”, “Below Average”, “Average”, “Good” and “Excellent”. Based on the feedback from the participating AMS through the AMS Questionnaire (Part 1), the general view is that the Contractor’s performance, in terms of their project management in resolving Member States’ problems encountered during the end-to-end testing, was “Average” to “Excellent”. Singapore commented that they felt the Contractor’s oversight of the end-to-end testing was lacking at times. For example, there was a lack of coordination and oversight on the different e-ATIGA Form Ds sent and received by the AMS. 5.1.1.1.3 Feedback from AMS Questionnaire (Part 2) The feedback from AMS Questionnaire (Part 2) relating to this section is provided below. Q11. How do you rate the Contractor’s performance in Project Management? Please note that the options were; “Poor”, “Below Average”, “Average”, “Good” and “Excellent”. Based on the feedback from the participating AMS through the AMS Questionnaire (Part 2), the general view is that the Contractor’s performance, in terms of their project management in resolving Member States’ problems encountered during the parallel testing, was “Average” to “Excellent”. 5.1.1.2 Efficiency 5.1.1.2.1 Feedback from AMS Questionnaire (Part 1) The feedback from AMS Questionnaire (Part 1) relating to this section is provided below. ASW Technical and Financial Evaluation – Final Report Page 28 of 80 Q12. How do you rate the Contractor’s efficiency in responding to your problems? Please note that the options were; “Poor”, “Below Average”, “Average”, “Good” and “Excellent”. Based on the feedback from the participating AMS through the AMS Questionnaire (Part 1), the general view is that the Contractor’s performance, in terms of their efficiency in resolving Member States’ problems encountered during the end-to-end testing, was “Average” to “Good”. Malaysia commented that the Contractor provided immediate support when issues were reported to them. However, they also felt that there was a lack of information on how to utilize the services provided by the ASW Regional Services and on the technical issues that were being faced by the other AMS. Singapore commented that they felt it would have been helpful if the Contractor had stationed staff at each AMS during the testing, so that issues could have been resolved immediately. In particular, they sometimes found it difficult to determine at what point messages were i.e. the ASW Gateway, Web Server Proxy or NSW. 5.1.1.2.2 Feedback from AMS Questionnaire (Part 2) The feedback from AMS Questionnaire (Part 2) relating to this section is provided below. Q12. How do you rate the Contractor’s efficiency in responding to your problems? Please note that the options were; “Poor”, “Below Average”, “Average”, “Good” and “Excellent”. Based on the feedback from the participating AMS through the AMS Questionnaire (Part 2), the general view is that the Contractor’s performance, in terms of their efficiency in resolving Member States’ problems encountered during the parallel testing, was “Average” to “Good”. Malaysia commented that the Contractor has continued to provide immediate support. Singapore re-iterated the points they made following the end-to-end testing (see section 5.1.1.2.1). They felt that the issues encountered during the parallel testing could have been conveyed in greater clarity to AMS and tracked more comprehensively by Axway. They also felt that it would have been helpful to have a single point of contact to be assigned to each AMS so that the assigned Axway personnel would be able to follow through the issues encountered and provide resolutions in a more holistic manner. 5.1.1.3 Effectiveness 5.1.1.3.1 Feedback from AMS Questionnaire (Part 1) The feedback from AMS Questionnaire (Part 1) relating to this section is provided below. Q13. How do you rate the Contractor’s effectiveness in resolving your problems? Please note that the options were; “Poor”, “Below Average”, “Average”, “Good” and “Excellent”. Based on the feedback from the participating AMS through the AMS Questionnaire (Part 1), the general view is that the Contractor’s performance, in terms of their effectiveness in resolving Member States’ problems encountered during the end-to-end testing, was “Average” to “Good”. 5.1.1.3.2 Feedback from AMS Questionnaire (Part 2) The feedback from AMS Questionnaire (Part 2) relating to this section is provided below. ASW Technical and Financial Evaluation – Final Report Page 29 of 80 Q13. How do you rate the Contractor’s effectiveness in resolving your problems? Please note that the options were; “Poor”, “Below Average”, “Average”, “Good” and “Excellent”. Based on the feedback from the participating AMS through the AMS Questionnaire (Part 2), the general view is that the Contractor’s performance, in terms of their effectiveness in resolving Member States’ problems encountered during the parallel testing, was “Average” to “Good”. 5.1.2 Assessment Overall, based on the feedback from the AMS, the Contractor’s performance has been “Average” to “Excellent”. The main concerns appear to be that the AMS felt that they were left alone at times and there was a lack of communication between the various parties involved. 5.2 Contractor’s compliance with the Request for Proposal The purpose of this section is to detail the findings of Task 3 of the TOR; “An assessment of whether the implemented ASW messaging system complied with the agreed Request for Proposal for the Development and Implementation of the ASEAN Single Window Software published and distributed in November of 2014 in preparation for the agreed cutover to live operation by 30 December 2015”. 5.2.1 Findings The findings below are based on the RFP, the “Contractor’s Technical Proposal in response to the RFP”, “Report of the 33rd TWG Meeting”, “ASW Gateway Technical Operations Guide”, “Operations Handbook”, “Change and Configuration Management Guide”, “Interoperability Specifications”, “ASEAN Administration Manual”, “User Manual” and the AMS Questionnaires (Part 1) submitted by Malaysia, Singapore and Thailand. 5.2.1.1 ASW Gateway / Software 5.2.1.1.1 Request for Proposal The following are extracts from the RFP relating to this section. “Assumption 1: Core characteristics of the proposed Component 1 technical design architecture are the following:  To provide the necessary mechanism to enable exchange-ready AMS to exchange the ATIGA Form D, electronically on an end-to-end basis from one NSW to other NSWs in an integrated, secured ASW environment adhering to the technical design of the ASW Pilot Project Component 1;  The ASW software shall implement functionalities that enable robust, consistent and secure application level information exchange (consisting of correct routing, 100% message delivery including processing of acknowledgement responses pushed by the receiving NSW, one time only delivery and security as per ASW Pilot Project Component 1 deliverables);  Software for the ASW must be hosted in each AMS;  The ASW software must employ queuing and message delivery re-try mechanisms to ensure delivery of messages and provide error response for unsuccessful delivery;  The ASW software must not initiate connections to the NSWs. Messages received from other ASW software must be put into queues for retrieval. The system should ensure that an alert message is generated after a specific time if the message is not retrieved by the NSW; ASW Technical and Financial Evaluation – Final Report Page 30 of 80  NSWs must employ a periodic calling mechanism to retrieve data from ASW software. NSWs must be able to acknowledge receipt of the message retrieved from the ASW software (the Contractor should revise the four interface scripts developed in the ASW Pilot Project Component 1 to include message reference number and a response back to the ASW software i.e. web services, JMS, file system and secure File Transfer Protocol);  The ASW software will not perform data encoding format and/or message standards conversions. The NSW of respective AMS is expected to perform this function by using the data encoding format and message standard tested during the scaled-down pilot;  The full ASW Regional Services functionality as indicated in the Component 1 design architecture must be implemented to cover the updating of the reference data (national reference data and common reference data) and Management Information Systems (MIS), including automated monitoring of message exchanges between AMS, and network activity and performance monitoring to ensure continuous capacity, statistical logs, network visibility;” “Assumption 2: Implementation of Component 2 must adhere to the core characteristics of the Component 1 technical design architecture as agreed upon by AMS and the message exchange protocol that was used during the ASW pilot component 2 scaled-down.” “Assumption 5: Local NSW perimeter networks may be implemented and the proposed solution should be able to operate in such an environment. Member States have the discretion to determine its perimeter network for the ASW software.” “Assumption 6: The ASW software, which is hosted by each individual Member State that has the full control of its security, must provide the capability to configure what information is stored permanently, temporarily or not stored at all on ASW software (it includes messages that are being exchanged through the ASW software). The use of such capability is optional and should be configurable by the Member States.” “Assumption 12: The contractor shall develop the capability within the RS Portal to support the optional use of PKI and digital certificates in the exchange of data by NSWs in accordance with the agreed technical design architecture (Component 1 Deliverables).” “The contractor will provide the NSW interface scripts installation manual that describes the step-by￾step installation covered for the NSW integration module at each of the AMS. Moreover, an administration manual will also be provided that describes the administration functions for the NSW integration module installation at each of the AMS.” 5.2.1.1.2 Contractor’s Technical Proposal in response to the RFP The following are extracts from the “Contractor’s Technical Proposal in response to the RFP” relating to this section. Assumption 1, 1st Bullet Point: “The Axway B2B software provides all the necessary functionality to securely exchange ATIGA Form D between AMS, offering each AMS a suite of NSW integration options to meet their specific needs, while adhering to the technical design of the ASW Pilot Project Component 1. As well as the exchange of ATIGA Form D, the Axway B2B software is designed to exchange additional message types, including XML, if required in the future.” Assumption 1, 2nd Bullet Point: “The exchange functionality performs correct routing, 100% message delivery with reliable messaging built-in. Receipt verification, with signed digital receipts, is also supported for non-repudiation, a key element of reliable messaging. The signed receipt contains the ASW Technical and Financial Evaluation – Final Report Page 31 of 80 unique identifier of the received message, digest value(s) of the message (parts), a timestamp when the event was generated, a signature covering the above. Enhanced Acknowledgement Status responses triggered by the receiving NSW are also included” Assumption 1, 4 th Bullet Point: “The exchange functionality has queuing and message delivery re-try built in to the protocol standard, ensuring delivery of messages and error responses for unsuccessful delivery. The number of times the ASW Gateway will retry connecting, if the initial attempt to connect and send the message fails, is configurable on a per AMS (trading partner) basis. Retries occur according to an algorithm. Timings for each retry attempt are also configurable for each AMS.” Assumption 1, 5 th Bullet Point: “The Axway B2B software was designed and configured from the outset not to initiate connections to the NSW. The solution includes alerting to generate an exception warning if the message is not received by the NSW within a configurable time period.” Assumption 1, 6 th Bullet Point: “For ASW Pilot (Live) an enhancement will be made to utilize a new bridge script, so the receiving NSW can generate the Acknowledgement Status Message (AS-3) when the message is retrieved from the ASW Gateway. This will be supported for the four protocol options for NSW Integration.” Assumption 1, 7 th Bullet Point: “The Axway B2B Software proposed for the ASW Pilot (Live) will not perform data encoding format and/or message standards conversions, although it has this functionality.” Assumption 6: “Access to all aspects of the Axway B2B software is strictly governed by a role-based authentication mechanism. Administrator and delegated user specific roles can be easily configured. Once configured, the role can then be allocated to trusted internal users. Such users are absolutely conditioned on the functions they can perform and information viewed. Each individual member state has complete control of their own configuration. For example, setting the duration of information (exchanged messages, related events) to be stored on the ASW Gateway server and related database.” Assumption 12: “Axway B2B Software for the ASW and Regional Services Gateways comes with a Public Key Infrastructure Support including Security Models: RSA, VeriSign, Entrust, Baltimore certificates Self-signed certificates, VeriSign XKMS and Entrust PKIX-CMP certificate Lifecycle management Security Algorithms Encryption: RC2, RC4, DES, 3DES, AES – 128, 192 and 256 key length Signature: MD5, SHAI. In the ASW Scaled Down Pilot all exchanges were conducted using secure electronic data messages, to ensure privacy and data integrity through encryption of data and digital signatures. Document level encryption, as well as encryption at the communication level was provided.” 5.2.1.1.3 Feedback from AMS Questionnaire (Part 1) The feedback from AMS Questionnaire (Part 1) relating to this section is provided below. Q14. Was the ASW server and software successfully installed onto your NSW network? This question refers to the 3rd bullet point of Assumption 1 (page 7) and Assumption 5 of the RFP (page 8). Indonesia, Malaysia, Singapore and Viet Nam all answered “Yes” to this question. This question is not applicable to Thailand, because they are using their own gateway. ASW Technical and Financial Evaluation – Final Report Page 32 of 80 Q15. Does the ASW software have a queuing mechanism for message delivery? This question refers to the 4th bullet point of Assumption 1 of the RFP (page 7). Indonesia, Singapore and Viet Nam all answered “Yes” to this question, while Malaysia answered “No”. Singapore commented that while the Contractor’s software does have a queuing mechanism for message delivery, it could be better enhanced for the ASW environment. This question is not applicable to Thailand, because they are using their own gateway. Q16. Does the ASW software have re-try mechanisms to ensure delivery of messages? This question refers to the 4th bullet point of Assumption 1 of the RFP (page 7). Indonesia, Singapore and Viet Nam all answered “Yes” to this question, while Malaysia answered “No”. Singapore commented that while the Contractor’s software does have re-try mechanisms to ensure the delivery of messages, it could be better enhanced for the ASW environment. This question is not applicable to Thailand, because they are using their own gateway. Please note that the re-try mechanism is briefly mentioned in the “ASW Gateway Technical Operations Guide” (section 17.6), provided by the Contractor. It states that “When submitting ATIGA Form D messages to another AMS, your ASW Gateway Software will go into ‘re-try’ mode if the external ASW Gateway is down for some reason. After the configurable number of re-try attempts has been attempted without success, then the ASW Gateway will classify the status of the transaction as ‘Failed’’. Q17. Does the ASW software provide error responses for unsuccessful delivery? This question refers to the 4th bullet point of Assumption 1 of the RFP (page 7). Indonesia, Singapore and Viet Nam all answered “Yes” to this question, while Malaysia answered “No”. Singapore commented that while the Contractor’s software does provide error responses for unsuccessful delivery, it could be better enhanced for the ASW environment. This question is not applicable to Thailand, because they are using their own gateway. Q18. Does the ASW software put messages received from other ASW software into queues for retrieval? This question refers to the 5th bullet point of Assumption 1 of the RFP (page 7). Indonesia, Singapore and Viet Nam all answered “Yes” to this question, while Malaysia answered “No”. Singapore commented that while the Contractor’s software does put messages received from other ASW software into queues for retrieval, it could be better enhanced for the ASW environment. This question is not applicable to Thailand, because they are using their own gateway. Q19. Do you receive alert messages generated after a specific time if the message is not retrieved by the NSW? This question refers to the 5th bullet point of Assumption 1 of the RFP (page 7). Malaysia, Singapore and Viet Nam all answered “No” to this question, while Indonesia answered “Yes”. ASW Technical and Financial Evaluation – Final Report Page 33 of 80 This question is not applicable to Thailand, because they are using their own gateway. The Contractor has advised that this feature has been developed and tested, and it is documented in the “Interoperability Specifications” (section 4.7.5). The document states; “This feature is optional and available for integration using web services, file system, and FTP connection. A SMTP server is required since this alert message will be pushed to a recipient using email. Axway does not provide an SMTP Server.” Q20. Are you able to acknowledge the receipt of the message retrieved from the ASW software using Web Services, JMS, FTP or sFTP? This question refers to the 6th bullet point of Assumption 1 of the RFP (page 7). Indonesia, Malaysia and Singapore all answered “Yes” to this question, while Viet Nam answered “No”. This question is not applicable to Thailand, because they are using their own gateway. Q21. Has the Contractor provided you with an installation manual and interface scripts? This question refers to the Scope of Work and Deliverables (page 14, 2nd paragraph). Malaysia, Singapore and Viet Nam all answered “No” to this question, while Indonesia answered “Yes”. This question is not applicable to Thailand, because they are using their own gateway. The Contractor has advised that installation guides for bridge scripts have been developed for Web Services, JMS and FTP. Interface specifications have also been documented in the “Interoperability Specifications” (section 4). Q22. Do you have full control of the ASW software’s security? This question refers to Assumption 6 of the RFP (page 8). Malaysia, Singapore and Viet Nam all answered “No” to this question, while Indonesia answered “Yes”. This question is not applicable to Thailand, because they are using their own gateway. The Contractor has advised that, while the AMS is responsible for the security of their own infrastructure / environment where the ASW Gateway is deployed, the ASW Gateway “provides comprehensive security features including the secure ebMS protocol, PKI certificate management, and the Secure Relay as a reverse proxy deployed in the DMZ”. Q23. Are you able to configure what information is stored permanently, temporarily or not stored at all on the ASW software? This question refers to Assumption 6 of the RFP (page 8). Malaysia, Singapore and Viet Nam all answered “No” to this question, while Indonesia answered “Yes”. This question is not applicable to Thailand, because they are using their own gateway. ASW Technical and Financial Evaluation – Final Report Page 34 of 80 The Contractor has advised that the “ASW Gateway Technical Operations Guide” (see section 14) includes a guide on how to configure the length of time for storage of messages and logs. According to this guide, “the ASW Gateway will be configured to keep 90 days of data, from commencement of message exchanges on the respective production gateway, after which it will be automatically archived. The Purge Data Configuration can be configured under Trading Configuration”. It also states that; “You can configure the ASW Gateway Software so that backed-up messages of a certain age are deleted. This lets you delete unwanted database records and backup files. When triggered, database records about documents and the corresponding files in the backup directory are deleted. Once deleted, you no longer can search for, view or reprocess these documents in Message Tracker”. Q24. Does the ASW software give you the option to use PKI and digital certificates? This question refers to Assumption 12 of the RFP (page 9). Indonesia, Singapore and Viet Nam all answered “Yes” to this question, while Malaysia answered “No”. This question is not applicable to Thailand, because they are using their own gateway. Interface specifications for the exchange of Public Key Certificates have been documented in the “Interoperability Specifications” (section 6). Q25. Does the ASW software perform any data encoding format and/or message standards conversions? This question refers to the 7th bullet point of Assumption 1 of the RFP (page 7). Indonesia, Singapore and Viet Nam all answered “Yes” to this question, while Malaysia answered “No”. This question is not applicable to Thailand, because they are using their own gateway. Q26. Has the Contractor provided you with an administration manual? This question refers to Scope of Work and Deliverables (page 14, 2nd paragraph). Indonesia and Singapore both answered “Yes” to this question, while Malaysia and Viet Nam both answered “No”. Singapore commented that it is incomplete. This question is not applicable to Thailand, because they are using their own gateway. The Contractor has provided the “ASW Gateway Technical Operations Guide”, which covers many administrative functions of the ASW Gateway / Software, including starting and stopping the ASW Gateway, setting up “Trading Partner Profiles”, archiving and purging, backing up and restoring etc. Q27. In your opinion, has the Contractor adhered to the technical design of Component 1? This question refers to the 1st and 2nd bullet points of Assumptions 1 of the RFP (page 7). Indonesia, Singapore and Viet Nam all answered “Yes” to this question, while Malaysia answered “No”. This question is not applicable to Thailand, because they are using their own gateway. Additional Comments ASW Technical and Financial Evaluation – Final Report Page 35 of 80 Malaysia commented that the Contractor did not have sufficient time to provide training on managing the ASW Gateway server. For example, this should include preparation for maintenance downtime, support / helpdesk during the pilot testing and understanding the B2Bi user manual. 5.2.1.1.4 Report of the 33rd TWG Meeting It was reported during the 33rd TWG Meeting that the Contractor had submitted the “ASW Gateway Technical Operations Guide” which provides “the technical day-to-day administrative aspects of operating the ASW Gateway and necessary guidance to complete related operational tasks”. It was also reported that the Contractor had submitted the “Operations Handbook”, which covered “the operational procedures of the ASW e-ATIGA Form D service and related ASW infrastructure with references, as applicable, to related documents, where more detailed information and operations would be available”. The Contractor agreed to provide hands-on training to Member States on ASW Gateway / Software prior to the cutover to live operation. 5.2.1.2 ASW Regional Services 5.2.1.2.1 Request for Proposal The following are extracts from the RFP relating to this section. “Assumption 1: Core characteristics of the proposed Component 1 technical design architecture are the following:  The full ASW Regional Services functionality as indicated in the Component 1 design architecture must be implemented to cover the updating of the reference data (national reference data and common reference data) and Management Information Systems (MIS), including automated monitoring of message exchanges between AMS, and network activity and performance monitoring to ensure continuous capacity, statistical logs, network visibility;” “Assumption 4: Contractor shall develop and maintain the ASW Regional Services (RS) Portal during phases 1 and 2 to support AMS risk management systems by providing synchronized reference data sets for validation and control. The contractor will be provided the code for the RS Portal that was used in implementing the Component 2 (Scaled Down) Pilot for review and possible use in implementing this contract.” “Assumption 7: The ASW Regional Services server shall be hosted at the ASEAN Secretariat (ASEC) during phase 1 and 2. Whether the server remains at ASEC or is moved to a data center thereafter will be determined at a later date.” “The Contractor shall operate the Regional Services and assist AMS in operation of their ASW software during the testing phase (Phase 1 and phase 2). The Contractor will provide a written manual on the operation and maintenance of the regional services software and any reporting systems.” “The Contractor will provide input to the Regional Operations team, which will succeed the VPT-IT Team mentioned in Assumption 11 once the latter is in place, in linking the ASW web portal (asw.asean.org) with the ASW RS portal” ASW Technical and Financial Evaluation – Final Report Page 36 of 80 5.2.1.2.2 Contractor’s Technical Proposal in response to the RFP The following are extracts from the “Contractor’s Technical Proposal in response to the RFP” relating to this section. Assumption 1, 8th Bullet Point: “In line with Component 1 design architecture the ASW Regional Services Portal manages the reference data for all AMS, including updates and authorization (ASEAN Administrator, AMS Administrator, AMS User) for each data set, including Common Reference Data ( ISO Country Codes, UN Location (Port) Codes, Package Type, Measurement Unit, ISO Country Code, Economic Operator ID Code) and National Reference Data (Harmonized System Codes, Customs Code, Government Agency Code, License Code List). The Portal also provides a Management Information Systems (MIS) Module with PKI Trusted Certificates, ASW Gateway monitoring for all participating AMS, Message Transaction Monitoring and Reporting (PKI Certificates, Transaction Reports) with real time visibility into the health of each ASW Gateways and exchange activity between AMS, including performance statistics. ” Assumption 4: “The ASW Regional Services (RS) Portal provides synchronized reference data set management for validation and control for all AMS covering Common Reference and National Reference Data.” 5.2.1.2.3 Feedback from AMS Questionnaire (Part 1) The feedback from AMS Questionnaire (Part 1) relating to this section is provided below. Q36. Has the ASW Regional Services (RS) Portal been installed as part of the ASW Regional Services? This question refers to Assumption 4 of the RFP (page 8). Indonesia, Malaysia, Singapore and Viet Nam all answered “Yes” to this question. Q37. Does the ASW Regional Services provide functionality to update reference data? This question refers to the 8th bullet point of Assumption 1 of the RFP (page 7). Indonesia, Malaysia and Viet Nam all answered “Yes” to this question, while Singapore answered “No”. Viet Nam commented that they were only able to update national reference data. The Contractor has advised that, while reference data is not required for the e-ATIGA Form D, a Reference Data Management system was developed and tested for ASW Pilot Project Component 2 (Scaled Down). These features have been documented in the ASW Regional Services Portal’s “ASEAN Administration Manual” and “User Manual”. Q38. Does the ASW Regional Services provide automated monitoring of message exchanges between AMS? This question refers to the 8th bullet point of Assumption 1 of the RFP (page 7). Indonesia, Malaysia and Viet Nam all answered “Yes” to this question, while Singapore answered “No”. The Contractor has advised that daily statistics are automatically sent from each ASW Gateway to the ASW Regional Services, which are then made available for viewing and monitoring. This has been documented in the “Operations Handbook” (section 4). ASW Technical and Financial Evaluation – Final Report Page 37 of 80 Q39. Does the ASW Regional Services provide network activity and performance monitoring to ensure continuous capacity, statistical logs and network visibility? This question refers to the 8th bullet point of Assumption 1 of the RFP (page 7). Indonesia and Viet Nam both answered “Yes” to this question, while Malaysia and Singapore both answered “No”. Viet Nam commented that the ASW Regional Services only told them whether the ASW Gateway was active or inactive. Q40. Has the VPT-IT been involved in the set-up of the ASW Regional Services (RS) Portal? This question refers to Scope of Work and Deliverables (page 14, 10th paragraph). Indonesia, Malaysia, Singapore and Viet Nam all answered “No” to this question. 5.2.1.3 Documentation 5.2.1.3.1 Request for Proposal The following are extracts from the RFP relating to this section. “The Contractor shall provide a written set of specifications to assist any AMS in the development of national software to exchange information through the ASW system;” “The Contractor shall submit comprehensive monthly reports on activities during the duration of the project;” “The Contractor shall submit a fortnightly statistics report with detailed analysis of issues encountered and proposed solutions during the testing process;” “The contractor must provide a version control procedure to track and merge changes in the operation of the ASW software and other software applications over time. This would provide a convenient and effective way of monitoring orderly changes or upgrades to minimize disruptions. Further, automate management of revision history in effecting upgrades and software patches of the ASW software or any other required software;” “The contractor shall provide an information security incident management plan. This will allow the effective handling of information security incidents. Lack of clearly defined organizational structure can create confusion, resulting in each phase of a response taking longer than necessary;’ 5.2.1.3.2 Feedback from AMS Questionnaire (Part 1) The feedback from AMS Questionnaire (Part 1) relating to this section is provided below. Q31. Has the Contractor submitted comprehensive monthly reports on activities during the duration of the project? This question refers to Scope of Work and Deliverables (page 14, 7th paragraph). Indonesia, Singapore and Viet Nam all answered “Yes” to this question, while Malaysia answered “No”. Singapore assumed that “test status matrix” meets this requirement. The Contractor has confirmed that the “test matrix”, which is produced on a weekly basis, was intended to meet this requirement. ASW Technical and Financial Evaluation – Final Report Page 38 of 80 Q32. Has the Contractor submitted a fortnightly statistics report with detailed analysis of issues encountered and proposed solutions during the testing process? This question refers to Scope of Work and Deliverables (page 14, 8 th paragraph). Indonesia, Singapore and Viet Nam all answered “Yes” to this question, while Malaysia answered “No”. Singapore assumed that the “weekly log” meets this requirement. The Contractor has confirmed that the “weekly log” was intended to meet this requirement. Q33. Has the Contractor provided a version control procedure to track and merge changes in the operation of the ASW software and other software applications over time? This question refers to Scope of Work and Deliverables (page 14, 9th paragraph). Indonesia and Viet Nam both answered “Yes” to this question, while Malaysia and Singapore both answered “No”. The Contractor has advised that a version control procedure has been documented in the “Change and Configuration Management Guide” (section 2.3). Q34. Has the Contractor provided an information security incident management plan to allow the effective handling of information security incidents? This question refers to Scope of Work and Deliverables (page 14, 11th paragraph). Indonesia, Malaysia, Singapore and Viet Nam all answered “No” to this question. The Contractor has advised that an information security incident management plan has been documented in the “Operations Handbook” (see section 3.6). Q35. Has the Contractor provided a written set of specifications to assist any AMS in the development of national software to exchange information through the ASW system? This question refers to Scope of Work and Deliverables (page 14, 6th paragraph). Indonesia and Viet Nam both answered “Yes” to this question, while Malaysia answered “No”. Singapore commented that this question is not applicable to them, as they have developed their own bridge script. The Contractor has provided a document called “Interoperability Specifications”. It covers various interface specifications, including specifications for bridge scripts using FTP, JMS and Web Services. 5.2.2 Assessment 5.2.2.1 ASW Gateway / Software In general terms, the Contractor’s software appears to comply with the requirements of the RFP for the ASW Gateway / Software. Also, based on the feedback from the AMS through the questionnaire, the Contractor’s software has been successfully installed onto the NSW networks. However, it is clear that the administration manual, the “ASW Gateway Technical Operations Guide”, does not meet the expectations of the participating AMS. There are several features, which are available in the Contractor’s software, that require more detailed explanation, especially how the ASW Technical and Financial Evaluation – Final Report Page 39 of 80 features are being used or can be configured within the ASW environment. As minimum, the features that should be documented are:  Queuing mechanism for message delivery  Re-try mechanisms to ensure the delivery of messages  Error responses for unsuccessful delivery  Messages received from other ASW software into queues for retrieval  Alert messages generated after a specific time if the message is not retrieved by the NSW  ASW software’s security  Configuring what information is stored permanently, temporarily or not stored at all  PKI and digital certificates  Data encoding format and/or message standard conversions The participating AMS also do not seem to be aware that the Contractor has developed installation guides and bridge scripts for Web Services, JMS and FTP. Recommendation No. 1 To ensure that all of the features of the ASW Gateway / Software, as stated in the RFP, are thoroughly documented in the “ASW Gateway Technical Operations Guide”. At a minimum, this should cover queuing and re-try mechanisms, error responses for unsuccessful delivery, alert messages, users and roles, archiving / purging of data, use of PKI and digital certificates, installation manuals and interface scripts. 5.2.2.2 ASW Regional Services In general terms, the ASW Regional Services appears to comply with the requirements of the RFP. There have also been no reported issues in installing the ASW Regional Services (RS) Portal as part of the ASW Regional Services. The Contractor has provided two manuals for the ASW Regional Services (RS) Portal, including the “ASEAN Administration Manual” and the “User Manual”. Both of these manuals cover the updating of reference data and how to produce transaction reports. However, it is clear that the ASW Regional Services is still cause for concern by the participating AMS, which could be partly due to the completeness of the documentation. Concerns were also raised during the end-to-end testing about the accuracy of the transactional reporting and the level of automated monitoring of message exchanges between AMS, network activity and performance monitoring. Perhaps the biggest concern is that the VPT-IT have not been involved in the set-up of the ASW Regional Services Portal and have also not been trained in how to use it. This issue is covered in more detail in Section 5.7. Recommendation No. 2 To ensure that all of the features of the ASW Regional Services, as stated in the RFP, are thoroughly documented in the “ASEAN Administration Manual” and “User Manual”. At a minimum, this should include; the updating of reference data, automated monitoring of message exchanges between AMS, MIS reporting and performance monitoring. ASW Technical and Financial Evaluation – Final Report Page 40 of 80 5.2.2.3 Documentation In addition to the documentation described in the previous sections for the ASW Gateway / Software and ASW Regional Services, the Contractor also appears to have met the other documentation requirements stated in the RFP, including:  Comprehensive Monthly Reports => Test Matrix  Fortnightly Statistics Report => Weekly Log  Version Control Procedures => Change and Configuration Management Guide (section 2.3)  Security Incident Management => Operations Handbook (section 3.6)  Written Specifications => Interoperability Specifications As a general comment, although the Contractor has provided a lot of documentation, the Consultant found it difficult to determine if / how the Contractor had complied with the RFP requirements. It is therefore not surprising that the AMS do not seem to be aware that many of the RFP requirements have been met and documented. One reason for this is that the full titles of the documentation are a bit misleading, as they imply that they have been written specifically for the e-ATIGA Form D. For example:  ASW Pilot e-ATIGA Form D - ASW Gateway Technical Operations Guide  ASW Pilot e-ATIGA Form D - Operations Handbook  Development and Live Implementation of the ASEAN Single Window (ASW) Software for the exchange of the ATIGA CO Form D - Change and Configuration Management Guide It should be noted that, when asked, the Contractor was very helpful in pointing out where the information could be found. That said, it is suggested that it would be easier for new users to find information if the documentation was renamed. A simple change to the titles of the documentation could make it easier for AMS users to find information in future. For example, the “ASW Pilot e￾ATIGA Form D - ASW Gateway Technical Operations Guide” could be renamed to something like the “ASW Gateway – Administration Manual”. The content, of course, is the most important issue. It is essential that the documentation is generic and relevant for all documents, not just the e-ATIGA Form D. For example, the “ASW Pilot e-ATIGA Form D - Operations Handbook” seems to have a mixture of specific e-ATIGA Form D information and information that should be more generic. It is therefore suggested that generic information, such as the support procedures and the security incident management, should be documented separately from e-ATIGA Form D specific documentation. Recommendation No. 3 To ensure that documentation is generic and relevant for all documents, not just the e-ATIGA Form D. For example, the support procedures and security incident management plan is relevant for all documents and should be documented separately from the “ASW Pilot e-ATIGA Form D - Operations Handbook”. It is also suggested that the “ASW Pilot e-ATIGA Form D - ASW Gateway Technical Operations Guide” should be renamed, so that it is clear that it is relevant for all documents, and that the specific ATIGA Form D related functionality should be consolidated into “ASW Pilot e-ATIGA Form D - Operations Handbook”. The title of the “Development and Live Implementation of the ASEAN Single Window (ASW) Software for the exchange of the ATIGA CO Form D - Change and Configuration Management Guide” may just need to be renamed to a more generic title. ASW Technical and Financial Evaluation – Final Report Page 41 of 80 5.3 User acceptance and success indicators of the ASW messaging system The purpose of this section is to detail the findings of Task 4 of the TOR; “An evaluation of the user acceptance process and success indicators of the ASW messaging system to ensure that all stated requirements are met”. 5.3.1 Findings The findings below are based on the RFP, “End-to-End Test Plan”, “End-to-End Test Report”, “Parallel Test Plan”, “Parallel Test Report”, “Report of the 33rd TWG Meeting” and the AMS Questionnaires (Parts 1 and 2) submitted by Malaysia, Singapore and Thailand. 5.3.1.1 End-to-End Testing 5.3.1.1.1 Request for Proposal The following are extracts from the RFP relating to this section. “Assumption 8: The development and testing in this Project will include a full/turn-around end to end testing of information/message exchange from one NSW to other NSWs i.e. activities in addition to the ASW software connectivity testing. A full/turn-around includes the utilization of the ATIGA Form D by the government agency and actors involved. The contractor should be able to provide an end-to￾end business process flow of the electronic ATIGA Form D including validation schemas for replacement, utilization, and cancelation procedures that must be implemented by an AMS NSW.” 5.3.1.1.2 End-to-End Test Plan According the “End-to-End Test Plan”, the objectives of the end-to-ending was: a) “To conduct end-to-end testing for the exchange of the electronic ATIGA Form D and the related Customs Responses, together with a sampling of corresponding paper versions, from the source application of the Certificate Issuing Agency (CIA), such as that used by Department of Foreign Trade for the approval/certification of the ATIGA Form D, to the destination application such as that used by Customs to process imports b) To review, and if necessary revise the agreed ASEAN data structure for the ATIGA Form D and e-ATIGA Form D Related Documents to ensure the data structures are compatible for all the AMS participating in the end-to-end testing c) To identify any data discrepancies in the area of the Sending AMS - for example, in the generation / processing of the ATIGA Form D and Related Documents by the applications or NSW of the Sending AMS d) To identify any data discrepancies or validation discrepancies in the area of the Receiving AMS – for example, in the processing / receiving of the ATIGA Form D and Related Documents by the applications or NSW of the receiving AMS e) To test the modified Acknowledgement (AS-3), MIS Reporting and Alerts f) To put in place action plans to address the data discrepancies identified – these action plans may cover the applications and NSW of each AMS participating in the tests, the ASEAN ATIGA Form D data structure and the ASW” The “End-to-End Test Plan” also lists the acceptance criteria as follows: 1. Successful end-to-end flows – Exporting Government to Importing Government systems (ATIGA Form D) between all participating AMS for scenarios involving all involved systems ASW Technical and Financial Evaluation – Final Report Page 42 of 80 NOTE: Each sending party is responsible to keep the transaction log of the message sent for tracking and post audit purpose. i. Original ATIGA Form D – all categories ii. Customs response – for all status (Received; Not Processed; Preferential Treatment Given; Preferential Treatment Not Given) iii. Technical acknowledgements: AS-1, AS-2, AS-3 iv. For 10% of the number of transactions executed in one month (volume to be confirmed and detailed by AMS). It is assumed there will be no end-to-end STRESS testing. AMS may conduct their own stress testing v. All critical and major issues identified during the test are resolved 2. Proven Harmonization of ATIGA Form D and Response message structures between all participating AMS 3. There is internal AMS agreement that test is successful and can proceed to the next stage. This may include internal assessment 4. The Disaster Recovery for the ASW Gateway is tested if a DR environment is available for the AMS (Internal to AMS including ASW Gateway) 5. Support Procedures Tested (Internal and Cross Border) 6. The daily reporting from the ASW Gateway, and the Regional Services MIS Reporting is correct 7. Note: It is agreed that there will be no consideration of potential business improvements, , in the acceptance criteria for the Start of the Parallel Test or the start of Live Operation” 5.3.1.1.3 Feedback from AMS Questionnaire (Part 1) The feedback from AMS Questionnaire (Part 1) relating to this section is provided below. Q1. Did you successfully connect to all participating AMS? Indonesia, Malaysia, Singapore, Thailand and Viet Nam all answered “Yes” to this question. Q2. How many ATIGA Form Ds did you send and how many responses did you receive? This question refers to item 1, bullet point (iii) of the acceptance criteria in the “End-to-End Test Plan”. Indonesia, Malaysia, Singapore, Thailand and Viet Nam all responded to this question. Tables 3-7 provide a breakdown of the number of ATIGA Form Ds they sent and the number of corresponding responses they received from the receiving AMS. Table 3: Indonesia’s breakdown of ATIGA Form Ds sent and corresponding responses received AMS ATIGA Form D Sent Technical Acknowledgements Customs Responses Received AS-1 Received AS-2 Received AS-3 Received Indonesia - - - - - Malaysia 1,497 1,497 1,497 1,497 0 Singapore 363 363 363 0 0 Thailand 1,408 1,408 1,408 1,408 0 Viet Nam 1,068 1,068 1,068 1,068 1,068 Total: 4,336 4,336 4,336 3,973 1,068 ASW Technical and Financial Evaluation – Final Report Page 43 of 80 Table 4: Malaysia’s breakdown of ATIGA Form Ds sent and corresponding responses received AMS ATIGA Form D Sent Technical Acknowledgements Customs Responses Received AS-1 Received AS-2 Received AS-3 Received Indonesia 11 11 11 11 0 Malaysia - - - - - Singapore 11 11 11 0 0 Thailand 12 12 0 12 0 Viet Nam 46 46 46 46 7 Total: 80 80 68 69 7 Table 5: Singapore’s breakdown of ATIGA Form Ds sent and corresponding responses received AMS ATIGA Form D Sent Technical Acknowledgements Customs Responses Received AS-1 Received AS-2 Received AS-3 Received Indonesia 89 - 88 - - Malaysia 92 - 21 - - Singapore - - - - - Thailand 93 - 20 - - Viet Nam 92 - 50 16 - Total: 366 - 179 16 Table 6: Thailand’s breakdown of ATIGA Form Ds sent and corresponding responses received AMS ATIGA Form D Sent Technical Acknowledgements Customs Responses Received AS-1 Received AS-2 Received AS-3 Received Indonesia 453 - - - - Malaysia 484 - - - 13 Singapore 240 - - - 89 Thailand - - - - - Viet Nam 455 - - - 99 Total: 1,632 - - - 201 Thailand noted that they did not collect all of the statistics during the end-to-end testing, as they were focusing on enhancing the system and fixing issues. Table 7: Viet Nam’s breakdown of ATIGA Form Ds sent and corresponding responses received AMS ATIGA Form D Sent Technical Acknowledgements Customs Responses Received AS-1 Received AS-2 Received AS-3 Received Indonesia 202 202 16 53 0 Malaysia 49 49 11 11 337 Singapore 22 22 0 0 0 Thailand 77 77 45 60 0 Viet Nam - - - - - ASW Technical and Financial Evaluation – Final Report Page 44 of 80 Total: 350 350 72 124 337 Q3. How many ATIGA Form D’s did you receive? Indonesia, Malaysia, Singapore, Thailand and Viet Nam all responded to this question. Tables 8-12 provide a breakdown of the number of ATIGA Form Ds they received and the number of corresponding responses they sent to the sending AMS. Table 8: Indonesia’s breakdown of ATIGA Form Ds received and corresponding responses sent AMS ATIGA Form D Received Customs Responses Sent Indonesia - - Malaysia 58 58 Singapore 10 10 Thailand 245 245 Viet Nam 831 831 Total: 1,144 1,114 Table 9: Malaysia’s breakdown of ATIGA Form Ds received and corresponding responses sent AMS ATIGA Form D Received Customs Responses Sent Indonesia 3,203 3,203 Malaysia - - Singapore 21 21 Thailand 612 612 Viet Nam 344 344 Total: 4,182 4,182 Table 10: Singapore’s breakdown of ATIGA Form Ds received and corresponding responses sent AMS ATIGA Form D Received Customs Responses Sent Indonesia 2,100 (received at Gateway but not to NSW) - Malaysia - - Singapore - - Thailand 84 56 Viet Nam 83 86 Total: 2,267 142 Table 11: Thailand’s breakdown of ATIGA Form Ds received and corresponding responses sent AMS ATIGA Form D Received Customs Responses Sent Indonesia 1,256 22 Malaysia 20 17 Singapore 24 11 ASW Technical and Financial Evaluation – Final Report Page 45 of 80 Thailand - - Viet Nam 167 94 Total: 1,467 144 Table 12: Viet Nam’s breakdown of ATIGA Form Ds received and corresponding responses sent AMS ATIGA Form D Received Customs Responses Sent Indonesia 4,232 4,232 Malaysia 876 876 Singapore 32 32 Thailand 518 518 Viet Nam - - Total: 5,658 5,658 Q4. Which categories of ATIGA Form D did you successfully test? This question refers to item 1, bullet point (i) of the acceptance criteria in the “End-to-End Test Plan”. Indonesia, Malaysia, Singapore, Thailand and Viet Nam all responded to this question. Table 13 provides a breakdown of the categories of ATIGA Form Ds that were successfully tested by the AMS. Table 13: Categories of ATIGA Form D successfully tested by the AMS Category Indonesia Malaysia Singapore Thailand Viet Nam Third-Country Invoicing (TCI) √ √ √ √ √ Accumulation (ACL) √ - √ - √ Back-to-Back CO (BCO) √ √ - - √ Partial Cumulation (PCL) √ - √ - √ Exhibition (EXH) √ - - - √ De Minimis (DMS) √ - √ - √ Issued Retroactively (IRA) √ √ √ √ √ Malaysia commented that, although they had successfully sent and received ATIGA Form Ds to all of the other participating AMS, they not able to determine whether the receiving AMS had identified the “right application case” had been ticked correctly in Box 13. Q5. Which Customs response statuses did you successfully test? This question refers to item 1, bullet point (ii) of the acceptance criteria in the “End-to-End Test Plan”. Indonesia, Malaysia, Singapore, Thailand and Viet Nam all responded to this question. Table 14 provides a breakdown of the Customs response statuses that were successfully tested by the AMS. Table 14: Customs response statuses successfully tested by the AMS Customs Response Indonesia Malaysia Singapore Thailand Viet Nam Received √ √ √ √ √ Not Processed √ - √ √ √ ASW Technical and Financial Evaluation – Final Report Page 46 of 80 Customs Response Indonesia Malaysia Singapore Thailand Viet Nam Preferential Treatment Given √ - √ - √ Preferential Treatment Not Given √ - √ - √ Malaysia commented that they only received Customs responses from Thailand and Viet Nam. Q6. Are you satisfied that the ATIGA Form D and Customs responses are harmonized? This question refers to item 2 of the acceptance criteria in the “End-to-End Test Plan”. Malaysia, Singapore and Thailand all answered “No” to this question, while Indonesia and Viet Nam both answered “Yes”. Malaysia commented that Customs’ responses were not harmonized according to what had been agreed between the AMS, although they noted that Customs responses were not mandatory for the end-to-end testing. Singapore commented that the Customs’ responses were not sent in some instance and the preferential and non-preferential statuses had not been tested yet. Thailand commented that some of the Customs’ responses they received did not enable them to identify which ATIGA Form D they related to due to “unspecified reference numbers”. Q7. Did you conduct your own stress testing? This question refers to item 1, bullet point (iv) of the acceptance criteria in the “End-to-End Test Plan”. Indonesia, Malaysia, Singapore, Thailand and Viet Nam all answered “No” to this question. Thailand commented that, as they are not using their own gateway to connect to the ASW, they did not carry out stress testing because their gateway has already gone through its own testing process. Q8. Are you satisfied that the reporting from the ASW Regional Services is correct? This question refers to item 6 of the acceptance criteria in the “End-to-End Test Plan”. Indonesia and Viet Nam both answered “Yes” to this question, while Malaysia and Singapore both answered “No”. Singapore provided an example that there was no data shown for their exchange with Thailand. Thailand explained that they were not using the not using the standard ASW Gateway and that they were developing their own reporting module, which is expected to be tested and deployed before they cut-over to the live operation. Q9. Are you satisfied with the support procedures? This question refers to item 5 of the acceptance criteria in the “End-to-End Test Plan”. Indonesia, Thailand and Viet Nam all answered “Yes” to this question, while Malaysia and Singapore both answered “No”. Malaysia commented that support procedures were not made available to them during the end-to-end testing. ASW Technical and Financial Evaluation – Final Report Page 47 of 80 The Contractor has advised that the support procedures have been documented in the “Operations Handbook” (section 3.4). Q10. Do you agree that the end-to-end tests have been successful? Indonesia and Viet Nam both answered “Yes” to this question with no further comments. Singapore answered “No” to this question. They do not agree that the end-to-end tests were successful, mainly because the “End-to-Test Report” (see section 5.3.1.1.4) identified what they considered to be major / critical issues for Singapore and Malaysia. They also felt that the Contractor should have provided a more detailed step-by-step guide on the subsequent processes to be implemented. They felt that the acceptance criteria lacked clarity in some areas and was left to the AMS to interpret for themselves, in order to assess whether the tests were considered to be successful or not. They suggested that it would have been helpful if the Contractor had provided a standardized report for all of the AMS to complete, rather than the AMS having to fill in the template themselves. Malaysia answered “Yes” to this question. However, from their perspective, they felt the tests could have been performed better. For example, they are not satisfied that the Customs responses were harmonized, users lacked information on how to utilize the services provided by the ASW Regional Services, there was no support procedures available during the testing, there was no information on the technical issues faced by the other AMS’ and there was no stress testing carried out. Thailand felt that the receipt of ATIGA Form Ds into their system was more successful than the sending of ATIGA Form Ds from their system, because Customs focused on the success of generating and sending the acknowledgement messages (i.e. AS-2, AS-3 and AS-4). 5.3.1.1.4 End-to-End Test Report The Contractor’s “End-to-End Test Report” provides a summary of the results, from their perspective, of the end-to-end testing carried out by the participating AMS. These results can be compared with the feedback received from the participating AMS through the AMS Questionnaire (Part 1). For ease of reference, the Contractor’s results have been summarized further into Table 15 below. Table 15: Summary of the end-to-end test results provided by the Contractor No. Test Indonesia Malaysia Singapore Thailand Viet Nam 1 Connectivity OK OK OK OK OK 2 e-ATIGA Form D Sent and Received OK OK Issue #1 OK OK 3 Customs Response Sent and Received Not Started Issue #2 OK OK OK 4 Technical Acknowledgement (AS-1) generated and received internally OK OK OK Not Required OK 5 Technical Acknowledgement (AS-2) Sent and Received OK Issue #3 OK OK OK 6 Technical Acknowledgement (AS-3) Sent and Received OK Issue #2 OK OK OK ASW Technical and Financial Evaluation – Final Report Page 48 of 80 No. Test Indonesia Malaysia Singapore Thailand Viet Nam 7 Regional Services MIS Reporting OK OK Manual In Progress Manual 8 Code Set and Economic Operator Id Administration Not Tested Not Tested Not Tested Not Tested Not Tested As shown in the table above, the “End-to-End Test Report” highlights three issues:  Issue #1 – There is an issue under investigation where not all of the e-ATIGA Form D’s generated by Singapore’s issuing agency application are sent out by their ASW Gateway.  Issue #2 - Malaysia is generating the Customs Response and the AS-3, but there is an issue with dispatch which is under investigation  Issue #3 – For Malaysia, the AS-2’s received from Thailand are not sent to Dagangnet. The report also points out that Indonesia were not planning to implement the Customs response until the 31st October 2015, which the Contractor indicates was optional for the end-to-testing. 5.3.1.1.5 Report of the 33rd TWG Meeting It was reported during the 33rd TWG Meeting that the end-to-end testing was carried out from the 24 th August 2015 to the 16th October 2015. The “End-to-End Test Report”, provided by the Contractor, was discussed and accepted during the meeting, which will include the final comments from the AMS. The Contractor will continue to resolve the outstanding technical issues identified during the end-to￾end testing, which will be carried over and tracked as part of the parallel testing. During the end-to-end testing, the “e-ATIGA Form D Process Specification and Message Implementation Guideline” and the corresponding XML Schema Definition (XSD) were amended to address issues raised by the participating AMS during the tests. Examples of these issues included “syntax errors for data elements due to different interpretation of the cardinality of such elements; correct inputs for data elements in RES message types; need to expand/improve the explanations of the tag name (i.e. “guideline” column) for clarity; clarification for “new” Cos; clarification on cancellation request and query request/response; declaration of invoice number in the case of multiple invoices; use of unique identification for each message”. 5.3.1.2 Parallel Testing 5.3.1.2.1 Request for Proposal The following are extracts from the RFP relating to this section. “Assumption 10: AMS will need to setup a dedicated team to manage both the electronic and paper￾based environments during the parallel testing phase to ensure a smooth testing phase prior to cutover to live implementation.” 5.3.1.2.2 Parallel Test Plan According to the “Parallel Test Plan”, the objectives of the parallel testing was: a) “To conduct parallel testing for the exchange of the e-ATIGA Form D and the related Customs Responses, together with a sampling of corresponding paper versions, from the source ASW Technical and Financial Evaluation – Final Report Page 49 of 80 application of the Certificate Issuing Agency (CIA), such as that used by the Agency for the approval/certification of the ATIGA Form D, to the destination application such as that used by the import customs to process imports. Optionally, if the exporting AMS is able to also send the corresponding paper version, this will provide an added assurance on cross checking between what is presented by the trader and that sent electronically and on paper from the exporting AMS. b) To review, and if necessary revise the agreed ASEAN data structure for the ATIGA Form D and e-ATIGA Form D Related Documents to ensure the data structures are compatible for all the AMS participating in the parallel testing c) To identify any data discrepancies in the area of the Sending AMS - for example, in the generation / processing of the e-ATIGA Form D and Related Documents by the applications or NSW of the Sending AMS d) To identify any data discrepancies or validation discrepancies in the area of the Receiving AMS – for example, in the processing / receiving of the e-ATIGA Form D and Related Documents by the applications or NSW of the receiving AMS e) To test the modified Acknowledgement (AS-3), MIS Reporting and Alerts f) To put in place action plans to address the data discrepancies identified – these action plans may cover the applications and NSW of each AMS participating in the tests, the ATIGA Form D data structure and the ASW” The “Parallel Test Plan” also lists the acceptance criteria as follows: 1. “Successful parallel flows – Exporting Government to Importing Government systems (ATIGA Form D) between all participating AMS for scenarios involving all involved systems NOTE: Each sending party is responsible to keep the transaction log of the message sent for tracking and post audit purpose. i. Original ATIGA Form D – all categories ii. Customs response – for all status (Received; Not Processed; Preferential Treatment Given; Preferential Treatment Not Given) iii. Technical acknowledgements: AS-1, AS-2, AS-3 iv. For up to 10% of the number of transactions executed in one month, subject to practical considerations. It is assumed there will be no parallel STRESS testing. AMS may conduct their own stress testing v. All critical and major issues identified during the test are resolved 2. Proven Harmonization of ATIGA Form D and Response message structures between all participating AMS 3. There is internal AMS agreement that test is successful and can proceed to the next stage. This may include internal assessment 4. The Disaster Recovery for the ASW Gateway is tested if a DR environment is available for the AMS (Internal to AMS including ASW Gateway) 5. Support Procedures Tested (Internal and Cross Border) 6. The daily reporting from the ASW Gateway, and the Regional Services MIS Reporting is correct 7. Mitigation measures when AMS are unable to receive/send the e-ATIGA Form D are carried out efficiently and effectively. 8. Note: It is agreed that there will be no consideration of potential business improvements, in the acceptance criteria for the Start of the Parallel Test or the start of Live Operation” ASW Technical and Financial Evaluation – Final Report Page 50 of 80 5.3.1.2.3 Feedback from AMS Questionnaire (Part 2) The feedback from AMS Questionnaire (Part 2) relating to this section is provided below. Q1. Have you been able to extract the data from production and create ATIGA Form D messages? Indonesia, Malaysia, Thailand and Viet Nam all answered “Yes” to this question, while Singapore answered “No”. Q2. How many ATIGA Form D’s have you sent so far and how many responses have you received? This question refers to item 1, bullet point (iii) of the acceptance criteria in the “Parallel Test Plan”. Indonesia, Malaysia, Singapore, Thailand and Viet Nam all responded to this question. Tables 16-20 provide a breakdown of the number of ATIGA Form Ds they sent and the number of corresponding responses they received from the receiving AMS. Table 16: Indonesia’s breakdown of ATIGA Form Ds sent and corresponding responses received AMS ATIGA Form D Sent Technical Acknowledgements Customs Responses Received AS-1 Received AS-2 Received AS-3 Received Indonesia - - - - - Malaysia 1,497 1,497 1,497 1,497 0 Singapore 363 363 363 363 0 Thailand 1,408 1,408 1,408 1,408 0 Viet Nam 1,068 1,068 1,068 1,068 1,068 Total: 4,336 4,336 4,336 4,336 1,068 Table 17: Malaysia’s breakdown of ATIGA Form Ds sent and corresponding responses received AMS ATIGA Form D Sent Technical Acknowledgements Customs Responses Received AS-1 Received AS-2 Received AS-3 Received Indonesia 304 304 304 379 21 Malaysia - - - - - Singapore 0 0 0 0 0 Thailand 134 134 0 0 0 Viet Nam 876 873 873 873 873 Total: 1,314 1,311 1,177 1,252 894 Malaysia commented that their statistics show that they did receive any responses from Thailand. Table 18: Singapore’s breakdown of ATIGA Form Ds sent and corresponding responses received AMS ATIGA Form D Sent Technical Acknowledgements Customs Responses Received AS-1 Received AS-2 Received AS-3 Received Indonesia 36 36 3 1 2 Malaysia 26 26 23 23 15 Singapore - - - - - Thailand 32 32 26 28 29 ASW Technical and Financial Evaluation – Final Report Page 51 of 80 Viet Nam 32 32 30 30 32 Total: 126 126 82 82 78 Table 19: Thailand’s breakdown of ATIGA Form Ds sent and corresponding responses received AMS ATIGA Form D Sent Technical Acknowledgements Customs Responses Received AS-1 Received AS-2 Received AS-3 Received Indonesia 1,111 N/A 422 4,886 415 Malaysia 1,358 N/A 960 960 667 Singapore 88 N/A 1 0 0 Thailand - - - - - Viet Nam 600 N/A 380 380 359 Total: 3,157 N/A 1,763 6,226 1,441 Thailand noted that the above table shows their statistics up to the 9th December 2015. Table 20: Viet Nam’s breakdown of ATIGA Form Ds sent and corresponding responses received AMS ATIGA Form D Sent Technical Acknowledgements Customs Responses Received AS-1 Received AS-2 Received AS-3 Received Indonesia 255 255 254 5,223 0 Malaysia 237 237 227 17 0 Singapore 225 225 202 96 140 Thailand 295 295 0 126 0 Viet Nam - - - - - Total: 1,012 1,012 683 5,462 140 Q3. How many ATIGA Form D’s have you received? Indonesia, Malaysia, Singapore, Thailand and Viet Nam all responded to this question. Tables 21-25 provide a breakdown of the number of ATIGA Form Ds they received and the number of corresponding responses they sent to the sending AMS. Table 21: Indonesia’s breakdown of ATIGA Form Ds received and corresponding responses sent AMS ATIGA Form D Received Customs Responses Sent Indonesia - - Malaysia 58 58 Singapore 10 10 Thailand 245 245 Viet Nam 831 831 Total: 1,144 1,144 Table 22: Malaysia’s breakdown of ATIGA Form Ds received and corresponding responses sent AMS ATIGA Form D Received Customs Responses Sent ASW Technical and Financial Evaluation – Final Report Page 52 of 80 Indonesia 5,059 2,772 Malaysia - - Singapore 35 22 Thailand 1,436 814 Viet Nam 43 22 Total: 6,573 3,630 Malaysia commented that their statistics for Singapore and Viet Nam are not accurate. They also commented that they are able to view their statistics through the ASW Gateway, but ASW Regional Services does not show any statistics for Thailand. Table 23: Singapore’s breakdown of ATIGA Form Ds received and corresponding responses sent AMS ATIGA Form D Received Customs Responses Sent Indonesia 601 224 Malaysia 0 0 Singapore - - Thailand 118 5 Viet Nam 29 192 Total: 748 421 Table 24: Thailand’s breakdown of ATIGA Form Ds received and corresponding responses sent AMS ATIGA Form D Received Customs Responses Sent Indonesia 5,054 3,405 Malaysia 129 116 Singapore 30 26 Thailand - - Viet Nam 33 57 Total: 5,246 3,604 Thailand noted that the above table shows their statistics up to the 9th December 2015. Table 25: Viet Nam’s breakdown of ATIGA Form Ds received and corresponding responses sent AMS ATIGA Form D Received Customs Responses Sent Indonesia 3,037 3,037 Malaysia 79 79 Singapore 66 66 Thailand 561 561 Viet Nam - - Total: 3,743 3,743 Q4. Which categories of ATIGA Form D have you successfully tested? This question refers to item 1, bullet point (i) of the acceptance criteria in the “Parallel Test Plan”. ASW Technical and Financial Evaluation – Final Report Page 53 of 80 Indonesia, Malaysia, Singapore, Thailand and Viet Nam all responded to this question. Table 26 provides a breakdown of the categories of ATIGA Form Ds that were successfully tested by the AMS. Table 26: Categories of ATIGA Form D successfully tested by the AMS Category Indonesia Malaysia Singapore Thailand Viet Nam Third-Country Invoicing (TCI) √ √ - √ √ Accumulation (ACL) √ - - √ √ Back-to-Back CO (BCO) √ - √ √ √ Partial Cumulation (PCL) √ - - √ √ Exhibition (EXH) √ - - √ √ De Minimis (DMS) √ - - √ √ Issued Retroactively (IRA) √ - - √ √ Thailand noted that some of their categories were generated using dummy data. Q5. Which Customs response statuses have you successfully tested? This question refers to item 1, bullet point (ii) of the acceptance criteria in the “Parallel Test Plan”. Indonesia, Malaysia, Singapore, Thailand and Viet Nam all responded to this question. Table 27 provides a breakdown of the Customs response statuses that were successfully tested by the AMS. Table 27: Customs response statuses successfully tested by the AMS Customs Response Indonesia Malaysia Singapore Thailand Viet Nam Received √ √ √ √ √ Not Processed √ - √ √ √ Preferential Treatment Given √ - - - √ Preferential Treatment Not Given √ - - - √ Q6. Are you satisfied that the ATIGA Form D and Customs responses are harmonized? This question refers to item 2 of the acceptance criteria in the “Parallel Test Plan”. Indonesia and Viet Nam both answered “Yes” to this question, while Malaysia and Thailand both answered “No”. Singapore sought clarification from the Consultant on what this question entails. As this question relates to the acceptance criteria of the “Parallel Test Plan”, it is understood that all participating AMS must be satisfied with both the ATIGA Form D and Customs response message structures. Thailand commented that they still have an issue with the matching of ATIGA Form Ds and Customs responses. In particular, when they send ATIGA Form Ds, they have not received the corresponding Customs response message from the recipient AMS. Q7. Did you conduct your own stress testing? This question refers to item 1, bullet point (iv) of the acceptance criteria in the “Parallel Test Plan”. Indonesia, Malaysia, Singapore, Thailand and Viet Nam all answered “No” to this question. ASW Technical and Financial Evaluation – Final Report Page 54 of 80 Thailand commented that, as they are not using their own gateway to connect to the ASW, they did not carry out stress testing because their gateway has already gone through its own testing process. Q8. Are you satisfied that the reporting from the ASW Regional Services is correct? This question refers to item 6 of the acceptance criteria in the “Parallel Test Plan”. Indonesia and Viet Nam both answered “Yes” to this question, while Malaysia and Singapore both answered “No”. Malaysia commented that the statistics shown in ASW Regional Services are not accurate. Thailand explained that they were not using the standard ASW Gateway and that they were developing their own reporting module, which is expected to be tested and deployed before they cut-over to the live operation. Q9. Are you satisfied with the support procedures? This question refers to item 5 of the acceptance criteria in the “Parallel Test Plan”. Indonesia, Malaysia, Thailand and Viet Nam all answered “Yes” to this question, while Singapore answered “No”. Q10. Do you agree that the parallel tests have been successful? Indonesia and Viet Nam both answered “Yes” to this question, while Malaysia answered “No” for the following reasons:  The ASW Gateway allows duplicate reference numbers  There are discrepancies in the statistics between the ASW Gateway and the transaction reports on the ASW Regional Services  ASW Regional Services records both successful and unsuccessful transactions  There are no responses to queries  Stress testing has not yet been carried out  The guidelines for ASW Regional Services were on received on the 23rd October 2015 Singapore commented that their ASW gateway was unable to transmit test cases without any reference documents after the B2Bi 2.2 software upgrade in late Nov 2015. As such, they have not been able to continue with the parallel testing as planned. Thailand commented that the acceptance criteria for the parallel testing has not yet been met, because there are still issues with determining the utilization status of ATIGA Form Ds. 5.3.1.2.4 Parallel Test Report The Contractor’s “Parallel Test Report” (Release 2) provides a summary of the results of the parallel testing carried out by Indonesia, Malaysia, Thailand and Viet Nam. The results of the parallel testing for Singapore, and further testing with Thailand, is expected to be covered in Release 3 of the report. The report covers the results of the parallel tests that were carried out between the 26th October 2015 and the 18th December 2015. It focuses on how the acceptance criteria of the “Parallel Test Plan” has been met, in order to determine whether the parallel testing has been successful. ASW Technical and Financial Evaluation – Final Report Page 55 of 80 The first two acceptance criteria cover the various categories of ATIGA Form D sent and received by the participating AMS. In order to meet the acceptance criteria, each participating AMS must send and receive a minimum number of ATIGA Form Ds for each category, as shown in Tables 28 and 29. Table 28 provides a summary of the results of the testing of each category of ATIGA Form D being sent by each participating AMS. Where live data was not available to carry out the tests, test data was used to simulate the live tests. Table 28: Send Original ATIGA Form D – categories as in the Live Data as below Category Acceptance Criteria Indonesia Malaysia Thailand Viet Nam Normal (NOR) 100 2459 > 100 > 100 104 Third-Country Invoicing (TCI) 10 1193 > 10 > 10 > 10 Accumulation (ACL) 5 6 5 (test) 5 (test) 5 (test) Back-to-Back CO (BCO) 10 1 > 10 > 10 5 (test) Partial Cumulation (PCL) 1 3 1 (test) 1 (test) 5 (test) Exhibition (EXH) 1 1 1 (test) 1 (test) 5 (test) De Minimis (DMS) 1 1 1 (test) 1 (test) 5 (test) Issued Retroactively (IRA) 5 2124 > 5 > 5 > 5 Drawback (DRW) 5 > 1 > 5 > 5 5 (test) Contractor’s Assessment OK To Confirm OK OK Table 29 provides a summary of the results of the testing of each category of ATIGA Form D being received by each participating AMS. Where live data was not available to carry out the tests, test data was used to simulate the live tests. Table 29: Receive and process by import Customs - Original ATIGA Form D Category Acceptance Criteria Indonesia Malaysia Thailand Viet Nam Normal (NOR) 100 > 100 > 100 > 100 > 100 Third-Country Invoicing (TCI) 10 > 10 > 10 > 10 > 10 Accumulation (ACL) 5 5 (test) 5 (test) 5 (test) 5 (test) Back-to-Back CO (BCO) 10 > 10 > 10 > 10 > 10 Partial Cumulation (PCL) 1 1 (test) 1 (test) 1 (test) 1 (test) Exhibition (EXH) 1 1 (test) 1 (test) 1 (test) 1 (test) De Minimis (DMS) 1 1 (test) 1 (test) 1 (test) 1 (test) Issued Retroactively (IRA) 5 > 5 > 5 > 5 > 5 Drawback (DRW) 5 > 5 > 5 > 5 > 5 Contractor’s Assessment OK Since ID sent all categories assumed OK for MY OK OK The Contractor’s assessment of the remaining acceptance criteria is shown in Table 30. Successful tests are indicated as “OK and unsuccessful, or incomplete tests, are indicated as “No”. ASW Technical and Financial Evaluation – Final Report Page 56 of 80 Table 30: Contractor’s assessment of the other acceptance criteria Category Indonesia Malaysia Thailand Viet Nam 1.3 For all status (Received; Not Processed; Preferential Treatment Given; Preferential Treatment Not Given, or Mixed). If an AMS has decided to implement the Customs Response messages, then the Customs Response is generated as indicated below for ATIGA Form D in a 2 week period. a) Customs Response is generated for each e￾ATIGA Form D received by importing Customs to indicate “Received” or “Not Processed” by importing Customs. OK OK No OK b) Customs Response may be generated for each e-ATIGA Form D that is received OK for processing by importing Customs to indicate Preferential Treatment Given; Preferential Treatment Not Given; or Mixed No No No No 1.4 Technical acknowledgements: AS-2, AS-3 are generated for each e-ATIGA Form D received by the importing ASW Gateway in a 2 week period and sent to the exporting ASW Gateway. OK OK OK OK 1.5 All critical and major issues identified during the test are resolved, as indicated in the AMS Weekly Log and confirmed to be resolved by the AMS Project Manager. OK OK OK OK 2 Proven Harmonization of ATIGA Form D and Response message structures between all participating AMS. This will be assessed by the e-ATIGA Form D being able to be received and processed by the import Customs – accordingly if items 1.1.and 1.2 are shown to be achieved then item 2 will also be achieved. OK OK OK OK 3 The Disaster Recovery for the ASW Gateway is tested if a DR environment is available for the AMS (Internal to AMS including ASW Gateway) Not Required Not Required Not Required To Be Done 4 Support Procedures Tested (Internal and Cross Border). This will be assessed based on the feedback from the designated Project Manager for each AMS. OK OK OK OK 5 The daily reporting from the ASW Gateway, and the Regional Services MIS Reporting is correct. This will be assessed based on a cross check for 3 days in a 2 week period, as well as the total for the 2 week period. This will be assessed by the Contractor where they have remote access OK OK OK OK ASW Technical and Financial Evaluation – Final Report Page 57 of 80 Category Indonesia Malaysia Thailand Viet Nam to the ASW Gateway, otherwise by the AMS themselves In addition to comparing the test results against the acceptance criteria, the Contractor’s report also provides detailed statistics of the various types of messages exchanged between Indonesia, Malaysia, Thailand and Viet Nam. It is understood that these statistics have been taken from the Regional Services reporting facility. Table 31 shows the detailed statistics of the messages exchanged between Indonesia and Viet Nam from the 28th November 2015 to the 10th December 2015. The statistics show that, from the 28th November 2015, all outstanding critical and major issues were resolved and that Indonesia and Viet Nam are ready to proceed to live operation. Table 31: Exchanges between Indonesia and Viet Nam Document Type Indonesia Viet Nam Viet Nam Indonesia Difference (Indonesia Sends, Viet Nam receives) Difference (Viet Nam Sends, Indonesia receives) Sent Received Sent Received 864 1,068 1,068 29 29 0 0 AS1 1,068 0 29 0 0 0 AS2 29 29 1,068 1,068 0 0 AS3 29 29 1,068 1,068 0 0 RES 18 18 1,068 1,068 0 0 Table 32 shows the detailed statistics of the messages exchanged between Indonesia and Malaysia from the 10th November 2015 to the 30th November 2015. The statistics show a difference between the number of AS3 messages sent by Indonesia and those received by Malaysia. This was reported as being due to Indonesia’s testing of their AS3 generation script, which were not recorded by Malaysia. Table 32: Exchanges between Indonesia and Malaysia Document Type Indonesia Malaysia Malaysia Indonesia Difference (Indonesia Sends, Malaysia receives) Difference (Malaysia Sends, Indonesia receives) Sent Received Sent Received 864 2,185 2,185 117 117 0 0 AS1 2,185 0 117 0 0 0 AS2 117 117 2,185 2,185 0 0 AS3 192 143 2,185 2,185 -49 0 RES 0 0 2,185 2,185 0 0 Table 33 shows the detailed statistics of the messages exchanged between Malaysia and Viet Nam from the 3rd December 2015 to the 17th December 2015. The statistics show that, from the 3rd ASW Technical and Financial Evaluation – Final Report Page 58 of 80 December 2015, all outstanding critical and major issues were resolved and that Malaysia and Viet Nam are ready to proceed to live operation. Table 33: Exchanges between Malaysia and Viet Nam Document Type Malaysia Viet Nam Viet Nam Indonesia Difference (Malaysia Sends, Viet Nam receives) Difference (Viet Nam Sends, Malaysia receives) Sent Received Sent Received 864 237 237 18 18 0 0 AS1 237 0 18 0 0 0 AS2 18 18 237 237 0 0 AS3 18 18 237 237 0 0 RES 9 9 237 237 0 0 Table 34 shows the detailed statistics of the messages exchanged between Indonesia and Thailand from the 26th October 2015 to the 11th December 2015. The statistics highlight that issues still need to be addressed by Thailand, especially the uploading of their statistics to Regional Services. Table 34: Exchanges between Indonesia and Thailand Document Type Indonesia Thailand Thailand Indonesia Difference (Indonesia Sends, Thailand receives) Difference (Thailand Sends, Indonesia receives) Sent Received Sent Received 864 5,326 0 0 472 -5,326 472 AS1 5,312 0 0 0 0 0 AS2 472 0 0 5,210 -472 5,210 AS3 5,437 0 0 4,607 -5,437 4,607 RES 580 0 0 0 -580 0 Table 35 shows the detailed statistics of the messages exchanged between Thailand and Malaysia from the 26th October 2015 to the 11th December 2015. The statistics highlight that issues still need to be addressed by Thailand, especially the uploading of their statistics to Regional Services. Table 35: Exchanges between Thailand and Malaysia Document Type Thailand Malaysia Malaysia Thailand Difference (Thailand Sends, Malaysia receives) Difference (Malaysia Sends, Thailand receives) Sent Received Sent Received 864 0 1,436 134 0 1,436 -134 AS1 0 0 134 0 0 0 AS2 0 134 1,038 0 134 -1,038 AS3 0 129 1,038 0 129 -1,038 RES 0 117 814 0 117 -814 ASW Technical and Financial Evaluation – Final Report Page 59 of 80 Table 36 shows the detailed statistics of the messages exchanged between Thailand and Viet Nam from the 26th October 2015 to the 11th December 2015. The statistics highlight that issues still need to be addressed by Thailand, especially the uploading of their statistics to Regional Services. Table 36: Exchanges between Thailand and Viet Nam Document Type Thailand Viet Nam Viet Nam Thailand Difference (Thailand Sends, Viet Nam receives) Difference (Viet Nam Sends, Thailand receives) Sent Received Sent Received 864 0 547 74 0 547 -74 AS1 0 0 70 0 0 0 AS2 0 74 414 0 74 -414 AS3 0 73 414 0 73 -414 RES 0 3 442 0 3 -442 5.3.1.2.5 Report of the 33rd TWG Meeting It was reported during the 33rd TWG Meeting that the parallel testing would be carried out from the 26th October 2015 to the 11th December 2015. The AMS agreed that the live operation would only proceed if two AMS successfully concluded the parallel testing. The “Parallel Test Plan" was also discussed and accepted during the meeting, which will include the final comments from the AMS. The Contractor updated the AMS on the results of the parallel testing as of the 16th November 2015, which had identified some further technical issues in addition to those identified during the end-to￾end testing. 5.3.1.2.6 Current Status According to the Contractor’s “Parallel Test Report” (Release 2), Indonesia and Viet Nam are ready to go live, subject to confirmation by Viet Nam. Malaysia will also be ready to go live, following some enhancements to their system to address multiple invoices per consignment item. Although Thailand still have some outstanding issues to address, they are expected to be the 4th AMS to be ready to go live. However, the report also highlights 4 open issues:  Viet Nam - AS2, AS3 messages are not sent to their back-end system  Malaysia - Multiple invoice references and multiple packaging details assigned to one consignment. Malaysia has advised that in practice this situation applies to invoice references only and for Third Country Invoicing  Thailand - not generating RES for all 864's from Indonesia, Malaysia and Viet Nam  Regional Services - In some cases, where there are duplicate 864 COO# it is not always clear if the reporting is correct or if there was a manual re-send It is assumed that all of the above issues will be addressed before the affected AMS will be able to proceed to live operation. ASW Technical and Financial Evaluation – Final Report Page 60 of 80 5.3.2 Evaluation 5.3.2.1 End-to-End Testing Based on the information available, specifically the “End-to-End Test Report”, end-to-end testing was not completed successfully. There are still some issues that are under investigation by the Contractor and the AMS. Firstly, the Consultant was informed that the Customs response was optional, according to the “ATIGA Form D Process Specification and Message Implementation Guideline”. However, the “End￾to-End Test Plan” clearly states that one of the objectives is “To conduct end-to-end testing for the exchange of the electronic ATIGA Form D and the related Customs Responses”. It is suggested that without the Customs response, there will be no audit trail if an ATIGA Form D is sent by an issuing authority and is not received by Customs. This would create problems for the importer, because they would not know who to contact i.e. the exporter or their NSW operator. Recommendation No. 4 To make the Customs response mandatory and to ensure that it is fully tested during the parallel testing prior to the cutover to live operation. The Consultant has also found it difficult to obtain statistics from the end-to-end testing, especially the number of ATIGA Form Ds sent and number of responses received. Singapore, for example, had reported that they were unable to get statistics from Thailand. However, on further investigation, the reason for this appears to be that Thailand had not been uploading their statistics to the ASW Regional Services Portal. After receiving feedback from Thailand, the Contractor confirmed that Thailand were not using the standard ASW Gateway, as they were using their own NSW gateway. The Contractor also advised that Singapore were not using the standard automated ASW Gateway mechanism to upload their message exchange information to the ASW Regional Services. Instead, the Contractor agreed to develop an alternative mechanism for Singapore, whereby the information was picked up automatically from the ASW Gateway. It is suggested that, until the alternative reporting mechanisms have been deployed and tested by Singapore and Thailand, during parallel testing, item 6 of the acceptance criteria cannot be met. It states; “The daily reporting from the ASW Gateway, and the Regional Services MIS Reporting is correct”. Recommendation No. 5 To ensure that the alternative reporting mechanisms of Singapore and Thailand have been fully tested, so that the number of ATIGA Form Ds sent by the sending AMS match the number received by the receiving AMS. Similarly, the number of responses sent by the sending AMS should match the number received by the receiving AMS. The fact that Thailand are not using the ASW Gateway also raises the question as to whether it is actually needed. While it is understood that Thailand are using an older version of the Contractor’s software, their experience proves that there is no need for a standard ASW Gateway. In fact, as long as each AMS follow the same standards (i.e. ebMS, ATIGA Form D and future ASW messages), each AMS should have the option to develop their own as part to their NSW system. ASW Technical and Financial Evaluation – Final Report Page 61 of 80 Recommendation No. 6 To ensure that all of the ASW technical specifications are platform neutral and can be easily implemented in any NSW environment without being dependent on any special software. This should include details of how the standard communications protocol (i.e. ebMS) is being used and all interfaces between the NSW and the ASW Regional Services Portal. 5.3.2.2 Parallel Testing As the Consultant has not received any feedback from Indonesia or Viet Nam, and the Contractor’s “Parallel Test Report” provides sufficient evidence to show the acceptance criteria has been met, they appear to be the first two AMS to be ready to go live. It is understood that the current target date for Indonesia and Viet Nam is the 8th February 2016. Based on the feedback received from Malaysia and Thailand, there appears to be more issues that need to be resolved than those reported by the Contractor (see section 5.3.1.2.6), as both felt that the parallel testing had not been successful when they submitted their completed questionnaires (see section 5.3.1.2.3). For example, Malaysia were not satisfied that the reporting from Regional Services reporting was correct and Thailand still had some issues in determining the utilization status of ATIGA Form Ds. Therefore, the Contractor’s target date of the 22nd February 2016 for Thailand may not be achievable. It is also clear from the feedback received from Singapore, and from the Contractor’s “Parallel Test Report” (Release 2), that Singapore still have some way to go before they are ready to go live. Recommendation No. 7 To ensure that the acceptance criteria in the Parallel Test Plan is met by resolving the outstanding issues, including; the testing of all categories of ATIGA Form D, the testing of all Customs response statuses, the harmonization of ATIGA Form D and Customs responses, and the accuracy of the ASW Regional Services reporting. 5.4 Network security, scalability and suitability of the Contractor’s software The purpose of this section is to detail the findings of Task 5 of the TOR; “An evaluation of the network security (in terms of security vulnerabilities), scalability and suitability of the Contractor’s software (a) for the exchange of the ATIGA CO Form D; and (b) to accommodate additional cross￾border documents”. 5.4.1 Findings The findings below are based on the Component 1 Deliverables and the “Contractor’s Technical Proposal in response to the RFP”. 5.4.1.1 Network Security 5.4.1.1.1 Component 1 Deliverables According to the Security Architecture Specifications for ASW, a key deliverable of ASW Pilot Project Component 1, the security aspects of the ASW should be segregated into 4 different layers:  Application Layer Security ASW Technical and Financial Evaluation – Final Report Page 62 of 80  Transport Layer Security  Network Layer Security  Physical Security In simple terms, messages should be sent between local installations of the ASW Software using the Simple Object Access Protocol (SOAP) over a secure Virtual Private Network (VPN). The use of Public Key Infrastructure (PKI) is also required at the network level to authenticate the “sender” i.e. in the context of the ASW, this is the ASW Software of the NSW sending the ATIGA Form D, for example. 5.4.1.1.2 Contractor’s Technical Proposal in response to the RFP The following are extracts from the “Contractor’s Technical Proposal in response to the RFP” relating to this section. “The Security framework in the Axway B2B platform mandates that any requesting system has the correct security credentials prior to processing any operational request on the ASW Gateway, e.g. routing, validation. The PKI in the Axway B2B solution can use self-signed certificates generated by the Axway B2B software, or certificates issued by certificate authorities (CAs) to provide authentication, confidentiality, integrity and non-repudiation of data. The Axway B2B engine supports x.509 public key infrastructure (PKI) to securely trade business documents using digital certificates and public key cryptography to secure transactions and communications, and allows certificates to be used for authenticating the identity of trading partners. Digital signatures can be applied to SOAP envelopes during the encryption routine. As utilized in the ASW Scaled-Down Pilot, Secure Sockets Layer (SSL) protocol authentication provides an added layer of security to trading relationships. Client-side certificate authentication can be used, meaning a partner’s certificate is used to verify the partner’s identity when a connection is made.” 5.4.1.2 Scalability 5.4.1.2.1 Contractor’s Technical Proposal in response to the RFP The following is an extract from section A-2.1 of the “Contractor’s Technical Proposal in response to the RFP” relating to this section. “Axway’s B2B software is field proven for mission critical, high performance, cross border B2B services, it is the low risk option to assure high quality, on-time delivery for the Live pilot and for the future expansion to mission critical document exchange with ASEAN – for reliability, performance, security, functionality, and interoperability with heterogeneous platforms. The Performance Benchmark figures in Appendix 2.4 using ebMS indicate performance for a single node over 100 transactions (10KB) per second for 10KB and over 70 transactions (100KB) per seconds. This compares favorably to the projected 3-4 messages per sec for the ASW Pilot (Live).” The Contractor has confirmed that their software can easily meet the projected volumes outlined in ASW Pilot Project Component 1 (Deliverable 05, Section 10.1), which states; “Sizing will be based on 300,000 messages of average size of 16KB per day, with a possible peak load of 100,000 messages per hour”. ASW Technical and Financial Evaluation – Final Report Page 63 of 80 5.4.1.3 Suitability 5.4.1.3.1 Contractor’s Technical Proposal in response to the RFP The following are extracts from the “Contractor’s Technical Proposal” in response to the RFP relating to this section. “The Axway B2B software is field proven for mission critical, high performance, multinational and cross border services such as for the Thai National Single Window, Dagangnet in Malaysia, EDI Indonesia, Shanghai E&P the e-port operator for the world’s largest port, Lenovo in China, and DB Schenker, one of the world’s largest logistics companies.” 5.4.2 Evaluation Based on the information available, the Consultant could find no reason to be concerned about the network security, scalability or suitability of the Contractor’s software. As pointed out by the Contractor, network security is the responsibility of the AMS. The Contractor’s responsibility is to provide a secure ASW network to enable the exchange of electronic documents (“messages”). By providing out-of-the box international security standards (e.g. SSL, PKI etc), they can easily meet the requirements of ASW Pilot Project Component 1. Therefore, if the AMS have any concerns about the network security, it may be necessary to engage an independent security expert to review the current ASW requirements. It is also advisable to test the exchange of digitally signed documents using PKI and digital certificates, as this was a requirement in the RFP. Scalability is more of an unknown factor, because the types and sizes of documents that will be exchanged via the ASW in future has yet to be determined. For example, in terms of size, a Certificate of Origin is one of the simplest documents used within the international supply chain. However, a sea cargo manifest can be very large, bearing in mind that container ships can now carry up to 19,000 containers. Therefore, in theory, if such a Pre-departure Cargo Report (see section 5.6.2.2.4) also included house level information, the maximum size of a message could be several thousand times larger than an ATIGA Form D. The sizing of 300,000 messages per day, as stated in ASW Pilot Project Component 1, is based on the total of the projected number of messages for seven AMS with Malaysia expected to have the largest volume of around 84,000 messages per day. Therefore, based on the methodology used in ASW Pilot Project Component 1 to estimate the peak load of 100,000 messages per hour (i.e. 300,000 divided by 3), the peak load on any specific instance of the ASW Gateway would not be expected to exceed 28,000 messages per hour (i.e. 84,000 divided by 3). This would be equivalent to 7.8 messages per second, compared to the Contractor’s benchmark of 70 transactions per second. Therefore, the Contractor’s software can easily meet the projected volumes stated in ASW Pilot Project Component 1. Should the AMS have any concerns about the scalability of the Contractor’s software, it is advised that they carry out their own stress testing. In fact, this would be strongly recommended. As a COTS solution, the Contractor’s software is also designed to meet the requirements of any B2B gateway, so it is certainly a suitable solution for the ASW. Although the only requirement for the ASW is to support Web Services using SOAP, the Contractor’s software also supports many industry standard transport protocols and standard message formats, including UN/EDIFACT and XML. ASW Technical and Financial Evaluation – Final Report Page 64 of 80 Recommendation No. 8 To plan for and carry out more comprehensive testing of the security features (i.e. use of PKI and digital certificates) and scalability of the Contractor’s software, including stress testing, different document sizes / types (e.g. Pre-departure Cargo Report) and different formats (e.g. UN/EDIFACT). In order to avoid any unexpected costs in future, the AMS may wish to consider defining at least three scenarios of small, medium and large transaction volumes. These scenarios should not be limited to the ATIGA Form D, as they would need also to take into account future documents, such as the Pre-departure Cargo Report. The Contractor should then be able to provide advice on the hardware and software requirements for each scenario, which would give the AMS some confidence that the Contractor’s software is scalable. Recommendation No. 9 To define three scenarios of small, medium and large transaction volumes and to ask the Contractor to provide the appropriate hardware / software requirements, together with the associated costs. 5.5 Draft transition and migration plan for the cutover to live operation The purpose of this section is to detail the findings of Task 6 of the TOR; “An assessment of whether the draft transition and migration (TM) plan for the cutover to live operation can be successfully implemented”. 5.5.1 Findings The findings below are based on the RFP, “Transition and Migration (TM) Plan” and the “Report of the 33rd TWG Meeting”. 5.5.1.1.1 Request for Proposal The following are extracts from the RFP relating to this section. “Assumption 9: The contractor will provide a post live operation efficient and effective transition management (TM) plan that includes the possibility of an eventual upgrade of the current network via https to a more secure virtual private network such as IPVPN if AMS decide to implement such a system. Moreover, this TM plan will include recommendations of public awareness activities, recommendations to be provided by ACTI, to be implemented by AMS to prepare users for the implementation of other cross-border documents, including the export information, agreed by Member States. It recognizes that the implementation of the ASW will have an impact on other stakeholders involved. Paramount in the implementation of the ASW beyond 2015 is its acceptance by all actors involved and the easy transition to new roles and tasks. The contractor is not tasked with implementing the TM plan.” 5.5.1.1.2 Transition and Migration (TM) Plan The Contractor has provided a document entitled “Plan for Transition to Live Operation of the e￾ATIGA Form D” and a more detailed “Schedule for Transition to e-ATIGA Form D Live Operation”. The documents outline a staged approach, starting with Indonesia and Viet Nam, then adding Malaysia, then Thailand and finally Singapore. It is also understood that Brunei Darussalam and the Philippines are planning to start testing in March 2016, when they are expected to join by mid-year, 2016. ASW Technical and Financial Evaluation – Final Report Page 65 of 80 The document also covers the following areas:  AMS Awareness Programs  Technical Deployment and Preparation for Production Operations for AMS  Deployment and Operations of Regional Services  Monitoring and Support Procedures for e-ATIGA Form D Live operation  Contingency Arrangements in case of disruptions to e-ATIGA Form D Service  Schedule for Transition to Live Operation of e-ATIGA Form D  Considerations for Implementing Expanded ASW Services The proposed schedule for the transition to live operation is provided in Table 37 below: Table 37: Schedule for the transition to live operation of e-ATIGA Form D No. Item Start End 1 Production environment deployment and configuration for first two AMS 8 Dec 2015 22 Dec 2015 2 Start Sending e-ATIGA Form D – Live Operation – (for first two AMS) 30 Dec 2015 On-going 3 Rely on e-ATIGA Form D for assignment of preferential duty (for first two AMS) 20 Jan 2015 On-Going 4 Phased in Live Operation for other three AMS 6 Jan 2015-10 Feb 2015 On-Going 5 Stop paper e-ATIGA Form D – all AMS 30 Mar 2015 6 AMS e-ATIGA Form D Awareness Program to Stakeholders Dec 2015 April 2015 5.5.1.1.3 Report of the 33rd TWG Meeting It was reported during the 33rd TWG Meeting that the Contractor had submitted a plan for the transition to live operation of the e-ATIGA Form D, together with the supporting “Schedule for Transition to e-ATIGA Form D Live Operation”. The current transition plan outlines the “approach, key activities, readiness checklists and schedule for the transition to live operation for the e-ATIGA Form D”. However, it was also reported that the transition plan would be subject to the signing of the amended Operational Certification Procedures (OCP), the full ratification of the Protocol on the Legal Framework (PLF) and agreement by two or more AMS to proceed to live operation. The AMS agreed that the transition to live operation would be carried out in stages. However, the participation by Malaysia, Singapore and Thailand will be subject to the consent of the traders. The Contractor suggested a standardized message structure for a cancellation request and query response, in response to a request by Malaysia. These have been included in the ATIGA Form D Message Implementation Guide, which are now available for testing by the participating AMS. The revised transition plan, including comments from the AMS, was accepted in principle. ASW Technical and Financial Evaluation – Final Report Page 66 of 80 5.5.2 Assessment Based on the information available, there are too many unknowns for the Consultant to be able to make a full assessment as to whether the transition and migration plan for the cutover to the live operation can be successfully implemented. One of the main concerns is that there are many activities left to the responsibility of the AMS, which are likely to take time to plan for, including:  Amendments to Operational Certification Procedures (OCPs)  Sign-offs by the relevant ministries  Bi-lateral agreements between the participating AMS to proceed to live operation  Public awareness activities  Installation and configuration of the ASW software in the production environment  Deployment of NSW software changes into production  Training / handover to production IT operations staff  Hosting of Regional Services Given the number of activities still to be carried out by each AMS, and also taking into account the current status of the parallel testing (see section 5.3.1.2.6) and the need for operational training (see section 5.7.2), the proposed schedule seems to be too ambitious. 5.6 Funding requirements needed to support the operation and expansion of the ASW The purpose of this section is to detail the findings of Task 7 of the TOR; “To identify any other funding requirements needed to support the operation and expansion of the ASW (including the electronic exchange of the export data and the four possible additional documents covered in the Information Process Modeling study) not included in the proposed funding requirements agreed by the 15th ASWSC Meeting”. 5.6.1 Findings The findings below are based on the “Design of Project Management and Planning Team (Project Management) Office”, the “ASW Regional IT Operational and Staffing Costs”, “Partnership Matrix for the Exchange of the ASEAN Customs Declaration Document (ACDD)”, the “Information Process Modeling Study” and the AMS Questionnaires (Part 1) submitted by Malaysia, Singapore and Thailand. 5.6.1.1 Operation 5.6.1.1.1 Design of Project Management and Planning Team (Project Management) Office It is understood that the Project Management Office (PMO) will not be in full operation when the ASW is cutover to the live operation for ATIGA Form D on the 30th December 2015 (see section 5.7). The ASEAN Secretariat will be hiring two staff for the PMO during 2016, who will be expected to take over the operation of the ASW Regional Services from January 2017. Initially, the PMO will consist of a Project Officer and a Technical Specialist, as the volume of transactions is expected to be low. Then, by 2019, when additional documents are expected, a Project Manager will be recruited. In the meantime, the VPT-IT will take on the responsibilities of the PMO during 2016. ASW Technical and Financial Evaluation – Final Report Page 67 of 80 5.6.1.1.2 ASW Regional IT Operational and Staffing Costs The estimated operational and staffing costs are shown in Table 38 below. The costs assume that the Project Officer and Technical Specialist will join the PMO in 2017 and the Project Manager in 2019. Table 38: Estimated operational and staffing costs from 2017 to 2019 2017 2018 2019 ASW Regional Services servers’ operational costs: 62,169.17 (low) 93,253.93 (high) 81,307.62 (low) 101,007.73 (high) 62,307.62 (low) 92,007.73 (high) ASW Regional IT operational costs for PMO set-up : 22,000.00 22,000.00 32,000.00 ASW Regional Staffing Costs: 135,000.00 115,000.00 210,000.00 Grand total: 219,169.17 (low) 252,253.93 (high) 218,307.62 (low) 238,007.73 (high) 304,307.62 (low) 334,007.73 (high) 5.6.1.2 Expansion 5.6.1.2.1 Partnership Matrix for the Exchange of the ASEAN Customs Declaration Document (ACDD) The ASEAN Customs Declaration Document (ACDD) is a harmonized paper document. It includes the data elements that are required to be submitted to Customs by both exporters and importers. The format of the ACDD was based on the European Union Single Administration Document (EU SAD). In total, the ACDD contains 48 data elements. However, many of these data elements are required by the importing country only and, as such, are not collected by the exporting country when the export declaration is submitted. It is understood that the AMS have agreed to develop a subset of the ACDD consisting of “export data” only, which would also need to be agreed by the participating AMS before it can be included in the ASW. 5.6.1.2.2 Information Process Modeling Study The “Information Process Modeling Study” identified four additional documents to be catered for in the ASW, including the Freight Booking Confirmation, Freight Loading Confirmation, Pre-departure Cargo Report and Sanitary / Phytosanitary Certificate. The study describes both the current “As-Is” processes and the future “To-Be” processes. It also recommends international standard message formats for each document. Freight Booking Confirmation According to the study, the current “As-Is” process is that the Freight Booking Confirmation is sent by the Shipping Agent to the Port Operator, and also to the Shipper. The “To-Be” process, as outlined in the study, proposes that the document is sent by the Shipping Agent to the NSW in the first instance, which can then be made available to the Port Operator, Customs and the ASW. The report also identifies an international message standard format, in the form of a UN/EDIFACT message called IFTMBC. The reason stated is that it is widely used by carriers and their agents. ASW Technical and Financial Evaluation – Final Report Page 68 of 80 Freight Loading Confirmation According to the report, the current “As-Is” process is that the Freight Loading Confirmation is sent by Port Operator to the Shipping Agent, which is the reverse of the process for a Freight Booking Confirmation. The “To-Be” process, as outlined in the study, proposes that the document is sent by the Port Operator to the NSW in the first instance, which can then be made available to the Shipping Agent, Customs and the ASW. The report also identifies an international standard message format, in the form of a UN/EDIFACT message called COARRI. The reason stated is that it is widely used by terminal operators. Pre-departure Cargo Report According to the study, there is no “As-Is” process for the Pre-departure Cargo Report. Instead, the study describes the “As-Is” process of the Cargo Manifest, which is sent by the Shipping Agent (at Origin) to the Shipping Agent (at Destination) and, in some countries, a copy is also sent to Customs. The “To-Be” process, as outlined in the study, proposes a new document, the Pre-departure Cargo Report, to be sent by the Shipping Agent (at Origin) to the NSW, which is then made available to Customs and the ASW. The report also identifies an international standard message format, in the form of a UN/EDIFACT message called IFCSUM. The reason stated is that it is widely used by carriers (assumed to be shipping lines) and their agents. Sanitary / Phytosanitary Certificate According to the report, the current “As-Is” process is that the exporter first makes an application for the Sanitary / Phytosanitary Certificate through the National Plant Protection Organization (NPPO), then the approved certificate is sent to the importer. The importer then presents the certificate to the NPPO in the importing country. The “To Be” process, as outlined in the study, proposes that the Exporter makes an application through the NSW, then the approved certificate is sent directly to the NPPO in the importing country. If required, the exporter can also get a paper copy of the certificate and send it to the importer. The report also identifies an international standard message format, in the form of the Electronic Sanitary and Phytosanitary Certificate (e-Cert), which is published by the United Nations Centre for Trade Facilitation and Electronic Business (UN/CEFACT), and the Electronic Phytosanitary Certificate (e-Phyto), which is published by the International Plant Protection Convention (IPPC). 5.6.1.2.3 Feedback from AMS Questionnaire (Part 1) The feedback from AMS Questionnaire (Part 1) relating to this section is provided below. Q28. Are Export declarations currently being submitted to Customs via your NSW? Indonesia, Malaysia, Singapore and Thailand all answered “Yes” to this question, while Viet Nam answered “No”. Q29. Which of the following documents are currently supported by your NSW? ASW Technical and Financial Evaluation – Final Report Page 69 of 80 Indonesia, Malaysia, Singapore, Thailand and Viet Nam all responded to this question. Table 39 below shows which of the additional documents, as identified in the “Information Process Modeling Study”, are currently supported by the NSWs of the participating AMS. Table 39: Additional documents supported by the NSWs of the participating AMS Document Indonesia Malaysia Singapore Thailand Viet Nam Freight Booking Confirmation - - - - - Freight Loading Confirmation - - - - - Pre-Departure Cargo Report - - - - √ Sanitary / Phytosanitary Certificate √ - - - √ Others √ √ √ √ √ In addition to Customs declarations, Singapore commented that their NSW supported Certificates of Origin. Malaysia commented that their NSW supported Manifests, Permits, Preferential Certificates of Origin, the Strategic Trade Act (STA) and Electronic Funds Transfers (EFT). Q30. Which of the following parties are currently connected to your NSW? Indonesia, Malaysia, Singapore, Thailand and Viet Name all responded to this question. Table 40 shows which parties, that are required to support the implementation of the additional documents identified in the “Information Process Modeling Study”, are currently connected to the NSWs of the participating AMS. Table 40: Parties connected to the NSWs of the participating AMS Party Indonesia Malaysia Singapore Thailand Viet Nam Customs √ √ √ √ √ Shipping Agents √ √ - √ √ Port Operators √ √ - √ √ National Authority Responsible for Food Safety (i.e. the authority responsible for issuing of Sanitary Certificates) √ √ - √ √ National Plant Protection Organization (NPPO) (i.e. the authority responsible for issuing of Phytosanitary Certificates) √ - - √ √ Others √ √ - √ √ 5.6.2 Funding Requirements 5.6.2.1 Operation Based on the information available, the current plan to phase in the PMO based on the volume of transactions certainly makes sense. However, the costs outlined in the paper entitled “ASW Regional IT Operational and Staffing Costs” assume a start date of January 2017 and not January 2016. ASW Technical and Financial Evaluation – Final Report Page 70 of 80 In fact, some of the costs outlined in the paper would be expected in 2016. For example, the Project Officer and Technical Officer would need to be recruited in the 4th quarter of 2016, so they can be trained. Therefore, there would be some staffing and training costs expected in 2016. There may also be hardware and software costs in 2016, which are not accounted for in the paper. Travel expenses to attend the ASW meetings should also be accounted for in 2016, unless the VPT-IT are prepared to attend at their own expense. Recommendation No. 10 To review the ASW Regional IT Operational and Staffing Costs to cover the operation of the ASW Regional Services during 2016. 5.6.2.2 Expansion Based on the information available, the current plans for the expansion of the ASW still seem to be more theoretical, because there are practical issues that do not appear to have been be addressed. Firstly, it is important to understand that no NSW is the same. Even those countries around the world who have implemented Common Off-The-Shelf (COTS) solutions have customized their systems to meet national requirements. Secondly, UNECE Recommendation No. 33, “Recommendation and Guidelines on establishing a Single Window”, implies that a Single Window is a central system that it is integrated with the various border control authorities. In fact, according to the World Bank’s Ease of Doing Business Report 2014, there were 73 of the 189 member economies surveyed who claimed to have Single Windows and yet only 18 have a Single Window “that links all relevant government agencies”. There have been no further updates in the more recent 2015 or 2016 reports. In order to expand the ASW, there would need to be at least two NSWs who share the same the same functionality and, where required, be connected to the relevant border control authority. For example, the Phytosanitary Certificate would be an obvious document to include in the ASW, yet most NSWs around the world do not cater for this document. Typically, exporters have to apply directly to their National Plant Protection Organization (NPPO) to obtain a Phytosanitary Certificate. Other countries have also tried to encourage industry stakeholders to send their commercial and transport documents to the NSW voluntarily, without much success. Generally speaking, industry stakeholders would expect some incentive to send copies of the documents to the NSW voluntarily, unless the government mandates it. The latter would require legislative changes. 5.6.2.2.1 Export Data Compared to other documents, which are explained in the following sections, export data should be relatively easy to include in the ASW. The main reason for this is that most NSWs, including Indonesia, Malaysia, Singapore and Thailand, are already connected to Customs via the NSW and export declarations are already mandated by governments. In this scenario, export data would be the data that is available on an Export declaration at the time of export. Such data could include the names and addresses of the exporter and importer, transport information (e.g. carrier, B/L number, port of loading, estimated departure / arrival dates etc) and commercial information (e.g. invoice number, total invoice amount, commercial description etc). ASW Technical and Financial Evaluation – Final Report Page 71 of 80 While it may include the AHTN code, it would not include any national extensions, applicable duty rates, duties / fees payable or permit / licence numbers. 5.6.2.2.2 Freight Booking Confirmation As the “Information Process Modeling Study” explains, the Freight Booking Confirmation is sent from the Shipping Agent to the Port Operator. In practice, it may not be a physical document, as freight bookings can also be made via a website, by email, by fax or even by telephone. The study also identifies an international standard message format, in the form of a UN/EDIFACT message called IFTMBC, which is commonly used within the maritime industry. However, the study does not take into consideration how this functionality could be extended to include other modes of transport, where other standards are used. For example, the air industry are arguably more technically advanced than the maritime industry, because they have had their own message standards for many years, in the form of Cargo Interchange Message Procedures (Cargo-IMP). These standards have recently been superseded by Cargo-XML. The Cargo-IMP message for a Booking Request is commonly known as the “FFR” and the Cargo-XML equivalent is called the “XFFR”. One of the biggest challenges for the ASW would be how to convince the Shipping Agents to send their documents to the NSW, unless this can be mandated by the respective governments or there is incentive for them to do so. In summary, there are still many practical questions that need to be answered before the ASW can be expanded to include the “Freight Booking Confirmation”, such as: 1. How do Shipping Agents currently send the document to the Port Operator? 2. Do the Shipping Agent’s systems support the exchange of electronic information? 3. If so, do they support UN/EDIFACT or XML, or perhaps just CSV or fixed length files? 4. Is there a need to support different message formats depending on the transport mode? 5. How will sending the document to the NSW benefit the Shipping Agent? 6. Can national regulations be changed to make this mandatory? 7. Who owns the data and can it be shared with other parties without their permission? The AMS may wish to consider carrying out a more comprehensive feasibility study to address the common practical issues of including the Freight Booking Confirmation in the ASW. 5.6.2.2.3 Freight Loading Confirmation As the “Information Process Modeling Study” explains, the Freight Loading Confirmation is sent from the Port Operator to the Shipping Agent. Again, in practice, it may not be a physical document. The study also identifies an international standard message format, in the form of a UN/EDIFACT message called COARRI, which is commonly used within the maritime industry. Similar to the case of the Freight Booking Confirmation, the air industry already have their own equivalent message to the COAARI, called the Status Message. This message is exchanged between the ground handling agents (GHAs), airlines and freight forwarders. Although the business process is slightly different to that of the maritime industry, the functionality is very similar. The Cargo-IMP message for a Status Message is commonly known as the “FSU” and the Cargo-XML equivalent is called the “XFSU”. One of the biggest challenges for the ASW would be how to convince the Port Operator to send their documents to the NSW, unless this can be mandated by the respective governments or there is incentive or them to do so. ASW Technical and Financial Evaluation – Final Report Page 72 of 80 In summary, there are still many practical questions that need to be answered before the ASW can be expanded to include the “Freight Loading Confirmation”, such as: 1. How do Port Operators currently send the document to the Shipping Agents? 2. Do the Port Operator’s systems support the exchange of electronic information? 3. If so, do they support UN/EDIFACT or XML, or perhaps just CSV or fixed length files? 4. Is there a need to support different message formats depending on the transport mode? 5. How will sending the document to the NSW benefit the Port Operator? 6. Can national regulations be changed to make this mandatory? 7. Who owns the data and can it be shared with other parties without their permission? The AMS may wish to consider carrying out a more comprehensive feasibility study to address the common practical issues of including the Freight Loading Confirmation in the ASW. Recommendation No. 11 To carry out a comprehensive feasibility study into the readiness and willingness of Shipping Agents, Freight Forwarders, Port Operators and Ground Handling Agents to send their electronic transport documents to the NSWs on a voluntarily basis, including the Freight Booking Confirmation and Freight Loading Confirmation. 5.6.2.2.4 Pre-departure Cargo Report As the “Information Process Modeling Study” explains, the Cargo Manifest is sent from the Shipping Agent (at Origin) to the Shipping Agent (at Destination), and sometimes to Customs. However, a Cargo Manifest is usually very different to a Pre-departure Cargo Report. Several countries have already introduced a “Pre-departure Cargo Report” with varying degrees of success. This is because one of the key differences between a “Cargo Manifest” and a “Cargo Report” is that the latter is required by Customs for risk management purposes. As such, Customs typically require more detailed information than is currently available on the “Cargo Manifest”, such as the HS Code, Commercial Description etc, and often “House” level information. One of the biggest challenges that other countries have faced, especially for maritime cargo, is that the Shipping Agent does not have all of the information required by Customs. Japan Customs, for example, have learned from the experiences of other countries, particularly the United States and Europe. First, instead of making the carrier legally responsible for submitting the “Pre-departure Cargo Report”, they have split the information into two levels. The carrier is legally responsible for the “Master” level “Pre-departure Cargo Report” and the Non-Vessel Operating Common Carrier (NVOCC) is legally responsible for submitting a “House” level “Pre-departure Cargo Report”. The second issue is with the timing, whereby Japan’s law requires the “Pre-departure Cargo Report” to be submitted 24 hours before departure and not 24 hours before loading like others. Another important point to note is that the current “As-Is” process for a “Pre-departure Cargo Report” in other countries is that the document is submitted directly by the Shipping Agent (at Origin) to the NSW (at destination) i.e. it is not submitted to the NSW (at origin). In fact, many Shipping Agents within the Member States are already submitting a “Pre-departure Cargo Report” to the United States, the EU and Japan, and possibly others. It is also worth noting that a lot of the information in the Pre-departure Cargo Report will also be available in the export data set, so there may be no need for both sets of information. In fact, one of ASW Technical and Financial Evaluation – Final Report Page 73 of 80 the principles of a Single Window, according the UNECE Recommendation No. 33, is that “individual data elements should only be submitted once”. Also, one of the key benefits of the ASW, which other countries do not have, is that it makes it easier for Customs administrations to share information. The AMS may wish to consider carrying out a comprehensive feasibility study to address the common practical issues of including the Pre-departure Cargo Report in the ASW. Recommendation No. 12 To carry out a comprehensive feasibility study into the readiness and willingness of national Customs administrations to request the Shipping Agents to send Pre-departure Cargo Reports to the NSW, which should also take into account any legal implications and any duplication of data being submitted. 5.6.2.2.5 Phytosanitary Certificate Of the four documents identified in the “Information Process Modeling Study”, the Sanitary / Phytosanitary Certificate is the only document which has a clearly defined international standard message format, in the form of an XML message called e-Cert. This multi-purpose message was initially published by the United Nations Centre for Trade Facilitation and e-Business (UN/CEFACT), which covers both Sanitary and Phytosanitary Certificates. More importantly, the International Plant Protection Convention (IPPC) have published their own international XML standard for the electronic Phytosanitary Certificate, known as e-Phyto, which is based on a subset of UN/CEFACT’s e-Cert’s standard. Given that there is already an international standard available, and as it is also a Government-to￾Government (G2G) document, this makes the e-Phyto an ideal candidate for the ASW. The AMS may wish to consider carrying out a comprehensive feasibility study to look at the practical issues of including the Sanitary / Phytosanitary Certificate in the ASW. Recommendation No. 13 To carry out a comprehensive feasibility study into the readiness and willingness of the certificate issuing authorities to exchange electronic Sanitary and Phytosanitary (e-SPS) certificates with their overseas counterparts via the ASW. 5.6.2.2.6 Other Documents In addition to those documents identified above, there are many other documents that could be exchanged by the ASW. The ASW software is certainly capable of supporting any document and in almost any standard format, such as XML, UN/EDIFACT, CSV etc. Firstly, there would need to be an identified business need. Certificates would be ideal candidates for the ASW, because many of them are issued by governments in the exporting country, which need to be presented to the government of the importing country. For example, international certificates are issued for the Convention on International Trade in Endangered Species of Wild Fauna and Flora (CITES) and there are common standards already available for this. There are also a number of international and industry initiatives, which may be relevant to the ASW in the future. For example, the World Customs Organization (WCO) has published a paper called the “Guide to the exchange of Customs valuation information”, which could also be considered. ASW Technical and Financial Evaluation – Final Report Page 74 of 80 The United States, for example, have also been working on a new security initiative called the “Air Cargo Advance Screening (ACAS)” programme. There are similar initiatives in Europe and Canada, called the “Pre-Departure/Loading Consignment Information for Secure Entry (PRECISE)” and “Pre Load Air Cargo Targeting (PACT)” respectively. IATA is working closely with all three countries to encourage them to accept the air industry standard messages, including the Flight Manifest, House Manifest, Air Waybill and House Waybill. Under this type of programme, the incentive is for airlines to have their cargo cleared faster. Recommendation No. 14 To carry out an analysis of the current international trends and industry initiatives to assess their applicability within ASEAN, and also to assess their potential impact on ASEAN in the future. 5.7 Adequacy of the plans for technical training for the Project Management Office (PMO) The purpose of this section is to detail the findings of Task 8 of the TOR; “An assessment of the adequacy of the plans for technical training for the Project Management Office (PMO) Staff in the operation of the ASW Regional Services”. 5.7.1 Findings The findings below are based on the RFP, the “Design of Project Management and Planning Team (Project Management) Office”, “ASEAN Administration Manual”, “User Manual”, information provided by the Contractor and “Scope of Work for the PMO Regional Services Training”. 5.7.1.1 Project Management Office (PMO) 5.7.1.1.1 Request for Proposal The following are extracts from the RFP relating to this section. The RFP states that; “The Contractor shall operate the Regional Services and assist AMS in operation of their ASW software during the testing phase (Phase 1 and phase 2). The Contractor will provide a written manual on the operation and maintenance of the regional services software and any reporting systems”. The RFP states that; “The Contractor will provide input to the Regional Operations team, which will succeed the VPT-IT Team mentioned in Assumption 11 once the latter is in place, in linking the ASW web portal (asw.asean.org) with the ASW RS portal”. 5.7.1.1.2 Design of Project Management and Planning Team (Project Management) Office The paper, entitled “Design of Project Management and Planning Team (Project Management) Office”, describes how the Project Management Office (PMO) should be set up to oversee the implementation of the future ASW activities. It explains that the PMO be set up in three phases;  Phase 1 – to be handled by the IT Virtual Project Team (VPT-IT) prior to the live operation  Phase 2 – will require the creation of a small ASW Project Team (ASW PT), consisting of a Project Officer and Technical Specialist, who will start operation from January 2017 to December 2018  Phase 3 – will require the establishment of the full PMO by 2018 ASW Technical and Financial Evaluation – Final Report Page 75 of 80 The paper also states that, for Phase 2, “Formal training and hands-on familiarization activities will be provided by the contractor specifically for the regional services operations”. On further clarification, the Consultant was advised that a decision had been made at the 15th ASWC Meeting, which would require the VPT-IT team to take on the functions of the PMO until December 2016, prior to the turnover of the ASW regional operations service to AMS by January 2017. It is therefore implied that formal training would be required by the Contractor to the VPT-IT prior to the cutover to the live operation by 30th December 2015. 5.7.1.1.3 Information provided by the Contractor Based on the feedback from the Contractor, they are not aware of any obligation under the RFP to provide formal training to the PMO. However, they have provided some technical training slides for the Regional Services, which could be used as a basis for the training. The Contractor was also not aware of any requirement to provide a technical training plan, as this is also not sated anywhere in the RFP. With reference to the RFP requirement to provide a written manual, the Contractor has provided two manuals for the ASW Regional Services Portal; the “ASEAN Administration Manual” and the “User Manual”. 5.7.1.1.4 Scope of Work for the PMO Regional Services Training Since the release of the first version of this Final Report, the Contractor has provided a draft Scope of Work for a “Training Program for PMO for Regional Services”. It should be noted that this training program is aimed at the AMS staff responsible for the operation and administration of the Regional Services Portal, until the PMO team has been established. The training is expected to be conducted over 4 full days, between 9:00am and 5:00pm, and the Contractor has assumed that the training will be carried out within the ASEAN Secretariat’s offices. It is understood that the training will be comprehensive and will delve into technical details, which will provide operations staff the opportunity to obtain a detailed understanding of the functionality of the Regional Services Portal and, where necessary, the ASW Gateway Software. ACTI have also requested that the Contractor includes the transition activities for the turnover of the ASW Regional Services to the PMO team and to cover version control and incident management procedures. A detailed agenda will be provided two weeks prior to the training. 5.7.2 Assessment The Consultant was not able to identify any plans for technical training for the PMO. While there may be an expectation by the AMS that the Contractor will provide some formal training, the Contractor seems to be under no obligation under the RFP to provide it. The need for a comprehensive “ASEAN Administration Manual,” to operate and maintain the ASW Regional Services Portal, will be essential in the future, especially for the new staff that will be recruited for the PMO. These new staff will have no background to the ASW, so they will depend on the written manual as their reference guide. ASW Technical and Financial Evaluation – Final Report Page 76 of 80 Recommendation No. 15 To reach an agreement with the Contractor to provide operational training to the VPT-IT team and to ensure that the “ASEAN Administration Manual” is comprehensive enough to be used as a reference guide for new PMO staff in the future. 5.8 Quality assurance procedures for the ASW Regional Services The purpose of this section is to detail the findings of Task 9 of the TOR; “An evaluation of the quality assurance procedures for the ASW Regional Operations Services covering: • Security quality – effectiveness of the security features implemented. • Information quality – accuracy, meaningfulness, and timeliness of the information produced. • Process quality – effect on information management process quality” 5.8.1 Findings The findings below are based on the RFP, the “Contractor’s Technical Proposal in response to the RFP” and other information provided by the Contractor. 5.8.1.1 Quality Assurance Procedures 5.8.1.1.1 Request for Proposal The following are extracts from the RFP relating to this section. “Bidders must provide a comprehensive written Project and Quality Management Approach, which should at least include:  Approach to technical activities (i.e. technical activities to execute the project);  Project Management procedures (day to day activities, project progress monitoring procedures, etc.);  Configuration and change management procedures;  Quality assurance procedures;  Security and confidentiality;” It should be noted that there is no reference to any specific requirement for the Contractor to provide any documented quality assurance procedures for the ASW Regional Operations Services. 5.8.1.1.2 Contractor’s Technical Proposal in response to the Request for Proposal The following is an extract from the “Contractor’s Technical Proposal in response to the RFP”. “In line with quality standards such as ISO9001:2000, the overall development, project and quality management methodology, to assuring the quality of the project deliverables includes:  Utilizing the right experts who have proven relevant experience and skills  The use of documented, field proven methodologies and the use of field-proven products  The use of established and proven relevant standards and conventions for design, development, configuration, testing, and documentation  The establishment of objective quality requirements, quality indicators, criteria, and metrics, which will be used to describe the customer’s requirements and against which the project quality performance will be monitored ASW Technical and Financial Evaluation – Final Report Page 77 of 80  The establishment of documented quality assurance and quality control processes that assess actual quality performance against the quality targets, requirements and criteria. This will include processes in the project plan that covers: o Quality and technical reviews of key deliverables by experts not involved in the day￾today work of the project, so as to provide an independent view o Inspections and Internal Quality Audits. In support of sustainability, deliverables, to the extent practical, will be provided only in electronic form.” 5.8.1.1.3 Information provided by the Contractor Based on the feedback from the Contractor, they are not aware of any obligation under the RFP to provide any specific quality assurance procedures for the ASW Regional Services. The Contractor also pointed out that the decision on where to host the ASW Regional Services will have some relevance to the quality assurance procedures, particularly in terms of security. 5.8.2 Evaluation The Consultant has not been able to identify any specific, documented quality assurance procedures for the ASW Regional Services. The Contractor also does not seem to be under any obligation under the RFP to provide them. Although there may be no need to provide separate written quality assurance procedures, they should be included in the ASW Regional Services Portal’s “ASEAN Administration Manual” and “User Manual” and, if possible, in the operational training to the VPT-IT team (see section 5.7). Recommendation No. 16 To reach an agreement with the Contractor to include quality assurance procedures in the ASW Regional Services Portal’s ASEAN Administration Manual and User Manual and, if possible, cover it in the operational training to the VPT-IT team. ASW Technical and Financial Evaluation – Final Report Page 78 of 80 6 Recommendations A summary of the recommendations identified this report is provided in Table 41 below, including a cross-reference to the sections containing the background information. Table 41: Summary of recommendations made in the report No. Recommendation Section(s) 1 To ensure that all of the features of the ASW Gateway / Software, as stated in the RFP, are thoroughly documented in the “ASW Gateway Technical Operations Guide”. At a minimum, this should cover queuing and re-try mechanisms, error responses for unsuccessful delivery, alert messages, users and roles, archiving / purging of data, use of PKI and digital certificates, installation manuals and interface scripts. 5.2.2.1 2 To ensure that all of the features of the ASW Regional Services, as stated in the RFP, are thoroughly documented in the “ASEAN Administration Manual” and “User Manual”. At a minimum, this should include; the updating of reference data, automated monitoring of message exchanges between AMS, MIS reporting and performance monitoring. 5.2.2.2 3 To ensure that documentation is generic and relevant for all documents, not just the e-ATIGA Form D. For example, the support procedures and security incident management plan is relevant for all documents and should be documented separately from the “ASW Pilot e-ATIGA Form D - Operations Handbook”. It is also suggested that the “ASW Pilot e-ATIGA Form D - ASW Gateway Technical Operations Guide” should be renamed, so that it is clear that it is relevant for all documents, and that the specific ATIGA Form D related functionality should be consolidated into “ASW Pilot e-ATIGA Form D - Operations Handbook”. The title of the “Development and Live Implementation of the ASEAN Single Window (ASW) Software for the exchange of the ATIGA CO Form D - Change and Configuration Management Guide” may just need to be renamed to a more generic title. 5.2.2.3 4 To make the Customs response mandatory and to ensure that it is fully tested during the parallel testing prior to the cutover to live operation. 5.3.2.1 5 To ensure that the alternative reporting mechanisms of Singapore and Thailand have been fully tested, so that the number of ATIGA Form Ds sent by the sending AMS match the number received by the receiving AMS. Similarly, the number of responses sent by the sending AMS should match the number received by the receiving AMS. 5.3.2.1 6 To ensure that all of the ASW technical specifications are platform neutral and can be easily implemented in any NSW environment without being dependent on any special software. This should include details of how the standard communications protocol (i.e. ebMS) is being used and all interfaces between the NSW and the ASW Regional Services Portal. 5.3.2.1 ASW Technical and Financial Evaluation – Final Report Page 79 of 80 No. Recommendation Section(s) 7 To ensure that the acceptance criteria in the Parallel Test Plan is met by resolving the outstanding issues, including; the testing of all categories of ATIGA Form D, the testing of all Customs response statuses, the harmonization of ATIGA Form D and Customs responses, and the accuracy of the ASW Regional Services reporting. 5.3.2.2 8 To plan for and carry out more comprehensive testing of the security features (i.e. use of PKI and digital certificates) and scalability of the Contractor’s software, including stress testing, different document sizes / types (e.g. Pre-departure Cargo Report) and different formats (e.g. UN/EDIFACT). 5.4.2 9 To define three scenarios of small, medium and large transaction volumes and to ask the Contractor to provide the appropriate hardware / software requirements, together with the associated costs. 5.4.2 10 To review the ASW Regional IT Operational and Staffing Costs to cover the operation of the ASW Regional Services during 2016. 5.6.2.1 11 To carry out a comprehensive feasibility study into the readiness and willingness of Shipping Agents, Freight Forwarders, Port Operators and Ground Handling Agents to send their electronic transport documents to the NSWs on a voluntarily basis, including the Freight Booking Confirmation and Freight Loading Confirmation. 5.6.2.2.2 & 5.6.2.2.3 12 To carry out a comprehensive feasibility study into the readiness and willingness of national Customs administrations to request the Shipping Agents to send Pre-departure Cargo Reports to the NSW, which should also take into account any legal implications and any duplication of data being submitted. 5.6.2.2.4 13 To carry out a comprehensive feasibility study into the readiness and willingness of the certificate issuing authorities to exchange electronic Sanitary and Phytosanitary (e-SPS) certificates with their overseas counterparts via the ASW. 5.6.2.2.5 14 To carry out an analysis of the current international trends and industry initiatives to assess their applicability within ASEAN, and also to assess their potential impact on ASEAN in the future. 5.6.2.2.6 15 To reach an agreement with the Contractor to provide operational training to the VPT-IT team and to ensure that the “ASEAN Administration Manual” is comprehensive enough to be used as a reference guide for new PMO staff in the future. 5.7.2 16 To reach an agreement with the Contractor to include quality assurance procedures in the ASW Regional Services Portal’s ASEAN Administration Manual and User Manual and, if possible, cover it in the operational training to the VPT-IT team. 5.8.2 ASW Technical and Financial Evaluation – Final Report Page 80 of 80 Appendix A – AMS Questionnaire (Part 1) Please refer to document “ASW Component 3 Questionnaire - ASEAN Member States - Part 1.docx” Appendix B – AMS Questionnaire (Part 2) Please refer to document “ASW Component 3 Questionnaire - ASEAN Member States - Part 2.docx”