PHOTO CREDIT TETRA TECH GEORGIA GOOD GOVERNANCE INITIATIVE Benchmarking Georgian Municipalities: Final Report JUNE 2016 This publication was produced for review by the United States Agency for International Development. It was prepared by Tetra Tech. Prepared for the United States Agency for International Development, USAID Contract Number AID-114-C-15-00001, Good Governance Initiative (GGI) in Georgia. This report was prepared by: Tetra Tech 159 Bank Street, Suite 300 Burlington, Vermont 05401 USA Telephone: (802) 495-0282 Fax: (802) 658-4247 E-Mail: international.development@tetratech.com Principal Contacts: David Smith, Chief of Party (dsmith@ggi.ge) Jesse Biddle, Project Manager (jesse.biddle@tetratech.com) Kelly Kimball, Deputy Project Manager (kelly.kimball@tetratech.com) GEORGIA GOOD GOVERNANCE INITIATIVE BENCHMARKING GEORGIAN MUNICIPALITIES: FINAL REPORT JUNE 2016 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES i TABLE OF CONTENTS 1.0 THE PROJECT ........................................................................................................ 1 2.0 THE PILOT STUDY ............................................................................................... 2 3.0 SELECTION OF SERVICES TO BE ASSESSED.................................................. 3 4.0 METHODOLOGICAL CHOICES.......................................................................... 4 5.0 DATA COLLECTION ............................................................................................ 5 6.0 TRAINING FOR MUNICIPALITIES ..................................................................... 6 7.0 BENCHMARKING METHODOLOGY IMPLEMENTATION: LESSONS LEARNED................................................................................................................ 7 8.0 ADAPTING QUESTIONNAIRES ......................................................................... 8 9.0 GOING BEYOND THE PILOT PROJECT: LEARNING FROM THE PILOT SURVEY TO DEVELOP A NATIONWIDE EFFICIENCY ASSESSMENT SYSTEM................................................................................................................... 9 9.1 PROMOTING COMPETITION BETWEEN MUNICIPALITIES ..................................... 9 9.2 ORGANIZATION...................................................................................................................... 9 9.3 HORIZONTAL EXTENSION: INCLUSION OF NEW MUNICIPALITIES (SEE APPENDIX III) ...........................................................................................................................10 9.4 VERTICAL EXTENSION: INCLUSION OF NEW SERVICES (SEE APPENDIX III)..................................................................................................................10 APPENDIX I: MODIFICATION TO THE QUESTIONNAIRES ................................. 12 A-1.1 LIBRARY......................................................................................................................................12 A-1.2 PRE-SCHOOL............................................................................................................................12 A-1.3 WASTE COLLECTION..........................................................................................................13 APPENDIX II: REVISED QUESTIONNAIRES ............................................................. 14 A-2.1 WASTE COLLECTION QUESTIONNAIRE.....................................................................14 A-2.2 LIBRARY QUESTIONNAIRE................................................................................................. 20 A-2.3 PRE-SCHOOL QUESTIONNAIRE ......................................................................................24 APPENDIX III: GOING BEYOND THE PILOT PROJECT: RECOMMENDATIONS ON THE IMPLEMENTATION OF A BENCHMARKING SYSTEM FOR IMPROVING MUNICIPAL SERVICES EFFICIENCY ........................................ 26 A-3.1 CURRENT STATE OF THE PROJECT...............................................................................26 A-3.2 ORGANIZATION....................................................................................................................26 A-3.3 MISSION OF THE TASK GROUP .......................................................................................27 A-3.4 TO KEEP IN MIND..................................................................................................................28 A-3.4.1 Work to be undertaken (MRDI and task group)...........................................28 GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 1 1.0 THE PROJECT The objective of the work assigned to the short term technical consultant was to support the Ministry of Regional Development and infrastructure (MRDI) in developing a benchmarking system for measuring, monitoring, and improving local government performance. This work is part of the Good Governance Initiative in Georgia (GGI). The Ministry of Regional Development and Infrastructure decided to start with a pilot project including three municipalities, and to draw lessons from the experience before extending the study to other municipalities. The benchmarking system will help MRDI to improve transparency, accountability and promote competition between Georgian municipalities. One can expect that this will translate into better service quality and lower costs. The external consultant assignment includes needs assessment, developing an efficiency measurement and benchmarking system for Georgian municipalities, designing questionnaires for data collection, providing training to the three pilot municipalities, and reviewing the lessons learned in the pilot study. GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 2 2.0 THE PILOT STUDY The pilot project included three of the twelve Georgian self-governing cities: Akhaltsikhe, Batumi and Kutaisi. Akhaltiskhe usual population is 21,600, Batumi 154,100, and Kutaisi 149,100 (2014). Kutaisi and Batumi are the 2nd and 3rd largest cities in Georgia. The country has a total of 64 self-governing municipalities. The Tetra Tech team ( ) met with representatives of the three pilot municipalities, Akhaltsikhe (December 2, 2015), Batumi and Kutaisi (December 3, 2015). The team took this opportunity to inform the three cities of MRDI’s wish to implement a municipal benchmarking system, of the objectives being pursued, and of the benefits the municipality could expect from the new assessment system, once operational. Unfortunately, the team was not able to meet employees familiar with the three end users’ services considered in the pilot study, with the exception of the library in Akhaltsikhe. None of the three municipalities expressed specific demand regarding the performance assessment system and the municipal services to be investigated. We therefore had to focus on the implicit needs deriving from general good governance principles. These imply accountability, transparency, responsiveness, as well as efficient, effective, and equitable provision of services. The availability of performance indicators, the acquisition of knowledge (data and analyses), and the exchange of best practices and expertise between municipalities would help achieve all these goals, with the exception of responsiveness (which requires institutional changes). Transparency is in the best interest of the citizen, but possibly not, in the short term, in the perceived best interest of elected municipal officials, as transparency can release both a positive and a negative image of the municipal management. GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 3 3.0 SELECTION OF SERVICES TO BE ASSESSED Another expected return of the pilot project is an increase awareness of the benefits municipalities could expect from a systematic measurement of resources (inputs), services (outputs) and policy goals achievement (outcome). Selected services have to be relatively simple, easy to compare between municipalities, and have a straightforward outcome. Most of the cultural services would not met these criteria, as cultural activities are too diversified and the outcome to express in physical units (library is however an exception). More generally, inclusion criteria can be found in the literature (e.g., the criteria for selecting the service provision sector in the report on government services, Australian government, Report on Government Services, Productivity commission (2016) ; Steering Committee for the Review of Government Service provision (2011) “Criteria for selecting service provision sectors for inclusion in the Australian Report on Government Services”- http://www.pc.gov.au/__data/assets/pdf_file/0010/116587/rogs-inclusion-criteria.pdf). • Services have similar objectives across municipalities (they should not be municipality-specific) to facilitate comparison; • There is a significant scope for efficiency, effectiveness, or quality improvement; • The service contributes significantly to the community’s well-being ; • The cost of data collection and reporting is reasonable (consider the cost–benefit relationship). At the beginning of the pilot study, Tetra Tech ( ) met with the representative of MRDI ( ) on November 26, 2015. MRDI expressed the wish that at least one (end-user) soft service (education, social services….) and one (end-user) hard service (waste collection, street cleaning….) be included in the pilot project. Finally, three end-users services or “core services” (household waste collection, pre-school, library) and four support services (finance, human resources, ITC, fleet management) were included in the pilot project. GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 4 4.0 METHODOLOGICAL CHOICES The methodology was extensively described in the 2nd intermediary report (Benchmarking: second report on methodology, January 2016, 47 pages). The basic model links the resources (or inputs) to the outputs and outcomes. Both the resources and the outputs are measured in monetary (GEL) and in physical units. Efficiency is measured in relative rather than in absolute terms. Efficiency degree vary between 0 (totally inefficient) and 1 (totally efficient). A service producing unit is declared (fully) efficient (in input) if it uses the least possible quantity of input to deliver a defined quantity of services (output), or efficient in output, if it delivers the highest possible quantity of services (output) with a defined volume of resource. The resources used to deliver a certain volume of services, or the volume of services delivered with a defined quantity of resources are compared against a standard (the best performing municipality or an international standard). The resources and the output(s) have to be adequately defined (there might be more than one input and more than one output). For end-user services, a similar indicator system can be applied to all services. The context is slightly different for support services because the administration is the recipient of the services. An indicator of the size of the administration is often used to measure the service volume delivered (New Zeeland Government (2015) Administrative & Support Services, Benchmarking Report for the financial year 2013/2014 - http://www.treasury.govt.nz/statesector/performance/bass/benchmarking; NOUSgroup (2014) Benchmarking with benefits: Sharing our learning – Driving efficiencies, NFP Benchmarking Project, Australia http://www.nfpbenchmarking.com.au/sites/default/files/Benchmarking%20with%20Benefits.pdf ). Several technics are available to measure the degree of efficiency and the potential of improvement (reduction of input consumption or increase in output) for each municipality and each service. I recommend using DEA (data envelopment analysis) as soon as data are available for about 15 municipalities (see for example, Norman, N. and Stoker, B. (1991) Data envelopment analysis: the assessment of performances, Wiley, New York; also Coelli, T.J. et al. (2005), An introduction to efficiency and productivity analysis, Springer – http://facweb.knowlton.ohio￾state.edu/pviton/courses/crp394/coelli_Intro_effic.pdf). GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 5 5.0 DATA COLLECTION To measure efficiency in service delivery, we need to collect three kind of data: the resources used to deliver the service (inputs), the volume of services (outputs), and the factors influencing production not under the control of the management (external or environmental factors). Quality of service need to be introduced as a “control variable” (it doesn’t make sense to simply compare resource use when the service quality differs significantly). When using DEA to assess efficiency, quality can be included (as an output). Additional information on processes and technology will help understand why some units are relatively efficient, while others are relatively inefficient. A questionnaire is the tool for data collection. Writing the questionnaires – one for each service - was a significant part of the work during the preparation of the survey. The questionnaires should be completed by a person in the organization familiar with the service. It is necessary to ensure that it is actually the case. The questionnaires should not simply be sent with a deadline for responding. It is important to make personal contact with the person filling the questionnaire before it is sent (to explain the purpose of the survey) and after the questionnaire has been received (to ensure that questions are clearly and properly understood). Completing the questionnaires took more time than was expected, the last two questionnaires having been received at the beginning of April. We found that some data were missing, and that the information provided was of uneven quality. The lack of personal contact with the employees completing the questionnaires is certainly one of the reason explaining the difficulties encountered in the data collection. One should avoid making the same mistake in the following surveys. Responses in the questionnaires have been entered into the data base. We have to be careful when analyzing the survey results, because some data are missing or may not be accurate (see database, missing data in yellow, suspicious data in cyan). GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 6 6.0 TRAINING FOR MUNICIPALITIES The training session for municipal employees took place in Kutaisi on April 18. It was possible to organize a unique session with participants from the three pilot municipalities. The objectives of the session were: • To explain the basic analytical model (inputs, activities, outputs, quality, customer satisfaction, outcome) • To learn how to identify inputs, outputs, quality, and factor influencing the production not under the control of management with practical exercises • To introduce and discuss the concepts of “frontier”, “best practice”, efficiency • To explain the idea behind benchmarking, and show how it works • To introduce participants to process analysis (to understand why the best performing unit is doing better than your own organization) • To propose practical exercises: designing an assessment tool for a municipal service, measuring quality • To discuss (some of) the pilot study results With a few exceptions, participants were not involved in the pilot study. There were no employees from the three end users services selected in the pilot study (waste collection, library, pre-school). Therefore, we received no feed-back on the questionnaires, the data collection process, and the indicators derived from the data base (see “Training session on performance measurement: Municipalities of Akhaltsikhe, Batumi and Kutaisi, April 18, 2016, Appendix IV). GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 7 7.0 BENCHMARKING METHODOLOGY IMPLEMENTATION: LESSONS LEARNED The pilot study was found to be very useful: • To improve the data collection tool (some of the questions were ambiguous or confusing, some were not really useful and could be dropped; before extending the study to other municipalities, the questionnaires should be submitted to a person familiar with the service and its technology); • To improve the process of data collection (the person completing the questionnaire is a key element in the process, she or he must be properly informed and have the opportunity to ask for clarification); • To show the usefulness of an in-depth analysis of municipal performances: the first results are encouraging and show a clear potential for improvement; To show that the process of data collection need to be improved: the person responsible for completing the questionnaire must be given clear instructions; a personal contact is essential GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 8 8.0 ADAPTING QUESTIONNAIRES On the basis of the lessons learned from the pilot project, we have introduced a few modifications in the questionnaires. The main changes are the following: we consider that all employees have full-time contracts (the notion of full-time equivalent is dropped), simplification (we don’t distinguish permanent and non-permanent employees anymore), some simplifications in questions regarding truck technology (front/rear loading), and the person completing the questionnaire is asked to provide a telephone number. All modifications are summed up in the document “Modification to the questionnaires”, Appendix I. Two options are open for the extension of the survey to other municipalities (same end-user services): to keep the same calendar year as in the pilot study (2014) or to shift the reference year forwards by one year (2015). The first option makeS it possible to use the data provided by the three municipalities in the pilot project (with some adjustments to take account of the few changes in the questionnaire). If the second option is retained (reference year 2015), new data will have to be collected in Akhaltsikhe, Batumi and Kutaisi (recommended) or data already collected could be used after being adjusted for price increase (monetary values only). The revised questionnaires can be found in the appendix section (Appendix II). GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 9 9.0 GOING BEYOND THE PILOT PROJECT: LEARNING FROM THE PILOT SURVEY TO DEVELOP A NATIONWIDE EFFICIENCY ASSESSMENT SYSTEM 9.1 PROMOTING COMPETITION BETWEEN MUNICIPALITIES The objective is to progressively implement a comprehensive benchmarking system for Georgian municipalities covering main municipal programs and services. It aims at helping municipalities develop a continuous improvement process. The simple fact that you are measuring performances and reporting the results can exert pressure on municipal authorities to improve service delivery, adopt good practices, and avoid excessive costs. Regular data collection (and dissemination) to assess whether municipal services are provided efficiently and whether desired results are achieved is playing about the same role, in public administration, as competition in the market, thereby acting as a form of surrogate competition (“yardstick competition”). The Netherland Bureau for Economic Policy Analysis has “characterized yardstick competition as the practice of linking financial consequences to benchmarking results” (CPB Netherlands Bureau for Economic Policy Analysis, “Yardstick competition, theory, design and practice”, The Hague, December 2000). To make yardstick competition work, the Ministry (MRDI) should modify the transfer formula design. Transfers to municipalities should not correspond to a defined percentage of actual expenditures (for pre-school, library or waste collection), but a defined percentage of standard expenditures for the service (expenditures of municipalities using good practices). 9.2 ORGANIZATION MRDI set up a task group whose mission is to develop an efficiency measurement system for Georgian municipalities based on the pilot project. A group of three, on a half-time basis (equivalent to 1.5 full time), should be able to implement the project during the first year. Once the project has attained its full rhythm of implementation, the workload might be moderately reduced (3 x 0.33 = one full time equivalent). The task group main assignments are: final control of the three pilot study questionnaires, collecting missing data for the three pilot municipalities and three core services, progressively extend the number of participating municipalities and the number of services, collect and proof-check the data, manage the data base, analyze the data, assess efficiency degree of each municipality and each service, determine the improvement potential, disseminate information, and show municipalities how to adopt better business practices and reduce costs. To assess efficiency in municipal service provision, the task group compare an efficiency indicator for a particular municipality to the same indicator for the best performing municipality (or an average of best performing municipalities). A comparison to international standards would also be helpful. Tetra-Tech external expert has agreed to look for international standards (for the three core services, when needed for the support services). For each service, relevant quality indicators have to be defined (two for each service, as quality has more than one dimension). Examples of quality indicators can be easily found in the international literature. GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 10 Every year, the task group will draw up a report that provides, for the mayor and the service managers, efficiency and quality of service indicators, as well as an estimate of improvement potential. The report will allow for comparison between Georgian municipalities, as well as comparison between Georgian municipalities and international standards. 9.3 HORIZONTAL EXTENSION: INCLUSION OF NEW MUNICIPALITIES (SEE APPENDIX III) Convince ten to twelve new municipalities to participate in the survey. Inform participants of the survey objectives and of the implementation procedure. With fifteen units, it will be possible to apply Data envelopment analysis (DEA) to measure the degree of efficiency and to identify the best performing units (benchmarks). Choose the reference year for the first wave of the survey. It can be 2014 (to be able to use the data provided by the three pilot municipalities) or 2015 (probably the best option). Before sending the questionnaires, call the contact persons and inform them about the modalities of the survey and about the questionnaire. Tell the contact person that she or he will be contacted again a few days after receiving the questionnaire (to ensure proper understanding of the questions). Then send the questionnaires: set a reasonable deadline for returning the questionnaire (about three weeks). Once the completed questionnaires have been returned, determine if data are missing, proof-check all data (for plausibility). When needed, call the contact person. Send necessary reminders. Prepare the data base (automatic calculation of indicators). Use efficiency indicators or DEA, identify the best-in-class (benchmark): data provided by the benchmark(s) have to be checked rigorously (as it becomes the metrics). Measure actual performance of each unit against the benchmark (or an international standard). Publish and disseminate results: all the main results must be published and provided to municipal authorities and to the management of the service. Change grant design to promote yardstick competition (see a) Promoting competition between municipalities). 9.4 VERTICAL EXTENSION: INCLUSION OF NEW SERVICES (SEE APPENDIX III) The scope of the survey has to be progressively extended horizontally to include all the main municipal services. We recommend a stepwise approach: two new services could be added each year (for example, one core service and one support service). Criteria: selected core service represent a significant part of the municipal workforce and of expenditures, homogenous output across municipalities. The task group devote the necessary time to acquire a good knowledge of the service, to understand its technology, the way to measure inputs and outputs, quality indicators and factor not under the control of the management influencing service provision and costs. Development of the survey questionnaire: we recommend measurING resources use (inputs) in physical units (employees, equipment capacity….) and in monetary units (expenditures in GEL). It is always useful to know the detailed composition of expenditures (personnel expenditures, leasing expenditures, and other expenditure). Existing questionnaire for support services can be used without major changes (it would however be useful to discuss the questionnaires with a person familiar with the service). GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 11 Each questionnaire should be submitted and discussed with a person familiar with the service. Ask the person if factors not under the control of management and quality of services are satisfactorily addressed. Also check the likely availability of data. C. Jeanrenaud/June 2nd, 2016 GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 12 APPENDIX I: MODIFICATION TO THE QUESTIONNAIRES All the changes are highlighted in yellow (see Appendix II) A-1.1 LIBRARY Simplifications: questions regarding full time equivalent employees and permanent and non-permanent employees have been dropped; questions regarding opening hours simplified. Reference period: 2014 (as in the questionnaire sent to the three municipalities) or 2015, I let the choice open. I propose to request the contact persons in the municipalities to give their telephone number. Municipal service or Municipally owned entity under 3. Organization It is central to obtain reliable data about output. For that reason, I propose to ensure that the library knows precisely how many books or other items are lent, and how many people effectively use the services of the library. New questions have thus been introduced. Output: I propose to make a difference between books and periodical borrowed for the reading room or to take home. The reason is that, in some library, users have direct access to books and periodical and the library personnel probably have no record of the number of items taken to the reading room. Active users (another indicator of output): I propose to make a distinction between those who are registered as users or have a registration card and those who are known because they have lent at least one item during the year. Expenditures: in the questionnaire, we asked two times total current expenditures; I kept it only one time. A-1.2 PRE-SCHOOL Simplifications: questions regarding full time equivalent and specialized classes dropped; enrolment rate, difficult to understand and not necessary to measure efficiency, dropped; number of pre-school age children attending a private kindergarten dropped (never got a response, municipality probably don’t know). The proper term in Georgian language has to be used to translate teacher. A new section is introduced for class size. I propose to request the contact persons in the municipalities to give their telephone number. We propose to always ask the total when there are several components (we have noticed that the sum of the parts is sometimes different of the total, which is a sign of a problem). Expenditures: we propose to separate expenditures for teachers (and assistant teaches) from expenditures for “specialized teachers” (such as music teachers). GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 13 A-1.3 WASTE COLLECTION Simplification: questions regarding full time equivalent employees and permanent and non-permanent employees have been dropped. Question regarding container overflow and spilled trash reformulated (was confusing). A question have been added to know if the deserved population is different from the municipal population. The wording of the question has to be checked, eventually reformulated by a person familiar with the institutional context. To be able to crosscheck the responses regarding the volume of waste collected, we propose to estimate the volume in two distinct ways: by asking the volume of waste in m3 dumped to the landfill, and by asking the number of truck trip to the landfill (with compaction mechanism and without compacting mechanism). Containers: I realized that some municipalities have different types (and size) of containers. I propose to first ask the municipalities to list the different type (size) of containers. In a second question, to ask the municipalities to estimate the average size of all neighborhood containers. Cleaning and discharge have to be translated using the proper term (was apparently a problem in the first questionnaire). We need to know the capacity of trucks, expressed in m3 non-compacted waste, to make comparison possible. For that reason, we need to know, for trucks with a compaction mechanism, the volume in m3 (compacted) and the compaction rate (see the new table under 6. Technology). To simplify, we suppressed some question related to the characteristics of the trucks (front/rear loading) CJ/29.9.2016 GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 14 APPENDIX II: REVISED QUESTIONNAIRES A-2.1 WASTE COLLECTION QUESTIONNAIRE Revised version April 29, 2016 Please complete the questionnaire electronically when possible, and send it to the following e-mail address xxxxxxxxxxxxxxxxx no later than xxxxxxxxxxxxxxx If you have any question, please call xxxxx or send a mail Required data and information concern year [year], unless otherwise specified 1. Person completing the questionnaire First name, surname, function, electronic address, and telephone number of the person filing out the questionnaire: First name: Surname: Function: E-mail: Tel.: 2. Organization Is household waste collected by The municipality ☐ A municipally-owned corporation/organization ☐ A subcontractor ☐ Is the unit collecting household waste also delivering another services, such as street cleaning? yes ☐ no ☐ If yes, what is (are) the additional service(s): Days of operation (per week, up to 7 if there is a collect on Sunday): Is the units collecting waste for villages which are not part of the municipality of [name]? Yes ☐ no ☐ GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 15 If yes, estimated population of the villages concerned: 3. Type of waste collected Only household waste ☐ Household and commercial waste ☐ Other type of waste ☐ If other type of waste collected by the unit, please specify: ……………………. Any waste separation program in place? yes ☐ no ☐ 4. Volume of household waste collected Number of trucks trips to the dumping site per month, trucks with no compaction mechanism Number of trucks trips to the dumping site per month, trucks with compaction mechanism Average capacity of trucks with no compaction mechanism in m3 Average capacity of trucks with compaction mechanism in m3 of compacted waste Average capacity of trucks with compaction mechanism in tons Estimate volume of waste transported to the landfill per year, in m3 5. Technology: Process More than one response possible Collection at drop-off points (containers) ☐ Door to door collection (waste not contained) ☐ Door to door or curbside (in bags) ☐ GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 16 If waste is collected in neighborhood containers: Number of containers on municipal area: Capacity in m3 per container (if more than one type, list the different size in m3 or liter): Average capacity of containers in m3: Container overflow and spilled trash How frequently do municipal employees face overflowed containers and spilled trash? Very rarely ☐ Less than 5% of cases ☐ More than 5% but less than 20% ☐ More than 20% of cases ☐ Frequency of container cleaning with water or chemical (per year): 6. Technology: vehicles Number of trucks used to collect waste in operating conditions (end 2014): Number of trucks effectively used to collect waste (end 2014): GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 17 Truck technology (please, put an X in the corresponding box) Fill one line per truck model Compaction ratio: If the volume of 4 m3 of household waste represent 2 m3 in the truck after compaction, the compaction ratio is 2. Thus, an 8 m3 truck can load 16 m3 of (non-compacted) waste Capacity: for compaction trucks, indicates the number of m3 of waste compacted Columns 3, 4 and 8: put an x in the box if the truck has the specified characteristic Model and year Automatic loading Compaction mechanism Number of trucks of this type Capacity in m3 (per truck) Compaction ratio Regularly used to collected household waste Usual staff (1; 1+1 ;1+2;1+3; 1+4….) 1 ☐ ☐ ☐ 2 ☐ ☐ ☐ 3 ☐ ☐ ☐ 4 ☐ ☐ ☐ 5 ☐ ☐ ☐ 6 ☐ ☐ ☐ 7 ☐ ☐ ☐ 8 ☐ ☐ ☐ 9 ☐ ☐ ☐ 10 ☐ ☐ ☐ GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 18 11 ☐ ☐ ☐ Total Add lines if necessary GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 19 7. Expenditures year], GEL Expenditures per year Personnel expenditures : waste collection staff Personnel expenditures: administrative personnel Personnel expenditure: technical staff (repair, maintenance...) Expenditures for consumption (fuel…) and material Leasing expenditures (trucks…..) Other expenditures (eventually amount paid to a contractor) Total 8. Waste collection staff, December [year] Number of employees Waste collection staff Administrative staff Technical staff Total staff 9. Process and workload Average number of hours a garbage trucks is operating weekly (collect and travel to the dumping site): …………9 hours……… Percentage of the households that are not deserved: …………………. 10. Transport Total distance in km to discharge all containers using the shortest route: Road distance to the dumping site, from the center of the city, in km: Are trucks going to the dumping site always with full bin? Yes, always with full bin ☐ Most of the time (>85%) ☐ Less than 85% of the time ☐ GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 20 A-2.2 LIBRARY QUESTIONNAIRE Revised version April 27, 2016 Please complete the questionnaire electronically when possible, and send it to the following e-mail address [……@....] no later than […] If you have any question, please contact [name, surname] per telephone [phone number] or e-mail. Required data and information concern year [year], unless otherwise specified 1. Person completing the questionnaire Municipality: First name, surname, function, electronic address, and telephone number of the person filing out the questionnaire: First name: Surname: Function: E-mail: Tel: 2. Services If service provided, put an X in the box Lending books ☐ Lending multimedia items (CD-Rom, DVD, ….) ☐ Lending E-books ☐ Newspaper, magazines for consultation ☐ Inter Library services* ☐ Electronic catalogue available ☐ Computer facilities for users ☐ Computers with internet access available for users ☐ Staff give time to respond to users’ enquiries ☐ Printing, copying facilities for users ☐ GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 21 * If a user of can borrow books or receive photocopies of documents owned by another library 3. Library organization Is library a municipal service (operated by municipal employees) or a service provided by a municipality owned corporation or organization? Municipal service ☐ Municipally owned organization ☐ Library branches Insert x in appropriate boxes One location only ☐ Multi-branch library (with neighborhood branch) ☐ Specialized library (children,…) ☐ If the library is multi-branch, number of branches: …………. Is registration mandatory to borrow books or other items? yes ☐ no ☐ Does the library keeps a record of the books borrowed? Validity of reader registration: One year ☐ More than one year ☐ Permanent ☐ Are books and periodical borrowed for consultation in the reading room registered? yes ☐ no ☐ 4. Statistics (31.12.2015) Number of books in stocks: Number of periodical, including newspapers and magazines (subscription ongoing): Number of books bought the last three years [year to year]: GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 22 5. Facilities Reading room(s) yes ☐ no ☐ If yes, number of reading places: Number of computers for the users: Have readers directly access to the collection: Yes, all collections ☐ Partial access ☐ No access ☐ 6. Activity [year] Number of books/periodical borrowed (not included those borrowed for use in the reading room): Number of books/periodical borrowed for use in the reading room [if registered]: Number of multimedia items (CD-ROMs, audio CDs, DVDs, Blue-Rays, video-games) borrowed: Have an inventory been carried out in [year]: yes ☐ no ☐ 7. Active users [years] Number of users who have renewed there registration card or membership in [year]: Number of users who have borrowed at least one item in [year]: 8. Opening hours If more than one branch, opening hours for the main branch Reading room Average number of opening hours per day (working days): Number of working days per week: Lending service Average number of opening hours (working days): Number of working days per week: GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 23 9. Library staff Library staff at end of [year] Number of employees Library staff, including administrative employees Technical personal Total 10. Expenditures [year], GEL Municipal library expenditures GEL Personnel expenditures Purchase of goods and services Others Total current expenditures Annual capital expenditure, average over the last three years [year to year]: GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 24 A-2.3 PRE-SCHOOL QUESTIONNAIRE Revised version April 28, 2016 Please complete the questionnaire electronically when possible, and send it to the following e-mail address xxxxxxxxxxxxxxxxx no later than xxxxxxxxxxxxxxx If you have any question, please call xxxxx or send a mail Required data and information concern year [year], unless otherwise specified 1. Person completing the questionnaire Municipality: First name, surname, function, and electronic address of the person filing out the questionnaire: First name: Surname: Function: E-mail: Tel: 2. Organization Is pre-school a municipal service (operated by municipal employees) or a service provided by a municipality owned corporation or organization? Municipal service ☐ Municipally owned organization ☐ 3. Statistics Pre-school age population (children eligible for enrollment in pre-school) [year]? …………… Number of children attending pre-school [year] Age range Number Less than 3 3 less than 4 4 less than 5 5 and more Total Are children from villages around [municipality] attending pre-school in [municipality]? If yes, how many [year]: ……………. GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 25 [To MRDI: The formulation of this (new) question has to be checked; the aim is to measure the deserved population) 4. Pre-school personnel, end of [year] Number Teachers [use the proper term in Georgian language] Assistant teachers [use the proper term in Georgian language] Specialized teachers (music, …..) Technical staff (cleaning, security…) Administrative staff (direction, secretariat…) Specialized personnel (nurses, speech-therapist, psychologists …) Other Total 5. Class size Total number of pre-school classes: Number of classes with more than 20 pupils (21 and over): 6. Expenditures [year], GEL Pre-school expenditures Expenditures for teachers and assistant teachers Expenditures for specialized teachers (music…) Expenditures for technical staff (security, cleaning...) Administrative staff expenditures (direction, secretariat …) Specialized personnel expenditures (speech therapist, nurses, psychologist…) Other current expenditures Current expenditures total Annual capital expenditure, average [last 3 years], GEL : ………………………….. GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 26 APPENDIX III: GOING BEYOND THE PILOT PROJECT: RECOMMENDATIONS ON THE IMPLEMENTATION OF A BENCHMARKING SYSTEM FOR IMPROVING MUNICIPAL SERVICES EFFICIENCY A-3.1 CURRENT STATE OF THE PROJECT The objective is to progressively implement a benchmarking system for Georgian municipalities covering main municipal programs and services. It aims at helping municipalities develop a continuous system of improvement. Performance measures can help reveal inefficiencies, encourage municipalities to adopt good practices and to provide services of good quality, and avoid excessive costs. A pilot study involving Akhaltsikhe, Batumi and Kutaisi was initiated at the request of Ministry of Infrastructure and Regional Development (MRDI) with the technical support of Tetra-Tech, an USAID contractor. The works began in November 2015. Once the strategic objectives had been defined, three core services (pre-school, library and waste collection) and four support services (finance, human resources, ITC and fleet service) were selected. The questionnaires for data collection were sent out at the end of 2015. Completing the questionnaires took more time that was expected, and the last two questionnaires were received beginning of April. Data collection has not been easy, as Georgian municipalities are not used collecting input and output data. Some data are still missing, and the data provided is of uneven quality. The difficulties experienced in data collection were not unexpected. First, it is first tentative survey to measure municipal services efficiency, secondly a personnel contact with the person filling the questionnaire is essential to a successful survey (and there was no such contact). The person in charge of the survey should call the municipal employee filling the questionnaire, before sending the questionnaire, again once the questionnaire has been received, to ensure that the questions are well understood, finally after the reception of the completed questionnaire if data are missing or seems not plausible. We learned from the pilot study that some questions were confusing or ambiguous, and that some other questions could be dropped. A new version of the questionnaires for the three core services has therefore been prepared. One must be aware that the questionnaires have never been discussed with a person familiar with the service (head of service, for example). The wording of a question is very important, and translation a difficult exercise: it need to be ensure that the questions in the Georgian version of the survey are unambiguous. Data from the three pilot municipalities are still incomplete (highlighted in yellow in the data base), some are probably inaccurate (suspicious data in cyan, such as 19.5 employees per waste collecting truck in Kutaisi). A-3.2 ORGANIZATION • Tetra Tech is forwarding the technical dossier to the Ministry of Infrastructure and Regional Development (MRDI), which is taking over the day-to-day operational management of the project: GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 27 final adjustments of the questionnaires, completing the data base for the three pilot municipalities and the three core services, horizontal (new municipalities) and vertical (new services) extension of the survey, training and coaching municipalities once the results are available. • MRDI set up a task group whose mission is to develop an efficiency measurement system for Georgian municipalities based on the pilot project. A group of three on a half-time basis (equivalent to 1.5 full time) will be able to implement the project during the first year. Once the project has attained its full rhythm of implementation, the workload might be moderately reduced (3 x 0.33 = one full time equivalent). A-3.3 MISSION OF THE TASK GROUP The task group main assignments are: final control of the three questionnaire, collecting missing data for the three pilot municipalities and three core services, progressively extend the number of participating municipalities and the number of services, collect and proof-check the data, manage the data base, analyze the data, assess efficiency degree of each municipality and each service, determine the improvement potential et disseminate information, show municipalities how to adopt better business practices and reduce costs. An alternative option would be to entrust the National Statistics Office of Georgia with the data collection and the management of the data base. MRDI or National Statistics Office need a legal basis for collecting the data. This issue has to be clarified. As a first step (first year), we recommend not to add new services and to extend the three core services to other municipalities. Adding twelve new municipalities to the three pilot units would allow to make comparison between fifteen units, enough to obtain meaningful results. In a second step (year 2), new services could be added (for example, one core service and one support service). The questionnaires for the support services in the pilot study are almost final and do not need much change (should however be discussed with a person familiar with the service). There is no need to collect data for each service every year. A rotating survey – each service being surveyed once every five years – largely fits the needs. Municipalities will thus have time to identify and adopt best practices. When the main services have been surveyed, MRDI would have to collect the data for a few services every year (3 if the total is 15). To assess efficiency in municipal service provision, the task group compare efficiency indicator for a particular municipality to the same indicator for the best performing municipality (or an average of best performing municipalities). A comparison to international standards will also be helpful. Tetra-Tech external expert has agreed to look for international standards (for the three core services, when needed for the support services). For each services, relevant quality indicators are need (2 for each services, as quality has several dimension). The selected quality indicators should be submitted to the managers of the services, this would improve the acceptability of the measurement system. Examples of quality indicators can be easily found in the international literature. Every year, the task group will draw up a report that provides, for the mayor and the person responsible for the service, efficiency and quality of service indicators, as well as an estimate of the potential of improvement. The report will allow for comparison between Georgian municipalities, as well as comparison between Georgian municipalities and international standards. GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 28 A-3.4 TO KEEP IN MIND The key factor of success is the quality of data. The method being based on comparison, the data constitutes the performance metric. If there is a bias in the performance metric, the whole exercise become meaningless. That is why using several measure of inputs (physical units and expenditures) and output (tons, cubic meter, population) is a useful precaution. This means also that it is important to proof-check the data when receiving the questionnaire. Among factor influencing output and resource use, those under the control of management are to be distinguished from those not under its control. Example of factor under the control of management: size of the containers used to collect waste. Example of factor not under the control of management: waste collection is easier if the density of waste is higher (m3 per km), it is more difficult if the streets of the city are often congested with traffic. If a municipality spends more than another one to collect one ton of waste because the organization of the service, it means that the municipality is not (fully) efficient. If it spends more because of congested road, one should not draw this conclusion. It is not always easy to decide if a factor is under control or not: are vehicles blocking the access to containers a factor under the control of the management (I would say yes, as the municipality could adopt regulatory measure to avoid cars being parked in front of containers). Considering pre-school, the characteristics of the population is the main factor not under management control (smaller classes are needed to achieve the same outcome when pupils belongs to a socially disadvantaged population). To our knowledge, there are no meaningful (external) factor influencing service provision of a library. A-3.4.1 WORK TO BE UNDERTAKEN (MRDI AND TASK GROUP) a) In the immediate future (MRDI) Clarify the legal basis regarding data collection: to facilitate data collection, there must be a legal obligation for the municipalities to provide data to MRDI or at least a strong incentive. If that would be helpful, the Tetra-Tech expert can do some research on the solution adopted in Switzerland and in some other European countries. Constitute a task group with an adequate time budget. b) In the immediate future (task group) Discuss each of the three core service questionnaires with a person familiar with the service (head of service or person with responsibilities in the service provision. This has not yet be done). If needed, make the required (small) changes in the questionnaire. There is no need to contact a person in one of the three pilot municipality, a person familiar with the service in Tbilisi would do the job. Submit for discussion (same persons): • Factors influencing service provision not under the control of management (at that stage, only waste collection). One possible indicator is the daily (or weekly) volume of waste in m3 per km of road (total volume of waste collected divided by the length of the trip to deserve all containers in the city, routing must be optimized); • Choice of quality indicators (to get a clearer idea about the availability of the data). • Population deserved if different from the "usual population" of the city (population deserved may vary from service to service). Introduce questions to collect indicators to assess the greater or lesser difficulty of service provision (factor not under control of the management) if not yet in the questionnaire. GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 29 c) Introduction new municipalities in the survey (horizontal extension) 1. First step is to convince ten to twelve new municipalities to participate in the survey. Inform participants of the survey objectives and of the implementation procedure. 2. Ask municipalities to appoint a contact person (one per service, head of service or person familiar with the service). 3. Choose the reference year for the first wave of the survey. It can be 2014 (to be able to use the data provided by the three pilot municipalities) or 2015 (probably the best option). If the first reference year is 2015, there are two options for the three pilot municipalities: use data which have already been collected (2014) and adjust monetary values for price increase, or collect data for 2015. I recommend the second option. 4. Before sending the questionnaires, call the contact persons and inform them about the modalities of the survey and about the questionnaire. Tell the contact person that she will contacted after the receipt of the questionnaire (to ensure proper understanding of the questions). 5. Send the questionnaires: set a deadline for returning the questionnaire (about three weeks). 6. Once the completed questionnaires are received, determine if data are missing, proof-check all data (for plausibility). When needed, call the contact person. Send necessary reminders. 7. Prepare the data base (automatic calculation of indicators). 8. Identify the best-in-class (benchmark): data provided by the benchmark have to be checked rigorously. 9. For each service, the data base should provide information on national benchmarks as well as on international benchmarks (or standards). 10. Publishing and disseminating results: all the main results must be published and provided to municipal authorities and to the management of the service. d) Adding new services 1. Choice of the new services to be investigated: two services per year (for example, one core service and one support service). Criteria: selected core service represent a significant part of the municipal workforce and expenditure, homogenous output across municipalities (culture would not be a good choice). 2. The task group devotes the necessary time to acquire a good knowledge of the service, to understand its technology, the way to measure inputs and outputs, quality indicators and factor not under the control of the management influencing service provision and costs. 3. Development of the survey questionnaire: we recommend to measure resources use (inputs) in physical units (employees, equipment capacity….) and in monetary units (expenditures in GEL). It is always useful to know the detailed composition of expenditures (personnel expenditures, leasing expenditures, and other expenditure). The existing questionnaire for support services (finance, human resources, ITC and fleet) can be used without major changes (it would be useful to discuss the questionnaires with a person familiar with the service). Do not write the questionnaire (core service) before having a good understanding of the service (technology, resources, output, quality measurement). 4. Each questionnaire should be submitted and discussed with a person familiar with the service. Ask the person if factors not under the control of management are satisfactorily addressed. Also check the likely availability of data. e) Disseminating information, training and coaching The MRDI task group help municipalities to interpret and use the survey results, encourage them to compare their own performances to those of best performing municipalities and to international standards. GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 30 The MRDI task group also provide training and coaching in process analysis and benchmarking. It should explain how to use the survey results to improve efficiency and service quality, using process reengineering and structural (regulatory) measures. The dissemination of the survey results has to reach all users, not only the management of the service (the mayor, municipal executive and council). Improving efficiency often requires painful and unpopular corrective actions that politician are reluctant to take, such as reducing the number of jobs. Governments do everything possible to avoid taking unpopular measures. Simply showing municipal governments that the cost of service provision could be reduced without affecting users' benefits will usually lead to no change in the way services are delivered. There are a multitude of examples showing that governments can very well live with an oversized and inefficient administration. Municipal governments will take tough action only if they have no other choice (serious financial distress) or if there is a strong incentive for change (intergovernmental transfers designed to promote efficiency). It would be worth thinking about how to bring underperforming municipalities to take action to reduce costs and improve service quality: give municipality sufficient time to adapt (if, for example, the workforce has to be reduced by 20%, this could be achieved not immediately but over five years, using natural turnover), share the benefits of cost reduction with municipalities (part of the money spared can be invested in new municipal projects)... C. Jeanrenaud/May 16, 2016 GOOD GOVERNANCE INITIATIVE (GGI) IN GEORGIA – BENCHMARKING GEORGIAN MUNICIPALITIES 31 U.S. Agency for International Development Good Governance Initiative (GGI) in Georgia 11, George Balanchine Street Tbilisi, 0131 Georgia Tel: 995-032-254-4000 www.usaid.gov/georgia