4. GENERAL EVALUATION OF THE PROGRAMME

The evaluation of the programme is based on 22 awarded projects. The programme represents large scale and comprehensive attempts to support improvements and capacity- building on several levels of government in CEE, but it is also an important tool in enhancing the capacities of participating NISPAcee and NASPAA partners. Taking into the account its focus and dimensions, both positive outcomes could be predicted, but it must be recognized that problems should occur during its realization. In Part 4.1, we analyze project implementation processes – to express main lesson to be learned in future. In Part 4. 2 we present the most important outcomes and some potential impacts that resulted from the programme.

4.1  Project implementations: evaluation and lessons to be learned

The success of the programme’s technical implementation can be measured by three indicators:

A) The level of fulfillment of the planned objectives or how the respective projects achieved their defined goals.

B) The quality of cooperation between partners and with government representatives.

C) The compliance to the time schedule: Was the sub-project realized according to the planned timeframe?

Level of achievement of planned goals

From the point of view of the programme as the whole, we have to stress that the level of achievement for the planned goals was very high and, in some cases, the projects delivered more than planned. Most projects were rated by NASPAA and NISPAcee as fulfilled; only one project as partly fulfilled. The exact data are provided by the Table 1, though some subjectivity in evaluating results cannot be avoided. “Best” projects will be stressed in Part 4. 2.

Table 1 Level of achievement of planned goals by projects

In evaluating the quality of cooperation between partners, the situation becomes a bit worthier. Two kinds of problems occurred (as expectable):

1. Limited cooperation between the project team and government representatives.

2. Limited cooperation between project partners.

Problems concerning cooperation between the project team and government representatives occurred almost entirely in  the “eastern” part of the CEE region; that is,  in countries where public administration systems are less developed than those of other new European Union members. This situation could be described as the outcome of old  and persistent administrative cultures, still existing in some regions of countries like Russia and Ukraine, where the necessity of open government and cooperation with academic institutions is not as important value as it should be.

Problems connected with the cooperation of partners cannot be necessarily connected with their respective regions. Perhaps surprisingly, the most visible case happened in Slovakia, one of more developed countries. Project reports list more reasons for cooperation problems between partners  such as limited time availability, illness, lack of funding (connected with a decreasing exchange rate of USD), changing position of partner, etc. Our opinion is that the scale of cooperation problems was not too large, given the size of the programme (see Table 2). We do not think it makes sense to evaluate reasons for such problems; it is to be expected that the evaluation favors the person or organization that prepared the report.

Table 2 Quality of cooperation between partners

Some projects may occur in column 2 and 3

The highest level of difference between plans and final results was achieved when evaluating timelines of respective projects (Table 3). However, this feature shall not be overestimated as some delays are typical of almost all projects in the academic environment. The most important is that time pressures did not influence the final achievement of planned goals in any case. And in some cases, the partners delivered planned outputs at their own cost as the project finances were not available after a certain time.

Table 3 Timelines of projects

The last issue to be mentioned here and probably the most important is finance, specifically the financial management of the programme. The system of financial management changed during the realization of the programme, as a reaction to the problems (time, achievement of goals, and quality of cooperation) of all three types previously mentioned.

During the first phase of the project, team members were paid one-half at the project’s mid-point, and one-half at the project’s conclusion. This policy was later changed, and no honoraria were provided before evaluation and approval of the project’s achievements. This change in the financial management system of the programme influenced the rate of success in a significant way; it also increased discipline of all partners involved, most specifically that of the responsible institution. This stricter system of financing is recommended for the future.

Another financial problem of the programme was the consequences of the significant decrease of exchange rate of USD within the programme duration, a decrease of almost 40 percent. Many projects were influenced, and some partial, non-core activities cancelled, though the currencies of most participating countries are linked to EUR and not to USD. It is difficult to advise on this issue. There is some chance of obtaining insurance covering risks of exchange rate fluctuations, but such mechanisms are difficult to implement in programmes of our type.

In conclusion, we wish to emphasize that from the point of view of programme implementation it was successful. No major problems occurred in the degree of quality of cooperation between previously unknown partners, in cooperation between academic institutions and government, and in following time schedules.

4.2 Most important outputs, outcomes and impacts of the programme

As already indicated, projects achieved planned goals and delivered planned outputs. From the point of view of outcomes, all reports stress that the respective projects increased the institutional capacity of project partners, and especially of the responsible institution (main goals in the TA projects; one of the goals of the PR projects).

The most visible and positive case of strengthening the institutional capacity of participants in TA part of the programme was project 1TA, realized by the Ukrainian Academy of Public Administration. The results of the project focusing on improvements in the MPA programme helped prepare the Academy for the process of international accreditation. In June 2004, UAPA was awarded status as the first CEE institution by accreditation. This was provided by the European Associating of Public Administration Accreditation (EAPAA). Europeanization and modernization of MPA programmes at UAPA might have important impacts on the quality of education and the training of public officials in the Ukraine—and also in surrounding countries—thus improving the public administration system in the country.

The common goal of the TA and PR projects was the development of cooperative relations between academic and training institutions within Europe, and between Europe and the USA. Almost all project reports that created or updated cooperative relations did not stop on the day of submission of the final report, but are expected to continue in future.

A significant issue of the PR projects was the improvement of links between the academic and training institutions of public administration and the government. In spite of some aforementioned problems, many barriers have been broken. In some cases, genuinely cooperative systematic relations between respective levels of government and the institutions were built; these are also expected to continue in the future. Three examples might be mentioned.

Example 1:

2-1PR: Cooperative Governance Improvement: Developing an Academic Advisor’s Practice for Politicians and Applied Policy Research: National University Centre “Public Sector Economics”, Bulgaria

The purpose was to train members of Parliament, a dream even for some developed countries. The task of trainers was rather uneasy one—to combine an introductory course with refresher knowledge as well as to systemise the practical knowledge of the members of the Parliament and to link it with modern economic science. As a sequence in the breaks and after the classes, intensive and vivid discussions took place. The training course achieved its results and was beneficial in bringing modern economic knowledge to both embers of Parliament and their advisors.

From July 2003 to January 2004 (excluding parliamentary vacations in August and December), the trainers and members of Parliament were constantly in touch–either in form of lectures or of seminars and parliamentary consultations. Two seminars, in July and November, were organised. The topics—“Economic crises in Bulgaria 1996-1997” and “New Management in the Public Sector”—were connected with adoption of the state budget for 2004.

Another target, Academic Consultation, was achieved. This took the structure of informal consultations in the Parliament with professors of the training team. In addition, the professors were present at themeetingsof the Budget and Economic Policy Commissions for ad hoc consultations or to answer questions during the discussions between members of Parliament and members of government for the adoption of the state budget. 

As suggested by the members of Parliament and their advisors, another tangible output, a book with teaching materials, was printed. At their request, the book includes not only topics from the training course, but also topics covered in the seminars and discussed during the project.

Example 2:

4PR: Improvement of regional and local health systems through attraction of critical mass of health managers to the policy-making process, Odessa Regional Institute of Public Administration, Ukrainian Academy of Public Administration, Office of the President of Ukraine

The project has had the important consequence of bringing the senior staff of ORIPA in to working in very close contact with senior Oblast’s officials in developing both general ideas about and a consensus for the need to reform the Oblast health system that follows national program for regional development. The policy paper was considered at the Oblast Health Agency meeting. As a result, the proposal to examine in depth relevant mechanisms of implementation of the two most important goals of the “Program for medicine service system reforming in the South Ukraine Region” was approved.

The project realization tends to make the work of ORIPA both more meaningful and practical. This, in turn, increases our influence in the shaping of public policy and our ability to provide our students with the best possible education. 

Partially as a consequence of the issues raised in discussion of the NASPAA/NISPAcee policy paper and other concerns, the Odessa Oblast government has appointed a Coordinating Council on Implementation of the Program of Reforming Medical Service in the Odessa Oblast which includes, among its membership, virtually all of the key officials concerned with health policy reform in the Oblast, including Dr. Alexander Korvetsky, Dean of Public Administration at ORIPA. 

Taking this into the account, the project was extended and financed by an extra $25,000 USD which continued cooperation and supported the growth of quality of regional public administration, specifically health care administration.

Example 3

3-2PR: Strengthening capacity-building of modern methods of budgeting, audit and control system at regional self-governmental level, University of Matej Bel, Faculty of Economics Banska Bystrica, Department of Public Economics, Slovakia

The project significantly helped improve contacts between academics and practitioners in the area of audits (a weak point of transitional public administrations systems). It also advanced contacts between schools in Slovakia and the Czech Republic dealing with the issue. New cooperation began with the potential of extensive influence and impacts on quality of public audits in respective countries

The University of Matej Bel, Faculty of Economics, department of Public Economics will benefit from this program through significant strengthening of its teaching and advisory capacities in the area of budgeting, control and audit in the public sector. The course on financial control, created as the result of the project and supported by regional self-government practitioners, was already introduced into the curricula.

Thanks to successful cooperation with regional self-governments, there is considerable potential for major impacts as the project continues into the future. Two forms are envisaged. One is training and lecturing for the purposes of the regional self-government. The preparation and realisation of these training courses will be realised in cooperation with regional self-governmental districts who originated the request for this manual preparation, and also with all Czech partners. Developed training curricula will be used also in the Czech Republic, and experts from both countries will exchange their capacities in those and during the delivery of courses. The second line is continuing in the further development of manuals, especially in the preparation of working guidelines for the concrete activities of regional auditors.