INTEGRAL E-GOVERNMENT DEVELOPMENT INDICATORS

dr. Mirko Vintar, Mitja Dečman, Mateja Kunstelj, Boštjan Berčič
1.	Introduction
E-Government development has become one of the most discussed topics among the practitioners and academics involved with development of public administration today. Goals are strictly set in most countries and different strategies are in full implementation. However, after one or two years under way many countries are confronted with very slow progress, development lagging far behind officially defined and announced timetables and agendas. Some of the countries and their governments even did not realise yet that they are lagging behind. One of the main reasons for this kind of ‘ignorance’ lays in absence of reliable and transparent development indicators and measuring tools, which would enable individual countries for:
•	concurrent tracking and evaluation of the development of e-government in the country and quantitative and qualitative comparisons with the goals set;
•	comparative evaluation and benchmarking of the results on the international scale.
In EU and in the world several activities and projects started recently in order to develop different more or less formal benchmarking methods. One of them, considering Europe and candidate countries, is a part of eEurope programme. Now already famous model of 20 selected services has been suggested and methodology for measuring progress in individual countries, based on those 20 services, has been put forward. But in accordance with our own experience this was only the first step. If we want to get more realistic picture and the integral perspective of one country's e-government progress and the level achieved, the suggested EU model will have to be broadened considerably and other indicators will have to be included in the benchmarking as well.

In the paper (which is a research in progress) we will try to analyse the so far proposed benchmarking models and describe their strengths and weaknesses. Further on we will purpose and describe an advanced model of e-Government development benchmarking indicators including the methodology for determining or evaluating them. The model that will be described is under development in Slovenia and results from our research work in the field for the last two years.

We want to present an advanced model of indicators and complementary methodology that enables measurement of e-government development of one country and qualitative and quantitative comparison with the other countries. The model is based on eEurope benchmarking, but it upgrades it in several dimensions, because the benchmarking of e-government in eEurope model does not give objective result of e-government level in one country and realistic comparison with the other countries.

The proposed model and corresponding methodology will cover three basic blocks of possible e-government development evaluation:
•	back-office development evaluation;
•	front-office development evaluation;
•	evaluation of the maturity of the environment.
It will also review it from supply and demand and point of view.

The advanced model takes into account information, communication and transaction services and concentrates on e-government services for the public. The model is still under development and might be used by Slovenian government and public administration in the first place. The methodology for defining indicators will be developed in the way that frequent measurements will be possible.
2.	Analysis of the existing benchmarking models
In this section we will survey existing benchmarking models against indicators of our own model which we are developing and which is presented elsewhere in this paper.
2.1.	Short description of the characteristics of surveyed models
We look at existing benchmarking models from two viewpoints: one is methodological, concerning how surveys were carried out, while the second one (contents) refers to what the surveys were measuring. We looked at the number of indicators in different indicator groups. We checked every benchmarking model we looked at for the presence or absence of particular indicator in particular group of indicators. All in all, we take into account 35 different benchmarking models - including our own. Matrix resulting out of this (with groups of indicators and indicators themselves and different models) is presented in Table 1.

Within methodological point of view, we take into account following groups of indicators: who is the author of the survey, what geographical area the survey refers to, what part of e-government (with respect to different clients) was examined, which technologies were taken into consideration, how the measurement itself was carried out and which user perspective tended to be the most important in the survey. Content wise, we divide groups of indicators into supply (referring to suppliers of e-services), demand (referring to users of e-services) and the overall maturity of the environment (see matrix in Table 1).

** Table 1**

Table 1: Overview of e-government models analysis
2.2.	In-depth survey
From methodological point of view, these benchmarking models were to significant extent authored by different (private) researchers, as well as official bodies of different countries. Among countries, most prominently represented are USA, followed by EU and others. Some international organizations such as UNO, PACE and others are also represented, but to much lesser extent. We also thoroughly examined our own environment and included benchmarking models developed in Slovenia.

From the geographical point of view, most models take into account USA as a country of survey, which reflects quite high level of sophistication of e-government and e-services in that country. Europe was second most surveyed area, probably reflecting the fact the some European states put up quite good e-services, second in sophistication only to USA. Slovenian state also figured strong in our survey of benchmarking models, which reflects our decision to take into account as many domestic benchmarking models as possible. Most benchmarking models focus solely on one state, but some of them survey more, thus giving a comparative picture across countries of e-government and e-services provided, based on one methodology. More than one benchmarking model deals also with Australia and Singapore.

With respect to group of indicators which indicate direction of relationship between e-government and different user groups, most benchmarking models took into account G2C relationship. This reflects a general perception that e-government's main focus is citizens and services for them. Second most represented relationship is that of G2B, reflecting importance of e- government services for private sphere (e-procurement etc). Other areas (C2G, G2G) figured less often in surveyed models. It is also true for this group of indicators that some benchmarking models took into account more aspects (relationships), while others focused on one relationship only (mostly G2C).

In the group of indicators representing information and telecommunication technology we see that almost all models put greatest weight on internet. They were much less concerned with infomats, mobile phones, cable TVs, teletext and other platforms for e-service delivery. Again, most of the models surveyed focused solely on internet while a few ones tackled more platforms for e-service delivery.

From the view of indicators regarding data collection, most used method was exploration of internet pages (mostly for e-services suppliers, which reflects the fact that most surveys were done in G2C area). Second most used method was phone conversation (mostly with users, such as citizens and businesses, but also with providers of e-services, such as civil servants), which also reflects the fact that G2C segment of e-government is very important. Interviews were also quite frequent. Less frequent were opinion polls, workshops etc. Surprisingly enough, very few benchmarking models decided to use e-mails, e-mail polls or internet questionnaires to collect data. They would be an obvious choice from the technological point of view, but they evidently don't provide enough quality (and quantity) of information to be more broadly used. Most of benchmarking models used one of the above methods for data collection, only very few relied on more than one means of collecting/processing data.

Looking at indicators for groups of subjects from whom data was collected, group of civil servants steps out from other groups. They are by far the most surveyed group among all. This probably reflects the fact, that surveyors found it most easy to talk with people which are directly in charge of various e-government projects, thus sidelining end-users. Although these (and the businesses) are also present among groups that were interviewed, questioned or polled they altogether do not reach the proportions of civil servants that were surveyed. This represents a potential shortcoming of existing benchmarking models and also weaker potential relevance of conclusions for the demand side of e-government, which is harder to measure and was thus gauged to great extent indirectly through responses and judgements of civil servants. One group, that also figures prominently among questioned subjects, is a group of researchers themselves viz. their opinions. This could also present danger to the objectiveness of the benchmarking results as not the end-users but researchers themselves judge quality of the services provided. This is of course true only for those surveys that rely only or predominantly on such sources, while it doesn't hold for other, more balanced models, because they take into account more than just one group of subjects. Most models indeed take into consideration more than just one group.

Within the contents focus, the first group of indicators refers to supply of e-government services. Some models measure the existence of technological infrastructure itself, while far less measure the actual usage of this infrastructure, which could also represent a considerable shortcoming. As far as the web pages of e-service deliverers are concerned, most benchmarking models concentrate on contents and characteristics of such portals, and a few less on mere existence of such pages. Some also measure sophistication of e-services. Almost none measure maturity of life-events found on these portals (life-events help citizens in finding out which services they need and in what processes to achieve some desirable state of affairs, e.g. get a job permission etc.). Life-event portals are becoming an important type of entry points for e-services, but since the idea of life-events is pretty new, it doesn't come as a surprise, that their sophistication can hardly be measured.

Demand for e-government services is measured through different indicators. Here we have again the mere existence of needed infrastructure and its actual use. Again, we can conclude that models that do touch upon this subject prevalently opt for existence, much less for actual use of such infrastructure. Other indicators in this group measure users' interest to use e-services, as well as the use itself of these services. Only small minority of benchmarking models measure quality of services (again reflecting the fact that e-services themselves are in the early stages of development). Less still measure wishes of users (reflecting the fact that e-services in this early stages are technology, not user driven) and none measure the desire to use life-event services (also reflecting the fact that life-events as such have not yet established themselves in the heads of the users as a potential vehicle for interacting with government and having delivered government services online)

 Last group of indicators surveys general maturity of the environment. This group represents some very sophisticated and advanced indicators of e-government sophistication that are tackled only by some models. Majority of those that do tackle them are concerned with obstacles to introduction of e-services. Quite a few of them also measure whether there are strategic and action plans for introduction and betterment of e-government. Others, such as advantages of e- government, motives for introduction of e- government and others are much less frequent.
2.3.	Conclusions
At the end, we can conclude that these benchmarking models are good in general, but more stress needs to be placed on user and on demand side of e-services delivery process. Methodologies for measuring e- government sophistication should also focus more on advanced services, such as life-events, which are yet to be fully appreciated and evaluated, and other advanced indicators such as obstacles when introducing e- government etc. We hope that the results of this survey will guide us well in the building of a reliable and holistic model of indicators of e-government.
3.	The analysis of models considering objectives of integrated model
3.1.	Objectives of the integrated model
Integrated e-government index is a complex set of indicators that represents the development state of a country's e-government in the way that corresponds to the objectives of the model. It is a demanding and comprehensive model that might be unfeasible or to expensive to measure as a whole in some countries. The objectives are strictly set since without them the indicators do not reach the needed effect and are practically unusable, because they might not be focused on the elements of e-government that represent the state and development of e-government. Therefore, our model was build to attain the following capabilities:
•	enable measuring of e-government development and be comparative to other models of Europe and the world;
•	including indicators, that can be measured in different countries, where e-government is introduced;
•	enable measurement of supply side of the state and demand side of the citizens and private sector;
•	enable repeatable measurement and evaluation;
•	include analysis and location of critical elements of e-government and determination of reasons for failure or unsuitable level of e-government; the weakest link of the e-government chain makes the whole chain unreliable and disused; nevertheless, if we find the weak link and remove the weakness, e-government can achieve full implementation and maximal efficiency and effectiveness as a whole.
E-government includes electronic execution of processes in public administration on local and state level. The legislative, judicial and executive powers are all included. It is usually partitioned on e-democracy and e-administration. E-democracy includes political information and participation using information technology (IT) and services like e-voting, e-forums, e-debates, etc. However, this is not the primary field of our integrated model. This module is focused mainly on the services of public administration and its users, therefore it is focused on e-administration, but throughout this paper we are using a more general term e-government instead.
3.2.	The pyramid of e-government levels
Considering the aims set, our integrated model is partitioned in different segments of indicators, where each segment represents a closed group. On the other hand, individual indicators in different segments could be linked with each other. The basic segments defined are supply segment of indicators, demand segment and environment maturity segment. The indicators in different segments correlate, e.g. services that are demanded by users and services supplied by e-government. Each segment is then divided to layers and indicators in different layers can be also linked horizontally or vertically. For example, e-payment functionality is offered for many services but users can not use it since e-banking is not trusted among them.

Considering the supply side, e-government is offering its services making them as useful and close to the end-user as possible. These services use different functionalities enabled by content and infrastructure. The end-user is a citizen, a person from a private sector or an employee of the government. Only the end-user oriented e-government can achieve success and serve the whole society, enabling it to become information society. Political or technological development of e-government can prosper only a short period but it is destined to fail. Politics change, technology advances, but citizens stay citizens.

User indicators, referring to individuals i.e. the citizens or private companies, represent the demand side. These indicators measure their needs and desires considering e-government services that are or should be available and the levels of usage of already implemented services of countries' e-government. Considering the levels of the pyramid below (see figure 1), we talk about infrastructure indicators (existence of telephone lines, broadband connections, computers…), availability indicators (attractive prices, accessibility, public access points…), contents (quality, quantities of data, applicability, up-to-date information…) and functionality (search engines, communication services, transaction services…).

** Figure 1 **
Figure 1: The pyramid of e-government indicators

The levels of pyramids are partitioned in the following way.

The basic level is represented by infrastructure, which is the ground element if e-government, since IT is the foundation for e-business. The infrastructure is also the foundation for other levels of pyramid, i.e. other levels of e-government. Indicators at this level are referring to telecommunication technology like networks, interfaces, converters in the field of wire and wireless technology. They include indicators for personal computers, servers, mobile devices… An important segment considers security includes indicators about technology and infrastructure that enables it. Indicators relate to usage of firewalls, smart card technology, public key infrastructure, certification authorities…

The following level is availability and usability since the mere existence of IT does not guarantee these two. Therefore, indicators on this level deal with ability of infrastructure use in different environments and the frequency of this usage. Indicators mainly consider availability of infrastructure elements to citizens, employees in private sector as well as in public administration.

The contents level includes indicators about the data and information available in different electronic formats, mainly the web sites and portals, the usage of it, linking and other content oriented indicators. This level includes indicators that deal with the public administration as a whole and indicators that are specific for individual organisations of public administration (parliament, administrative districts, courts…). The level focuses on different the content offered, i.e. the data available, catalogues, foreign languages, news... The stress is on integrity, comprehension, up-to-date-ness…

The functionality level is the upgrade of the contents level since the use of modern techniques of using IT enables additional services and functionalities for different parts of e-government. Among other things this level includes indicators about structured form of data like databases, XML documents… 
Functionality indicators also refer to the electronic implementation of a certain administrative procedure and the level of communication, data exchange or transaction abilities. These functionalities of different services are measured in different benchmarking models, among others, the eEurope benchmarking of 20 selected basic public services. eEurope model of levels was the foundation of our level partition but we trued to expand it in greater detail. In eEurope model the four levels model is used, where the level 1 – data and content matches our content level of the pyramid. Others are defined as followed:
•	level 2 - one-way interaction is one-way transfer of data, usually toward the user and includes transfer of different documents, forms and other structured data, that exist in public administration;
•	level 3 - two-way interaction is two-way transfer of data or communication that includes indicators that refer to modern IT that supports the exchange of data between users and government (C2G, B2G and vice versa) and administration internally (G2G). At the same time this transfer of data includes exchange between users themselves (C2C) in the administration frame of contents. These functionalities include e-mail, web forums, chats, newsgroups, interactive forms…
•	level 4 - transactional level is the highest level according to eEurope model (not by ours) including services that incorporate user identification, e-payment, implementation of the whole process from application to the end in the electronic form…
These levels are used in eEurope and in a bit expanded way in our model (inside the functionality level), which enables the comparison of indicators and results of measurements between eEurope and our model. However, our model compared to eEurope model further divides these levels and incorporates more, thus enables a more comprehensive overview. Beside the evaluation of 20 selected services, using indicators on other levels and segments, it shows the integral overview of development state of e-government and enables its analysis. If the level of e-income tax service in one country is implemented with transaction level (according to eEurope model 100%), the analysis is probably unnecessary since the maximum is achieved (unless end users are not happy with it). However, if it is implemented on the content level (according to eEurope model 0-25%) additional indicators are needed to ascertain the precise reason for this state (poor infrastructure, service not interesting for users, lack of users' trust…) and these exist in our model.

The last level of our model (not included in eEurope model) is the integration level that incorporates the entire pyramid and represents its top and a final goal. Integrated service truly represents the most useful implementation of a service for the user as "one-stop-shop" service. Only in this way can e-government rely upon the interest of users since it represents an easy and understandable implementation of administration processes in electronic environment, which is for many users today still unknown or "hard to cope with" territory. The so-called "life-events" are measured on this level and as shown in many life-event e-government portals of the world, present an ideal front end type of e-government of the future.

Indicators on each level can be dissected according to the segment or sphere of e-government they specifically relates to. These groups can be divided organisationally or functionally. These dissections can be done in the same way on different levels and therefore enable individual vertical linking. Doing this, specific segments can be thoroughly analysed and dependences between indicators on different levels can be studied. If we for example select or define indicators that relate to parliament, we can study the correlations of indicators across the levels through their vertical linkage (i.e. members do not use e-mail to communicate with the citizens – functionality indicator, since their e-mail addresses are not published on the web site – content indicator).

The demand and supply side indicators can be correlated as well thus enabling the comparison between supply and demand sides, which is one of the goals of this model. Because of the same structure of levels for majority of indicators on both sides, these two segments can be compared. The comparison between the supply side indicators of e-government and the demands side indicators of end-users can be made as well (for example citizens do not use the functionality of electronic payment of car registration, since the have no interest in it – horizontal link, or because they do not have the safe and trusted technology for electronic payment – vertical link)

** Figure 2**
Figure 2: Service (left) and life event (right) representation

Indicators describing individual service can be distributed among more different levels and a service can be shown as vertical slice of the pyramid (see figure 2, left). If for example e-government offers e-income tax, the competent public administration organisations need infrastructure that has to be available to employees, a web server is needed and it has to be available to users. Appropriate content has to be published (procedure information, legal background…) and additional functionalities like e-mail addresses for questions and solving problems, secure user identification and secure payment of income-tax.

Speaking about life events, one can imagine them as integrated services offered in a suitable way, available to the user in one place. If we consider the life event "moving house", life event portal would gain the relevant data needed for different services from the user and according to them produce some kind of output. The services like application for new permanent address, change of identification card, search for schools and kindergartens at new location are hidden from the user. For the whole integrate scheme (see figure 2, right) it is not necessary that all the services included are developed to the same level and each can be formed in its own contents and functionality level.

Beside mentioned segments, our model includes the needed environment maturity that indirectly influences on the services of e-government and is in that way linked to the indicators of other two segments. The environment segment is divided to political, legal and social indicators. While first two consider mainly the supply side, the last one (considering people) relates to supply and demand side. The legal and political aspect can greatly influence and help or hinder the e-government development. Lacks of political support, unsolved legal issues are big issues that represent major problems in many countries implementing e-government. Even political strategies that exist sometimes need further action plans and other initiatives to force e-government development. The human i.e. social factor can influence the development itself, changes its course or speeds or slows it. The social indicators also deal with digital divide issues. Since many of e-government services are offered to citizens, they should be able to use them. Indicators such as feelings about improvements of the processes, lower costs, tax relieves, etc. are measured here. On the public administration employees side indicators measure their readiness for new ways of work, demands on new knowledge, fear of discharge…
3.3.	Comparison of existing researching according to our goals

It can be found out that none of known (abovementioned) researches corresponds wholly to objectives we set. The majority of them are concerned with the measurement of particular narrow fields of e-government (namely, only some of the researches try to cover all key e-government fields and consider factors of environment), and on the other hand none of them combines direct comparison between supply and demand.
Researches (ICMA, 2001), (UNPAN, 2002) and (NAO, 1999) focus mainly on the demand and represent a good base for the forming of key groups of indicators in our model. The benchmarking of 190 UN countries (UNPAN, 2002) defines the so called ‘e-gov index’, the numeric value, which intends to reflect the degree of economic, social and democratic development of particular analysed country. E-government index is the average of three basic indicators: presence on the Web (measured by analysing the web sites), telecommunication infrastructure (measured through indicators of IT equipment of end-users and their internet usage) and human capital (measured with the help of UNDP index of human development, index of information access and population density). Although the methodology for e-gov index calculation is not presented in the whole, many questions about reasonableness of aggregating measurement results into single-number value are rising up. We could ask ourselves if in this way the results are generalized too much and if particular indicators are weighted correspondingly. But, on the other way, from non-aggregated results the general situation is seen with much more difficulty. The solution, which would be suitable for our model is a partial aggregation of particular groups or indicators. For example, some researches show interesting examples of aggregating only web site indicators (NAO, 1999; UNPAN, 2002, Cap Gemini Ernst & Young, 2002; Cullen et all, 2000; West, 2002).
For Slovenia, as future member of European Union, the comparison with other EU countries as well as candidate countries is of great importance. EU already executes many measurements in the field of information society development and in e-government within them (Benchmarking eEurope, 2003). From this point of view, the corresponding existing indicators should be included in the suggested model. Apart from other studies, the measurement of 20 public services on the web (12 for citizens and 8 for businesses) developed within eEurope model (Cap Gemini Ernst & Young, 2002) must be mentioned in this place as one of the few, which measures the degree of e-services development and relates to the contents and functionality level of our suggested model of indicators (Figure 1). Services are measured by means of 4-stage model of e-services on-line sophistication (
Table  2).

** Table 2**

Table 2: The four-stage model of on-line sophistication of e-services (source Cap Gemini Ernst & Young, 2002)

The criticism of the methodology of the above mentioned part of eEurope model has two key elements: definition of sample of public services (which should represent all public services) and the definition of measurement model. Both sides have, in our opinion, some weak points. For example, in the list of 20 services there is the birth and marriage certificates service, which is, in our opinion, unnecessary. The service is in the first place intended mostly for the public servants (and not citizens), which need data from those certificates in performing different other administrative procedures. We are convinced that citizens must not request those data, because government already has them and public servants can gain them by themselves simple accessing corresponding database. Therefore, the countries that have ‘not relevant’ mark at this service have in fact the most developed service and not countries with the service in fourth stage of development. On the other side, the greatest weakness of this four-stage model is the fact that first three stages are relatively well partitioned, while the fourth stage covers too extensive part of service execution. Therefore, it should be divided into smaller parts in order to estimate the stage of service development more accurately. Consequently, adapted four-stage model should be used in our model of indicators so that the results will still be comparable and at the same time more accurate and unambiguous.
As for the fact that recent development of services based on life events is in progress, it is interesting that there are almost no researches covering them. Actually, we found only one such study (see TietoEnator Trigon AB, 2001), but this includes only analysis of best practices in particular countries. As life events are nothing else than integrated services, above mentioned four-stage model should be broadened to enable measuring life events too. First attempt of this broadened model was presented in (Vintar et al, 2002).
Considering methodologies of measurement, it can be seen that for the measurement of such broad spectrum of indicators the combination of different ways of collecting data should be used. In existing studies web analyses by researchers themselves, inquiries and interviews of public servants (particularly when additional clearances or interpretations of results are needed) are most frequently used. It can also be found out that the model of indicators, which tries to be integral as much as possible, is quite extensive and thus from the viewpoint of implementation very demanding. Besides, there are many practically standard studies (for example about IT equipment and internet use by citizens), which are regularly executed in different countries for years (also in Slovenia). Therefore, it is logical to use these particular indicators in our model as well.
4.	Integral e-gov index model

We suspect that at this moment there is no objective and integral benchmarking model, which could be used to objectively estimate the level of e-government development in one country and compare it to others. Existing benchmarking models provide only a partial picture. Therefore, we tried to solve this problem integrally and develop the draft for the model, presented here.

During the definition and selection of indicators we derived from existing benchmarking researches of School of public administration in Slovenia (Vintar et all. 1999, 2000, 2002) and researches made in EU (Cap Gemini Ernst & Young. 2001,2002; Espana 2002, Fundacion Retevision. 2002). We want to keep the model comparable to models used elsewhere (mainly EU and candidate states) and have results comparable with results of our studies from previous years. At the same time we followed the examples of different institutions, research institutes and non-governmental organisations in the world and their findings in this field.
4.1.	Structure of the model
Our integrated model is based on the pyramid structure described above and aimed to the objectives of e-government development benchmarking already mentioned. It is divided in three main segments of indicators, where supply and demand segment are further divided according to the pyramid levels while environment maturity segment contains political, legal and social indicators.

On the supply side, we measure different indicators of e-government services that are offered to the users. In the infrastructure level these are:
•	basic technology, i.e. existence of intranet, local area networks on organisational and country level inside the public administration, IT equipment of working posts, Internet access, mail servers…
•	security, i.e. public key infrastructure, firewalls, smart cards technologies…
On the availability level the indicators refer to measurement of the percentage of working posts that have the available infrastructure and the frequency of its use. Indicators correlate to those from infrastructure level (e.g. indicator existence of Internet access – infrastructure level, percentage of employees using it, frequency of usage – availability level).
The contents level relates to the content that e-government is offering to the user in the electronic form. These are mainly the contents of web sites, accessible through different types of clients (internet browsers on computers, mobile phones, PDAs…). Beside general indicators relating to administration as a whole, there are specific indicators for individual organisations inside public administration. Therefore, these specific groups of indicators are focused on e-government of:
•	parliament;
•	ministries;
•	local communities;
•	government;
•	administrative districts.
The functionality indicators supplement the contents indicators and refer to specific functionalities above the contents. They are not measured individually, instead they upgrade specific contents indicators and they are linked to them and grouped in the same way (e.g. published legal bases for building permit – contents indicator, submitting application for building permit – functionality indicator). There are general indicators and specific ones for individual organisations. According to the comparative quality of the model to the eEurope benchmarking of 20 e-government services, similar but more detailed division and levelling is defined. Since our detailed levels correlate to different eEurope levels, these two are comparable. Functionality is therefore divided to:
•	basic functionalities on transfer of data (searching, saving data, printing data, but sending data (e.g. application) back to e-government in classical paper form) - equals eEurope level 2;
•	additional functionalities on transfer of data (filling electronic forms, sending data by e-mail) - equals eEurope level 2;
•	basic interaction (interactive and web forms) - equals eEurope level 3;
•	advanced interaction (forms with user identification) - equals eEurope level 3;
•	basic transaction (realisation of the process including payment) - equals eEurope level 4;
•	advanced transaction (realisation of the process including payment and electronic issue of resolution) - equals eEurope level 4.
Basic functionalities on transfer of data do not mean the transfer of content itself since this functionality is default functionality in electronic world (e.g. saving a web page or copying to the word processor from the browser). The indicators in this case measure transfer of structured data like forms, documents… The basic functionalities are defined by indicators that test search capabilities on web sites, personalisation and customatisation of showed data and the form of it as well. Additional functionalities of data transfer include indicators measuring capabilities of filling electronic forms and sending data by e-mail. Basic interaction includes indicators measuring existence and capabilities of different forums, discussion groups, chat channels, use of e-mail in the scope of C2G, B2G and vice versa and exchange of data like receiving and sending structured form of data, attaching documents with forms to e-mail, use of interactive forms… Advanced interaction also includes authentication and authorisation of users if necessary. Transaction indicators refer to functionality of services like electronic payment, identification of end user (using digital certificate), e-referendum… Advanced transaction indicators upgrade these and deal with capabilities to conclude the whole process in electronic way (if possible) so that no paper was needed from the beginning to the end of the process.

The last level measured is the level of integration that is missing in eEurope level. These indicators measure integration of different areas and services and in this manner represent the user-friendliest way of offering services in e-government. According to the lately well know word of "life-event" services, integration represents exactly that; the organisation and integration of different services in the frame of life-event. The user is not interested in specific services that he has to accomplish but wants to solve his life problem in one place and get the solution out of the e-government system. The life event solution therefore integrates different solutions of e-government that include contents and functionalities of different organisations in public administration in different places. The long door to door, building to building, city to city travelling is over in this way. Since our model needs to be comparable with eEurope model 20 selected services will be measured as well but ranked according to our defined levels.

The demand side includes indicators that link with the indicators of supply side and enable comparison between these two segments establishing a form of correlation. Even on the demand side, the indicators are grouped in equal levels and groups but concentrate and try to solve the following issues:
•	wishes and expectations of end users;
•	level of applicability;
•	measure the level of satisfaction of end users considering the offer.
The infrastructure level indicators on the supply side measure the number of households with personal computers, access to Internet, the number of internet providers, level of broadband lines, the price of access compared to BDP and other indicators that measure the affect of infrastructure on use of e-government services. The availability level includes indicator dealing with number of internet service providers, public hosts, percentage of users and private companies planning to use e-government services, communicate with employees of public administration, etc. Functionality level measures what functionalities users need or use, which are the most wanted, which are trusted, etc. Users are asked which services they would like to use and how often they use one or more out of 20 selected services (if available).

The indicators of environment maturity are divided on legal, political and social group and partitioned considering internal and external view of e-government as well. The legal indicators are focused on legal acts and frameworks considering e-government, i.e. personal data protection act, acts on the right of access, copyright and IT misuse... Political indicators are dealing with strategies and initiatives, budget funding for e-government and other political factors that influence on the e-government development. Social indicators refer to different social aspects of demand side, i.e. end users, as well as those from the supply side, i.e. employees from public administration. These indicators measure motives for e-government implementation and development. They also focus on obstacles that are seen by the people from internal (employees) and external (end-users) point of view. Social actions and activities in the filed of e-government are considered as well, e.g. changes in employment structures, additional educations, policies of internet use and emails for employees and digital divide indicators, like IT access and knowledge for end users.
4.2.	Methodology

Compared to the other developed states of EU there is no quantitative or qualitive benchmarking of e-government development in Slovenia and no suitable methodology for doing such benchmarking. However, this kind of measurement would enable Slovenian government to react fast and quickly remove the obstacles. Unperformed activities of e-government strategies and policies would be detected before the critical situations. The process of benchmarking has to be developed wholly and based in indicators that could give results in short intervals (fast and easy frequent benchmarking). These would give sufficient data for analysis and overview of the e-government state. Information gained this way would be also beneficial to project managers of different strategies, definition of priorities, modifying project activities to current situation, discovering weaknesses of e-government development. They would be suitable for managers finding best practices, calculating implement-ability and discovering the factors that influence on the success of projects. Even the Bangeman report (Bangemann, 1994), which is one of the basic documents of information society warns that those who would not develop quick enough can lag behind and disable the fast economic development, society prosperity and competition. And these can influence on the country and even drag it in the serious crisis. Therefore the methodology has to enable fast and frequent but adequate measurement to achieve aimed objectives. Just enough indicators has to be set, not to much, not to few.
Our methodology is divided into 3 segments:
•	checking web sites and portals of public administration and its organisations;
•	testing communication between end users and e-government;
•	interviewing participators of e-government processes from supply and demand side.
Checking of web sites is timely consuming process and an extensive measurement that at the same time demands a lot of knowledge from the researchers and good knowledge on e-government and its services. It demands a good selection of web sites since all could not be scanned and definition of indicators that could give exact results and at the same time be aggregated to give a more global overview. Without exact rules and definitions there can be big differences between results of different people measuring indicators (human factor) and the differences could disable the representation of the actual situation. These mistakes could also be important if benchmarking is done in different countries, since unclear definition of measuring methodology would in this case give different and incomparable results.

Testing communication is effective and simple way of measuring functionality of some services that are offered by e-government. One of the most common ways of electronic communication is definitely e-mail. The implementation and usage are not complicated. Therefore, most users on supply side know how to use it. The same goes for the demand side. The e-mail addresses that are available on web sites for communication C2G or C2B are similar to addresses used for regular (snail) mail. The difference is usually in the reaction to the message. The paper form is more official in the minds of employees so every one message is reacted upon sooner or later. The electronic form of communication does not have the same kind of respect or culture and can be easily ignored. The simulation and measuring of this kind of communication would show the real situation, which might differ to the numbers of published e-mail addresses on the web. We know it is a fact that sending e-mail is easier then paper one, but if the address is officially published it should carry some responsibility with it. That is the reason for doing such a benchmarking.

The interviewing or questioning participant in the e-government processes considers demand and supply side. The hardest task in this form of benchmarking is finding a representative sample since all employees or end users cannot be surveyed. The situation is a bit less complicated on the supply side since the employees questioned want to cooperate or have to cooperate because of the policies of administration or superiors. Usually they are willing to cooperate if they receive the final results, since they can use them to compare their situation with others. On the demand side, defining the representative sample is more problematic. The people are not willing to cooperate, because of the high number of different inquiries in everyday life and the big number of questions (indicators) for this model. The financial burden for the institution doing the benchmarking is also high, especially if there is no government aid or sponsorship.
4.3.	Aggregation of indicators

As already mentioned the aggregation of indicators is necessary and demanding task. It is necessary since the results of individual indicators are not clear and this task is too extensive. The results should be presented in its final form in some kind of aggregated form and usually in the numerical scale. On the other hand, this kind of representation can often be misleading or incorrect. In the process of aggregation and calculation from the results of individual indicators we can change or loose their real meaning or values. Therefore, the task of aggregation has to be applied by exact and well-defined rules that have to be available to those reading the result as well. Only in this way can the readers verify the aggregated results in find out how the researchers got them.
5.	References

Bangemann M. et.al. (1994). Europe and the Global Information Society – Recommendations to the European Council. European Commission, Brussels.
Benchmarking eEurope (2003). http://europa.eu.int/information_society/eeurope/benchmarking/text_en.htm.
Cap Gemini Ernst & Young. (2001). "Summary Report: Web-based Survey on Electronic Public Services (results of the first measurement October 2001)". European Commission, DG Information Society, http://www.europa.eu.int/information_society/eeurope/benchmarking/list/2001/index_en.htm.
Cap Gemini Ernst & Young. (2002). "Summary Report: Web-based Survey on Electronic Public Services (results of the second measurement April 2002)". European Commission, DG Information Society, http://www.europa.eu.int/information_society/eeurope/benchmarking/list/2002/index_en.htm.
Cullen, R., Houghton, C. (2000). "Democracy Online: An Assessment of New Zealand Government Web Sites". Government Information Quarterly. Vol. 17/3, 243–267.
Espana 2002, Fundacion Retevision. (2002). "Evolution of eGovernment in the European Union." Report commisioned by the Spanish Presidency of the Council of the European Union. Madrid, 25.–26. april 2002.
IICMA International City/County Management Association, PTI Public Technology, Inc., (2001). "Is Your Local Government Plugged In?" Highlights of the PTI/ICMA 2000 Electronic Government Survey.
NAO National Audit Office, (1999). "Government on the web. Comptroller and auditor general", LONDON: The Stationery Office. URL = http://www.nao.gov.uk/home.htm, http://www.governmentontheweb.org
TietoEnator Trigon AB (2001). "Benchmarking of Electronic Service Delivery in the Public Sector: Executive Report". www.europa.eu.int/ispo/ida.
UNPAN United Nations Division for Public Economics and Public Administration (2001). "A Worldwide Survey on E-government: Benchmarking E-government: A Global Perspective". United Nations Division for Public Economics and Public Administration American Society for Public Administration. URL = http://www.unpan.org/egovernment2.asp#survey 
Vintar, M., Dečman, M., Kunstelj, M. (1999). "Telematics in the Service of Democracy: the Slovenian Parliament and other Slovenian Public Institutions on the Internet." In Parlamentary Affairs: a journal of comparative politics, Vol. 52/3, 451–463.
Vintar, M., Kunstelj, M., Leben, A. (2002). "Delivering Better Quality Public Services Through Life-event Portals". In NISPAcee conference presentation, Krakov, Poland.
Vintar, M., Dečman, M., Kunstelj, M. (2000). "Public sector and internet: the case of Slovenia". In Uporabna informatika, Ljubljana, Slovenia. Vol. VIII/3, 146–158.
West, D.M. (2001). "Urban E-Government: An Assessment of City Government Websites". Brown University, Providence, USA. http://www.insidepolitics.org/egovt01city.html.

Table1: Overview of e-government models analysis
	Research	
1	Cap Gemini Ernst & Young (2001). Summary Report: Web-based Survey on Electronic Public Services (Results of the first measurement: October 2001).	
2	IPMIT. (2002). Information society indicators in public administration in Slovenia.	
3	TietoEnator Trigon AB (2001). Benchmarking of Electronic Service Delivery in the Public Sector: Executive Report.	
4	EOS Gallup Europe. (June 2001). FLASH EUROBAROMETER 103: Internet and the public at large - report.	
5	INRA (EUROPE), European coordination office. (MARCH 1999).EUROBAROMETER 50.1: 
Measuring Information Society – Report.	
6	INRA (Europe), European Coordination Office. (Spring 2000). EUROBAROMETER 53: MEASURING INFORMATION SOCIETY 2000 – Analytical REPORT.	
7	Hart-Teeter. (September 2000). E-GOVERNMENT: THE NEXT AMERICAN REVOLUTION. 	
8	John Clark. (September, 2000). Citizens Expectations for Electronic Government Services.	
9	Government Internet Use Survey (1999).	
10	Cullen, R., Houghton, C. (2000). Democracy Online: An Assessment of New Zealand Government Web Sites.	
11	National Association of Counties (NACO). (April 2000). 2000 E-Government Survey	
12	Cook, M.E. (October 2000). What Citizens Want From E-Government: Current Practice Research.	
13	Strover, S., Straubhaar, J. (June 2000). E-Government Services and Computer and Internet Use in Texas.	
14	West, D.M. (September 2000). Assessing E-Government: The Internet, Democracy, and Service Delivery by State and Federal Governments. Brown University, Providence, ZDA.	
15	West, D.M. (September 2001). State and Federal E-Government in the United States.	
16	West, D.M. (September 2001). Urban E-Government: An Assessment of City Government Websites.	
17	IICMA (International City/County Management Association), PTI (Public Technology, Inc.). (February 2001). Is Your Local Government Plugged In?	
18	UNPAN. A Worldwide Survey on E-government  	
19	COMNET-IT (The Commonwealth Network of Information). (2000). Technology for Development Foundation) .GLOBAL SURVEY ON ON-LINE GOVERNANCE: FINAL REPORT	
20	Ouzounis, V., Hardey, C. PACE SURVEY OF DEVELOPMENTS AND USE OF E-COMMERCE WITHIN EUROPEAN PUBLIC ADMINISTRATIONS. ECCO, GMD-FOKUS.	
21	Momentum Research Group (2000). Benchmarking the e-Government Revolution, year 2000 Report on Citizen and Business Demand Momentum Research Group of Cuningham Communication.	
22	OGC (Office of Government Commerce). (July 2001). e-Government Benchmarking Electronic Service Delivery. Office of the e-Envoy.	
23	NAO (National Audit Office). (1999). Government on the web.	
24	The Henley Centre, MORI Government research. (2000). E-Government: ready or not?.	
25	World Markets Research Centre and Brown University, USA (2001). GLOBAL E-GOVERNMENT SURVEY 	
26	AUDIT COMMISSION. (2002). Councils and E-government Research So Far.	
27	MORI. (January 2000). Assessing Attitudes to the Change of Address Function: Government Portal Research.	
28	U.S. General Services Administration, Intergovernmental Advisory Board (1999).Integrated Service Delivery: Governments Using Technology to Serve the Citizen.	
29	National Research, Inc. (1999).Technology in America’s Counties A Survey of Use and Application. NACO - National Association of Counties, N.W.Washington, ZDA.	
30	Stowers, G.N.L. (1999). Becoming Cyberactive: State and Local Governments on the World Wide Web.	
31	Cohen, S., Eimicke, W. (2001). The Use of the Internet in Government Service Delivery. The Pricewaterhouse Coopers Endowment for The Business of Government. endowment.pwcglobal.com	
32	Vehovar, V., Vukčevič, K. (oktober 2001). Digital divide – Slovenia 2001. University of Ljubljana, Faculty of Social Sciences, Centre for Methodology and Informatics, Project RIS.	
33	Vehovar, V., Kuhar, M. (October 2001). Indicators of IT in Slovenia and EU. University of Ljubljana, Faculty of Social Sciences, Centre for Methodology and Informatics, Project RIS.	
34	CATI (January 2002). Government web sites analysis in Slovenia. Ministry of the Information Society.	

Table 2: The four-stage model of on-line sophistication of e-services (source Cap Gemini Ernst & Young, 2002)

Stage	Definition	
0	No publicly accessible website(s) or the website(s) do not qualify for any of the criteria for the stages 1-4	
1	Information necessary to start the procedure to obtain the service available on the website(s)	
2	Interaction includes downloadable or printable form to start the procedure to obtain the service on the website(s)	
3	Two-way interaction comprehends electronic forms to start the procedure to obtain the service on the website(s)	
4	Transaction covers full electronic case handling of the procedure by the service provider (including decision, notification, delivery and payment if necessary)	

Figure 3: The pyramid of e-government indicators

Figure 4: Service (left) and life event (right) representation