1.5 The Scientific Advisory Groups (SAG) surveys

As pointed out earlier, the basic intention of the European System of Environmental Pressure Indices is to provide decision-makers and the public with a tool that enables them to come to better decisions in a democratic process. Comprehensiveness and objectivity are necessary to ensure that such a tool enjoys the same acceptance and trust that the SNA and its top aggregate, GDP, are enjoying in economic policy.

When designing this tool, however, it quickly becomes evident that there is a strong “human element” determining the views on the right way to do it. The mental distance between those who are shaping the scientific debate, and those who decide on budgets and environmental standards, is often extremely great. Using indicators and indices to bridge the gap between science and politics is the attempt to enable communication between two communities that have their own dynamics and values. This is complicated by the fact that indicators don’t fall from heaven; the data needed for calculating them have to be produced by “the statistical system”, that is, by human beings who may also have their own views on what is sound or unsound.

For producing a convincing indicator tool, three communities with entirely different “system dynamics” must be integrated:

- scientists should discover something revolutionary every year;

- statisticians should consistently reproduce indicators year by year with the same methodology;

- politicians need simple and strong messages to talk to their voters.

The picture of the “information iceberg” demonstrates the efforts needed to produce a comprehensive system of indicators and indices. A lot of “invisible work” is necessary to provide good data for indicators, and we must therefore evaluate carefully how to spend the limited resources available for such work. Since Eurostat does no own data collection, the national statistical services will have to bear most of the burden of this task. The European Commission can give a modest financial support in the development phase of such a project, but the resources needed for a regular data collection in fifteen Member States exceed the limits of the budget allocation foreseen in COM (94) 670.

1.5.2 Selection of the panels

Statistical data collection behaves like an oil tanker: if the route is clearly defined, the cost of transport (of data) is very low; but quick changes of direction are impossible. Likewise, establishing a new data collection in all Member States of the European Union requires typically five to ten years (calculated from defining the needs to obtaining first data).

Sometimes, statisticians receive a phone call from a policy-maker, saying “ I have a meeting on xyz tomorrow, can you quickly send me some data for that? ”. The appropriate answer would be “ certainly, call me again in five or ten years ”. Of course, statistical data collection should serve the political process. However, asking policy-makers for their specific data needs is rarely a good strategy for achieving that, given that it takes a lot of time between the discovery of an environmental problem, and the moment when a politician wants to see the indicator describing that problem. For example, Climate Change or Ozone Layer Depletion were discussed among scientists already in the early seventies; nowadays these issues are firmly placed on the political agenda, but the data situation is still far from satisfactory.

This is the precise reason why we chose senior natural scientists for the Scientific Advisory Groups; we hoped that they would help us to look into the future, and to identify the indicators that will be demanded by politicians in the coming years. Most of our SAG experts do have regular contacts with policy-makers, and they will often be arguing with them on which are the priorities of the future. By “tapping” this debate at the source, we hoped to gain time, so that we might have the right indicators ready when the issue reaches the political agenda, in spite of the inertia of the data collection process.

The second reason for choosing natural scientists is of course that the detail required for a comprehensive system of indicators goes beyond the knowledge of the average decision-maker. Only a toxicologist, for example, can define the relative importance of heavy metals, POPs, pesticides etc., and the most appropriate indicators for describing these issues.

The selection of the panels was performed by the ten Specialized Institutes (SI) of the Pressure Indices project. They had clear instructions for doing this selection, in particular regarding qualification of the scientists, political neutrality, and geographical coverage. For each Member State of the EU, we asked for 30 scientists per policy field. This proved to be difficult; within the projects’ budget, we could not afford having one SI per country and policy field, so each of the SIs had to cover fifteen Member States. Finding 30 biodiversity experts in Finland, or 30 waste experts in Portugal, is theoretically possible (there are many university teachers specialised in environmental issues in all EU states), but certainly not an easy task. On average, we got 16 experts per country and policy field, a total of more than 2,400. Below the distributions by country and by policy field:

It is worth noting that finding the correct postal addresses for 2,400 scientists in fifteen countries with eleven different languages was, although intellectually not very stimulating, by far the most difficult and most expensive step in the indicator selection process. There was a theoretical risk that the SIs might have introduced a political bias by favoring industry or NGO experts (Eurostat told the SIs to be neutral, of course); in practice, they were so grateful for any hint to an expert name (on workshop participation lists, from scientific literature, EEA network, national research boards, ...) that they certainly found no time to reflect on the experts’ political positions. Nonetheless it is correct to say that in spite of the large number of experts and the good geographical and thematic distribution, the SAG list remains a “hand-made” selection. Ideally, the expert selection should have been done by the most relevant societal actors (industry/farmers associations, environmental NGOs, environment ministries). Given the practical problems, that would have seriously delayed the project. However, in view of the more controversial process of indicator aggregation, we still plan to broaden the political basis through such a process as soon as possible.

1.5.3 Survey I: Brainstorming

The first round of the SAG consultations started with a “blank page”. After some general explanations (purpose of the project, PSR model, definition of a good pressure indicator), they were invited to list five indicators that would be necessary to describe pressures within their area of expertise.

Alternatively, we could have provided the experts with a preselection of indicators that are either available or in development by international organizations. This would have biased the selection towards data availability, and thus saved us a lot of work... at the expense, however, of sacrificing the principal goal of providing policy with a comprehensive system of indicators. Since the Pressure Indices are a long-term project, we needed a proper Cost-Benefit Analysis defining data needs (benefits for the political process) and costs of data collection efforts independently.

Since many indicators were mentioned more than once, the total number of different proposals was approximately one-thousand, still far too many for the purposes of the project. The Specialized Institutes therefore had to “boil down” these 1,000 proposals to a manageable number of 300, that is, 30 for each policy field, by:

Within the indicator selection process, the “boiling down” of the initial proposals was the most sensitive phase, since it could not be absolutely transparent [8]. The SIs were fully aware, however, of the risk to introduce a personal bias at this stage, and the necessity to remain as neutral as possible, in order to respect the views of the SAG.

Preserving the original proposals was not always easy; when looking at the current list, many indicator experts would still feel a burning desire to improve it. The problem is that while each single expert may be able to produce a better, more consistent list of indicators, it is almost impossible to obtain a list that looks perfect to more than one expert...

Once we received a letter by a SAG member saying “ This survey is useless. We should simply gather the six real experts for xyz in Europe, and within a couple of days we will produce a much better list of indicators ”. We did not accept this generous offer (although maybe he would really have been able to produce a perfect list), because his letter implied that a) at least nine of the fifteen EU Member States did not have a single expert good enough to define a European indicator list for the policy field xyz; and b) that six persons would be able to provide a representative picture of the European indicator needs, speaking thus simultaneously for fifteen countries as well as for industry, agriculture, environmental NGOs and other relevant actors of environmental policy.

Certainly, the list of indicators can be improved. But it should be done gradually through a transparent process that involves many of those who will use the future indicators and indices for a better environmental policy debate. The acceptance of the Pressure Indices by their users will require a difficult compromise between scientific quality and democratic legitimation. Transparency and openness through the SAG surveys are an essential element for ensuring that this compromise will lead to useful indicators.

1.5.4 Survey II: Choosing the essential indicators

With the lists of 30 indicators per policy field, a second survey was started in October 1996. The SAGs were thus confronted again with “their” indicators, this time condensed to a list that fit on one page. The questionnaire asked the scientists to judge the proposed indicators according to a number of criteria [9] described below.

1.5.4.1 Policy Relevance

As mentioned in the introduction, indicators should feed non-controversial information into controversial political debates. If the information is controversial, it will not help the debating parties. If the debates are so peaceful that everybody agrees with the opposite side, then developing indicators would be a waste of human and financial resources. We have therefore asked the SAG explicitly how they judge the “Policy Relevance” of the proposed indicators. The results were slightly disappointing, because the experts tended to see most indicators as policy-relevant. Furthermore, the ranking obtained was practically identical with the one obtained by the “core indicator” question (see below).

1.5.4.2 Analytical Soundness

Does the indicator measure the problem? Or rather something else? Ideally, the indicator should directly at the causes of the problem to be solved. For example, health impacts may be caused by particles. An indicator “Emissions of particles” would point fairly directly at the cause; an indicator “number of cars per household”, however, although showing a significant correlation to the problem, would give a misleading message to the public and the decision-makers: “ reduce the number of cars, and you will solve the health problem ”. The correct message “ reduce the particle emissions ” would instead encourage decision-makers to examine all options, from less traffic to technological solutions, and to choose the policy mix with least cost and fastest results.

Unfortunately, we still have plenty of indicators for which the “Analytical Soundness” is rather low, e.g. the pesticide indicators (which do not distinguish between more or less harmful pesticides, good or bad application practice etc.) or the water consumption indicators (which do not take into account the scarcity or abundance of water resources in a given region). Such indicators give only indirect hints to the real problems.

1.5.4.3 Responsiveness

The most important anthropogenic pressure on the environment is the fact that human beings are living on this planet. That is a tautological statement, of course, but if we took it seriously, “population density” should be a component of any pressure index... and it would probably occupy most of the pie chart showing the weights of the components in that index. However, that index would be happily ignored by politicians. Decision-makers must see a chance to improve the indicator, otherwise they will not act. Therefore, indicators must be “responsive” to political action. Instead of pointing at population density, they should point at those aspects of the presence of human beings that somehow can be eliminated without eliminating people.

While the “population density” example is an abstract one, there are many other practical examples where indicators point at trends that cannot be strongly influenced by environment policy, for example the trend in overall transport volumes. “Transport volume” as a component of an Air Pollution index would occupy, given its importance, a high share; but political action to reduce Air Pollution would concentrate on those index components that could be more easily reduced.

1.5.4.4 Core indicators

Ideally, we should identify all relevant pressures on the environment, and aggregate them carefully into pressure indices at environmental policy field level. As the results of the first survey showed, however, that would require too many indicators – the environment is a highly complex exercise. However, producing e.g. 300 indicators would go far beyond the capacities of the statistical system including the EEA and other international organisations. What we can hope for will be approximately 60 pressure indicators in the short term, and about 80-100 in the next century. Furthermore, there are significant differences in importance between, for example, “CO ₂ emissions” and “Losses in the water supply network” or “Density of marine transport”, which would make the contribution of more “exotic” indicators to an overall policy field index so small that they might as well be dropped. The optimum number of indicators depends also on their usage in the political debates. Currently, the small number of indicators, and often their poor statistical quality, prevents decision-makers from using them seriously as a reference. The relevance of indicators, however, grows with the extent to which they can cover the issues discussed in public:

1.5.4.5 Improving the policy field coverage by re-clustering

During the second survey, we had tried to offer as much choice as possible to the SAG experts, e.g. by providing several versions of “CO ₂ emissions” to the Climate Change experts (per capita, per GDP etc.). We hoped that we could thus identify the variant of the indicator that would be most appropriate for describing the issue to be measured.

As a side effect, however, each single CO ₂ emission indicator received only a fraction of the votes for core indicators that a unique CO ₂ emissions indicator would have got. Therefore, we had to add together the results for all variants, and count them as if there had only been one proposal. This “re-clustering” has been done also for indicators pointing at slightly different issues (e.g. different types of hazardous wastes), because otherwise important issues would not have been included in the core list. Thus, the coverage of the policy field (measured through the number of times the indicators were declared as essential in the “core” question) could be improved without increasing the number of total indicators.

1.5.5 Survey III: Improving the indicator set, and first steps towards aggregation

1.5.5.1 Learning by doing: the roles of continuity and evolution

As already described in Chapter 1.5.1, bridging the gap between science and politics is a task that must take into account the differing dynamics of the systems involved. One of the interesting properties of the indicator-politics interface is that continuity is needed to achieve an impact. “One-off” indicators, presented once and then never again, have no significant impact - they are as short-lived as a sensational “chemical-of-the-month” story. The popular economic indicators derive their strength from the fact that the public, including decision-makers, are used to them, and are waiting for them. Even indicators that are meaningless for 90% of the population, like the Dow Jones Index, start to develop a life of their own, because everybody sees it daily in the evening news. Nowadays, a falling Dow Jones has become a yardstick for the severeness of the latest crisis, be it a crash of the Russian or Asian markets, a civil war, or a scandal in Washington - a meaning that was never intended. Likewise, GDP was never intended to be a welfare indicator, but through eternal repetition, the original meaning got lost. There are two lessons to be drawn from this observation:

Since the latter effect is undesired, we must develop a mechanism that ensures that the Pressure Indices become policy-relevant through continuity and steadiness, but at the same time maintain the positive effects of giving science a stronger voice in the democratic debate. In other words:

In practice, this can and will be done by asking the SAG panels regularly (yearly or biannually) to check the scientific validity of the indicator set, and to replace a few obsolete indicators with new ones that reflect the latest state of the scientific (and political) debate. This mechanism is not entirely new - national accountants do that all the time in order to keep track of changes of economic reality (a GDP not including computer manufacturing would probably be considered “economically unsound”).

Although the current set of 60 indicators selected for the first publication still looks convincing for many experts, it has one major shortcoming: the selection process was not based on knowledge of real figures, but on perceptions of the SAGs. This is a dangerous argument, since most of the involved scientists would rightly insist that they know their policy field very well, and have a better access to relevant figures than statisticians; but nonetheless it can be expected that a look at the set of six indicators per policy field, covering several EU Member States, may provide some new insights. Most of the scientific debate on environmental and SD indicators, valuation and aggregation has been rather abstract, but we are now approaching the phase of “learning by doing”, of checking abstract concepts against real figures.

Therefore, the 2,300 SAG experts will be once more asked for their advice. They will receive an excerpt of the first pressure indicators publication containing the indicators for their policy field, and will be asked to tell us, on the basis of a thorough look at the “real” figures:

Based on the response, the indicator set for the update of the first publication will be cautiously adapted, by replacing approximately 5-10 of the sixty indicators with those most frequently suggested for upgrading. Thus, the indicator users will be given the chance to detect changes compared to the first publication ( continuity), while at the same time the soundness of the indicator set is being steadily improved ( evolution).

1.5.5.2 Some principles of aggregation

Step 1: choice of index components : which elements are needed to answer the question “Is landfilling better than incineration” ? Since the list of 100 indicators has been selected in order to give a comprehensive coverage of all environmental problems, it makes sense to consider this list as a “menu” from which to choose the necessary indicators. For example,
A ( landfilling) produces methane (CC-2), heavy metal emissions to water (TX-4), land use (UP-6), BOD (WP-6);
B ( incineration) produces NOx, NMVOC, SO2 and particle emissions (AP-1 to AP-4), CO ₂ emissions (CC-1), heavy metal emissions to air (TX-5), noise (UP-5), some land use (UP-6) and hazardous waste (WA-3). [10]

Step 2: weighting: which is the share an indicator (issue) should be given in order to give a representative picture of the overall pressure in the respective policy field? Are NOx emissions (AP-1) in general a bigger problem than particles (AP-4)? The Pressure Indices project has not yet produced agreed weighting factors, but as a first proxy one could look at the frequency the SAG expert included the respective indicators in their core list of “essential” indicators, for example for the policy field Air Pollution :

The Scientific Advisory Group (SAG) for Air Pollution had to decide which 5 indicators from the proposal list of 30 indicators they considered as “essential” to give a good description of the total pressure. Although the question did not ask directly for weighting coefficients, it can be assumed that the distribution shown here corresponds roughly to the “political weight” of the indicator. NOx and NMVOC’s would thus account for approx. half of the “problem pressure” associated with the policy field “Air Pollution”, while the other four indicators are of less importance. As long as new SAG survey results are not available, the distribution of core votes could be used as a first proxy to calculate indices.

Step 2: valuation: how should a given figure be judged? Is a decline of NOx emissions by 1% compared to last year “good”, “satisfactory”, or still “serious”? While on a national level the trend (x% per year) should be judged by the SAGs, on a project level another type of “normalisation” or valuation unit, must be found, e.g. “g NOx emissions per kg of waste incinerated”. Typically, one would examine the technological options, and group them into quality classes, e.g.
0-a g NOx/kg waste Advanced but expensive technology (“very good”)
a-b g NOx/kg waste Best Available Technology (“good”)
b-c g NOx/kg waste good standard technology (“satisfactory”)
c-d g NOx/kg waste cheap but outdated technology (“serious”).

Note that all figures, especially for step 2 (weighting), can of course be modified according to local or regional requirements. However, in order to maintain consistency and the possibility to compare one policy decision with another one in a different region or city, each deviation from e.g. national or international weighting coefficients should be justified explicitly. It is important to remain transparent when doing this, otherwise conflicts between national priority setting ( “methane emissions must be reduced according to the Kyoto Protocol” ) and local policy ( “methane has no effect on the health of our citizens and is therefore not our problem” ) could arise, e.g. by tacitly excluding methane from the choice of indicators for waste landfilling. If all x-thousand mayors of a country agreed to exclude methane, than this country could not fulfill its international contracts regarding greenhouse gas reductions.

1.5.5.3 First steps towards a consensus on weighting procedures

This distinction takes account of the discussion on critical loads, carrying capacity etc., which suggests that a simple “averaging” of emission trends would be misleading, because some types of pressures, although important, can be tolerated or solved through a slow, economically acceptable process, while others need immediate attention because critical levels may have been reached. Furthermore, simple multiplication is also not possible because of the differing responsiveness of the indicators. For example, an annual reduction of CO ₂ emissions by 1% would probably judged as “good”, given the enormous difficulties to adapt the whole economy to a different attitude towards energy use; a reduction of lead emissions or CFCs by 1%, in contrast, would probably only be considered “satisfactory”, since it is relatively cheap to do that, as demonstrated in the past.

Independently of the still controversial methodological debate on the advantages and disadvantages of aggregated indices, their acceptability as policy tools depends very much on whether experts with different political backgrounds can agree on weighting systems to be used for aggregation. There are observable differences in the valuation of environmental issues between experts, which sometimes seem to be correlated with “closeness” to the most relevant policy actors, such as:

1. where exactly scientists with different political affinities disagree, regarding e.g the importance of indicators, the weights to be used in a pressure index for their policy field, or the valuation of trends (questions A-C); this would allow us to prioritize such issues for further in-depth research;

2. where experts with different political affinities are closer to a consensus; this would allow us to start carefully testing the pressure indices system, and to gain practical experience with the use of indices for communication to the public and as support tools for decision-making purposes.

Of course, scientists tend to say (and believe) that they are totally independent, objective and neutral. However, nobody is free of personal beliefs and values, and quite naturally we trust some people more than others. Could you please tell us below to which of the political actors you personally have more confidence? Like all your answers to this questionnaire, we will treat this information absolutely confidential.

At first sight, this question may look very distant from the normal thinking of a statistical office; but the goal to provide objective, neutral and non-controversial figures (e.g. Pressure Indices) implies a need to assess how controversial such figures would be. Without empirical evidence, it would be almost impossible to distinguish between readily publishable results, and those that require in-depth research (presumably outside the statistical system) in order to gain the acceptance of the actors of environmental policy.

It can be expected that the “political sensitivity” of the results is low for the weighting part (see Valuation: the problem volume-problem share model ), while the valuation part will reveal substantial differences between e.g. experts close to economic sectors and those closer to environmental NGOs. This is a problem inherent to all valuation methods (monetization by WTP or avoidance cost, distance to goals etc.), but there has never been a systematic effort to quantify the degree of controversy, at least not at the level of detail provided by the set of 60/100 indicators. The responses may thus also provide valuable information for alternative aggregation methods, e.g. in the context of the Commission’s initiative on Satellite Accounting.

[10] Note that “Waste landfilled” (WA-1) and “Waste incinerated” (WA-2) have not been included in this list. At national level, these two indicators represent the impacts of landfilling and incineration in a very general way (i.e. the various types of pressures generated); however, for comparing two concrete options at local level, the individual impacts should be studied.

A	B	D	DK	E	F	FIN	GR	I	IRL	L	NL	P	S	UK	EU
158	170	220	187	166	170	116	147	188	106	34	205	158	196	201	2422

AP	CC	LB	ME	OD	RD	TX	UP	WA	WP	Total
234	248	284	270	176	175	215	288	252	280	2422