The METRONOME project (A Methodology for Evaluation of Project Impacts in the
Field of Transport) was financed under the 7th Framework Programme (FP) of the
European Commission. The project aimed to develop a methodology for evaluating
project achievements supported in FP5 and FP6 with particular focus on:
Strengthening industrial competitiveness (IndCo); Contributing to sustainable
development (SuD); and Improving community and public policies (CPP). The
expected impact set for the project was to use the results and findings from
the FP5 and FP6 projects to contribute to the definition of intermediate
performance targets for FP7 and new research policy objectives.
The developed METRONOME screening, selection and evaluation methodology has
three main phases (Figure 1):
Identification of European transport research and policy objectives for
Industrial Competitiveness, Sustainable Development, and Community and Public
Screening and selection of FP5 and FP6 themes and projects for the evaluation
Evaluating project impacts through the METRONOME impact model, using a
The methodology takes a two-dimensional approach to project impact evaluation.
On the one hand, it evaluates the projects’ achievements in light of the FP5
and FP6 Work Programme objectives and targets set for IndCo, SuD and CPP
themes. On the other hand, it evaluates, through the METRONOME impact model,
the impacts of the FP research projects according to four impact groups and
related indicators. The impact groups are (1) impacts on manage-ment and
co-ordination, (2) scientific impacts, (3) customer/end user impacts, and (4)
societal impacts (Figure 2).
The METRONOME methodology proposes the following four complementary approaches
(elements) to finding values for the impact indicators within the above
Project evaluation and dissemination quality matrices
Coordinator questionnaires (and complementing interviews)
Lead user interviews
In the course of the METRONOME project, the developed methodology was applied
to a sample of 100 FP5 and FP6 transport projects in the fields of industrial
competitiveness, sustainable development and community and public policies.
For each of the evaluation themes, a specific application with different mixes
of evaluation methods was developed and applied.
In general, the METRONOME evaluation methodology proved to be useful in
producing information for the definition of intermediate performance targets
for FP7 and of new research policy objectives from the perspectives of: (1)
achieving the FP5 and FP6 objectives and targets, (2) FP5 and FP6
implementation and operational environment, and (3) research project outcomes
and impacts. Testing the methodology illustrated that different mixes of
evaluation methods (both reviewing and surveying) are needed to gain in-depth
information on both FP target achievement and potential project impacts over
various time perspectives.
The main difficulty encountered during the METRONOME methodological
development was the availability of project result (e.g. Final Report) data. A
structured, up-to-date FP project result database that is ready and available
for the evaluators would enable more reliable, less time-consuming and less
costly FP impact evaluations. Other major difficulties identified during the
methodology development were the relatively low response rates in co-ordinator
surveys and interpretation of the multi-level objective and target structures
of the FPs as the basis for evaluation. As regards the future methodological
development, only a repeated (and simultaneously elaborated) evaluation
process can provide more detailed analysis of project or programme impacts.
Further, and as a complement to the former, more emphasis and resources are
needed in integrating such future elements into evaluation methodologies that
can better support the future WP objective and target setting in the changing
European transport and research environment.
Based on the METRNOME project reviews, it can be claimed that in general,
achievement of FP objectives in both FP5 and FP5 has been good throughout, and
in some cases very good. The vast majority of the strategic project objectives
of the project sample were fully met. This indicates that on individual
project levels, in terms of both substance and practicalities, the projects
worked well. A surprising finding was, however, that in the fields of SuD and
CPP, both FP5 and FP6 projects were considered to have contributed more to
higher level Work Programme objectives than the lower level Key Action or
Programme Subdivision objectives, which could be considered more directly
applicable to the projects commissioned. This suggests that there could be
considerable discrepancies between the different levels of objectives and
targets set for the FPs.
Based on the surveys, project coordinators presented slightly more positive
views of the achievements of FP5 projects than of FP6 projects. In the case of
lead users, however, the result was reversed. Consequently, no meaningful
distinction could be made between the two FPs in either the objective
achievements or the impacts. The differences could even be due to the temporal
implementation of FPs, in a sense that recent FP6 projects are better recalled
than the more distant FP5 ones.
Keeping the above in mind, it could, however, be said that in the field of
IndCo, major achievements were found in the fields of development of advanced
technologies, processes and services, and in the contribution to societal,
environmental (e.g. safety, traffic congestion) and financial issues. These
same fields were emphasised in both FPs. The main contributions of the SuD
projects in both FPs were identified as developing, integrating and managing a
more efficient, safer, more secure and environmentally friendly transport
system to provide user-friendly door-to-door services. Contribution to the
development of decision-making tools was the main achievement of CPP projects
also in both FPs.
Scientific publications and a high level of scientific expertise in general
were considered as the main immediate impacts of FP5. Improved networking
between researchers and public/private organisations and strengthened networks
between international parties were seen as the major immediate impacts of FP6.
Also patents and standards produced in the IndCo projects (especially in FP6)
represent the immediate impacts, even though the transport industry and
service sector seem not to have been greatly involved .
As regards the intermediate impacts, the major impacts of both FPs can be seen
as strengthened expertise, development of decision-making tools, co-operation
with end users in the projects, and usage of project results by the public
sector or other societal actors. Contributions (often indirectly) to new
transport policy development but also to new products or service developments
were also seen as somewhat positive. The evoked networking or co-operation
seems to be strongest among project research partners, but can be also
identified among stakeholders in both the public and private sectors.
Transferring project results into standards, norms or regulations and failure
to raise new unsolved research questions were considered weaknesses of the FP5
and FP6 transport projects. This indicates that even though tools for
decision-making have been developed and some contribution made to e.g.
transport and SuD policies and strategies, the practical, regulatory outcomes
have been modest or are not known. In addition, the identified discrepancy
between the FP objectives of different levels and the low level of achievement
of European level objectives by projects in matrix evaluations supports this
finding. The above illustrates that the impacts of projects in both FPs have
been strongest within the impact group of management and coordination. Also
scientific and end-user impacts have been adequate, but wider societal impacts
If one looks at the potential impacts of FPs on the shaping of the European
Research Area (ERA), the most critical issue is the availability and
dissemination of FP project result data. This concerns both lower, individual
project level and centralised EC level. Currently the project results are not
easily available for the use of individual projects/persons or for FP
evaluation purposes. Consequently, the FP output quality needs improvements
both on a project level (e.g. longer supported maintenance of web sites) and
community level (centralised FP project output database).
Another important issue is the lack of consistency identified between the
different levels of objectives set for FP Work Programmes. Only a few of the
evaluated projects met simultaneously their own strategic objectives, Work
Programme (WP) objectives on two levels and relevant European policy
objectives. In order to clarify the FP future evaluation in terms of objective
achievements, the consistency of the WP objective structure should be
Other aspects identified as relevant for future FP evaluations, which
METRONOME time frame and resources did not allow, were the following: Closer
co-operation with technology platforms and EC officials in project
evaluations, detailed investigations regarding follow-up research project
paths, including transport projects commissioned under other programmes than
transport (e.g. Information Society, Environment and Security) into the
evaluation, and considering very carefully the point in time for FP evaluation.
To conclude, it seems that FP5 and FP6 have certainly played a significant
role in the European science and technology agenda. For evaluation of the FPs’
role on the global map or on the contribution to EU research competitiveness
at international level, the project sample does not give a representative
insight. The finding of the METRONOME evaluation should, how-ever, be relevant
to all national and European civil servants and researchers both within
indus-try or public bodies and interested in the future of European transport
research, both in the-matic and practical terms.