The METRONOME project (A Methodology for Evaluation of Project Impacts in the Field of Transport) was financed under the 7th Framework Programme (FP) of the European Commission. The project aimed to develop a methodology for evaluating project achievements supported in FP5 and FP6 with particular focus on: Strengthening industrial competitiveness (IndCo); Contributing to sustainable development (SuD); and Improving community and public policies (CPP). The expected impact set for the project was to use the results and findings from the FP5 and FP6 projects to contribute to the definition of intermediate performance targets for FP7 and new research policy objectives.

The developed METRONOME screening, selection and evaluation methodology has three main phases (Figure 1):

  1. Identification of European transport research and policy objectives for Industrial Competitiveness, Sustainable Development, and Community and Public Policies

  2. Screening and selection of FP5 and FP6 themes and projects for the evaluation

  3. Evaluating project impacts through the METRONOME impact model, using a multifaceted approach

 

 

The methodology takes a two-dimensional approach to project impact evaluation. On the one hand, it evaluates the projects’ achievements in light of the FP5 and FP6 Work Programme objectives and targets set for IndCo, SuD and CPP themes. On the other hand, it evaluates, through the METRONOME impact model, the impacts of the FP research projects according to four impact groups and related indicators. The impact groups are (1) impacts on manage-ment and co-ordination, (2) scientific impacts, (3) customer/end user impacts, and (4) societal impacts (Figure 2).

 

The METRONOME methodology proposes the following four complementary approaches (elements) to finding values for the impact indicators within the above groups: 

  1. Project evaluation and dissemination quality matrices

  2. Coordinator questionnaires (and complementing interviews)

  3. Lead user interviews

  4. Workshop(s)

In the course of the METRONOME project, the developed methodology was applied to a sample of 100 FP5 and FP6 transport projects in the fields of industrial competitiveness, sustainable development and community and public policies. For each of the evaluation themes, a specific application with different mixes of evaluation methods was developed and applied. 

In general, the METRONOME evaluation methodology proved to be useful in producing information for the definition of intermediate performance targets for FP7 and of new research policy objectives from the perspectives of: (1) achieving the FP5 and FP6 objectives and targets, (2) FP5 and FP6 implementation and operational environment, and (3) research project outcomes and impacts. Testing the methodology illustrated that different mixes of evaluation methods (both reviewing and surveying) are needed to gain in-depth information on both FP target achievement and potential project impacts over various time perspectives.

The main difficulty encountered during the METRONOME methodological development was the availability of project result (e.g. Final Report) data. A structured, up-to-date FP project result database that is ready and available for the evaluators would enable more reliable, less time-consuming and less costly FP impact evaluations. Other major difficulties identified during the methodology development were the relatively low response rates in co-ordinator surveys and interpretation of the multi-level objective and target structures of the FPs as the basis for evaluation. As regards the future methodological development, only a repeated (and simultaneously elaborated) evaluation process can provide more detailed analysis of project or programme impacts. Further, and as a complement to the former, more emphasis and resources are needed in integrating such future elements into evaluation methodologies that can better support the future WP objective and target setting in the changing European transport and research environment.

Based on the METRNOME project reviews, it can be claimed that in general, achievement of FP objectives in both FP5 and FP5 has been good throughout, and in some cases very good. The vast majority of the strategic project objectives of the project sample were fully met. This indicates that on individual project levels, in terms of both substance and practicalities, the projects worked well. A surprising finding was, however, that in the fields of SuD and CPP, both FP5 and FP6 projects were considered to have contributed more to higher level Work Programme objectives than the lower level Key Action or Programme Subdivision objectives, which could be considered more directly applicable to the projects commissioned. This suggests that there could be considerable discrepancies between the different levels of objectives and targets set for the FPs.

Based on the surveys, project coordinators presented slightly more positive views of the achievements of FP5 projects than of FP6 projects. In the case of lead users, however, the result was reversed. Consequently, no meaningful distinction could be made between the two FPs in either the objective achievements or the impacts. The differences could even be due to the temporal implementation of FPs, in a sense that recent FP6 projects are better recalled than the more distant FP5 ones.

Keeping the above in mind, it could, however, be said that in the field of IndCo, major achievements were found in the fields of development of advanced technologies, processes and services, and in the contribution to societal, environmental (e.g. safety, traffic congestion) and financial issues. These same fields were emphasised in both FPs. The main contributions of the SuD projects in both FPs were identified as developing, integrating and managing a more efficient, safer, more secure and environmentally friendly transport system to provide user-friendly door-to-door services. Contribution to the development of decision-making tools was the main achievement of CPP projects also in both FPs. 

Scientific publications and a high level of scientific expertise in general were considered as the main immediate impacts of FP5. Improved networking between researchers and public/private organisations and strengthened networks between international parties were seen as the major immediate impacts of FP6. Also patents and standards produced in the IndCo projects (especially in FP6) represent the immediate impacts, even though the transport industry and service sector seem not to have been greatly involved .

As regards the intermediate impacts, the major impacts of both FPs can be seen as strengthened expertise, development of decision-making tools, co-operation with end users in the projects, and usage of project results by the public sector or other societal actors. Contributions (often indirectly) to new transport policy development but also to new products or service developments were also seen as somewhat positive. The evoked networking or co-operation seems to be strongest among project research partners, but can be also identified among stakeholders in both the public and private sectors.  

Transferring project results into standards, norms or regulations and failure to raise new unsolved research questions were considered weaknesses of the FP5 and FP6 transport projects. This indicates that even though tools for decision-making have been developed and some contribution made to e.g. transport and SuD policies and strategies, the practical, regulatory outcomes have been modest or are not known. In addition, the identified discrepancy between the FP objectives of different levels and the low level of achievement of European level objectives by projects in matrix evaluations supports this finding. The above illustrates that the impacts of projects in both FPs have been strongest within the impact group of management and coordination. Also scientific and end-user impacts have been adequate, but wider societal impacts quite modest.

If one looks at the potential impacts of FPs on the shaping of the European Research Area (ERA), the most critical issue is the availability and dissemination of FP project result data. This concerns both lower, individual project level and centralised EC level. Currently the project results are not easily available for the use of individual projects/persons or for FP evaluation purposes. Consequently, the FP output quality needs improvements both on a project level (e.g. longer supported maintenance of web sites) and community level (centralised FP project output database).  

Another important issue is the lack of consistency identified between the different levels of objectives set for FP Work Programmes. Only a few of the evaluated projects met simultaneously their own strategic objectives, Work Programme (WP) objectives on two levels and relevant European policy objectives. In order to clarify the FP future evaluation in terms of objective achievements, the consistency of the WP objective structure should be increased.  

Other aspects identified as relevant for future FP evaluations, which METRONOME time frame and resources did not allow, were the following: Closer co-operation with technology platforms and EC officials in project evaluations, detailed investigations regarding follow-up research project paths, including transport projects commissioned under other programmes than transport (e.g. Information Society, Environment and Security) into the evaluation, and considering very carefully the point in time for FP evaluation.

 

To conclude, it seems that FP5 and FP6 have certainly played a significant role in the European science and technology agenda. For evaluation of the FPs’ role on the global map or on the contribution to EU research competitiveness at international level, the project sample does not give a representative insight. The finding of the METRONOME evaluation should, how-ever, be relevant to all national and European civil servants and researchers both within indus-try or public bodies and interested in the future of European transport research, both in the-matic and practical terms.