New Activity Launched at ELRA, European Language Resources Association and ELDA, Evaluations and Language resources Distribution Agency

ELRA & ELDA to devote more resources to the evaluation

of Human Language Technologies (HLT)

If at the very beginning the main activity of ELRA & ELDA in the framework of the evaluation task was to supply the language resources appropriate for testing and evaluation, both are now getting involved in the evaluation process itself, the evaluation of products, systems, and applications developed for HLT.

The evaluation has become a major activity in the field of HLT. This activity is highly critical, as its main objectives are to check the quality of the developed applications and systems, and ensure that these are ready for the market. To evaluate a specific technology means measuring the progress achieved, comparing different approaches to a given problem, and choosing the best solution, assuming that its advantages and disadvantages have been analysed. The evaluation also involves the assessment of the availability of technologies for a given application, product benchmarking, and assessment of system usability and user satisfaction.

ELDA actively participates in evaluation projects, at French (e.g. Amaryllis), European (e.g. CLEF, Cross-Language Evaluation Forum), and world (e.g. Aurora) levels.

The main objective of the Amaryllis project is to evaluate information retrieval systems with French text documents (corpora and questions and answers), and to provide a common methodology for the evaluation of such search tools. The corpora designed for these purposes are available in the ELDA catalogue, at www.elda.fr/catalog.html.

The main task of the CLEFproject consists of providing an infrastructure for the evaluation of information retrieval systems that operate on European languages, in multilingual, monolingual, and cross-language contexts. Test-suites of reusable data will also be developed for benchmarking purposes.

In the framework of CLEF, ELDA is in charge of conducting the user needs survey, identifying the data and negotiating their distribution rights with the owners, and participating in the production of an exit plan that will set some evaluation procedures and recommendations regarding the evaluation of multilingual and cross-lingual systems.

As for the Aurora project, it aims at establishing a worldwide standard for the feature extraction software in a DSR (Distributed Speech Recognition) system (evaluation of algorithms for front-end feature extraction in background noise on one hand ; on the other hand, the evaluation and comparison of the performance of speech recognition algorithms’ robustness. The databases which have been created in the framework of Aurora are distributed by ELDA.

Indeed, several resources developed in the framework of these evaluation projects are already available in our catalogue, which may be consulted at the following address : www.elda.fr/rubrique6.html.

In the near future, as we are getting more and more involved in the evaluation activity, ELRA & ELDA will add in its catalogue further resources and tools related to evaluation, and a new team, whose task will include the organisation of evaluation campaigns and every other aspects of the evaluation activity, will join the agency.