Evaluation started some time ago by supplying language resources for evaluation projects (Amaryllis, Aurora, CLEF). Since then, our evaluation activities have been extended to include the management of projects dedicated to the evaluation of HLT systems and products.

Evaluation is important to the Language Engineering industry on many levels. It enables research and development teams to validate research hypotheses and assess progress and system development. It also identifies promising research directions or technology with a view to bringing it to market. Evaluation also enables funding agencies to determine whether their investment has led to significant progress. Other advantages of evaluation campaigns include the production of high quality training/test data, evaluation software, methodologies, metrics and protocols, all of which may be made available in the form of evaluation packages and distributed in the same way as traditional language resources. Such evaluation packages would enable all research teams in a field to compare and benchmark their systems.

Evaluation may be carried out on many levels :

  Basic Research Evaluation (validating research directions)
  Technology Evaluation (assessment of solutions for well defined problems)
  Usage Evaluation (end-users in the field)
  Impact Evaluation (socio-economic consequences)
  Programme Evaluation (funding agencies)

All information available on HLT Evaluation can be found on the ELRA HLT Evaluation Portal.