This section only summarises the more complete ASR Evaluation Plan document to be found here.
TC-STAR Evaluation Run #1 for ASR will take place from March 1, 2005 to March 25, 2005. The complete schedule can be seen here, but we can outline the important dates for ASR:
- Last week of February: ELDA sends a DVD with audio test data to participants.
- March 1, 2005: Beginning of ASR run by participants
- March 11, 2005: End of ASR run - participants send their output to ELDA
- March 18, 2005: End of ASR scoring by ELDA - results are sent to participants
- March 19, 2005: Beginning of adjudication phase
- March 25, 2005: End of adjudication phase - results are definitive
Before the proper evaluation run, participants have access to training data and development data composed of audio files [DVD sent by RWTH? Sent by ELDA? or download?] and transcriptions. See below training data and development data.
ASR evaluation will be run in 3 languages: English, Spanish and Chinese Mandarin.
English and Spanish are run on recordings from the European Parliament Plenary Sessions (EPPS), while Chinese is run from recordings of Voice of America.
- Declare your interest to the project's coordinator.
- ELDA send you an End-Used Agreement to have access to the data. You must fill in, sign and return this agreement to ELDA.
- Get access to training and development data:
- EPPS training and development audio files will be sent to you by RWTH
- EPPS training and development transcriptions can be downloaded (see section training data and development data)
- Chinese training data can be any of those listed as training in section Resources.
- Chinese development and test audio files have to be bought to LDC.
- Chinese development transcriptions can be downloaded (see section development data)
- ELDA will contact you as any other participant for the evaluation run.
- You must release the data after the end of the evaluation run, as said in the agreement.