TC-Star Evaluation Information (WP4)

2nd Campaign Events

Date	Event
2006-02-01	Beginning of ASR Run
2006-02-12	End of ASR Run - output sent back to ELDA - word graphs available to SLT
2006-02-15	Preliminary ASR results sent to participants + reference transcriptions
2006-02-24	Final ASR results sent to participants
2006-02-15	Beginning of SLT Run
2006-03-01	End of SLT Run - output sent back to ELDA
2006-03-05	Preliminary SLT automatic results sent to participants
2006-03-15	Final SLT results sent to participants
2006-02-24	Beginning of TTS Run - Voice conversion evaluation
2006-02-27	Beginning of TTS Run - Prosody module, Acoustic module, Expressive Speech, TTS component evaluations
2006-03-01	Beginning of TTS Run - Text processing module evaluation
2006-03-02,03	End of TTS Run - Text processing, Prosody, Acoustic, Voice conversion, Expressive Speech, TTS component evaluations - Output sent back to ELDA
2006-03-03	Beginning of TTS Run - End-to-End evaluation
2006-03-03,04	End of TTS Run -End-to-End evaluation - output sent back to ELDA
2006-03-08	ELDA conducts Subjective tests
2006-03-14	End of Subjective tests
2006-03-17	TTS Results sent to participants- End-to-End evaluation , Prosody, Acoustic, Voice conversion, Expressive Speech, TTS component evaluations
2006-03-31	TTS Results sent to participants - Text processing evaluation
2006-06-19,20,21	TC-STAR Evaluation Workshop in Barcelona

 

Tasks

DONE

IN PROGRESS

FUTURE

 

Task	start	end	Ongoing
ASR
Packaging raw Dev audio data	18 Oct 05	20 Oct 05	Done.
ELDA send Dev audio data	20 Oct 05	20 Oct 05	Done.
Spanish Parliament Dev (stm + uem)	19 Oct 05	10 Nov 05	Done
English EPPS Dev (stm + uem)	18 Oct 05	10 Nov 05	Done
Spanish EPPS Dev (stm + uem)	18 Oct 05	10 Nov 05	Done
Validation of Dev sets	11 Nov 05	18 Nov 05	Done
EPPS English Test	14 Nov 05	25 Nov 05	Done
EPPS Spanish Test	14 Nov 05	25 Nov 05	Done
Spanish Parliament Test	28 Nov 05	23 Dec 05	Done
Validation Test data	21 Dec 05	31 Dec 05	Done
ELDA send Test Data (audio + uem)	01 Feb 06	01 Feb 06	Done
Participants run	01 Feb 06	12 Feb 06	Done.
Evaluation run	12 Feb 06	15 Feb 06	Done.
Final results	15 Feb 06	24 Feb 06	Done.
SLT
Mandarin VoA FTE Dev (src+ref)	17 Oct 05	10 Dec 05	Done.
EPPS English FTE Dev (src+ref)	17 Oct 05	25 Nov 05	Done.
EPPS Spanish FTE Dev (src+ref)	17 Oct 05	25 Nov 05	Done.
Spanish Parliament Verbatim Dev (src+ref)	17 Oct 05	25 Nov 05	Done.
EPPS English Verbatim Dev (src+ref)	14 Nov 05	10 Dec 05	Done.
EPPS Spanish Verbatim Dev (src+ref)	14 Nov 05	10 Dec 05	Done.
Mandarin VoA Test (src+ref)	12 Dec 05	13 Jan 05	Done.
EPPS English FTE Test (src+ref)	12 Dec 05	13 Jan 05	Done.
EPPS Spanish FTE Test (src+ref)	12 Dec 05	13 Jan 05	Done.
Spanish Parliament FTE Test (src+ref)	12 Dec 05	13 Jan 05	Done.
Spanish Parliament Verbatim Test (src+ref)	12 Dec 05	13 Jan 05	Done.
EPPS English Verbatim Test (src+ref)	12 Dec 05	13 Jan 05	Done.
EPPS Spanish Verbatim Test (src+ref)	12 Dec 05	13 Jan 05	Done.
ELDA send Dev data	18 Nov 05	10 Dec 05	Done.
ELDA send Test data	15 Feb 06	15 Feb 06	Done.
Participants run	15 Feb 06	01 Mar 06	Done.
Automatic evaluation	01 Mar 06	05 Mar 06	Done.
Final results	05 Mar 06	15 Mar 06	Done.
TTS
TTS - End-to-end
ELDA sends Test data	03 Mar 06	03 Mar 06
Output sent back to ELDA	03 Mar 06	04 Mar 06
Subjective tests	08 Mar 06	14 Mar 06
Results	15 Mar 06	17 Mar 06
TTS - Text processing module
ELDA sends Test data	01 Mar 06	01 Mar 06	Done.
Output sent back to ELDA	01 Mar 06	02 Mar 06	Ongoing
Evaluation	03 Mar 06	10 Mar 06
Primary results	10 Mar 06	10 Mar 06
Adjudication	13 Mar 06	24 Mar 06
Final results	31 Mar 06
TTS - Prosody module
ELDA sends Test data	27 Feb 06	27 Feb 06	Done.
Output sent back to ELDA	27 Feb 06	02 Mar 06	Ongoing
Subjective tests	08 Mar 06	14 Mar 06
Final results	15 Mar 06	17 Mar 06
TTS - Acoustic synthesis module
ELDA sends Test data	27 Feb 06	27 Feb 06	Done.
Output sent back to ELDA	27 Feb 06	02 Mar 06	Ongoing
Subjective tests	08 Mar 06	14 Mar 06
Final results	15 Mar 06	17 Mar 06
TTS -Voice conversion
ELDA sends conversion directions	24 Feb 06	24 Feb 06	Done.
ELDA sends Test data	01 Mar 06	01 Mar 06	Done.
Output sent back to ELDA	01 Mar 06	03 Mar 06	Ongoing
Subjective tests	08 Mar 06	14 Mar 06
Final results	15 Mar 06	17 Mar 06
TTS -Expressive speech
ELDA sends Test data	27 Feb 06	27 Feb 06	Done.
Output sent back to ELDA	27 Feb 06	02 Mar 06	Ongoing
Subjective tests	08 Mar 06	14 Mar 06
Final results	15 Mar 06	17 Mar 06
TTS - Whole TTS component
ELDA sends Test data	27 Feb 06	27 Feb 06	Done.
Output sent back to ELDA	27 Feb 06	02 Mar 06	Ongoing
Subjective tests	08 Mar 06	14 Mar 06
Final results	15 Mar 06	17 Mar 06