1 SPEECH AND SPEAKER RECOGNITION EVALUATION&Sadaoki Furui

1．Introduction

2．Principles of Speech Recognition

3．Categories of Speech Recognition Tasks

4．Evaluation of Speech Recognition Systems

5．Principles of Speaker Recognition

6．Categories of Speaker Recognition Tasks

7．Normalization and Adaptation Techniques

8．Evaluation of Speaker Recognition Systems

9．Factors Affecting the Performance and Evaluation Paradigm Design for Speech and Speaker Recognition Systems

10．System-Level Evaluation of Speech and Speaker Rognition

11．Conclusion

References

2 EVALUATION OF SPEECH SYNTHESIS&Nick Campbell

1．Introduction

2．Components of Computer Speech

3．Evaluation Methodologies

4．Organised Evaluations and Assessment

5．Speaking to (and on Behalf of) People

6．Conclusion

References

3 MODELLING AND EVALUATING VERBAL AND NON-VERBAL COMMUNICATION IN TALKING ANIMATED INTERFACE AGENTS&Bj？rn Granstr？m and David House

1．Introduction

2．KTH Parametric Multimodal Speech Synthesis

3．Data Collection and Data-Driven Visual Synthesis

4．Evaluating Intelligibility and Information Presentation

5．Evaluating Visual Cues for Prominence

6．Evaluating Prosody and Interaction

7．Evaluating Visual Cues to Sentence Mode

8．Evaluation of Agent Expressiveness and Attitude

9．Agent and System Evaluation Studies

10．Future Challenges in Modelling and Evaluation

References

4 EVALUATING PART-OF-SPEECH TAGGING AND PARSING&Patrick Paroubek

1．POS Tagging

2．Parsing

3．Evaluation and Natural Language Processing

4．POS Tagging Evaluation Methodology

5．Methodology and Evaluation Measures for Parsing

6．Conclusion

References

5 GENERAL PRINCIPLES OF USER-ORIENTED EVALUATION&Margaret King

1．A Historical Note

2．What is User-Oriented Evaluation?

3．A First Principle: Quality is Decided by Users

4．A Second Principle: Users do not Have the Same Needs

5．A Third Principle: Quality can be Characterized

6．A Fourth Principle: Quality can be Measured

7．Combining the Particular and the General: The Ideal

8．Conclusion

References

6 AN OVERVIEW OF EVALUATION METHODS IN TREC AD HOC INFO-RMATION RETRIEVAL AND TREC QUESTION ANSWERING&Simone Teufel

1．Introduction

2．Evaluation Criteria

3．Evaluation Metrics

4．Real-World Performance

5．Conclusion

References

7 SPOKEN DIALOGUE SYSTEMS EVALUATION&Niels Ole Bernsen,Laila Dybkj？r and Wolfgang Minker

1．Introduction

2．Evaluation Methods and Criteria

3．Evaluation of the NICE Hans Christian Andersen Prototype

4．Evaluation of the SENECA Prototype

5．Conclusion

References

8 LINGUISTIC RESOURCES,DEVELOPMENT,AND EVALUATION OF TEXT AND SPEECH SYSTEMS&Christopher Cieri

1．Introduction

2．The Linguistic Resource Landscape

3．Background on Linguistic Data and Annotation

4．Data Planning for Technology Development and Evaluation

5．Finding Resources

6．Building Resources

7．Conclusion

References

9 TOWARDS INTERNATIONAL STANDARDS FOR LANGUAGE RESOURCES&Nancy Ide and Laurent Romary

1．Introduction

2．Background

3．The Linguistic Annotation Framework

4．Putting it All Together

5．Conclusion

References

Index