Chapter 1 Introduction

1.1 Rationale for the study

1.2 Objectives of the study

1.3 Organization of the thesis

1.4 Definition of terms

1.4.1 Online

1.4.2 Marking

1.4.3 Online marking

1.4.4 Online Marking System(OMS)

1.4.5 Local Area Network(LAN)

Chapter 2 Research Questions and Methodology of the Study

Chapter 3 Issues in the Direct Testing of EFL/ESL Writing Ability

3.1 Introduction

3.2 What is a direct writing test?

3.3 EFL/ESL writing ability:What shall we test?

3.4 Issues in validity

3.4.1 What is validity?

3.4.2 Types of validity

3.5 Issues in reliability

3.5.1 What is reliability?

3.5.2 Methods of judging reliability of writing assessments

3.6 The relationship between validity and reliability

3.7 Four components of a direct writing test

3.7.1 The task

3.7.2 The writer

3.7.3 The scoring procedure

3.7.4 The rater

3.8 Washback

3.8.1 Washback in general

3.8.2 Washback of direct tests of writing

3.9 Practicality

3.10 Summary

Chapter 4 The CET Writing Test

4.1 Introduction

4.2 The writing test required by the CET

4.2.1 A direct test

4.2.2 Positive washback

4.3 The scoring of CET compositions

4.3.1 The scoring approach currently adopted

4.3.2 Procedures involved in scoring CET essays

4.3.2.1 Scoring Principles and Marking Scheme

4.3.2.2 Range-finders and sample essays

4.3.2.3 Rater training

4.3.2.4 Rating process

4.3.2.5 Monitoring raters'scoring during the scoring sessions

4.3.2.6 Recording essay scores

4.4 Computer-aided adjustment of writing scores

4.5 Discussion

Chapter 5 The First Experimental Study

5.1 Introduction

5.2 Compositions

5.3 Participants

5.4 Data collection procedure

5.5 The introspection and retrospection studies

5.5.1 Introduction

5.5.2 Data elicitation

5.5.3 Tape transcription

5.5.4 Data analysis

5.6 The questionnaire studies

5.6.1 Design of the questionnaires

5.6.2 Analysis of questionnaire responses

5.7 Findings from the introspection,retrospection and questionnaire studies

5.7.1 Issues and problems in rating CET essays online

5.7.2 Decision-making behaviors while rating CET-4 essays

5.7.3 Summary of comments made by the raters on essays

5.7.3.1 Overall summary

5.7.3.2 Variations in raters'comments

5.7.4 Essay elements'influences on raters'decision-making

5.7.5 Elements of good CET essays in the raters'eyes

5.8 Analysis of writing scores

5.9 Summary and discussion

5.9.1 About the issues and problems involved

5.9.2 About the raters'scoring decisions

5.9.3 About the writing scores

Chapter 6 The Second Experimental Study

6.1 Introduction

6.2 Compositions

6.3 Participants

6.4 Data collection procedure

6.5 Problems encountered

6.6 Data analysis

6.7 Results

6.8 Summary

Chapter 7 Design of the OMS

7.1 Introduction

7.2 Literature review on online marking of compositions

7.2.1 Automated scoring of essays

7.2.1.1 Overview of four major automated scoring methods

7.2.1.2 Analysis of the four major automated scoring methods

7.2.1.3 Summary

7.2.2 Online scoring of essays by human raters

7.2.2.1 Overview of online scoring of essays by human raters

7.2.2.2 Empirical research on online scoring of essays by human raters

7.2.2.3 Summary

7.3 A preliminary model of marking essays online

7.4 Overview of the CET Online Marking System(OMS)

7.4.1 The data management module

7.4.1.1 Basic information management

7.4.1.2 Essay management

7.4.1.3 Search and report

7.4.2 The training module

7.4.3 The rating module

7.4.4 The monitoring module

7.5 Operation of the OMS and the rater interface

7.5.1 Overview of the operation of the OMS

7.5.2 The OMS rater interface

7.6 Main features of the CET OMS

7.6.1 Random distribution of scripts

7.6.2 Efficient score recording

7.6.3 Online real-time monitoring of scoring

7.6.4 Quality control of raters

7.6.4.1 Adherence to the CET Scoring Principles and Marking Scheme

7.6.4.2 Rater training

7.6.4.2.1 Compulsory training

7.6.4.2.2 Individual rater's self training

7.6.4.2.3 Forced training

7.6.4.3 Online discussion

7.6.4.4 Back-reading and score revising

7.6.4.5 Time control

7.7 Advantages of the CET OMS

7.7.1 Real and efficient random distribution of scripts at the national level

7.7.2 Real-time online monitoring of raters

7.7.3 Assured quality control of scoring

7.7.4 Overall efficiency

7.7.5 Efficient and economical storage of scripts

7.7.6 Express retrieval of scripts and scores

7.7.7 Efficient management and potential utilization of test data for research

7.8 Limitations of online scoring and solutions

7.9 Summary

Chapter 8 The Third Experimental Study

8.1 Context of the experiment

8.2 Participants

8.3 Compositions

8.4 Data collection

8.4.1 Step 1:Online marking

8.4.1.1 The first round online marking

8.4.1.2 The second round online marking

8.4.2 Step 2:Conference marking

8.5 Data analysis

8.6 Results

8.7 Summary and discussion

Chapter 9 Data Analysis Using FACETS

9.1 FACETS and method

9.2 The first approach:comparison of rater severity and consistency from the online setting and the conference setting

9.2.1 Rater severity and consistency:the online setting

9.2.1.1 Rater severity:the online setting

9.2.1.2 Rater consistency:the online setting

9.2.2 Rater severity and consistency:the conference setting

9.2.2.1 Rater severity:the conference setting

9.2.2.2 Rater consistency:the conference setting

9.2.3 Comparison of rater severity and consistency in two settings

9.2.4 Comparison of rater severity change between two settings

9.3 The second approach:bias analysis

9.3.1 Bias analysis:rater by essay interactions

9.3.2 Bias analysis:rater by setting interactions

9.4 Conclusion

9.5 Discussion

Chapter 10 Summaries,Discussions,Implications and Recommendations

10.1 A refined model of online scoring of CET essays and its main features

10.2 Benefits proceeding from online scoring

10.3 Practicality

10.4 Scoring quality

10.5 Raters'comments

10.6 Suggestions for the improvement of the Online Marking System

10.7 Implications for other writing tests

10.8 Suggestions and recommendations for future research

10.8.1 Suggestions for future research in online marking of compositions

10.8.2 Recommendations for future research in EFL writing assessment

10.9 Theoretical and practical significance of the study

References

Appendices

后记