INTRODUCTION

PART Ⅰ LITERATURE REVIEW

CHAPTER 1 LITER ATURE REVIEW I—LANGUAGE TESTING AND VALIDITY EVIDENCE

1.0 Introduclion

1.1 A Short History of Language Testing

1.2 Validity and Validation

1.3 Aspects of Validity Evidence

1.3.1 Theory-based Validity

1.3.2 Context Validity

1.3.3 Scoring Validity

1.3.4 Consequential Validity

1.3.5 Criterion-related Validity

1.4 Summary

CHAPTER 2 LLTERATURE REVIEW Ⅱ—PARAMETERS FOR A VALIDATION FRAMEWORK FOR AWT

2.0 Introduction

2.1 Test-taker Characteristics

2.2 Theory-based and Context Validity Evidence

2.2.1 The Definition of Academic Writing

2.2.2 The Cognitive Process Models in Writing

2.2.3 A Socio-cognitive Model of Writing:Process and Context

2.2.4 Text Types and Text Construction in Academic Writing

2.2.5 A Summary of Theoretical Models of Academic Writing

2.2.6 The Specific Parameters of Theory-based Validity and Context Validity

2.3 The Parameters of Sconng Validity Evidence

2.4 The Parameters of Consequential Validity Evidence

2.5 The Parameters of Criterion-related Validity Evidence

2.6 A Summary Prototype Framework for Validating Academic Writing Tests

2.7 The Situation of Academic Writing Tests at Tertiary Level in China

2.7.1 General Requirements for Academic Writing

2.7.2 Specifications for Academic Writing in TEM4 and TEM8

2.7.3 Additional Syllabus Requirements for the University Graduation Dissertation

2.8 General Research Question and Research Sub-questions

2.9 Implications of the Research

2.10 Summary

PART Ⅱ PILOT STUDIES

CHAPTER 3 THE DEVELOPMENTAND PILOTING OF INSTRUMENTS AND PROCEDURES FOR APPLYING THE VALIDATION FRAMEWORK TO TEM8E AND GRADUATION DISSERTATIONS

3.0 Introduction

3.1 Pilot Study One:The Development and Piloting of the Instrumentsand Procedures on Test-taker Characteristics

3.1.1 Sampling Participants

3.1.2 Establishment of Variables in Test-taker Characteristics

3.1.3 Development and Piloting of Instruments

3.2 Pilot Study Two:The Development and Piloting of the Instruments and Procedures for the Theory-based Validation of TEM8E and the Dissertation

3.2.1 Participants

3.2.2 Establishment of Variables in Theory-based Validity Evidence

3.2.3 Development and Piloting of Instruments

3.3 Pilot Study Three:The Development and Piloting of the Instruments and Procedures for the Context Validation of TEM8E and the Dissertation

3.3.1 Participants

3.3.2 Establishment of Variables in Context Validity Evidence

3.3.3 Development and Piloting of Instruments

3.4 Pilot Study Four:The Development and Piloting of the lnstruments and Procedures for the Scoring Validation of TEM8E and the Dissertation

3.4.1 Participants

3.4.2 Establishment of Variables in Scoring Validity Evidence

3.4.3 Development and Piloting of Instruments

3.5 Pilot Study Five:The Development and Piloting of the lnstruments and Procedures for the Consequential Validation of TEM8E and the Dissertation

3.5.1 Participants

3.5.2 Establishment of Variables in Consequential Validity Evidence

3.5.3 Development and Piloting of Instruments

3.6 Pilot Study Six:The Development and Piloting of the Instruments and Procedures for the Criterion-related Validation of TEM8E and the Dissertation

3.7 Summary

PART Ⅲ MAIN STUDY

CHAPTER 4 DISCUSSION OF THE RESULTS CONCERNED WITH THE EVIFENCE OF TEST-TAKER CHARACTERISTICS

4.0 Introduction

4.1 Sampling Participants

4.2 Discussion of the Results from Questionnaire A1(QA1)

4.2.1 Discussion of Learner Background Variables

4.2.2 Discussion of Motivation

4.2.3 Discussion of Orientation to the Language Study Context

4.2.4 Discussion of Learning Strategies

4.2.5 Discussion of Learning Approaches

4.3 Discussion of the Results Obtained from Questionnaire A2(QA2)

4.3.1 Discussion of Academic Writing Leaming Strategies

4.3.2 Discussion of Use/Process Strategies in Academic Writing

4.4 Summary

CHAPTER 5 DISCUSSION OF THE RESULTS CONCERNED WITH THEORY-BASED VALIDITY EVIDENCE

5.0 Introduction

5.1 Participants

5.2 Discussion of the Results from Questionnaire B1(QB1)

5.2.1 Statistical Analysis of the Results from Factor Analysis of QB1

5.2.2 Quantitative Analysis of the Results from Regression and ANOVA on QB1

5.2.3 Brief Qualitative Analysis of Academic Writing Process Feedback from QB1

5.2.4 Discussion of the Results from SEM on QB1 and QA2

5.3 Discussion of the Results from the Think Aloud Protocol Experiment

5.4 Discussion of the Results from Questionnaire B2(QB2)

5.4.1 Discussion of the Results from Factor Analysis of QB2

5.4.2 Discussion of the Comparison of the Responses to QB1 with those to QB2

5.5 Discussion of the Results from Focus Groups(FG)

5.5.1 Participants and Administration of FG

5.5.2 Discussion of the Feedback from FG

5.6 Discussion of the Feedback from Diary B

5.6.1 Participants and Administration of Diary B

5.6.2 Discussion of the Feedback from the Diary

5.7 Summary

CHAPTER 6 DISCUSSION OF THE RESULTS CONCERNED WITH CONTEXT VALIDITY EVIDENCE

6.0 Introduction

6.1 Participants

6.2 Discussion of the Results from Questionnaire C1a (QC1a)

6.3 Discussion of the Results from Questionnaire C1b(QC1b)

6.4 Discussion of the Results from Questionnaire C2a(QC2a)

6.5 Discussion of the Results from Questionnaire C2b (QC2b)

6.6 Discussion of the Results from Interview C1

6.7 Discussion of the Results from Interview C2

6.8 Summary

CHAPTER 7 DISCUSSION OF THE RESULTS CONCERNED WITH SCORING VALIDITY-,CONSEQUENTIAL VALIDITY-,AND CRITERION-RELATED VALIDITY EVIDENCE

7.0 Introduction

7.1 Scoring Validity Evidence

7.1.1 Participants

7.1.2 Discussion of the Results from Questionnaire D1(QD1)

7.1.3 Discussion of the Results from Questionnaire D2(QD2)

7.1.4 Summary of 7.1

7.2 Consequential Validity Evidence

7.2.1 Participants

7.2.2 Discussion of the Results from Questionnaire E1a(QE1a)

7.2.3 Discussion of the Results from Questionnaire E1b(QE1b)

7.2.4 Discussion of the Results from Questionnaire E2a(QE2a)

7.2.5 Discussion of the Results from Questionnaire E2b(QE2b)

7.2.6 Summary of 7.2

7.3 Criterion-related Validity Evidence

7.3.1 Participants

7.3.2 Discussion of the Results from the Correlations of the two Assessments

7.4 Summary of Chapter 7

CHAPTER 8 CONCLUSIONS AND IMPLICATIONS

8.0 Introduction

8.1 Research Questions Revisited

8.1.1 Test-taker Characteristics

8.1.2 Theory-based Validity

8.1.3 Context Validity

8.1.4 Scoring Validity

8.1.5 Consequential Validity

8.1.6 Criterion-related Validity

8.2 Limitations of the Study

8.3 Implications

8.3.1 Implications for Test Developers in General

8.3.2 Implications for Exit Assessment at Universities

8.3.3 Implications for University Students

8.4 Future Research Directions

8.5 Concluding Remarks

REFERENCES

APPENDICES

List of Tables

CHAPTER

Table 2.1 Definitions of learning strategies

CHAPTER

Table 3.1 Some demographical characteristics of the seniors at YNU in the pilot study

Table 3.2 The main demographical characteristics of 11 participants in TA experiment

Table 3.3 Transcription conventions (adapted from Bracewell and Breuleux 1994:59)

Table 3.4 Think-aloud essay marking scheme

Table 3.5 Cronbach alpha coefficients for marking essays in piloting TA

Table 3.6 Frequency of score distribution of theessay in piloting TA

Table 3.7 Pearson correlations between TEM8E and the dissertation at YNU in 2003

Table 3.8 Spearman rho between the ranks of TEM8E and the dissertation at YNU in 2003

Table 3.9.1 Instruments developed and piloted for test-taker characteristics

Table 3.9.2 Instruments developed and piloted for theory-based validity

Table 3.9.3 Instnunents developed and piloted for context validity

Table 3.9.4 Instruments and/or procedures developed and/or piloted forscoring,consequential and criterion-related validity

CHAPTER

Table 4.1 The demographic overview of student-participants in the main study(QA1.1.02,03,04,05,11)

Table 4.2 Data collection list in the Main Study on test-taker characteristics

CHAPTER

Tables 5.1-31 appear inAppendix 5.

Table 5.32 Data collection list in the Main Study on theory-based validity evidence

CHAPTER

Tables 6.1-4 appear in Appendices 6.1-6.

Table 6.5 Data collection list in the Main Study on context validity evidence

CHAPTER

Tables 7.1-11 appear in Appendices 7.1-7.

Table 7.12 Data collection list in the Main Study on scoring-,consequential-,and criterion-related validity evidence

List of Figures

CHAPTER

Figure 2.1 The external characteristics of test-takers (O'Sullivan 2000;O'Sullivan and Weer 2002)

Figure 2.2 Structure of the writing model(Flower and Hayes 1981:370)

Figure 2.3 Structure of the knowledge-telling model(Bereiter and Scardamalia 1987:8)

Figure 2.4 Structure of the knowledge-transforming model(Bereiter and Scardamalia 1987:12)

Figure 2.5 Model of writing as communicative languageuse(Grabe and Kaplan 1996:226)

Figure 2.6 Elements of text structure(Grabe and Kaplan 1996:63)

Figure 2.7 A model of text construction(Grabe and Kaplan 1996:81)

Figure 2.8 A summary model of academic writing

Figure 2.9 Theory-based validity and context validity(adapted from Weir 2005:47)

Figure 2.10 Components of language knowledge(adapted from Bachman 1990:87;Bachman and Palmer 1996:68)

Figure 2.11 Sub-framework of scoring validity(Weir 2005:47)

Figure 2.12 Sub-framework of consequential validity(adapted from Weir 2005:47)

Figure 2.13 Sub-framework of criterion-related validity(Weir 2005:47)

Figure 2.14 A socio-cognitive framework for validating writing tests(adapted from Weir 2005:47)

CHAPTER

Figure 3.1 Q-Q Plot of essay scores from 11 participants in the TApilot

CHAPTER

Figure 4.1 Distribution of participants:Gender by University(QA1.1.01)

Figure 4.2 Distribution of participants:Ageby University(QA1.1.03)

Figure 4.3 Distribution of participants:Region by University(QA1.1.04)

Figure 4.4 Distribution of participants:Type of secondary school by University(QA1.1.05)

Figure 4.5 Distribution of participants:Beginning time of studying English by University(QA1.1.11)

Figure 4.6 Linear and curvilinear regression on QA1.3.08 'If we had no tests,I think I would actually learn more'in the variable of test-related anxiety in orientation

Figure 4.7 Linear and curvilinear regression on QA1.3.09 'During importanttests,I often feel so nervous that I often forget what Ireally know'in the variable of test-related anxiety in orientation

Figure4.8 Linear and curvilinear regression on QA1.3.10'Even when I am well prepared for a test,I feel very worried aboutit'in the variable of test-related anxiety in orientation

CHAPTER

Figure 5.1 Scree plot for component number on QB1 in Factor analysis

Figure 5.2 The model for causal relationships between academic writing strategies,sub-processes and score gains

Figure 5.3 Scree plot for component number on QB2 in Factor analysis