This study employed Bachman and Palmer's (2010) Assessment Use Argument framework to investigate to what extent the use of a second language oral test as an exit test in a Hong Kong university can be justified. It also aimed to help test developers of this oral test identify the most critical areas in the current test design that might need improvement. Candidates' oral responses to five integrated speaking tasks in this oral test were rated on five dimensions: Task fulfillment and relevance (TFR), Clarity of presentation (CoP), Grammar and Vocabulary (GV), Pronunciation (Pron), and Confidence and Fluency (CoFlu).

To provide backing for the meaningfulness of interpretations, confirmatory factor analysis (CFA) and item response theory (IRT) analyses were used to analyze 999 candidates' scores and raters' verbal reports were also analyzed to provide complementary information to the results of the quantitative analyses. Several CFA models were first tested and compared in terms of their statistical fit and substantive interpretability. And a graded response model was applied to the test data. The CFA results showed that the superior fit of the Higher-order trait-Uncorrelated Method model validated the test design, confirmed the current multicomponential view of language ability in the literature, and provided the most parsimonious explanation of the relationships among the five dimensions and overall speaking proficiency. The analytic scores were found to have much larger factor loadings on the trait factors than on the method factors, providing evidence that the component test scores could be meaningfully interpreted as indicators of the five dimensions. The presence of a higher-order speaking ability factor governing the five trait factors also supported the practice of reporting one composite score. Task Fulfillment and Relevance (TFR) measured on Task 4 had the highest method loading (.60) on Task 4 and the lowest trait factor loading (.36) on TFR, which suggested TFR4 might be too task specific and weak in measuring students' speaking ability to fulfill a speaking task in a relevant way. The trace lines of the graded response model also confirmed this. The raters' verbal reports showed that most raters did not have much difficulty differentiating across the performance levels. Hence, the problem of TFR4 can only be due to the nature of the task itself and its low discrimination. Both CFA and IRT results indicated that task types had great effects on test takers' speaking abilities especially TFR and that this language ability component might be too task specific.

In order to investigate the impartiality of interpretations, multi-group CFA and differential item functioning (DIF) were conducted to examine the extent to which the oral test had test bias and item bias across (1) gender and (2) disciplines. The multi-sample CFA results indicated that the factor structure was significantly different between males and females. However, the comparison of the factor loadings between females and males showed that only the factor loading of one item for the male group was significantly different from the female group at the 0.05 level. DIF results also suggested that the majority of the items displayed no DIF. The source of DIF may be attributed to the group mean difference on the latent trait and their real differences on certain aspects of language ability measured in this test. This provided backing for the impartiality of score interpretations, indicating that the rating-based interpretations from GSLPA SLT are impartial to a large extent across subgroups of test takers (males vs. females; business vs. non-business).

In order to examine the consistency of test scores, Generalizability theory (G theory) analyses were performed to investigate whether the test was dependable and whether the five dimensions were separable. G theory results showed that the phi coefficient for the whole test fell between .76 and .85 and Grammar and Vocabulary and Pronunciation proved to be the most dependable dimensions. G theory and CFA results both confirmed that the five speaking dimension were highly correlated with each other. The possible reasons of these findings were further discussed with reference to the raters' verbal reports.

Based on the above results, it can be concluded that the meaningfulness, impartiality, and consistency could be justified to a large extent. Some critical areas to be improved in the test design and administration were identified. Theoretical and practical implications were addressed and methodological limitations were also discussed. Overall, this study highlights the usefulness of Bachman and Palmer's Assessment Use Argument (2010) to justify the use of an existing language assessment.

This study is an attempt to measure the weighting patterns of the raters in a large-scale English as a Foreign Language (EFL) speaking test, classify these raters according to their weighting patterns, characterize the different types of raters in the rating process, and associate the rater types with different patterns of rater variability. The context was the Test for English Majors - Band 4, Oral Test (TEM4-Oral), a high-stakes certification test administered to college EFL majors in China toward the end of their sophomore year. To quantify the weighting patterns, 126 nonnative-speaking college teachers of English who served as TEM4-Oral raters in 2010 were requested to judge the EFL oral proficiency of 120 hypothetical test-takers with computer-generated score profiles featuring strengths and weaknesses in various criteria. Their relative weights on the criteria were derived from regression analyses, and then fed into cluster analyses to classify the raters into three types. To characterize different types of raters, a sample of 21 raters were involved in verbal protocols and requested to rate the performance of five real test-takers and justify their ratings. To associate the rater types with the different patterns of rater variability, the real ratings of 33 raters including all three types were analyzed through Many-Facet Rasch Measurement, Hierarchical Linear Modeling, Generalizability Theory, and Confirmatory Factor Analysis. The cluster analyses classified the raters into three types according to whether they gave the largest weights to form-related criteria or content-related criteria, or were balanced in the weighting patterns, and the three types were named form-oriented, content-oriented, and balanced respectively. In the verbal protocols, the form-oriented raters were found to be most severely subject to the anchoring and masking effects of pronunciation and intonation whereas the content-oriented raters displayed the strongest mitigation of such effects. The balanced raters came in between, but shared more similarity with the content-oriented raters. In association with rater variability, the form-oriented raters were found to be most severe among the three types and the content-oriented raters most lenient. On specific TEM4-Oral subscales, the form-oriented raters were unexpectedly severe in pronunciation and intonation, but unexpectedly lenient in grammar and vocabulary, whereas the content-oriented raters were unexpectedly lenient on the content-related subscale of discussion but unexpectedly severe on the subscale of grammar and vocabulary. However, no clear-cut relationship was found in reliability and restriction of range, and mixed results were reported in terms of halo effect.

At the University of California, Los Angeles, the Test of Oral Proficiency (TOP), an internally developed oral proficiency test, is administered to international teaching assistant (ITA) candidates to ensure an appropriate level of academic oral English proficiency. Test taker performances are rated live by two raters according to four subscales. While the subscale scores have potential as valuable feedback to major stakeholders, only a weighted average of the four subscale scores are currently reported and used. This study presents a way of extracting valuable information from the TOP subscale scores. In particular, it investigates an approach to obtaining oral English proficiency profiles based on the subscale score patterns of 960 TOP test takers.

This study utilized item response theory and finite mixture modeling to investigate profiles of academic oral English proficiency in terms of the TOP subscales. A higher-order generalization of the graded response model was formulated to estimate subscale scores that accounted for structural dependencies and rater effects in the observed TOP scores. The estimated scores were clustered using a multivariate normal mixture model to yield subscale score profiles. The mixture model suggested seven profile groups and classified the TOP test takers into the seven groups. The profile groups were then interpreted and labeled based on characteristic score patterns and linguistic background shared by group members.

To achieve a thorough understanding of the resulting profiles, discourse features of test taker performances sampled from different profile groups were closely examined. A small corpus was constructed based on the sampled test taker performances and compared to a reference corpus to explore the overall pattern of TOP test takers' language use. The comparison showed that the TOP test takers tended to use relatively fewer function words than speakers in the reference corpus. Characteristic features of each profile group's discourse were investigated through an identification and examination of discourse organizing lexical bundles. The results suggested that the use of metadiscourse and textual reference bundles with an explicit past reference point might be related to test takers' academic oral English proficiency.

One of the fundamental problems in language testing is the lack of adequate generalizability between what a test is measuring and what fulfills the learners' real world language use needs. It is important to recognize that no matter how precise a test measures a construct, if the way that a construct is defined and the way that test tasks are specified do not correspond to the domain of generalization in a meaningful way, test scores may never become adequate indicators of what learners can do with English in real life. This study investigated constructs and tasks of the General English Proficiency Test (GEPT) high-intermediate reading test to explicate the issues involved in generalizing test scores to non-test situations.

The study identified and demonstrated quantitatively and qualitatively the way and extent to which two distinct ways of conceptualizing reading constructs, the trait/curriculum-based and the task/domain-based approaches, could lead to divergent construct specifications, difficulty levels, item/text characteristics, and underlying factor structures, using approaches of expert judgments and confirmatory factor analysis. A total of 242 university students and six trained raters participated in the study. All the participants took the GEPT reading test and a task-based reading test developed based on the can-do statements in the Common European Framework of Reference (CEFR) and its designated Target Language Use (TLU) domains.

It was found that when items are more task-based and workplace specific, the less similarity they share with trait/curriculum based test items. The nature and the constituents of the reading comprehension construct shift. Not only do task-based and workplace specific items require a significantly higher amount of complex propositional content to be interpreted rather than recognized, they also demand a wider range and extent of language abilities (ideational, functional, and sociolinguistic) and strategic competence when making such interpretations in relation to context. Among all the combinations of language abilities, that of manipulative function and strategic demand appear to have the most effect on the complexity of reading construct. The ability to comprehend texts then is different from the ability to comprehend texts in context. The very nature of contextualization changes the nature and constituents of the comprehension construct.

Using Bachman and Palmer's (2010) Assessment Use Argument (AUA) framework, this study strongly suggests that the GEPT is not as meaningful or generalizable as the Language Training and Testing Center (LTTC) claims it is. GEPT test scores do not provide stakeholders with sufficient information about the ability to be assessed in the TLU domain, and the GEPT tasks do not have a sufficient degree of correspondence to the TLU tasks. Due to inadequate sampling of the target constructs and its task characteristics, GEPT test scores do not appear to generalize to performance in the target domain.

Research suggests that listener perceptions of a speaker's oral language use, or a speaker's comprehensibility, may be influenced by a variety of speaker-, listener-, and context-related factors. Primary speaker factors include aspects of the speaker's proficiency in the target language such as pronunciation and grammatical accuracy, and domain-related skills (e.g., teaching skills). Listener factors include proficiency in the target language, attitudes towards the speaker, familiarity with the speaker's native language or accent, familiarity with the speaker's topic, and familiarity with non-native speakers or speech in general. Finally, a variety of contextual factors may play a role including the norms of interaction or interpretation, speech form and content, and purpose of communication. Although previous research has identified many of these factors, none of the studies reviewed have attempted to integrate many of these factors into a larger conceptual model. Research that has examined the relationships between several of these factors has been limited by small sample sizes, constrained or inauthentic speech samples, and homogenous groups of speakers or listeners.

The primary purpose of this study was to examine the relationships among naïve listener perceptions of oral language use and speaker, listener, and contextual factors within a single model, using the framework of structural equation modeling. The conceptual model evaluated by this study may be viewed in several different ways: first, as an interactive model of oral language use centered on listener perceptions of the speaker; second, as a model of the relationships among teacher- and student-related factors that influence a teacher's comprehensibility to students in an academic domain.

The results of this study support an interactional perspective on oral language use which holds that both speaker- and listener-related factors influence comprehensibility. In this study, comprehensibility was predicted by speaker-related factors including components of oral proficiency and teaching effectiveness, and listener-related factors such as perceptions of the speaker's personality, attitude towards the speaker, and interest in the speaker's topic. The results have practical implications for language teaching and education policy, and emphasize an important consideration for oral proficiency assessments in this domain: providing sufficient information about "real world" constructs to make appropriate decisions.