UC Berkeley

Mixed item formats are in extensive use in large-scale assessment today and widely accepted as a means to improve assessment validity. Many studies have investigated the differential effect of Item Format on gender and ethnic subgroups in mathematics, yet few of these studies have attended to their impact on students with limited language proficiency and linguistic abilities in the area of reading. As educational policy increasingly mandates the inclusion of minorities such as English Language Learners (ELL) and students diagnosed with Specific Learning Disability (SLD) in federal and state assessment, many question the validity of achievement test scores because the degree to which the test score is a function of language proficiency is not clearly understood (Mahoney, 2008). To fill that research void, this study investigated Item Format effect and its interaction with (a) group membership as an ELL and/or SLD student and (b) assessment format differences related to genre and reading content. The software program ConQuest (Wu et al., 2007) was used to conduct Item Response Modeling on the 2007 NAEP 4th grade reading achievement data. Analysis demonstrated an overall hierarchy of Item Format difficulty. Contrary to expectation, DIF analysis demonstrated that flagged CR items favored both ELL and SLD students, while flagged MC items generally favored the Non-SLD group. But, multidimensional regression analysis demonstrated that there is no interaction between Item Format and group membership because the effect was virtually equivalent across MC and CR formats for the reference and focal groups. In looking to the future of assessment design, more research is needed to fully understand how Item Format differences contribute to assessment difficulty, the limited application of various Item Formats, specifically how they are suited to particular content, and how to blend Item Formats in a manner that utilizes their unique benefits while also producing fair results (Hastedt & Sibberns, 2005).