Jinyan Huang

Turgay Han

In the assessment of ESL/EFL writing, there has been much debate about the differences between holistic and analytic scoring. Empirical studies reported mixed results regarding the impact of scoring methods on the variability and reliability of ESL/EFL writing scores. These mixed research findings continue to raise a dilemma for institutional writing assessment policy makers in deciding an appropriate and effective method to score ESL/EFL students’ essays. Using generalizability (G-) theory as a theoretical framework, this study was intended to provide a solution to this dilemma through the investigation of the rating variability and reliability of EFL writing by undergraduate students at a Turkish university. Nine EFL essays written by Turkishspeaking students were scored first holistically and then analytically by five raters. The results showed greater rater variation for holistic scores than for analytic scores of EFL papers. Further, there was a large difference in the G-coefficients between holistic (with a G-coefficient of .64) and analytic scoring (with a G-coefficient of .90) and this difference had tremendous impact on the reliability of holistic scoring of EFL papers. The findings of this study provide evidence for policy makers that analytic scoring is more appropriate and effective than holistic scoring in institutional English writing assessment context. Important implications are discussed.

Kaycheng Soh

Different democracy ranking systems have different conceptualizations of ‘quality of democracy’ using different sets of indicators. The systems vary in approach on a minimalist-maximalist continuum. This study reanalyzed the data of 94 countries common to the Democracy Index, Democracy Ranking, and World Governance Index for 2011. It looked into the factor structures of each system by pooling the data of the three systems. When considered separately, the three systems all showed internal cohesiveness in terms of factor structure. However, the indicators showed distinctive nature with one focusing on political processes, another on social benefits, and a third on political atmosphere. It is argued that combining the indicators and calling the outcome ‘democracy ranking’ causes confusion when comparing rankings across systems. It is suggested that the three aspects of Polity (of-democracy), Benefits (for-democracy), and Atmosphere (by-democracy) need be kept distinct when describing and comparing countries on democracy.

Thomas J. Sheeran

Vincent J. Rinaldo

Jason Blokhuis

Robert Michael Smith

Walter S. Polka

Sylvia Valentin

Although there seems to exist a consensus regarding what constitutes the elements of “good teaching,” qualities such as curriculum in the content areas have been traditionally safeguarded by academic freedom. Issues related to grading, program acceptance, program continuation and recommendation for licensure were matters for institutional determination not judicial intervention. Recently, however, American courts have signaled a change in judicial oversight of these decisions that will force teacher preparation programs to re-examine the ways and means that “intangible” expectations of candidates can be measured in a legally defensible fashion. This paper seeks to provide some insight and understanding of means by which pre-service teacher candidates’ dispositions can be measured and evaluated in a legally defensible fashion.

S T. Mulder

HUANG, JINYAN. Fairness in Large-scale ESL Writing Assessments: Investigating Evidence through Generalizability Theory. Saarbrücken: Lambert Academic Publishing, 2012. Pp. 236. $99.00, paper. ISBN: 978-3-659-19364-4.

UI Research Center

UI JOURNALS
With the goal of advancing research in the social sciences, Untested Ideas (UI)Research Center provides the researchers and scholars worldwide with resources,grants, and academic exchange platforms and channels through research interestgroups, memberships, funded research projects, annual conferences, trainingworkshops, and publications. The mission of UI Research Center is to serve the socialsciences researchers and scholars in the world and aim to promote advanced andcutting-edge research methodology, publish investigations on new and untested ideas,and disseminate research findings that make original and significant contributions tothe social sciences.