|
|
Tweet |
|
The Effect of Scoring Order on the Independence of Holistic and Analytic Scores
By: Nancy Robb Singer, Paul LeMahieu
Publication: The Journal of Writing Assessment
Date: July 19, 2012
Summary: This research explores the matter of independence of scores and the effects of scoring order upon those judgments, affirming that when readers will be asked to score both ways, holistic scoring should precede analytic scoring. It also suggests interesting insights into the cognitive processes engaged by scorers as they score holistically and analytically.
Excerpt from Article
When assessing student writing, there are often legitimate reasons for seeking both a holistic score as well as a set of analytic scores. Holistic scoring refers to assessments in which a single summary judgment of quality is rendered, albeit often guided by a conceptual framework that articulates essential dimensions upon which quality is to be defined (Huot, 1988; White, 1984). In analytic scoring, writing is described in terms of essential attributes and individual judgments recorded for several constituent attributes. Holistic scoring is thought by some to enable a more complete and appropriate depiction of a phenomenon as complex as writing, which they feel can never be adequately deconstructed into several parts (White, 1994). Moreover, singular summary judgments may fulfill certain assessment purposes perfectly well, while being quicker and, therefore, more economical (Spandel & Stiggens, 1980). Analytic scoring, on the other hand, may be considered more informative and therefore potentially relevant to instructional and programmatic decision making, though it is more time consuming and therefore generally more expensive to conduct (Cooper & Odell, 1977; Rabianski, 1979).
As a practical matter, instructional interests coupled with economic or logistic concerns often dictate that both types of scores be obtained in a single common scoring session. A recent survey of state writing assessment systems in the United States (National Writing Project [NWP], 2008b) shows that 46 of the 50 states have direct writing assessments. Of these, 32 (67%) assign holistic scores and 21 (44%) assign analytic scores. Seven states assign both types of scores in their writing assessment systems. When both kinds of scores are desired, questions reasonably arise regarding the independence of the scores obtained. Because holistic and analytic scores have very different conceptual meanings, it is essential to the validity of each type of score evidence that it not be unduly influenced by the other. For example, to those deeply concerned about holistic assessment, such scores are not merely the aggregate (be it a sum or average) of some set of analytic scores. Rather, a holistic score is conceived of as something qualitatively different than the mere aggregate of many parts. The written product may weave those parts together in a way that creates a greater (or conceivably lesser) whole.
To preserve the greatest degree of independence of holistic judgments, rational analysis would suggest that the "whole" of the writing should be judged first before scoring component parts through analytic scoring. However, little is known about the effects of scoring order on the scores obtained or if true holistic scoring is even possible from the mind of a scorer who has already been trained to and will be asked to provide analytic scores at the same time. This research explores the matter of independence of judgments and the effects of scoring order upon those judgments.
Copyright © 2011 The Journal of Writing Assessment. Reprinted with permission.
Singer, Nancy Robb and Paul G. LeMahieu. 2011. "The Effect of Scoring Order on the Independence of Holistic and Analytic Scores." The Journal of Writing Assessment 
.
Download the Article
Download "The Effect of Scoring Order on the Independence of Holistic and Analytic Scores "
