new SAT I writing section - biased and unreliable??

College Search & Selection
1

<p>I would be interested in hearing opinions about the new SAT I writing section that was introduced this year. Students who took the new test don't yet know their scores. The SAT II has had a Writing test that has been required by some colleges for several years. I personally will not have to take the new SAT but I question the validity of these tests, the scoring methods, and the potential for bias. </p>

<p>How are these tests scored? How can 300,000 handwritten essays be scored with validity in a few weeks? Will scores be lower for essays that do not follow a formula? that show creativity? that come from students with bad handwriting? I am especially concerned that scoring will favor students with nice handwriting. How are the readers trained? How reliable are the readers? </p>

<p>For those of you who have taken the SAT II writing, was your score an accurate reflection of your writing skill? How well do the SAT writing scores agree with the SAT verbal score?</p>

<p>I have heard that some colleges are ignoring the writing section score. True?</p>

2

<p>I took the SAT II Writing and received an 800 and a subscore of 10 on the essay (max = 12). I did not think that the essay I shovelled together in twenty minutes was an accurate reflection of my writing ability. I'm the type of person who begins thinking about writing my essay as soon as I get the assignment. When I begin writing I take a minimum of three hours, and a day later I read over my essay and revise. I cannot even begin to piece together a well-formed, impromptu essay in twenty minutes. I realize that graders take the extreme time limit into consideration, but I still think my essay did little to distinguish me from any other test-taker.</p>

<p>I really would not be surprised if it were found that scorers favored students with better handwriting. They're going to be grading a LOT of essays written by students with poor penmanship; when a student with legible handwriting comes along they're probably going to feel very relieved. </p>

<p>The SAT essay is only going to test basic fluency and help colleges determine the sizes of their remedial writing classes, IMO. Overall, the writing test is not a great idea, but all standardized tests have some inherent flaws.</p>

3

<p>I got an 800 writing as well. I have a very skewed view on my writing skill. I am a science student at a school known for "breeding" some of the best writers in the nation. I pretty much always get Bs in english. That being said, last summer when I was at a journalism program at Columbia (just so you know, I do photography and layout) I had to write some articles, and the teacher told me I was one of the best writers in the class, which was for people who were gonna be the Editor in Chief during the school year. ( I was the only one from my school to go, so they put me into the EiC class instead of a layout class where I should have been)</p>

<p>Also, what makes you think the new SAT will be any different from the old SAT and SATII writing. Maybe I'm misunderstanding you, but it seems to me like you're implying that now with the combination of these two, the scoring will be biased and unfair, whereas it wasnt like this before.</p>

<p>Also, I should say, I usually got Bs in english mainly because of poor textual analysis, not poor writing style. Since the writing SATII is basically an SATII in bull*****ting and grammar, I found it very easy. Actually no i didn't, I thought I bombed it cuz i spent too much time on my essay, and didnt have any time to check over any multiple choice answers.</p>

4

<p>"Also, what makes you think the new SAT will be any different from the old SAT and SATII writing. Maybe I'm misunderstanding you, but it seems to me like you're implying that now with the combination of these two, the scoring will be biased and unfair, whereas it wasnt like this before."</p>

<p>I don't think she's implying this. Now that the SAT II writing has been synthesized into the SAT I, it's become a much bigger issue. All the criticisms concerning objectivity are the same; they've just gotten bigger.</p>

5

<p>Well, it's no more unfair than assessing candidates based on how many words they happen to know. At least this is a skill they're testing.
I had no problems with the SAT II Writing. How is this any different from the essay questions on an exam, where again one has to write a coherent response in a very limited amount of time?</p>

6

<p>I believe both the new SAT writing section and the SAT II Writing test are probably invalid and inaccurate as an overall assessment tool, although a certain percentage of the scores are probably accurate. They are probably least accurate for students with bad handwriting. </p>

<p>As far as I know, the College Board has not demonstrated that the writing scores are valid. They could demonstrate validity with at least a sample of test-takers that the writing scores agree with other indicators of writing ability such as English grades or other standardized tests such as their own verbal subtest. They could demonstrate scoring accuracy by having two or more people score the tests independently to see if they agree. They could demonstrate that handwriting has no affect on scores by having different people score handwritten and typed versions of the same essay. It would be easy for them to establish whether the writing tests are valid and accurate but they have not publicized any such documentation. </p>

<p>The College Board should also have a certain percentage of essays from every SAT administration routinely scored twice to see if the scores agree. </p>

<p>The essays tests that are graded by high school teachers suffer from the same pitfalls but the SAT has greater consequences and is scored under worse conditions. Teachers have the option to take their time, to read carefully, to ask questions, and to put the essay in context. I just think teachers are more likely to value creativity and originality, to work through the bad handwriting and overlook the handwriting when they grade for style and substance.</p>

<p>The number of vocabulary words you can define, although limited in scope, is at least a valid measure of something.</p>

<p>On the positive side, I think the SAT writing test is preferable to the application essays and personal statements that can be totally faked.</p>

7

<p>"As far as I know, the College Board has not demonstrated that the writing scores are valid."</p>

<p>Collegehelp, the guiding element of your position is indeed "as far as I know".</p>

<p>Before starting to write a litany of "The College Board could and should...", you ought to consider researching the issue, even just a bit. To get started, here are a few comments on the essay grading. </p>

<p>ETS and its contractor, Pearson Education Assessment, will scan the essays and distribute them online to individual scorers hired as independent contractors and paid a variable per-hour rate, depending on experience and performance. The scorers are, typically, high school and college English teachers and must qualify for employment by completing a detailed questionnaire about their experience teaching and assessing writing. In addition, they must complete an online training course to become familiar with the holistic scoring method and rubric to be used in scoring the essays and also must score at least seventy percent of forty prescored sample essays in exact agreement with preassigned true scores. </p>

<p>Scorers are expected to read each essay quickly to gain an impression of the essay as a whole and to score essays immediately, without rereading, using a 1-6 point rubric. The rubric instructs scorers to treat the essays as first drafts but to determine, among other criteria, students levels of mastery of writing as shown in the essay, with levels of mastery ranging downward from clear and consistent to reasonably consistent to adequate to developing to little to, at the very bottom, very little or none. The rubric is substantially similar to the rubric used to score essays written for the now-superfluous SAT II: Writing test. </p>

<p>New SAT essay scorers are advised not to judge essays solely by length, to try to be mindful of the time pressures and other conditions under which the essays were written, to read supportively (i.e., to look for what writers did well), to ignore handwriting quality, to excuse minor errors, and to understand that no one aspect of writing (coherence, diction, grammar, etc.) is more important than another, and that no aspect of writing is to be ignored. Each essay will be scored by two scorers; thus will receive a score ranging from two to twelve points; however, if the two scorers? scores for an essay differ by more than one point, a third scorer will be used. </p>

<p>Scorers performance will be monitored and reevaluated during the scoring process by random use of calibration essays; these are prescored essays considered paradigms of various scores along the 1-6 scale. Scoring leaders will monitor individual scorers performances in real time and collect data about inter-rater reliability; as part of the effort to ensure uniformity, these scoring leaders may provide feedback via phone and the Web when appropriate to individual scorers.</p>

8

<p>i really don't see how combining the SATII writing with the SATI makes the situation any more or less troublesome.</p>

<p>And comparing it to English Grades wouldnt work, because not all schools grade the same.</p>

<p>And the skills you need to do well in English class are not required on the SATII Writing.</p>

9

<p>xiggi-
Thanks for the information! Where did you find it? I actually tried once to get information from the College Board about their SAT II writing test scoring and received no reply. I really appreciate your excellent response. It makes me feel better about the writing test. At least they are making a strong effort to ensure scoring consistency. </p>

<p>However, a minimum 70% agreement with pre-scored tests during training leaves me a little uncomfortable. It tells me there is room for error in scoring. The fact that the essays are scored by two people who must agree...that's a very good procedure to maintain consistency. The objective sections of the SAT test (verbal and math) are scored with 100% consistency.</p>

<p>I am still not satisfied that bad handwriting is not penalized unfairly although the two-rater system probably makes it less likely. Two scorers could both downgrade an essay because of poor handwriting and still be consistent. Simply telling trainees to ignore handwriting is not enough. As I said, it would be easy to check for handwriting bias in a sample of essays by having two different versions of the essay scored independently...one version handwritten and the other typed. Do they do that for samples of essays? Not to my knowledge.</p>

<p>I stand by my statement that the College Board has not demonstrated the validity of the writing test...at least not made it public knowledge. The writing test scores may have some consistency and still not be meaningful. For example, I could try to find out how smart people are by measuring their height (an extreme example, I admit). Two different people would agree on height but height has nothing to do with how smart you are (I hope!). Consistency is different from validity. </p>

<p>The SATs are designed to predict success in college. The verbal and math subtests are related to whether a student graduates and to their gpa in college. You'd expect writing test scores to also predict success, especially for writing intensive majors. Do they? Or, do they to some extent measure something else such as neat handwriting. The College Board should have some idea of this from the SAT II writing test. The College Board should tell the public about it.</p>

<p>Bottom line, though, I feel better about the writing tests after learning about the efforts made to score them fairly. I am still concerned that international students are going to have a hard time with this part of the test and colleges will have to make allowances for international students and perhaps other groups. At least, they should interpret the results in context.</p>

<p>I heard that some colleges plan to ignore the writing score. Has anybody else heard this? If so, why? Has anybody questioned the validity of their own SAT II writing test score? I know someone (not me) who was one of the best creative writers in school but who had a mediocre score. There was no question about it...this person was an outstanding writer but had a mediocre score. But did have bad handwriting. This is part of where I am coming from on this.</p>

10

<p>
[quote]
However, a minimum 70% agreement with pre-scored tests during training leaves me a little uncomfortable. It tells me there is room for error in scoring.

[/quote]
</p>

<p>It would be nearly impossible to get 100% of the scores correct! The scale takes objectivity into account, and that is why on the Writing SAT II you could still get an 800 even if you got less than a 12 on the writing sample.</p>

11

<p>Another question that has occurred to me is whether the writing test really adds anything to the SAT verbal and math tests. On the surface it would seem like it should but does it? Did the verbal SAT test score not help colleges tell whether a student had English writing skill? Does the writing test really add anything new? If the verbal and math sections did a good job predicting success, then why was the writing test needed? If the writing test agrees with the verbal test, then that supports the validity of the writing test but it also makes you wonder what the writing test tells you that the verbal does not? If the writing test disagrees with the verbal score then the writing test is measuring something different than verbal aptitude, but what? Don't writing ability and verbal ability go hand in hand? Were students with high verbal scores arriving at college unable to write???</p>

12

<p>Well then you'd need to ask the same thing about the PSAT which has had a writing section for many years. The bulk of the writing section on the SATI is not much different from the writing section on the PSAT, with the addition of the essay. In all, the essay counts for about 10% of your total score. I personally think it's a great idea to have grammar and writing skills included on a test for prospective college students. All one has to do is read some of the essays kids post here on CC and you will quickly realize how many high scool students are never taught the basics of grammar and writing in their "English"classes. And, yes, students are definitely arriving at college unable to write. I just read in an article this morning, for instance, that the UC and Cal State system has found that only 46% or so of freshman are able to pass a basic writing class. I suspect it is the same at many, many colleges. Being able to read and being able to write are two different skills, IMHO.</p>

13

<p>Here is a link to a page on the College Board website that addresses some of my questions about the usefulness of the Writing test:</p>

<p><a href="http://www.collegeboard.com/prod_downloads/about/news_info/cbsenior/yr2002/pdf/seventeen.pdf%5B/url%5D"&gt;http://www.collegeboard.com/prod_downloads/about/news_info/cbsenior/yr2002/pdf/seventeen.pdf&lt;/a&gt;&lt;/p>

<p>This page answered many of my questions about Writing test validity. It shows that the SAT Writing test was able to do a pretty fair job of predicting college gpa, college English grades, and performance in Freshman composition courses. I think the correlations of about .3 would be considered on the low side in some fields but are pretty good in the field of education (with all its complexity).</p>

<p>The SAT I verbal test was not able to predict college English grades as well as the SAT II writing test, but almost as well. SAT II writing scores were least valid for African American males. I was not able to find anything about international students.</p>

<p>I still have some concerns about the impact of handwriting quality on Writing test scores and about the inherent subjectivity and potential for bias of the writing test scores but overall I am now satisfied that the College Board was diligent in developing the writing test. One year from now, we will probably know more about the fairness and validity of the new SAT I writing test.</p>

<p>I also have some lingering doubts about whether the writing test adds enough information about abilities in writing to what is already available. Is it worth it? I got the impression from the College Board website that the writing test can replace the costly and laborious local writing tests that many colleges administered on their own. It could be that this is as much a motivation for introducing the writing test as anything.</p>

<p>It is hard to imagine that the writing test would improve on the SAT's ability to predict graduation rates. The correlation between the average verbal and/or math SAT score and the graduation rates at colleges is already about .9 (the maximum is 1).</p>