Interesting discussion! Thanks @Plotinus for the thoughtful analysis. As to the question in the title of this thread, I remember seeing a job posting for SAT question writers on the Khan Academy site back when the CB/KA partnership was first announced. Of course, that posting no longer exists, but thanks to the magic of internet archiving, we can still see it:
Thank you, PWNtheSAT, that job announcement is interesting in a couple of ways:
It asks only for MATH question writers. Was there a separate announcement for VERBAL question writers? I have only looked at a couple of verbal questions on the Khan Academy site, but at least the one timed reading practice I did seemed like genuine College Board material. The questions were twisted and the English was perfect. So CB contracted out only the math…? This makes sense in a way, except that SAT Math questions are so peculiar that, as I have shown above, it is incredibly difficult to write them the same way that the ETS people do.
(2) The announcement pretty much proves that many of the Khan writers are not the same as the ETS writers who are writing the real tests.
(3) The qualification requirements look low, but I can’t tell if they are lower than those for ETS question writers. It also could depend upon who is doing the selecting and editing.
I just want to say that I have no problem with the fact that Khan Academy is offering free SAT prep. I think it is a great resource. It is always better to do some practice than no practice of any kind, and Khan Academy has really a huge number of questions in many different areas. I haven’t yet found any published material that is better to prep for the redesigned SAT, apart from the 1 PSAT and 4 SAT’s published by CB.
My problem is that questions not written by the test-maker seem to have taken the place of the Official Online Course, that is, the course that had dozens of quizzes and 10 full practice tests ALL WRITTEN BY THE PEOPLE WHO WRITE THE TEST. CB used to sell the Online Course to whole states. It was really well done. The questions were great, and there were lots of complete practice tests.
I think people should let Khan Academy and College Board that students deserve more official material in the old sense of official. Students should not be put in a worse position than they were when the Online Course existed.
Interesting analysis, @Plotinus, and interesting job posting @PWNtheSAT.
quote The qualification requirements look low, but I can’t tell if they are lower than those for ETS question writers. It also could depend upon who is doing the selecting and editing.
[/quote]
True, but we don’t know how selective or difficult it is to become a problem writer. For example, I would’ve been eligible to apply in high school.
According to the SAT [test specifications](https://collegereadiness.collegeboard.org/pdf/test-specifications-redesigned-sat.pdf), under “Data Analysis Domain,” it says "The student will summarize categorical data or use categorical data to calculate conditional frequencies; conditional probabilities; association of variables; independence of events. "
So my best guess is yes, but maybe rarely. Of course you might have to use the notion of independence itself, for example, flipping a fair coin and flipping another coin are assumed independent, even though the problem doesn’t explicitly say so.
I looked at the Khan Academy questions you recently posted and here were my thoughts:
Mother’s day:
I’m not a huge fan of randomly taking out table entries to create problems (in real life, you would have collected all the data, and done some analysis based on that). I would actually use “compare…with,” not “compare…to.” since you are comparing two different sets (see [here](http://www.grammarly.com/answers/questions/9719-compared-to-compared-with-in-comparison-with/)).
I didn’t have my calculator with me (granted, I could just input the addition into my browser) but it’s quite doable without a calculator if you have good mental math skills.
Sugar:
“Approximately what is the relative frequency…” seems rather awkward to me. Anyone else feel the same way?
And another “randomly take out table entries to make it a bit harder” question. This one was a bit harder for me to solve without a calculator, since I was only able to estimate (11.4% of 61.6 is about 7, so around 69 kg/capita total in 2011, 69 - 10 - 30 = 29. 29/69 is very close to 3/7 or 42%). I would pick C, although since B is pretty close, I would probably get my calculator and do all the calculations as a check. Not the most interesting IMO.
Yes, people (including myself) do write bad math questions, and it’s pretty common to write several bad questions before coming up with a very good one. The process lies in how exactly they select and revise the questions.
I would assume they have test-solvers do the entire test to check that the distribution of topics is appropriately balanced (not just for topics, but also difficulty, how much of it requires/recommends calculator use, the answer choices, whether too many problems of a certain type are asked). Someone should also revise the test to clear any ambiguous questions or awkward wordings, which I’m not really sure if it was done here.
“I didn’t have my calculator with me (granted, I could just input the addition into my browser) but it’s quite doable without a calculator if you have good mental math skills.”
Whether a problem is easy to do without a calculator varies enormously from one person to another. My point was not that you have to or do not have to use a calculator. My point is that the Khan numbers are much worse that the CB numbers for the vast majority of test-takers. The ability to judge the numerical difficulty for the average test-taker should be a pre-requisite for question writers. Just because the question writer can multiply four-digit numbers in his head does not mean that he should write SAT questions with four-digit number multiplications.
“I would assume they have test-solvers do the entire test to check that the distribution of topics is appropriately balanced (not just for topics, but also difficulty, how much of it requires/recommends calculator use, the answer choices, whether too many problems of a certain type are asked). Someone should also revise the test to clear any ambiguous questions or awkward wordings, which I’m not really sure if it was done here.”
Are you talking here about how CB checks an official test or about how the Khan Academy material is checked? There are no complete practice tests produced by Khan Academy. The questions I have presented are individual questions from 5-question theme-based practice sets, all at level 4, All the questions are of the same difficulty and are in the same area, so there is no need to check for a balance of difficulty or topic distribution. However, it would be good if someone checked that the questions had the same content, language, and style as CB questions. I think this would have to be done by an ETS editor.
"According to the SAT test specifications, under “Data Analysis Domain,” it says "The student will summarize categorical data or use categorical data to calculate conditional frequencies; conditional probabilities; association of variables; independence of events. "
So my best guess is yes, but maybe rarely. Of course you might have to use the notion of independence itself, for example, flipping a fair coin and flipping another coin are assumed independent, even though the problem doesn’t explicitly say so."
“Independence of events” was tested even on the old SAT in the sense you describe (i,e., that if we flip two coins, the two events are independent), but there were never questions of the kind I posted with table data about first born children going to college (How many first born children have to go to college to show that going to college is independent of being a first born child?). There are zillions of those table data independence questions on Khan Academy. There is not one question like that on the 4 Redesigned SAT’s. What I wanted to know is whether questions like the Khan Academy ones could be on the real test.
This is really interesting stuff @Plotinus and @MITer94. Now I’m not so aggravated that my D has not worked on the Khan material. It sounds like things were kind of rushed (like real life projects), and proper due diligence and oversight were not performed.
Wow, I am not an English major and I am really feeling self-concious about every word I type here! Please don’t try to diagram my sentences.
Yeah, this is probably true - another thing to look out for when test-solving. Although one difference I noticed is that the new CB questions tend to save any calculator work for the end. Consider this question:
https://collegereadiness.collegeboard.org/sample-questions/math/calculator-permitted/24
Lots of numbers including pi and sqrt(3), but you can solve without a calculator and then use the calculator to compute the answer. But yeah, I’ll agree with you that this is a completely different type of calculator question.
I don’t have the 4 redesigned SATs, so I’ll have to take your word for it. I didn’t really like that independence question about firstborns and college.
As for whether questions similar to Khan might appear on the test, I don’t know. Maybe once the new SAT comes out, Khan will step it up a bit. Or maybe they’ll fall into the fake category.
https://collegereadiness.collegeboard.org/sat/practice/full-length-practice-tests
College Board made the 4 SAT’s available on its website.
There are many independence questions like the one about the first borns inr the Table Data section on Khan Academy.
@Plotinus ah, okay thanks. I’ll look at them when I have time to solve the questions and compare.
“Wow, I am not an English major and I am really feeling self-concious about every word I type here! Please don’t try to diagram my sentences.”
Hey, this is a forum We are just chatting, We are not writing Official SAT questions… We are not getting paid We are not the College Board… But if you are the College Board and you call a question an official SAT question, you better be ready to have it diagramed.
@Plotinus I did the calculator math section from the Practice Test 1 released from CB (only the calculator one since we were talking about questions with minute calculations), and no questions required abnormally tedious calculations. However question 19, section 4 (a typical system of 2 equations) is easily done by back-solving and plugging in the answer choices, which is how I suggested solving the original question you posted.
Are the Khan Academy questions you are getting coming from that long list of math topics found [url=<a href=“https://www.khanacademy.org/mission/sat/practice/math%5Dhere%5B/url”>https://www.khanacademy.org/mission/sat/practice/math]here[/url] (after the diagnostic tests)? I looked at a few sections (including “Interpreting Linear Functions”) and noted that many of the questions were very similar to the ones that you posted and didn’t like. I get the feeling that those aren’t really SAT-like and are better used for learning/reviewing basic concepts, and wouldn’t expect to see many of them appear on the actual test.
Did anyone else notice that Khan has added the number of problems need to complete the each level?
For example
Radical and Rational Exponents has the completed four section arch and now under it it says level four completed completed: 0 out of 21
Operations with rational expressions has the three section arch and says Level three questions completed: 1 out of 35
Before it just had the progress arch, I like this much better!
But the BULK of the Khan Academy questions are in the sections you said were “not really SAT-like”: So that means that students now have virtually NO official practice questions. Or in any case, many fewer than they had until 2015.
This is why students are going to flock to the ACT.
You are right to point out that Test 1 question 19 has bad numbers, or much worse numbers than we would have found on the old SAT. On the old SAT you could usually solve systems easily by elimination.
For question 19, backsolving will definitely work well. You can also copy the equations into Solve if you have an algebra calculator.
In fact, I think it is wrong to put this question in the calculator section when people are allowed to have algebra calculators. Anyone who can write the equations and copy them into the calculator can answer the question. People without an algebra calculator will be at a disadvantage. I think if this question is going to be on the test, it should be a grid-in in the non-calculator section. Then you might actually have to be able to do some algebra and arithmetic to solve it!
There are still four full-length practice tests by CB. In any case, I wouldn’t expect a plethora of good test materials until sometime after the new SAT has been implemented.
That seems to have been the trend for the past several years.
Four full length tests are not so many compared to 10 Blue Book + 10 Online Course + 2 Preparation Booklets=22…
There is only 1 PSAT test. Students (especially if aiming for NMS) are using the 4 SAT’s just to prepare for the PSAT.
Do you think there will be a plethora of good SAT practice test materials later on? How much later?
That isn’t going to help the class of 2017.
It will be interesting to see what percentage of the class of 2017 switches from the SAT to the ACT in Spring 2016.
When does the PSAT booklet with the sample test come out? I should ask our school if they have the booklets yet. If that isn’t the same as the 1 sample PSAT, it will double the number of available sample tests.
This goes without saying, but already having a good command of the math and verbal skills needed will greatly reduce the amount of SAT prep needed (and hence, reliance on such resources). If you have a solid command of the English language and all of the commonly-tested grammatical rules, you shouldn’t need to work through a dozen practice tests. Or if you paid attention and did well in your math classes, and thoroughly understand the formulas, theorems, etc. There are many places where you can pick up these skills besides an SAT book. Ideally, the classroom should be the first.
But in any case, the 2017’s are going to have the disadvantage of fewer authentic SAT practice materials. Nothing I can do about that. My best guess is that there will be a greater proportion taking the ACT.
Later on, I’m pretty sure. I don’t know when. According to the [url=https://en.wikipedia.org/wiki/SAT#2005_changes]Wikipedia[/url] article, the last major change to the SAT was in 2005, where analogy and quantitative comparison questions were removed, and the writing section was added. Perhaps you can try finding resources from 2005 and 2006 and compare their quality…
“having a good command of the math and verbal skills needed will greatly reduce the amount of SAT prep needed (and hence, reliance on such resources). If you have a solid command of the English language and all of the commonly-tested grammatical rules, you shouldn’t need to work through a dozen practice tests. Or if you paid attention and did well in your math classes, and thoroughly understand the formulas, theorems, etc. There are many places where you can pick up these skills besides an SAT book. Ideally, the classroom should be the first.”
What you say is true for people who
- have good skills
- go to good schools
- near test-taking time are already in the percentile range they want to be in.
Many schools are not doing a good job preparing students for tests like the redesigned SAT. The new test definitely looks better aligned with the school curriculum than was the old test, and this is great for students who study well and go to schools in which the Core Curriculum is taught well. But many teachers are overworked and underpaid. Classrooms are overflowing. Public schools are underfunded. Private schools coddle the students. Students spend all their time on Facebook and no time reading. The schools can’t assign serious readings to students because the students will fail. Look at the controversy over Common Core. Scores on the Common Core tests are horrific. Competition for selective universities means that a relatively small difference in standardized test scores can change an admissions verdict.
The CB idea that SAT prep can be reduced or eliminated by writing a test aligned with the Core Curriculum is premised on a high-functioning student body, a high-functioning high school system, and a high-functioning college admissions system: three false premises. The US high school system is dysfunctional. The college admissions procedure is insane. Students vary. Many approach test-taking time to find they do not have the skills they need to be in the percentile range they need to be in to get into the university they would like to attend. It is too late to go back and redo high school more seriously or at a better school. That is why they or their parents look for high quality SAT prep. The problems are deep.
You can’t fix these deep problems in the educational system just by rewriting a test. You have to triple teachers’ salaries. You have to get your kids off Facebook. And this is just for starters.
It is well-known that CB recentered the SAT scoring curves in 1995 to cover up the fact that median scores were plummeting. Scores were falling in part because more lower-scoring students were taking the test, and in part because average students and schools were just getting worse. This recentering meant that overnight, a 650 in reading became a 750. A student who previously was just pretty good suddenly became great. There was no way to distinguish between the student who had 800 on the old curve and the student who had 700 on the old curvet: they both had 800 on the new curve. So many people now had really high scores. The SAT no longer distinguished well among students at the top of the scale. Today, there are so many people who score 2200, 2300, 2400. I don’t think that is going to change with the new SAT. People will aim to get perfect or super-high scores, and a lot of people are going to get them. The test is too easy to function well for selective college admissions (which was its original purpose). Students applying to very selective schools will still want to prep to get perfect scores or as close to perfect as possible. Other students will want to prep to make up for gaps in their high school educations. If CB doesn’t provide a reasonable number of high quality practice materials, I wouldn’t be surprised if even more students (including the Ivy Leaguers and MITers) flock to the ACT.
“When does the PSAT booklet with the sample test come out? I should ask our school if they have the booklets yet. If that isn’t the same as the 1 sample PSAT, it will double the number of available sample tests”
The PSAT Preparation Booklet has been out for a long time. The sample test in it is the same practice test that was released by CB back in April. There is only 1 Official PSAT Practice Test.
Here is an interesting study CB might carry out:
Keeping all other variables are constant, if x>y and x and y are positive integers, do students who do x Official Practice Tests followed by review and targeted skill practice in weak areas raise their SAT scores more than students who do y Official Practice Test followed by review and targeted skill practice in weak areas?