Who is Writing the Questions for the Khan Academy Redesigned SAT Prep?

Test Preparation SAT Preparation
61

Yeah, we have all sorts of education-related problems, no doubt about that.

Which is partly why more and more schools are test-optional now. Hampshire College now doesn’t even accept SAT/ACT scores, last time I read on the news (although not really because the test is too easy, but for other reasons [here](https://www.hampshire.edu/news/2014/06/18/no-to-satsacts-not-even-optional-at-hampshire-college)).

Basically, is there a positive correlation between number of official practice tests taken and the amount that students increase their scores. Might be interesting but lots of other factors at play. I’m simply curious as to what the score cutoffs will be (e.g. how many for 700 or 800).

62

There’s a lot of anecdotal evidence on this forum that the answer to your question is yes. In fact, I think that more than 50% of the discussions on this forum are based on the premise that the answer to your question is yes.

63

@MITer94 - of course you’re correct, but with all due respect (and that’s significant given how much help you provide on this forum) I think the comment is irrelevant. This is a SAT prep forum and as I stated in the post directly above many or most people come here to learn how to rapidly prep for the SAT. People who already have the requisite math and verbal skills have no need of this forum, unless like most of the people posting on this thread, they’re sharing knowledge for the benefit of others.

64

“Basically, is there a positive correlation between number of official practice tests taken and the amount that students increase their scores.”

The answer is yes. If CB says otherwise and you believe it, I have a bridge I’d like to show you.

If your kid were in the class of 2017, you would want CB to provide more official practice tests.

The Khan Academy SAT prep was supposed to level the playing field by providing WORLD CLASS
SAT prep for free, so wealthy kids would not have an unfair advantage. But now it turns out that the Khan Academy SAT Prep is not so world class after all, and not even as classy as are the Blue Book and the SAT Online Course, which have questions and tests WRITTEN BY THE PEOPLE WHO WRITE THE TEST.
The rich kids are going to get help from their tutors anyway. And the other kids?

65

For anyone interested in this topic but without the time to read all the prior posts on this thread, the last paragraph above does a great job IMHO of summarizing the point and conclusions of all the prior posts. Well done @Plotinus .

66

quote I think the comment is irrelevant.

[/quote]

@CHD2013 yes, I think you’re right. Ideally the above would be the case, but after seeing a lot of practice questions by Khan Academy, I agree they’ll have to step it up to level the playing field (especially since CB partnered with them).

67

“they’ll have to step it up”

As a first step up, there should be much more transparency on the Khan website regarding which material is really official. Students who might think that all the questions are genuine SAT questions, and this could skew their expectations and study plans.

As a second step up, there should be at least as much official practice material as there is on the Official SAT Online Course. I don’t see how anything less could be dubbed “world class”.

68

Perhaps. There’s a place for you to provide feedback on these issues ([url=https://www.khanacademy.org/mission/sat/practice/math]here[/url], where it says “Let us know…”).

I feel like this will happen eventually, but official, full-length practice tests take awhile to write (finding passages, writing problems + answer choices, test-solving, revising, making sure topics are well distributed, etc.). I don’t know how much time it took to compile the four practice tests, and I don’t know how much official practice material will be out when the new SAT rolls out next year.

69

There’s a place for you to provide feedback on these issues (here, where it says “Let us know…”).

Don’t you think that Khan and CB knew all along that most questions were not going to be written by CB? Look at all the pages I had to write to get to something like the truth. Look at the comments of parents who just assumed that CB was writing the questions. Look at the answer of CB to my inquiry with its “either…and/or”. Let’s not be ingenuous.

This is not something Khan and CB want to wear on their sleeves.

“I feel like this will happen eventually … I don’t know how much official practice material will be out when the new SAT rolls out next year.”

I think CB should have waited to roll out the new test until more practice materials were available. Maybe it would not have been good for the bottom line in the short run, but in the long run it might have helped to maintain its customer base more effectively.

70

I would think so. However, I see a couple potential drawbacks if CB were to write most or all of the practice material at this time, provided the new SAT rolls out in 2016. Of course, postponing the first “new” test would give more time for more and better practice materials, although CB might have its own reasons why they are implementing in March 2016.

I don’t feel like I have much more to add to this topic, even though it’s a very interesting one indeed and should be discussed about, kudos for initiating this thread. Others will probably know more about the Khan Academy materials than I do. But in any case, I’m curious as to what will happen in the next year or so…

71

Thanks for your comments, information, and insights MITer94. I have benefited greatly from them. In particular, I am going to reevaluate my view of backsolving . In one of his books, Gruber actually distinguishes between higher- and lower-functioning test-takers based on whether they forward-solve or backsolve. You have definitively refuted this metric, not only for the old SAT but also for the new one.

72

I would like to make one more observation about “stepping up the game” of automated online test preparation. This observation applies equally to Khan Academy preparation, the SAT Online Course, and the myriad other online courses.

As a teacher, I am much more interested in HOW a student solves a problem than in WHETHER THE ANSWER IS CORRECT.

Often students produce the correct answer with a poor or even completely wrong method.

Students also often use a very good method but make a slight careless error.

The main role of a teacher is to evaluate and teach reasoning, concepts and methodology, not to score answers.

Suppose a student does a diagnostic test with an online preparation course. She gets some questions right and some questions wrong, and then the software designs a personalized program of additional practice in the subject areas of the wrong answers.

But what if the student got a question right using a wrong or poor methodology? She needs work in this area. She needs someone to explain to her why her methodology is poor and what the correct methodology would be. This is not going to happen. The software will determine that she has good skills in this area.

And what if she got a question wrong due to a sign error, and she has great skills and methodology in that area? The software is going to propose that she work on this area when she does not need to. This is an inefficient use of her time.

And what about the areas in which she needs better concepts, skills, and methodology, and she got the questions wrong? She has read the correct solutions, maybe watch a video or two, get some hints, and do more problems. She has to teach herself from the materials available.

My experience is that some students can teach themselves about their weak areas from printed solutions and videos, but many can’t or won’t. Many need a person to explain to them the correct concepts, skills, and methods in their weak areas. There is also the problem of motivation. A live instructor worth her salt is more motivating than are energy points and badges, especially when you have to face the areas you hate the most.

When a teacher does a diagnostic test, she evaluates not so much which answers are right and wrong, but what concepts and methods the student used to produce the answers. She designs a personal program of study to improve the student’s concepts and methods. This involves PERSONALIZED EXPLANATIONS. She explains the concepts and methods in a way personalized to the level of understanding of the individual student. She checks whether the student is following the explanations. A personal relationship develops. This is not going to happen with an automated system.

So in the end of day, I don’t think you can get world class instruction from an online course alone, even if you have official questions.

For world class instruction, you need a world class teacher.

73

Interesting! I had the same reaction when I went though Khan: so many calculations with hard numbers and no way to simplify. However, I thought that it was following the common core trend or whatever it is of having the kids work with hard numbers all the time. I personally hate it but I see it a lot. One of my kids is in Algebra 2 this year and in the unit about systems half of the coefficients were mixed numbers. So much time to do that homework, pure headache. I am glad to hear that the actual test might not be so bad.

74

College Board and Khan have to specify which questions on the Khan site are genuine SAT questions and which are skill-building questions. Parents, teachers, and kids deserve to know whether the new SAT is going to have as much data entry and number-crunching as the Khan problems do.

75

I’m a junior and just took the October SAT bc I don’t know what the new one will be like… planning to take the ACT in the spring, but I’m leaving the option open to take the new SAT in the fall, depending on how I do on both. Thanks to everyone who has written on this thread. TBH, I think it’s hard for Khan and the College Board to write so many new questions and test quickly before the rollout is supposed to happen in March. So I am going to see if more practice tests or copies of the real ones that are given next spring come out before I might study for the new one (maybe by summer next year). I think a lot of test prep places have their people take the test and then order the “Question and Answer Service” to get copies of it to show students.

76

“I think it’s hard for Khan and the College Board to write so many new questions and test quickly before the rollout is supposed to happen in March.”

So why didn’t they wait to roll out the new test until after sufficient practice materials were available?

“I think a lot of test prep places have their people take the test and then order the “Question and Answer Service” to get copies of it to show students.”

I am sure that this will make the test prep companies even more popular if CB doesn’t make more official practice tests available.

77

This is true. Studies of great schools always come up with a long list of factors that must go well to make things really work right, but at the end, they almost always end up with teacher quality as the single biggest factor.

78

Yeah, this is an issue not only with online platforms in general, where trying to ascertain where the student messed up is often far too complex, but also with multiple choice tests. For one, it becomes more difficult to tell whether the student actually solved it using mathematically valid techniques, or used a “cheap” and/or invalid process of solving (not just backsolving/plugging in choices, but making faulty assumptions or just simply guessing). Making the entire math test grid-in or AIME-style (all answers integers between 000 and 999) helps IMO, but only solves half the problem.

I will also agree with this, but there are many places in the country/world where that opportunity is difficult to come by…

Interesting – just curious, what did it say specifically (e.g. higher-functioning test takers tend to forward solve more, or back-solve more)?
I generally try to solve the problem directly, and don’t typically back-solve unless it’s a much faster solution (e.g. that 2x2 system with ugly numbers) or if the problem seems to encourage it in some way.

79

" just curious, what did it say specifically (e.g. higher-functioning test takers tend to forward solve more, or back-solve more)?"

Ok, I just threw this out from memory, since I am not writing a research paper. Gruber does not use the exact words that I used.He does not come right out and say that backsolving is never a good or the best strategy. But you might infer something like this position from his strategy diagnostic test. Here is a footnote since you asked for one:

I have Gruber’s Complete SAT Guide 2009.

At the beginning, there are two strategy diagnostic tests, one for verbal and one for math. There are standard questions, but after each question there is the second question: “How did you get your answer?” The answer choices for the second question give a variety of strategy options that students might be likely to use for that specific question. The backsolving strategy, when offered, (whether verbal or math) always is counted by Gruber as wrong and the forward-solving strategy for the same question always is counted by him as right. There are no questions in which backsolving counts as a correct strategy.

If Gruber thought that the highest-functioning students sometimes backsolve, then he should have included some questions in which backsolving is the correct strategy. The fact that there are NO questions in the diagnostic that would be correct to backsolve entails that according to Gruber, students who never backsolve use the best strategies.

As far as I can remember, there are also no example questions later in the book for which he recommends backsolving, no explanations of which questions are best to backsolve, and no demonstration of how backsolving works. (But I read the book 6 years ago, so I could be wrong about this. I remember the diagnostic tests more clearly.)

Gruber’s civilized approach is a stark contrast to say, MGMAT Advanced GMAT Quant or the MIT street-fighting mathematics, in which approximation, guessing, and backsolving are treated as valuable strategy tools.

I have to confess that I thought Gruber was on to something in the case of the old SAT, and I thought that the new SAT was going to be even more civilized than the old. But you have changed my mind, at least for the math. It looks like the HIGHEST functioning students are going to street-fight the math as needed.

But kudos to Gruber for diagnosing strategy and not just answers. I can’t agree more with THAT. Talk about stepping it up. Khan, take note.

80

“this is an issue not only with online platforms in general, where trying to ascertain where the student messed up is often far too complex, but also with multiple choice tests. For one, it becomes more difficult to tell whether the student actually solved it using mathematically valid techniques, or used a “cheap” and/or invalid process of solving (not just backsolving/plugging in choices, but making faulty assumptions or just simply guessing). Making the entire math test grid-in or AIME-style (all answers integers between 000 and 999) helps IMO, but only solves half the problem.”

I did not mean to critique to test itself, just computerized prep courses for the test. Of course, the test itself is not perfect.

You are never going to write a computer-scored verbal test that is anything like a classroom English or history test, no matter what you do. Those multiple choice answer choices are really bizarre.

There are plenty of other testing models out there in the rest of the world. Usually tests taken by the entire student population are written by the country’s Department of Education, not by a private company (i.e., national school-leaving tests). In the US, much more is left to the private sector, so there are more business interests at play. See health care.)

In most of the countries I know, there is not ONE test for everyone. For example, in England, different people take A-levels in different areas, and some people just take O-levels. This is more like the IB or AP programs in the US.

Some countries also have separate exams or a separate selection procedure just for the very selective universities, or for private universities that specialize in a certain field. It is true that people do test prep for these exams, but only a relatively few people choose to take them. In the case of the super-selective universities, the tests are really, really hard. I am thinking here of l’Ecole Normale in Paris, or the Scuola Normale in Pisa. Cambridge and Oxford have their own professors interview and give an oral examination to candidates to make the final sections. You can’t get into those schools because you have 2400 but the other guy has 2300 on a test as easy as the SAT. The scores at the top are more spread out. The differences are clearer. The exams are written and graded by professors in the field, not by testing companies.

There is a big divide between the top and the middle/bottom.
I sometimes think that CB revises the SAT every x years in part so that it can recenter the curves without admitting what is happening. CB recentered the curves in 1995 without revising the test, and it looked pretty bad.

It would probably not be politically correct in the US to emphasize an increasing divide by creating a two-tier selection process for the most selective universities. And it probably would also not be good for CB’s and ACT’s bottom lines, especially if they lost control of the second round.