Who is Writing the Questions for the Khan Academy Redesigned SAT Prep?

Test Preparation SAT Preparation
1

I think free online SAT Prep is a great idea.

I always have done all my SAT prep with official College Board questions from the Official Guide and the Online Course. I had assumed that Khan Academy Prep would be like the old Online Course Prep, with material very, very similar or identical to the material on real tests. With this in mind, I began to work through the Khan Academy SAT math practice questions.

WHAT A DISAPPOINTMENT! So many questions that are NOTHING like the questions in the 4 official tests College Board has published.

The Khan Academy questions have crazy numbers requiring minute calculations. It took me several minutes to work out many of the Khan Academy questions, while I was able to do the questions in the published redesigned SAT’s in at most 30 seconds.

EIther Khan Academy is using very poor quality imitation questions, or else it knows something we don’t about what the questions will be like on the new SAT.

It is nice to have some free prep for people who like to crunch numbers, get a head start on the GMAT, or prepare to study engineering at MIT, but this is no substitute for the real thing.

Come on College Board, BRING BACK THE COLLEGE BOARD ONLINE COURSE!!! It was worth the 60 bucks! Give us OFFICIAL QUESTIONS written by the ACTUAL TEST WRITERS!

2

Do you have a couple example questions in mind? It’s possible that they appear to require tons of calculations, but don’t.

3

I did not say that the Khan Academy problems require “tons of calculations”. I said that the problems require
MINUTE numerical calculations.

Here is an example:

"Miles conducted an experiment to determine if there was a connection between a person’s age and the person’s typing rate. During the experiment, all subjects were asked to type the same essay. Their age, a, and typing rate r, in words per minute, were recorded in the scatterplot below.

(I can’t post the scatter plot, but some points are pretty closely bunched around the line of best fit, which is not drawn, and then there are several outliers above and below. The scale on the x-axis is one box= 5 years, and on the y-axis is one box=5 minutes. The x-axis goes from 0 to 180, and the y-axis from 0-100. The y-intercept of the trend line is going to be off the grid).

According to the experiment, which of the following equations models the relationship between age and typing rate?

A. r= -0.88z+67
B. r=-0.88a+103
C. r=-1.15a+94
D. r=-1.15a+108

For anyone who knows how to draw a line of best fit and to determine the equation of a line from the graph of a line, it is clear that the correct answer is either B or D. However, since the practice is done online, it is not possible to actually draw the line of best fit on the graph. The line of best fit has to be estimated visually from the plot on the screen. Copying the scatterplot with its 16 data points onto paper would be time-consuming and very difficult to do accurately, because not even one of the points lies on a grid cross-hatch. (On the old SAT, CB usually gave a scatter plot with a couple of data points on the cross-hatches that were also on the line of best fit, so that it was easy to get an accurate calculation of the slope.)

Under these conditions, it is very difficult and time-consuming to determine reliably whether the line of best fit is going to pass through y-intercept 103 or 108 (both off the grid), or whether the slope is -1.15 or -.88. This is going to be very difficult and time-consuming even for a student who has the required skills and knowledge.

What is the point of using these strange (not round) numbers that are so close to each other (especially since the scale is 1 box= 5 units, the plot is available only a screen and not on paper, the grid ends below the y-intercept, there are outliers, and all the points miss the cross-hatches)? How is the student supposed to calculate the slope to a precision of 0.27? He can’t even draw the line of best fit on a piece of paper, and even if he could, it is not clear that it would pass through any grid-intersection points, so the co-ordinates of the points on the line of best fit have to be estimated as well.

Even the SAT Math Level 2 doesn’t get to that level of number fiddling.

The College Board has never done number minutiae on the SAT. The College Board has always concentrated on testing concepts, not messing around with hundredths. Is College Board moving towards numerical minutiae, or is this just an artifact introduced by Khan Academy?

There are many Khan Academy questions like this one. This is just one example. When I go through the Khan Academy questions, I get a funny feeling. They feel like someone not from College Board wrote them.
Maybe I am just imagining this. Maybe that’s how the redesigned SAT will be: convoluted numerical minutiae. That’s what I would like to find out.

So my question is: is it true that College Board is not writing the Khan Academy practice questions. And if College Board does not write the Khan Academy questions, WHO IS WRITING THEM? And will students get more OFFICIAL practice questions written by College Board?

The bottom line is that for years and years College Board prided itself on offering test preparation questions and also whole tests WRITTEN BY THE SAME PEOPLE WHO WRITE THE TEST. “Buy this book because the same people wrote the book who wrote the test,” College Board said this because as we all know, it is really, really difficult or impossible to write questions the same way that the test-maker does. Every test has its own style, and students prepare best by practicing with questions with EXACTLY the same style.

This kind of exactness is more important than whether the slope of a line is .88 or 1.15, because students are preparing for a test, not building a rocket.

I really miss the Online Course. It was $60 well spent.

4

@Plotinus I don’t have access to those problems, but because -1.15 < -1 < -0.88, shouldn’t it at least be possible for a capable student to determine whether the slope is greater or less than -1? Maybe visualize the least squares line, pick two easy points, and find their slope? The x- and y-axis both represent 5 units, so this doesn’t seem hard, in theory at least. I don’t know how hard it actually is since I can’t see the graph. It shouldn’t be “time-consuming” at all, although the tradeoff is it’s a little more likely that one would pick the wrong answer.

That said, I compared this question with the linear regression questions found [url=<a href=“https://collegereadiness.collegeboard.org/sample-questions/math%5Dhere%5B/url”>https://collegereadiness.collegeboard.org/sample-questions/math]here[/url] from CB (particularly 3,4,5,20). CB’s appears more conceptual in nature; you have to be careful on 3 and 5 but they can be solved with certainty. But I don’t know if CB will throw more minute problems out there (perhaps), and I don’t know who writes the Khan Academy questions.

5

I have a pdf of the problem. I will try to send it to you by PM.

In my opinion it is not easy to visualize the line to that degree of accuracy from the computer screen. There are no easy points because all the points are between grid lines. The Khan Academy problem diverges from previous College Board problems in many, many subtle and not so subtle ways.

In any case, the point is not about this specific problem. The point is about the overall look and feel of Khan Academy SAT questions. To me, they don’t look and feel like the current College Board SAT problems, and they also don’t look and feel like the problems on the 4 redesigned SAT practice tests College Board has released. Since College Board has been publicizing that Khan Academy offers “Official” SAT prep and has suggested that students are now much better off than they were with the former College Board prep options, I think College Board should also tell us whether College Board is writing the questions of the Khan Academy “Official” SAT Prep. If College Board is NOT writing these questions, in what sense is this prep “official”? “Official Prep” = “Written by the same people who write the test,” If College Board IS writing the Khan Academy questions, then students should know this because it implies that the SAT is going to change in ways that do not emerge from the 4 published Official Practice Tests.

6

http://im***gur.com/kGNOhHU

Here is a link to the pdf. Delete the *** before using the link.

7

I just PM’ed you the solution. It is indeed solvable without taking too much effort.

8

Thanks for your PM and for taking the time to analyze the problem. You say that the x-intercept of the line of best fit is 80 or at most 90. I printed out the scatterplot and I drew in the line of best fit with a ruler. I got an x-intercept over 100. Maybe the computer or the printer changes the aspect ratio? I don’t know. But my line of best fit came out with a y-intercept of approximately 102 and a slope of approximately -1.02. These values are in the middle of the values in the choices.

There is clearly not much effort in drawing the line and determining the equation. The effort is in determining why one choice is better than the other. They look to me like they are both well within the margin of error given the tools available.

9

Perhaps my lower bound of 80 was a bit too low. But an x-intercept of 90 to 95 or 100 seems reasonable.

As I indicated in my last PM, try sketching choice B in the same xy-plane as the scatterplot. The x-intercept is around (117,0) and the vast majority of the points lie below the line of best fit. This is why I still say the answer is D.

10

I have never seen a College Board problem in which the only way to determine the equation of a line of best fit is to work backwards from the answer choices and check the x-intercept match. I also don’t think working backwards from answer choices is a skill required by the Common Core, which is what the Redesigned SAT is supposed to be testing. I thought working backwards from answer choices was one of those test-taking tricks taught by Princeton Review et al., tricks that the Redesigned SAT was supposed to undercut. I thought the Redesigned SAT was supposed to be aligned with the curriculum and methods taught in school. In school, students are given a scatterplot, they sketch the line, they pick two points, and then they determine the slope and the equation of the line. Any student who follows this procedure is going to be scr—ed by this question. She is going to come up with an equation between the two answer choices, and then she will have to start from the beginning to find a way to distinguish between them. You yourself. who clearly are a very experienced math person, only came up with the plug in the answer choices and test the x-intercept match on the second stab at the question. This is a strategy that will only work if the answer choices are given. It is not a standard high school skill.
I can’t imagine College Board putting a question like this on its Online Course, and then explaining in the solutions section that to find the equation of the line of best fit, you have to test the equations in the solutions and see which equation gives the best x-intercept.
I think that if this identical question were put on an actual SAT, there would be many, many protests. That’s why I ask who is writing the questions for Khan Academy. It looks like the Khan Academy writers can’t see the difference between a high skill and an engineering problem.

11

@Plotinus I understand your points - one thing I should’ve mentioned earlier is that we don’t even need to observe that the x-intercepts don’t match. Simply sketching the line y = -0.88x + 103 (or perhaps some points on it) is sufficient to rule out B, since it lies way above the majority of the data points, taking outliers into account. There are also many other ways one could eliminate B.

Possibly, although there have been worse cases (including incorrect math questions on the SAT).

12

There have certainly been SAT questions where you had to look at a data set and eyeball the slope and intercept. But in the past, it is also true that the wrong answers were more starkly wrong. Still, I agree with @MITer94 's earlier post: a quick glance shows a line of best fit that clearly is steeper than 45 degrees with a y intercept that has to be more than 100. So while this is a “tighter” look, I don’t think it is unreasonable.

BUT…

I share @Plotinus 's basic level of skepticism regarding the Khan questions. I can imagine a few years from now, when QAS have been out in the world and books of real tests are available that the common wisdom will be to only use Khan for skill practice and stick with the real stuff for practice tests.

13

It is not only a “tighter” look, but a very, very different numerical and graphical approach from anything on the 4 Redesigned practice tests CB has published.

Here is a link to two line of best fit problems from the CB Redesigned SAT’s:

http://im***gur.com/a/7vfoE

I think anyone who compares these questions to Khan Academy will see that the style is very, if not completely, different. In the College Board questions, you don’t have to draw an approximated line of best fit and then decide if the slope is -.88 or -1.15, or if the y-intercept is 103 or 108. There is nothing even remotely like that. The College Board questions might even be harder conceptually than the Khan Academy ones, but they don’t mess around with numerical minutiae.

The numerical minutiae might not matter to an engineering person or someone very good in math, but I am pretty sure that a lot of students who are not engineering types would be flustered by the small numerical differences and the unusual numbers in the answer choices.

I have a lot of other examples of Khan Academy questions with this different style, but I am not going to put them all up. Everyone can go to the Khan Academy website, look at the CB practice tests, compare the two, and judge for himself or herself.

What matters to me is that Khan Academy at present seems a poor substitute for the old SAT Online Course, which had 10 Official Practice Tests over and above the ones in the Official Guide. It’s very nice to have an automated personalized study program, all those badges and energy points, the screencasts, and of course it’s all free. However, nothing can make up for the paucity of “real stuff”. I had thought Khan Academy was going to provide the digital platform, and College Board was going to provide the material. The way things are, there just is not that much “real stuff” out there, especially for the class of 2017. A lot of these students are planning to take the ACT. I can understand why.

14

@Plotinus I’m still not really sure why you’re making such a huge deal over one little math problem. Yes, I do agree that the styles are different and that the Khan Academy question you posted involves a lot more numbers. Is it an unreasonable question? I don’t think so, and I still wouldn’t be surprised if a similar question appeared on the new SAT. I don’t think it’s a “very, very, different approach” as you stated - quite honestly, it’s a pretty standard problem on linear regressions.

For one, as @pckeller said, it might be better to use Khan Academy for learning/reviewing concepts while using official material for practice tests. You don’t want to teach/learn to the test - first, it’s harder to learn the material well, and second, if the SAT decided to throw a slightly different type of problem, you’d be far less prepared to adapt. In fact, I firmly believe that solving a wider variety of problems will better prepare you for the SAT (provided, of course, they’re decent quality). Also, real life is not like the SAT. You will have to solve lots of problems that you’ve never seen before, given what you already know.

15

Fwiw, for the original problem, here are all the methods I’ve thought of that can let a student correctly deduce the answer is D (note that A and C are obviously incorrect). Of course there may be others.

  1. Carefully drawing the line of best fit, we see that the line's slope has absolute value greater than 1 (you can check by picking two points).
  2. The method I initially proposed, which involved simply sketching the line corresponding to choice B and seeing that it clearly cannot work since nearly all points lie below the line.
  3. Take two points spaced far apart that aren't obvious outliers (e.g. (28,75) and (50,45)) and compute their slope. More guesswork than the other solutions though.
  4. If you've taken a statistics course, a well-known fact about the least squares regression line is that it must pass through the point (xbar, ybar) where xbar is the average of the x-coordinates of the datapoints and ybar is the average of the y-coordinates. xbar is roughly 40 or 45. If B is correct, then ybar is roughly -0.88(40) + 103 which is around 70. Doesn't make much sense.
16

I am using this problem just as an illustration, I would not make a fuss if there were only a couple of Khan Academy problems like this one.

I don’t think this problem is at all unreasonable in itself. All the solutions you propose are great. However, I don’t know whether it is realistic to use these approaches given the timing and technological constraints of the context in which the problem is presented. In a different context and with different instruments, it would be a really banal problem.

  1. You can’t draw a line of best fit on a computer screen. The Khan Academy practice is online. I don’t know whether the aspect ratio of my computer screen or the way my software produced the image is affecting how the graph looks. I don’t know enough about computers to evaluate this.
    I did print out the graph (since on the SAT the graph would be on paper) and drew the line of best fit without a ruler (as would have to be done on the SAT). The slope came out around 45°. It was not clear whether the slope was more or less than -1, or whether it was closer to -.88 or -1.15. The line I drew had about the same number of point above and below it.
  2. I sketched line B using the following method: I plotted a y-intercept of 103 estimated by eyeballing, and then did a rise over run of 9 down and 10 to the right, also estimated by eyeballing, and then connected the two points with a line sketched free-hand without a ruler. This is supposed to be a valid way to draw a line with the tools that are available during the SAT. It was not clear that line B had more points above it. It looked to me like line B sketched this way passed around through the middle of the points. In addition, this solution requires backsolving. I thought CB wanted to move away from backsolving.
  3. You are supposed to be able to use any two points to determine the equation of a line.
  4. This is a very nice solution, but it still involves backsolving. You can backsolve even more easily by taking a data point near the line of best fit, plugging it into the two answer choices, and seeing which equation works the best.

You are completely right that CB puts odd-ball, unpredictable math questions on the SAT (some of the level 5 questions). It would be really fantastic if Khan Academy had some of these level 5 questions, and this would really help people get better scores. But surely the question I am complaining about is not a level 5 question, and not at all like an odd-ball CB math question. It is a very straightforward cookbook level 3 line of best fit question with two answer choices that are so close to one another that to decide which choice is right you have to backsolve.

I found a lot of problems with bad numbers that require minute calculations in the Khan Academy SAT prep, but I haven’t seen any conceptually unusual, creative math problems. Of course I have only seen a small part of what is available. I did notice that even if you get every question right at level 4, you stay at level 4.I did several practices in a row at level 4 just to make sure of this and to try to find some questions that look like the old CB level 5. Does this mean that there are no level 5 questions available on Khan Academy? Or are these levels different from the CB difficult levels?

I didn’t think students are going to go to the Khan Academy SAT prep to prepare for real life. If you want to be an engineer and make things, you take calculus, computer science, and statistics, you don’t do SAT prep. SAT prep is to help you get a good score on the exam. “Official” SAT prep is supposed to allow you to practice with questions that have the same style and content as the real exam. Skill-building is great. But there is no substitute for practice with questions written by the test-maker. This is what the CB SAT Official Guide and Online Course used to provide. Now that these have been replaced by Khan Academy, who is going to fill this void?

17

This analysis is really interesting–thanks to all of you for doing it, and know that if you have time and inclination to do more of it, some of us are paying close attention and learning a lot.

18

Thank you for your kind comment. I am glad if this analysis is in some way helpful to students preparing for the new SAT. I think the class of 2017 has been put in a difficult situation and needs as much help it can get.
With this in mind, here are some more line of best fit questions from the Official Redesigned SAT practice tests:

http://im***gur.com/a/E1wJ7

and here are some from the current SAT format:

http://im***gur.com/a/6VKUk

There are clearly differences between the old and new SAT questions:

  1. On the new SAT, students are expected to know the shape of exponential functions.
  2. On the new SAT there are questions asking to interpret the meaning of elements in an equation.
  3. On the new SAT, there are equations with non-round numbers. For example, the line of best fit in problem 27 (the hardest one of the new lot) is y=0.0125x-61

However, there are no questions at all like the Khan Academy question I posted above. None of the questions requires any number crunching or backsolving. By “number crunching” I mean a numerical operation that is difficult to do quickly and correctly without a calculator.

Many of the CB questions require nothing more than reading the graph carefully and paying attention to the scale of the axes.

A couple of questions require calculating the slope of the line of best fit. However, in these questions either the line is already drawn and so it is easy to calculate the slope exactly, or the answer choices are very far apart, so it is very easy to match any reasonable approximation with the best answer.

The “hard” question (number 27 our of 30 multiple choice calculator questions) has the equation y=0.0125x-61, with numbers that look similar to the numbers in the Khan Academy question. However, the CB question asks the student to interpret the meaning of the number “61”. It does not ask the student to do ANY calculations with these numbers. It also does not ask the student to calculate the equation of the line of best fit and match this to equations with these kinds of numbers. The CB question is conceptual and interpretative. It requires high level verbal skills to understand the wording. The Khan Academy question requires backsolving and number crunching, but little or no verbal or interpretative skill.

This is what I mean when I say that the Khan Academy questions do not look and feel like College Board questions. It is very difficult to produce College Board-style questions. Just the use of language is really unusual.

19

In this case, no. If you pick two outliers and try to estimate the line of best fit, your answer would be way off. Or if your choice of points was less than ideal. This is why I stated it’s more guesswork.

Can you provide a few more examples of Khan Academy questions that don’t seem to resemble official CB new SAT test questions? So far we’ve only seen one. I solved about 7 or 8 last night on their website but my skills of determining exactly whether a question is suitable for the new SAT using tiny nuances is probably not as good as yours.

20

I meant that you should be able to pick any two points on the sketched line of best fit to determine the equation, not that you can pick any two data points on the scatterplot to determine the line of best fit. Obviously some data points are far away from the line.

I will try to collect some more illustrative material from Khan Academy if there is interest.