PSAT / NMSF cutoffs: Concordance tables vs. percentiles

Test Preparation
1

Not sure if this should be a separate thread or not, but I think this is the key point in trying to determine NMSF cutoffs.

As everyone has noticed, there are 2 primary pieces of data from collegeboard - the concordance tables and the percentiles (both given in https://collegereadiness.collegeboard.org/pdf/2015-psat-nmsqt-understanding-scores.pdf ). They disagree at the top end of the range.

I just wanted to point out that they’re measuring different things, and the data for them are generated in different ways.

Percentiles are simple - what fraction of test takers did better / worse. Once all the scores are known, you can know exactly what everyone’s percentile is. And even with small (representative) samples, you can get it almost exactly right.

Concordance tables are more complicated. As best as I can understand, concordance tables are trying to relate how a score on one test relates to a score on another test. IOW, if the same person took both tests at the same time, what would they get on each. My understanding is that early concordance tables were relating SAT to ACT scores - by using data from kids who had actually taken both. I think concordance tables are built by looking at scores from kids who took multiple tests. If you have a kid that takes the PSAT, and then takes the SAT and the ACT at about the same time, that’s a data point for concordance.

Once the PSAT is done, and all scores are in, you do not have full data for the concordance table. You need to wait until more kids who took the PSAT also take other tests. Which is why they loudly state that the concordance tables are preliminary.

So, 2 points.

  1. The concordance tables are necessarily preliminary, despite all PSAT scores being known.
  2. The percentile table - which are potentially final - is what you want to use for NMSF cutoffs regardless. NMSF goes to the top n students. It doesn’t matter if this year’s kids are much smarter or dumber than last year (which is kindof what concordance tables are trying to represent) - it’s still the top n. If n divided by the number of kids that took the test is exactly 1%, then exactly the top percentile would be NMSF (though that ignores the state-specific reality).

Therefore, I will present the sure-to-be-popular :slight_smile: optimistic cutoffs based on the percentile tables I posted previously:
http://talk.collegeconfidential.com/discussion/comment/19181229/#Comment_19181229

If your 2015 score is 228, add 12 to get a corresponding 2014 score of 240.
If your 2015 score is 214, add 10 to get a corresponding 2014 score of 224.
If your 2015 score is 205, add 8 to get a corresponding 2014 score of 213.
If your 2015 score is 202, add 4 to get a corresponding 2014 score of 206.
If your 2015 score is 200, add 2 to get a corresponding 2014 score of 202.
If your 2015 score is 198, add 0 to get a corresponding 2014 score of 198.

IOW, the higher your score, the more points you add. (Obviously, if your 2015 SI is between those numbers, you add some number in the middle. So for say a 209, you’d add 9 to get a 218.)

This would imply that the CA cutoff is 213. (Because adding (slightly less than) 10 gives you a 223, which was the 2014 cutoff.) Texas, which was 218 in 2014, would be 209 on the 2015 test. North Dakota, which was 201 in 2014, might be 199 or 200 in 2015.

Let me know what you think, if you agree, and/or if I’ve made any mistakes.

Mike

2

@thshadow this is really helpful - thank you! After seeing the SI percentiles, the only thing that makes sense at the very high end (exactly where you would expect to see cut-offs) is that the preliminary concordance tables are flawed at best and most likely inapplicable - perhaps they are meant more for the median score and not the top. It will be interesting to see if they work any better once the final tables are released. The “sliding-scale” seems to fit the SI data better. @GDadwith3more was saying this as well.

3

Assuming the percentile table adequately reflects the statistics of the students taking the 2015 PSAT, I think this is solid.

4

@theshadow Thank you! As a parent of a 221 in the highest cutoff,I hope you are right! I’m sure shortly someone will come along with another theory as to why the top cutoff will be 222 or 223 (like yesterday), but like you, I tend to believe that the % tables are fixed, and the concordance are variable and may be wrong.

The only thing that concerns me is the state by state specific cutoffs… but I’m thinking that in the high cutoff states, MOST juniors were already taking the test, so any additional juniors taking it this year because it fell on a weekday would not necessarily rise to the top as top scorers. At least not enough of them to have a material affect on the cutoff. I’m hoping…

5

I don’t think this prediction will hold because “it is too good to be true” :wink: such as 209 in Texas and 213 in CA. That’s just too good :wink:
Sorry I’m a bit tired of number crunching by now, so won’t say my reasons. Time to do laundry.

Where is @GMTplus7 who could do distribution curves?
Now we have some data to slog through. Pg 8.
https://collegereadiness.collegeboard.org/pdf/2015-psat-nmsqt-understanding-scores.pdf

6

While at it, here is another theory based on percentiles only, not concordance
http://talk.collegeconfidential.com/discussion/comment/19181083/#Comment_19181083

7

@payn4ward Yup, that’s the one I was talking about. I’m really not smart enough to figure out if those stats make sense or not…its another post the highest cutoff could be 222. I really need to stop reading this forum, its up and down, all day long!

8

@payn4ward The theory in that post assumes the scores follow a Gaussian distribution. I don’t think that’s the case, and it’s a very key assumption in the analysis.

9

@suzyQ7 Yes, we should stop speculating and all go back to our former lives :wink:
Chances of predicting correct cutoff is harder than winning the power ball it seems.

10

I Agree - these are not normally distributed aka Gaussian curves. There is relative clumping of the data that is occurring between the 96-98th percentile so when you use multiple standard deviations from the mean to estimate it overshoots the 99th percentile and results in an estimate that is too high for the data. You can see that the estimates are way higher than the actual SI percentiles because of this.

11

<<yes, we="" should="" stop="" speculating="" and="" all="" go="" back="" to="" our="" former="" lives="">></yes,>

My former life includes a dentist appt so fitting clumps of data to a Gaussian curve is actually more fun :wink:

12

@GDadwith3more @bucketDad That makes me feel better. So you both think the highest cutoffs are closer to @theshadow calculations (i.e. top cutoff closer to ~215 instead of 222)?

If your 2015 score is 228, add 12 to get a corresponding 2014 score of 240.
If your 2015 score is 214, add 10 to get a corresponding 2014 score of 224.
If your 2015 score is 205, add 8 to get a corresponding 2014 score of 213.
If your 2015 score is 202, add 4 to get a corresponding 2014 score of 206.
If your 2015 score is 200, add 2 to get a corresponding 2014 score of 202.
If your 2015 score is 198, add 0 to get a corresponding 2014 score of 198.

@Dave_N Thoughts on @GDadwith3more @bucketDad comment’s above?

13

OK, never mind @GMTplus7
https://www.■■■■■■■■■■■■■■/blog/2016/01/14/understanding-your-new-psat-score/

14

@theShadow, your analysis looks spot on.

Assuming between 1.5 and 1.7M juniors took the PSAT, the 99+% contains about 7500-8500 students. Those are the highest scoring students in the country based on SI. Most of those students SHOULD be NMSF. The state cutoffs need to be low enough to capture most of those students. That means the cutoffs need be to below 214 in most states.

Last year, 49 out of 50 states had cutoffs low enough to capture all students with 99+%. The only exceptions were NJ and DC. - and their cutoffs were just above the bottom 99+% range.

I think the idea that the test was much easier this year and thus the number of students with very high scores must be exceptionally high isn’t true. Only 7500-8500 students could get a score of 214 or higher.

15

The blog linked above https://www.■■■■■■■■■■■■■■/blog/2016/01/14/understanding-your-new-psat-score/ claims that CB has inflated the higher percentiles, and that GCs are reporting far more students in the 99 percentile than before. If CB has indeed grossly inflated the 99 and 99+ percentile groups then I don’t think you can conclude much by looking at percentiles. For all we know there are 30,000 kids, not 7500-8000 kids, being told they are at the 99+%.

16

@mathyone but are GC’s looking at the total score percentiles or the SI percentiles when they are are reporting so many more 99%?

17

@suzyQ7 SI percentiles

18

I actually don’t get it… are they saying that the 15 rows of 99+ on page 11 of the report are inflated - meaning actually not 99+%, but instead some lower number - or are the fact that there are more 99+% is because the test was too easy? Are they saying that the Page 11 percentiles are wrong?

19

@suzyQ7 Yes, they are saying that page 11 is wrong. They are predicting much higher cut-offs this year. I used their methodology and ran my daughter’s numbers, and they are saying a 1470 would not make cutoff based on concordance with 2014 cut-offs (for Texas).

20

@suzyQ7 The article considers multiple possible problems. Quoting below:

Why the increase in percentiles?

There are several possible explanations, and we don’t claim to have the answers. A few possible explanations might be the following:

  • The “nationally representative sample” used to develop the percentile tables wasn’t particularly representative;
    • The students who took the test this year varied in a statistically significant way from students who have taken the test in years past; -The preliminary concordance tables are specious and will be significantly reworked at some point (revised tables are slated to arrive in May, following the administration of the March and May SATs);
    • The College Board’s new percentile charts fail to properly account for the change in distribution that results from eliminating the guessing penalty and shifting the number of answer choices from 5 to 4;
  • Something else entirely is going on.

The first bullet suggests the page 11 percentiles could be wrong. That’s just one of several possible explanations.

If the page 11 percentiles are right, then I think the analysis discussed in this thread holds up and the cutoffs go down. That said, I’m having a hard time trusting any info coming from the College Board. The fact that their job has been crowd-sourced to internet speaks volumes.