Clear as mud indeed! @suzyQ7 When you end up with an 2015 SI of 221 equaling a 2014 SI of 221-228 there’s no way to judge what that 2015 211 REALLY means. All these concordances (which don’t even agree!) but the key one is missing - concord the 2015 SI to the 2014 SI. That’s not the same as just converting the score and using THAT as the SI. Is this REALLY that much to ask?
I found an SAT percentiles document (from ~2006, I think) that had numbers of test takers at each SAT score level, the total number of test takers, and the percentile including 99 and 99+. Adding up the number of 99+ scores did indicate that 99+ means 99.5 and up.
@mamelot Maybe you posted this somewhere else, but I agree that the SI percentiles are probably correct. CB took the time to determine how many wrong in each section gives a certain score, and then the percentile of that score (whether research sample or whatever), which was probably difficult and time-consuming. So, even if these are a little off - more people scored a 36 in reading than 1% (which I know doesn’t make sense mathematically), they still then calculated SIs. So, why once you have 1.5 million SIs would you not just rank them and say this is 95.5 and this is 99%, etc.? So, I agree with you that the total score, subject score, and combine R & W score might be based on a research sample, but the SIs might be actual.
@suzyQ7 mentioned that the percentile table might be right because than CB would have to revise. I agree with that because if you go by testmaster/concordance, it would have to be revised SIGNIFICANTLY. Whereas, if you revise the concordance tables somewhat siginificantly, I doubt that many people will give you a hard time.
I probably won’t say this right, but I think you guys will understand this. One analysis (that I don’t have time to do today) would be to add the number of students from each of the high cutoff states that make NMSF. Once you get to 8,000, you could say that would be a conservative 99+ (which I still don’t think you would need in some if not all of those states). Fortunately, for this analysis, some of the high cutoffs states are also the ones with high numbers of NMSF. So CA - 2000; TX - 1700; NY - 1000. What I’m trying to say is if you get to 8,000 and that predicted SI is 216 based on testmasters, than CB page 11 would have to be adjusted quite a bit. And why, when you have the actual data, would you take on that risk?
^^To your point, @micgeaux, about which tables they would be revising, I would add the obvious comment that the one marked “preliminary” is the one most likely to be subject to revision. BTW, I concluded the same thing about the SI percentiles being actual - since they don’t provide a definition (and they do for all other tables in that report) then that’s really the only conclusion we can make. My guess is that they removed the bullet point because it’s too confusing (what does “previous year” mean, anyway?). The simplest answer is sometimes the best one.
@Mamelot I was basing this off of YOUR conclusion (in another thread). You are very observant! I just so happened to agree with you.
I’m not a computer programmer, but I think it would be difficult as a statistician and as a programmer to get some of the other scores to work out. But, I think it would be super easy, once you have SIs, to write a simple program to number them and sort them from high to low, so that you know if you have 150 (228s) etc.
In other words, @micgeuax, CB isn’t trying to “fit” those SI’s to anything like previous score distributions or theoretical curves based on research studies. The underlying test scores, of course, ARE based on those but that’s because those scores are also addressing a different issue which is college readiness tracking - and they can’t really be based on previous years results because it’s a new test.
Of course this just points out how irrelevant are the preliminary concordance tables to this exercise of assessing cut-offs. BTW, I took my D2’s PSAT scores from last year and “con corded” it to this year’s then compared the percentile and guess what? It works! (off but not by much). And why? Most likely because she’s at the 93rd percentile NOT the 99 - 99+ like my D3. What works for most of the curve might not be applicable to the tails.
I did listen to the CB webinar today (just ended) - seems the schools & GC’s should have access to reports online (they referred to Roster Reports and there being a “Report Selector”) that indicate how a student’s and the school’s performance compares to the state performance. I did screen shot some slides but not sure how to share them here.
Another comment they made was about the concordance tables - they were apparently based on study from 2014 - Feb 2015 & will be updated after the SAT in March - at least that’s what I thought I heard.
I missed that webinar. Is it recorded? Have you a link?
NM - I think I found it: https://www.youtube.com/watch?v=HgQfbryW2qg
The link to join the CB webinar is here: https://lp.collegeboard.org/help-resources-accessing-scores
There will be one on Tuesday too - not sure when the archived one will be posted.
Thanks @CA1543. Did anyone ask about the SI tables or cut-offs or how to understand current SI’s in light of previous NM qualifying scores or anything like that?
Here is a link that shows many of the types of reports & info the educators may access:
https://collegereadiness.collegeboard.org/educators/k-12/reporting-portal-help
I submitted a question about SI’s - bc they said GCs could get info comparing students & their school to a state’s data but it was not answered (I felt I could not ask about NM qualifying scores bc wont really be out till Sept.) But the concordance tables are only based on a study CB did from part of 2014 and through Feb 2015.
Yes, the one I found was older but it did talk about the drill down level of detail available to the schools which does include Roster Reports. Thank you.
I called NMSC to get some history for TX & was told that the number of NMSF’s don’t change that much (sorry I didn’t ask about other states) but I would venture to say the number awarded for each state doesn’t change that much from year to year. Also she confirmed to me that it doesn’t change that much because it’s based on population ( I think graduating seniors is what she meant). The number awarded to TX has been running about 1300.
As far as concordance tables she told me not to use them as a way to figure out cutoffs for this year! Those tables were designed more or less for GC’s & sophomores to prep for junior year. Of course she told me the only thing I can do is wait, LOL! Well I know I’m NOT going to waste time on those tables unless I have a sophomore & I don’t.
I’m going to post 3 email messages between Jed Applerouth and me regarding his article, “Can You Trust Your PSAT Score?”
First email:
Jed,
I have a question about the paragraph below. Your article says the SI Percentile was calculated using a research sample. I reviewed College Board “Guide to Understanding PSAT Scores”. I don’t see any reference by them to the SI percentiles using a research sample. Can you please point me to where that information comes from? Why wouldn’t I assume the SI Percentile is from actual data?
I agree other data, such as National Sample Percentile and User Percentile do make that claim. But my question is on the SI Percentile.
Thanks,
(my name)
“NMSC Selection Index Percentile: When scores were released last week, educators were provided with a third set of percentiles, the “Selection Index” percentile. This was still calculated using a research sample, but the sample was limited to 11th grade students. This percentile is based on a student’s NMSC Selection Index (48-228) rather than the student’s scaled score (320-1520). For example, if a student earned a selection index score of 205+ out of 228, they scored in the 99th percentile using the selection index percentiles. If you don’t know what your selection index percentile is, you can find it on page 11 of College Board’s Guide to Understanding PSAT scores.”
Second email:
Hey (my name),
Thanks for your email.
When it comes to the projected national and user percentiles, the CB clearly states that it gathered data between Dec 2014 and Feb 2015 from its research sample of 90,000 8-12th graders:
https://collegereadiness.collegeboard.org/pdf/college-board-guide-implementing-redesigned-sat-concordance-installment-3.pdf
Pages 17 and 20 spell out the target sampling procedure.
To your point, in terms of the SI percentiles, technically there isn’t anywhere in the document that explicitly states from which sample these percentiles were derived, but in fact very little documentation about the NM SI percentile exists. There is no mention that the SI percentiles are in fact “real percentiles” derived from the October testing pool. Based on everything we have read, we believe those scores are research based as well.
The National Merit Selection Index percentile has always been described as the percentile based on juniors taking the PSAT. This translates into the same definition of the new “User Percentiles”. If the CB had the data to publish “real” (read not study based) selection index percentiles, there would be no reason for the rest of the user percentiles to be based on the study.
In fact the National Merit Selection index percentiles differ by less than 2 percentage points on average from the total user percentile. Even if they were based on the same sample, we would expect to see this because Math is a bigger part of the total score than reading and writing (Math is doubled). Any cases where the User percentile for the total score and the NMSI percentile differ significantly, the student indeed performed significantly better or worse on Math than Reading or Writing.
In a nutshell, although its not explicitly stated in the documentation, it seems the SI percentile also comes from the research sample.
Hope this helps.
Jed
Third email:
Jed,
Thanks for the response and the analysis.
I’ve contacted College Board to specifically ask whether the SI % table is based on real data from the 2015 PSAT or the “user” numbers from their research sample.
I agree with you, the 2015 CB Guide does not specifically state were the SI % Table numbers come from. Some people, like you, have assumed they came from the “user” numbers and others, like me, have assumed they came from the real PSAT scores.
In contrast, the 2014 CB Guide, specifically states the SI % Table numbers came from the previous year’s actual results.
If I can get a response from CB, I’ll let you know what they say.
Again, thanks for the response.
(my name).
So, what exactly happened to the super high (221-223) cut offs some people were predicting? It seems like no 210 should do it in a lot of states. Correct?
@Speedy2019 – Thanks for all the outreach and the info you shared. I had really hoped the SI percentiles in the CB’s recent report were based at least on a sample of the students in 11th grade who took the October 2015 test (rather than a research study group of students who did not take the test). I am not sure how they can be particularly helpful if the NM cut offs in September are typically based on the actual test takers in the prior Oct. PSAT or off of some sample - seems the cut offs must be based up actual test takers of the 2015 PSAT,right?
@Jay12321 Those super high cutoffs are still in play and cannot be ruled out. They go from being probable, to improbable, to possible. Stay tuned.