MIT reveals data discrediting the Revealed Preferences rankings

Colleges and Universities A-Z Harvard University
21

<p><a href="sakky:">quote</a>No, that is a completely false. The RP study NEVER relies on cross-admit data. The RP study uses information about admissions decisions (but NOT cross-admit information) as raw data.

[/quote]
</p>

<p>What you are calling "admissions decisions" I have been calling "cross-admit decisions" (matriculation decisions of students accepted to two OR MORE schools). Other posters apparently understood that, but to remove any ambiguity let's call it "multiple-admit matriculation decisions". Given that phrasing, do you have any further disagreement with the statement that the RP rankings are derived entirely from a list of a few thousand such decisions (i.e., that those decisions are the sole data from which the ratings are computed by some statistical algorithm)? </p>

<p>
[quote]
It then MODELS the data to fill the missing gaps,

[/quote]
</p>

<p>Everyone understands that, and the discussion prior to your arrival was about other matters. You are attacking a strawman in belaboring this point. Notice, for instance, that from the first posting in the thread (the one you replied to when entering the discussion) I referred to the "numerical weights" of schools in the posting you first replied to here; the weights are the (estimated) parameters of the MODEL of the data, as you call it.</p>

<p>
[quote]
Hence, the modeled data is BETTER than the cross-admit data as long as the model holds, because cross-admit data, by definition, does not include missing data.

[/quote]
</p>

<p>Whether it holds is the 64 dollar question. The RP paper is silent on that question, since they don't publish the cross-admit data, and they don't provide any measure of the quality of the model as a predictor of the cross-admit decisions.</p>

<p>If the model were good, the ranking should linearize nicely. What the confidence probabilities from the MCMC simulation seem to show is that it doesn't; there are distinguishable tiers of schools, as we knew without the RP study, but there isn't necessarily more ranking potential beyond that.</p>

<p>
[quote]
"Actual choices are the absolute CRUX of the paper"</p>

<p>Uh, wrong. There is a world of difference between somebody preferring, say, Harvard and actually having the CHOICE of Harvard. Just because you don't have the actual choice of a particular school doesn't mean that you don't want it.

[/quote]
</p>

<p>You're addressing a different issue. I'll come back to it later, this posting is too long as is. </p>

<p>
[quote]

[quote]
Maybe you didn't read the paper. Where people apply is not a form of revealed preference that they attempt to model, and the RP ranking rewards specialty schools that are negatively "preferred" by a majority who would never apply there (BYU, Caltech, and others). In the other direction, a school that is "preferred" enough to be a favorite safety school will suffer in the rankings.

[/quote]

Perhaps you didn't read the paper. Specifically, you may not have read section 7 in which the authors explicitly discuss the notion of self-section and perform a RP study that measures only those students who are interested in technical subjects,

[/quote]
</p>

<p>You did not understand the paper. Section 7 <em>does not</em> model the "revealed preference" information disclosed by non-applications, nor does any other section of the paper attempt to model that form of revealed preference. The authors do not propose any model to deal with that question. All they do in section 7 is repeat the RP ranking for the subset of students who indicated an interest in (for example) science. The RP methodology, as it stands, does not even try to deal with the particular form of revealed preference contained in an applicant's selection of target schools. </p>

<p>
[quote]
Now, I agree with you that Caltech's ranking in the RP probably is inflated relative to the entire set of schools, and in particular, may well be inflated relative to those schools that have little overlap with Caltech (i.e. more humanities-oriented schools). But that's not the point that we're discussing. The point we are discussing is what is Caltech's RP ranking relative to MIT, both of which are obviously technically oriented schools.

[/quote]
</p>

<p>No, that wasn't the question at all. It has nothing to do with the fact that two particular schools, happen to be misranked relative to each other. The problem is that the ranking methodology used is stablest and most accurate for the schools that will end up on top. If there is possibility for a school with less than 20 matriculation decisions to swing into the second place (other schools in the top 6 had hundreds of applications and dozens of matriculation battles covering all possible pairings), and to have its desirability mis-estimated so badly (50 Elo points higher than MIT when it's more like 200 lower), that tends to confirm that the model is unstable. If it is unstable at the very top, it only gets worse going down the list.</p>

<p>
[quote]
What counts for the purposes of this discussion is whether Caltech's RP ranking relative to MIT is inflated.

[/quote]
</p>

<p>No, Caltech vs MIT alone doesn't matter, it's the implication for the whole model. If Caltech beating MIT were happening with both schools below number 10, it would be a minor point. That it happens at number 2 and number 4, with a large Elo point discrepancy, combined with other information (Yale-Stanford and others), corroborates pre-existing suspicion from the mathematics, that the model is unstable with this amount of data.</p>

22

<p>
[quote]
What you are calling "admissions decisions" I have been calling "cross-admit decisions" (matriculation decisions of students accepted to two OR MORE schools). Other posters apparently understood that, but to remove any ambiguity let's call it "multiple-admit matriculation decisions". Given that phrasing, do you have any further disagreement with the statement that the RP rankings are derived entirely from a list of a few thousand such decisions (i.e., that those decisions are the sole data from which the ratings are computed by some statistical algorithm)?

[/quote]
</p>

<p>Sure, but that's entirely different from what is traditionally known as 'cross-admit' data, which is strictly speaking, the comparison of just 2 schools. </p>

<p>
[quote]
Everyone understands that, and the discussion prior to your arrival was about other matters. You are attacking a strawman in belaboring this point. Notice, for instance, that from the first posting in the thread (the one you replied to when entering the discussion) I referred to the "numerical weights" of schools in the posting you first replied to here; the weights are the (estimated) parameters of the MODEL of the data, as you call it.

[/quote]
</p>

<p>Ok, you could say that. But the key difference is that the weighting of those model parameters is entirely endogenous to the model itself. Contrast that to, say, USNews, where the weighting parameters are entirely exogenous and hence entirely arbitrary. </p>

<p>
[quote]
Whether it holds is the 64 dollar question. The RP paper is silent on that question, since they don't publish the cross-admit data, and they don't provide any measure of the quality of the model as a predictor of the cross-admit decisions.</p>

<p>If the model were good, the ranking should linearize nicely. What the confidence probabilities from the MCMC simulation seem to show is that it doesn't; there are distinguishable tiers of schools, as we knew without the RP study, but there isn't necessarily more ranking potential beyond that.

[/quote]
</p>

<p>Nobody, least of all me, is saying that the model is perfect. What I am saying is that it is more informative than just looking at pure cross-admit (or multi-admit) data alone. For example, at least the RP model takes a stab at filling in the missing gaps. Multi-admit data alone tells you nothing about the missing gaps. </p>

<p>
[quote]
You did not understand the paper. Section 7 <em>does not</em> model the "revealed preference" information disclosed by non-applications, nor does any other section of the paper attempt to model that form of revealed preference. The authors do not propose any model to deal with that question. All they do in section 7 is repeat the RP ranking for the subset of students who indicated an interest in (for example) science. The RP methodology, as it stands, does not even try to deal with the particular form of revealed preference contained in an applicant's selection of target schools.

[/quote]
</p>

<p>Section 7 itself doesn't deal with the modeling of non-application. But the entire paper *attempts * to do so implicitly, through revealed preferences. That is how the paper can compare schools that are clearly in different leagues. For example, I would surmise that very few people are applying to both Harvard and Colorado State. Yet the model is able to infer that Harvard is more desirable than Colorado State through intermediary schools via the matriculation tournament. </p>

<p>
[quote]
No, that wasn't the question at all. It has nothing to do with the fact that two particular schools, happen to be misranked relative to each other. The problem is that the ranking methodology used is stablest and most accurate for the schools that will end up on top. If there is possibility for a school with less than 20 matriculation decisions to swing into the second place (other schools in the top 6 had hundreds of applications and dozens of matriculation battles covering all possible pairings), and to have its desirability mis-estimated so badly (50 Elo points higher than MIT when it's more like 200 lower), that tends to confirm that the model is unstable. If it is unstable at the very top, it only gets worse going down the list.

[/quote]
</p>

<p>Again, nobody is saying that the model is perfect. But I would hardly use the phraseology of 'fatal' or 'mis-estimated so badly'. After all, again, we have to compare the RP to the available alternatives. Honestly, what's better? USNews? Gourman? WSJ? Newsweek? Shanghai Jiao Tong? Whatever you might say about the RP, I would argue that it is at least as useful as those other rankings, unless you are willing to take the position that all of those other rankings also suffer from fatal errors. </p>

<p>
[quote]
No, Caltech vs MIT alone doesn't matter, it's the implication for the whole model. If Caltech beating MIT were happening with both schools below number 10, it would be a minor point. That it happens at number 2 and number 4, with a large Elo point discrepancy, combined with other information (Yale-Stanford and others), corroborates pre-existing suspicion from the mathematics, that the model is unstable with this amount of data.

[/quote]
</p>

<p>Again, I would argue that the model is no less unreliable than the alternatives. There was one particular year in USNews when Caltech was ranked #1. I would call that rather unreliable. </p>

<p>Look, siserune, the truth is, we suffer from a paucity of highly reliable college rankings. The RP isn't perfect, but it at least tries to be scientific, with a genuine theoretical workup and reasonably defensible model that is based on well-understood academic techniques. The results aren't perfect, but at least it's a good start. The other rankings don't even try to construct a theoretical workup. They don't even try to build an academically sound methodology. I would much prefer the RP, and follow-on papers that I hope will be inspired by it, over the other rankings.</p>

23

<p>haukim: Stanford focuses much of its recruitment effort in California (40%+ of Stanford admits are California residents). </p>

<p>Any person who has spent even 5 minutes talking with california applicants will tell you that Stanford is anti-California and has really no interest in doing anything that won't help their diversity stats. don't feel like getting into it, but California is a big state and most all the hooked applicants already live there.</p>

24

<p><a href="haukim:">quote</a>
I figure to rank such an extensive list of colleges, one school being listed above another should work something like this:</p>

<p>% of cross-admits to school A and school X won by school A
vs.
% of cross-admits to school B and school X won by school B</p>

<p>It is impossible to utilize head to head matchups between schools to formulate such a ranking due to natural matriculation advantages.

[/quote]
</p>

<p>Let me elaborate (what I think is) your point in a different way.</p>

<p>The problem with head to head matchups is that the types of students who get to choose between A and X can (and often will) differ systematically from those who choose between X and B. It means that RP ratings are most reliable in comparing similar schools which have highly overlapping applicant pools and a large number of mutual admits. So one expects that they ranked and got the cross-admit percentages largely correct for Harvard/Yale/Princeton, and probably got the ranking right for the battle of the top 3 LACs. As you add MIT and Stanford, the heterogeneity in the applicant pool should start to misalign the model with reality (unless given a mountain of data), and this will continue as one brings in Duke, the lower Ivies, the LACs, the second-tier research schools, and so on. Heterogeneity of applicants, as a source of instability in the model, increases as you go down the list of schools, as does the more basic problem of depletion of data at the lower ranks. That's the theoretical expectation, and to the limited extent that cross-admit rates are published or inferrable, it seems to be borne out.</p>

<p>(added: )</p>

<p>The effect of head-to-head comparisons is not in itself a criticism of the RP model. It is an analysis of what one would expect the behavior of the model to be, what it is good at and what its limitations might be. The modelers can use whatever method they like, based on head-to-head matchups, or the weather or the zodiac, and that's fine. What matters is how well the model describes the data (again, the ability to directly compare the ratings with cross-admit data is the only philosophical difference between RP and the more conventional rankings including USNews, JiaoTong, Newsweek and Gourman). The import of the theoretical observations is that they help make sense of what the available data tell us about the behavior of the model, given that the authors have not released the information to make that judgement directly.</p>

25

<p>replying to (first half of) sakky posting :</p>

<p>
[quote]
But the key difference is that the weighting of those [Revealed Preferences] model parameters is entirely endogenous to the model itself. Contrast that to, say, USNews, where the weighting parameters are entirely exogenous and hence entirely arbitrary.

[/quote]
</p>

<p>That's not a difference. Where US News makes choices from a 15-dimensional space of parameters, RP has an even larger space of modelling choices. One selects the shape of the probability distributions of desirability parameters, the multiple-comparison model (that of Luce), and any number of other things.</p>

<p>The only difference is that RP has a built-in performance metric, because it predicts certain observables: the cross-admit rates. It doesn't matter whether the model is philosophically justified or not, one can test whether it works. The problem is that the paper provides no direct indicator of how well or how badly the model describes the data. </p>

<p>
[quote]
What I am saying is that it is more informative than just looking at pure cross-admit (or multi-admit) data alone.

[/quote]
</p>

<p>Informative means information can be obtained from it. Consider the following thought experiment: an alien comes to earth, knowing nothing of colleges or their rankings. He is given the tables published in the Revealed Preferences study. What can he reliably conclude (reasonable certainty, but not necessarily 100 percent) about applicant behavior either in the sample cohort or for applicants nationwide? What can he conclude that isn't already in USNews, and when at variance with USNews and the other ratings, how can he evaluate which to trust?</p>

26

<p><a href="sakky:">quote</a>
Section 7 itself doesn't deal with the modeling of non-application. But the entire paper attempts to do so implicitly, through revealed preferences.

[/quote]
</p>

<p>That's incorrect. The preference information contained in the pattern of applications and non-applications has an overall NEGATIVE correlation with the rankings produced by the RP study.</p>

<p>To say that the RP model implicitly takes application decisions into account (through its analysis of matriculation decisions) would mean that adding school X to some of the matriculation battles should generally elevate X in the ratings, as it shows those consumers preferred X over some other schools (enough preference to apply there, if not necessarily to win the battle for X). In fact, the opposite is usually true. </p>

<p>Examples: </p>

<ol>
<li><p>A school gets publicity that makes it a very popular safety school. Lots of people add it to their application list due to this preference, but few opt to go there (it being a safety, and the RP list being derived from multiple acceptances, the people who got in only there don't count). The sudden popularity lowers the rating of this school, as it will lose a lot of tournaments that it wouldn't have participated in otherwise.</p></li>
<li><p>A student applies to all the top 10 schools except UPenn, is accepted everywhere, and selects the highest ranked school from the list (Harvard). Now suppose this person had added UPenn to the application list; the result would be to give even more points to Harvard for beating a larger roster of competitors, and to take away points from UPenn for having been "preferred" enough to gain an application.</p></li>
<li><p>A very strong student knows he wants either MIT or Stanford, will get into either, and doesn't care about any other schools. Applies to MIT, Stanford, and one in-state safety; gets in to all and chooses MIT. This student has a negative preference for Harvard, Princeton and all the others. However, not applying to those schools helps their ratings, as they don't lose to MIT; it hurts MIT, which would have collected more points by beating a larger list of schools; and it hurts Stanford, whose loss to MIT is not "distributed" among several fellow losers. So the strong preference for MIT and Stanford revealed in the application pattern actually hurts both schools and helps all others (except the safety) in the RP ratings.</p></li>
</ol>

27

<p>Thanks for the examples in post #26. I hear by email from one of the working paper </p>

<p><a href="http://www.economics.harvard.edu/faculty/hoxby/papers/revealedprefranking.pdf%5B/url%5D"&gt;http://www.economics.harvard.edu/faculty/hoxby/papers/revealedprefranking.pdf&lt;/a> </p>

<p>co-authors that the working paper is now being prepared for formal, peer-reviewed journal publication. I've been surprised over the last couple years that there hasn't been more published back-and-forth about the working paper's methodology. It should be helpful to scholarly discussion for a current, updated (but presumably based on the same data set) draft of the paper to go through the peer review process for journal publication, because then perhaps some of these kinds of questions will have to be confronted in that draft. If they are not confronted in the published version of the paper, I suppose those kinds of questions would be issues to bring up in a reply paper in the same journal or in another journal in the same discipline.</p>