I’ll remind users that College Confidential is not a debate society. Make your point then move on, please.
I don’t think the authors of this study would call an Iowa State Engineering degree worthless. It is anything but worthless. However, the authors of this study do feel that someone’s chances to hold certain important positions, and to make an income in the upper most tiers, are greater from an Ivy + university.
The Iowa State degree is not bereft of any advantages or potential advantages. An Ivy+ degree offer advantages and potential advantages that the Iowa State degree does not. Hence, the term, “It is the student, not the school” would not be entirely accurate. It is the student and the school.
If I haven’t made this clear by now, I’ll let it go.
1 Like
It’s not surprising that GREs are really bad as predictive tools for PhD programs. Success in a PhD program is dependant on research abilities, which are not something that GREs test for. Of course there is also the fact that a PhD is supposed to be a body of original work, and standardized tests aren’t really the best way to test whether a person has the ability to create something original.
3 Likes
There were some interesting summary statistics about how 8% of the top 1% of households in income went to an Ivy+ undergrad and 13% of the top 0.1% of households.
I wonder if there was data available that showed a summary for the top 1% just in terms of SAT/ACT scores above 1500.
And if there was data showing what the percentage break down was for the top 0.01% of incomes that went to an Ivy undergrad. I suspect that the percentage would be even higher than 13% but I’d be curious to see whether the trend continued.
A household income of $650k is the top 1%
A household income of $3m+ is the top 0.1%
Right, and that’s GRE/PhD mismatch is a contrast with law and medical schools, which are training you to practice a profession that requires internal and external exams along the way. Standardized tests make a lot more sense there.
I’ve been a law school administrator, law school admissions consultant, and LSAT tutor over the last 24 years. My theory, with no empirical support, is that the LSAT would predict bar exam passage even if they made it into a test of math or some other content less related to law. A lot of people are just strong or weak standardized test takers.
4 Likes
Not quite directly on point, but this article on time-limited tests has a lot of empirical data on what time-limited tests are really testing for:
I know there is a lot more out there on this issue, but I think this at least indirectly confirms your theory that similar time-limited tests are often more testing for a common ability to do well on similar time-limited tests than anything very specific to the content.
1 Like
Tests really only need time limits when they’re too simple. Math Olympiads (both national and international), for example, give contestants 9 hours over 2 days to solve 6 problems (if a contestant knows how to solve a problem, it wouldn’t take more than 5 minutes each).
2 Likes
In some things, timing matters. Surgeons. Airline pilots.
1 Like
Yep, speed itself is a valuable skill in some professions.
1 Like
Yes, and here is a relevant passage from the paper on that subject:
Untimed power tests have been regarded, for nearly a century, as “the ideal” (Peak & Boring, 1926, p. 72) because they best assess the “complexity of … thought-processes” (Line & Kaplan, 1932, p. 1). In contrast, time-limited tests are good only for assessing tasks that are so cognitively simple that everyone would respond correctly if no time limit was imposed (Danthiir, Wilhelm, & Schacht, 2005; DuBois, 1932; Mead & Drasgow, 1993).
This is all part of why I find it hard to take tests like the PSAT, ACT, SAT, LSAT, and so on seriously in terms of predicting ability at any sort of real world task involving “complexity of thought processes”. As in, pretty much all the real world tasks I actually care about.
2 Likes
Sports too. The NFL has famously used the Wonderlic Test since the 1960s as part of the draft evaluation process.
And empirical studies have found it is pretty much useless in predicting NFL performance.
This suggests the ability to do these sorts of tests quickly is not predictive of being able to do other sorts of cognitive tasks quickly. As I understand it, there is actually a lot of study about what, say, makes for a good fighter pilot, and they need all sorts of attributes like situational awareness, being able to mentally model dynamic 3D spatial relationships using individually more limited information sources, and so on.
There are surely versions of this for anyone who has to perform rapid tasks in complex real world situations, including in sports and so on. And so it makes sense something like the Wonderlic would be far too limited to be usefully predictive.
Why would sports have rapid cognitive response times required? Pilots may have a few seconds to correct before a plane crashes; surgeons need to ensure patients do not bleed out.
AFAIK only the NFL requires the Wonderlic test. Rapid cognitive response time is a definite advantage for many positions…think QB reacting to a change in the positioning of the defense, everyone on both sides of the ball when a QB calls an audible, etc. With that said, many of the best QBs totally bombed the Wonderlic test…Marino, Bradshaw, Kelly, I’m sure there are more.
Then it appears what it is testing has little correlation to ability in football
So, consider an NFL quarterback. On a given play, that have to survey the initial defensive alignment, potentially change the play call . . . and the defense then might do a last second shift.
The ball is then snapped, and the defense might then do further optional things like blitzes, fake blitzes, and so on. Meanwhile, different things are happening all over the field: some pass rushers might be beating the offensive line, some wide receivers might be beating their cornerbacks, some linebackers and safeties might be reacting to help, and so on.
The quarterback has to read all this and then make decisions about what to do, including whether to hand off (if that was a play option), whether and where and how to pass, whether and where and how to run, and so on.
And the quarterback has to keep updating these decisions as the play continues to develop, say running out of the pocket to escape pressure, while looking for options to find an open receiver, or potentially an angle to try to run, or potentially throwing the ball safely out of bounds. And they have to make these decisions understanding how this could all affect the game given the context of the down, the score, the play clock, and so on.
And then do it all again next play.
It is no wonder the NFL wants a test of whether a quarterback has a fast mental processing speed. It just turns out the Wonderlic isn’t working. Which again makes sense, because so much of what I just described isn’t plausibly going to be tested by something like the Wonderlic.
This sort of thing happens all over sports. There is an admittedly overused phrase in basketball called “basketball IQ”, which is similarly about being able to rapidly process dynamic spatial relations, with a defense or offense trying to trick you if possible, and make good decisions about what sorts of plays to execute. This is typically most associated with “point guards”, the sort of QBs of basketball, but increasingly in modern basketball players all over the floor are involved in complex playmaking, and defending complex plays.
And on and on.
Indeed, I would suggest many professional team sports are specifically designed to de facto test for this sort of thing. The game rules are set up to make complex and variable interactions between the players the key to success, offenses and defenses are intentionally given all sorts of opportunities for trickery, and time pressure is imposed in various ways including with play clocks and so on. Of course some sports, and some sports positions, involve this more than others.
4 Likes
Exactly.
The problem with tests like the Wonderlic is they test whether you are good at taking tests like the Wonderlic.
They are not good at testing your ability to do other significantly different sorts of rapid cognitive tasks, including the sorts of rapid cognitive tasks required of professional quarterbacks.
By the way, I just learned the NFL in 2022 finally abandoned the Wonderlic, after an internal audit of combine assessments. Apparently they are now using proprietary tests that emphasize spatial awareness and other attributes specific to fast decision making in athletic contexts.
I believe that is more or less the history of the Wonderlic in military aviation too. Like, the Navy tried it out for aviators, but then (quickly) abandoned it when it proved to have very little predictive ability for things like flight failures.
3 Likes
In many sports, the player needs to make instant decisions on which of various choices of action to do, often based on what the opponents are doing. Does the quarterback being seeing defensive players approaching throw the ball, and to which receiver, or hold on to it and scramble? Does the basketball player with 3 seconds on the shot clock try the shot or try to pass it to another player who may get a better shot?
1 Like
Absolutely.
Moreover, as you likely know better than I do, law and medicine require the ability to memorize large amounts of data, and to be able to access those memories when faced with a situation which is similar. Exams can indeed test the extent of these skills.
Since these occupations require a standard license, the tests for these licenses have to be standardized.
1 Like
Just adding a source on the Wonderlic’s history with Naval aviators:
According to Fiske (1947b), the initial pilot selection battery consisted of three tests, all of which were developed and/or validated with the help of the CSTAP. The first was the Wonderlic Personnel Test, a brief (12 min.) intelligence test. The biserial correlation for the least restricted cohort to pass/fail from ground school was rbis = .31 and rbis = .12 for flight failures with 2,356 students. The low predictive validity to flight failures combined with the demonstrated nonequivalence of the three versions of this test led the Navy to search for another test with better properties. Consequently, the Wonderlic was replaced in October, 1942 with the Aviation Classification Test, a longer (45 min.) intelligence test. Fiske presents no data on the predictive validity of this test.
As I understand it, they kept modifying that testing regime and eventually developed what was called the Aviation Selection Test Battery, which is a series of subtests specifically designed to test different attributes relevant to pilots and flight officers. The ASTB continues to be refined, and is now on “series E”:
As explained there, among other things:
The ASTB-E now includes the Performance Based Measures (PBM) Battery, which assesses the examinee’s ability to think in three dimensions, physical dexterity, eye-hand coordination, and ability to divide attention among different tasks.
I’m pointing all this out just to note that the Navy, like the NFL, has not abandoned testing entirely. Instead, they have adopted testing regimes specifically designed to test for the attributes relevant to their particular rapid cognitive tasks.
Speaking just for the law part, I would suggest in modern legal practice what you really need is more of what I think of as an indexing memory.
Because of the rise of massive and yet searchable electronic information storage, clients, courts, jurors and such now have a quite high standard for what breadth and depth of information they expect. No one can really carry that around in their heads (or at least virtually no one), so you have to be good at being able to quickly access, and re-access, needed information on demand.
The term I am using, indexing memory, comes from articles like this:
https://www.sciencedirect.com/science/article/pii/S0896627320305286
Relevant quote:
A few keywords typed into the Google search bar will, more often than not, immediately lead to the exact piece of information we are seeking. While the details of how this magic happens are proprietary, the general idea is transparent; Google has managed to index vast swaths of the internet and uses our search terms to quickly point to the most appropriate information (Google Search - What is Google Search and How Does it Work). The system is surprisingly flexible, using history, context, or location to hone results; completing or anticipating partial bits of information; and finding and separating similar items by detecting small differences. These properties, which underlie both its efficiency and popularity, echo the abilities of the memory systems operating in our own brains, particularly the episodic memory circuits dependent on the hippocampus (Squire et al., 2004; Tulving, 2002). The hippocampus is crucial for the encoding of memory, it is adept at integrating and interpreting contextual cues to drive recall, and it is efficient at both discrimination and association (Maren et al., 2013). Thus, much like how Google works as an index of information, one parsimonious explanation for hippocampal function is that it functions as an index of memories (Guo et al., 2018; Miller and Sahay, 2019; Tanaka, 2020; Tanaka and McHugh, 2018; Tanaka et al., 2018; Tonegawa et al., 2018).
I wouldn’t offer this as a definitive theory, but it definitely seems to me like increasingly the people who are good at the sorts of things the hippocampus apparently does for our internal memory are best suited for navigating the demands of the modern legal system.
You know, until AIs replace us all in a year or so.
4 Likes