Dear Current, Past or Future College Applicants and their parents:
US News & World Report (and other college ranking systems) gives the appearance that the colleges and universities are being compared using the same data. Even those examining the formulas used may not realize that different information is being used across schools in a way that makes them appear as if they are using comparable data but are not. I am calling for those involved with the Common Data Set and the other ranking systems to establish definitions for their variables that are used across all schools. That would be easy for these researchers to accomplish and would keep schools from playing with their data.
Applicants and parents have a right to know what information is being used to generate the ranking lists. Please weigh in here by contributing to this thread in an effort to help make the process more transparent for future applicants.
What is an applicant to each school? If some schools use the term to include anyone who pays a deposit and other schools use the term to include only those who submitted all the material they list as "required" the resulting list is deceptive because are being compared using apples and oranges. This is easy enough for these lists to systematize/operationally define.
When objective data (Test scores, grades) are included, it should include data from all students who submitted the data so schools can't just include cherry picked information. If all data isn't included, those reading the list should have a way to know that a school failed to comply and instead, cherry picked data. If is rumored that at least one state school fails to include scores from a considerable number of students including EOP students, Advantage Program (and other program) students and International students when the international student submitted SAT or ACT scores for review. If this is true, the 25%-75% is skewed far upwards and does not represent the scores of the students who end up in seats at that university. If other schools are misrepresenting their data in a similar or different way, the data is inaccurate and should not be included. The premise, and the assumption of those who use the ranking lists, is that the data used includes scores for all students who submitted scores as part of their application regardless of whether or not the scores were required.
When scores are depicted, are they the highest scores each student submitted, super-scored scores or the first set submitted? The numbers that are used that constitute each variable needs to be clearly defined.
This call is a request that those involved in research related to data from the Common Data Set and data used for other purposes provide operational and precise definitions of each of the variables used so that each college conforms to the standard or indicates its refusal to do so. That would “clean up” the data and ensure that schools are being compared on comparable data.
At this point US News and World Report College Ranking data includes too much noise to be valid. Those involved in generating the report know where the error lies. The fact that each variable contains a mixture of kinds of data is unacceptable and therefore the lists are deceptive. The definition of each variable differs across school. This fact is obscured because the researcher uses the data as if it represents the same information for each institution. Researchers have an obligation to make known sources of error clear. (For example, it would be incorrect to state that the average height of a group of 6 people is 6 feet after averaging numbers using the following-3 people at 5" s and 3 people at 7 feet)
These lists now have too much power to allow them to continue to contain known sources of error without correcting them. The correction is relatively easy. If the schools can’t be forced to submit data that conforms to a standardized definition of each variable they can be asked to be transparent about what it is that is included. Respondents can be asked to place a check next to one of a list of possible definitions for each variable. For example: "X defines an applicant as someone who submitted all required material for admission. Please indicate the way your institution defines an applicant: A) all material required for the application, b) any student who indicates interest or who indicates a plan to apply, c)…
Define your variables!
Dear Applicants and Parents of Applicants, please indicate what you think about this call in the discussion below.
I’ll start. When applicants read about the % of applicants accepted to a particular school in the Common Data set they should know what that means.
School A: 15,000 indicate interest by sending in a postcard 10,000 send in application fees 5,000 completed applications 3,000 get acceptances
Is the acceptance rate:
A. 3000/15000=20%
B. 3000/10000=30%
C. 3000/5000=60%
School A looks different depending upon which of the calculations are used. Don’t you want to know how applicant is defined by each school you are considering. Even if you don’t care, these numbers are used for other purposes too. The rankings often consider these numbers. Can you believe that they don’t consider how the school is defining applicant. They just use whatever number the college puts into the line for # of Applicants:___________________?
It is easy enough for the researchers to ensure they are using the same formula for % accepted for each school by defining the term and asking schools to include only numbers from “applicants” who fit their definition. What do you think? Is it time for applicants to request more transparency when it is easy enough for the researchers to be clearer?
Schools are going to game because they have an incentive to game because applicants use gameable data and rankings to decide which school is better. The proper thing to do then is to judge schools (if you are going to) by non-gameable criteria.
The real problem to me is that US News and World Report is asking for data from the colleges when the data has already been collected and overseen by the department of education. US News says it collects the data from colleges but for those colleges that refuse to give data they pull it from NCES - The National Center for Education Statistics - specifically IPEDS which is data used for reporting is through the Integrated Postsecondary Education Data System. This is the data that is required by the Department of Education and is reported to the feds at three different times each year.
IPEDS has a glossary which is used by each school to (supposedly) ensure that data is correctly defined and that apples are indeed compared to apples. Any school that uses federal funding is required to input data or risk loosing federal aid. This is true for the little school that teaches cosmetology to the large state schools, to the elite private schools.
My question is, if the department of ed us already supervising the collection of data in an apples to apples format, then why not just use this data. It is publicly available! I would assume that most schools will use IPEDS data for US News and other ranking systems, since it is easy to do and already collected and packaged but why not take away that option and used federally supervised and collected data.
People may be better served by the College Navigator which is based off of IPEDS data. It does not rank, but instead uses apples to apples data which allows a student to search out colleges and weigh differences. It it not a perfect system because IPEDS is problematic as well, but it is overseen and evaluated.
I was an institutional researcher for several years and my job was to report this data to various entities. We were an unranked school so I did not report to US News but I did report to the college board and other places. I always used IPEDS data since it was already collected and easier but I had the ability to pick and choose. Why not take that ability away and make it all the same?
LKnomad, Where exactly does the data for the NCES get collected. Are you sure that the colleges aren’t simply using the same data they use for the Common Data Set? Is that data complete? Is there a way to determine if all scores are included for any given school for instance? A comparison of the data published in US News to the NCES data would be very interesting. I have to go out but will look at your links when I return. Thank you for posting that.
I was curious and so did a comparison between the NCES and the Common Data set that which I know excludes some relevant data and they match. My guess is that a school that defines a applicant one way on the NCES does the same for the CDS. And, if they conveniently omit scores they don’t want to count (Advantage program, EOP, Internationals, and others) on one they do the same on the other. I will look more closely later.
@lostaccount I have more info but I am at a show rehearsal! Will check back for questions. I know this stuff well and can answer anything you are interested in. This was my job for quite awhile.
“An individual who has fulfilled the institution’s requirements to be considered for admission (including payment or waiving of the application fee, if any) and who has been notified of one of the following actions: admission, nonadmission, placement on waiting list, or application withdrawn by applicant or institution.”
Does this mean that when a student’s data is missing anything listed by the college or university as required, that person is not considered an applicant and that person does not add anything to the variable called “# of applicants”?
I actually had a long conversation with the IPEDS people about this. They told me that I should count people who received some sort of response/letter - accepted, denied, waitlisted. We hada wierd admission procedure so our office went round and round about this.
But there is also a catch. IPEDS only wants a count of first time students. For our instition this meant anyone who entered without any credits. (Unless they earned then the summer before the fall start date.) So this could mean an instition may not be including students with dual enrollment credits OR it might mean they are! It could mean many different things.
Without any AP credits either? Can they leave out EOP students or the scores for international students if the students send them despite not being required? Seems to me that if the Common Data Set clearly defined each variable then the data would be even cleaner than what you are describing. The NICEs data and Common Data Set are identical for a school I know excludes student scores from their 25-75% and from the averages when shown. I don’t know if they fudge the number of admissions but I would not be surprised.
My school did not accept APs but our interpretation would not have included AP credits as college credits. We had a high rate of students who attended other colleges but were not able to make it into their health science programs - nursing specifically - so we had a lot of students who started elsewhere then came to us. None of those students were counted in our numbers because they were not first time students.
There would be no reason to leave out other types of students and I don’t think IPEDS would allow it. There is a count later which further divides enrolled students including internationals.
You may want to actually look at the paperwork that the schools fil out. If you go to the IPEDS data center and look up a specific school you will get a better idea of what they ask.
Choose look up an institution - Then type in an institution - when you have the institution selected then choose reported data. When you look at the data you will see the actual forms that are used, with the instructions. The data is a bit older than if you were to look up the info in other parts of the data center but this is how to view the actual forms.
You can also explore the entire data center. It will allow you to pull large amounts of data to view in Excel or Stats programs.
ALso to answer your earlier questions. @lostaccount - the data gets collected online through an interactive IPEDS system. You also have one thing backwards, the common data set would use data from IPEDS not the other way around. The actual CDS questions were fashioned after IPEDS. IPEDS came first and schools have been reporting to IPEDS for years. All institutions are required to report to IPEDS BUT not all institutions choose to fill out the CDS, it is voluntary and not overseen. You can also see IPEDS data from years past if you want to look at trends.
A smart institutional researcher would just pull their IPEDS data for the CDS or their fact book- that is what I did. The IPEDS data is easy to use but there is no rule requiring it.
It would also be helpful if graduation rates were reported in number of school terms (semesters or quarters) as well as calendar years. At schools where many students do co-ops or otherwise take time off of school, that can give a distorted impression of graduation rates.
Can a college simply decide that they won’t include data from special programs (say EOP) in their SAT data? I have it from an authority that one particular school did not include SATs from some applicants in the computation of their scores (for US News). Since the IPEDs and CDS match, that implies that the school is also not reporting all scores they have from admitted students in the IPEDS. I assume there is no test of the veracity of each school’s report.
@lostaccount. No they cannot pick and choose. They risk losing federal funding if they do. IPEDs does look for irregularities (changes from year to year for example) and will contact a school if they feel something is off.
I am afraid that LKNomad overstates the usefulness of the IPEDS/NCES data for ., the common mortals. Inasmuch as school officials have the ability to pull data, most people will find the data published by the government to be both older and less complete than the data culled by the joint efforts of USNews, Peterson, et al.
The IPEDS data is still for the Fall of 2013. One cannot find numbers about early applications.
The reason why the CDS came into existence was to shore up the deficiencies and timelessness of the government data publishing.
There is a lot to government could do, and the easiest step would be to FORCE schools to release complete and accurate data with 90 days of collection via its PUBLIC website. Full data about admissions, including early applications, deferrals, and all that good stuff. Obviously, it is too simple for the morons who work for the government or for institutions who prefer to hide behind the government minimum requirements.
With public data on the websites, there would be no need to even look at the NCES data.
Actually the schools do collect and report data quickly. The data on admissions for Fall is collected in October. It includes enrollment data so there is a wait time so that schools can count students that actually show up the first day of class. That cannot be collected until the schools make a final count. Many have to run a census first.
Data on finance is collected in winter. Data on grad rates is collected in Spring.
There is also significant data collection for accrediting agencies that is never made public. WASC required significant data but it is not made available. Also individual programs from within an instition report data to their accrediting agency. Nursing for example is collected on a state and national level.
This data is available to schools but it does take awhile to show up for the public. As the “keyholder” I had access to early data from all institutions. As I mentioned in another post, the data is checked and there are around 7000 schools. The comon data set, would typically be pulled from the IPEDS data which is pulled on the above schedule.
The problem is that must people are only looking at a handful of schools and they want their answers now. In reality the government is overseeing thousands of schools from the tiny plumbing program in a rural town to the large schools such as UCLA. Huge systems are actually more complex because they report to their parent and the parent reports to IPEDS. There is an extra step and an extra check.
This data collection is very complex and takes time. It is not as simple as counting a bunch of applications.
You are talking about apples and oranges, and mixing them!
For starters, the CDS organization does not rely on the IPEDS as a primary source. The instrument used is the Common Data Survey. While the information should in theory be identical, the reports --as you must know if you submitted data to both-- are different. Here is a link: http://www.commondataset.org/
The discussion about what is available to “insiders” who work in institutional research of reporting offices at the universities and college is IRRELEVANT to the applicants and … people discussing such matters here. The bulk of the information is not that important but salient and timely data on admission rates, graduation, class sizes, etc IS important.
All here understand that the final numbers cannot be collated until they are … in! However, there are no reasons whatsoever what schools are not forced to disclose their EA or ED rates by March 31 and preliminary admission numbers by June 1st including WL movements. If there are schools that cannot complete such reports, chances are that they are … not exactly the schools discussed with fervor around this site. On the other hand, most selective schools should have no problems disclosing the information … if forced to do so. This will be eliminate schools a la Columbia, Chicago and other WUSTL that prefer to keep all data on FYEO!
What the government collects via NCES and IPEDS is mostly out of date and not so relevant to … students. It is typical government data: mostly comprehensive, mostly interesting to professionals, and of very little use to people who should gain information from it, namely applicants!
Again, the solution is simple. If a school receives federal funding, they could be forced to disclose preliminary data on a timely basis. The reports could be segregated between the trivial details about faculty resources and other numbers most people never look at --and leave it to the data junkies-- but have basic numbers disclosed earlier that COULD be helpful!
This is not rocket science, but as usual, the combination of government employees and school administrators remains as lethal and unproductive as ever.