Reuters: How Asian test-prep companies swiftly exposed the brand-new SAT

Test Preparation
1

“Booklets for the redesigned exam leaked online within days of the test. The ongoing failures to secure the SAT are prompting some college officials to question the validity of exam scores.”

http://www.reuters.com/investigates/special-report/college-sat-two/

CC gets mentioned in passing. Looks like College Board has a major issue to deal with, and that improved technology is the only solution. Obviously, single-use tests are essential. But, as long as the exact same tests are administered across the globe in different time zones, leakage will be a problem.

Certification tests for companies like Microsoft and Cisco have used randomized tests offered on-demand electronically for many years. That would be a huge change and would require major infrastructure investment, but it would avoid test-leaking scandals like this one.

It probably wouldn’t work for TCB, but language instruction firm Duolingo is allowing people to take tests on their own device. They reduce cheating by monitoring the test taker in real time from a central proctor who views the device’s front-facing cam. One proctor can monitor multiple test-takers.

2

As I wrote in the thread about the test registration cancellations, barring adult test-takers was just for PR and not at all an effective measure against cheating. At least Reuters has brought a couple of the gaping holes in SAT test security to public attention. There are still others.

There are different test security models used by different tests. The other main paper test is the LSAT - a one and done test. Obviously, it is also a much more expensive test. However, I don’t see how CB could move the SAT online, even to the test-taker’s own device, in the near future. That would go completely against the CB campaign to “open doors” to underprivileged students. Would underprivileged students have suitable devices and know how to use them sufficiently well? Would not digitalizing the SAT to own-devices reintroduce special hurdles for the underprivileged?

3

Good points, @Plotinus . Of course, mobile devices have been used to make services like banking available to millions on underprivileged people with limited access to physical banks. Obviously, this is a hugely complex effort with different security concerns. I wouldn’t expect to see it happen for many years, even if a practical approach emerged.

One option that would provide easy access would be a proctored exam using physical computers or tablets in existing schools. Test times could be booked in advance and conducted over a period of days or longer so you wouldn’t need one device per test taker. Randomization of questions from a large database would be essential, though, to prevent earlier test takers from providing significant help to later ones.

4

Underprivileged people and especially developing countries have been on the forefront of mobile technology because that’s often the only bank, source of information, source of education, map… there is. Literally.
(Kenya was about 5 years ahead of the US in terms of mobile banking, for instance).

ETS digitalized the TOEFL years ago - and its format is similar enough to verbal (CR) on the SAT that I don’t see why the SAT is on paper, not
Devices could be provided - which school in the US doesn’t have computers/tablets? A time to practice with the device would be included before each test, just in case.
Abroad, the situation could be case-by-case, depending on country. Kenya, Germany, China, and Nepal have different problems.
Front-facing cameras monitoring test takers and randomization of questions would be an obvious solution, too.

5

The TOEFL has a very different target population of test-takers than does the SAT.
Do you think so many underprivileged internationals are taking the TOEFL and the SAT? The problems for digitalizing the SAT are mainly the underprivileged in the US.

6

If the real concern were test security, wouldn’t it be easier and less expensive to stop recycling tests than to move the SAT to a digital platform? Oh, I forgot. Moving to a digital platform would mean more money for Big Data. Bill Gates must be smiling.

7

I do wonder if a digital test wouldn’t be more vulnerable to cheating, in some ways. I can see two.

  1. Questions would have to be separated into different levels of difficulty. Otherwise, a "true 600" student might see more difficult questions than the average and get a 500 on a given section, or a "true 700" student could dodge the handful of questions that your average test would include to separate 700s from 800s, and get an 800. Rather than separate questions into tests (300-400 questions each), the CB would have to separate them into difficulty levels (each a large fraction of the question bank, likely 10-20%) in order to randomize the SAT. Leaking a single difficulty level compromises the test until the full level can be rewritten.
  2. The question bank would have to be small enough for the CB to write the questions and test their difficulty with an extensive control group. That fact limits its size; compiling 5,000 questions would be a time-consuming and expensive endeavor. This in turn makes it beneficial for a student to book a time at the end of the test period and read questions as they're posted online. This is true whether the test is on-demand for 9 months of a year or for any shorter period - say, a week. An on-demand test would mean a larger question bank, but also give cheats more time. A 5-day test period (for instance) gives cheaters less time, but also makes their job easier by limiting the size of the question bank, because the CB would need several sittings each year to accommodate all test-takers.

Realistically, a student can cover 100 questions in a day, at the very least - the SAT itself asks them to answer far more, in far less time. 300-400 is a more likely figure. This means a 1,000-question database for a 2-3 day test period is wholly inadequate to prevent cheating; even 5,000 questions would be iffy, and I doubt the College Board has the time, the resources, or the will to produce and test 35,000 questions and accommodate 7 yearly sittings.

The other option is a massive question bank, well into the tens of thousands, available on demand. But this couldn’t be updated all that often, making it vulnerable to cheating, an issue solved only by reducing the benefits of such an approach with 100,000+ questions.

8

I posted this stuff in another thread, but one need only compare what the ACT has been doing right to what the College Board hasn’t been doing right in order to understand that the fault is completely the CB’s own!

In the summer of June 2014, when rumors of leakage pertaining to the ACT first surfaced in South Korea, the ACT took drastic preventive measures in its subsequent administration of the September ACT exam in South Korea and decided to use two different forms or tests in South Korea alone (60E in South Jeolla-do Province and 69F in the Seoul district), not to mention the fact they mixed up at least 5 different forms in Asia for that particular month. The message from ACT was clear: whether the rumors were true or not, it was not going to take any chances.

In that same year, the College Board used the December 2012 exam internationally in January of 2014, an exam that previously had been recycled on at least two occasions both in the U.S. and elsewhere, even despite overwhelming evidence obtained from the South Korean Prosecutors’ Office of South Korea, District of Yeok-sam-dong (Prosecutor **** oversaw the confiscation and seizure of illegal materials from at least 44 teachers/academies in South Korea, 2013 and this exam was one of many included in that blacklist) that the exam was seriously compromised and should never be used again.

Moreover, what a hoax this whole “lock-boxed” security procedure for the SAT is!

In 2015, when it was announced that the College Board or ETS would be sending “padlocked boxes” to Asia in an effort to thwart the theft of tests from the boxes in which they were delivered, it did so only for fewer than 5 schools (Shanghai American School in China and Seoul International School in South Korea being the main two in which they suspected leakage) in Asia, while the remaining thousand schools dispersed throughout Asia simply received “taped” boxes that any teacher or proctor could have easily stolen a test from. What a publicity stunt! And one wonders why the May 2015 administration of the SAT was leaked in Egypt, the U.S, and all over the world, and I will not even go into what happened in Oct 2015, Nov 2015, and Dec 2015, and Jan 2016.

Conversely, the ACT actually dispatched a team of “security agents” throughout Asia to oversee the administration, storage and proper distribution of its tests and provided detailed training to supervisors, not to mention everything else
they did to prevent any type of leakage from anywhere in Asia!

AND HOW COME THE COLLEGE BOARD ISN’T DOING ANYTHING ABOUT ITS OWN SUPERVISORS AND PROCTORS RESPONSIBLE FOR THE SECURITY OF THE TESTS WHO ARE THE ONES ACTUALLY SELLING THE TESTS DEMANDING RIDICULOUS SUMS OF MONEY, SUCH AS $50,000, FROM THE TEACHERS/BROKERS IN KOREA.CHINA???

It’s so sickening how complicit the College Board and ETS are in this whole scam that I’m shocked to see why nobody sees what is really going on. And the blame is put on the “cartel-like” organizations who will stop at nothing “to cheat, lie and steal.”? ARE YOU KIDDING ME???

9

Ugh. The more you know…

Also, the venerable prosecutor’s last name isn’t going to escape CC’s filters.

10

CC is mentioned more than just in passing. Aside from other references about how test material is shared on this Board, the closing of the article states: “Have you read College Confidential?” Roy said in an interview. “All the detailed material is there.”
This makes me sick.
Is this Board perpetuating the cheating as the article suggests?
If so, what is being done to stop it?

11

Reddit had SO MUCH more detailed information than CC.

I don’t get why CB does not use several different forms abroad - randomized by location. That would help. Isn’t that what they did here in March?

12

I understand the need for security and absolutely agree that no tests should ever be recycled by CollegeBoard. However, I don’t know if giving the test online or on a electronic device is really the answer.

I’m sure it is possible, but it would be complicated and might lower scores for students. My D17 has to take state testing online every year and she hates it. Often times, the reading or science passage is not visible with the questions, so it requires clicking back and forth between windows. The math sections are quite annoying when scratch work has to occur completely separately from the actual problem, requiring the student to look from the screen to the paper over and over again, transferring the information between each surface. Also, staring at a computer screen for 4 hours is enough to strain anyone’s eyes.

I’ll be interested to see what ends up happening in the realm of electronic testing, but one thing is for sure, CB needs to stop recycling tests.

13

http://twelvebytwelve.net/xts/finger-of-blame.html

Written in 2013.

ABSOLUTELY PROPHETIC.

“And the contemporary testing regime inflicts injury even when no cheating has taken place. For example, a test may be cancelled in anticipation of cheating, with all examinees being forced to take the test at a later time, which may result in the missing of application deadlines and therefore delay of further study by a year. Or an examinee who scores extraordinarily high may be suspected of cheating, and distrusted, even though he came by his score honestly.”

“The Educational Testing Service has historically accepted no responsibility for the cheating that takes place on its exams, instead blaming students and their tutors. However, today’s students do not have to take this blame lying down because the Internet has made available to them information that students in earlier times lacked — today’s student writing an ETS test is more likely than ever before to know that some, perhaps many, of the students being tested along with him are cheating. In some examination settings considered in a broader discussion of cheating, an examinee’s wondering whether he is the only one in the hall not cheating may spring less from paranoia than from being well-informed. It is the arrival of examinee awareness of massive cheating that is able to reverse the attribution of responsibility, as explained below.”

http://www.reuters.com/investigates/special-report/college-sat-one/

According to the PowerPoint, College Board officials also considered another approach: Push ahead with all scheduled tests in every country, regardless of the security risks. Under this scenario, one of the “benefits” listed was giving the “appearance that security situation is under control.”

UNBELIEVABLE.

14

@NotVerySmart,

According to their 990 for the year ending in June 2014, the College Board made a profit of about $99 million in that year. Total assets in that year were about $1 billion.

I think the College Board has the resources to create one-time tests. If they want to.

I think the US Congress should give them the will. The College Board is a non-profit. It and the ACT are effectively a duopoly. The security of college admissions tests should be their highest priority. The sloppiness with which the College Board approaches this sacred trust should be reason enough to remove their tax-exempt status.

Or they could tighten up security. Their choice.

15

Lets again blame CB for the test hacked and copied . Its always their fault. No matter how tight they make their security, someone (s) from certain countries will hack/steal/compromise it.

16

@periwinkle

Exactly!

Congress’s role is to force companies to create better products and services!

That is exactly why Congress was created.

After Congress fixes this, I hope they can force Apple to not have my apps update so often on my phone!! Senator McCain asked about this but Congress has NOT yet “give them the will” to fix this!!

http://politicalticker.blogs.cnn.com/2013/05/21/mccain-why-the-hell-do-iphone-apps-need-updating/

17

Exactly. The CB is on the record as saying that it costs about $1 million to create a test. With clear, untaxed profit of $99 million, the CB absolutely has the resources to use unique tests in every test administration. The College Board must stop recycling tests.

18

Good for Reuters for doing some real reporting here, but the March 2016 cover page they reproduce in their story looks fake to me. A scan wouldn’t look that crisp, and I don’t think College Board says things like “SAT Test,” which sounds more like non-native English. I could be wrong, of course–but it looks to me like someone took the cover page of one of the College Board’s practice tests and just doctored it a bit.

19

They’re on the record saying that the contents of that exam match reports of the test that was given in March and have run it by students who took the March exam, all of whom said it was the exact same test. I’m very inclined to believe the report, partly because of what I’ve heard from students here in Asia, but also because, hey, this is Reuters, not Buzzfeed, and Steve Stecklow has won a Pulitzer and three Polks. @jgoggs

20

I’m not doubting the report. I’m only doubting that cover. But again, I could be wrong.