There is a large variety of obsessions at display here at CC forums and most of us wonder at other people’s amazing obsessions while completely unaware of our own obsessions. Not any different from any other sample group of humans.
There are many categories which make up any given rating & ranking system. Some categories are very important to applicants & their families–such as average net cost & average class size & job placement statistics.
So much is major dependent. And the specific courses taken even within a major. Columbia traditionally offered A+ grades for exceptional work - not sure if it still does - so that can up the perception of difficulty for the average student (just like at all the Ivies, the very best kids are in a totally different class from the average student there - and the A+ at Columbia recognized at least some of that).
All that being said, the answer to which elite school is the easiest is always the same: Brown 
In terms of academic rigor, Caltech probably has no equal. Part of the reason is everyone there has to take the same physics sequence, regardless of majors. The famous physics problem of toilet paper unrolling while dropping is just one of the examples. Unlike Harvey Mudd or MIT, Caltech is on a quarter system and its courses tend to proceed at a faster pace. Most of its students also take at least 5 full courses each term (quarter).
On the other hand, UChicago doesn’t seem to be as rigorous as many people thought:
http://talk.qa.collegeconfidential.com/university-chicago/2155668-uchicago-core-sequences.html#latest
It will be hard to rearrange this list, I would have to add some universities/colleges that aren’t here.