“Colleges are about academics, but at times it seems that being intelligent is only a prerequisite to elite admissions, but the main focus.”
I think you mean not the main focus. I agree with that. But the reason is pretty clear. US high school coursework, AP exams and standardized tests are quite easy (though include a lot of busy work), relative to exam systems in other countries, and don’t identify the most brilliant students very effectively. And in reality, US colleges don’t just want the most academically able students, because they have social objectives as well (if you want to see pure academic selection, go to Oxbridge, and try having dinner with a bunch of their mathematicians 
).
So elite colleges require you to spend a lot of time (say 20-30 hours per week for many years) on ECs, on the basis that if you can do that and still get excellent results in your academic work then you must be good (and dedicated). And if that 20-30 hours leads to some recognizable achievement in sports (especially ones they favor), arts or academic competitions then all the better for their social objectives.