You are mistaken, as others have pointed out, if you think that athletes and musicians are chosen based on pure ability.
In the United States, elite college admissions is like casting a play. Yes, academic achievement is important. But beyond that, the admissions committee is forming a class. That’s the system.
By the way, the system you propose is exactly the way it works for admission to the three elite NYC high schools – Stuyvesant, Bronx HS of Science, and Brooklyn Tech. Admission is based on a single test. Current demographics at Stuyvesant are 73% Asian, 19% White, 3% Hispanic, 1% Black. Overall, 70% of NYC public school students are Black or Hispanic.