I am sorry, but none of that should matter. If you have physical disabilities for example, you have a different pool to compete in for the Olympics. To compete in the main Olympics, you have to make the cut. You have to be really good. Where you came from, How you got there, what your family story was, none of that matters. Only how you perform matters. The top 20 schools in the US (just to keep this argument simple) have around 30,000 slots for freshman. Who should get these slots is what we are discussing here. I am saying, they should go to the most academically gifted kids we can find. They can be gifted in Humanities, Arts, Science, Language whatever (based on their interests), but must be extraordinarily gifted and this gift must be measured on a national scale based on testing criteria that is meaningful, tough and challenging.
If we want to give lower SES kids a chance (an admirable social goal), then lets invest public money in giving them a good education, giving them special training, I am ok with that, but the rules of the race have to be the same for everybody, once you get to the start line.