Williams, like other elite institutions, is happy to indulge the assumption that black or Latino = low SES.
To his credit, Anthony Jack in his The Privileged Poor, points out his own experience at Amherst and research at an unidentified elite (presumably Harvard, where he is an assistant professor) confirming that more than half of the black and Latino kids are from privileged upper income backgrounds. Fully half of the remaining black kids and about a third of the remaining Latinos were also from privileged backgrounds in the sense that they attended prestigious prep schools on full scholarships, usually as part of programs like Prep for Prep.
There has been no serious attempt by the elite schools to find talent in the vast swath of non-URM poor, and only modest attempts to identify talent in the poor but unprivileged URM groups (which Jack terms the “Doubly Disadvantaged”).
It’s mostly optics.