I think university prestige has to do with a lot of different factors. Among them are age, history, quality of the professors, quality of the student body, exclusivity, resources provided by the university, selectivity, networking, and success of alumni, both financially and professionally. Sports can also be a factor for certain universities.
You can have an old university that doesn’t ooze prestige. For example, The College of William and Mary is the second oldest university in the U.S. It includes among its alumni 3 U.S. presidents and 16 signers of the declaration of independance. Yet, most in the U.S. probably haven’t heard of it, and even less are aware of its history.
On the other end of the spectrum we have a school like Stanford. Stanford’s only around 125 years old. However, Stanford’s alumni have been very successful financially, were involved in the development of Silicon valley, and comprise a large part of the ‘new money’ associated with the western U.S. (In contrast to the ‘old money’ associated with Ivy-league schools and eastern U.S…) So, this has helped it become an extremely prestigious university in a short amount of time. Other factors that have helped (like I illustrated above) include quality of professors, selectivity, networking, financial resources of the university, and so on.
I agree that the average person isn’t familiar with the most prestigious schools in the U.S. But that doesn’t mean that they can’t observe that X is a school where smart people go, that X is hard to get into, and that students who go to X have successful careers. Just because most people don’t have the intricate and detailed knowledge of CC posters doesn’t mean they’re oblivious.
To a lesser degree, I think the media play an important role in how we perceive universities. If professors appear from a small list of universities on TV and are hailed as experts, that improves the image of the universities that they’re affiliated with. And universities with more resources (i.e. money) are able to provide larger marketing budgets to improve their image and help shape their perception. (c.f. UCLA’s optimists campaign.)
As a final point, a central factor prestigious universities share is money. For example, many universities held as prestigious received large donations from billionaires in the past (Stanford, Chicago, Duke, Emory, etc.) and presents (Johns Hopkins, UCLA, etc.) Or, they were able to accumulate a lot of wealth through their long history (c.f. Most of the Ivy league.) And many state universities considered to be prestigious received extensive support from the state. One can see this in the UC System and in schools like UVa, Michigan, Texas, etc.