<p>For graduate school admissions in general, where you go to school is less important than what you do there. In some ways, being a big fish in a small pond can be good - since you will be competing with fewer people for slots in research labs, and will be more likely to compare favorably with your classmates when your recommenders are writing your letters. (That’s not to say that any old school will do, but a public university or small college can be as good a choice as a big private for certain students).</p>
<p>Also, as another note - even if there are very famous physicists at a given college, it is unlikely that a physics undergrad student will work closely with them. I am a grad student who works with a famous methodologist in my field; the undergrads are mostly mentored and directed by postdocs and grad students. These famous researchers also often buy out their teaching time with research funds, so often they are not teaching or they teach one class a year to graduate students. Not saying that having access to these guys is not a good thing, but it may not be as much of a boon as people think.</p>