Just as easy, IMO. PhD programs pretty much don’t look at the ranking of physics undergraduate programs. They care more about potential and the work that you did as an undergrad given the resources that you had.
For instance, here’s a list of all of Caltech’s PhD students:
http://www.pma.caltech.edu/GSR/gradclasslist.html
Many from abroad, but you’re not going to tell me that WashU offers physics students less opportunity than Cal State-Fullerton, Nevada-Reno, and Oregon St., are you?