I agree with other answers that there are likely to be many different universities where you can do well as an undergraduate student and then go on to a highly ranked graduate program in physics. Which university you get your bachelor’s from is likely to be less important than how well you do, what sort of research you get involved in, and the references you get from your professors and from the people that you do research with.
Also, look at the physics program at universities that you are considering rather than the overall ranking of the university.
Of course top schools such as Stanford, Caltech, MIT, and Harvard are very good for physics. However, so are many public universities such as UCB, UIUC, UC Boulder (perhaps the “other UCB”), UMD, Stony Brook, at least two different UW’s, and many others. Your in-state public universities are probably a good place to look. Quite a few LACs would also be very good.
Top graduate programs will have students from a very wide range of undergraduate programs.
To be more useful, we would need to know more about you, such as which state you are from, what your budget is, and what your high school stats look like.
Another important thing to consider is where you would be comfortable. Fit is much more difficult to determine that “US News ranking of the physics program”, but fit is more important.
I would also not count on continuing with physics until at least after you complete special relativity and quantum physics. People who are good at physics are usually also good at math and computer science and probably several other things. Fortunately universities that are good at physics are usually also good at math and computer science and quite a few other things. This is likely to allow you to switch if you want to when you are in university.
I also agree with the comment above that MIT is hard (as would be Caltech, Harvey Mudd, …). You have to be sure that you want to work very hard for four years without a break if you are hoping to attend MIT or Caltech or a similar school.