I’m going to caveat this by saying that I’m not in that field, and that astronomy may be one of those few fields (like animal science, which I learned here) where undergrad training is more niche and going to one of the few highly specialized programs is important. But I will say that generally speaking, you don’t have go to a university with a program that’s highly ranked or “the best” in your field.
First of all, undergrads don’t really go to a “program.” You’re not headed off to grad school, where the vast majority of your primary interactions will be with your department. Probably about 1/3 of your classes will be taken in your major and the rest will be taken across the college or university; you’ll have friends and classmates in all departments. You have to think about the holistic experience, but just one department.
Secondly, most rankings of fields like astrophysics are done at the graduate level, and really apply to the doctoral programs. While a lot of that will trickle down, some of the things that doctoral programs are ranked on don’t apply to undergraduates. For example, some of those elite professors who train doctoral students don’t even teach undergraduate courses or even really interact with undergrad RAs (although their presence in the department and ability to do research with their grad students and postdocs can have a more indirect effect on your experience). Sometimes professors who are preoccupied with publishing and grant-writing don’t want to spend a lot of time training and mentoring undergrads. (I have known many professors who explicitly avoid teaching undergrads, or any students for that matter, because teaching takes a lot of time and impedes one’s ability to write grants and papers.)
This also means that only universities with doctoral programs in the field are included in the rankings, even if there are colleges with fantastic BA and/or MA programs in the area. For example, I know that Agnes Scott College (a small women’s college in Decatur, GA) has a great program in astrophysics, with a state-of-the-art science center; their own telescope, observatory, and planetarium; and membership in a consortium that allows students telescope and observatory access across the globe. And since their physics/astrophysics program is quite small and undergrad-only, undergrads probably have more access to it than they would have access to the facilities at Berkeley or UCLA. But Agnes Scott will never appear on any of these lists because they don’t have a doctoral program.
That said, I know that astronomy is one of those fields that requires specialized and very expensive equipment like fancy telescopes and observatories and stuff. I don’t know how much undergrads actually get to use any of that stuff, but having the opportunity to at least observe it in practice could give you a leg up in graduate admissions IF graduate school and a research career is something you’re interested in. And the top-rated programs are the ones most likely to have fancy stuff like that.
A quick glance at the NRC rankings for astronomy and astrophysics shows that UC-Santa Cruz, in addition to UCLA and UC-Berkeley, also has a great astronomy/astrophysics program at the doctoral level.