OP–you’re right in considering whether you’ll be happy at a school more than the rankings etc IMHO. If you’re not happy, grades tend to slip and there you go. It’s that much harder to continue forward.
As for grad schools, while not dipping toe into the ongoing rankings debate here, my understanding of many grad schools is that the comment that came before the rankings debate is more accurate. That comment said that grades, tests, statement, and your professional accomplishments (inc. research or internships) are important. Many Ph.D.-level programs at least the students are basically entering a mentorship with a professors. Often, though admissions offices handle the application, it’s the professors who screen or help screen whether they want to mentor someone 5-7 years. It’s a relationship. So, if you’re brilliant on your application–shown through the above-listed items–the professor will care much less what school you went to as undergrad. Most profs that I know do not keep a list of USNWR ranking in their brains. They are more concerned about other things–their research and mentoring, for example. They might give extra consideration to the Harvard undergrad application, because that name stands out, but if the application is thin, then there’s a high chance that Mr or Ms Harvard will be set aside. Schools like Smith vs Hamilton–few professors really care about the rankings IMHO at that level of granularity. While I haven’t extensively studied every single masters program, it’s telling that Harvard Law School accepts people from a wide swath of undergraduate programs–from tiny religious schools you’ve probably never heard of to state directional schools to USNWR high-ranked LACs and from Harvard itself. It’s worth googling for Harvard’s current list of undergrad schools they pulled from. In other words, most people know that the school you attend is just one of many indications of brilliance but the best indicator is what you produced on your application, in your work, in the interview and elsewhere.
So go be happy.
Best of luck to you.