Most schools have a larger proportion of students from the states immediately surrounding it, even Ivy League schools like Columbia and Penn. It comes partially from familiarity and partially from distance - some students don’t want to travel very far from home, and that’s okay. That doesn’t mean you have to stay in the Midwest after graduation at all - a lot of students probably choose to stay in the Midwest because they are from the Midwest. Also, as for the winter - it’s a valid concern, and it really just depends. I have a friend from South Carolina who escaped the Southeast for Boston because she loves the cold and the winters and the changing of the leaves. But I’m from the Southeast, did grad school and a postdoc in the Northeast, and can’t really deal with the cold - I kind of ruled out most Northeastern cities wrt settling down because I didn’t want to do it. (I live in Seattle now, which still has the dark but not the cold.) Some people like having four seasons and colder weather and some people don’t. It’s hard to tell up front who you are unless you’ve experienced a mix of things.
Every college is still ‘figuring itself out.’ No college has remained exactly the same since its inception, and changing times call for a change in offerings and admissions rates and type of student. Newer colleges may have some advantages to them, too. And Pitzer’s not “poorly rated.” On USNWR it’s #36, which puts it in the top 3% of all four-year colleges in the United States. Also, USNWR values inputs and not outputs; Pitzer was less selective than the other colleges until recently. Honestly, differences between rankings within the top colleges of USNWR is splitting hairs in some senses. Pitzer and Grinnell are both excellent colleges; you’ll get a great education at either, and will have a great pick of post-graduation options. So I wouldn’t worry about the prestige factor or opportunities after graduation - I’d worry more about fit and where you want to be.
An overnight can be a decent way to get a vibe for a college, but one bad event or weird person can kind of ruin it for you. Oberlin is a really quirky school with lots of quirky students, so I think that part isn’t going to change. And no, you shouldn’t ignore your feelings for the academics - you have two other excellent options and there’s no point in going somewhere you won’t enjoy.
If it were me I would choose Pitzer, but that’s because of my own personal values (the weather, proximity to a large great city, the Claremont Consortium increasing the number of college students to have social experiences with, the vibe of the Claremont Colleges, Pitzer’s social values and liberalness).