Is it worth it to transfer to a "less prestigious" college?

I am currently a freshman at Vanderbilt University, and I’ve been thinking for a while that Vanderbilt isn’t for me. I haven’t been very happy with the overall attitude of the school, and while there is a great sense of community, it’s rather intense for someone like me. Even more, I am very far from home, and while I know that college is about being independent, I sometimes have anxiety and other mental health issues that make it very difficult to be away from my primary support system without the option to visit. I’ve also been considering an environmental science degree, but Vanderbilt does not have a very large or diverse program for it.

I’ve been doing a lot of research into other schools with good environmental programs that I think I would enjoy on the East Coast (closer to where I live), but many either don’t offer merit scholarships for transfers or likely won’t give me as much need-based aid as Vanderbilt. The result is that I have been considering many schools that are viewed as being “less prestigious” (i.e. public/state schools with much higher acceptance rates), and I am afraid of the impact this may have on my future. I have never viewed these schools as inherently worse, and I believe I would be very happy in a less stressful academic environment, but adults in my life have cautioned me about giving up an education at a school like Vanderbilt where I have received such a good financial aid package. (It is also worth noting that I have friends at many of the schools I am looking at, so I know I would have people that would make it a more enjoyable experience).

Overall, the idea of spending four years here does not make me happy, but I can’t tell if I’m making a good decision by leaving or not. Is what I’m feeling normal and something I should push through, or is it reasonable to at least consider transferring? And, if transferring is a good option, is it a mistake to go to a more average institution where I’ll be paying the same amount as I am now?