It’s certainly not insane. It’s hard for anyone to advise. You know how it feels. You know the situation, including financial considerations.
There’s always the dilemma, is it best to ride it out, often things don’t click right away, that’s perfectly normal, and probably true for more students than not, even if it doesn’t feel like it; OR it really is a poor fit and time to move on.
I’ll just say one of mine is currently a senior. They decided to transfer a month into freshman year. They applied to a similarly ranked national university, was accepted, and started there in the spring. They are super involved–in a lab, working on campus, in a Greek organization, and in an honors society. They love the decision. That first spring semester was a little slow socially, but they just said, okay I’ll just kill it in my classes. And they did. By fall of the next year, everything was clicking. So it can work out fine.
So good luck. Stay positive, that helps as much as anything. Stay engaged, explore both options as you move forward. Make the best choice for yourself and run with that. Just know your feelings are natural and shared by many. Good luck!