OP, I would recommend you stay at your current school. At least it has a physics program.
I go to a local school and I initially applied here for bio, and the school itself is made up of something like 70% bio majors. Now that I want to do physics I’m basically forced to transfer out, since they don’t offer the major here. I was doing some research on schools and your school came up on some lists, it’s definitely not unheard of, so even prestige-wise you should do fine. Don’t worry about it.
Me though… Well since I have to transfer out, I figured I might as well apply to top physics schools. After doing a lot of research though, I realize that the undergraduate physics curriculum is basically the same everywhere (especially since I’m a self-learner) and what really differentiates schools is the opportunities they offer and the people you meet, but more importantly how you as a student take advantage of such things.
Since I’m in dire need of courses and opportunities that my current school doesn’t offer at all, even though I have some nice stats and essays, I really feel like I’d be happy just taking what I can get. I’m really pushing the envelope where I am currently.
That being said, I’m still personally applying to elite schools because it’s another shot at the admissions game, which I heavily neglected during HS, so I’m finding some excitement in it now. I would say if you don’t really feel the need or desire to transfer though, don’t.
Also, unlike you OP, I haven’t had much luck making friends where I am currently. 
So for me I have no real reservations about getting out there and being surrounded by new people.
Btw, I’m also interested in going to graduate school for physics at a top university. Like PurpleTitan mentioned, it’s possible to get in from a “low-ranked” university, you just gotta do your best and play your cards right given what you have available. Best of luck to you.