Transferring from Colby

Hi. I’m currently an undergrad at a very selective and prestigious (~25% acceptance rate) liberal arts college. I got a 2000 on the SAT, and did kinda well (top 25% of my class) in a very competitive public high school, but a huge reason I got into my current college is because I’m a double legacy. The main reason I applied ED is because I’m a double legacy. But I’ve decided that I don’t like it here, and I want to leave. So my question is… will being at a very prestigious university help me get into other prestigious schools like Tufts, Rice, Pomona, etc. even though I’m getting mostly B+/A- at my current school and given my high school performance? Based solely off of my high school performance, it’s unlikely I would have gotten into any of these. Some of my extracurriculars:
Club tennis (HS/College), volunteering (HS/College), Ultimate frisbee (College), treasurer of Make a Wish at my current College, I had a job for 2.5 years at home.
Some other facts: member of LGBT, 770 on Math section of SAT, hoping to be a Math/Stats major. 5 on AP Stats/Ap Psych.

My dad said I could only transfer if I go somewhere as good or better than my current school. I won’t go too into why I want to leave (greek life is a major aspect), but my question is: is leaving realistic? Is it worth trying? Thank you!