That was the primary reason why my GC stressed on the fact that students should choose a college that offers a wide array of disciplines across the board. Just like you, apparently many students change their interest and pursuit, either forcing the students to transfer to another college or simply switching to another department.
Luckily, you have not matriculated anywhere just yet. Of the colleges you have applied to and got accepted, I would rearrange the interest level based on the strength of STEM (physics) or if it isn’t too late, you may be able to apply to other schools that have a strong physics department.