If money’s not a problem, which one would be better for me to attend?
Are both equally affordable? What are your career plans? Private colleges often have research opportunities for freshmen as well as upperclassmen. Public schools may offer a broader range of courses, but the number of students per class may be higher.
Have you visited both campuses? If not, you could ask to visit and sit in on a class.
It all depends on what your eventual goals are and what you want to get out of college. UF is a Ph.D. granting university and you will have the opportunity to get involved in funded research with graduate students. UF will also have a full physics curriculum that is preparatory for graduate school. It is likely that Vassar will have a very similar curriculum because physics degrees are pretty similar at all universities, however, if the number of physics majors is too small, they may not be able to offer the same range of courses as UF.
Bottom line is that you can get a good physics education at both places but you need to choose the one that fits you best.