Looking at your list of pros and cons, Wesleyan seems to be a much better choice.
You like the housing, you like the academic opportunities, you like the social structure. You don’t like one think which is irrelevant (rankings), and one thing which is rumor.
For Vassar, you like how the campus looks, you think that there are more LGBTQ students, and and has a possible minor that you like, You dislike the area around the college, and the gender ratio. Neither are all that important either.
I think that neither set of cons is all that critical, but the pros that Wesleyan has seem to far outweigh those of Vassar.
However, on the up side - no wrong choices. You’ll do great in either.