You can’t go wrong. Both excellent schools, and if your child finds a telling detail that makes one school “fit” - then great!!! Go with it, no regrets.
That said - I have a son at Vassar, and am thus partial to that school, and think that sometimes folks underestimate how intellectually focused the school is, as opposed to its well-earned artsy/creative vibe.
Note that Vassar is particularly recognized for Psychology as one of the nation’s leading feeder schools (per capita) for Psychology PhDs.
Swarthmore publishes lists of the per-capita PhD production rate of colleges and universities derived from published NSF statistics (in which Swarthmore always does quite well 
).
Doctorates Awarded :: Institutional Research :: Swarthmore College
In looking through this document, you’ll see that in the category Psychology and social science, Vassar ranks 13th in the US (Harvard ranks 14th) among the percentage of its graduates who go on for a PhD in these fields, with Hamilton a respectable 75.
In Humanities and the Arts, Vassar ranked 12th in the nation for per capita graduates going on for a PhD. Hamilton ranked 63rd (still phenomenal, considering the more than 3500 universities in the US).
Looked at another way, not per capita, but just comparing raw numbers among liberal arts colleges, a higher ed blogger has used the NSF data to make a searchable table where you can adjust type of degree, date range, type of college, and so on, to investigate undergraduate institutions of people who get PhDs in the US. Baccalaureate origins of doctoral recipients (highereddatastories.com)
Playing around with this tool, you can see that from 2008-2018 (the most recent years for this tool), Vassar had 109 graduates who went on to graduate schools and received PhD’s in Psychology (the most of any liberal arts college, or baccalaureate college, in the US). In that same period, Hamilton (a smaller school) had 39 graduates receive PhD’s in psychology.
In that same period, Vassar had 57 students who received PhD’s in literature (or “letters” as the search tool lists it) second most behind Oberlin among US liberal arts colleges, while Hamilton had 22.
Your child may not be interested in potentially going to graduate school - but what these numbers indicate is the likelihood that your child would be in classes with students that were deeply engaged in the field, doing research, and with the intent of continuing on after their BA. This shapes the climate of the classroom discussion, faculty advising, and the peer climate.
It does NOT measure intelligence, drive, achievement, quality of faculty, etc. - but I still find the data useful for what they are.