I went to Williams and my spouse went to Vassar. (Wife of husband’s cousin went to Tufts, but I do not know it well enough to comment.)
Both my spouse and I loved our college experiences. I think you will have a great academic experience at either one. Williams is well known for art history, but Vassar is more artsy overall. Both have good sciences-- a slight edge to Williams, but Vassar just built a beautiful new science facility.
Both have small classes, smart kids, great professors, intellectual conversations that spill over into dorms and dining halls.
Williams has distribution requirements that are pretty minimal and easy to meet. Vassar has even fewer requirements- foreign language and quantitative.
I think your biggest differentiator will not be academics, but environment. Which “feels” better to you when you visit?
Williams- rural, small one-horse town but with great art museums and a theatre festival. Surrounded by mountains, great natural beauty and outdoor opportunities. Far from anything else. Eclectic buildings, very nice (see new library) , but Vassar has an overall edge in architecture. Freshman entries build a community for new students-- a very special part of the Williams experience. Students then move on to live in other dorms and most students live in on-campus dorms for all four years. Many, many single rooms. 3 dining hall choices. Liberal overall but definitely more traditional than Vassar.
Vassar-- suburban school, within two hours of Manhattan. Some of the most beautiful college buildings every built. An artsy, liberal school, where you will encounter more political protests than at Williams. Great theatre. Students are assigned to a pretty dorm where they will live for their first three years, then many move to on-campus school-owned apartments for senior year. Many single rooms in later years. One central All Campus Dining Center (AC DC).
You can’t go wrong here. Let’s put it this way: neither my spouse nor I think our son, now a high school junior, would be better off at one of the schools than the other. We would be happy for him to attend either college, or another school altogether; it is all a matter of personal preference.