I agree that Boston city drivers are in a class by themselves. But @ClassicRockerDad notes, the highways in the NE are really ok.
OP, your son and my daughter have similar taste in colleges! Vassar and Amherst are among her favorites as well (of those she has seen so far). Clearly great minds think alike! 
I like the idea of your flying in to Hartford or Albany. Do NOT fly in to Boston. It’s really not near Amherst and it’s a difficult area to navigate. If for some reason you have to fly in to Boston, try to avoid having to drive through the O’Neill Tunnel. Boston is a great city, but it is more fun if you can take public transportation or in a worst case scenario taxi/Uber.
We did a tour this summer that included Amherst, Smith, Wesleyan, Vassar, Hamilton. I wish we had also seen Skidmore; we will have to check it out on a future trip because I’m suspecting it belongs on my daughter’s list. We drove through rural Pennsylvania on the way home to Maryland and that was EASY driving, much easier than the Merritt or I-95 through Connecticut. I think you could get from Vassar to Haverford without having to negotiate NYC/northern NJ if you planned it well.
I hear you on the disadvantages of touring schools when college is not in session, but there were information sessions and tours in the summer, so you do get to do more than just self-tour.
Albany, as a town offset from the Eastern megalopolis, offers a city-in-the country feel. If you center your trip around this area, your traveling should be enjoyable.
Check EACH school schedule…some won’t be on break that week, and others will still have tours and info sessions anyway.
If you are looking at more urban schools on the east coast, be prepared for traffic…and tolls. Sure, you can drive “around” to avoid the tolls, but it will take you a lot longer to get where you are going.
Does your college wannabe have an IPhone or smartphone? Tell him to get familiar with google maps. And bring a charger for the phone. I’m betting he can use the map function on those phones!
I live here…it’s just not worth it to drive around and around to avoid some tolls.