Here you have the typical LAC vs. U differences: class size differences (advantage Williams), course and major offerings (advantage Vanderbilt), familiarity with and access to profs (Williams), D1 sports and more varied social scene (Vandy), close-knit (Williams) vs. relatively more space/anonymity (Vandy). Those are typical LAC/U differences and they apply here as well.
Both will set you up well for whatever you’d like to do afterward, whether work or further studies.
So to me it’s all about what you value and going to the place that offers (more of) it. And if you can’t decide based on academics and academic vibe, let the differences in sports/social scene and setting help you make the choice.