So, if I wanted to major in physics, which one should I go to? (if we consider money not to be a problem and we talk purely academical/research opportunities/opportunities after college).
You will have good opportunities after both. You may wish to decide by the environment and the experience you want to have for four years of your life. Small college vs. large. Warm weather all year vs. four seasons. Big sports vs. division 3 sports and no football team.
If you can visit both this week, great. If not, take out the Fiske Guide to Colleges or the Princeton Review or the Ultimate Guide, and read the descriptions. Decide which environment sounds more appealing to you.
Congratulations on having two nice options!
Both S1 and S2 majored in physics at Vassar and went to a top engineering grad school. The Vassar math and physics background served them well in grad school. Two totally different schools!