My suggestion: If you don’t know what you want to major in, don’t twist yourself in knots trying to evaluate relative department strength between the 2 schools.
You already mentioned the fit issue…so pick the environment you want to be in more: More isolated and near the great outdoors, or suburban with a 20 minute train ride to a big city?
As for rankings, if you need a reason to focus on Haverford look at Forbes (#25 vs. Colby @ 61) or Washington Monthly (#6 LAC vs. #25).