If your goal is a PhD program, they don’t care about extracurricular activities; they care only about your academic and research background. PhD programs are not as picky about GPA as medical and law schools – a 3.3+ overall GPA and 3.5+ GPA in your major is perfectly sufficient for good programs.
You’re likely to do well where you’re happiest. If you think that’s Smith, I’d go and not look back. It’s a great school, and I’m sure you could do well there.