For your career, if you want to enter academia, who your advisor is and what lab you work in is key.
In terms of superficial stuff, Madison is a fun college town that is gorgeous in the summer but with brutal winters while SD has nearly perfect weather the whole year round.