Austin and Boston are not “college towns” but are large cities that happen to have colleges located there. “College town” has never in my experience been equated to “great place to go to college” but has always meant “big town/small city that is culturally dominated by a college or university located there.” If we are going to include Boston and Austin, why in the world is New York not listed? Or Los Angeles? I am not even sure that Madison should really qualify given that it is the state capital first and home of UW second, though enough people do think of Madison for UW first that it is the exception that proves the rule.
Otherwise I agree with a few other posters–Athens, GA and Bloomington, IN immediately come to mind, and people from the northeast likely have a few nominees (but I am not really familiar with ne college towns).