I would be hesitant to call places like Boston or Madison, WI. college towns. Seems to me like a college town is one where the university or college is the main employer and gives the town its identity. I grew up in Columbus, Ohio, and always thought it was silly when people called it a college town because Ohio State was there.