In general, the fan sports culture is not prominent at any of these schools. Vassar and Swarthmore don’t even have football teams.
Harvard and Yale are both DI schools, and occasionally have “big” sports events. The annual Harvard-Yale football game, for example, is a big deal. It would also be a big deal if either school qualified for the NCAA men’s basketball tournament, as Yale did in 2016.
But even without a sports fan culture, you could still find a high percentage of athletes at these schools, especially the smaller ones. At Swarthmore, for example, 27% of students are on a varsity sports team. They are probably smart jocks though, or they wouldn’t be at Swarthmore.