Most rural schools are going to have quite a bit of drinking (not much to do).
Colby and Lafayette both have big drinking cultures. Williams does as well. Skidmore has a pretty big drug scene (mainly pot). If you don’t want to, you don’t have to get involved, but it is a pretty big thing on campus. I’m sure it happens at other schools on your list, too, I’m just not as aware of it.
Bowdoin is not particularly rural- at least, not in the way that Colby or Williams are.