If you choose your friends carefully, you can avoid most of the problems associated with drugs, alcohol, and tobacco. Of those three substances, alcohol is the most prevalent but plenty of students do not partake. Look for those students. Personally, I found parties boring in college. A movie with friends was much more enjoyable.
Also, I agree that larger schools will mitigate this problem, for the simple reason that peer pressure will be lower. At a small liberal arts college, peer pressure can be quite high. At larger schools it is easier to choose your friends, and ignore the people you don’t like. There is an old say “you can make a big school small, but you can’t make a small school big.”
By the way, “in the real world” you need to learn to avoid drugs too. The notion that tolerating substance abuse is a part of becoming an adult is upside down.