In practice there are most definitely affinities between high schools and universities. It is however a factor I would not consider too much when drawing up the list of colleges to apply to. Maybe one would however want to consider this as a tie breaker when deciding where to apply ED.
About those interactions between high schools and colleges: “The Gatekeepers: Inside the Admissions Process of a Premier College” by Steinberg is a good read. Old book but no doubt still very relevant.