How much does college choice matter for careers (prospective neurosurgeon)?

I’m currently in my junior year at high school. I know for a fact that I want to get into the medical field (currently completing an AS in biology at local community college), and I think I want to get into neurosurgery. As I’m not in medical school yet I obviously can’t be certain but this has always intrigued me, though I suppose this question can be answered for anything in the surgical spectrum.

What I’m wondering about is how much college choice matters in terms of likelihood to start my residency outside of medical school. Does going to an Ivy League immediately place you Astro ahead of others? Even if they were to go to a top university as well (though not Ivy)?

I plan on applying to Stanford, Cornell and Dartmouth as far as ivy leagues go, but given as that is a high reach at best I also plan on applying to UCLA and Berkeley (I am a resident of California)… if I wer to be accepted into say Stanford and Berkeley, would choosing Stanford immediately impact my chances of a job outside of school, or are other factors weighed heavier than the school itself? What about if I were to go to a “mediocre” school? Does that put me at a severe disadvantage?