The tip top colleges look to see if you took the most rigorous schedule available to you, so not going down the honors track would cut out the core of what they look for.
It won’t hurt your pre-med or business aspirations.
Except…I have noticed that people who take calculus for the first time in college don’t do as well as those who have seen it before and THAT could hurt your college GPA which would affect medical school admissions.