Leaving Honors Program and Graduate/Medical/Transfer Applications

I’m currently in a rather difficult but not particularly well known Honors Program at a fairly good private university. If I decide the workload becomes too burdensome or the grades a little low, how much would it hurt me to leave the Honors Program? It will show all honors classes first semester and then none potentially for the rest of the four years. Will med schools or graduate schools view that negatively? Also, what happens if I consider transferring next year? Will leaving it hurt me in the long run?