Well, we are not from a highly competitive area, so my comments may not matter. Neither of my older sons did any summer programs during high school. During the summers, my oldest son worked at Mathnasium, played baseball, and played chamber music. Oh, and the summer before his senior year, he did research at a local State University. He had been invited by the professor because he had audited a class during the school year and he did well enough to be invited to do research, apparently.
My middle son worked as a cellist, did chamber music, was on a bowling team, and took a couple of art of problem-solving math classes to catch up since he was a little behind due to his math disability.
My oldest son just graduated from MIT, and my middle son will be attending the University of Pennsylvania, so it did not seem to hurt them to have skipped summer programs. YMMV.