IMO, you want to err on the side of taking more rigorous classes, not less, as long as you can do well in them. You get one shot at college applications and class rigor is just one of many factors in admissions. Every year, it’s get more and more competitive for the most popular colleges.