Some college applications will ask why you switched. A reasonable explanation such as a shorter commute, moving to a new house, being able to participate in Z sport/activity that’s only offered at the new school will not penalize you. "It was too hard " or things along these lines won’t reflect positively on you. So, what explanations could you give?
Ultimately you’ll be compared to the students in the school you’re graduating from. If the other (less competitive)school is a better fit, offers as many courses, and has an environment conducive to doing well, it sounds like it’d benefit you and as a result would lead to a better experience and college outcomes.