"Will colleges think I’m not challenging myself enough, even though my course rigor is near/at the top of my grade? "
Read this again and recognize your logic flaw in the question. Of course they won’t hold it against you. By definition, you’re at the top. Anything else is absurd