Because you were not placed in advanced math in middle school, you are on the normal on-level sequence of:
9th: algebra 1
10th: geometry
11th: algebra 2
12th: precalculus
You would not be stopping taking math if you followed this sequence. You would be taking all of the math available to you and expected of you based on your middle school math placement.
In contrast, someone placed in advanced math in middle school (i.e. algebra 1 in 8th grade) would have:
9th: geometry
10th: algebra 2
11th: precalculus
12th: calculus
If this latter student decided to take no math course in 12th grade, the more selective colleges with holistic review may question why s/he stopped taking math even though there is another more advanced math course to take.