Change in Guidance Counselor

Hello! I am in my senior year, and I was wondering if I should request for a new guidance counselor?
-Freshman Year: My counselor would not let me take physics I honors. I had a space for a class and met the prerequisites. Although it is rare for criminal justice students to take it (I go to a vocational school and mostly engineering/CS students take it, but a few business/ graphic arts/ medical students there). I had to email my principal/ talk to my science teacher/ email the science director/ eventually get the class transfer from another counselor. (My mom eventually talked to him too, and then he started to respond to my emails much faster, which would usually take 2 weeks). I admit I was kind of being a pest, but I was going to be put into a class I did not want, but I really wanted physics.
-Sophomore Year: I was put into a regular class and was not allowed to take an advanced science class because he said it was full. (I skipped pre-calc because I took it as a summer course, which made my scheduling a bit complicated.) When I had to take the pre-calc final exam at my school in the summer, I was temp. assigned to a summer counselor, who told me my schedule was a mess, and he put me into the science course (I am not sure how this happened).
-Senior Year: He is in charge of all criminal justice students, which is about 300-400 students. I know this is a lot, but I know many students who were forced to take a regular class over an honors class, were not having their questions answered, and so on. He is just really hard to get a response from. This year they added a new counselor for our shop, and I was put into her group (luckily :)). She was so awesome! She responded to my emails the day of, and sent my transcripts when I needed it. But since I am applying for Questbridge, a supervisor switched me back because it would be best if my former counselor wrote my recommendation letters. Although I did have him for 3 years, I do not really know if it would really make a difference in my letter considering the circumstances. Also, I am having a hard time reaching him (he has varying office hours and takes a while to respond to my emails), especially when I need him for application requirements.

I do not hate him because he is an alright person, but I am worried that the college process will be more stressful and perhaps be hindered by this. The other counselor is better in terms of convenience and helpfulness. Would this be rude? I do not want to be mean or anything, but I am a bit overwhelmed by this, you know? Sorry for the long passage.