The key thing for graduate study is going to be how well you are playing coming out of undergraduate, period, if you go into the audition and play well, teacher wants you, you get in, where you went to school itself isn’t going to mean much other than how well you play (there is some truth, obviously, that if you apply for grad school at the school you went to UG, it will likely be easier to get in, especially if you stay with the same teacher, but in your case you would be going to a different school).
Another key factor is, how far are you behind the kids who were good enough to get into the ‘top tier schools’ and what instrument you are. On instruments like violin and piano and flute and cello, the bar to get into the top programs is extremely high, as is the competition, and starting late like you did can be more of an issue, because violin and piano and cello students start much younger than let’s say a brass player, and as a result it may be a lot harder to ‘catch up’ on strings or piano in your UG to those who have been playing a lot longer and at a high level, whereas with brass or woodwinds it may not be as big a gap and easier to make it up.
Then, ask yourself what drives you personally forward and what environment best suits that? Do you need to be surrounded by a lot of top level players to help push you? Then pick a program that seems to have a solid studio on your instrument and good teachers. If you are someone who might be distracted by being in a city area, might be better to go to a school with less distractions, if you are someone who feels you would moult in a rural area, maybe a program in a city area would be better.
The biggest factor is going to be the teacher, and I recommend highly, if at all possible, to do sample lessons with potential teachers and see which seems to be the best fit. If you can’t do that, then you may want to talk to your primary teacher now, or try and do research on the department to see who you might study with. That is probably going to be the single biggest factor,with the ultimate question being, what drives you (as a person) forward the best, and since teaching music is not like teaching calculus, that fit is critical.