Elena, that’s really only a “problem” per se if you continue to perceive it as a problem 
You’ve mentioned valuing “prestige” (by which I hope you actually meant rigor) but in the discipline you’ve cited, you also need resources, opportunities, access to people in a research environment working at the top of their field…
IME, small schools are well suited for liberal arts, in part because they don’t necessarily have the scope, resources or research capability for deeply specialized programs. While there are exceptions, generally, it’s a trade off.
My son attended a small, rigorous magnet school in HS - 85 grads in his class. The school was very focused on Socratic-style learning in an IB curriculum – but as a result, always had fewer course choices than a typical comprehensive sized school. So naturally, he was attracted to smaller schools initially for college. But none had quite the specific, “deeper” area of specialization at a very high level in which he was already fairly accomplished (a specific combo of music and engineering).
For him, Umich was perfect, despite its size, or BECAUSE of it The size, scope and caliber meant he was able to pursue his specialized interest. In his case, many, if not the majority, of classes were in fact quite small and intimate. And where they weren’t, he didn’t seem to have much difficulty reaching out and finding any support he needed, including large ENG intro classes. Michigan works hard to create a “smaller” community in a number of ways…think of it as a collection of communities within a larger network of communities
So that means that while ENG is a community of roughly 6,000, sub-disciplines form smaller communities
Another thing to consider at college level is that by now, you will or should be learning as much from collaboration among peers as from any lone mentor. A larger student body inside your discipline, while possibly more competitive, is also more robust in terms of opportunity for collaboration. Navigating this may give you more than you realize in terms of securing your future.
So while fit is important, so too curricular diversity in specialized disciplines, and the latter is more difficult to make work among a small student body. In your case you may need to prioritize which is more important to you.
I think visiting a few of your selected schools overnight, talking with existing students and sitting in on a range of intro AND upper level courses would benefit you enormously. You might be surprised at your favorites after so-doing. It made the world of difference to my son