When I was in my PhD program, I didn’t see any difference between those of us who went to elite schools and those of us who went to non-elite schools - and we ranged the gamut.
It’s also possible that the culture of talking about your struggle is different - when I was working with undergrads at an elite school, they were reluctant to admit to struggling with anything, because the culture there was to fake it until you make it and not to admit weakness. The culture at my (less elite, but still good) undergrad college was much more collaborative and people talked more freely about their challenges, so it’s also possible that the students from “relatively unknown schools” (which is not an indication of prestige or rigor) are more likely to be vocal about struggling with the coursework, especially when they believe they are in a safe space with their own classmates.
I’m not going to say that certain colleges don’t have features that maybe help develop a likelihood or propensity to go to grad school. Sure, students who are already the type to want PhDs or consider PhDs are more likely to be drawn to Swarthmore or Reed, probably; but schools like Swarthmore and Reed also provide experiences - small, discussion based classes; close relationships with professors; an atmosphere that values and promotes intellectual depth; more circumscribed social activities - that are more likely to promote interest in getting a PhD. Also, since Swarthmore and Reed attract more students who want a PhD already, that means that their peers who maybe haven’t figured out what to do yet or are more malleable may be more likely to be influenced to also want PhDs.
Some colleges also probably prepare students for their PhD better than other ones - through rigor in the coursework, through the kinds of research they offer, through the professors that they retain on staff to train and mentor their students. But part of that is also selection effect as well.
I think the takeaway is - what you do is more important than where you go to undergrad. But where you go can have at least an indirect effect on your desire to get a PhD, your competitiveness for PhD programs and your performance in the program once you get there. That effect, though, is not a simple one to explore.