You are overestimating the difference in size between advanced undergraduate classes at UNC & the others. There won’t be a significant difference in most cases.
I used to live near Worcester, & Clark’s rough neighborhood has long been its main drawback. I would pick UNC if it’s affordable. Holy Cross would be fine (much better neighborhoid than Clark), but UNC is a special place, and I would jump at the chance to go there.