You are obviously a really smart person. I love that you have done so much research and you know so much about each of these 4 colleges that you are considering (med school acceptance rates etc). I love that you reassessed what you really want in a college/university after not getting into Brown ED1. You have a really good head on your shoulders and I think you should be confident in yourself to make the right decision for you regarding ED2.
No college is perfect, and from your personal situation, each of these 4 colleges have some great things about them, and each of them have a couple concerns.
Honestly if I were you, I would not apply ED2 to any of them. You are an extremely strong applicant, and I think you are very likely to get into Haverford applying ED2 or RD. By applying ED2, you are just boxing yourself in and eliminating your other options. By not applying ED2, you have the chance to be accepted to two, three or all of them. Applying RD would also give you more time to think about what to do. You may hear back from the other coaches in the next couple months (especially if a student athlete they expected to apply to their college ED2 ends up not applying ED2) and it may become obvious to you over the next 4 months that one of these 4 colleges is your definite top choice, or that one should be eliminated.
If you were not such a strong candidate, I would say ED2 to Haverford because it sounds like you would be happy there overall. But I think you can afford to apply RD to Haverford and see what happens with the other 3 in the RD round.
I think Pomona is a great option for you, and I don’t think you should be too worried about not liking California. The truth is that Pomona may feel more like home than the culture of a place like Swarthmore, which is a very unique place and a little more “love it or hate it” compared to Pomona where I think a greater variety of people can be very happy. The neuroscience opportunities at Pomona would be better than Haverford, and the geographically unified consortium of Pomona all on one campus is a major asset compared to Swarthmore and Haverford. I also don’t think that you should be concerned at all that Pomona’s physical closeness to the other Claremont colleges will feel like one (too big) university.
Unlike ED2 to the other colleges you are interested in, ED2 to Pomona usually offers very little boost to the acceptance rate compared to RD, so applying RD doesn’t really hurt you very much at all with Pomona. However, this opinion is based on acceptance rates for the general population, and it may not necessarily reflect the past experience of applicants from your particular high school - and based on your comments, it is possible that applying ED2 to Pomona would make a difference coming from your particular school, I don’t know.
So my advice is buy yourself some more time to think and ponder each of these 4 schools. You said you felt like you didn’t really understand what you actually wanted until you received the Brown ED1 news, so don’t rush yourself into making a hasty decision now. Good luck!