Swarthmore has a more active CS department, to be sure. However, Wesleyan has a very strong physics department - two students in the past 10 years have won the Apker Prize in undergraduate physics (two recipients nationwide per year), and the best part is that Wes competes in the research institution category because it has a tiny graduate program. Those Wes students beat out the large university students.