If you would “ideally” go to Swarthmore in a general sense, then you should not be deterred by unreliable impressions regarding the breadth of its CS offerings. However, it might be fair to say that the popularity of CS at Swarthmore could lead to restricted enrollment in some classes, an aspect that you may want to research further.
In Wesleyan’s favor, it would likely be stronger with respect to your interest in music.