I recommend you consider your academic interests, then base your college choices on those interests.
For physics, you could find excellent choices among schools without engineering programs. These schools often offer 3-2 or 2-1-1-1 programs with universities that offer engineering. However, these associated programs will not necessarily be supported through financial aid.
For engineering specifically, you should definitely look into schools that offer direct programs. Swarthmore, Union and Trinity should be among those you consider.
In general, you should extend your research as far a possible before committing to an ED II application.