You could get into Engineering grad school with a degree in math or physics, but there would most likely be some engineering pre-requisite courses you would need to take before you could be admitted. You may or may not be able to take them as part of your undergrad - majors typically have priority getting into courses so you could be shut out of some engineering courses if you are in a different major.
Not to say it can’t be done - S was looking at a similar path - but do be aware it might require some additional coursework after you complete your undergrad.