English Scores and applying to tech schools

I am an IB student. My IB score is 40/42, in which I have a 7 in Physics HL, Math HL, Chemistry HL, Psychology SL and Spanish. However I am on the boundary of 4 and 5 in English SL.

I have a very strong math and science background of achievements, grades and extracurricular. I am also applying to tech schools like MIT, Caltech e.t.c. Will the bad English grade affect my application? If I am applying to Ivy League universities, will it affect my application too?

I have upto the mark SAT, ACT and SAT Subject Test Scores.