Will my SAT II scores hurt me when applying for top 20 schools?

Hello, I am wondering just as the title says. I have a 750 on Math 2, 640 on U.S. History, and 580 on physics. I’m mainly concerned about my physics score, since I took the USH SAT II for fun and performed OK on Math 2.
Here is some backstory to this: I had no idea what the SAT II’s were until the middle of my junior year. I go to a (primarily low income and minority student) high school where absolutely nobody talks about it and generally people don’t take it (since most people are expecting to attend a CSU school). After looking at some college application requirements, I realized I’d have to take a science SAT II (I’m interested in majoring in bio at the major schools). I was taking AP Physics 1 and thought that would be enough prep for the SAT II physics exam. I was clearly wrong because the SAT II covered topics over AP Physics 1 AND 2. Now, I’m here wondering if my scores will hurt me significantly or if college admissions will acknowledge this mishap and let it slide.
As for more information on the side, I am a decently accomplished student. I have a 3.9 UW + 4.2 W gpa and have scored over 1500 on the SAT recently (I hope to get a 1600 by the end of this month, though). I participated in extracurricular activities (math competitions and social activism) at the state level, not to mention a bit of scientific research and much volunteering. I did score a 3 on the AP Physics 1 exam, which isn’t too shabby considering the curve for AP Physics 1 is nuts (there’s a super high fail rate). I also scored 4’s in AP U.S. History and AP Calc AB. I guess what I am trying to show is that these SAT II scores might stick out like a sore thumb in my applications, but at least super poor performance isn’t a total pattern.