What if I fail my AP Calculus BC exam?

I'm currently a Junior, taking AP calc BC. I barely squeaked by this semester with a 93 average. (I got an 87 on my mid-term exam). I completely skipped pre-calc because my math teacher and guidance counselor advised me to go right to calculus. It's been a battle to keep up with the rest of the class, but I've managed to do okay, despite some holes in my knowledge. I'm afraid that I won't pass my AP exam because I've recently learned that I'm bad with standardized tests. Despite finding success in various AP and honors classes, I scored a really low SAT score. I'm concerned this pattern will happen with my AP exam. 

If I fail my AP exam, but pass the class (with say, a B), will that ruin my chances of getting into a good pre-med program? I’m planning on applying to Quinnipiac University, the University of Vermont, and the University of Florida (in Gainesville). All of those schools supposedly admit holistically, so I’m praying my 4.0 GPA and extracurricular activities out weigh my 1580 SAT score and predicted bad AP exam score.

My understanding is that your AP score only matters for getting credit for the course. You won’t get credit if you don’t pass, but I don’t think it affects admissions.

So colleges won’t care if there’s a horrible difference between my Exam score and my final class grade?