I failed my first semester of AP calculus, but passed the 2nd and the exam.
I heard colleges require 3 years of math, and sometimes even 4; do I need to retake AP Calculus because of failing first semester with a D?
For UCs, passing second semester calculus validates the entire math requirement except if you are missing geometry (although a D or F in geometry can still be validated by passing a higher level course, and geometry taken in middle school can count toward the requirement).
http://admission.universityofcalifornia.edu/counselors/files/validation_matrix_math_lote.pdf
For other schools, check their policies.
So since I passed calculus, does that count as 4 years of math due to fact I passed all the previous maths? So I am fine not taking math senior year?
As of other college policies, do I just search up “validation matrix”
Yes, for UC you are fine with the math requirement, as long as you have taken a year of geometry previously as well. Validation means that the lower level courses could have D or F grades or be missing (except for geometry) but are assumed completed by passing the higher level course. However, the D grade still affects your GPA. Note that the minimum math requirement for UC and CSU is just geometry and algebra 2, though more competitive applicants will have completed higher levels like precalculus or calculus.
“Validation” is probably a term unique to UC (and CSU). You may need to check each college’s admissions web site and/or ask the college directly.
I took geometry in my 8th grade of middle school, I hope that counts.
Sorry, I didn’t see your middle school part. I tried searching up validation for schools such as USC and other privates, but I can’t find any. 
May you help me on that part?
Instead of looking for specific “validation” policies, I would look more broadly. The UC system has a lot of vocabulary terms that are pretty unique. I would look for pages that detail the required high school curriculum and policies on failed classes. If all else fails, contact the admissions offices, but beware that they are fairly inundated this time of year.
USC does not have a specified high school curriculum, but says at http://www.usc.edu/admission/undergraduate/firstyear/prospective/components.html :
So the main thing to worry about for USC is the D grade and how it affects your GPA and looks to an admissions reader. UC admissions readers and admissions readers at other holistic admission schools will also see the D grade. Note that many private schools are somewhat more flexible on the high school curriculum, often saying something similar to what USC says (or giving a recommended-but-not-required pattern), but without a hard list of required courses to be considered.
If you do not retake the class, your D will be calculated into your UC and Cummulative GPA which could impact your chances at these competitive schools.
I am unable to retake the class this summer due to a trip to a foreign country for the whole summer. Is it possible to retake it my senior year?
I’ve done many extracurricular that I hope could be helpful due to it relating to my interest in my major. This includes a paid part time software engineering job aside the CEO and helping him develop a product, and many more.
I heard many people tell me that UC’s tends to be more “academic” and privates having more liniency towards students who strives to be successful outside of academics. I hope this could help in applications.
You can repeat it senior year, but UCs (and CSUs) will not see the grade in time for it to replace the D grade in your GPA when you apply to college (and the admissions readers at UCs will still see the D you got the first time in any case). Some other schools which include midyear reports in admission decisions may see the passing grade from a first semester senior year retake, but they will still know you got a D the first time but passed the second semester, so the value of the repeat is probably not that high.
“Students should successfully complete at least sixteen yearlong college preparatory courses while in high school:
Four (4) years of English (composition and literature)
Two (2) years of foreign language
Three (3) years of mathematics (algebra, geometry, algebra II, trigonometry, or calculus)
Two (2) years of natural science (e.g., biology, chemistry, physics)
Two (2) years of social sciences (e.g., U.S. history, government, economics, geography)
Three (3) years of additional courses in subjects above or academic electives (such as computer science, art, music, speech)
Students should pursue Advanced Placement or International Baccalaureate classes whenever possible and if offered by high school.”
I found this on the USC website. “3 years of math”, but does really inform about validations. 
Validation is a UC concept. USC is not a UC school.