I don’t have a dog in this fight, so this is unbiased.
Surely your son would get a strong education at either school. But in terms of prestige, Colgate holds the edge.
Obviously, that should only come into play if all else that’s important is equal (namely academics, fit, and finances). If he prefers the academic opportunities or cultural vibe at F&M, and the two offer similar bottom lines, skrew prestige…