Do we consider our 23 yr old son a family member on FAFSA if we don't claim as exemption on return?

If you rightfully claim him, he could prepare his tax return both ways, file the correct one, and you gift him his increase in tax. (Assuming it is less than or equal to your own tax savings by claiming him as your dependent.