Agree with @ucbalumnus that it’s early (interest and grade wise) - but you might consider Washington & Lee and for less competitive Sewanee (which has four semesters of Japanese).
Colorado College has a minor in Japanese.
U of Arizona for a big school as well as CU Boulder. You didn’t ask big but some big schools have strong physics/astronomy and combining with Japanese will be more difficult.
Good luck. But I think you are - years ahead of even needing to think about this - like two. For real.