Pass/no pass hurting graduate school admissions?

<p>I've been told that taking too many classes for pass/no pass will hurt your chances at graduate school. So I am a chemical engineer, but i decided to take an advanced mathematics course that isn't a requirement for anything, but i took it just for the self benefit of learning. Now i regret it, as it's extremely difficult, and for the fear of hurting my GPA, i want to switch this class to pass/no pass. Would it look bad towards graduate school admissions that its p/np since it's not a random GE, but rather a math course?</p>