They’re right that it’ll be a distraction. The question is if that distraction is worthwhile.
If you’re studying a humanities discipline that’ll mean a fairly low entry-level salary, maybe you do want to minimize debt. If you’re studying engineering or computer science, I’d advise against working in college. With your degree in hand, you’ll earn more in a year than in 4 years of part-time work as a student - if you have a good GPA. Working during college may well sabotage your GPA, costing you far more in the long term than you’ll earn from the work. That’s why many work-study positions are sinecures - e.g. sitting in a library and answering a question every 10 minutes or so - that allow a student to spend 90% of his shift studying.