That depends on the situation. There are certainly still physical phenomena for which we have no in-depth understanding. There are many, many concepts that we do understand fairly completely. however, that understanding was reached, generally, using a whole lot of experimenting and math.
Unfortunately, you can’t. No one can. Ultimately, having some qualitative physical insight into a situation can often help a scientist or engineer come up with the proper model to describe it. On the other hand, there are many, many situation where a great deal of insight into a phenomenon can be obtained by examining the governing equations. There will be times where you may, as a scientist or engineer, come across a new application on the job. Sometimes you can just find a governing equation already derived and other times you can derive one (at least approximately) from first principles. Either way, if you are able to look at the equation and recognize certain traits about it, you can get a lot of insight into how the actual physical system behaves. For example, if you see that the governing equation is essentially a diffusion equation, you can expect certain things to be true about it and apply that knowledge accordingly.
In short, ideally there is a back and forth relationship between qualitative physical insight and the insight gained by mathematics. They can reinforce each other quite a bit, and very good engineers understand that. Of course, you can be an engineer and bring home a paycheck every two weeks without being able to do that, but it is certainly beneficial to be able to do that sort of thing.
I can count the number of times on one hand that I have had to do an optimization problem on some kind of physical dimensions outside of the classroom. Experimental science does feature a lot of qualitative analysis, but it features far more quantitative analysis. It’s still very math-heavy on account of the central goal being to help develop a mathematical model that describes a phenomenon. There is a type of research, generally called testing and evaluation, in which designs are tested and evaluated against a set of performance benchmarks and tweaks are made in order to improve it, but those designs weren’t initially created by guesswork. You may find some that were, but by and large, it was done using existing mathematical approximations (models) of the situation in order to come up with the design in the first place, then going back to those in order to tweak the design.