When two tones are generated, their phase relationship with one another matters because of their interaction with one another. If they are out of phase, they will cancel each other out and the sound will drop off. If phase moves in and out of destructive and constructive interference, then there can be rhythmic throbbing of cancellation and reinforcement. But if the movement in phase is such that no destructive interference results, then phase is imperceptible. The bottom line is that if you can hear something as a result of phase changes, then what is heard is cancellations that cause frequency anomalies - not the phase.