You'll notice I am careful to describe phase interaction problems as summing problems. This is because the thing I am most concerned about is the notches formed by incoherent summing. That is the thing you most want to avoid. The thing is, to me it is important not only to avoid cancellation notches straight on the forward axis, but also at angles within the coverage angle of the loudspeaker. This raises the bar, because there are more places in 3D space where interactions between drivers are important than just one on-axis point.To illustrate the issues a little better, here's a chart that shows electrical phase of various crossover slopes:
1st order - 45 degrees at crossover, 90 apart for adjacent drivers
2nd order - 90 degrees at crossover, 180 degrees apart
3rd order - 135 degrees at crossover, 270 degrees apart
4th order - 180 degrees at crossover, 360 degrees apart
5th order - 225 degrees at crossover, 450 degrees apart
6th order - 270 degrees at crossover, 540 degrees apart
7th order - 315 degrees at crossover, 630 degrees apart
8th order - 360 degrees at crossover, 720 degrees apart
This gives you an idea of the amount of phase movement each crossover slope introduces. Combine this with the path length disfferences between each driver and the listener, and you can see how complex summing is in the acoustic realm. At locations where the path length difference plus the phase difference from the crossover combines to make the sound sources any multiple of 180 degrees apart, there is destructive interference which causes a notch. The goal of the designer is to place any notches off-axis, away from the intended listening position. If the speaker is designed for uniform coverage through some angle, then notches must be off-axis far enough to be outside the edge of the coverage angle.
By the way, I agree with you completely on the matter of absolute phase verse moving phase. I made the exact same observation you did with synthesizers, as I'm sure everyone has that has ever been exposed to VCO or any signal generator that can produce a sawtooth wave. I played piano as a boy and got a MOOG synthesizer when I was a teenager. I was initially surprised that a sawtooth sounds the same regardless of its polarity. This made me realize that our ears aren't particularly sensitive to phase by itself - we hear phase change only because of the amplitude response anomalies it creates. I've heard that snakes and reptiles can't see a non-moving object very well, and in a way, we're like that with phase. We cannot hear phase relationships very well unless they're moving.