The issue is that as source impedance is increased, the resulting circuit acts more like one having constant current. This makes the woofer excursion vary purely because of the electronics, which sets the amount of motor drive and damping. What you have essentially is a voltage divider, where both impedances are fairly significant, and one of them varies considerably at different frequencies.

If the amplifier has very low source impedance, then it acts like a constant voltage source. It acts like there is no voltage divider, because the source is essentially "shorted." When full power is applied, the voltage across the motor goes to the rails. And when no voltage is applied, it acts like the motor is shorted. So the load impedance doesn't interact with the source impedance very much as a voltage divider.

These effects are quite noticeable when you take a motor with high Zmax and compare its behaviour in a circuit having strong drive/damping ability to another having high source impedance. You can drive a speaker having low Zmax with either of two amps having different damping factors and you might not hear much difference if the amps are comparible in other respects. But put a speaker with high Zmax on the amps, and you'll definitely notice it.

Try this experiment: Take a high-power solid-state amp and drive a speaker with low Zmax. Now take the same amp and put a series resistor inline, of say 10 ohms. The low Zmax speaker won't peak too much, and you won't notice the difference in frequency response. You might hear the reduction in output, but the response curve will be fairly similar so the speaker will sound similar. But do the same thing with a high Zmax speaker. You'll hear the difference now. There are some SET amps that have high enough source impedance that they act this way. So they probably are best paired with speakers having low Zmax.