The main point is that "speed" is a rather inaccurate term to use, since all speakers will move their diaphrams the same speed at a given frequency and a given excursion. But just like mass is only one property, so too is inductance. Each of these is a reactive property, and that's what is really importnat here. Some of the reactive components in a loudspeaker include the inductance of the motor, the mass of the diaphragm, and the stiffness of the suspension.In a sealed cabinet, the list grows to include the added stiffness of the trapped air; In a bass-reflex cabinet, you must include the Helmholtz resonator. And in a horn you must include several resonant peaks starting at the flare frequency. None of these reactive properties are bad in themselves, but they do form a filter function. So each of these things is significant to the performance of the loudspeaker.
The speaker's motor exhibits inductance, its diaphragm has mass and its suspension has compliance. Most speakers use electrical damping to act as a sort of "motor braking" to control the cone, but its mechanical features are relevant too. Particularly with lower power amps like SET amps - In this case, the amplifier cannot be expected to electrically damp the cone as well as a high-power amp can. The output circuit doesn't have low enough impedance, and so it isn't particularly good as a current sink for back-EMF. So there is reason to examine the physical features of a woofer, especially if it is used with amplifiers that don't source and sink a lot of current.