Its a matter of how well the driver can convert electrical energy into cone motion. Think of the magnet/voice coil as a motor; the larger than magnet, the smaller the voice coil gap and the better the voice coil topology the more power the motor is able to develop from the energy fed into it by the amp. Then that power has to move the cone. The less resistance the cone has to movement(caused by the combination of its mass, the stiffness of the suspension and a few other factors) the easier the job the motor has moving it. It's like a car- big motors in small cars equal fast; small motors in big cars equal slow. BUT- what it takes to gain efficiency also can lead to a loss of response in the bass end, as both the lower Qts of a strong motor and the higher Fs of a light cone limit bass response.