The dynamic range and low distortion characteristic of high-efficiency designs are what do it for me. That and the fact that the better drivers tend to be built for prosound, so high quality and high efficiency tend to go hand in hand.As an example, the first drivers I ever saw that used shorting rings were prosound units. That made them waaaay better where distortion was concerned because for one thing they were very efficient, so required low drive levels for a given SPL and for another thing, they resisted flux modulation distortion, so even-orders were like 20dB less.
Now days, I think there are some high quality low-efficiency drivers. High excursion woofers have relatively low efficiency because the geometry of the coil and magnet makes high excursion and high flux density mutually exclusive. You can increase magnetic strength, but if the flux is spread out over more space to make high excursion possible, that reduces magnetic density. This requires more power in the coils to move the diaphragm greater distances. So even if the build quality is there, the efficiency is not.
I suppose some of those high-power, high-excursion designs can be made to provide high output, and maybe low distortion. With greater power, they can be expected to provide high SPL, which then means it is possible to get them to produce high dynamic range. There is the compression issue to deal with, but I am not closed minded that it couldn't be done. So I suppose there are possible solutions using low-efficiency designs that provide high dynamic range and low distortion.
But most low-efficiency designs aren't particularly impressive. The newer high-excursion woofers of the last ten years or so are a possible exception, but most home hifi speakers in the 85dB-90dB range are just average devices with 100dB maximum power limits. That's where high efficiency speakers start, with a watt or so, just loafing. I have personally never heard a low efficiency design that was of exceptionally good quality. So I guess that's where my observation ends.