This isn't just a rule of thumb, it's actually a characterization based on physics. Hoffman's Iron Law is a mathematical formula that was later refined by Thiele and Small, whose work now forms the basis of all modern loudspeaker design. Hoffman's Iron Law states that the efficiency of a woofer system is directly proportional to its cabinet volume and the cube of its cutoff frequency (the lowest frequency it can usefully reproduce). The obvious implication is that to reduce the cutoff frequency by a factor of two, e.g. from 40 Hz to 20 Hz, while still retaining the same system efficiency, you need to increase the enclosure volume by 23=8 times. In other words, to reproduce ever lower frequencies at the same output level you need an extremely large box!

However, box size isn't the only variable. You can continue to use a small box by accepting a much lower efficiency. In order to retain the same sound pressure level (SPL, meaured in dB's), though, this requires both a very large amplifier and a driver that can handle a lot of power and move a lot of air (requiring high excursions). Furthermore, it must be able to do so with minimal distortion. This is exacerbated by power compression, a phenomenon where the power heating of the driver's voice coil results in a non-linear relationship (read "distortion") between the electrical power in and the acoustical power out.