A horn that's undersized becomes a sort of horn/transmission-line hybrid. That in and of itself isn't bad, and in fact, with the right tuning can be very good. But there's a difference in implementation that is significant. Transmission lines are damped to reduce unwanted standing-wave modes. They are designed to encourage some modes and attenuate others, using careful driver placement and damping. Horns, on the other hand, are usually made without damping and with the diaphragm at or near the apex of the horn. They are so-called wide-bandwidth resonators and so resonant modes aren't damped. The designer intends for the horn to load the diaphragm resistively, not reactively, so it is expected to act as if it weren't a resonator. But the undersized horn does act reactively at low frequencies. That's why I've characterized it as a horn/transmission-line hybrid.

A horn decreases diaphragm motion in its passband, so distortion is reduced. It increases diaphragm load and that improves efficiency. Between increased efficiency and increased directivity, a horn amplifies sound volume in its passband. All that is good news. But when a basshorn is undersized, its loading decreases as frequency drops. Worse than that, it fluxuates loading instead of being uniform. This tends to create ripples in response.

Another thing that can hurt performance of an undersized basshorn is using it below the horn's passband, in a direct radiator mode. This is tempting to do in order to increase bass extension. It may be a worthwhile compromise in some cases, but it should be noted that this mode increases distortion more than a direct-radiating front-loaded design. The reason is pretty simple - The horn amplifies harmonics, so it amplifies distortion. Fundamentals in direct-radiator mode are not amplified by the horn, but the harmonic distortion is. This can result in harmonic distortion that is actually louder than than the fundamental.

A well-executed basshorn provides high-output and low-distortion. If made large enough, it can provide deep response. Sometimes smaller horns can be used in groups to achive the necessary size. But when a basshorn is used too low in frequency, it can actually sound worse than a direct radiator. It is good to look at the matter realistically when deciding what kind of bass system is best for any particular installation.