1. The electro-mechanico-acoustic response of the driver(s)
2. The acoustic filter function of the cabinet/horn/chamber(s)
3. The directivity of each component
4. The environment

The raw driver has its own characteristic response. This is modified by the cabinet and/or horn, if used. But the driver sort of sets the baseline.

If a horn is used, it modifies the directivity, usually making it more uniform but not always. Constant directivity horns tend to make the directivity more constant as the name implies. They usually have straight side walls that set the radiation angle. But they cannot make the pattern consistent through the entire audio band. They lose pattern control at low frequencies and the radiation angle widens, approaching omnidirectional.

Horns with narrow throats and expanding flares tend to increase directivity as frequency rises. This is known as collapsing DI, and it serves to provide acoustic EQ. High frequencies are augmented on-axis because they are beamed more narrowly as frequency goes up. But this also serves to reduce off-axis energy at high-frequencies.

Direct radiators have collapsing DI also. As the radiated sound frequency gets close to wavelength proportions, the pattern narrows. When the diameter of the radiator is approximately one wavelength across, the pattern narrows to approximately 90 degrees. At lower frequencies it becomes more and more omnidirectional and at higher frequencies the pattern becomes more and more narrow.

If the loudspeaker is used outdoors or in any anechoic environment (such as a treated room), then the reverberent field is of reduced energy. The reverberent field is formed by reflected energies, so in an anechoic environment, there is less energy in the reverberent field. In such an environment, off-axis response has less of an impact to a listener on-axis. Certainly, off-axis response is important to a wider audience, but not so much to an on-axis listener in an anechoic environment.

But nearly all indoors environments are far from anechoic. The reverberent field energies are hear by the listener even when on axis. If the speaker has peculiar directional response, then the reverberent field may be non-uniform and the sound will be unnatural. So it is best to match directionality between subsystems at the crossover point to avoid abrupt transitions in directionality.