Actually, a first-order crossover only shifts phase 90o at the asymptote, which is a fancy way of saying the stop band. The stop band is where the driver is attenuated, so this is sort of a "mute" point. Pun intended.Please notice the graph of phase shifts of first-order filters in the post called "Phase, delays and offset baffle spacing." These shifts are an inevitable part of the nature of filters. Both low-pass and high-pass filters are shown, and equivalent baffle offsets are shown at various frequencies.
You can line up the tweeter and woofer (or fullrange) so that they are in phase at one frequency and at one specific listening position. Move off axis and the whole thing shifts. But I'm not sure this is really a huge problem when you consider that there are few sound sources in a home sound system, and there are walls and furnature and things of various sizes in the room to reflect different frequency components at different amounts and positions. The numbers of phase modifiers from the environment usually makes much more irregularity than do the electronics.