I agree with you, 100%. The condition of rooms in most North American homes is such that a good quality loudspeaker will usually sound pretty good, but there is definitely more influence by the room than anything else. There are walls and tables and pets inside, so it's a reverberent chamber with lots of fill and plenty of oddly shaped reflective and absorbant surfaces.Having said that, most of the time I've found that a good quality loudspeaker will sound pretty good in most rooms. The exception for me are those that aren't on concrete slabs - raised hardwood floors always seem to really screw up the bass and and upstairs rooms do too in some cases. Compared to this, framed drywall construction on a concrete slab has always been pretty acoustically friendly. But there are definitely artifacts caused by the environment that are greater than those caused by the speaker or electronics.
That's why it's tough to measure stuff in one room and use it to compare with somebody else's system in another room. If we're discussing rooms, that's one thing. But if we're discussing speakers, we have to find a reference environment, which is usually anechoic.
Anyway, one of the best references is included in Andrew Hon's paper, and it is the link to a Master's thesis paper written by Daisuke Koya called "Aural phase distortion detection." Koya goes to great lengths to be objective and in so doing, is quick to point out the limitations of his test. In doing so, he really addresses the issue, in my opinion, and makes his work probably the most telling. It is neither a pronouncement of the importance of phase distortion, nor does it say that it is unimportant. It simply illustrates - and pretty accurately, I think - where people are most likely to notice phase distortion and why. I also believe he has done a good job of isolating purely time-related artifacts from those that are manifested in frequency response. That is very important because, for example, if there is 180o phase shift between two tightly coupled sound sources then you would expect audible cancellation which is not purely a time-related effect.
In conclusion, Koya writes:
"The human auditory system was found to be extremely tolerant of even gross phase distortion effects. Although the impulse test signals were very revealing of the presence of phase distortion, more refined research utilizing an improved selection, broader frequency range, and various all-pass filtering implementations (wider range of phase distortion levels and Q) of test signals are necessary to ascertain more accurate permissible levels of audible phase distortion. Improved irradiation methods, such as the use of phase-equalized loudspeakers in an anechoic environment, may also aid in ascertaining more accurate permissible levels."