I think what you're saying is that the term "total harmonic distortion" is somewhat ambiguous, so measurements of this value don't mean much. They don't tell us what levels of each harmonic is present, just the combined total. If that's what you mean, I think maybe I'd agree that an ambiguous number is a waste of time.But to me, harmonic distortion is one of the most offensive commonly occuring anomalies created by audio reproduction equipment. The most offensive is response anomalies. Things like a complete loss of treble or a large peak in the midrange, that sort of thing. That bothers me the most. But clipping is definitely high on the "ugly" list, and it is characterized by exaggerated amounts of odd harmonics.
Certainly, low levels are imperceptible. I doubt anyone can hear a response anomaly of 0.1dB and few can hear 1dB, but most everyone can hear a 10dB swell or dip in a particular band. Midrange is easier to notice, just like the phon curve shows. Same must be true of distortion. Probably nobody can hear harmonics that are -50dB, but -10dB is another matter. And same as the phon curve, the band which harmonics fall in is important, as is their relation to the fundamental.
So what I'm saying is that I think reductions in harmonic distortion are some of the best improvements one can make in an audio system. The figure of "total harmonic distortion" might not be particularly relevant, but I am certain that moderate to high distortion levels are very offensive, and probably one of the worst things a component in an audio system can cause.