Audio advice and discussions with audiophiles and professionals. Free discussion forums, open to all and frequented by industry experts. Ask questions and get advice on stereos, speakers, hi-end amplifiers, pre-amps, home theater, tube hi-fi, turntables, vinyl, tweaks, music, films, video, DVDs and more.
I need an honest answer. ABX testing is a methodology that is constantly disputed by the different experts in the field of psychoaucoustics. Just one hit on Google brings up dozens of current research papers from respectable science departments of known universities. At what point does the concept that this is an evolving discipline and will experience revisions of the data and conclusions the same way every other science tool does become important to the discussion? I may be wrong here but it seems that the proponents of the ABX tests as they are currently performed considor the results they obtain to be absolute and incontrovertible. Is that possible in any science? Is this a dumb question? I mean every day they discover aspects of the brain that completley refutes long held beliefs. Look at the phenomenon of phanthom pain; can a science explain that? I know the cursorary explanation; a matrix of synapses that retain a memory of the missing limb and then creates situational responses to something that no longer exists? But the real explanation has not been discovered. In this they are clueless. Thanks; J.R.
Double-blind testing is definitely a good way to get the least psychological bias from people. Blind tests using placebos and test groups are used in the medical field to remove psychological bias. To me, this kind of a testing is a no brainer. But what is also a no brainer is that this is an entertainment industry and that the equipment tested is purchased just as much for its aesthetic value as for its performance. People buy what they like, and sound quality is obviously important but so is looks. I guess that doesn't really address the psychology of the test and whether there is validity in it or not. Still, it just doesn't matter to me. If you want an unbiased answer, do a blind test. To me, the people that argue about this are wasting their time.
Thanks for the reply Wayne; I understand this is just not something you have an interest in and I respect that.
I'm interested in the results of studies, but not the bickering about whether blind testing is effective or not. To me, it's like arguing the color of the sky. One camp insists it's blue and the other insists it's black. Both know that one is talking about night and the other about day, but they seem to like arguing so much, they prefer to continue the "fight" ad nauseum anyway.
It's redundant; thats for sure. I geuss I have that terrier mentality; I can't let stuff go. Always been a personality failure on my part. I just can't get my head around this as an unsoluble problem; the chicken or the egg. Like when you were a kid and tried to imagine the concept of infinity argueing with your friends. I think it's the thought that something so subjective like music can be reduced and quantisized to a stream of data. It's like when animal research tries to dictate that there can be no anthropomorphizing of the subject,that it is projection to ascribe emotional components to animal behaviour. I disagree.
In my local hifi forum[commercial equipment not DIY] they have an ABX forum where no one posts! Being a student with debt, why spend more If it isnt better.
Is that really the claim? I think the best argument for the validity of ABX is a test done some 20+ years ago (I remember the article but unfortunately don't have the reference). A bunch of golden ears participated in an amplifier comparison and, no surprise, null results. They couldn't distinguish their ass from their elbows, with one exception. The exception was J Gordon Holt - he hit like 19 out of 20 or something reasonably similar. That told me something - that a guy who was never accused of being full of shit as far as I know showed he could walk the walk, unlike his contemporaries who just talk the talk. Now he's a crafty old bastard, and he may have found a reflection in a window someplace that allowed him to see the signal input lights on the amp front plates or something like that. But assuming it was on the up-and-up I would claim that it's a clear demonstration that ABX does not obsecure the details as claimed by its detractors - you just have to be as good as you claim to be when somebody is watching (as opposed to all the crazy claims they make in reviews of stuff they hear when nobody is watching).
I've seen people really get up in arms on this topic, and it always leaves me stunned. I mean, why bother? To me, certain things can be quantified, and others can't. I can measure resistance, capacitance, power, response, distortion, etc... It's a little harder to quantify beauty, although we've all seen some "10's", know what I mean? The double blind test is a way to put a reasonably objective metric on a subjective human evaluation. It is a way to ask things about percieved quality without exposing any details that might affect the outcome of the test. The idea is so obviously simple, I don't see anyone objecting to it. Maybe some of the mechanics of the test procedure can be flawed, like the switching device or whatever. But the basic concept of blind testing is good. The information it provides is useful. Then again, if a person chooses a product because he digs it, that's cool too. I don't give a rat's tail if he can hear the quality difference or if it's partly aesthetics or maybe he's just keeping up with the Joneses. The choice may be purely psychological. Imagine choosing a sex partner by blind testing; It's a rather risque example, but I think you get my point. Maybe that isn't the way most people would prefer to choose their mate. As for reviewers and "golden ears," I can't be sure about their tastes or their motives, so that's a different story. We talked about that in the thread called "Reviewers - Is it possible for them to be unbiased?." But as for the simple question of whether blind tests are useful, I'd have to say they obviously are, because they're a way of introducing a degree of objectivity. It's additional information, that's all.
I see now. The ABX argument for me is not about whether performing those tests can provide a reasonable approximation of an absolute truth; which is what they are designed to do;no? I am more concerned about relying on them to prove a point; which is pointless. The concept that since a difference is not quantified by an ABX test it cannot possibly exist is the shibboleth I am after. I don't see it as a competition and there I think is the issue. I see it as a design tool; and that is why it needs to be addressed. Processing sensory input to the CC is so little understood and lacking in a firm basis for making predictions that I am sure in the future the whole concept of ABX testing will be revamped. The best at this research cannot even agree on threshold limits. These tests are rudimentary at best and those who accuse skeptics of the methodology of being reactionists are themselves reactionists. The sad part is depending on this type of testing to make design descisions will once again; like perfect sound forever set us back on the road to great and natural music reproduction.
HI John, Here is a thought that may be helpful: Measurement tools are far more sensitive at detecting differencees between components than the human ear. Acoustic signal fidelity is not terribly complicated, and can be characterized by a small number of metrics. So, components that measure the same along these metrics will sound the same. Now, they will be other factors such as how it looks, the manufacturer's name, what your friends like, etc. that can change how something sounds to you. MEasurement tools can't capture those. thsoe would be the psychological factors. AS Wayne said, it really doesnlt matter why yu like something. Just have fun with it. thanks -akhilesh
AK; Hello; Is that true? I am not sure I follow you here. If that were so, why not find the speaker that sells the most units; define that speakers metric through testing , then build all speakers utilising that metric? You could corner the market in loudspeakers. I thought the concept that all loudspeakers that measure the same sound the same was a dead issue by now. I have to ask; are you saying that the human ear is less sensitive to airborne vibration than a microphone? And that the brain has less power to resolve data than an oscilliscope? Correct me if this is wrong. I'm sorry; I know your putting me on and I appreciate your sense of humor believe me. One thing audio definately needs is a sense of humor.
One things for sure,I trust AB blind testing rather than sighted tests ! Thats the funny thing,its so logical for a magazine based on profit to encourage spending more and more,and more in order to get a 'better sound' Note that the guys who make the best reviewed speakers in stereophile like Dunlavy audio labs,they firstly use measurements to see if a system is good,then they listen if it measures well
Since we hear things logarithmically, we don't hear subtle differences in levels. For example, you have to double power to get a 3dB increase, which sounds like a small change to us. A 10% increase is 0.5dB, and that's just too small to hear. But 10% is pretty significant, really, and is easy to measure. Budget equipment is not always accurate because it isn't calibrated, but it will surely show a 10% change in levels. But man, is it hard to hear a change in levels that get much below 3dB (200%) unless you're quickly changing between the two and listening closely.
Same thing with distortion: hard to discern, but much much easier to measure. -akhilesh
Thats a fact. Level matching would be a priority in any test for audio. I think there exists; within the incredibly complicated, poorly understood capacities of the brain; the ability to resolve and detect nuances of sensory input that our very crude and simple measuring devices cannot detect. There are aspects of testing that focus on certain pieces of the experience and enhance those artifacts to coincide with the testing modality; which is arranged to help define the outcome. We make predictions that re-inforce our original postulates then assign values to the outcome. Then some poor unsuspecting music lover comes along; who is unaware of this and says hey I really like the sound of my NONABX brand speakers, they let me hear the music; unlike those other ones in your test. Well; then by definition he is deluded and a sucker. And we still can't get around the logic; if things measure the same then they must sound the same. We know; we know that isn't true. So, now what? We convince ourselves there is no difference between amps that measure well and ressurect Julian Hirsch as an audio god. Meanwhile isolated Japanese guys listen to Western Electric gear; and when you enter their home; your amazed at what music you have been missing all this time with Perfect Sound Forever.
If it is hard to discern; why do we care? If we can't hear it Then everything should sound fine.
Consider this: Most electronic equipment today that is engineered by competent engineers (and that includes most of the mass market stuff) offers perfect fidelity as far as the human ear is concerned. Some audiophiles (SET lovers) find this perfect signal reproduction not as appealing as SETs, in other words these audiophiles (myself included) LIKE the audible second order distortion introduced by SETs, as well as the non-flat frequency curves. To these people, regular equipment (which is perfectly well engineered) begins to sound "thin" or "metallic" becuase it lacks the "richness" and "naturalness" of SETs. Now extend this thought to analog sound versus digital sound (digital sound offers far more signal fidelity than home analog reproduction). Same logic. Some audiophiles (not me) actually PREFER the audible distortions introduced by analog reporoduction. Are all these audiophiles delusional? Not at all! They prefer something that is not true to the original, and that's fine! I don't really care that my 45 SET amp stinks in terms of measurements, or that an engineer who designed and built it today for any mass market company would be fired on the spot...i still love listening to it in the upper bass - midrange-lower treble. But to expect all the engineers of this world to start deliberately introducing distortion and uneven freq curves just becuase a few of us like it is unreasonable, don't you think? And criticizing the equipment they make is also somewhat unfair, since all they are doing is designing perfectly competent amps & digital sources for as low a cost as possible. And finally, expecting everyone to have the same preference as I do or you do may also be somewhat unreasonable, since we all differ in our likes. SO bottom line, it's probably wise to just chill out, spend less time wondering why the heck the rest of the world can't enjoy the same distortions & aberrations we do, and just enjoy the music. Don't you think? -akhilesh
AK; I knew I could count on you to follow this through. We are attempting to provide the absolute best possible musical experience we can and that's why we do this; right? It doesn't matter that good engineers build mass market equipment that measures well; they have been doing that for 40 yrs. However; there is something better out there and we try to get it. If I follow your logic; then we are done; it's over. The vanishingly low distortion and perfect freq. response graphs mean the mass market stuff sounds as good as it gets. So why are we not out fishing? There is a component of music that is not being measured with the crude devices we are using; that explains why music sounds different than hi-fi. If hi-fi sounded exactly like music; that would be easy to prove; have the Symphony lip-synch the music while a stereo of enough power plays at Carnegie Hall. Saves a ton of money. Analogy; here in NY it can be 98% humidity/70o's/and low Barometric pressure. Does that tell you if it's raining? Does it tell you if it is a nice day? I know it's a crude analogy but it is all I could come up with. Also; I don't take this personal; I like you guys. No one is building anything now so this kills the time. Unless you want me to post pictures of my house and every piece of gear I own? Yawn.
"The vanishingly low distortion and perfect freq. response graphs mean the mass market stuff sounds as good as it gets." John, thanks for your reply. My point was, that perfect measurements don't sound good to many of us. We are both saying the same thing: you are saying that measurements don;t tell the story, and I am saying the same thing, i.e. measuremrnts don't predict how it will sound! All measurement do is predict fidelity to the original signal, not how good it will sound to you or me. I think we are in agreement! -akhilesh
Wheew! Thats a load off my mind. The only reason I pursue this is I just can't understand how there can be such a discrepancy between what constitutes good sound for one person and not another. You would think there would be at least some basis upon which to define why there is a difference. You hear what sounds real to you while someone else finds that same sound to be unacceptable. How can it be. This was brought home to me when I auditioned the Decware SD speaker. How can anyone say that plays music? It only sounds good on one type of music and abysmal on everything else. Now I have to examine this; so I am going to investigate building Martin's T-Line for the Lowthers. What do you think. I am only doing this as a result of his reasonable and methodical treatment of the subject. How about it; any opinions?
HI John, I heard Bob Brines' lt2000 lowther system based on MArtin's work. It sounded very good, but in my opinion will work best as a widerange, not a full range. It will need help below about 80 Hz, and above about 4,000 Hz to sound really good. It's really hard to get a full range that will sound good on every kind of music. Try it out and tell us what you think. If all you listen to are small musical arrangements & vocals, then a full range makes much more sense than if you lsiten to large orchestras, rock, etc. Again, that's just my opinion. -akhilesh
I am not sure if it was the same thing, but there was a Stereophile (I think) article "about twenty years ago" describing a double-blind test that was put on at some audio show or convention. (It may have been Atkinson, not Holt.) About 50 people participated. There were five tests. Two got all of them right. One of them was the author of the article, who was now convinced forever that he had a golden ear. The asshole did not ever stop to realize that pure chance would have made about two people get everything right. I am sure that for the rest of his life he has felt that he had license to pontificate on all topics because he had a golden ear. That was one of the reasons that I quit subscribing to Stereophile, and have not taken it up for even just $10 per year. I wonder if we are talking about the same thing. (No offense, but I doubt that Holt could score 19/20 when everyone else failed.)