Good question.
Most of the time I have had to do A vs B listening, it has been in loudspeaker or driver development. In that case, I eventually reached sort of a dilemma.
After some time in development, one reaches a point where any change you make may well make a “difference” BUT often one then finds that the change actually “helps” some recordings but “hurts” others.
In which case, one case has different flaws than the other but there is often no way to tell which one is actually more accurate (that is assuming accuracy is the target).
By listening, your judgement A over B or B over A may depend entirely on what you play through them. Specific coloration’s in a persons speakers can even shape their taste in music.
At this point, I usually decided to go whichever way appeared to be the most accurate when considering the various TEF measurements.
It is kind of funny too, many people think of speakers as being “pretty good” reproducing the signal, yet if compared to even junky electronics, most speakers (measured the same ways) would appear to be seriously broken AND they spread the energy out in time significantly (even before being put in a room).
The problem is that some things that look like show stoppers in the measurements are hardly audible, other things are audible but are much more subtle in the measurement.

I would not want to scare anyone away from listening tests just because they are not blind.
In hifi just as it is in Pro-Sound, there is a huge degree of preconceived notion and expectation which is normally the result of a great deal of proper and costly marketing.
Even if one is aware, side by side comparisons often show large differences in loudspeakers, large enough to clearly “hear” the difference between acoustically based and marketing based performance.

For amplifier testing, on the live audio board amp shootout for example, when “blind” most (all but one) of the people (who were all in audio) couldn’t hear any difference between most modern pro amplifiers.
In a different blind test, through revealing horn speakers, no one present including myself could hear any difference between my Threshold Stasis and a QSC pl-236, until a medium volume, when the vastly larger power of the QSC was audible (in a positive way) in the dynamics. I used pro sound amps for “home” use from then on.

Many modern tube amps (as opposed to the older designs) tend to have very significant harmonic distortion, levels well above what are clearly audible in testing.
It also seems like pretty often the more exotic and expensive the amp is, the more exotic the reviewers language becomes and the greater the distortion and poorer the bandwidth is too.
Here, since many parts previously thought to be “bad” (like iron core inductors, transformers etc) have significant built in non linearity, many of these designs are refined by “taste”, balancing all the “flavors” into a gourmet’s reconstruction of the original input signal.
Personally, as opposed to a guitar amplifier etc where the right sound is everything, a re-producing amplifier is supposed to be a “straight wire with gain”, not create rich. warm tapestry with everything you feed in.
The flip side is that Tubes can also be made to deliver low distortion AND can have a small distortion spectrum (both very desirable from an accuracy point of view).
Do tube amps sound better? Is it because they are more accurate or are is it euphonic coloration?
I don’t think one can make a blanket statement about that or much else, it depends case by case.

Assuming that you have already past the point where differences are obvious, then what ever it is you want to find out by testing remember you need to at least have A or B to chose from. People have a poor acoustic memory, you can’t have much time between A and B and remember you can be easily fooled by not matching the levels first.
Listening for a fine detail improvement alone, after you have modified something is sort of like galloping really fast on a horse, unless you have 2 horses, there can’t be a race or statistical winner, even if your unable to imagine anyone riding any faster.

Does that mean that last little detail you modified helped?
Maybe yes, maybe no, the only way to know statistically (as opposed to just an unshakable belief) for sure is to compare side by side, “mit and mitout” as they say it Bratwurst country north of me.

When the differences are small enough, then you also have to remove prior knowledge and inputs from the senses unrelated to the pressure going in the ear holes.

There is a surprisingly large amount of money spent on things where the difference “goes away” when the test is limited this last way. Strong motivation to discredit the method.

Tom Danley

“What if” the current “gas crisis” was actually some sort of real shortage, or was not so subtle “pressure” from the Arab world (OPEC), circumventing our “unbiased” news media, instead of just another big profit grab like the other gas crisis’s in the past turned out to be? Umm.