In fact, the way I do it is: start with the amp, then build a speaker using T/S params, and a passive netwoork, so that the speaker mates well with the amp.

I have personally followed MArtin;s advice and used a passive network, and it improved my speakers considerably, with no sonic deteriorations that I could hear. You may want to try and find scientific studies that investigate if phase shifts (without frequency shifts) are audible. My own experience tells me: no.

IMHO the major determinants of speaker AUDIBLE performance are frequency curve, distortion and efficiency (with efficiency being a distant third, provided reasonable amplification is used).
-akhilesh