I think you are referring to adding extra mass to the cone? You're 100% right - adding extra mass will lower Fs, lower sensitivity and will also raise Qms and hence Qts. It will reduce excursion-limited power handling though, if your intent is to use it at lower frequencies, which is what I think you want to do as you wouldn't add extra mass unless you wanted it to do deeper.

Adrian