Those little sponges wouldn't hurt anything. They add comfort actually. I don't know if the manufacturers took moisture into account when making headphones, but extra protection wouldn't hurt.
Headphones usually don't have the extra sponge for protection like earphones. That's why I asked if I need to add one because my friends and I are daily active and we have headphones most of the times. I thought the sweat may destroy the pads quicker without protection.