If you have your woofer producing higher frequencies, the damping material will help obsorbe those frequencies inside the box, preventing standing waves and smearing from those frequencies eminating from the port. It will also serve to enlarge the cabinet somewhat as the woofer perceives it.

If you add to much stuffing however, it destroys the function of the ported box. It'll run more like a sealed box. That might be your problem.

P.S. polyfill in a sealed box is by no means necessary, but usually won't hurt.

Adam