You might think about this - though Wayne hates it. Turn the speaker around - make the back the front. cut the hole to the inside diameter of the basket rim, maybe a bit smaller (a bit, like a 1/16" on the radius).

Put some scabs around the opening of the box that would have received the front, and plan to screw the back onto the scabs to make it removable. Now you can access the driver after installation.

You can still mount your port to the now-removable back - it really doesn't matter much where the port opening is unless you plan to flush them back against a wall.