Why not use your router to cut the hole for the 3 3/4? I used a home made router jig for making circles and just marked it accordingly for the port size.
How are you now going to put in the insulation with the baffle/face on the 1pi now?
I built my 1 and 2pis leaving the baffle/front off the box. I also cut the baffle about a 1/2" larger so I could flush cut it with a router when I was done in case I was a tad out of square. I then did all the insulating and such. I also used a drill press and used "T-nuts" for holding the woofers in. Are you just going to screw them to the wood? I am sure its fine perhaps I am just anal =P Once that was all done I glued on the baffle and was ready to rock =)