For vented cabinets, port position won't matter unless airflow is restricted. So if the port is small and/or displacement of the driver is high, vent airspeed may cause turbulence from a nearly spaced wall. It's probably a good idea to keep a distance of at least port diameter or so from any obstruction.

Another thing to consider is standing wave nodes. If all dimensions of the box are small compared to the wavelengths presented to it, then standing waves won't form. But if a box dimension becomes acoustically large - 1/4λ or greater - then standing waves will form inside. Port and driver position are important in a box like this. Once the box is large enough in any dimension, it develops pipe modes and if you put the port or driver in a standing wave node, you'll see ripples in response as a result.