Noise can certainly come from inductive or capacitive coupling, so you could expect to pick up noise when a high-impedance circuit is placed near something like a large power transformer, fluorescent light or electric motor.But you might also check your wiring. If a cable has a loose connection - particularly the ground - then these kinds of symptoms often result. It will be perfect in one position because the ground connection is solid. Then you move it and the ground opens up, so the circuit becomes noisy.
You can also find examples that go the other way. If a ground loop condition exists, then removing a ground connection is actually preferred. You only want a single point ground, or at least, that no current flows between ground connections. When there is current flow along a ground conductor, then any resistance in this conductor will cause a voltage drop. So connection at different places along this condutor will then have different voltage potentials, instead of the expected 0.0v reference ground.
I think some experimentation is in order. Try different cables, and try different sources and amps. We want to isolate the problem. You can also try different positions, but I think you'll want to verify that your connections are proper first. Otherwise it's easy to chase your tail.