Can I move a bird box?

Bought a bird box yesterday and left it on our balcony to put up in the next couple of days. Just noticed that a bird has started building a nest in it.

It is right near our balcony door which gets used a lot so could/should I move it to the other side of the balcony, or just leave it where it is and hope they don't mind us coming and going?