Nesting wood pigeons

Hello, a pair of wood pigeons have set up a nest in a small Salix tree in my garden. The nest is eye level and there is now one egg in it. The problem is that my garden is so tiny that every time I go out there? It’s scaring them and causing them to flee their nest. One of them does seem to return to the egg but I’m worried about how long they’ll be tolerant of activity in such close proximity. Could somebody please advise me as it’s going to be impossible not to use the garden space with summer on the horizon.