Wood pigeon nest outside window

Hello! A couple of wood pigeons have started to nest right outside our upstairs bedroom window on some temporary scaffolding.

However after a walk at lunchtime I walked into the bedroom to find the nest empty. Is it likely they have abandoned the nest or is this normal nesting behaviour?

How long should we wait before we can remove the nest?

Thanks Slight smile