Sparrow nesting times

Hi. 

I have a huge bush in my garden that seems to be the local hangout of all the sparrows. Over the summer, the bush gets huge but I am loathe to cut it back due to the nests, buts it makes access to my house tricky. When is the latest in the year I can cut it back as not to then disturb the sparrows and their nests? I am in SE England if that is relevant....