We have had sparrows nesting in our boxes and now it looks like they are returning to roost over winter. I have not yet cleaned the boxes out. Shall I leave it or clean them anyway?