Removing nesting box? or admitting to trespassing?

can anyone help me please? A few months ago I put up 2 bird boxes on my neighbors trees, wrong thing to do I know as she was on holiday and I didn't have her permission. Anyway my neighbor has now come back from holiday and is demanding that I remove the bird boxes ...... one of them has blue tits nesting in it, I do not think there is anything in the other box ...... where do I stand? I feel I should refuse to remove the boxes and suggest that they should phone the Police and report me for trespassing in the first place? but would welcome any views/comments.