Miserable Neighbours

A little advice if possible please. I have been feeding birds in my garden for years, this year we have had a group of starlings visit, not a problem to me, but my neighbour as now threatened that if I don't stop feeding the birds, he'll 'take the matter further' as they are 'sh*tting all down his fence', as he so eloquently puts it.

Personally I'd ignore him and carry on feeding as I'd sooner have the birds as friends than the petty neighbour, but my wife has spoken and to keep the peace she has told me to stop putting food out for them, trouble is, it breaks my heart watching them look for food that is no longer there.

My question therefore, if I carried on feeding, what course of action can this miserable git take and where do I stand in regard to the feeding of birds in my own back garden?

Thanks in anticipation.