I personally, as yet, had any problems with feeding birds and neighbours objecting, but then the immediate neighbours also feed the birds with one exception, the young family on one side.
While I'm not fully aware of the past history, other than you mention this is an annual thing, which suggests to me this probably has ongoing issues.
I appreciate it isn't always easy to talk to neighbours, there are those who will did their heels in irrespective, but sometimes, and often, communicating with the neighbour can help to ease the problem. The best way to achieve that is to let them tell you why they don't like you feeding the birds. They may have a genuine concern, rats, mice and other rodents, or they may feel the birds are dirty, in that they carry disease.
If they are genuinely having a rodent problem, and it isn't uncommon after all, bird food on the ground is easy pickings, then you could work things out in that they talk to you, you can aim to work a solution out.
It could be the pest control vehicle is for a totally unrelated problem. I would have thought a pest control person would be calling on you if they felt, or knew, that the origins were with your feeders, otherwise it would be a wasted job, because the source hasn't been controlled.
Here, living in a very built up area, I use trailcams, mainly to monitor bird and other wildlife activity, but at times, a rodent can be spotted, and at the point, the food is withdrawn for three weeks or so, depending on rodent observations following that first sighting.
The birds will soon find food, they won't starve, and to be honest, we leave empty feeders, even now during breeding season, from time to time to encourage them to seek other food sources, and doing it during the breeding season also helps to encourage youngsters to seek food from other sources.
It could also be something as basic as the bird poo left behind.
Trying to understand their concerns, could help to resolve the issue, but there's no guarantee or magic wand.
But the birds won't starve, they will seek out other food sources if the feeders are empty.
Mike
Flickr Peak Rambler