Deterring squirrels from ground feeding tray

I have a ground feeding tray in my garden, which quickly attracts squirrels whenever I put any food out for the birds. I invested in an RSPB adjustable ground feeding sanctuary but the problem I have is setting the holes to an appropriate size. If the holes are set too big then the squirrels soon raid any food left out and if the holes are set too small then this stops larger birds such as blackbirds from feeding. I have tried sprinkling cayenne pepper on the bird food to deter squirrels but this doesn't seem to stop them and I've tried putting food out elsewhere in the garden for squirrels but they still come to the ground feeding tray (as well as their own feeder). Does anyone have any ideas on how to stop squirrels from eating the food on the ground feeding tray whilst still allowing larger birds to do so?