When to switch over to winter feeding?

Over the winter I enjoyed a huge number and variety of small birds in my garden. Over the spring and summer the numbers tapered off so I adjusted the feeding accordingly, for most of the summer I've just left out mixed seed for the 40 or so house sparrows which live in the tree bordering mine and my neighbour's garden. Now I hope to attract all the finches and tits back as autumn/winter starts to bite but my question is when do I start providing the sunflower hearts, niger seeds, fat balls etc to bring these visitors in? I don't want to waste them as feral pigeon feed!