To remove or leave alone

Hi, I have a fledgling in my garden, which the parents are keeping fed, last night I went out the back garden to check all was OK and spotted 2 cats in my garden, I chased them away and brought the pigeon inside wrapped in a  towel and placed in a box, I put it straight back outside this morning and a parent came to feed straight away, I'm concerned to leave it outside tonight in fear that the cats will return, is it OK to bring it indoors at night to keep safe or should I just leave it and let nature take its course? 

Thanks!!