You could do it with them in, but I'd prefer not. I was swiching my bright red gravel for natural colored gravel in my five gallon, and when the space is that small, theres nowhere for the fish to hide and it might get a little too invasive for them.
I would just take enough water out for the fish to be happy in your holding tank/bucket, put the fish in, and put in gravel/sand, then dump them back in, but be careful, there will be left over water.