You won't lose all the bacteria, since there is some in the gravel, decorations, walls, filter itself, etc. But you will lose a good portion. Can you stick in the new cartridge at the same time for a few days or a week, so that the bacteria have a chance to populate both? That way there will be less of a jump. Or, you could swirl the used cartridge around in the tank so most of the bacteria goes into the water and then can settle onto the sponge cartridge.