It's the nitrites. They're more poisonous, and cause long-term damage to the fishes' ability to process oxygen. You need to do water changes to keep them under 1ppm, preferably less than 0.25 ppm. Add a pinch or two of aquarium salt (just a pinch or two) to help reduce the effects of nitrite poisoning.
Basically, the nitrites affect the hemoglobin, and your fish are suffocating.