Brown is good. It's the stuff that keeps your cycle going. But you may notice a fast buildup if your water has a lot of particles in it, which can happen if you are overfeeding or frequently disturbing your substrate.
Don't ever wash your filter media in tap water to try and 'clean' it - just swish it out in old tank water to loosen the large particles so the water flows freely.