Hmm...well, ammonia can definately make a tank leathal to goldfish and you want to check your water chemistries with a test kit...
...but if it -looks- dirty (brown scum on glass/gravel/decorations, brown or green cloudy water, white fuzzy stuff floating around or on the bottom, filter grimy, water smells..etc), it is usually because you need to clean it.
I have six fantail goldfish that are of a rather large (tennis ball) size in a 29 gallon tank at the moment (YES JOSH I KNOW I NEED A BIGGER TANK), but can't afford or provide space to give six fish 20 gallons each (that would be 120 gallons for those fish). So, as a compromise, I clean their tank twice a week. I do a 5 gallon water change with siphoning on Wed, and a 10 gallon water change on Saturdays.
I also have trained my fish to be neat eaters. Since they like to eat from the bottom, I've put a plasic "dish" there and put their food on the dish (yes I put my hand in the tank to get it there). I feed soaked pellets, and only enough that each fish gets a mouthful of it. I feed twice a day.
Goldfish are one of those fish that like large and frequent water changes and lots of fresh water. A powerful filter is also the key, I have a Marineland Penguine 330 on my tank, and the biowheels do a great job at helping to reduce ammonia levels, and water changes will help remove nitrate build up.
Good luck
~~Colesea