It depends. Sometimes, they're just playing. I have a good-sized Moor and a couple of my medium/small orandas like to swim up through her skirts while they're playing 'follow the leader'. (yes, Mushroom is definitely a "she") Some fish are just pests -- I kept a ryunkin for a friend for a couple months while she got her tank going and he was constantly chasing and pecking at everyone. Thank goodness he wasn't staying!
If no one here has the answer you're looking for, you can ask that question 100 times and still not get it. Aside from the breeding spots, goldfish are pretty subtle, just like many fish. There are ways of determining angelfish sex outside the breeding season, but they too are very subtle and mistakes are often made. With many fish, this is determined solely by the "plumpness" of the female. Some other fish are very obvious due to size, shape or color. You cannot mistake what sex most cichlids are. Black phantom tetras have different fin colors according to the sex. Some male fish have larger or pointier dorsal fins. Livebearers tend to have the goniopodium. What sex are your goldfish? You may not be able to determine this until they are much larger, if at all. I bet *they* know, but, again, it would likely be a moot point in such a small tank.