With that amount of fish in a new small tank it sounds like ammonia poisoning, but without a filter it could easily be low oxygen levels. Is it possible to test the water for ammonia? You said they are also not eating as much, or at all? That sounds a lot more like a sickness than low oxygen, get the 10 gallon as soon as you can and switch them but that won't fix it immediately. You will still have to cycle the 10 gallon and with that load you would be better off switching the mollies to the 10 and leaving the platies in the 5.5 for a while while they cycle.
Also are you using a heater? Warmer water has lower oxygen levels but it really sounds more like "new tank syndrome". If you added the mollies and tried to feed them immediately they usually don't eat for 24 hours to adjust to the stress or being moved, they can appear to be sick too but they are just stressed and vulnerable. If you have any kind of ammonia neutralizer like ammo chips or ammolock I would use that as well as daily water changes until the 10 gal comes, then split them up and test both tanks regularly while they cycle.