I think it depends on the plants. I've heard of certain plants anchoring differently in different substrates, due to the gaps in the rocks.
Another thing I hear, but don't know too much about is that nutrients will more readily fall into the gaps in gravel, while they won't in sand. Don't take my word as gospel, but look into it for yourself.
Just remember that as much as you gravel vac you're going to stir up extra junk when you remove the gravel. One option I've heard of is to just add sand to the top.
I'm someone more experienced will come along and set me straight.