I don't have issues with gravel roads really. I think a mistake many riders make is going too slow, the tires sink too far in instead of gliding across the surface like a speed boat. There is a limit though, if it's too loose, the tire will sink and throw you, but compacted dirt isn't an issue for me.
I rarely ride on gravel but when I do I loosen up the way I handle the front. I try to sit up instead of forward and let the front tire be more of a rudder and just stay away from the front break. I also slow down to 55 or so... (just kidding) Mostly I stay aways from it, though, as my bike has enough scratches already...