Headlight shuts off, then on, then off...

My left headlight bulb seems to to burn orange sometimes while the right side burns bright. Changing from dim to bright doesn't change anything at all. I just seems to not to work sometimes while other times it works just fine. I can wiggle the housing a little bit whereas the other side is solid as a rock. The doesn't go out, it just glows orange and the other side has no problems. Is this a wiring problem? I figured maybe a R/R if both did that but it's only one.