I know what you're talking about, but I've not had that problem with mine (day one purchase). Maybe you got a bad one? ...
It did it on the red one I had on the gold 5S's I had too...maybe I'm just being overly anal about it, but it was enough to make me search out another case (when I bought the Spigen case pictured above).
I also noticed a few other issues with the leather case...
- When propping the phone between my cheek and shoulder (talking on the phone while doing something else), people said they couldn't hear me talking at all or that my voice was very faint. This hasn't happened at all with the Spigen case (my guess because of the open design around the mic vs. Apple's little holes).
- Some 3.5mm headphones jacks would not fully engage the input and would cause crackling and audio distortion from playback...I found this to happen with my friends tape adapter in his old truck and the headphones we rented on a delta flight (tried them out of curiosity more than anything).