New device, new hardware. The codec in the 5 drives me nuts with the click/pop every time the IC goes into or out of shutdown. If I've got my headphones on but the device is silent (like right now) it makes me crazy when the keyboard slides on or off. I actually replaced my first phone, thinking it was a problem with the device, but it's present on my new phone and also my new iPad 4.
I'm pretty sensitive to this problem, because I work on these and other audio semiconductors for the mobile industry. Seriously, the requirements that some manufacturers demand regarding issues like this make it hard to believe that the new iOS devices went with this solution. The iPhone 4 was pretty solid, so it almost seems like a step down in the 5. It sounds "crunchy" every time it happens.
I spend much of my time evaluating audio quality with precision test equipment and listening tests, and to be honest, I usually find something that bothers me with every part. I've long gotten over how the built in speakers sound, and audio over Bluetooth has always sounded bad to me. Most of the content is MP3, anyway, so it's going to sound kind of crappy anyway, even over USB.
I'm not an audio snob, but my job requires me to evaluate every nuance of a part and identify any weak spots or bugs that can be fixed before a part can be introduced. I've simply gotten really good at doing this over the last decade.
In a way, it's kind of nice, because I find something wrong with everything. Expensive gear sounds just as bad to me as cheap stuff, so I don't bother spending the extra money!
My advice is to listen to another iPhone 5 to determine if yours is worse. If they're all the same, either learn to live with it or get a different device before your return window runs out.