When you're listening to music over BT, you're getting MP3-like audio compression over A2DP. Thats good enough for most peoples ear's, and Apple's AAC over BT should improve upon that.
When you're talking, audio doesn't fare so well. Your voice gets processed by the headset's DSP, and downsampled to either 16khz or 8khz mono, passed through a lossy encoder, and then on to your phone. Thats before it even gets to the phone, which then has to re-encode your voice with a different lossy codec, generally at 8khz, before passing it on to your carrier.
Sometimes its the DSP that misbehaves, sometimes its the BT radios, sometimes the second compressor chokes on artifacts from the first compressor. Sometimes though it works just fine.
See, its the future.