This is normally due to you connecting the adapter to the cable before connecting it to the phone. If you follow the instructions on the document i have attached below you shouldn't have this issue any more.
http://www.pac-audio.com/PACProductData/PXAMG/3_Tech_Briefs/lightning_adapter.pdf