In your music app depending on what you are pleasing be it a song or the radio you have to turn it off from what you are playing - the cross arrows is what you want to tap on.
I do tap on that to switch off shuttle but it makes no difference. My albums just keep shuffling - not good when listening to a symphony. Tapping the cross arrow just makes album jump to another track.