I have the same problem, and it all started after the last update! At first there was a microphone on the right side and I could easily dismiss the stupid thing by holding it down and swiping up. Now it just shows the timer. I am going to restart my phone to reboot, and hopefully that fixes (temporarily) the problem. I agree that it is most annoying, and I too wish there was a way to permanently disable the feature until I wanted it!