I've got a bumper on my phone and I love it. I thought I wouldn't, and only got it to have some sort of protection until better/more case options came out. I think I'm sticking with the bumper, though. The rubber edges keep the phone raised just enough when you set it down that you don't worry about scratches. I haven't dropped my phone yet, but I'm confident that the phone with the bumper will survive 95% of the falls it would take. From watching all those drop videos, it looks like the weakest parts of the phone are the corners/lip of the glass. It seems that the phone always shattered when it was dropped on it's side/corner and the bumper provides perfect protection for that area while keeping a very slim form factor.
To answer your question directly- I've only noticed difficulty in pulling my phone from the pocket of athletic shorts. Khakis, jeans, everything else has been fine.