My Opinions:
Android phones have cheap feeling hardware. Plastic and stuff makes it feel chinsey, I really like the iphone 4 hardware, it is solid, and has a premium feel to it that I have not found in any other phone.
Android as an OS is awesome, except it has NO regard to how it uses your resources (battery, data, etc). All these cool widgets and some great apps that constantly run in the background draining your battery, which most users have no idea how to stop them. There are FAR too many tweaks that need to be done in your settings to get optimal battery life etc. Most users will not even bother or know how to do this, so they get $hi++y battery life and a sluggish phone, I have had NON of these issues with my iphone 4.
BLOATWARE, BLOATWARE, BLOATWARE.... Android, c'mon, it is freakin' ridiculous. We pay hundreds of dollars for a top of the line phone that the carriers FILL WITH GARBAGE APPS that we DO NOT want, and cannot delete. These apps start on their own and run in the background and the ONLY WAY to stop it is to root the phone. This is probably one of the top annoyances I had when I had an android (Evo 4G).
Apple may not be as open, and I am not proclaiming Apply Fanboism, because I am a fan of cool gadgets, not from a particular manufacturer. I may someday switch back to Android, but in my opinion, they have a LOT of work to do first. And the fragmentation issue is real, there is just no way they are going to maintain a smooth user experience with so many different handsets out there, and then let HTC, Samsung, Moto, etc. slap a new interface on top of it (sense, motoblur, etc).
I am pretty sure I will be sticking with apple at least through the iphone 6.... then I will have another look at Android.
EDIT: Maybe my biggest annoyance was actually battery life... real life tests (with standard battery) at the end of a typical day, my Evo battery would be down to 30% (or less if I was in an area with poor network coverage), my iphone 4 has never dipped below 60%.... take that for what you will. Android battery life sucks compared to iphone. Now what if I stay out late and can't connect to a charger? The android phone is DEAD.... iphone can go all night, many times I will go 2 full days between charges.