I have a Note 3 and was undecided about removing screen protector. Read the posts here and decided to REMOVE it. Took a while to get the glue out and happy with the end result. Back to my bright screen and quick touch sensitivity. Using the Verizon tempered glass protector and works great.
A few things to add..
There will be glue residue along the edges and not easy to remove it with cleaner or cloth.. Using Goo Gone will help to remove it, you dont want that sticky stuff on your phone.
I think the decision comes down to this..
If you are in a work environment that is dusty or wet sometimes then LEAVE it on. If you are either home or in your car, or office environment and don't have lots of risk with day to day elements, then remove it to get better clarity and touch sensitivity. Just have to weigh out whats more important,.
Not clear if screen can be easily re-applied, might be difficult to get glue to stay within the small edge and not smear into screen.