I just took off my anti glare and I'll never use one again. It really messes with the retina display. It takes all the clarity and beauty away. I love anti glare too but WOW you can see so much better with a crystal.
Same experience here. I initially went for an SGP Anti Fingerprint/Glare as I'd used similar on my 3GS. However, the 4G is a different kettle of fish, with it's retina display, I found it took too much away from the display. So I replaced it with the SGP Crystal version. Sure the fingerprints are annoying, but I'd rather that annoyance then a display that is hindered and held back by the protector.
You have to decide which is more important to you; a crystal clear view with a ton of fingerprints and oil spots, or a slightly less clear view and very limited fingerprints and smudges. Right now, i'm sticking with the anti-glare because i simply can't stand fingerprints. I will admit that i miss the beautiful screen, but i spent the majority of my day wiping the screen rather than playing with it when i had the crystal on.