Hey Joe, I was thinking the same thing you were in my related post "wait for 4.1?" but I bit the bullet last night and did it!! It took about an hour to download the 380ish mb OS then about 15 minutes to install. It runs awesome and I notice my wi-fi is faster than before! The biggest plus for me was the "all inboxes" for mail, I have 3 accounts and wow is this a lot simpler especially when it links all threads together!! I love it! No issues so far!!
I haven't upgraded to iOS 4, and unless there is an untethered jailbreak on it's way I don't plan to upgrade either. I've come to rely on lockinfo, sbsettings, notifier, and other jb programs. Even then I was never that impressed with iOS 4 so if an untethered jailbreak does come out I don't see myself upgrading unless applications I rely on no longer support 3.1.3.