I had a Note 4 and also returned it. It definitely lags, not as bad as older Samsung phones, but it's noticeable. In my case I disabled all motion gestures and s voice, turned off s health, disabled s finder from the home button, swapped to Google Keyboard and the Google Now launcher, and despite all that slimming it still lagged so i can only imagine how bad it is for people that just use it the way it comes out of the box.
What really got me was the TERRIBLE a use of the screen size. Samsung seems to inflate the DPI higher than the resolution/size calls for naturally and it hurts. 5.7" screen with a 4x4 icon grid? Even the LG G3 allows for more with it's 5.5" screen... I know you can use after market launchers to get larger grids, but I'm talking about stock screen usage. The scaled DPI settings are hugely noticeable in the browser (stock or Chrome), apps with lists of content like Twitter and Facebook, etc. I did an in store comparison and the iPhone 6 Plus displayed more content on the same web pages than the Note 4 did despite having the smaller screen. Reducing the text size in the system menu is only a band aid as well because it does nothing for images and other GUI elements. The upper bar in Facebook was larger on the Note 4 than on the Nexus 7, for example.
Anyway, for a device hailed as the king of phablets, it sure does make terrible use of the screen. They should have options for scaling/zooming like the 6 Plus does, not just font size, instead of assuming everyone wants to have everything blown up all the time. I personally use big screen phones to have more on the page, not just blown up versions of the same amount of content I get on smaller phones.
Regarding the 6 Plus, I actually picked one up yesterday. I was in the T-Mobile store asking about the Nexus 6 and they just happened to finally have some 6 Plus in stock. I was using an old Nexus 4 since returning the Note and I figured I'd give it a shot despite my draw to the Nexus 6. I like it so far, but there is still a chance I may return it for the Nexus 6. No matter what, I am very pleased with its much better use of its screen size when compared to the Note 4.