If you're moving from a 6+, as long as you like the smaller size, it should be an improvement all the way around... faster processor, better camera, etc. If you're moving from the 6S+, then it will be all about the form factor trumping things like 3D Touch, a nicer FFC, and having a barometer.
I went from a 6S to the SE because it was too practical of a choice to avoid. I don't care about the front facing camera, I don't care about a barometer, and 3D Touch is nice, but not habitual for me. I love the form factor and slightly improved battery life on the SE. The screen isn't as nice, but that's something you won't even notice unless you sit there and look at both phones constantly. Once you're off of one screen for a few days, your eyes will forget what the former screen looks like.
If you're on a 6+ and want the smaller form factor, IMHO the SE is a no-brainer unless you want to see what 4.7" form the 7 brings. If you're on a 6S+ it's a little bit more of a decision, and it should be one made under the assumption that you REALLY want the 4" form factor. If you're not sure on that, the move from a 6S or 6S+ probably isn't worth the hassle.