I switched to the SE from a 6, I previously had a 6 Plus. I'm over big screens. A small, portable, compact yet powerful iPhone is perfect for me. The battery life is fantastic, the camera is fantastic, I can't praise it enough.
Ultimately though you need to decide what will work best for you.