I went from a 5s to a 6s and have no regrets. I considered the 6s plus, but I have small hands and it was too much, plus I carry my phone in pockets much of the time, and I just didn't want something the size of a plus. That said, I do see the merits of the SE... I don't think I'd consider going back, but it's a personal preference.
Rumour of the resurrection of the 4 inch iPhone has been simmering for quite some time. Hell, it even started as a possible 3.5 inch phone (I do believe Apple once considered it. If go small, go really small and the 4s is still popular
. But I was not sure and going back and forth between the 6s and the 5s 3 times in the last 6 months alone!, poor Apple LOL.
The 6s is not really "too large" and actually started growing on me as the sweet spot, when SE came true. Then I jumped on it. I'll see how it goes and if I really need a 6s size (for typing and aging eyes), I will rather wait and grab a 7. That's my strategy now. The SE is serving me really well now, particularly in combination with my iPad Mini 4.
But I fully understand that 4 inch is a controversial size, sitting right between the camps for small and large screen sizes.