It's really just a matter of preference really...but I tend to agree with you in that if you don't use features like this and you have a desire to turn them off completely instead of trying them out at all to see how they work in your every day life, why even buy the device in the first place? The same kinds of threads popped up when Siri was first introduced, and then when the finger print scanner came out, and then when Apple Pay came out, and I'm sure there will be dozens more.
In the end, it really just comes down to personal preference and decisions on what you want to do with your own device, but at the end of the day, is it really worth the investment in these devices if you're not going to try and get the most use out of the spectrum of new features on them? I think having Siri in an "always on" state makes it INFINITELY more usable than before, and I intend on beating up Siri with requests like never before!