Both my iPhone 7 and iPhone X are on 11.1. However, under the Message app in Settings, neither has the option to sync messages to iCloud which I thought was a feature of iOS 11. Not just greyed out, it's not even there. I'm not clear why this option is absent.
It's an option that Apple is, apparently, still working on. It's not even in the beta version right now (other options we've been waiting for are also missing). I'm married to a programmer/software engineer, and sometimes things are just not ready when the deadline arrives. It's not supposed to happen and certainly not planned that way. Hopefully Apple isn't having so much trouble with it that they decide to scrap the idea altogether.
Technically speaking, we don't recommend using any 3rd party software not officially supported by Apple. At this time there is no Apple supported way to only backup or restore specific things from or to an idevice. If you do use 3rd party software that is not supported by Apple, you do so at your own risk.
Perhaps you can reset everything listed under Reset
with the exception of erase all content, uninstall all 3rd party apps that Apple did not already add, clear browser data etc, move your photos to your computer and then delete them from the phone, remove everything from wallet, etc. etc. etc on your old phone and just leave the messages. Then do a new backup and restore that backup to your new phone. Big pain in the butt to do, but at least you can get what you want without putting your security at risk.