According to Apple:
"
Your Apple TV (4th generation) can support several Bluetooth accessories simultaneously, up to this many: - One Siri Remote or Apple TV Remote*
- One Bluetooth keyboard
- Four MFi (Made for iOS) Bluetooth controllers (Unless you also connect a Bluetooth audio accessory. If so, you can connect only one MFi controller).
You might be able to pair more accessories, but at some point you might need to unpair others to allow for new ones."
Source: https://support.apple.com/en-us/HT205302
With that in mind, try disconnecting the remote and see if that helps.