In the early days, absolutely. I was really concerned about putting insecure, rushed-to-market devices on my home network. I didn't even consider them getting bot'ed and used in DDoS attacks, but I didn't like the idea of opening holes on my otherwise secure network.
It was one of the reasons that, when I saw all those articles saying HomeKit was delaying vendors and Apple was being jerks by forcing them to use end-to-end encryption and secured Wi-Fi chipsets, I had to laugh. That was exactly what I wanted.
So, when HomeKit came out, that's when I started amping up my home automation.
I don't currently have any security concerns over HomeKit.