I would say false chum my device gets a little dust under my leather case however it is minimal and I don't think it should damage the cosmetics of the hardware at all .
Fine dust or sand does have the potential to scratch your screen, but if we talking about general dust from day to day use then everything will be fine
Maybe just take your phone out the case once a week and just blow into it. That should be enough to keep it looking clean. Hope you enjoy your iPhone SE! Don't stress about dust!