Welcome to iMore! It does seem unfair. But it's part of the security and the check and balance of verification. It's an inconvenience, but not the end of the world. You're fortunate to have what you do. Good luck!
Sent from my ancient but trustworthy iPhone 5.