I've had a few issues where one or the other don't work (very rare, and normally because I'm taking them out a bit too often), but for me popping the offending AirPod out for a few seconds, then popping it back in resolves the issue (might take a while to start working, i.e. 5 seconds).
If it is more frequent than this, I'd try to reset the AirPods and re-pair them:
1. Open the AirPods case so you can see the status light
2. Press and hold the white circular button on the back of the AirPods case for at least 15 seconds, the status light inside should start flashing amber.
3. Ensure that you AirPods are in the case, then simply open them like you did the first time to re-pair
(you might have to re-choose any specific settings you made, for example double tap preference).