If you are still within the contact just press cancel, if its empty it's gone again.
So you could try just to clear the field and press save, maybe that will also remove it.
Empty fields are not shown on the contact card, so is it then still an issue?