I take photos with the buttons on top (where they should be) and they show in Windows upside-down. It's something to do with the meta data. They show normally in photo apps like Photoshop, it's just Windows that has issues.
If you open the pic in photoshop and immediately save it (without doing anything), they suddenly appear correctly in Windows.
Beamed over the interwebz from my iPhone 4S using Siri.