Regardless of what you're doing on the phone, that doesn't sound right. The battery may be bad. To check the battery, try this test. Many Genius Bar techs use it.
Write down your usage and standby time, press the sleep/wake button (or lock button, as some call it) to put the device to sleep, and set the device down for five minutes. When you come back, take note of the change in time. If your device is sleeping properly, then the Standby time should have increased by five minutes and your Usage time by <1 minute . If your Usage time rises by more than one minute, you have a drain problem. Something is keeping your device from sleeping properly, significantly shortening the time it will last.