In the grand scheme of things, the display brightness doesn't make that much of a big deal.
The battery load between lowest and highest screen setting only changes about 100mA. While it will absolutely cause your battery to drain faster, it still doesn't affect the overall battery range as much as using it does.
For example, an iPhone 4S with all of the radios off and displaying a static screen at lowest brightness: 87mA (16 hours)
Same conditions, full brightness: 189mA (7. 5 hours)
The current draw is liner throughout the range, so you can estimate different settings pretty easily.
For perspective, simply typing text into the notes app causes the current draw to climb an additional 300mA, and an additional 25mA if you have keyboard taps enabled.
If you factor in the wifi, cellular, and GPS radios turned on and working, you can start to understand why the battery life is so short with these devices.
It seems counterintuitive that typing a note or swiping your home screen will drain your battery faster than watching a movie, but Apple made the decision (and a good one at that) to sacrifice battery endurance in order to max out the CPU on user interactions. There's a reason why it appears to run so smoothly - and it's not because of special hardware.
Your battery could have lasted longer, at the expense of lags in the user interface.