Not worth it. I was in the same boat as you and the screen real estate is just perfect for what I use the ipad for.
If you have the iPhone 5, ipad 3 is perfect. The mini sits too much in between too have all three. If you didn't have an iphone 5 and had the iPhone 4/s the ipad mini would of made more sense. Leave the ipad 3 for home/work, use i4 as you do normally, and carry the ipad mini around.
Also not sure where you live but carrying an ipad mini during winter might be easy as you can put it in your jacket/coat but come summer time where is it gonna go? Your back pocket?
I used to do that with the Galaxy Tab 7" but it got old quick and the screen size gets annoying fast as well. Constantly have to zoom in.
Most of the time my iPad is in my laptop bag. I usually don't carry my iPad unless I have a bag of some sort to carry it around. It's too big to just be carrying it around.