It's not really true that they don't affect performance. I think the common belief is that they aren't really doing anything in the background because some articles have been written "explaining" how multitasking in ios works. However, that's not always the case.
There are a huge number of people that complain about poor battery life; some have seen drops of 10% an hour while in standby overnight. From what I've observed, that drain is caused by apps that are misbehaving in the background.
I keep a pretty tight leash in what my 4S does, and I spot all kinds of things that cause improper battery drain.
I use BatteryDetective on my jailbroken phone and can see the actual battery current being consumed. When everything is running correctly, the phone draws an average of 4mA while in standby. With a 1400 mA/hr battery, it should be able to run a LONG time in standby.
Every once in a while, I'll notice that the battery seems to be draining faster than it should, and I'll open BatteryDetective and notice that it's burning 200mA or more during standby. That will drain it pretty quickly. In EVERY case, I can pull it back down to normal by closing apps in the multitask bar. Believe it or not, one of the biggest offenders is the stock Music app. Leaving that one sitting there is - in my opinion - probably the biggest culprit for most battery issues.
Another one that causes problems, oddly enough, is NewsRack. I use it daily, but every once in a while, it will get stuck on something and burn 30mA in the background. Not much, but enough over normal to notice decreased life over time.
A couple of months ago, I had a problem with iTunes Match when something in my Cloud data became corrupted. It would crash iTunes on any computer that I logged in with, and the Music app would burn >400mA if I opened it. My phone would start getting hot and the battery would drain almost fast enough to watch. Apple had to delete my Cloud data to restore operation, but it was a great example of how background apps can severely impact the device.
Bottom line, closing apps in the multitask bar is a good idea, no matter what anybody else claims.