^ That. Some see it as a badge of honour and feel they need to go (or be called names or something!?! I dunno). If they were watching something on tv which was dire and boring as fuck they would stop and watch something else. If the pub they went to started selling piss warm jango they would go somewhere else. Just about anything else people don't enjoy anymore they don't continue it. Football is a different fish tho, not sure why!?
Probably because a football club is something you have supported with family and mates, it's a shared thing, passed down through generations usually.
That said, I agree with you, if you don't want to go you shouldn't go.