Being a pansy

The word pansy has always had a negative connotation when describing someone. It implies that the person is a weakling.

Yet every winter I'm always impressed with how these *delicate* flowers hold up in the freezing temperatures. They are way tougher than most.

So, how is it that this association came to be? We should be calling tough people pansies. That would just make more sense.