According to the Kern River Preserve in in southern California, if you live in a very cold climate it is recommend that the perches be removed from your hummingbird feeders in the winter. Doing so will help ensure that hummingbirds keep their metabolism up while feeding in the cold temperatures. It is possible for a hummingbird to get too cold and die while perched in the winter.
The site unfortunately does not define the phase "a very cold climate." I will have to leave that determination individually up to those of you who live wherever the temperatures are extremely cold. My personal advice would be to use your best personal judgement on the matter, after all no one who is a hummingbird enthusiast would wish to do any harm to these amazing birds.