It actually is. Although I don't have a link for it, there's the theory that too much handwashing dries out the skin, creating cracks and more places for germs to hang out, and instead frequent rinsing/drying/moisturizing is the way to go.

Door handles anywhere: the worst.

As for men, if we blithely will put our dicks in someone else's mouth, is it compulsory to wash after merely touching it? I do, but that's because I handle food, and being seen washing my hands is not only being professional but good for my cred.