Cloth wearing seems so natural that everyone is self-deemed knowledgeable and has some expert opinions about it. However, to clearly explain the physics involved, and hence to make predictions for clothing design or selection, it turns out to be quite challenging even for experts. Cloth is a multiphased, porous, and anisotropic material system and usually in multilayers. The human body acts as an internal heat source in a clothing situation, thus forming a temperature gradient between body and ambient. But unlike ordinary engineering heat transfer problems, the sign of this gradient often changes as the ambient temperature varies. The human body also perspires and the sweat evaporates, an effective body cooling process via phase change. To bring all the variables into analysis quickly escalates into a formidable task. This work attempts to unravel the problem from a physics perspective, focusing on a few rarely noticed yet critically important mechanisms involved so as to offer a clearer and more accurate depiction of the principles in clothing thermal comfort.