Articles of clothing like shirts and pants cannot (or at least should not) be removed during the day, nor do most people remove them to eat, drink, talk, etc. A mask covers parts of the body (nose and mouth) that are not typically covered because of the functions they are designed to perform. Unlike masks, which are required only where social distancing cannot be maintained, our culture has historically frowned upon appearing in public without pants.
Of course masks can be mandated by governments or businesses, but those entities may be reluctant to do so if they do not believe the requirements can be adequately enforced for the reasons stated above. It is much easier and more likely for a person to pull down their mask for a moment or two, thereby eliminating its usefulness, than it is for a person to remove their shirt or pants.