Using the same towel for both face and body will increase the chance of skin damage like acne, irritation, rashes, etc. by transferring oils, bacteria and other cosmetic products like moisturizers from your body to face.