I still can't even tell how it's such a big deal. Everyone should be able to handle wearing a small piece of cloth over their faces.

Some people are easily distracted by an article of clothing they never had to wear before. Some people are convinced that the mask is unnecessary and are "rebelling" against pressure. Some people think that the masks have political/societal implication and symbolism that they don't want none of.
Note that all of these mindsets don't seem to prioritize the main purpose of the masks, which is tied directly to the wellbeing of themselves and their families.