That’s because fedoras are dress hats. They’re deeply associated in our minds with suits. Wearing one with a T-shirt doesn’t make you look like an innovator — it makes you look confusing and a little off-putting to people’s habits of thought. And wearing one all the time tells people that you don’t understand any other headgear options.
It’s not that you have to follow the rules all the time. It’s that you have to understand how the rules work in people’s minds, and what they’ll see if you break them. No one gets to rewrite style overnight. Even celebrities and designers get mocked if they push too far, too fast.
Have some respect for the existing traditions. They’re there for a reason.