When the user base so used to getting content free, it's difficult to find ways to then charge for the same thing. When that happens they naturally conclude they're being ripped off. A solution in that situation might have been modders opting to lock content at a given version and then offer an updated version at cost. That way Users would have the option to continue using what they had access to (same game-play, same feature), or paying to update. It appears this didn't happen though and modders chose to just offer what they already had at cost (don't know if Steam allows for pseudo-duplicates like that).
The broader issue with the system revolves around Copyright and using other peoples content, it brings a whole massive set of complications to the table; when things are free it doesn't matter; when money is involved, it does. For example, if a popular mod earns significant revenue and has a
DMCA Take Down Notice issued against it, Valve is obligated to temporarily remove from, or deny access to, the infringing content in Store
AND the purchasers computer(s). If the Notice isn't challenged within 10 days, it implies the content is 'bad' and Valve is then further obligated to issue refunds to all those whom purchasing the item
AND find a way to deny access to, or remove the content, permanently from computers.
Furthermore, if the mod that's taken down is used by other mods, it results in a whole mess where people that might have paid for "X" mod now can't use it because "Y" is no longer available. Users would blame Valve for that, demand refunds, and vocally blame the company when that does not happened - Valve is not actually obligated in those instances to issue refunds because the onus of mod "X's" operability would be on Users doing their due-diligence before purchase, or accepting their own liability for acknowledging the mod may not work (this is of course notwithstanding the author of the mod that was taken down.. are
they then obligated and liable to refund those users now with broken products).
Paid Mods would work, after all it's no different than other forms of purchasable User Generated Content, but not in an environment where the marketplace is literally a free-for-all full of disproportionate cross-contamination.
The only other alternative is for content authors to sell their wares directly from their own websites, but that brings with it its own set of problem, mainly administrative and management.