One additional factor to consider is that in most cases those setting the leisure hours (i.e. employers) are not the same ones enjoying the leisure (i.e. employees). While the leisure/productivity tradeoff applies to an individual, an economically rational employer only really values productivity and will only offer as much leisure time as necessary to attract and retain employees. So while social forces do generally push for additional leisure over time, such as shorter work weeks, it's often challenging for people to find the type of employment situation where they have significant flexibility in trading off income for leisure time.

As an example, I have a pretty good paying, full-time white collar job. It would be much more challenging if not impossible to find an equivalent job making half as much working 20 hours a week. Of course I could probably find some way to apply the same skills half-time as a consultant or whatever, but that comes with a lot of tradeoffs besides income reduction and is less readily available to a lot of people.

Maybe the real exception here is at the top of the economic ladder, although at that point the mechanism is slightly different. Billionaires have pretty infinite flexibility on leisure time because their income is almost entirely disconnected from the amount of "labor" they put in.