A big part of that is the regulatory environment. For example - I have the interest and means to get my PPL. I'd love to do so, but the FAA considers ADHD to be a disqualifying medical condition.
https://www.faa.gov/licenses_certificates/airmen_certificati... implies there may be a path (depending on what "special issuance" means).
The cost is a much bigger factor for most
Ultralight and sport (no medical clearance) can still land on airfields, no?