The main thing is the old forums were sorted by recency (how new a post was or how recently it was replied to) rather than some AI-driven engagement mechanism. They were structured (you'd have several rooms for different topics on the same forum).
This probably isn't totally untrue, but, for example, 4chan is sorted by recency and has a lot of the same problems as other large sites online, which to me says that it can't just be complex recommendation systems driving problems.
I'm not aware of any "AI-driven engagement mechanism" for subreddits. You can sort by new, top, hot, best. Hot/Best are opaque heuristics, but they function reasonably and you aren't forced to use them. And for most communities, tags function as adequate topic groupings.
You're rewarded for participation with fake nonsense points, same as all the forums of yore.