Yes, SVMs and Random Forests still rule the many worlds of classification problems often not in the spotlight.

Perhaps the optimizer is worse but the architecture or objective function is better.

Imagine how good it would be with derivatives

Bayes Optimal Classifier?