Yeah, this was my thought (as a layman) as well.

It seems some level of interpretability, even if it requires tooling, could result in the ability to spin out synthetic training data sets from other models in an automated fashion.