Some thoughts on this.

The Pile was created by high school students. I was impressed how good it was when it came out given the team and how many resources they had.

Including 200k pirated books seems like a problem. It should be removed. If I was an author, I would be concerned by a process that allows this much content to be effectively published with no filter on the internet.

I personally think there is a lower bar for researchers and high school students to clean datasets used for research or class projects given that there is no commercial harm and cleaning huge datasets takes engineering resources that not everyone has.

In this case, it looks like it was removed by https://huggingface.co/collections/common-pile/common-pile-v...

Companies that commercialize the data should have a higher bar because they have more resources. It seems like Anthropic paid $1.5B for not cleaning the data before releasing a model based on it.