If company like Meta are downloading pirated books etc.. to train their AI, they will surely honor robots.txt.