For me it was a key DB file inside the Photo library which Time Machine omitted from all backups and prevented me from restoring the library. Not fun.