Having built something similar with CLIP on an M1, frame sampling rate is the whole ballgame. One frame a second on 12k videos is days, keyframes only got me to an overnight run.
Have you tried scene detection? I would guess camera cuts are even less frequent than keyframes.
> 12k videos ?
you are pirating first-release movies for commercial purposes?
[flagged]
Maybe you need a minimal downscale version as well, I heard is very common technique in the video editing world.
Based on my experience, sampling rate can be tricky if what you are looking for lasted less than interval period.