If I remember correctly, the whisper documentation actually recommends to trim non-speech portions as the models halucinate heavily during those portions.