Baryon and lepton number conservation are what's called "accidental symmetries" in the standard models, meaning there is no real underlying symmetry that would conserve them. In fact many extensions of the Standard Model don't, while still retaining unitarity. The problems already start once you try to calculate any time evolution of anything, because the Hilbert operator not being unitary means that everything breaks. Even the total energy in a closed system might vanish or blow up to infinity. You can't calculate anything under these conditions.