It's frustrating when it eats into your safe following distance. The driver merging in ahead of you is being dangerous and not leaving a safe following distance for themselves (or you).
The buffer exists to be used. Allowing people to merge into it makes the lane that they came from safer. Build a new buffer.
When traffic is heavy, everyone is likely following at [what they perceive to be] a minimum following distance.
It's simply not possible to merge during heavy traffic without eating into someone's safe following distance.