This is because big corporations supporting Brexit figured out it will be better for their bottom line if they could source labour from wider pool and have it tied to visa. Something EU workers would never be comfortable with. Hence you had the so called Boriswave - an influx of workers paid below market rates supporting big corporations able to navigate Home Office corrupt system. Conservative party never told the public what it was really about - bringing in very much slave workforce to exploit - at the expense of working class and SMEs.

By the looks of it, Conservative party will never recover from this betrayal and soon followed by Labour who decided to maintain the status quo.