That's not surprising. At the speed DDR4 runs, PCB design has to account for all sorts of weird effects. Adding the additional traces on the adapter probably pushes all sorts of timing over acceptable thresholds. There's a reason one of the big textbooks in the area is subtitled "A Handbook of Black Magic". [1]

[1] https://www.amazon.com/High-Speed-Digital-Design-Handbook/dp...