If you can do inference on the CPU, drop the GPU : it should be faster.