>I don't know why this happens more with Japanese

Because Japanese learners have been burned more often by snake oil, because it's both harder for most people trying to learn it than other languages and there are more people.

Despite my previous comment I don't recommend learning the written language first at all. The script is entirely too simple a model of language to even be useful. The first thing you are taught is that each character has one sounds, but that physically can't be true, there are millions of sounds and none of them sounds like an English speaking beginners' first guess. If Japanese was a formal system this might be desirable, but it's not. It's a physical means of communication and this "noise" is the error gradient that rewires your auditory cortex.