In the US, "ginger" is a seasoning, not a color. Like, in general; we do not refer to anything as being ginger-colored unless it's a sort of pale yellowish tan.
People will still refer to redheads as "ginger" but only some of the time
In UK English ginger isn't exactly a color either. It's not like yellow or orange. It's a state of being.
I don't know why this is being downvoted. Does "ginger-colored" sound right to you? It doesn't quite work.
If you would then please explain the "kick a ginger day" from SouthPark
The redhead, ginger, color link comes from early season ginger, which has reddish pink tips around the leaves right above the fresh root.