This is the code I use since 20 years back:
http://move.rupy.se/file/base64.txt
If this part becomes the bottleneck; so I need to write a GPU accelerated version I'll eat my own hat.
Vectorized base64 can reach speeds in the proximity of memory copy.
I've easily reached a point where it was worth it to go AVX route in my "for funsies" game that I'm making.
Your decoder implementation accepts all sorts of non-Base64 strings and decodes them to whatever. Among other things, it also accepts MIME Base64 (with line breaks and whitespace) and base64url strings, but decodes them to the wrong values. This is a classic example of insufficient input validation that can lead to parser mismatch vulnerabilities or worse.