if the input is larger than ~1.5GB.
...on a system where ints are 32 bits. The bottom line is that this doesn't appear to be easily exploitable and IMHO it should be the callers of EVP_EncodeBlock which need to be inspected to determine the actual risk.
using a nontrivial formula
It's base64. 4/3 expansion; round it up and add padding. What's "nontrivial" about that?
and the caller does not pass in the output buffer length, so the implementation cannot validate it
Not necessary, because the caller is responsible for allocation.
> It's base64. 4/3 expansion; round it up and add padding. What's "nontrivial" about that?
If you mean ceil(inlen * 4/3) + 1 (even if you use exact arithmetic), then you are passing too small a buffer and you will overflow. Maybe your “add padding” part fixes it? In any case, I think that the fact that this confusion is even possible proves my point.
You should not need to read the manpage of a function you are calling in order to determine whether you have passed a large enough buffer to avoid overflow.
Reading the manpage to get the function to not return an error would be a different story.
The question can be reduce to "who do the length validation".
In my case, if i write a public-facing API, i never trust the caller will do the correct thing, thus the caller must provide all the critical parameters, and i must validate all of them.
I am confused why you make out 32 bit int as being rare. Almost all devices that you'd reasonably connect to the internet at this point have int as 32 bits