The thing that jumps out to me is that it returns “the size of the output data minus one” but if I’m passing in a pointer it’s on me to make sure there’s enough memory allocated isn’t it? So the caller must make sure it’s calculated this correctly - in which case why return the size? Taking the size as an argument could be safer as you’d be checking the caller calculated enough space too.
(Honest questions in this it’s probably been 20 years since I did c)
That part of the API seems particularly random indeed, especially when comparing it to other EVP_{En,De}code* functions which use the returned int for success/error.
The only reason for such a design I could think of is that the caller would be able to "detect" the overflow by comparing the returned value against the length of the buffer. But that is of course pointless, as the compiler is free to assume a buffer overflow cannot happen and thus optimize the check away...