There is no proprietary blobs in the Signal Server source code.
The build of Signal Server from public source does not mean that's what they deploy in production.
For security analysis the Signal Server is immaterial, all claims of security can be verified from the client. For all intents and purposes the server is just a dumb pipe.