alt Hacker NewsGo is portable, until it isn't
Comments
I ran into this issue when porting term.everything[0] from typescript to go. I had some c library dependencies that I did need to link, so I had to use cgo. My solution was to do the build process on alpine linux[1] and use static linking[2]. This way it statically links musl libc, which is much friendlier with static linking than glibc. Now, I have a static binary that runs in alpine, Debian, and even bare containers.
Since I have made the change, I have not had anyone open any issues saying they had problems running it on their machines. (Unlike when I was using AppImages, which caused much more trouble than I expected)
[0] https://github.com/mmulet/term.everything look at distribute.sh and the makefile to see how I did it.
[1]in a podman or docker container
[2] -ldflags '-extldflags "-static"'
Was there not a third option: Calling the journalctl CLI as a child process and consume the parsed logs from the standard output? This might have avoided both the requirement to use CGO and also to write a custom parser. But I guess I am missing something.
Once you use CGO, portability is gone. Your binary is no longer staticly compiled.
This can happen subtley without you knowing it. If you use a function in the standard library that happens to call into a CGO function, you are no longer static.
This happens with things like os.UserHomeDir or some networking things like DNS lookups.
You can "force" go to do static compiling by disabling CGO, but that means you can't use _any_ CGO. Which may not work if you require it for certain things like sqlite.
Hashicorp's Vault go binary is a whopping 512Mb beast. Recently considered using its agent mode to grab secrets for applications in containers but the size of the layer it adds is unviably big. And they don't seem interested into making a split server/client binary either...
I think this is true for nearly all compiled languages. I had the same fun with rust and openSSL and glibC. OP didn’t mentioned the fun with glib-c when compiling on a fairly recent distro and trying it to run on an older one. There is the “many Linux” project which provides docker images with a minimum glib c version installed so it’s compatible with newer ones. The switch to a newer open ssl version on Debian/Ubuntu created some issues for my tool. I replaced it with rust tls to remove the dynamic linked library. I prefer complete statically linked binaries though. But that is really hard to do and damn near impossible on Apple systems.
I tink the title is a bit misleading. This is about very low level metrics collection from the system which by definition is very system dependent. The term “portable” in a programming language usually means portability for applications but this more portability of utilities.
Expecting a portable house and a portable speaker to have the same definition of portable is unfair.
FWIW I maintain an official implementation of the journal wire format in go now.
https://github.com/systemd/slog-journal so you can at least log to the journal now without CGO
But that's just the journal Wire format which is a lot simpler than the disk format.
I think a journal disk format parser in go would be a neat addition
And a set of people rediscovered why cross compiling only works up to certain extent, regardless of the marketing on the tin.
The point one needs to touch APIs that only exists on the target system, the fun starts, regardless of the programming language.
Go, Zig, whatever.
If you really need a portable binary that uses shared libraries I would recommend building it with nix, you get all the dependencies including dynamic linker and glibc.
You hit this real quick when trying to build container images from the scratch. Theoretically you can drop a Go binary into a blank rootfs and it will run. This works most of the time, but anything that depends on Go's Postgres client requires libpq which requires libc. Queue EFILE runtime errors after running the container.
Cross-compiling doesn't work because you're not defining your dependencies correctly and relying on the existence of things like system libraries and libc. Use `zig cc` with Go which will let you compile against a stub Glibc, or go all the way and use a hermetic build system (you should do this always anyhow).
Use dlopen? I haven’t tried this in Go, but if you want a binary that optionally includes features from an external library, you want to use dlopen to load it.
Interesting that it uses the C API to collect journals. I would’ve thought to just invoke journalctl CLI. On platforms like macOS where the CLI doesn’t exist it’s an error when you exec, not a build time error.
Go was never truly portable on Linux unfortunately due to its dependency on libc for DNS and user name resolution (because of PAM and other C-only API). Sure, pure Go implementation exists, but it doesn't cover all cases, so, in order to build a "good" binary for Linux you still needed to build the binary on (oldest supported) Linux distro.
If your production doesn't have any weird PAM or DNS then you can indeed just cross-compile everything and it works
I’ve had some success using Zig for cross compiling when CGO is required.
so like every other language
This stuff is out of my frame of reference. I've never used Go before and have never had the need to go this low level (C APIs, etc); so please keep this in mind with my following questions, which are likely to sound stupid or ignorant.
Can this binary not include compiled dependacies along side it? I'm thinking like how on windows for portable apps they include the DLLs and other dependant exes in subfolders?
Out of interest, and in relation to a less well liked Google technology, could dart produce what they are after? My understanding is dart can produce static binaries, though I'm not sure if these are truly portable compile once run everywhere sense.
i wonder, for their use case, why not just submit journal in binary format to the server and let the serve do the parsing?
This seems to imply that Go's binaries are otherwise compatible with multiple platforms like amd64 and arm64, other than the issue with linking dynamic libraries.
I suspect that's not true either even if it might be technically possible to achieve it through some trickery (and why not risc-v, and other architectures too?).
There's no such thing as a portable application; only programs limited enough to be lucky not to conflict with the vagaries of different systems.
That said, in my personal experience, the most portable programs tend to be written in either Perl or Shell. The former has a crap-ton of portability documentation and design influence, and the latter is designed to work from 40 year old machines up to today's. You can learn a lot by studying old things.
This is an (organizational) tooling problem, not a language problem - and is no less complicated when musl libc enters the discussion.
Well, that was pretty obvious that the portability is gone, especially when you start linking into systemd, even on the host system you have to link with the shared libs into systemd, you cannot link statically.
Go is portable until you have to deploy on AS/400
Well now you've gone and linked to a fascinating tool which I'm going to have to dive into and learn: https://kaitai.io/
Thanks.
Cgo is terrible, but if you just want some simple C calls from a library, you can use https://github.com/ebitengine/purego to generate the bindings.
It is a bit cursed, but works pretty well. I'm using it in my hardware-backed KMIP server to interface with PKCS11.
From the article:
> In the observability world, if you're building an agent for metrics and logs, you're probably writing it in Go.
I'm pretty unconvinced that this is the case unless you happen to be on the CNCF train. Personally I'd write in Rust these days, C used to be very common too.
> We did not want to spend time maintaining a backward compatible parser or doing code archaeology. So this option was discarded.
Considering all of the effort and hoop-jumping involved in the route that was chosen, perhaps this decision might be worth revisiting.
In hindsight, maintaining a parser might be easier and more maintainable when compared to the current problems that were overcome and the future problems that will arise if/when the systemd libraries decide to change their C API interfaces.
One benefit of a freestanding parser is that it could be made into a reusable library that others can use and help maintain.