logoalt Hacker News

jcgl • today at 12:30 PM • 2 replies • view on HN

I think you're mistaking text-with-structured data for structured data itself.

Because unix shell is irrevocably text-oriented, kludging in something like JSON is basically the best that can be done when you start to want to do structured operations on structured data. (I'm sympathetic to your point about the AWS CLI tools doing JSON by default though--that just sounds like bad design.)

Being text-oriented imposes drastic limits on composability. Because there is no structure, every element of a pipeline needs to do its own parsing of the input data. This leads to brittle pipelines where every element is tightly coupled to its input's textual representation.

As an exercise, try to write a pipeline that sorts podman images by size without removing the column headers[0]:

  $ podman image ls --all 
  REPOSITORY                                 TAG         IMAGE ID      CREATED       SIZE
  docker.io/prom/prometheus                  latest      937690d77350  2 months ago  367 MB
  quay.io/keycloak/keycloak                  latest      da9433c9fac3  2 months ago  466 MB
  registry.fedoraproject.org/fedora-toolbox  43          a32da54355ca  4 months ago  2.19 GB
  docker.io/powerdns/pdns-auth-49            latest      8c1385c9deed  4 months ago  208 MB
  docker.io/testcontainers/ryuk              0.13.0      b75bc7ce94c3  6 months ago  7.21 MB
As far as I can tell, there is no way to do this in a manner that's even remotely composable. Your best bet is to basically do everything from within awk. Whatever the result would be, it certainly won't be pretty!

Contrast that with what you can do in PowerShell. You can write a couple of standalone functions[0] that are readable and composable, resulting in this pipeline:

  podman image ls --all |
      Replace-SpacesWithTabs |
      ConvertFrom-Csv -Delimiter "`t" |
      Sort-Object -Property {Convert-HumanSizeToBytes -Size $_.size} -Descending
[0] Repurposing this from a blog post I wrote: https://www.cgl.sh/blog/posts/sh.html#this-should-be-basic

Replies

alloyed • today at 3:34 PM

the point is well-taken, but i do want to show the bash version just for fun:

    podman image ls --all | sed 's/\s\s\+/\t/g' | tee >(head -n 1) >(tail -n +2 | sort -hrk 5) >/dev/null
this is _still_ all text, and we're relying heavily on sort to do a bunch of internal parsing and be in agreement with podman about how sizes should be formatted. also, for "real world" work, i dunno if the tee trick here has any kind of order guarantees, just that it works fine in this case. I'd probably just end up dropping the header and living with worse output in reality
➕ show 2 replies
unfunco • today at 4:13 PM

    $ podman image ls --all --sort=size
…or was the point more about doing it in a pipeline?
➕ show 1 reply