Is Nix really appropriate for building a complete rootfs? Could I start from scratch and write a nix build script or whatever and end up with a rootfs I could run on some aarch64 SBC or SoM?
And how is the Nix story for going from a rootfs to a bootable image? On e.g Rockchip you need to make an image where there's some firmware blob in one are of the flash which jumps to a uBoot in another area which loads the kernel/initramfs/devicetree, how does that look in Nix land?
How does it handle building vendor kernels or carrying kernel patches instead of mainline?
Yes, most of this is all supported. There is a existing recipe for building an image that you just dd to and SD card and can boot in a Raspberry Pi (and other devices supported by mainline kernel + u-boot).
Kernel is a package just like another; so you just write your own recipe. Patching kernel recipe works just like patching any Nix recipe.
Not sure about the rockchip-proprietary boot parts; you may be on your own.
I don't really like Nix but it's probably fine for this - you have files that come in, a build step, and output files. Your derivations (build recipes) are functions that can take arguments to e.g. provide a set of patches to apply. There are recipes for building a kernel in nixpkgs already which you could fork.
The main annoyance I'd have in your case is probably the number of file copies of large images it may do to keep things isolated, but most hermetic build systems will do something similar.