Unicode scalars are not so fast. But yes working directly with UInt8/Data bytes is efficient.
That’s how I took over maintenance of SwiftSoup and made it over 10x faster according to my benchmarks. Besides various other optimizations such as reducing copying, adding indexes, etc.