logoalt Hacker News

jrmg • yesterday at 1:57 PM • 1 reply • view on HN

Is the fact that the original did [compress base]+[compress seq] rather than [compress [bytes + seq]] not important?

(I honestly don’t know is gzip does something different when presented with two chunks as opposed to one, or, if it does, if bz2 has equivalent behaviour - but the difference in the code did stand out to me, and it does seem related to ‘extending the token sequence’)

Replies

networked • yesterday at 3:48 PM

This difference doesn't matter because of how zlib works. At least by default, zlib divides the input data into its own blocks independent of the caller. If you don't feed it enough data to complete a block, it waits until you feed it more or finish the stream.

We can test it by going back to zlib:

       def length_for(seq: bytes) -> int:
  -        return len(bz2.compress(context + seq, level))
  +        return len(zlib.compress(context + seq, level))
At temperature zero, this outputs the same sample as commit 3734bf6, the most recent commit upstream:

  MENENIUS:
  'Though all at once cannq

  MARCIUS:
  I'll fight
  'Though all at once cannq

  MARCIUannq
  
  MARCIUS:
  I'll fight
  'Though
  
  AUFIDIUS:
  If I fly, Marci
  
  AUFIDIUS:
  If I fly, Marci
  
  AUFID
  
  AUFIDIUS:
  If
  If I fly
I also tried LZMA for good measure:

       def length_for(seq: bytes) -> int:
  -        return len(bz2.compress(context + seq, level))
  +        return len(lzma.compress(context + seq))
The sample at temperature zero:

  MENENIUS:
  'Th
  
  A carbuncle enti
  
  , as big as thou
  
  
  A aa
This is followed by a lot of whitespace.

python-lz4 gives you all newlines after the prompt. I tried debugging it, and the compressed length of different candidate seqs is the same.