alt Hacker NewsYes: you can classify a test file by topic with gzip as follows:
gzip -9 sports.txt testfile.txt
gzip -9 politics.txt testfile.txt
gzip -9 business.txt testfile.txt
The test file belongs to the topic with the smallest size *.gz file.
Witten's group at Waikato uni were perhaps the first to work on this.
Also check out the Hutter prize if you are interested in this.
Replies
There are some deep connections between machine learning, compression, and cryptography with information theory as a common thread.
Also, I’ve never seen “ass.” Used to shorten “aside” — I typically use N.B. but perhaps only for important ones.
aka Normalized compression distance (NCD). Its close cousin: Normalized Google distance (NGD) is also super interesting!
https://en.wikipedia.org/wiki/Normalized_compression_distanc...
You might also want the topic files to be compressed against each other to get a baseline matrix and then multiply any results by the inverse, assuming equal priors on the topics.
We used a similar technique for a class project (N decades ago) to test this:
https://en.wikipedia.org/wiki/Baconian_theory_of_Shakespeare...
By looking at mutual information from different authors on the same topic vs same author on different topics. As I recall, it convincingly disproved the hypothesis.
Nitpick: Doing it exactly like this is flawed because you let the compressibility of your references taint the result; what you would prefer is the compressed size of testfile given sports.txt/... as a dictionary without accounting for the compressed size of that, no?
Really interesting approach though.
we were doing this in Qualcomm 20 years ago
I seem to remember it being shown for character recognition via JBIG. Maybe in Managing Gigabytes?
There would be some overlap with business sports analogies - eg team huddle.
Is pigz faster?
[dead]
Back in the day - maybe two decades ago - I implemented language detection like this.
I seeded gzip compressors’ dictionaries with Wikipedia articles in different languages.
I would then try to use said dictionaries on any random text, and the one that was best able to compress it, was the correct language.
Absolutely totally not the best approach, but very fast and super simple to implement.