Experiment № 081 · The story
Only Rearranged
Why I picked it
Most compression I can explain in a sentence: spot something frequent, spend fewer bits on it. The Burrows–Wheeler transform doesn't do that at all. Michael Burrows and David Wheeler published it in 1994 as a Digital Equipment research report, and on its own it compresses precisely nothing — the output is the same length as the input, with the same characters in it and the same count of each. It is a permutation. And yet it is the reason bzip2 beat everything around it for years. I wanted an algorithm whose whole contribution is rearranging, and which is honest about the fact that the rearranging is where all the cleverness lives.
What I enjoy about it
The moment the rows land. Before the sort you're looking at a diagonal staircase — every rotation shifted one place from the one above — and the last column is meaningless confetti. Then the rows slide into alphabetical order and that column falls into blocks. The reason is oddly human: sorting groups rows by what follows, and in real text the same context tends to be preceded by the same letter, so every row starting he almost certainly has a t hanging off its end. The other half I like is undoing it. Turn on keep only the last column and watch the matrix evaporate — and then press undo it anyway. From that single column the whole text walks back out, right to left, one jump per character. All that scaffolding was never stored; it was only ever a way of describing which permutation to apply.
Something you might take away
A reversible rearrangement cannot add information and cannot destroy any — that's what reversible means. So the transform doesn't make the text more compressible in any deep sense; it moves the redundancy that was already there into a form a much dumber algorithm can see. Run-length coding is too stupid to notice that abcabcabc… repeats — it only ever sees a, then a different character, and gives up. After the transform it doesn't have to be clever, because every a in the text has been stacked into one pile in front of it. Paste in random letters and watch the counters refuse to move: there was nothing to expose, so nothing came out. That's the general shape of the thing — most of the value in a pipeline is usually in the preparation, and preparation that looks like it achieved nothing is often the step that did all the work.