Intro to Compression

Compression can be used to reduce the size of data. This can be particularly important when transmitting data, as a reduced size enables communication over connections of limited strength and speeds up the transmission.

HTTP Compression

Compression can be applied at the application layer level. In a web context, this means applying compression at the HTTP level. More specifically, HTTP requests can be compressed by the webserver. This is indicated by the Content-Encoding HTTP header. This header can be set to the values gzip, compress, or deflate to inform the web browser what kind of compression method was used to compress the data. The web browser is then able to unpack the compressed data and display the web page correctly.

If compression is applied at the HTTP level, the compressed response looks similar to this:

Since the compression is applied only to the HTTP body, all headers are transmitted uncompressed and in their original state.

TLS Compression

Instead of applying compression at the application layer level, it can also be applied at the TLS level. This means that not only the application layer payload but all application layer data is compressed. In a web context, this means that the whole response is compressed, including all HTTP headers.

Since the compression is applied at the TLS level, it is completely transparent to any web server or web proxy such that we cannot detect it in Burp. However, whether TLS compression is used or not is negotiated in the TLS handshake.

We can see the compression methods supported by the client in the ClientHello message in the Compression Methods Field:

The compression method is then chosen by the server in the ServerHello message:

Example: LZ77

As an example of how compression works, let's look at the LZ77 algorithm. LZ77 works by keeping a dictionary of recently encountered character strings and replacing repeatedly encountered sequences with a reference to the first occurrence. As a simplified example, consider the following sentence: I like HackTheBox's HackTheBox Academy. This would be compressed as I like HackTheBox's <13,10> Academy. We can see that the second occurrence of HackTheBox has been replaced with a back reference of two numbers, the back pointer and the length. To unpack the original sentence, we follow the back pointer by moving backward 13 characters and replacing the reference with the following 10 characters, resulting in the word HackTheBox.

It is important to understand that LZ77 uses a sliding window, so it only considers a recent history of words for compression and does not operate on the text as a whole. This is important for the upcoming compression attacks.