Internals: TLS over TLS

The ssl application that comes with Erlang/OTP implements TLS using an interface equivalent to the gen_tcp interface: you get and manipulate a socket. The TLS encoding and decoding is applied transparently to the data sent and received.

In order to have a TLS layer inside another TLS layer we need a way to encode the data of the inner layer before we pass it to the outer layer. We cannot do this with a socket interface. Thankfully, the ssl application comes with options that allow to transform an sslsocket() into an encoder/decoder.

The implementation is however a little convoluted as a result. This chapter aims to give an overview of how it all works under the hood.

gun_tls_proxy

The module gun_tls_proxy implements an intermediary process that sits between the Gun process and the TLS process. It is responsible for routing data from the Gun process to the TLS process, and from the TLS process to the Gun process.

In order to obtain the TLS encoded data the cb_info option is given to the ssl:connect/3 function. This replaces the default TCP outer socket module with our own custom module. Gun uses the gun_tls_proxy_cb module instead. This module will forward all messages to the gun_tls_proxy process.

The resulting operations looks like this:

Gun process <-> gun_tls_proxy <-> sslsocket() <-> gun_tls_proxy <-> "inner socket"

The "inner socket" is the socket for the Gun connection. The gun_tls_proxy process decides where to send or receive the data based on where it's coming from. This is how it knows whether the data has been encoded/decoded or not.

Because the ssl:connect/3 function call is blocking, a temporary process is used while connecting. This is required because the gun_tls_proxy needs to forward data even while performing the TLS handshake, otherwise the ssl:connect/3 call will not complete.

The result of the ssl:connect/3 call is forward to the Gun process, along with the negotiated protocols when the connection was successful.

The gun_tls_proxy_cb module does not actually implement {active,N} as requested by the ssl application. Instead it uses {active,true}.

The behavior of the gun_tls_proxy process will vary depending on whether the TLS over TLS is done connection-wide or only stream-wide.

Connection-wide TLS over TLS

When used for the entire connection, the gun_tls_proxy process will act like a real socket once connected. The only difference is how the connection is performed. As mentioned earlier, the result of the ssl:connect/3 call is sent back to the Gun process.

When doing TLS over TLS the processes will end up looking like this:

Gun process <-> gun_tls_proxy <-> "inner socket"

The details of the interactions between gun_tls_proxy and its associated sslsocket() have been removed in order to better illustrate the concept.

When adding another layer this becomes:

Gun process <-> gun_tls_proxy <-> gun_tls_proxy <-> sslsocket()

This is what is done when only HTTP/1.1 and SOCKS proxies are involved.

Stream-wide TLS over TLS

The same cannot be done for HTTP/2 proxies. This is because the HTTP/2 CONNECT method does not apply to the entire connection, but only to a stream. The proxied data must be wrapped inside a DATA frame. It cannot be sent directly. This is what must be done:

Gun process -> gun_tls_proxy -> Gun process -> "inner socket"

The "inner socket" is the socket for the HTTP/2 connection.

In order to tell Gun to continue processing the data, the handle_continue mechanism is introduced. When gun_tls_proxy has TLS encoded the data it sends it back to the Gun process, wrapped in a handle_continue tuple. This tuple contains enough information to figure out what stream the data belongs to and what should be done next. Gun will therefore route the data to the correct layer and continue sending it.

This solution is also used for receiving data, except in the reverse order.

Routing to the right stream

In order to know where to route the data, the stream_ref had to be modified to contain all the references to the individual streams. So if the tunnel is identified by StreamA and a request on this tunnel is identified by StreamB, then the stream is known as [StreamA, StreamB] to the user. Gun then routes first to StreamA, a tunnel, which continues routing to StreamB.

A problem comes up if an intermediary is a SOCKS server, for example in the following scenario:

HTTP/2 proxy <-> SOCKS proxy <-> HTTP/1.1 origin

The SOCKS protocol doesn't have a concept of stream, therefore when we refer to request to the origin server they are [StreamA, StreamB], not [StreamA, StreamB, StreamC]. This is a problem for routing encoded/decoded TLS data to the SOCKS layer: we don't have a built-in way of referring to the SOCKS layer.

The solution is to have a separate handle_continue_stream_ref value that assigns a reference to the SOCKS layers. Gun will then be able to forward handle_continue message, and only them, to the appropriate layer.

Gun therefore has two different routing avenues, but the mechanism remains the same otherwise.

gun_tunnel

In order to simplify the routing, the gun_tunnel module was introduced. For each intermediary (including the original CONNECT stream at the HTTP/2 layer) there is a gun_tunnel module.

Going back to the example above:

HTTP/2 proxy <-> SOCKS proxy <-> HTTP/1.1 origin

In this case the modules involved to handle the request will be as follow:

gun_http2 <-> gun_tunnel <-> gun_tunnel <-> gun_http

The gun_http2 module doesn't do any routing, it just passes everything to the gun_tunnel module. The gun_tunnel module will then do the routing, which involves removing StreamA from [StreamA, StreamB] where appropriate.

The gun_tunnel module also receives the TLS encoded/decoded data and forwards it appropriately. When it comes to sending data, it will return a send command that allows the previous module to continue sending the data. The gun_http2 module will ultimately wrap the data to be sent in a DATA frame and send it to the "inner socket".