Getting Started With HTTP/2

The web is built on small HTTP exchanges. HTTP runs on TCP, which is aimed at long running transfers — it is not cut out for small exchanges that are common with HTTP.  There is a lot of overhead caused by TCP in a modern website where only a fraction of time is spent downloading content. HTTP/2 addresses these performance problems. In this article, we will review how HTTP/2 works under the hood and introduce tools to get started with HTTP/2.

You may also like: TLS/SSL Explained: TLS/SSL Terminology and Basics.

TCP

The Internet consists of five layers of TCP/IP. All applications run on the Transport layer, which, for HTTP, is TCP.

The first thing that takes place when establishing a TCP connection is what is known as Three-Way handshake.

Every time a new HTTP request is made, the TCP Three-Way handshake adds this overhead before HTTP Request can be made. Every TCP connection is then closed with a Three-Way handshake too. This adds latency in order of tens of milliseconds for every HTTP request.  The HTTP Header Connection: Keep-Alive introduced in HTTP/1.1 solves this problem to some extent.

TCP is a reliable protocol — every packet sent by the client is acknowledged. This relieves applications from the complex problems of loosing packets or receiving them out-of-order. This, however, introduces another problem — Head-of-line blocking in addition to increasing the overhead. 

TCP deals with network congestion using flow control. TCP slow start is a flow control mechanism where it gradually increases the TCP window size until it finds the network's maximum carrying capacity. The TCP slow start adds to the latency in HTTP requests and responses.

HTTP/2

HTTP/2 addresses the shortcomings of TCP. It is a binary protocol that keeps the same semantics as HTTP 1.x, meaning that all the headers, verbs, resources, etc. work in the same way. It is more about solving issues with TCP than fixing problems with HTTP 1.x.

HTTP/2 Connection

There are two ways to make an HTTP/2 connection

1. Establishing an h2c connection. HTTP/2 can work over HTTP connection over plain text.

2. Establishing an h2 connection. Predominant way of using HTTP/2 is to establish connection over HTTPS.

Frames constitute the basic protocol unit in HTTP/2 and are used to communicate data — request, reponse, header, or body.

The header frames are compressed using HPACK.

Streams

Streams represent a logical single request/response communication. A stream is just a bidirectional sequence of frames that share a common identifier. This enables interleaving of frames from multiple streams together that allows for multiplexed communication over a single connection.

HTTP/2 uses single TCP connection per host. It provides bi-directional multiplexing via streams allowing for concurrent requests and responses. Streams can be prioritized as well as subjected to flow control.

Server Push

Server push is a function available in HTTP/2 whereby a server can push resources to the client without having client request them. It still provides clients with control whether it wants to accept the data or not.  Although, server push eliminates the need to inline resources, e.g., JavaScript, CSS, it does not replace other technologies, e.g., WebSockets.

Tools

We will use tools that support HTTP/2 to investigate HTTP/2 connections.

1. cURL. HTTP/2 support was added in version 7.33+. It uses nghttp2 library under the covers. The command line parameter --http2 can be used to make HTTP/2 requests, e.g., curl --http2 domain.com. Alternatively, a Docker image can be used for cURL that supports HTTP/2.

 docker run -t --rm badouralix/curl-http2 \
--http2-prior-knowledge -i -k \
    https://http2.golang.org

 2. Chrome Tools. Chrome provides protocol information under the Network tab.

Chrome also provides chrome://net-export  to save netlogs that can be used to view HTTP/2 sessions using external netlog viewer — https://netlog-viewer.appspot.com.

3. Wireshark. Wireshark can be used to view HTTP/2 exchanges, as it supports decrypting of SSL. To do so, we need to export the environment variable SSHKEYLOGFILE from the browser and configure Wireshark to use this sslkeylog file.

HTTP/2 in Action

We will use tools to do exercises that will help us understand HTTP/2 better.

1. HTTP/2 demo. Let us start by checking if our browser supports HTTP/2. Open the Akamai website in your Chrome browser. Check out the demo that shows how HTTP/2 improves performance and load time.

2. HTTP/2 Connection. Use cURL to connect to an HTTP/2 server serving plain text. The client first connects to the server over HTTP/1.1 and sends an upgrade request. The server responds with 101 Switching Protocols, and then the connection upgrades to HTTP/2.

> curl -vs -o /dev/null --http2 http://nghttp2.org
*   Trying 139.162.123.134...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x66aba0)
* Connected to nghttp2.org (139.162.123.134) port 80 (#0)
> GET / HTTP/1.1
> Host: nghttp2.org
> User-Agent: curl/7.64.0
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
>
< HTTP/1.1 101 Switching Protocols
< Connection: Upgrade
< Upgrade: h2c
* Received 101
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=33
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< date: Sat, 05 Oct 2019 04:59:34 GMT
< content-type: text/html
< last-modified: Mon, 19 Aug 2019 13:20:36 GMT
< etag: "5d5aa224-19d8"
< accept-ranges: bytes
< content-length: 6616
< x-backend-header-rtt: 0.001353
< server: nghttpx
< via: 2 nghttpx
< alt-svc: h3-23=":4433"; ma=3600
< x-frame-options: SAMEORIGIN
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
<
{ [6616 bytes data]
* Failed writing body (0 != 6616)
* stopped the pause stream!
* Connection #0 to host nghttp2.org left intact

Next, let's connect to an HTTP/2 server over TLS. To decrypt TLS traffic in Wireshark, set the environment variable 

set SSHKEYLOGFILE=%USERPROFILE%\tools\ssl\keylog.log

or, launch Chrome Canary with flag  --ssl-key-log-file.

>"AppData\Local\Google\Chrome SxS\Application\chrome.exe" \
--ssl-key-log-file=%USERPROFILE%\tools\ssl\keylog.log

To launch Wireshark, go to Edit->Preferences->Protocols. Select TLS, and set the logfile name to the same file as set in the environment variable SSHKEYLOGFILE. With this set, we'll be able to decrypt TLS traffic. Start traffic capture in Wireshark, and in your Chrome browser, open https://http2.golang.org/. 

3. HTTP/2 Stream Exchange.  Go to https://http2.golang.org/gophertiles in Chrome browser and see the HTTP/2 Streams in Wireshark. You can also use Chrome Developer Tools to see the HTTP/2 exchanges under Network tab.

4. HTTP/2 Priorities. nghttp2 not only provides a library that cURL uses, but it also provides an HTTP/2 server that implements HTTP/2 priorities based on content-type — it first serves HTML, followed by CSS, and then JavaScript. It is available as a Docker image.

>docker run --rm -it svagi/nghttp2 nghttpd
Usage: nghttpd [OPTION]... <PORT> [<PRIVATE_KEY> <CERT>]
HTTP/2 server
Too few arguments

The nghttp client can be used to connect to an HTTP/2 server.

>docker run --rm -ti svagi/nghttp2 nghttp -nv https://nghttp2.org
[  0.604] Connected
The negotiated protocol: h2
[  1.326] recv SETTINGS frame <length=24, flags=0x00, stream_id=0>
          (niv=4)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):1048576]
          [UNKNOWN(0x08):1]
          [SETTINGS_HEADER_TABLE_SIZE(0x01):8192]
[  1.326] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
          (niv=2)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[  1.326] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  1.326] send PRIORITY frame <length=5, flags=0x00, stream_id=3>
          (dep_stream_id=0, weight=201, exclusive=0)
[  1.326] send PRIORITY frame <length=5, flags=0x00, stream_id=5>
          (dep_stream_id=0, weight=101, exclusive=0)
[  1.326] send PRIORITY frame <length=5, flags=0x00, stream_id=7>
          (dep_stream_id=0, weight=1, exclusive=0)
[  1.326] send PRIORITY frame <length=5, flags=0x00, stream_id=9>
          (dep_stream_id=7, weight=1, exclusive=0)
[  1.326] send PRIORITY frame <length=5, flags=0x00, stream_id=11>
          (dep_stream_id=3, weight=1, exclusive=0)
[  1.326] send HEADERS frame <length=39, flags=0x25, stream_id=13>
          ; END_STREAM | END_HEADERS | PRIORITY
          (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
          ; Open new stream
          :method: GET
          :path: /
          :scheme: https
          :authority: nghttp2.org
          accept: */*
          accept-encoding: gzip, deflate
          user-agent: nghttp2/1.12.0
[  1.629] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  1.629] recv (stream_id=13) :method: GET
[  1.629] recv (stream_id=13) :scheme: https
[  1.629] recv (stream_id=13) :path: /stylesheets/screen.css
[  1.629] recv (stream_id=13) :authority: nghttp2.org
[  1.629] recv (stream_id=13) accept-encoding: gzip, deflate
[  1.629] recv (stream_id=13) user-agent: nghttp2/1.12.0
[  1.629] recv PUSH_PROMISE frame <length=47, flags=0x04, stream_id=13>
          ; END_HEADERS
          (padlen=0, promised_stream_id=2)
[  1.683] recv (stream_id=13) :status: 200
[  1.683] recv (stream_id=13) date: Sat, 05 Oct 2019 11:12:30 GMT
[  1.683] recv (stream_id=13) content-type: text/html
[  1.683] recv (stream_id=13) last-modified: Mon, 19 Aug 2019 13:20:36 GMT
[  1.683] recv (stream_id=13) etag: "5d5aa224-19d8"
[  1.683] recv (stream_id=13) accept-ranges: bytes
[  1.683] recv (stream_id=13) content-length: 6616
[  1.684] recv (stream_id=13) x-backend-header-rtt: 0.001768
[  1.684] recv (stream_id=13) strict-transport-security: max-age=31536000
[  1.684] recv (stream_id=13) server: nghttpx
[  1.684] recv (stream_id=13) via: 2 nghttpx
[  1.684] recv (stream_id=13) alt-svc: h3-23=":4433"; ma=3600
[  1.684] recv (stream_id=13) x-frame-options: SAMEORIGIN
[  1.684] recv (stream_id=13) x-xss-protection: 1; mode=block
[  1.684] recv (stream_id=13) x-content-type-options: nosniff
[  1.684] recv HEADERS frame <length=238, flags=0x04, stream_id=13>
          ; END_HEADERS
          (padlen=0)
          ; First response header
[  1.684] recv (stream_id=2) :status: 200
[  1.684] recv (stream_id=2) date: Sat, 05 Oct 2019 11:12:30 GMT
[  1.684] recv (stream_id=2) content-type: text/css
[  1.684] recv (stream_id=2) last-modified: Mon, 19 Aug 2019 13:20:36 GMT
[  1.684] recv (stream_id=2) etag: "5d5aa224-98aa"
[  1.684] recv (stream_id=2) accept-ranges: bytes
[  1.684] recv (stream_id=2) content-length: 39082
[  1.684] recv (stream_id=2) x-backend-header-rtt: 0.001364
[  1.684] recv (stream_id=2) strict-transport-security: max-age=31536000
[  1.684] recv (stream_id=2) server: nghttpx
[  1.684] recv (stream_id=2) via: 2 nghttpx
[  1.684] recv (stream_id=2) alt-svc: h3-23=":4433"; ma=3600
[  1.684] recv (stream_id=2) x-frame-options: SAMEORIGIN
[  1.684] recv (stream_id=2) x-xss-protection: 1; mode=block
[  1.685] recv (stream_id=2) x-content-type-options: nosniff
[  1.685] recv (stream_id=2) x-http2-push: 1
[  1.685] recv HEADERS frame <length=63, flags=0x04, stream_id=2>
          ; END_HEADERS
          (padlen=0)
          ; First push response header
[  1.685] recv DATA frame <length=6616, flags=0x01, stream_id=13>
          ; END_STREAM
[  1.685] recv DATA frame <length=9431, flags=0x00, stream_id=2>
[  1.972] recv DATA frame <length=15732, flags=0x00, stream_id=2>
[  2.259] recv DATA frame <length=13919, flags=0x01, stream_id=2>
          ; END_STREAM
[  2.259] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
          (last_stream_id=2, error_code=NO_ERROR(0x00), opaque_data(0)=[])

We'll create a simple application consisting of HTML, CSS and JavaScript and will have nghttpd server, which is a HTTP/2 server, serve it over TLS.  Create folders — certs and docs. The certs folder will store private key and certificate; docs will store the application.

Generate a self-signed certificate.

openssl req -newkey rsa:2048 -nodes -keyout certs/nghttpd.key \
-x509 -days 365 -out certs/nghttpd.crt

Create the application.

<!DOCTYPE html>
<html>
<head>
  <title>HTTP/2</title>
<link rel="stylesheet" type="text/css" href="style.css" />
</head>
<body>
<h1>I am a headline made with HTML</h1>
<p>And I am a simple text paragraph. The color of this text is styled with CSS. 
      Click the button below to remove me through the power JavaScript.</p>
<button>Toggle the text above</button>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js"></script>
<script src="script.js"></script>
</body>
</html>

body {
font-family: sans-serif;
text-align: center;
padding: 3rem;
font-size: 1.125rem;
line-height: 1.5;
transition: all 725ms ease-in-out;
}

h1 {
font-size: 2rem;
font-weight: bolder;
margin-bottom: 1rem;
}

p {
margin-bottom: 1rem;
color: tomato;
}

button {
cursor: pointer;
appearance: none;
border-radius: 4px;
font-size: 1.25rem;
padding: 0.75rem 1rem;
border: 1px solid navy;
background-color: dodgerblue;
color: white;
}

$('button').on('click', function() {
$('p').toggle();
});

Now launch nghttpd.

>docker run --rm -it -p 443:443 -v D:\nghttpd\certs\nghttpd.crt:/ssl/nghttpd.crt \
-v D:\nghttpd\certs\nghttpd.key:/ssl/nghttpd.key \
    -v D:\nghttpd\docs:/docs svagi/nghttp2 nghttpd 443 \
    /ssl/nghttpd.key /ssl/nghttpd.crt -d /docs

Open the browser and hit https://127.0.0.1/index.html. Usechrome://net-export  to save netlogs and then view the netlog file using netlog viewer — https://netlog-viewer.appspot.com. We can see that HTML is served first, followed by CSS, and then JavaScript — the priority determined by the nghttpd server.

5. Server Push. We can configure server to push a resource when it gets request for another resource. Let us push CSS file when we get request for HTML.

Start nghttpd server with -p  option to indicate that it should push style.css when a request is made for index.html file. Export the netlog file and view using netlog viewer to see the session showing PUSH details.

>docker run --rm -it -p 443:443 -v D:\nghttpd\certs\nghttpd.crt:/ssl/nghttpd.crt \
-v D:\nghttpd\certs\nghttpd.key:/ssl/nghttpd.key \
    -v D:\nghttpd\docs:/docs \
    svagi/nghttp2 nghttpd 443 /ssl/nghttpd.key /ssl/nghttpd.crt \
    -d /docs -p/index.html=/style.css

 6. Working with HTTP/2 Library. Java library okhttp can be used to connect to an HTTP/2 server. Code illustrating HTTP/2 Request using okhttp can be downloaded from my GitHub. Remember to  pass-Xbootclasspath/p:alpn-boot-8.1.13.v20181017.jar  as VM Option in IntelliJ Run Configuration if you are using Java version 8. Java 9 and above has native ALPN support and this VM option is not supported. 

Conclusion

In this article, we looked at how HTTP/2 addresses some of the performance issues seen in modern web applications. We introduced few tools to get up to speed with HTTP/2.  There are many resources on the web to learn more. 

Further Reading

• Introduction to HTTP.