Introduction to HTTP

By Mick Knutson

The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web.

Hypertext is a multi-linear set of objects, building a network by using logical links (the so-called hyperlinks) between the nodes (e.g. text or words). HTTP is the protocol to exchange or transfer hypertext.

RFC 2616 Hypertext Transfer Protocol: http://www.w3.org/Protocols/rfc2616/rfc2616.html

Uniform resource identifier (URI) is a string of characters used to identify a name or a resource.

RFC 1630: Universal Resource Identifiers (URI): http://tools.ietf.org/html/rfc1630

Uniform resource name (URN) is a uniform resource identifier (URI) that uses the urn scheme and does not imply availability of the identified resource. Both URN's (names) and URL's (locators) are URI's, and a particular URI may be a name and a locator at the same time.

BASIC SYNTAX:

EXAMPLE:

urn:isbn:9781849683166

The URN for 'Java EE6 Cookbook for Securing, Tuning and Extending Enterprise applications.'

RFC 1737: Uniform Resource Names (URN): http://tools.ietf.org/html/rfc1737

Uniform resource locator (URL) is a specific character string that constitutes a reference to an Internet resource.

BASIC SYNTAX:

EXAMPLE:

http://baselogic.com:80/blog/?param1=value&param2=value#anchor

RFC 1808: Relative Uniform Resource Locators (URL): http://tools.ietf.org/html/rfc1808

Typical Schemes

QUERY STRING ENCODING USES THE FOLLOWING RULES:

• Letters (A-Z and a-z), numbers (0-9) and the characters '.','-','~' and '_' are left as-is
• SPACE is encoded as '+' or hexadecimal (%20)
• All other characters are encoded as %FF hexadecimal representation with any non-ASCII characters first encoded as UTF-8 (or other specified encoding)
• The typical maximum value for a single GET parameter value is 512 bytes

Request Methods

HTTP defines methods (sometimes referred to as "verbs") to indicate the desired action to be performed on the identified resource. What this resource represents, whether pre-existing data or data that is generated dynamically, depends on the implementation of the server. Often, the resource corresponds to a file or the output of an executable residing on the server.

The HTTP/1.0 specification: section 8 defined the GET, POST and HEAD methods and the HTTP/1.1 specification: section 9 added 5 new methods: OPTIONS, PUT, DELETE, TRACE and CONNECT. By being specified in these documents their semantics are known and can be depended upon. Any client can use any method that they want and the server can choose to support any method it wants. If a method is unknown to an intermediate it will be treated as an un-safe and non-idempotent method. There is no limit to the number of methods that can be defined and this allows for future methods to be specified without breaking existing infrastructure. For example WebDAV (RFC5789) defined 7 new methods and RFC5789 specified the PATCH method.

RFC 2616-sec9: HTTP Method definitions: http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html

URI Length Limits

Request Message

The request message consists of the following:

The request line and headers must all end with <CR><LF> (that is, a carriage return character followed by a line feed character (\r\n)). The empty line must consist of only <CR><LF> and no other whitespace. In the HTTP/1.1 protocol, all headers except Host are optional.

A request line containing only the path name is accepted by servers to maintain compatibility with HTTP clients before the HTTP/1.0 specification in RFC1945.

RFC 2616-sec5: HTTP Request: http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html

Response Message

The response message consists of the following:

The Status-Line and headers must all end with <CR><LF> (a carriage return followed by a line feed). The empty line must consist of only <CR><LF> and no other whitespace.

RFC 2616-sec6: HTTP Response: http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html

The Status-Code element is a 3-digit integer result code of the attempt to understand and satisfy the request. The Reason-Phrase is intended to give a short textual description of the Status-Code. The Status-Code is intended for use by automata and the Reason-Phrase is intended for the human user. The client is not required to examine or display the Reason- Phrase.

The first digit of the Status-Code defines the class of response. The last two digits do not have any categorization role. There are 5 values for the first digit:

Common Status Codes

RFC 2616-sec10 HTTP Status Code Definitions: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

Header Fields

HTTP header fields are components of the message header of requests and responses in the Hypertext Transfer Protocol (HTTP). They define the operating parameters of an HTTP transaction.

The header fields are transmitted after the request or response line, the first line of a message. Header fields are colon-separated name-value pairs in clear-text string format, terminated by a carriage return (CR) and line feed (LF) character sequence. The end of the header fields is indicated by an empty field, resulting in the transmission of two consecutive CR-LF pairs. Long lines can be folded into multiple lines; continuation lines are indicated by presence of space (SP) or horizontal tab (HT) as first character on the next line. A few fields can also contain comments (i.e. in. User-Agent, Server, Via fields), which can be ignored by software.

Request Headers

Response headers

RFC 2616-sec14: Header Field Definitions: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

An Internet media type is a two-part identifier for file formats on the Internet. The identifiers were originally defined in RFC 2046 for use in email sent through SMTP, but their use has expanded to other protocols such as HTTP. These types were called MIME types, and are sometimes referred to as Content-types, after the name of a header in several protocols whose value is such a type.

A media type is composed of two or more parts: A type, a subtype, and zero or more optional parameters. For example, subtypes of text have an optional charset parameter that can be included to indicate the character encoding (e.g. text/html; charset=UTF-8).

Common application MIME Types

Common multipart MIME Types

Common text MIME Types

Cookies

A cookie, also known as an HTTP cookie, web cookie, or browser cookie, is usually a small piece of data sent from a website and stored in a user's web browser while a user is browsing a website. When the user browses the same website in the future, the data stored in the cookie can be retrieved by the website to notify the website of the user's previous activity.Cookies were designed to be a reliable mechanism for websites to remember the state of the website or activity the user had taken in the past. This can include clicking particular buttons, logging in, or a record of which pages were visited by the user even months or years ago.

SERVER SETTING COOKIES IN RESPONSE:

Set-Cookie: name=value

Set-Cookie: name2=value2; Expires=Tue, 26 Jun 2012 19:19:47 GMT

BROWSER SENDING COOKIES TO SERVER IN REQUEST:

Cookie: name=value; name2=value2

Data URI's

Data URL scheme can be useful for embedding images into HTML/CSS/JS to save on HTTP requests instead of referencing remote files.

STANDARD <IMG> TAG:

<img width="99" height="99" alt="BASE Logic, Inc. logo" src="http://baselogic.

com/images/BLiNC_logo.png" />

DATA URI IMAGE IN <IMG> TAG:

<img width="99" height="99" alt="BASE Logic, Inc. logo" src="data:image/

png;base64,R0lGODlhEAAQAMQAAORHHOVSKudfOulrSOp3WOyDZu6QdvCchPGolfO0o/XBs/

fNwfjZ0frl3/zy7///

Avoiding caching

If a web server responds with Cache-Control: no-cache then a web browser or other caching system must not use the response to satisfy subsequent responses without first checking with the originating server. This header field is part of HTTP version 1.1, and is ignored by some caches and browsers. It may be simulated by setting the Expires HTTP version 1.0 header field value to a time earlier than the response time.

The request that a resource should not be cached is no guarantee that it will not be written to disk. In particular, the HTTP/1.1 definition draws a distinction between history stores and caches. If the user navigates back to a previous page a browser may still show you a page that has been stored on disk in the history store. This is correct behavior according to the specification. Many user agents show different behavior in loading pages from the history store or cache depending on whether the protocol is HTTP or HTTPS.

The header field Cache-Control: no-store is intended to instruct a browser application to make a best effort not to write it to disk. The Pragma: no-cache header field is an HTTP/1.0 header intended for use in requests. It is a means for the browser to tell the server and any intermediate caches that it wants a fresh version of the resource, not for the server to tell the browser not to cache the resource. Some user agents do pay attention to this header in responses, but the HTTP/1.1 RFC specifically warns against relying on this behavior.

RFC 2616-sec13: Caching in HTTP: http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html

HTTP Conditional GET

A conditional GET is an HTTP GET request that returns an HTTP 304 response (versus HTTP 200). An HTTP 304 response indicates that the resource has not been modified since the previous GET request and the resource will not be returned to the requesting client as part of the response.

BASIC SYNTAX:

[Last-Modified / If-Modified-Since]

[ETag / If-None-Match]

CLIENT REQUEST:

[MickKnutson]$ curl --silent --head --header 'If-Modified-Since: Tue, 19 Jun

2012 10:10:10 GMT' http://baselogic.com/images/BLiNC_logo.png

SERVER RESPONSE HTTP 304:

HTTP/1.1 304 Not Modified

Date: Tue, 26 Jun 2012 19:13:29 GMT

Server: Apache/2.2.21 (Unix)

Connection: close

Expires: Wed, 27 Jun 2012 19:13:29 GMT

Cache-Control: max-age=86400

transfer Layer Security (TLS / SSL)

transfer Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL), are cryptographic protocols that provide communication security over the Internet. TLS and SSL encrypt the segments of network connections at the Application Layer for the transfer Layer, using asymmetric cryptography for key exchange, symmetric encryption for privacy, and message authentication codes for message integrity.

RFC 5246 transfer Layer Security (TLS) Protocol v1.2: http://tools.ietf.org/html/rfc5246

Basic access Authentication (BASIC)

In the context of an HTTP transaction, basic access authentication is a method for a web browser or other client program to provide a user name and password when making a request. This is the most basic way of implementing authentication for a web application and is suitable when we are accessing the application both using browser and other software such as scripts and so on. In this mode, when accessed by a browser, the browser will use its standard dialog to collect the credentials. It is easy to implement, but the credentials will be transmitted as plain text and anyone can collect them if we do not have TLS/SSL or some network level encryption in place.

CLIENT REQUEST (NO AUTHENTICATION):

GET /secured/index.html HTTP/1.1

Host: localhost

SERVER RESPONSE:

HTTP/1.1 401 Authorization Required

Server: Apache

Date: Tue, 26 Jun 2012 19:19:47 GMT

WWW-Authenticate: Basic realm="Secure Area"

Content-Type: text/html

Content-Length: 311

​

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"

 "http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd">

<HTML>

  <HEAD>

    <TITLE>Error</TITLE>

    <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1">

  </HEAD>

  <BODY><H1>401 Unauthorized.</H1></BODY>

</HTML>

​

Before transmitting the username and password entered by the user, the two are concatenated with a colon separating the two values; the resulting string is Base64 encoded. For example, given a username mick and password secret key, the string "mick:secret key" will be encoded with the Base64 algorithm resulting in bWljazpzZWNyZXQga2V5. The Base64- encoded string is transmitted in the HTTP header and decoded by the receiver, resulting in the decoded colon-separated username and password String. Encoding the username and password with the Base64 makes them unreadable visually, but they are easily decoded. Confidentiality is not the intent of the encoding step, rather, the intent is to encode non-HTTPcompatible characters that a username or password may contain, into those that are HTTP-compatible.

CLIENT REQUEST "MICK:SECRET KEY" (USER NAME "MICK", PASSWORD "SECRET KEY"):

GET /secured/index.html HTTP/1.1

Host: localhost

Authorization: Basic bWljazpzZWNyZXQga2V5

SERVER RESPONSE:

HTTP/1.1 200 OK

Server: Apache

Date: Tue, 26 Jun 2012 19:19:47 GMT

Content-Type: text/html

Content-Length: 10476

Digest access authentication (DIGEST)

This method is similar to the BASIC authentication method, but instead of the password a digest of the password is transmitted.

Digest communication starts with a client that requests a resource from a web server. If the resource is secured with Digest Authentication, the server will respond with the http status code 401, which means Unauthorized to access this resource.

CLIENT REQUEST (NO AUTHENTICATION):

GET /dir/index.html HTTP/1.0

Host: localhost

In the response from the initial request, the server indicates in the HTTP header with which mechanism the resource is secured.

SERVER RESPONSE:

HTTP/1.0 401 Unauthorized

Server: Apache

Date: Tue, 26 Jun 2012 19:19:47 GMT

WWW-Authenticate: Digest realm="baselogic.com",

         qop="auth",

         nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093",

         opaque="5ccc069c403ebaf9f0171e9517f40e41"

Content-Type: text/html

Content-Length: 311

​

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"

 "http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd">

<HTML>

  <HEAD>

    <TITLE>Error</TITLE>

    <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1">

  </HEAD>

  <BODY><H1>401 Unauthorized.</H1></BODY>

</HTML>

Now the user can enter the credentials using mick as the username and secret key as the password.

CLIENT REQUEST (USERNAME "MICK", PASSWORD "SECRET KEY"):

GET /secured/index.html HTTP/1.0

Host: localhost

Authorization: Digest username="mick",

                     realm="baselogic.com",

                     nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093",

                     uri="/secured/index.html",

                     qop=auth,

                     nc=00000001,

                     cnonce="0a4f113b",

                     response="35f308904a9a9623498f358d1cb10afd"

SERVER RESPONSE:

HTTP/1.0 200 OK

Server: Apache

Date: Tue, 26 Jun 2012 20:19:47 GMT

Content-Type: text/html

Content-Length: 7984

​

You should notice the term Digest in the response which indicates that the resource requested by the client is secured using Digest Authentication. The server also indicates the type of Digest Authentication algorithm used by the client with Quality Of Protection (QOP) and the nonce string, which is a Base64 encoded timestamp and private hash generated by the server.

String nonce = Base64.encode(new Timestamp() : "Private MD5 Hash")

The private hash is created by the server, and the Base64 encoding allows for decoding of the timestamp and private hash, even though the private MD5 hash is a one-way encryption.

An internet browser responds to this by presenting the user a dialog, in this dialog the user is able to enter a username and password for credentials. The dialog does not show the warning about transmitting the credentials in clear text as with a Basic Authentication secured site.

The response is generated by several digest properties sent from the client, with the addition to an HA1 and HA2 value concatenated together, then MD5 hash encrypted. The algorithm is the following:

Response = MD5( "HA1:\

                nonce:\

                nc:\

                cnonce:\

                qop:\

                HA2" )

The HA1 hash is the username, realm, and password separated by colons.

HA1 = MD5( "mick:baselogic.com:secret key")

The MD5 hash for the HA1 is: 935f1e7b1582ffd6e05e7dc8e949ac6f

The HA2 hash is the initial HTTP GET request made to the server.

HA2 = MD5( "GET:/secured/index.html" )

The MD5 hash for the HA2 is: cc21ab6caf04c32228f0250c9eb48705

The final response MD5 hash algorithm would look like this:

Response = MD5(                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       935f1e7b1582ffd6e05e7dc8e949ac6f:\

dcd98b7102dd2f0e8b11d0f600bfb0c093:\

00000001:0a4f113b:auth:\                

cc21ab6caf04c32228f0250c9eb48705")

This would result in 35f308904a9a9623498f358d1cb10afd which is the value for the response to the client sent to the server for authentication.

Listing: Submitting DIGEST authentication credentials.

Form-Based Authentication (FORM)

In this mode, we can use our own form to collect the username and password. It is very flexible in terms of implementation and how we ask for the username and password, but requires extra work for implementing the forms.

This method suffers from the same security risk as the BASIC method because the credentials are transmitted as plain text.

This is however, the most user friendly type of authentication as it allows site owners to control the look-and-feel over the user experience during site navigation.

<form method="post" action="/secured/login">

        <input type="text" name="username" required>

        <input type="password" name="password" required>

        <input type="submit" value="Login">

</form>

BASE Logic, Inc: http://baselogic.com

Tcp: http://en.wikipedia.org/wiki/Transmission_Control_Protocol

RFC 793 TCP Connection states: http://tools.ietf.org/html/rfc793

RFC 2397 The "data" URL scheme: http://tools.ietf.org/html/rfc2397

RFC 2459 Internet X.509 Public Key Infrastructure: http://tools.ietf.org/html/rfc2459

RFC 4918: HTTP Extensions for Web Distributed Authoring and Versioning (WebDAV): http://tools.ietf.org/html/rfc4918

cURL: http://en.wikipedia.org/wiki/CURL