Understanding the HTTP 431 Error: A Developer's Guide

The 431 HTTP status code, also known as "Request Header Fields Too Large," is a client-side error that indicates the server refuses to process a request because its header fields are too large. This error can be caused by either the total size of the headers or a single header field exceeding the allowed size.

The HTTP 431 status code was first introduced in RFC 6585, published in April 2012. This was part of an effort to standardize additional responses for HTTP/1.1, addressing issues that were not covered by the existing status codes at the time. Introducing this specific error code was crucial for enhancing server security and preventing malicious attacks like buffer overflow attacks, which could exploit large headers.

Understanding the HTTP 431 Error

When a 431 error occurs, it means that the server has blocked the client's request due to excessively large header fields. Each server has its own policies regarding the maximum allowed size for HTTP header fields, which helps protect against denial-of-service attacks.

Common Symptoms of HTTP 431 Errors

• The total size of request headers is too large: This error occurs when the cumulative size of all request headers exceeds the limit.
• The size of individual header fields is too large: In this case, a single header field is too large to process, leading to the 431 error.

Common Causes of HTTP 431 Errors

• Too many cookies: Having excessively large or many cookies can cause the 431 error. This can happen if the same cookies are set multiple times or if multiple cookies are combined in a single request or response. One common scenario is reusing the same localhost port across multiple projects and sending all of those projects' cookies along with a request.
• Improper header field caching: Caching large headers incorrectly can lead to the 431 error. Improper caching can increase the header size, causing a buffer overflow.
• Unnecessary or poorly formatted request headers: Overusing HTTP headers can result in requests that exceed the size limits set by the server, leading to a 431 response. Additionally, ensure your headers are properly formatted. Having a Referer URL that's too long can trigger a 431.

How to Avoid the 431 Error

Clear Cookies

Regularly clear cookies on the client side (Chrome dev tool > Application > Cookies) and restrict the number of cookies used on the server.

Cache Headers

Implement header caching correctly to ensure the cached headers reflect the actual request and compression response. Adjust ETags if you rely heavily on them for conditional requests, as large ETags can contribute to the error.

Minimize HTTP Headers

Simplify your HTTP headers to include only necessary information. Avoid including redundant metadata, especially in headers like Authorization, Referer, and User-Agent.

Compressing the Header Size

Sometimes, you aren't sending enough headers and your headers are bloated due to constraints out of your control (e.g., Your JWT token encodes a ton of information). Header compression can significantly reduce the size of your request headers. Utilize protocols like HTTP/2 and HTTP/3, which support header compression by default. This can automatically reduce the size of your headers without requiring additional configuration.

Dealing With Server Max Header Size Limits

The HTTP spec does not define a limit on header size; however, many servers do. Here are the limits for various popular web servers/hosts:

• Apache: 8K
• nginx: 4K-8K
• IIS: 8K-16K
• Tomcat: 8K-48K
• Node: (<13) - 8K; (>13) - 16K
• Cloudflare: 16K per header, 32K total

Most servers allow some form of configuration, and different versions of the server software might have lower or higher limits. You should also check the latest documentation to see if these limits include other parts of the request (e.g., cookies) or just the regular headers.

Increasing the Request Header Size Limit

Sometimes, increasing the request header size limit may be necessary. You can typically do this from the web console or CLI for your server. If you are developing locally, there are typically CLI flags that can configure this value. Here's the flag to configure request header size limits in Node JS:

--max-http-header-size=16384

Note: Increasing the header size limit should be done with caution, as larger headers can consume more memory and degrade performance.

How to Send HTTP 431 Error Responses

You might not want to rely on your server to determine your request header size limit, or maybe you want to enforce a custom header size limit per API endpoint. Follow the steps below to learn how to send your own 431 errors.

Building Request Header Size Limits Into Your API

As mentioned above, your host + server combination will likely automatically enforce a limit. To build your own custom request header size limit capability in your API, you can use libraries like express in Node.js. Here is an example illustrating how to enable limits on request header sizes:

const express = require("express");
const app = express();
app.get("/", (req, res) => {
  const headerSizeInBytes = JSON.stringify(req.headers).length;
  const maxHeaderSizeInBytes = 2000; // example limit
  if (headerSizeInBytes > maxHeaderSizeInBytes) {
    res.status(431).send("Request Header Fields Too Large");
  } else {
    res.send("Hello World!");
  }
});
app.listen(3000, () => {
  console.log("Server is running on port 3000");
});

Differentiating Error Responses Between the Total Size of Request Headers and a Single Header Field

When handling the HTTP 431 error, make sure to differentiate responses between two scenarios:

1. The total size of request headers is too large: Return a response indicating that the cumulative size of the headers is too large.
2. The size of the individual header fields is too large: In this case, provide an error response that states which specific header field exceeds the allowable size.

Modify the example from the previous section to achieve differentiated error responses:

const express = require("express");
const app = express();
app.get("/", (req, res) => {
  const headerSizeInBytes = JSON.stringify(req.headers).length;
  const maxHeaderSizeInBytes = 2000; // example limit
  const exceededHeaderField = Object.keys(req.headers).find(
    (key) => req.headers[key].length > maxHeaderSizeInBytes * 0.1, // example individual field limit
  );
   if (exceededHeaderField) {
    res
      .status(431)
      .send(
        `Size of Individual Header Field '${exceededHeaderField}' Too Large`,
      );
  } else if (headerSizeInBytes > maxHeaderSizeInBytes) {
    res.status(431).send("Total Size of Request Headers Too Large");
  } else else {
    res.send("Hello World!");
  }
});
app.listen(3000, () => {
  console.log("Server is running on port 3000");
});

Example of an HTTP 431 Response

We recommend responding to your users using the Problem Details format API response.

HTTP/1.1 431 Request Header Fields Too Large
Content-Type: application/problem+json
Content-Language: en
 
{
  "type": "https://httpproblems.com/http-status/431",
  "title": "Request Header Fields Too Large",
  "detail": "Size of individual header field 'referer' too large",
  "instance": "/account/12345/msgs/abc",
  "trace": {
    "requestId": "4d54e4ee-c003-4d75-aba9-e09a6d707b08"
  }
}

Letting Your Gateway Handle Request Header Size Limits

If you use an API gateway, you can easily add a policy to your request pipeline to handle this. Here's how to do it using Zuplo:

Navigate to your route in the Route Designer and click Add Policy on the request pipeline.

In the Choose Policy modal, we have two options depending on what you want to do.

1. Limit the entire request size with the Request Size Limit Policy.
2. Use the Custom Code Inbound Policy and copy the code sample from above (minus the express bits) into a typescript module that will be connected to the policy.

Conclusion

The HTTP 431 error is often triggered by overly large request headers. You can avoid encountering this error by optimizing your headers, compressing them as needed, and implementing request header size limits in your APIs.

Additionally, implementing header size limit checks in your API is simple. Most servers already include defaults, and you can also do it yourself at both the cumulative and individual header levels from within your API routes.